Sizing Up the Gap: Evaluating Sample Size Justification in Prospective Real-World Studies
Author(s)
Sarah Bandy, PharmD, PhD1, Jaymin Patel, PharmD2, Daniel Sheinson, PhD3, Danny Yeh, PhD1;
1AESARA, Value Evidence, Durham, NC, USA, 2AESARA, Durham, NC, USA, 3Genentech, San Francisco, CA, USA
1AESARA, Value Evidence, Durham, NC, USA, 2AESARA, Durham, NC, USA, 3Genentech, San Francisco, CA, USA
Presentation Documents
OBJECTIVES: Sample size calculation is not only important for clinical trials but also for prospective real-world data (RWD) studies testing hypotheses. The goal of this study is to review whether recent prospective RWD studies published in high impact journals described their methods to determine sample size.
METHODS: We conducted a search of PubMed to identify prospective RWD studies in full-text articles that were published in the top 5 highest impact factor general medicine journals (New England Journal of Medicine, Lancet, JAMA, Nature Medicine, Annals of Internal Medicine) and Value in Health from January 1 - December 31, 2023. Prospective RWD studies included cross-sectional or longitudinal cohort studies and case-series in which hypotheses were tested going forward in time.
RESULTS: A total of 374 papers were identified. After title/abstract review, 62 were selected for full-text review, and 24 were included. Data sources included patients from medical centers/clinics (n=14; n=3 single-center), registries (n=2), community (n=5), and surveys (n=3).
Most studies (15/24, 63%) did not include a description of methods on sample size calculation. Of the 9 studies that did, 7 used prior real-world studies, 6 leveraged author/expert opinions, and 5 referred to clinical trials to inform rationale (some studies referenced multiple sources). Two studies increased the number of enrollees after calculations revealed a shortfall in the number required for a well-powered study. None of the reviewed studies discussed considerations of overfitting issues for future regression adjustment in their sample size calculation.
CONCLUSIONS: Among recent prospective RWD studies identified in top-tiered journals, providing rationale for sample size is not common. Among those that did, accounting for sample size shortfall was rare, leading studies to potentially be underpowered to explore their objectives.
METHODS: We conducted a search of PubMed to identify prospective RWD studies in full-text articles that were published in the top 5 highest impact factor general medicine journals (New England Journal of Medicine, Lancet, JAMA, Nature Medicine, Annals of Internal Medicine) and Value in Health from January 1 - December 31, 2023. Prospective RWD studies included cross-sectional or longitudinal cohort studies and case-series in which hypotheses were tested going forward in time.
RESULTS: A total of 374 papers were identified. After title/abstract review, 62 were selected for full-text review, and 24 were included. Data sources included patients from medical centers/clinics (n=14; n=3 single-center), registries (n=2), community (n=5), and surveys (n=3).
Most studies (15/24, 63%) did not include a description of methods on sample size calculation. Of the 9 studies that did, 7 used prior real-world studies, 6 leveraged author/expert opinions, and 5 referred to clinical trials to inform rationale (some studies referenced multiple sources). Two studies increased the number of enrollees after calculations revealed a shortfall in the number required for a well-powered study. None of the reviewed studies discussed considerations of overfitting issues for future regression adjustment in their sample size calculation.
CONCLUSIONS: Among recent prospective RWD studies identified in top-tiered journals, providing rationale for sample size is not common. Among those that did, accounting for sample size shortfall was rare, leading studies to potentially be underpowered to explore their objectives.
Conference/Value in Health Info
2025-05, ISPOR 2025, Montréal, Quebec, CA
Value in Health, Volume 28, Issue S1
Code
MSR95
Topic
Methodological & Statistical Research
Topic Subcategory
Confounding, Selection Bias Correction, Causal Inference
Disease
No Additional Disease & Conditions/Specialized Treatment Areas