Beyond Leave-One-Out: A Ranked Leave-k-Out Method for Detecting Influential Studies
Author(s)
Neha Tripathi, MPH1, Akanksha Sharma, MSc1, Parampal Bajaj, BTech1, Supreet Kaur, MSc1, Shubhram Pandey, MSc2.
1Heorlytics Pvt. Ltd., Mohali, India, 2Heorlytics Pvt. Ltd., SAS Nagar, Mohali, India.
1Heorlytics Pvt. Ltd., Mohali, India, 2Heorlytics Pvt. Ltd., SAS Nagar, Mohali, India.
OBJECTIVES: Meta-analytic results can be susceptible to the inclusion of influential studies. Although traditional leave-one-out (L1O) diagnostics effectively assess individual study impact, they do not account for combined effect of multiple studies.This analysis presents a ranked combinatorial sensitivity framework that systematically excludes studies—individually, in pairs, and in triplets—based on outlier rankings to better capture their cumulative influence.
METHODS: A random-effects meta-analysis was conducted in R using the metafor package, with the Restricted Maximum Likelihood (REML) method for estimating between-study variances. To assess study influence, standardized residuals and Cook's distance were calculated. Studies were then ranked based on these diagnostics, and sensitivity analyses were performed by systematically excluding one (L1O), two (L2O), and three (L3O) studies at a time. For each exclusion scenario, pooled effect estimates were recalculated to evaluate the stability of findings. For instance, in a case example from rheumatoid arthritis trials assessing the efficacy of biologic therapies, this approach revealed that a small cluster of studies disproportionately influenced the overall treatment effect
RESULTS: The L1O analysis showed that removing the top-ranked outlier increased the pooled effect estimate to 0.4371, while excluding the second-ranked outlier reduced it slightly to 0.3899. In the L2O analysis, excluding top-ranked pairs led to a wider range of effect estimates, from 0.3708 to 0.4623. The L3O analysis, involving the exclusion of the most influential triads, produced an even broader range (0.3493 to 0.4898), underscoring the synergistic impact of multiple high-leverage studies
CONCLUSIONS: The ranked Leave-k-Out framework enhances traditional sensitivity analysis by incorporating outlier-driven prioritization, allowing for the detection of both individual and combined study influences on meta-analytic estimates. This approach offers a robust and reproducible method for evaluating the reliability of evidence in the presence of heterogeneity and potential study-level bias
METHODS: A random-effects meta-analysis was conducted in R using the metafor package, with the Restricted Maximum Likelihood (REML) method for estimating between-study variances. To assess study influence, standardized residuals and Cook's distance were calculated. Studies were then ranked based on these diagnostics, and sensitivity analyses were performed by systematically excluding one (L1O), two (L2O), and three (L3O) studies at a time. For each exclusion scenario, pooled effect estimates were recalculated to evaluate the stability of findings. For instance, in a case example from rheumatoid arthritis trials assessing the efficacy of biologic therapies, this approach revealed that a small cluster of studies disproportionately influenced the overall treatment effect
RESULTS: The L1O analysis showed that removing the top-ranked outlier increased the pooled effect estimate to 0.4371, while excluding the second-ranked outlier reduced it slightly to 0.3899. In the L2O analysis, excluding top-ranked pairs led to a wider range of effect estimates, from 0.3708 to 0.4623. The L3O analysis, involving the exclusion of the most influential triads, produced an even broader range (0.3493 to 0.4898), underscoring the synergistic impact of multiple high-leverage studies
CONCLUSIONS: The ranked Leave-k-Out framework enhances traditional sensitivity analysis by incorporating outlier-driven prioritization, allowing for the detection of both individual and combined study influences on meta-analytic estimates. This approach offers a robust and reproducible method for evaluating the reliability of evidence in the presence of heterogeneity and potential study-level bias
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
SA16
Topic
Study Approaches
Topic Subcategory
Meta-Analysis & Indirect Comparisons
Disease
No Additional Disease & Conditions/Specialized Treatment Areas