Presentation (CTI)

OBJECTIVES: Sensitivity analysis is an important approach for assessing the “robustness” of observational study results. However, how sensitivity analyses are performed and the extent to which the results and interpretations differ between sensitivity analyses and the primary analysis remain poorly understood.
METHODS: We systematically searched observational studies for assessing treatment effects of drugs (2018-2020) from PubMed core clinical journals. Information on sensitivity analyses was extracted using pilot-tested, standardized data collection forms. Characteristics of sensitivity analyses were summarized, and treatment effects estimated from primary and sensitivity analyses were compared. Multivariable logistic regression was used to examine the association between study characteristics and the consistency between the results of the primary analysis and the sensitivity analyses.
RESULTS: Of the 256 included studies, 152 (59.4%) studies performed sensitivity analyses (median:3, IQR: 2 to 6), and 131 (51.2%) clearly reported the results. Among these 131 studies, 71 (54.2%) studies showed significant differences between the primary analyses and sensitivity analyses, with an average 24% effect size difference (95% CI 12% to 35%). Within the 71 studies, 145 sensitivity analyses with alternative study definitions (n=59), alternative study designs (n=39), and alternative statistical models (n=38) showed significant inconsistencies. Only 9 of the 71 studies discussed the implications of inconsistencies, while 62 dismissed or ignored them. Heterogeneity of results was more likely in studies with ≥3 sensitivity analyses, a small effect size (ratio 0.5-2; standardized difference ≤3), or the use of blank controls.
CONCLUSIONS: There are significant differences between primary and sensitivity analyses in observational studies. However, there has been no comprehensive assessment and management of these divergences. There is an urgent need for improved practice in the conduct of sensitivity analyses and the management of their results, especially when they are inconsistent with the results of the primary analysis.