OVERDOSE EARLY WARNING SYSTEMS: KEY LIMITATIONS AND POTENTIAL ALTERNATIVE APPROACHES
Author(s)
Joel Segel1, Qiushi Chen, PhD2, Holly Nguyen, PhD2, Glenn Sterner, PhD3, Yifang Yan, MS2, Paul Griffin, PhD2.
1Associate Professor, University Park, PA, USA, 2Penn State University, University Park, PA, USA, 3University of Kentucky, Lexington, KY, USA.
1Associate Professor, University Park, PA, USA, 2Penn State University, University Park, PA, USA, 3University of Kentucky, Lexington, KY, USA.
OBJECTIVES: Policymakers and public health practitioners have a strong interest in creating early warning systems (EWS) to identify emerging opioid and other substance-related threats to guide interventions. While some national or regional surveillance models exist, many state and local governments desire predictive models for their local jurisdictions. However, there are a number of inherent difficulties in developing accurate prediction models at this level of granularity.
METHODS: Our primary predictors included opioid and other substance-related overdose deaths, overdose-related emergency department visits, overdose response incidents, drug arrests, and opioid prescriptions in Pennsylvania at the county level and quarter level for 2018-2022. To identify the potential need for intervention, we defined an overdose death “spike” if the number of overdose deaths from any drug was greater than one standard deviation above the past-year average and increased from the previous quarter, although varied in sensitivity analyses. We estimated multiple prediction models using two primary types of predictive methodologies: (1) supervised learning using machine learning models (e.g., random forest, gradient boost, spatial neural networks) and (2) early detection algorithms based on statistical modeling (e.g., Farrington algorithm, CUSUM). For machine learning models, data were split into the first three years for training and the last two years for testing.
RESULTS: Our computational results showed only moderate prediction performance across methods, input features, and model configurations. Machine learning models showed an AUROC up to 0.72, and historical overdose deaths contributed the most to the models’ predictive value. Statistical early detection algorithms achieved sensitivity of 0.36-0.48 while maintaining a high specificity of 0.96-0.97.
CONCLUSIONS: Across models, we find it very difficult to identify overdose spikes. This highlights broader concerns about a local-level EWS, namely, since data are often not granular enough, rare outcomes such as overdoses are hard to predict, and EWS require a plan for responding to a spike if detected.
METHODS: Our primary predictors included opioid and other substance-related overdose deaths, overdose-related emergency department visits, overdose response incidents, drug arrests, and opioid prescriptions in Pennsylvania at the county level and quarter level for 2018-2022. To identify the potential need for intervention, we defined an overdose death “spike” if the number of overdose deaths from any drug was greater than one standard deviation above the past-year average and increased from the previous quarter, although varied in sensitivity analyses. We estimated multiple prediction models using two primary types of predictive methodologies: (1) supervised learning using machine learning models (e.g., random forest, gradient boost, spatial neural networks) and (2) early detection algorithms based on statistical modeling (e.g., Farrington algorithm, CUSUM). For machine learning models, data were split into the first three years for training and the last two years for testing.
RESULTS: Our computational results showed only moderate prediction performance across methods, input features, and model configurations. Machine learning models showed an AUROC up to 0.72, and historical overdose deaths contributed the most to the models’ predictive value. Statistical early detection algorithms achieved sensitivity of 0.36-0.48 while maintaining a high specificity of 0.96-0.97.
CONCLUSIONS: Across models, we find it very difficult to identify overdose spikes. This highlights broader concerns about a local-level EWS, namely, since data are often not granular enough, rare outcomes such as overdoses are hard to predict, and EWS require a plan for responding to a spike if detected.
Conference/Value in Health Info
2026-11, ISPOR Europe 2026, Vienna, Austria
Value in Health, Volume 29, Issue 12S
Code
EPH15
Topic
Epidemiology & Public Health, Real World Data & Information Systems, Study Approaches
Topic Subcategory
Public Health
Disease
Mental Health (including addiction)