Evidence-Driven Simulated Agents to Train Reinforcement Learning for Personalization in mHealth Interventions
Author(s)
Juan Carlos Caro, Ph.D Health Policy and Management, Giorgio Galgano, B.S. Industrial Engineering.
Universidad de Concepcion, Concepcion, Chile.
Universidad de Concepcion, Concepcion, Chile.
OBJECTIVES: Reinforcement learning allows for contextual personalization in mHealth interventions, particularly key for behavior change. Initial training requires novel data for new users, facing the cold-start problem. This study evaluates the training performance of simulated agents, generated with calibrated parameters from empirical studies framed under the Integrated Behavioral Change Model for physical activity. We focus on difficulty personalization, an important feature for interventions targeting behavior adherence.
METHODS: Based on the Integrated Behavioral Change Model, metadata was compiled from available studies from 2011 to 2024 in public access repositories, extracting sample statistics. The metadata allows to parametrize a joint distribution for the key contextual variables: psychological constructs (Likert-type scales) and demographic characteristics. The simulation structure allows to simultaneously generate agents, as well as predict their behavioral outcome (model reward), using the contextual variables and personalization feature (exercise difficulty), with three levels (easy, medium, hard). To evaluate performance, we compare the training results from offline policy algorithms using simulation versus real data from a validation pilot.
RESULTS: The metadata collected was sufficient to parametrize a joint distribution consistent with the Integrated Behavioral Change Model. The simulated users were randomly allocated to each difficulty level, consistent with the protocol from real data available, matching average reward to each difficulty level. In both simulated and real data, the results from the offline policy algorithms showed important differences between random and optimal difficulty allocation, consistent with the psychological constructs in the model. Training performance showed a similar average loss in both datasets.
CONCLUSIONS: Population-based simulations using empirical results from validated behavioral theories constitutes a viable and efficient solution to generate synthetic agent data to address the cold start problem. Evidence-driven simulated data reduces significantly initial implementation costs of reinforcement learning in personalization for mHealth, facilitating the scalability and accessibility of digital personalized interventions for behavioral change.
METHODS: Based on the Integrated Behavioral Change Model, metadata was compiled from available studies from 2011 to 2024 in public access repositories, extracting sample statistics. The metadata allows to parametrize a joint distribution for the key contextual variables: psychological constructs (Likert-type scales) and demographic characteristics. The simulation structure allows to simultaneously generate agents, as well as predict their behavioral outcome (model reward), using the contextual variables and personalization feature (exercise difficulty), with three levels (easy, medium, hard). To evaluate performance, we compare the training results from offline policy algorithms using simulation versus real data from a validation pilot.
RESULTS: The metadata collected was sufficient to parametrize a joint distribution consistent with the Integrated Behavioral Change Model. The simulated users were randomly allocated to each difficulty level, consistent with the protocol from real data available, matching average reward to each difficulty level. In both simulated and real data, the results from the offline policy algorithms showed important differences between random and optimal difficulty allocation, consistent with the psychological constructs in the model. Training performance showed a similar average loss in both datasets.
CONCLUSIONS: Population-based simulations using empirical results from validated behavioral theories constitutes a viable and efficient solution to generate synthetic agent data to address the cold start problem. Evidence-driven simulated data reduces significantly initial implementation costs of reinforcement learning in personalization for mHealth, facilitating the scalability and accessibility of digital personalized interventions for behavioral change.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
EPH96
Topic
Epidemiology & Public Health, Health Service Delivery & Process of Care, Health Technology Assessment
Topic Subcategory
Public Health
Disease
No Additional Disease & Conditions/Specialized Treatment Areas, Personalized & Precision Medicine