EVALUATING AI-ASSISTED TIME-AWARE EVIDENCE MAPPING FOR HEALTH TECHNOLOGY ASSESSMENT: A PILOT REVIEWER AUDIT OF TRACEABILITY, TEMPORAL VALIDITY, AND JURISDICTIONAL ALIGNMENT
Author(s)
Annabelle I. Day, Master's Student1, Yi-Ting Jonathan Lin, PhD Candidate2, Chun-Wei Hsu, PhD Candidate1, Kuan-Ju Wang, PhD3, Chao-Hsin Michael Ding, MS4, Wei-ming Huang, PharmD, MD5, Ching-Po Lin, PhD1.
1Institute of Neuroscience, National Yang Ming Chiao Tung University, Taipei, Taiwan, 2Ph.D. Program of Interdisciplinary Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan, 3PanCAD.ai Co., Ltd., Taipei, Taiwan, 4Department of Education and Research, Taipei City Hospital, Taipei, Taiwan, 5Institute of Health and Welfare Policy, National Yang Ming Chiao Tung University, Taipei, Taiwan.
1Institute of Neuroscience, National Yang Ming Chiao Tung University, Taipei, Taiwan, 2Ph.D. Program of Interdisciplinary Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan, 3PanCAD.ai Co., Ltd., Taipei, Taiwan, 4Department of Education and Research, Taipei City Hospital, Taipei, Taiwan, 5Institute of Health and Welfare Policy, National Yang Ming Chiao Tung University, Taipei, Taiwan.
OBJECTIVES: HTA review requires evidence claims that are accurate, source-traceable, temporally valid, and aligned with jurisdiction-specific HTA and reimbursement requirements. In this pilot reviewer audit study, we developed and evaluated a temporally anchored, AI-assisted evidence-mapping workflow for generating auditable HTA evidence fields. The evaluation focused on source traceability, temporal validity, jurisdictional alignment, reviewer agreement, correction burden, and error patterns.
METHODS: We selected eight high-cost oncology medicines with publicly available regulatory, clinical, and HTA evidence. The workflow extracted structured fields covering indication, population, comparators, trials, endpoints, safety, regulatory/HTA status, reimbursement restrictions, and uncertainty. Time-aware mapping was implemented through a temporal evidence graph linking each statement to source type, document date, page/location, domain, and jurisdiction, enabling assessment at a prespecified index date. Two trained reviewers independently audited 288 fields against source documents using predefined criteria. Jurisdictional alignment was assessed against relevant HTA decision contexts.
RESULTS: Of 288 fields, 91.7% were traceable to correct sources and 86.8% were temporally valid. Jurisdictional alignment was achieved in 82.6%; non-aligned fields most often involved comparator selection, reimbursement restrictions, or outdated HTA wording. Reviewer agreement was substantial (κ=0.78). Major and minor corrections were required for 8.3% and 21.5% of fields, respectively. Among identified errors, missing temporal context (31.0%), oversimplified HTA judgments (26.4%), numerical extraction errors (18.1%), source mismatch (14.8%), and other errors (9.7%) were observed. Median generation time was 11.4 minutes per medicine, excluding reviewer audit.
CONCLUSIONS: AI-assisted time-aware evidence mapping produced structured HTA evidence fields with high source traceability, temporal validity, and jurisdictional alignment, supporting auditable evidence preparation and potential efficiency gains. Evaluation of AI-assisted HTA evidence should extend beyond factual accuracy to include these dimensions and reviewer correction burden. Human oversight remains essential, while the workflow may reduce routine workload and allow reviewers to focus on targeted verification and contextual judgment.
METHODS: We selected eight high-cost oncology medicines with publicly available regulatory, clinical, and HTA evidence. The workflow extracted structured fields covering indication, population, comparators, trials, endpoints, safety, regulatory/HTA status, reimbursement restrictions, and uncertainty. Time-aware mapping was implemented through a temporal evidence graph linking each statement to source type, document date, page/location, domain, and jurisdiction, enabling assessment at a prespecified index date. Two trained reviewers independently audited 288 fields against source documents using predefined criteria. Jurisdictional alignment was assessed against relevant HTA decision contexts.
RESULTS: Of 288 fields, 91.7% were traceable to correct sources and 86.8% were temporally valid. Jurisdictional alignment was achieved in 82.6%; non-aligned fields most often involved comparator selection, reimbursement restrictions, or outdated HTA wording. Reviewer agreement was substantial (κ=0.78). Major and minor corrections were required for 8.3% and 21.5% of fields, respectively. Among identified errors, missing temporal context (31.0%), oversimplified HTA judgments (26.4%), numerical extraction errors (18.1%), source mismatch (14.8%), and other errors (9.7%) were observed. Median generation time was 11.4 minutes per medicine, excluding reviewer audit.
CONCLUSIONS: AI-assisted time-aware evidence mapping produced structured HTA evidence fields with high source traceability, temporal validity, and jurisdictional alignment, supporting auditable evidence preparation and potential efficiency gains. Evaluation of AI-assisted HTA evidence should extend beyond factual accuracy to include these dimensions and reviewer correction burden. Human oversight remains essential, while the workflow may reduce routine workload and allow reviewers to focus on targeted verification and contextual judgment.
Conference/Value in Health Info
2026-11, ISPOR Europe 2026, Vienna, Austria
Value in Health, Volume 29, Issue 12S
Code
HTA374
Topic
Health Technology Assessment, Methodological & Statistical Research, Organizational Practices
Topic Subcategory
Decision & Deliberative Processes, Value Frameworks & Dossier Format
Disease
No Additional Disease & Conditions/Specialized Treatment Areas