OBJECTIVES: SLRs are labor-intensive and time-consuming, however, they are required for regulatory and health technology assessments (HTA). We have previously published the application of LiveSTART^TM in title abstract review stage. We now seek to explore whether LiveSTART^TM could be used for LiveRef^TM limited extraction to increase efficiency of a global value dossier (GVD).

METHODS: Data were extracted manually by two independent and experienced analysts. Seventy-eight datasets containing extractions from 6,377 congress abstracts and curated library references were used to train (4,782) and validate (1,595) an AI model in LiveSTART^TM for data extraction. Objective data including indication, population, country, study category, study design, treatment products, sample size, data source and reported variables were prepared. Subjective interpretation as quantitative and qualitative summaries were also included. The accuracy and Hamming scores were calculated. Accuracy is a measure of concordance between prediction and actual results for binary results, whereas Hamming score is used for non-binary outcomes.

RESULTS: LiveSTART^TM validation showed an accuracy = 0.76 for the sample size variable, and Hamming scores of 0.9, 0.84, 0.65, 0.64, 0.53 and 0.5 for indication, category of evidence, treatment products, population, study design and reported variables, respectively. For subjective interpretations, LiveSTART^TM demonstrated good grammar, syntax and logical editing as assessed by a senior medical writer and value communication specialist. LiveSTART^TM extracted data from 1,595 abstracts in ≈30 minutes. LiveRef^TM GVD update demonstrated a 264.5-hour reduction (99.8%) in time as compared to manual extraction. With the addition of the LiveSTART^TM title abstract review tool, a total of 3.29 weeks were saved in a GVD update project.

CONCLUSIONS: With the combination of machine-assisted title and abstract review, and data extraction, LiveSTART^TM AI could potentially yield comparable accuracy with considerable time savings in an SLR project. However, full utilization will require the adoption by regulatory and HTA authorities.