Natural History of Creutzfeldt-Jakob Disease: A Comparison of Manual vs. Manual Plus AI-Assisted Approaches in Literature Review
Author(s)
Setareh A. Williams, PhD1, Richard J. Weiss, MD2, Russell Vincent Becker, MA3, Guy Rayford Mitchell, Jr., MPH4.
1President and CEO, Star Biopharma Consulting, LLC., Star Biopharma Consulting, Malvern, PA, USA, 2Star Biopharma Consulting, Malvern, PA, USA, 3Russell Becker Consulting, Mobile, AL, USA, 4Star Biopharma Consulting, LLC., New York, NY, USA.
1President and CEO, Star Biopharma Consulting, LLC., Star Biopharma Consulting, Malvern, PA, USA, 2Star Biopharma Consulting, Malvern, PA, USA, 3Russell Becker Consulting, Mobile, AL, USA, 4Star Biopharma Consulting, LLC., New York, NY, USA.
OBJECTIVES: To compare the results of a literature review on the natural history of Creutzfeldt-Jakob Disease (CJD)-a rare, fatal neurodegenerative brain disorder-conducted manually vs. manually plus AI assistance.
METHODS: A step-by-step approach was used to compare the two methods for identifying CJD natural history articles. Abstract and full-text screenings were conducted independently with two individuals conducting the abstract screening and another two, who had not reviewed the abstracts, conducting the full manuscript review. AI assisted review was carried out, in accordance with ISMPP guidance.
RESULTS: In Step 1 (PubMed Literature Search), 51 articles were identified using predefined search strings. This increased to 58 after adding 10 AI-suggested terms. Step 2 (abstract screening) involved manually reviewing 58 abstracts. Of these, 33 case studies on diagnostic challenges were excluded and of the remaining 25, 18 were relevant, covering prognosis, phenotypic variations, genetics, and immune factors. Seven were off-topic (eg, ALS, Alzheimer’s, delirium, dementia, BSE, animal studies, and general guidelines) but AI, via a Python-coded program, falsely marked 3 as relevant. Among the 7 AI-identified abstracts, 5 were case studies, 1 focused on delirium, and 1 was relevant. AI rankded 58 abstracts as important (scores 8-10) except for 5, which were scored 2-5. Manual screening also deemed these 5 irrelevant but found only 18 abstracts relevant overall, not 53 as AI suggested. In Step 3 (Full-text screening), 18 manuscripts were manually reviewed, with 15 found relevant. AI assigned high relevancy scores to 13 of the 18. While AI agreed on exclusion of 3, it also excluded 2 that manual review found relevant.
CONCLUSIONS: While AI may have the potential to enhance literature review, it is advised to be cautious and use AI to complement, not replace, human reviewers to maintain accuracy and reliability at every stage of the literature review.
METHODS: A step-by-step approach was used to compare the two methods for identifying CJD natural history articles. Abstract and full-text screenings were conducted independently with two individuals conducting the abstract screening and another two, who had not reviewed the abstracts, conducting the full manuscript review. AI assisted review was carried out, in accordance with ISMPP guidance.
RESULTS: In Step 1 (PubMed Literature Search), 51 articles were identified using predefined search strings. This increased to 58 after adding 10 AI-suggested terms. Step 2 (abstract screening) involved manually reviewing 58 abstracts. Of these, 33 case studies on diagnostic challenges were excluded and of the remaining 25, 18 were relevant, covering prognosis, phenotypic variations, genetics, and immune factors. Seven were off-topic (eg, ALS, Alzheimer’s, delirium, dementia, BSE, animal studies, and general guidelines) but AI, via a Python-coded program, falsely marked 3 as relevant. Among the 7 AI-identified abstracts, 5 were case studies, 1 focused on delirium, and 1 was relevant. AI rankded 58 abstracts as important (scores 8-10) except for 5, which were scored 2-5. Manual screening also deemed these 5 irrelevant but found only 18 abstracts relevant overall, not 53 as AI suggested. In Step 3 (Full-text screening), 18 manuscripts were manually reviewed, with 15 found relevant. AI assigned high relevancy scores to 13 of the 18. While AI agreed on exclusion of 3, it also excluded 2 that manual review found relevant.
CONCLUSIONS: While AI may have the potential to enhance literature review, it is advised to be cautious and use AI to complement, not replace, human reviewers to maintain accuracy and reliability at every stage of the literature review.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
MSR152
Topic
Methodological & Statistical Research, Organizational Practices, Study Approaches
Disease
Infectious Disease (non-vaccine), Neurological Disorders, Rare & Orphan Diseases