OBJECTIVES: As current health technology assessment (HTA) frameworks do not provide specific guidance on the assessment of medical artificial intelligence (AI), this study aimed to propose a conceptual framework for a broad HTA of medical AI.

METHODS: A targeted literature search of policy documents was conducted to distill the medical AI issues relevant for impact-assessment. Publications by national (UK, ) and supranational (European Union) governing bodies, non-profit organizations and academic institutions were obtained from their websites and through a Google search. Three exemplary cases were selected to illustrate the relevance of the extracted issues: (1) An application supporting radiologists in stroke-care (2) A natural language processing application for clinical data abstraction (3) An ICU-discharge decision-making application.

RESULTS: A total of 31 publications were selected, from which a long-list of extracted issues was reduced to a short-list of 25 issues, based on an iterative deductive and inductive approach. The issues were grouped by four focus areas: (1) Performance & Evidence, (2) Human & Organizational, (3) Legal & Ethical and (4) Transparency & Usability. Although some issues were relevant to all three case studies (e.g., clinical effectiveness, cost-effectiveness), they also showed variability, with workflow issues being particularly important in stroke-care, generalizability in clinical data abstraction and explainability & fairness in ICU-discharge decision-making.

CONCLUSIONS: The current methodology of HTA requires extension to make it suitable for a broad evaluation of medical AI technologies. The application of the 25-item assessment list that we propose should be tailored to the intervention being assessed, since the case studies showed that conceptualization of the issues differs across applications.