OBJECTIVES: The NICE evidence standards framework (ESF) for digital health and care technologies was originally published in 2019. It provides a pragmatic set of evidence standards for digital health technologies (DHTs) in the UK health and care system, including risk-stratified requirements for clinical evidence and graded levels of economic evaluation based on the scale of commissioning. In 2021, NHSx commissioned the NICE Office for Digital Health to update the ESF and expand it to include evidence standards for DHTs that include artificial intelligence algorithms (AI-DHTs) that may change over time (adaptive algorithms).

METHODS: NICE commissioned an academic consortium including experts from Imperial College London, the University of Birmingham, and the Alan Turing Institute to inform the design of the AI standards. The NICE team and academic partners conceptualised draft AI standards, which were iterated through surveys, semi-structured expert interviews and a modified Delphi consensus process. NICE held stakeholder workshops on the AI standards and iterated them according to feedback. A public consultation on the AI standards was held in March 2022, and feedback was incorporated into the standards.

RESULTS: A literature review, expert survey and semi-structured interviews identified the OECD Framework for the Classification of AI Systems to be the most appropriate system to use as a framework. Key areas identified for AI-specific evidence standards include, among others, (1) the quality and representativeness of training/test data, (2) measuring changes in performance (across sites and over time), and (3) the need for transparency in understanding the limitations of the AI-DHT, what its outputs are and how these should be interpreted.

CONCLUSIONS: The AI expansion to the NICE evidence standards framework will be published in June 2022. It provides a clear and usable set of evidence standards for AI-DHTs. NICE will continue to iterate the ESF as this fast-moving field develops further.