The Role of Education in Shaping an Open-Source Future
Shannon Kindilien, PhD, University of New Mexico, Albuquerque, NM, USA; Diana Alecsandra Grad, MPH, Department of Public Health, Babes-Bolyai University, Cluj-Napoca, Cluj, Romania; Xavier G.L.V. Pouwels, PhD, University of Twente, Department of Health Technology & Services Research, Enschede, The Netherlands
Transparency, Built In
Health policy making is increasingly informed by the results of simulation models. Distrust concerning the results of such models is nevertheless on the rise as their development and underlying methods are often confidential. This contradicts (1) the societal wish for more openness concerning policy making, and (2) the principles of the Open Science movement,1 which have been steadily embraced by the scientific community. In this article, we focus on the potential contribution of the role of education in developing and using transparent health economic analyses to improve the transparency of health policy decisions, but note that this is not the sole element affecting the transparency of decisions.
While openness is considered valuable from scientific and societal points of view, current health economics and outcomes research (HEOR) education devotes little, if any, attention to training students to perform evaluations adhering to the Open Science1 or FAIR (Findability, Accessibility, Interoperability, and Reusability) Guiding Principles.2 According to the UNESCO Recommendation on Open Science, open science includes efforts to make scientific knowledge “…openly available, accessible and reusable for everyone…, and to open the processes of scientific knowledge creation, evaluation and communication to societal actors….”1 Also, FAIR comprise guiding principles for data management and stewardship that promote the findability, accessibility, interoperability, and reuse of digital assets.2
Today’s students will be tomorrow’s analysts and policy makers. Hence, the HEOR community should give more attention to developing open-source modeling skills. Future professionals should learn to perform health economic analyses that are transparent, reproducible, widely accessible to all (ideally without restrictions), and interoperable.2 Besides modeling skills, this requires the ability to make one’s code readable, understandable, and findable by others.
Open science principles1 also encourage more intensive stakeholder engagement in the practice and communication of HEOR evaluations; these provide a critical means of reinstating trust in science and increasing support for health policy. Existing initiatives within the HEOR community are contributing to this paradigm shift. These include, among others, the Innovation and Value Initiative, the Peer Models Network, the Open Source models Clearinghouse, and ISPOR’s Open Source Model Special Interest Group. However, greater visibility and engagement is still needed, particularly with junior HEOR colleagues and students.
"While openness is considered valuable from a scientific and societal point of view, current health economics and outcomes research education devotes little, if any, attention to training students to perform evaluations adhering to the Open Science or FAIR principles."
Developing an Open-Source Skillset
Putting the principles of open science into practice requires both technical and social skills. Concerning the technical skills, acquaintance with open source programming languages with high-quality version control, such as Python3 and R,4,5 may seem the most adequate medium to prepare future professionals to meet the goals of open science.
Transparency and reusability in modeling can be partially achieved by providing access to source code. Appropriate documentation about a given analysis (eg, inputs, workflows, outputs) and how to modify the model for other purposes allows for peer review. Learning to concisely describe and assess the underlying theory, assumptions, and variables of models is also critical to participating in open science and should be emphasized in academic curricula.
Exposure to data visualization techniques and tutorials provides means of acquiring open-source skills but learning how to develop and deliver these is also critical to contributing to HEOR’s open source future. Interactive data visualization, in particular, can improve public engagement; it is an increasingly important part of health communication in our technologically rich age. The skills to visually represent data and provide interactive or exploratory views of a study should be encouraged among modelers and HEOR scientists. The R package Shiny, for example, supports the development of apps or dashboards. Alternatively, recorded audiovisual documentation of a study or tutorials for the use of a model can be made available through academic homepages, websites, or social media platforms. Videos or audio annotated PowerPoint presentations can be effective ways to engage with others; they present opportunities to demonstrate a functioning model in real time, to create a dialogue about updates or modifications to an existing open-source resource, or to document the outcomes of an individual analysis for stakeholders and the public.
Limitations of Open-Source Resources
It is important to recognize there are barriers to using existing open-source models. It may not always be possible to find a resource that satisfies the needs of a current decision problem, or to publicly share content developed for a specific research question.
Barriers to using existing models may include a lack of documentation, a lack of coding convention, a context-specific design, a lack of validation studies, and confusion around the presence of multiple versions of the model without any curation of each successive release. Researchers may also be discouraged by programs that rely on multiple underlying pieces of code without clear distinctions between what is a functioning or legacy component, or that are written in multiple programming languages. In these instances, a research team may be forgiven for thinking the cost of retrofitting an open-source model is equal to or greater than doing the same to a proprietary model.
"By making graduates literate in programming language including code testing, version control, licensing, distribution, data visualization, and public health communication, we prepare them to be better peers and more effective professionals who can engage with both HEOR stakeholders and the public."
The decision to make one’s own code open source, while morally admirable and consistent with the spirit of open, peer-reviewed scientific work, also involves barriers. Within HEOR, much of the data used are confidential, personal, or proprietary. In these cases, creating documentation, tutorials, or providing a test sample of data for others to use as proof of concept may be impossible or represent a significant additional resource burden for the developer. The expectation to maintain and update any resource made public, as well as the responsibility to address any potential issues of liability, may also act as deterrents.
The Future of HEOR Open Source
Health economists have drawn attention to the need for a health economic model registry,6,7 which would retrospectively and prospectively contain structured information on existing health economic models. This type of platform could address many known problems within HEOR, including the lack of documentation on existing models, the failure to curate open-source model versions, the potential publication bias for proprietary resources, intellectual property disagreements, as well as offer a better way of determining which models or model versions have been subjected to validation studies. In addition, by allowing protocols of health economic models to be cited separately from journal articles, a more robust conversation about transparency and credibility will be possible.
Improving and assessing existing models is and will remain part of the HEOR community’s responsibility to stakeholders and the public. Continuous efforts that address model transparency, validation, and reproducibility already exist, but open-source–specific guidelines and assessment tools will also be needed. Proposed coding frameworks,8 checklists such as TECH-VER (Technical Verification)9 and AdviSHE (Assessment of the Validation Status of Health-Economic decision models)10 validation assessment tools all have a place in managing the open-source assets of the future.
Academic programs and training courses in HEOR should incorporate open science and FAIR principles into their curricula. The ability to communicate about, engage with, and contribute to this part of our professional community is an increasingly important and hirable skill. Enhancing transparency and resource availability is consistent with the goals of higher education worldwide. By making graduates literate in programming language including code testing, version control, licensing, distribution, data visualization, and public health communication, we prepare them to be better peers and more effective professionals who can engage with both HEOR stakeholders and the public.
References
1. United Nations Educational S, and Cultural Organization. Draft text of the UNESCO Recommendation on Open Science. Published 2021. Accessed May 11, 2022. https://en.unesco.org/science-sustainable-future/open-science
2. Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:160018.
3. Seabold S, Perktold J. Statsmodels: Econometric and statistical modeling with python. Paper presented at: Proceedings of the 9th Python in Science Conference 2010.
4. Hart R, Burns D, Ramaekers B, et al. R and shiny for cost-effectiveness analyses: why and when? a hypothetical case study. Pharmacoeconomics. 2020;38(7):765-776.
5. Racine JS. Reproducible econometrics using R. New York, NY: Oxford University Press; 2019.
6. Sampson CJ, Wrightson T. Model registration: a call to action. Pharmacoecon Open. 2017;1(2):73-77.
7. Sampson CJ, Arnold R, Bryan S, et al. Transparency in decision modelling: what, why, who and how? Pharmacoeconomics. 2019;37(11):1355-1369.
8. Alarid-Escudero F, Krijkamp EM, Pechlivanoglou P, et al. A need for change! a coding framework for improving transparency in decision modeling. Pharmacoeconomics. 2019;37(11):1329-1339.
9. Büyükkaramikli NC, Rutten-van Mölken MPMH, Severens JL, Al M. TECH-VER: a verification checklist to reduce errors in models and improve their credibility. Pharmacoeconomics. 2019;37(11):1391-1408.
10. Vemer P, Corro Ramos I, van Voorn GA, Al MJ, Feenstra TL. AdViSHE: a validation-assessment tool of health-economic models for decision makers and model users. Pharmacoeconomics. 2016;34(4):349-361.
