OBJECTIVES:

To critically appraise the quality of radiotherapy cost studies published between 2004 and 2015.

METHODS:

Building upon a recent systematic literature review looking at the provider cost in PubMed Medline, Embase (Defourny, 2016), selected studies were examined in-depth focusing on the study quality. The review was performed by three independent reviewers and restricted to studies using cost data published after the QHES grid became available. Selected studies were critically assessed based on 3 existing instruments: CHEERS, CHEC, and QHES.

To compare the instruments (inter-instrument agreement) and the agreement between rater, the Kendall coefficient of concordance was calculated.

A further refined examination of the references was conducted, focusing on the guidelines’ items relevant for costing.

RESULTS:

The selection restricted our analysis to 15 articles to be studied in-depth, out of the 52 studies previously selected.

The mean score across all articles were respectively 70%, 81%, 72%, mirroring variation in scope of the articles. Over time, instruments’ scores rise with a higher tendency for QHES.

Comparison of the instruments per assessor yielded a significant coefficient of 94%, 84% and 87% while the coefficient per rater was significant at 69%, 81% and 77% for each instruments. The inter-rater analysis yielded 10 percent point variation, meaning that they agreed most in rating articles against CHEC, QHES and lastly CHEERS.

While the overall mean score was fairly good (74% [31%-93%]), in-depth examination highlights the omission (in the instruments) of criteria covering the type of cost analysis (mean, incremental or full costing) as well as whether the analysis features were modelled or reality-based; these define methodology transparency.

CONCLUSIONS:

Cost studies’ quality was fairly good while influenced by the instrument more than by the assessor.

Existing cost guidance establishes an outline framework while leaving a high degree of freedom in the reporting by the researchers which jeopardize comparability across studies.