OBJECTIVES: Rare disease studies often suffer from a lack of power to detect differences across treatment arms. This lack of power can stem from testing multiple endpoints, which requires adjustments for multiplicity. However, these adjustments can be too conservative, obscuring true treatment effects. This is especially concerning in rare disease studies, where it is difficult or impossible to improve power by sampling more subjects. The objective was to show that, compared to traditional hypothesis testing, permutation tests can improve power while still adjusting for multiplicity (i.e., controlling Type I error).

METHODS: Permutation tests differ from more well-known hypothesis testing approaches by estimating the null distribution via resampling instead of assuming that the null distribution of the test statistic follows a theoretical distribution (i.e., t-statistic is compared to a t-distribution for a t-test). Resampling creates a more accurate null distribution, which leads to more powerful hypothesis tests. Simulation studies based on empirical PRO data were used to illustrate the advantages of the permutation test in terms of power and Type I error. The sample size, data distribution, and number of hypothesis tests were varied.

RESULTS: The results of the simulation studies showed that, compared to the traditional t-test, the permutation test led to substantial gains in power while controlling Type I error. For example, although both the permutation test and the Bonferroni adjustment controlled Type I error at 0.05, the permutation test demonstrated a power of 0.76 to identify a true treatment effect, whereas the Bonferroni adjustment had a power of 0.59.

CONCLUSION: Permutation tests were well-established in the statistical literature at a time when lack of computing power prevented their widespread adoption. Revisiting the literature suggests that this approach is not only feasible to implement but can lead to meaningful improvements in power and precision in rare disease studies.