OBJECTIVES: MAIC controls for population differences by weighting patients in the index trial to match the average characteristics of the comparator trial. Weighting leads to a loss of precision, which is quantified in terms of the effective sample size (ESS). We examined whether low ESS in MAICs can also lead to bias despite complete adjustment for imbalanced population characteristics.

METHODS: We simulated unanchored MAICs of time-to-event (TTE) and binary outcomes varying sample sizes, event rates, and levels of population overlap. Relative effect estimates were measured as marginal log hazard ratios from Cox models and log odds ratios from logistic regression models (conventional and penalized likelihood estimation). Scenarios with a bias of 10% or more were examined to understand the determining factors.

RESULTS: Bias was observed in MAICs of both TTE and binary outcomes when the absolute ESS was low (<~30-35), regardless of whether low ESS resulted from low sample size or poor overlap between populations. In TTE outcomes, bias occurred when the effective event count (EEC) – the number of events in the weighted index sample – became very low (EEC<~10). Bias was avoided when the event rate for the TTE outcome was high, even with low ESS. For binary outcomes, traditional logistic regression was susceptible to large bias when ESS<30-35, particularly when the event rate was either low or high. Use of a penalized likelihood approach largely addressed this issue, except where the event rate was high.

CONCLUSIONS: ESS is a useful metric to gauge the potential for bias in MAICs. Across the scenarios we investigated, low ESS was a necessary but not sufficient condition for bias to occur, however. A high event rate protects against potential bias for TTE outcomes but has the opposite effect for binary outcomes. Caution is advisable when interpreting MAICs where ESS is less than ~30.