OBJECTIVES: Because different methods for propensity score (PS) matching (PSM) exist, the objective of this study was to assess whether different PSM methods differ in terms of matching quality or study results.

METHODS: We used an anonymized claims dataset of type-2-diabetes-mellitus patients, who were treated with Sulfonylureas (SU; n=904), or Metformin (MET; n=7,874). Associations between treatment assignment and macrovascular outcomes (MACE) and all-cause-survival were analyzed.

Three different sets of baseline variables were used for PS calculation (all available 10 variables, variables significantly associated with group exposition, age/gender/CCI only). To these, we applied the optimal without replacement (O) and the nearest neighbor with replacement (NN) matching algorithm. Caliper widths were defined as fixed (0.001) or determined by PS (0.2*standard deviation of LOG(PS)). In a further scenario, PSM was done within 5-year-age/gender classes. Matching quality was assessed by comparing differences in (1) number of matched patients, (2) baseline characteristics similarity, (3) bias reduction and (4) differences in pneumonia/arm fracture/back pain rates between groups.

RESULTS: In 18 different PSM calculations, between 726 and 904 matched pairs could be derived. Percentage of baseline variables/non-study-related outcomes still significantly different between PSM samples ranged from 0%-40%/0%-20%, depending on PSM method. Highest impact on matching quality showed caliper definition and whether matching was/was not done within fixed age/gender classes. Best matching quality was achieved by using an O approach with caliper 0.001 without matching within pre-defined age/gender classes. In only 10 out of the 18 comparisons, all-cause mortality showed a significant difference between SU/MET exposition (MACE: 4 out of 18 comparisons).

CONCLUSIONS: Because different PSM methods are associated with different matching quality and strongly affect the outcomes of retrospective comparative analyses, we recommend to (1) carefully choose the used PSM method, and (2) to apply different PSM-methods in scenario analyses to test robustness of study results.