A significant but commonly overlooked limitation of propensity score methods (PSMs) is the assumption that any unobserved characteristics are not related to an individual’s choice of treatment and the effect of that treatment. We evaluate the performance of two PSMs - matching and inverse probability weighting (IPW) - for estimating an average treatment effect (ATE) in plausible scenarios of unobserved covariates. We use Monte Carlo simulations to assess how PSMs balances study treatment and comparator arms and impacts the ATE estimate. We consider 3 scenarios: 1) the unobservable covariates affect the probability of treatment only, 2) the unobservable covariates affect the outcome of interest only, 3) the unobservable covariates affect both the probability of treatment and the outcome of interest. We compare the true, unadjusted, and regression-calculated ATEs to the ATEs estimated using matching and IPW. We simulate data analogous to administrative claims with fields for age, gender, and illness severity. The simulated data also includes a covariate that would be unobservable by a researcher but may affect the probability of receiving treatment or the magnitude of the outcome. These simulated covariates are used to construct treatment assignment and a measure of spending as the outcome of interest. We find that PSMs succeed at balancing the treatment and comparator populations across observed covariates, as expected. However, neither matching or IPW succeed at balancing the population to control for the influence of an unobserved covariate on treatment assignment. Moreover, ATE estimates from regression adjustment and both PSMs are comparable when an unobservable covariate affects the outcome but not the treatment. PSMs do balance observable characteristics between treatment and comparator arms of non-randomized treatment but are do not balance unobserved covariates affecting treatment assignment. Researchers should take care to understand potential unobserved variables that affect treatment assignment that would bias study results.