Evidence synthesis based on the quantitative analysis (indirect treatment comparisons ITC) of data from randomized controlled trials (RCTs) is the gold standard in healthcare decision making. However, when ITC of RCTs may not be practical or possible, comparative observational data may provide a valuable alternative despite potential challenges with internal validity. The present project aims to develop a conceptual framework from which valid quantitative assessment of real-world evidence (RWE) can provide necessary insights into treatment effects outside the RCT settings. Considerations of quantitative evidence synthesis in RWE were identified using published best-practice guidelines for quantitative analysis of RCT data and applied  to two test cases, atrial fibrillation (AF) treated with novel oral anticoagulant (NOAC) and type II diabetes treated with antihyperglycemic agents. Data sources of RWE, sample selection, outcome definitions, and statistical method variation were assessed. Significant variations across all key factors were observed in studies for both indications. In the case of NOAC treated AF, homogeneity of the data source and geographic location did not guarantee appropriate quantitative comparison. Several factors unique to assessing observational database study comparability for meta-analyses were identified including a need for a greater focus on selection bias inherent to the type and location of the database and increased sensitivity of estimates to the index dates particularly in relation to local and international product launches, as well various outcome definition and reporting complexities. Valid ITC could be performed using RWE from comparative observational studies. Careful selection of studies is required to mitigate factors that would otherwise compromise results through uncontrolled bias. Researchers need to go beyond the obvious individual study reporting biases and fully assess the complexities of comparative data sources to be able to appropriately select studies for ITC.