OBJECTIVES: Veterans Administration Corporate Data Warehouse stores databases with standardized structure that could be used for automated data extraction, reviewer abstraction, and text mining to determine the association between health outcomes and disease-specific factors. This retrospective study provides assessment of VA administrative data used to examine the impact of pharmacological therapy on complications in ulcerative colitis (UC). METHODS:  Previous studies investigating the effect of 5ASA on the risk for colorectal cancer (CRC) in UC patients have reported conflicting results. We obtained nationwide UC and CRC data from VA healthcare system for the period 2001-2011. Secondary relational databases were searched for clinical variables based on standardized criteria - ICD9 diagnoses, procedural and medication codes. Data extraction captured demographics, clinical information and pharmacy record for a cohort of 37,191 UC cases. We constructed a dataset of potential ulcerative colitis cases with CRC (n=1,087) defined by ICD9 codes 556.x for UC, and 153.x,154.x and 159.0 for CRC. A random subsample of 100 non-5ASA users with CRC was compared to 100 controls without CRC. RESULTS: Diagnosis of ICD9 code for CRC had PPV 79% and NPV 100% in the random sample. Within the 1087 potential CRC cases, only 500 (46%) were found to have evidence of both conditions on chart review with kappa agreement between automated and manual abstraction 0.73 (95% CI:  0.70-0.76) for CRC and significantly lower for UC - 0.60 (95% CI: 0.57-0.63). The initial overall prevalence of CRC in the UC cohort was 2.9% and decreased to 1.34% after human text search verification. CONCLUSIONS: Automated extracts have great potential for diseases surveillance but manual review yields more reliable data. Pre-defined diagnostic algorithms based on a combination of methods as well as further technology development like natural language processing and longitudinal patient record will improve accuracy of retrospective databases.