OBJECTIVES:  Systematic reviews are a core, albeit labour intensive, component of evidence based medicine. Recently a number of, mainly commercial, automation tools have been developed to aid the processes of systematic review focusing on study selection, article review and data extraction. We sought to develop an open-source, freely available program using the statistical package R to undertake data extraction. To test our program we selected the Cochrane risk of bias tool for RCTs, an established short questionnaire, answered with yes/no/unclear responses. Our aim was to accurately automate responses using a bespoke R program. METHODS:  The Cochrane risk of bias tool comprises 5 components: (1) randomisation, (2) allocation concealment, (3) blinding, (4) incomplete data and (5) selective reporting. We developed an algorithm programmed in R using the ‘tm’ package to data mine publications, responding yes/no/unclear to each of the 5 components; results were generated in both a tabular format and in a risk of bias summary figure. Results were compared to abstraction conducted independently by an experienced systematic reviewer. A selection of multiple myeloma RCTs provided our case study. RESULTS:  Abstraction conducted by the reviewer was used to check the accuracy of response to each risk of bias questions. Accuracy of abstraction generated by our R program varied considerably between questions. For the first 3 questions, accuracy was high where there are expected stock phrases as responses. Lower accuracy was seen for questions 4 and 5; language around these responses is more nuanced requiring higher level interpretation across multiple criteria. CONCLUSIONS:  Accurate automation of data extraction is achievable for ‘box ticking’ aspects of systematic review, but still requires human validation to ensure data is interpreted correctly. A potential use at this early stage is validation of human data extraction. Future work will widen the scope of our program to other aspects of systematic review.