OBJECTIVES: The tobacco industry is actively developing modified risk tobacco products (MRTPs) with the potential to reduce tobacco-related mortality and morbidity in the population. The 2012 FDA Draft Guidance on MRTP applications asks applicants to conduct consumer-based research on the perceptions of risk associated to tobacco products. This study reports on how qualitative methods described in the 2009 FDA Guidance on Patient-Reported Outcome Measures can be used with cognitive debriefing interviews (CDI) to understand how consumers interpret the complexities of risk ratings. METHODS: Reanalysis of CDI transcripts on a new risk perception measure, the Perceived Risk Instrument (PRI), was used to explore ways that consumers evaluate tobacco product health risk. 48 interviews were conducted in two US cities over four days. 24 individuals completed the PRI as applied to conventional cigarettes (CC) and 24 as applied to MRTPs. Codification and thematic frequency analysis followed commonly accepted standards. RESULTS: Thematic saturation occurred on the first day and by the end of the second day all content themes were defined. Daily refinements of the PRI lead to a reduction of the number of concerns expressed over time. The frequency of some risk evaluations for CC and MRTP were similar: For example the general degree of difficulty rating risks (CC, 58% vs. MRTP, 46%); and the need for clear time frames to evaluate risk (CC, 21% vs. MRTP, 29%); while other evaluations differed by product, such as unfamiliarity with product risks (CC, 0% vs. MRTP, 25%). CONCLUSIONS: Qualitative methods can be used with CDI data to refine risk perception measures. Understanding the ways consumers evaluate the risk associated with tobacco product use is necessary to create instruments that provide accurate data to inform public health policies. Such considerations also help improve population-based research and evidence-based product assessment.