Validation of 5 key colonoscopy-related data elements from Ontario health administrative databases compared to the clinical record: a cross-sectional study
===========================================================================================================================================================

* Jill Tinmouth
* Rinku Sutradhar
* Ning Liu
* Nancy N. Baxter
* Lawrence Paszat
* Linda Rabeneck

## Abstract

**Background:** Colonoscopy is used widely, but its quality is highly variable, which may adversely affect patients. Research and quality-improvement initiatives in a variety of jurisdictions have sought to address this issue, often supported by the use of health administrative data. As these data are generally not collected for these purposes, it is critical to measure their validity before use. The aim of this study was to validate health administrative data definitions for 5 key colonoscopy elements through comparison to the clinical record.

**Methods:** In a cross-sectional study, we randomly sampled 1968 colonoscopy and noncolonoscopy procedures performed at 23 hospitals and 5 nonhospital endoscopy clinics between April 2008 and March 2009 in Ontario. We compared definitions for 5 key colonoscopy elements (colonoscopy case, colonoscopy setting, colonoscopy completeness, anesthesiologist assistance and polypectomy) derived from the health administrative data to the clinical record. We calculated weighted and unweighted sensitivity, specificity and positive predictive value, adjusted for clustering of patients within physicians, for each definition relative to the reference standard.

**Results:** We abstracted 1845 records; in 1282 cases (69.5%), colonoscopy was intended or performed. The weighted sensitivity and specificity of colonoscopy case, nonhospital colonoscopy setting and anesthesiologist assistance exceeded 95%. The weighted sensitivity for colonoscopy completeness and polypectomy exceeded 95%, but specificity was less than 90%.

**Interpretation:** Ontario health administrative data definitions for 5 key colonoscopy data elements performed well, with sensitivity and specificity values acceptable for use in research and quality-improvement initiatives. In jurisdictions where health administrative data are similarly used for research or quality improvement, similar studies could be considered.

Colonoscopy is common, essential for the management of gastrointestinal diseases. The procedure has important risks, including perforation (0.85/1000), bleeding (1.64/1000) and death (0.074/1000).1 Furthermore, there is considerable variation in the performance and quality of colonoscopy,2,3 with attendant consequences for the health care system and patients. For example, although colonoscopy can be safely and comfortably performed with moderate sedation administered by the endoscopist, increasingly in some jurisdictions, anesthesiologists are providing deep sedation for the procedure;4–6 however, there are added costs6 and potentially complications7,8 associated with this practice. Patients are adversely affected by poor-quality colonoscopy; such procedures are associated with missed colorectal cancers and cancer-related death.9

Initiatives related to colonoscopy practice standards,10,11 research12,13 and policy14,15 have been implemented to improve quality. The use of routinely collected data16 such as health administrative data for these purposes is highly attractive as they are inexpensive to use and readily available and can be captured uniformly across a health care payer or system. These data have been used to study and measure the quality of colonoscopy, including completeness,17 type of setting,4,18 polypectomy rate/adenoma detection rate,9,19 anesthesiologist assistance,4–8 complications1,8 and missed cancers.9,18 Health administrative data are also used for funding and accountability and to measure performance and quality of care by government agencies.20 As these data are often collected for other reasons, 21,22 it is critical to ensure their validity when using them for these purposes. In other jurisdictions, colonoscopy validation studies have generally focused on procedure indication; 23–25 other aspects of colonoscopy have not been widely evaluated. In Ontario, the Institute for Clinical Evaluative Sciences houses health administrative databases containing the health care records for the population of Ontario. These databases have been used extensively in colonoscopy research1,4,7,17 and quality improvement,26 but, to date, they have not been validated. The objective of this study was to validate health administrative data definitions for 5 key colonoscopy elements: colonoscopy case, colonoscopy setting, colonoscopy completeness, anesthesiologist assistance and polypectomy.

## Methods

### Overview

In this multisite chart abstraction study, we created health administrative definitions of 5 colonoscopy data elements: a “colonoscopy case,” colonoscopy setting, colonoscopy completeness, anesthesiologist assistance and polypectomy. We compared these definitions to reference standards: clinical data obtained via chart abstraction at 23 hospitals and 5 nonhospital endoscopy clinics in Ontario. For some data elements, we included more than 1 health administrative definition and/or more than 1 reference standard (see below and Table 1⇓ for detailed descriptions). The clinical data largely comprised medical records of complete and incomplete colonoscopy procedures but also included gastroscopy and flexible sigmoidoscopy procedures to allow estimation of the true-negative rate for the case definition of colonoscopy.

View this table:
[Table 1:](http://www.cmajopen.ca/content/6/3/E330/T1)

Table 1: 
Description of the cohorts, administrative data definitions and reference standards for 5 colonoscopy data elements

### Sources of administrative data

We used the Ontario Health Insurance Plan (OHIP) database and the Canadian Institute for Health Information databases housed at the Institute for Clinical Evaluative Sciences. The OHIP database contains physician billing data on inpatient and outpatient visits and procedures including colonoscopy since 1991. The Canadian Institute for Health Information databases comprise diagnosis and procedure (both inpatient [Discharge Abstract Database] and same-day [Same Day Surgery database]) codes for all hospital admissions in Canada since 1988. These databases were linked at the individual level by means of an encrypted version of the provincial health card number.

### Hospital and nonhospital endoscopy clinic sites

We randomly selected 23 hospital and 5 nonhospital facilities in Ontario to participate in the study (Figure 2). The hospital sites were divided into 4 strata, and from those that performed more than 200 colonoscopy procedures in the prior year (*n* = 106/115), we selected a random sample from each stratum as follows: rural (*n* = 2), urban and lowest tertile of annual colonoscopy volume (*n* = 8), urban and middle volume tertile (*n* = 7), and urban and highest volume tertile (*n* = 6). With the assistance of endoscopy equipment manufacturers, we identified 34 nonhospital clinics active during the study period. We randomly selected 5 clinics from among those who performed more than 200 colonoscopy procedures in the prior year and indicated willingness to participate in the study (*n* = 21/34).

![Figure 2:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/6/3/E330/F1.medium.gif)

[Figure 2:](http://www.cmajopen.ca/content/6/3/E330/F1)

Figure 2: 
Flow chart showing sampling of facilities and procedures.

### Medical record abstraction

We identified all outpatient procedure visits for gastroscopy, flexible sigmoidoscopy and colonoscopy at the 28 facilities from Apr. 1, 2008, to Mar. 31, 2009, using health administrative data (hospitals) or self-reported billing data (clinics). These visits constituted the sampling frame (Figure 2). We used OHIP codes to identify procedures as this approach allowed identification of procedures performed outside of hospitals as well as the stratified sampling for colonoscopy procedures described below. Colonoscopy procedures were identified with the base code (Z555A), which was common to all colonoscopy procedures. From the 144 078 procedures in the sampling frame, we then randomly selected 1968 medical records for abstraction using a stratified sampling strategy.27 Briefly, the procedures in the sampling frame were divided into 6 strata based on billing data (complete colonoscopy, incomplete colonoscopy to descending colon, incomplete colonoscopy to splenic flexure, incomplete colonoscopy to hepatic flexure, gastroscopy, flexible sigmoidoscopy). We then randomly sampled procedures in unique patients from each stratum, aiming for our intended sample sizes as described below. We deliberately oversampled, expecting missing charts at the time of abstraction. The final sample included complete colonoscopy procedures (*n* = 794), incomplete colonoscopy procedures (*n* = 806), gastroscopy procedures (*n* = 128) and flexible sigmoidoscopy procedures (*n* = 240). Our sampling strategy ensured adequate inclusion of less common events, such as incomplete colonoscopy, while maintaining a sample size feasible for medical record abstraction.

At the 28 sites, trained medical record abstractors from the Institute for Clinical Evaluative Sciences, blinded to the administrative data, abstracted from 3 prespecified sources (the endoscopist’s procedure note, the anesthesiologist’s record and the pathologist’s record) using a standardized data collection protocol. The abstractors collected the data in a standardized fashion using a customized data collection platform residing on encrypted laptop computers. The data were then transmitted from each site to the Institute for Clinical Evaluative Sciences via a secure virtual private network. If the endoscopist’s procedure note could not be found at the sites or if the dates for the procedure found in the medical record at abstraction did not agree with those in the administrative data, the case was excluded.

### Cohort creation

We created 3 cohorts for the analyses. The first cohort comprised all successfully abstracted medical records and was used to assess the health administrative data definition for colonoscopy case. The second cohort comprised medical records in which colonoscopy or flexible sigmoidoscopy was intended and colonoscopy codes were billed; we used this cohort to evaluate the definition for colonoscopy completeness. Ontario physicians bill colonoscopy per segment reached; this data structure allows measurement of colonoscopy completeness.17 Although there is a separate code for flexible sigmoidoscopy, anecdotally, colonoscopy codes are often used to bill this procedure as remuneration is better. As this practice may result in misclassification of incomplete colonoscopy procedures when Ontario administrative data are used, we included flexible sigmoidoscopy procedures billed with colonoscopy codes in the cohort. We excluded patients with prior total colectomy or right hemicolectomy, as these patients are typically excluded when administrative data are used to measure colonoscopy completion. Finally, the third cohort comprised medical charts in which colonoscopy was intended or performed and was used to test the remaining 3 colonoscopy data elements (colonoscopy setting, anesthesiologist assistance and polypectomy).

### Administrative data definitions and reference standards for colonoscopy data elements

Descriptions of administrative data definitions and reference standards for each data element are given in Table 1. There were 14 alternative definitions for colonoscopy case because of the structure of OHIP colonoscopy codes: a base code (Z555A) must be used indicating that the scope was inserted to the level of the descending colon. Up to 4 additional “E” codes are then used for every additional segment of colon or terminal ileum visualized. Figure 1 provides a listing of the specific OHIP codes used in each definition. We evaluated colonoscopy completeness using 2 definitions of colonoscopy case: achieving the cecum or terminal ileum among 1) colonoscopy procedures identified using the most sensitive definition and 2) those identified using the most accurate definition, based on the analyses described below.

![Figure 1:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/6/3/E330/F2.medium.gif)

[Figure 1:](http://www.cmajopen.ca/content/6/3/E330/F2)

Figure 1: 
Weighted sensitivity and specificity of 14 administrative data definitions using Ontario Health Insurance Plan (OHIP) codes of colonoscopy case compared to the reference standard of colonoscopy intended or performed according to the medical record. Note: CI = confidence interval.

We compared health administrative data definitions to relevant reference standards obtained via chart abstraction by the chart abstractors. The reference standard for colonoscopy case was the intent to perform or performance of a colonoscopy according to the endoscopist’s procedure note in the medical record at the institution where the colonoscopy was performed. The “intention to perform a colonoscopy” criterion allows the distinction of an incomplete colonoscopy from a flexible sigmoidoscopy. There were 2 reference standards for anesthesiologist assistance (presence of anesthesiologist record, indicating an anesthesiologist attended the procedure, and propofol use documented in anesthetic record or endoscopist’s procedure note) and for polypectomy (polyp documented by endoscopist and histologic confirmation of adenoma, including advanced adenoma, or sessile serrated adenoma/polyp documented). We included 2 reference standards for these data elements as anesthesiologist assistance and polypectomy are often used as surrogates for the use of propofol and adenomas, respectively. Reference standards were not validated; however, the medical record is generally considered an acceptable reference standard for studies validating health administrative data.28 The specific OHIP billing codes used and their definitions are given in Supplementary Tables A1–A3, Appendix 1.

### Sample size

We performed a sample size calculation a priori for the key data elements of colonoscopy case and colonoscopy completeness. Using methods for 2-sided binomial tests,29 we estimated that we would need at least 600 complete colonoscopy cases, 600 incomplete colonoscopy cases and 300 noncolonoscopy cases to have over 80% power (α = 0.05) to detect an absolute difference of at least a 4% in coding accuracy (assuming a coding accuracy proportion, determined by comparing the medical record data with the administrative data, of 0.85) within each group.

### Statistical analysis

We calculated sensitivity and specificity with 95% confidence intervals for each administrative data definition relative to the relevant reference standard. We adjusted the 95% confidence limits for clustering of patients within physicians using the Taylor series expansion method.30 Because we oversampled incomplete colonoscopy procedures, we performed both unweighted and weighted analyses, where the weights reflected the distribution of procedures in the sampling frame relative to those in the sample. Weighted results are presented unless there was important variation between weighted and unweighted results. For colonoscopy case, we created a receiver operating curve by plotting the sensitivity (on the y-axis) and 1-specificity (on the x-axis) for the 14 definitions. We defined the definition located in the upper left-hand corner of this curve as the most accurate as per the Youden method.31 We performed all sampling and analyses using SAS version 9.3 (SAS Institute, Inc.).

### Ethics approval

Ethics approval was obtained from the Sunnybrook Health Sciences Centre Research Ethics Board as well as from the research ethics boards at the 23 hospitals where chart abstraction was performed.

## Results

Among the 1968 randomly selected endoscopy procedures, 1845 charts (93.8%) were successfully abstracted (see Figure 2 for exclusions). Colonoscopy was intended or performed in 1282 (69.5%) of the 1845 cases. About 25% of patients undergoing the included procedures were less than age 50 years, and over half were women; 284 procedures (15.4%) were performed in nonhospital endoscopy clinics (Table 2). Seven to 153 charts were abstracted per hospital, and 31–86 charts were abstracted per clinic.

View this table:
[Table 2:](http://www.cmajopen.ca/content/6/3/E330/T2)

Table 2: 
Patient and procedure characteristics for all successfully abstracted charts and for charts in which colonoscopy was intended or performed

### Colonoscopy case

We evaluated 14 definitions of a colonoscopy case (Figure 1, Figure 3). The most sensitive definition was the base colonoscopy code, Z555A, with or without 1 or more additional E codes. However, this definition was also the least specific. The most accurate definition was the base code plus the code “to hepatic flexure” (E741A), with or without additional E codes. Positive predictive values exceeded 95% for all definitions (Table 3).

![Figure 3:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/6/3/E330/F3.medium.gif)

[Figure 3:](http://www.cmajopen.ca/content/6/3/E330/F3)

Figure 3: 
Receiver operating curve of the 14 definitions for colonoscopy case. Upper left point of curve corresponds to Ontario Health Insurance Plan (OHIP) definition 6: OHIP codes Z555A + E741A ± other E codes.

View this table:
[Table 3:](http://www.cmajopen.ca/content/6/3/E330/T3)

Table 3: 
Positive predictive values of OHIP definition of colonoscopy, nonhospital setting, colonoscopy completeness, anesthesiologist assistance and polypectomy compared to reference standards

### Nonhospital clinics

All 3 administrative data definitions for nonhospital clinic setting were found to have sensitivity, specificity and positive predictive values in excess of 95% (Figure 4, Table 3). The first 2 definitions applied criteria using OHIP codes alone or Canadian Institute for Health Information codes alone; when used simultaneously, there was minor loss of sensitivity.

![Figure 4:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/6/3/E330/F4.medium.gif)

[Figure 4:](http://www.cmajopen.ca/content/6/3/E330/F4)

Figure 4: 
Sensitivity and specificity of nonhospital setting, colonoscopy completeness, anesthesiologist assistance and polypectomy compared to reference standards. Note: CI = confidence interval, CIHI = Canadian Institute for Health Information, OHIP = Ontario Health Insurance Plan.

### Colonoscopy completeness

The sensitivity and specificity for colonoscopy completeness differed depending on the definition of colonoscopy case and whether they were weighted or unweighted (Figure 4). Regardless of the colonoscopy case definition, the weighted sensitivity exceeded 95% and the weighted specificity was poor (< 80%). All unweighted estimates were about 95% or higher, with 1 exception: the unweighted sensitivity using the most sensitive colonoscopy definition was 70.2%. Positive predictive values exceeded 95% for all definitions (Table 3).

### Anesthesiologist assistance

The sensitivity and specificity of the administrative data definition for anesthesiologist-assisted colonoscopy exceeded 95% with either presence of anesthesiologist record or the use of propofol as a sedating agent (Figure 4) as the reference standard. Positive predictive values were less robust (83.8% and 77.2%, respectively) (Table 3).

### Polypectomy

Three administrative data definitions of polypectomy were compared to 2 reference standards, polyp documented by endoscopist and histologic confirmation of adenoma, or sessile serrated adenoma/polyp documented (Figure 4). Using more codes improved sensitivity but worsened specificity. These definitions were more sensitive but less specific when the reference standard was histologic confirmation of adenoma versus polyp seen/removed. Positive predictive values were higher when the reference standard was polyp documented versus histologic confirmation of adenoma (Table 3).

## Interpretation

We found that health administrative data definitions of colonoscopy case, colonoscopy setting and anesthesiologist assistance performed well compared to the medical record. The weighted definitions of colonoscopy completeness were sensitive but not specific. The definitions of polypectomy performed less well for the identification of the more clinically relevant reference standard, histologically significant polyps, than they did for polyp documented by endoscopist.

In the current study, the most accurate definition of colonoscopy incorporated codes indicating that the endoscopist reached the hepatic flexure. However, in practice, other definitions may be used depending on context, e.g., for studies on perforation (as the procedure may be aborted if perforation is recognized).7 Li and colleagues32 also found that health administrative data identified colonoscopy procedures accurately.

The weighted and unweighted results for colonoscopy completeness differed. The weighted results should be more valid than the unweighted results as long as the distribution of procedures in our sampling frame is representative of the distribution in the underlying population. The weighted results for colonoscopy completeness indicate that the administrative data definitions are sensitive but less specific, which would occur if the endoscopist billed for a complete colonoscopy procedure but, in fact, the procedure was not complete. In a study of 15 168 colonoscopy procedures in which Medicare claims were matched to records in the Clinical Outcomes Research Initiative database, the Medicare data also failed to identify incomplete colonoscopy procedures accurately.33

We found that the administrative data definition comprising codes for removal of a polyp 3 mm or larger, for fulguration and for removal of polyps greater than 3 cm was highly specific, but the sensitivity was 76.9%. This specificity value is comparable to that reported in the study of Medicare data33 and in a study from Quebec;34 however, the sensitivity values reported in those 2 studies were better (92% and 86%, respectively). In the current study, this administrative data definition performed reasonably well for the clinically relevant reference standard of adenoma or sessile serrated adenoma/polyp, although the false-negative and false-positive rates were 20% and 13%, respectively.

### Strengths and limitations

Strengths of our study include chart abstraction by trained, blinded abstractors who used a standardized data collection tool, a large sample that was representative of regional and facility differences, and use of a rigorous sampling strategy that randomly selected institutions and procedures and ensured adequate inclusion of rare events. However, the latter approach, which necessitates the use of weighted cases, also introduces a potential limitation, as inaccuracies may occur if the selected cases are not representative of the underlying sampling frame. Although we sampled procedures from 2008/09, our findings remain relevant, as recent studies on colonoscopy in Ontario used data from this year and earlier years.7,35 In addition, although some new codes for anesthesiologist assistance and colonoscopy (by colonoscopy indication) have been introduced, none of the codes we validated have been removed, and the structure of the codes remains the same (see Supplementary Table A3, Appendix 1, for an explanation of the changes). We would expect minimal effect on the test characteristics reported above as long as these new codes are included where relevant in the administrative data definitions above. As some of the clinics we approached did not agree to participate, there is a risk of volunteer or selection bias if the nonparticipating clinics were more likely than the participating clinics to code inaccurately, which would make our algorithms less accurate. However, as the nonparticipating clinics would have contributed a small number of colonoscopy procedures to the total sample, any effect would have been small. Cases were excluded from analysis if the medical record could not be located, if the endoscopist procedure note could not be located at the time of chart abstraction, if the dates did not match or if records could not be linked to administrative data, and cases were excluded from the polypectomy analyses if the pathology report could not be located. Given the small proportion of cases in which these records were missing, we expect that the impact on our findings is small.

### Conclusion

We rigorously validated 5 colonoscopy data elements that are routinely used in health administrative data studies of colonoscopy, for quality-assurance purposes and to guide health policy. Validation enhances the credibility and transparency of studies using health administrative data, which is important for uptake of findings36 by scientific and policy consumers and for application of algorithms by other scientists.28 We expect that our findings will be of interest to scientists and policymakers interested in using health administrative data to study and improve the quality of colonoscopy.

## Acknowledgements

The authors acknowledge Refik Saskin for his analytic assistance and thank the participating nonhospital clinics for contributing their data to this study.

## Footnotes

*   **Competing interests:** Cancer Care Ontario is a provincial government agency that acts as primary advisory on matters related to cancer to the Ontario Ministry of Health and Long-Term Care. Jill Tinmouth is employed by Cancer Care Ontario as the lead scientist of Ontario’s colorectal cancer screening program, ColonCancerCheck. Nancy Baxter is employed by Cancer Care Ontario as the provincial lead for the Gastrointestinal Endoscopy Quality-Based Procedures program. Linda Rabeneck is employed by Cancer Care Ontario as the vice-president of Prevention and Cancer Control. No other competing interests were declared.

*   This article has been peer reviewed.

*   **Contributors:** Jill Tinmouth, Lawrence Paszat and Linda Rabeneck were responsible for the study conception and design. Jill Tinmouth drafted the manuscript, and Rinku Sutradhar, Ning Liu, Nancy Baxter, Lawrence Paszat and Linda Rabeneck revised it critically for important intellectual content. All of the authors analyzed and interpreted the data, gave final approval of the version to be published and agreed to be accountable for all aspects of the work.

*   **Funding:** This study was funded by the Canadian Cancer Society Research Institute and Cancer Care Ontario. It was also supported by the Institute for Clinical Evaluative Sciences, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care. Jill Tinmouth is the recipient of a Canadian Institutes of Health Research Embedded Clinician Researcher Award.

*   **Disclaimer:** The opinions, results and conclusions reported in this article are those of the authors and are independent from the funding sources. No endorsement by the Institute for Clinical Evaluative Sciences or the Ontario Ministry of Health and Long-Term Care is intended or should be inferred. Parts of this material are based on data and information compiled and provided by the Canadian Institute for Health Information (CIHI). However, the analyses, conclusions, opinions and statements expressed herein are those of the authors and not necessarily those of the CIHI.

*   **Supplemental information:** For reviewer comments and the original submission of this manuscript, please see [www.cmajopen.ca/content/6/3/E330/suppl/DC1](http://www.cmajopen.ca/content/6/3/E330/suppl/DC1).

## References

1.  Rabeneck L, Paszat LF, Hilsden RJ, et al. (2008) Bleeding and perforation after outpatient colonoscopy and their risk factors in usual clinical practice. Gastroenterology 135:1899–906.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1053/j.gastro.2008.08.058&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=18938166&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000261762200051&link_type=ISI) 

2.  Hilsden RJ, Dube C, Heitman SJ, et al. (2015) The association of colonoscopy quality indicators with the detection of screen-relevant lesions, adverse events, and postcolonoscopy cancers in an asymptomatic Canadian colorectal cancer screening population. Gastrointest Endosc 82:887–94.
    
    

3.  Bowles CJ, Leicester R, Romaya C, et al. (2004) A prospective study of colonoscopy practice in the UK today: Are we adequately prepared for national colorectal cancer screening tomorrow? Gut 53:277–83.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiZ3V0am5sIjtzOjU6InJlc2lkIjtzOjg6IjUzLzIvMjc3IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby82LzMvRTMzMC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

4.  Alharbi O, Rabeneck L, Sutradhar R, et al. (2009) A population-based analysis of outpatient colonoscopy in adults assisted by an anesthesiologist. Anesthesiology 111:734–40.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19741486&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000270210100014&link_type=ISI) 

5.  Dominitz JA, Baldwin LM, Green P, et al. (2013) Regional variation in anesthesia assistance during outpatient colonoscopy is not associated with differences in polyp detection or complication rates. Gastroenterology 144:298–306.
    
    

6.  Khiani VS, Soulos P, Gancayco J, et al. (2012) Anesthesiologist involvement in screening colonoscopy: temporal trends and cost implications in the Medicare population. Clin Gastroenterol Hepatol 10:58–64 e1.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.cgh.2011.07.005&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=21782768&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

7.  Bielawska B, Hookey LC, Sutradhar R, et al. (2018) Anesthesia assistance in outpatient colonoscopy and risk of aspiration pneumonia, bowel perforation, and splenic injury. Gastroenterology 154:77–85.e3.
    
    

8.  Wernli KJ, Brenner AT, Rutter CM, et al. (2016) Risks associated with anesthesia services during colonoscopy. Gastroenterology 150:888–94, quiz e18.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1053/j.gastro.2015.12.018&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=26709032&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

9.  Corley DA, Jensen CD, Marks AR, et al. (2014) Adenoma detection rate and risk of colorectal cancer and death. N Engl J Med 370:1298–306.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1056/NEJMoa1309086&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=24693890&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000333830000008&link_type=ISI) 

10. Rees CJ, Thomas Gibson S, Rutter MD, et al. (2016) UK key performance indicators and quality assurance standards for colonoscopy. Gut 16:1923–9.
    
    

11. Tinmouth J, Kennedy E, Baron D, et al. (2014) Colonoscopy quality assurance in Ontario: systematic review and clinical practice guideline. Can J Gastroenterol Hepatol 28:251–74.
    
    

12. Kaminski MF, Anderson J, Valori R, et al. (2016) Leadership training to improve adenoma detection rate in screening colonoscopy: a randomised trial. Gut 65:616–24.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiZ3V0am5sIjtzOjU6InJlc2lkIjtzOjg6IjY1LzQvNjE2IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby82LzMvRTMzMC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

13. Coe SG, Crook JE, Diehl NN, et al. (2013) An endoscopic quality improvement program improves detection of colorectal adenomas. Am J Gastroenterol 108:219–26.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1038/ajg.2012.417&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23295274&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

14. (2016) Quality assurance (Forzani & MacPhail Colon Cancer Screening Centre, Alberta Health Services, Edmonton) Available: [www.albertahealthservices.ca/info/Page8336.aspx](http://www.albertahealthservices.ca/info/Page8336.aspx). accessed 2016 Sept 21.
    
    

15. Quality Management Partnership Colonoscopy (Cancer Care Ontario, Toronto) Available: [https://www.qmpontario.ca/colonoscopy/](https://www.qmpontario.ca/colonoscopy/). accessed 2016 Mar 10.
    
    

16. Benchimol EI, Smeeth L, Guttmann A, et al. (2015) The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS Med 12:e1001885.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1371/journal.pmed.1001885&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=26440803&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

17. Baxter NN, Goldwasser MA, Paszat LF, et al. (2009) Association of colonoscopy and death from colorectal cancer. Ann Intern Med 150:1–8.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.7326/0003-4819-150-1-200901060-00306&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19075198&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000262655100001&link_type=ISI) 

18. Bressler B, Paszat LF, Chen Z, et al. (2007) Rates of new or missed colorectal cancers after colonoscopy and their risk factors: a population-based analysis. Gastroenterology 132:96–102.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1053/j.gastro.2006.10.027&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=17241863&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000243843500016&link_type=ISI) 

19. Baxter NN, Sutradhar R, Forbes SS, et al. (2011) Analysis of administrative data finds endoscopist quality measures associated with postcolonoscopy colorectal cancer. Gastroenterology 140:65–72.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1053/j.gastro.2010.09.006&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20854818&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000285503200022&link_type=ISI) 

20. 1.  Fantuzzo J, 
    2.  Culhane DP
    
    Rothbard A (2015) in Actionable intelligence: using integrated data systems to achieve a more effective, efficient, and ethical government, Quality issues in the use of administrative data records, eds Fantuzzo J, Culhane DP (Palgrave Macmillan US, New York), pp 77–103.
    
    

21. Logan JR, Lieberman DA (2010) The use of databases and registries to enhance colonoscopy quality. Gastrointest Endosc Clin N Am 20:717–34.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20889074&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

22. De Coster C, Li B, Quan H (2008) Comparison and validity of procedures coded with ICD-9-CM and ICD-10-CA/CCI. Med Care 46:627–34.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1097/MLR.0b013e3181649439&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=18520318&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000256507000011&link_type=ISI) 

23. Groenen MJ, van Buuren HR, van Berge Henegouwen GP, et al. (2010) Validation study of automatically generated codes in colonoscopy using the endoscopic report system Endobase. Scand J Gastroenterol 45:1121–6.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20504245&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

24. Lee JK, Jensen CD, Lee A, et al. (2015) Development and validation of an algorithm for classifying colonoscopy indication. Gastrointest Endosc 81:575–82.e4.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25577596&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

25. Fisher DA, Grubber JM, Castor JM, et al. (2010) Ascertainment of colonoscopy indication using administrative data. Dig Dis Sci 55:1721–5.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1007/s10620-010-1200-y&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20393875&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

26. Quality Management Partnership (2015) Building on strong foundations: inaugural report on quality in colonoscopy, mammography and pathology, Available: [https://www.qmpontario.ca/common/pages/UserFile.aspx?fileId=350473](https://www.qmpontario.ca/common/pages/UserFile.aspx?fileId=350473). accessed 2016 Mar 10.
    
    

27. Steele LS, Glazier RH, Lin E, et al. (2004) Using administrative data to measure ambulatory mental health service provision in primary care. Med Care 42:960–5.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1097/00005650-200410000-00004&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=15377928&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000224196900004&link_type=ISI) 

28. Benchimol EI, Manuel DG, To T, et al. (2011) Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. J Clin Epidemiol 64:821–9.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.jclinepi.2010.10.006&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=21194889&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

29. Rosner B (2006) Fundamentals of biostatistics (Duxbury Press, Boston), 6th ed.
    
    

30. Casella G, Berger RL (1990) Statistical inference (Duxbury Press, Belmont (CA)).
    
    

31. Youden WJ (1950) Index for rating diagnostic tests. Cancer 3:32–5.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=15405679&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A1950UD97200004&link_type=ISI) 

32. Li X, Hilsden R, Hossain S, et al. (2012) Validation of administrative data sources for endoscopy utilization in colorectal cancer diagnosis. BMC Health Serv Res 12:358.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23062117&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

33. Ko CW, Dominitz JA, Green P, et al. (2011) Accuracy of Medicare claims for identifying findings and procedures performed during colonoscopy. Gastrointest Endosc 73:447–53.e1.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20950800&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

34. Wyse JM, Joseph L, Barkun AN, et al. (2011) Accuracy of administrative claims data for polypectomy. CMAJ 183:E743–7.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTgzLzExL0U3NDMiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzYvMy9FMzMwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

35. Murthy SK, Benchimol EI, Tinmouth J, et al. (2018) Temporal trends in postcolonoscopy colorectal cancer rates in screen-eligible persons: a population-based study. Gastrointest Endosc 87:1334.e4.
    
    

36. Morrato EH, Elias M, Gericke CA (2007) Using population-based routine data for evidence-based health policy decisions: lessons from three examples of setting and evaluating national health policy in Australia, the UK and the USA. J Public Health (Oxf) 29:463–71.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1093/pubmed/fdm065&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=17942850&link_type=MED&atom=%2Fcmajo%2F6%2F3%2FE330.atom) 

*   Copyright 2018, Joule Inc. or its licensors