Using physician billing claims from the Ontario Health Insurance Plan to determine individual influenza vaccination status: an updated validation study
=======================================================================================================================================================

* Kevin L. Schwartz
* Nathaniel Jembere
* Michael A. Campitelli
* Sarah A. Buchan
* Hannah Chung
* Jeffrey C. Kwong

## Abstract

**Background:** Owing to the absence of a vaccination registry in Ontario, administrative data are currently the best available source to determine population-based individual-level influenza vaccination status. Our objective was to validate physician billing claims for influenza vaccination in the Ontario Health Insurance Plan database against the Canadian Community Health Survey.

**Methods:** We used self-reported seasonal influenza vaccination status of Ontario residents surveyed between 2007 and 2009 as the reference standard. The survey responses were linked to physician claims database records to validate billing codes for influenza vaccination. We calculated sensitivity, specificity, positive predictive value and negative predictive value with 95% confidence intervals (CIs). We stratified the data by several covariates and comorbidities to determine stratum-specific performance characteristics. We used these estimates to adjust an estimate of influenza vaccine effectiveness for the 2010/11 influenza season.

**Results:** For the 47 301 patients included in the analysis, the sensitivity for the billing codes was 49.8% (95% CI 49.0%-50.5%), specificity was 95.7% (95% CI 95.5%-96.0%), positive predictive value was 88.4% (95% CI 87.8%-89.0%) and negative predictive value was 74.5% (95% CI 74.0%-74.9%). Performance measures were optimized in patients aged 65 years and older, particularly those with comorbidities.

**Interpretation:** Although administrative data have limitations for determining influenza vaccination status, owing to the high positive predictive value, they are well suited for self-controlled study designs that are often used to assess vaccine safety. For studies of coverage and effectiveness, restricting the cohort to patients aged 65 years and older will minimize misclassification bias. Performance characteristics from this study can be used to mitigate misclassification bias.

Influenza continues to pose a major public health burden in Canada. It is estimated that 5%-10% of the population has a symptomatic influenza infection each year.1 Since 2000, the province of Ontario has offered free influenza vaccines to residents aged 6 months and older in a variety of settings, including physician offices, community-based public health clinics, healthcare facilities, workplaces, schools and pharmacies. However, the absence of a comprehensive vaccination registry that captures influenza vaccines delivered in all settings has hindered efforts to evaluate the program in terms of vaccine safety, effectiveness and coverage.

We previously validated physician billing claims for influenza vaccination submitted to the Ontario Health Insurance Plan (OHIP) against self-reported influenza vaccination from the Canadian Community Health Survey cycle 1.1, conducted in 2000/01.2 We found high specificity (97%) and positive predictive value (91%), moderately high negative predictive value (79%), but lower sensitivity (56%). Sensitivity was higher for adults aged 65 years and older and for patients who reported having chronic medical conditions. Previous studies have found self-reported vaccination status to be valid.3-10 The low sensitivity of physician billing claims is partially explained by patients receiving influenza vaccines outside of physician offices.2

The objective of this study was to update the previous validation with more recent data, and to estimate performance measures of OHIP billing claims for patients with a more comprehensive (and more rigorously determined) list of risk factors for serious influenza infections.

## Methods

### Study population and setting

This study included Ontario residents who responded to the Canadian Community Health Survey between Jan. 1, 2007, and Sept. 30, 2009, and who agreed to have their survey data linked with provincial health administrative data. These data sets were linked using unique encoded identifiers and analyzed at the Institute for Clinical Evaluative Sciences (ICES). We excluded patients surveyed on Oct. 1, 2009, or later because 2 vaccines were used during the 2009/10 influenza season (the monovalent pandemic A/H1N1 vaccine and the trivalent seasonal influenza vaccine), and we were unable to differentiate between them using the OHIP data because the same billing codes were used for both vaccines. Data from more recent cycles of the survey were not yet available in linked format at ICES at the time of manuscript submission. This study was approved by the institutional review board at Sunnybrook Health Sciences Centre, Toronto, Ontario.

### Data sources

#### Canadian Community Health Survey

The Canadian Community Health Survey is a national cross-sectional survey that collects health-related information on patients aged 12 years and older through telephone and in-person interviews. The first 3 iterations in 2000/01 (cycle 1.1), 2003 (cycle 2.1) and 2005 (cycle 3.1) were biennial surveys of about 130 000 respondents. In 2007, Statistics Canada changed the survey design so that data would be collected from about 65 000 respondents each year. The survey excludes people residing on aboriginal settlements, full-time members of the Canadian Armed Forces and people living in institutions (less than 3% of total population). Details of the survey methodology have been described elsewhere.11 The response rates for the 2007/08 and 2009/10 cycles were 77.6% and 73.2%, respectively. The linkage rate between Canadian Community Health Survey and ICES data was 83%.

#### Ontario Health Insurance Plan database

The OHIP database contains billing information from about 94% of Ontario's physicians.12 It excludes physicians not paid through fee-for-service methods. OHIP provides virtually the entire Ontario population with universal insurance coverage for physician services and hospital care, excluding new residents during their initial 3 months in the province.

### Outcomes

#### Influenza immunization status from the Canadian Community Health Survey

Respondents were asked "Have you ever had a flu shot?" Those who stated that they had were then asked "When did you have your last flu shot?" Respondents who specified that they had received the vaccine within the previous 12 months were then asked which month; if they answered the current month, they were asked "Was that this year or last year?" 

We classified patients who reported receiving a flu shot within the last 12 months as having been immunized as follows: those whose month of vaccination differed from the month of interview; and those whose month of vaccination matched the month of interview and was in the same year. Because the questionnaire did not determine the exact date of vaccine receipt, patients whose month of vaccination matched the month of interview, but in the previous year, may have received their vaccine more than 365 days earlier. For these specific patients, we classified those whose interview occurred during the first 15 days of the month as having been immunized and those interviewed after the first 15 days of the month as not immunized. We conducted a sensitivity analysis restricting the survey dates from Feb. 1 to Aug. 31 of each year to minimize the risk of immunization year misclassification for patients surveyed during influenza vaccination campaigns in Ontario (usually September to January).

#### Influenza immunization status from OHIP

To identify influenza vaccination status in the OHIP database, we used the billing codes for vaccination with influenza vaccines, G590 (influenza vaccination plus visit) and G591 (influenza vaccination only). We also included the tracking code Q130 (influenza vaccine tracking code), which is used when a patient has undergone vaccination elsewhere. Physicians belonging to certain remuneration plans receive financial incentives for attaining prespecified targets for influenza vaccination of their patients aged 65 years and older, and all 3 codes are included in the numerator for those calculations. Using the Canadian Community Health Survey interview date as the reference date, we considered the presence of any of the influenza vaccination codes over the previous 365 days to be actively immunized.

#### Other variables

We determined neighbourhood income quintile using residential postal codes, and defined rural residence as a community size of fewer than 10 000 residents. Having a regular physician was determined from the Canadian Community Health Survey question "Do you have a regular medical doctor?" We evaluated patients for the presence of a number of potential risk factors for serious influenza infection, including chronic cardiovascular diseases (congestive heart failure, history of acute myocardial infarction or acute ischemic stroke, and hypertension), chronic respiratory diseases (asthma and chronic obstructive pulmonary disease), diabetes, chronic kidney disease, cancer, immunosuppression (resulting from infection with HIV or from immunosuppressive therapies), dementia, morbid obesity (body mass index > 40 calculated from the height and weight provided in the survey) and pregnancy (derived from the MOMBABY database). Most of these conditions were defined using previously validated algorithms applied to administrative data sets housed at ICES, including the OHIP database, the Canadian Institute for Health Information Discharge Abstract Database, the National Ambulatory Care Reporting System, the Same Day Surgery database, the Canadian Organ Replacement Register, the Ontario Renal Reporting System, the Ontario Diabetes Database, the Ontario Cancer Registry, the Ontario Myocardial Infarction Database, the Chronic Obstructive Pulmonary Disease database, the Ontario Drug Benefits database, the Ontario Congestive Heart Failure database, the Ontario Hypertension Database and the Ontario HIV database.13-29 These databases and the definitions used are described in Appendix 1 (available at [www.cmajopen.ca/content/4/3/E463/suppl/DC1](http://www.cmajopen.ca/content/4/3/E463/suppl/DC1)).

### Statistical analyses

We set self-reported influenza immunization status from the Canadian Community Health Survey as the reference standard. We calculated performance measures (sensitivity, specificity, positive predictive value and negative predictive value) with 95% confidence intervals (CIs) for OHIP physician billing claims for influenza vaccination. We stratified the results by survey cycle, age group, sex, rural versus urban residence, having a regular physician and presence of risk factors for serious influenza infections. We further stratified some of these groups by age (< 65 v. ≥ 65 yr). Statistical analyses were conducted using SAS Enterprise Guide version 6.1 (SAS Institute Inc., Cary, NC).

### Application example

To illustrate the applicability of our results, we used the values for sensitivity and specificity to correct the bias arising from misclassification of influenza vaccination status based on OHIP physician billing claims. We applied a SAS macro developed by Fox and associates30 to results from a previous study by Kwong and associates that assessed vaccine effectiveness against admissions to hospital for laboratory-confirmed influenza among older adults during the 2010/11 influenza season.31 This macro uses a probabilistic method for conducting a sensitivity analysis using individual-level data. Using the overall sensitivity and specificity from this study's results for influenza immunization status in Ontario patients aged 65 years and older, we calculated vaccine effectiveness corrected for misclassification of the exposure variable (i.e., influenza vaccination).30

## Results

There were a total of 48 426 survey responses, of which 1122 were excluded for either refusal or an inability to answer the influenza vaccine question and 3 were excluded for invalid birthdates, leaving 47 301 Ontarians included in our analysis (Table 1). Based on survey results, about 40% of patients reported having undergone vaccination against influenza, ranging from less than 25% among patients aged less than 50 years to 68% among those aged 65 years and older. Vaccine coverage was higher among women, older adults, patients with a regular physician and patients with risk factors for serious influenza infection, except for pregnancy.

View this table:
[Table 1:](http://www.cmajopen.ca/content/4/3/E463/T1)

Table 1:   Demographic characteristics of the study population and percentage receiving influenza vaccine within the previous 12 months

The combined sensitivity for influenza OHIP billing codes was 49.8% (95% CI 49.0%-50.5%), specificity was 95.7% (95% CI 95.5%-96.0%), positive predictive value was 88.4% (95% CI 87.8%-89.0%) and negative predictive value was 74.5% (95% CI 74.0%-74.9%) (Table 2). The sensitivity ranged from 20.3% in adolescents (aged 12-17 yr) to 68.9% in patients aged 65 years and older, whereas specificity was high for those less than 65 years of age (≥ 96.0%) and declined to 82.7% for those aged 65 years and older. Similarly, positive predictive value increased with age, whereas negative predictive value decreased.

View this table:
[Table 2:](http://www.cmajopen.ca/content/4/3/E463/T2)

Table 2:   Performance measures of Ontario Health Insurance Plan physician billing claims compared with self-reported influenza vaccination using Canadian Community Health Survey data

Having access to a regular physician substantially improved the sensitivity of OHIP influenza vaccine billing codes, but with some decrease in specificity. The validity of the OHIP influenza immunization codes was fairly consistent across a variety of risk factors, as long as the cohort was restricted to patients aged 65 years and older. For chronic conditions, the sensitivity ranged from 68.9% to 74.3% and dropped to 60.5% in patients without any comorbidities. The specificity ranged from 73.8% to 90.0%. The sensitivity decreased for all conditions to 40.4%-57.4% in patients less than 65 years of age, but remained significantly higher than for younger patients without any comorbidities (29.1%). The specificity was high across all comorbid conditions in the younger cohort. The positive predictive value was high for all groups except those aged 12-17 years.

In the sensitivity analysis restricted to those who were surveyed between February and August, the overall positive predictive value increased from 88.4% to 93.2% and the specificity increased marginally from 95.7% to 97.5% (Table 3). The improvements in both specificity and positive predictive value were seen in all subgroups. Results for patients aged less than 65 years who have risk factors for serious influenza infection are not presented owing to the presence of numerous small cells (i.e., cell size < 6 patients).

View this table:
[Table 3:](http://www.cmajopen.ca/content/4/3/E463/T3)

Table 3:   Performance measures of Ontario Health Insurance Plan physician billing claims compared with self-reported influenza vaccination using Canadian Community Health Survey data, restricted to patients surveyed between Feb. 1 and Aug. 31

We incorporated our results into a misclassification bias adjustment sensitivity analysis to show the utility of these results when using administrative data for determining individual-level influenza immunization status for vaccine effectiveness studies. We input a sensitivity of 68.6% and a specificity of 89.9% (from Table 3). Figure 1 shows a significant underestimation of influenza vaccine effectiveness for the 2010/11 season before adjusting for the misclassification of immune status. Vaccine effectiveness increased from 42% (95% CI 29%-53%) to 68% (95% CI 61%-78%) after adjustment.

![Figure 1](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/4/3/E463/F1.medium.gif)

[Figure 1](http://www.cmajopen.ca/content/4/3/E463/F1)

Figure 1 
Influenza vaccine effectiveness estimates for the 2010/11 season, with 95% confidence intervals, from Kwong and associates31 before and after adjustment for misclassification bias using the validation results from this study.

## Interpretation

We found that OHIP billing claims had only moderate performance characteristics to correctly identify influenza vaccination status in Ontario, compared with self-report. For children and adults less than 65 years of age, sensitivity was under 50%, but specificity was greater than 90%. Among those aged 65 years and older, sensitivity was higher, but with lower specificity. The sensitivity was generally higher for patients with comorbid conditions and those with a regular physician. These subpopulations had the most accurate OHIP influenza vaccination billing claims. The performance characteristics were better when restricted to Canadian Community Health Survey respondents who were surveyed between February and August, suggesting the presence of some misclassification by influenza season when including respondents surveyed during months when influenza vaccines are generally given.

There are a number of potential explanations for the low sensitivity of OHIP billing claims. A substantial minority of individuals are vaccinated outside of physician offices, and we would not expect their vaccinations to be captured in health administrative data, despite the existence of an influenza vaccine tracking code. These people include those who received the vaccine at workplaces, schools or public health clinics. In 2012, pharmacists began providing vaccination, and these are captured in the Ontario Drug Benefits database, which may improve the performance of Ontario administrative data in subsequent years. In addition, remuneration per vaccination is low (ranging from $0.68 to $9.60, depending on the family practice funding model), possibly resulting in missed billings.

The lower specificity in the older population is more difficult to explain. It is possible that proportionally more older adults forget while responding to the survey that they had received the influenza vaccine that year. However, previous studies have found self-reported influenza vaccination status to be reasonably valid.3-9 Alternatively, billing errors or medical fraud could explain a proportion of the false-positive results.

### Limitations

The Canadian Community Health Survey excludes children younger than 12 years and older adults living in long-term care facilities. These are important high-risk groups to include in studies, and it is unfortunate that we are unable to quantify the validity of influenza vaccination in these groups. However, our study does characterize the validity of influenza vaccination, as captured by administrative data, in virtually all other high-risk groups.

We used self-report as the reference standard in this analysis, and although verification of responses was not possible, 8 previous validation studies comparing self-report to medical records suggest that sensitivity of self-report is high (86%-100%); both specificity and positive predictive value are more variable, but are generally lower (38%-98% and 62%-96%, respectively).3-10 However, the specificity and positive predictive value of self-report may be artificially reduced when using medical records as the reference standard if patients can receive influenza vaccine through alternative providers (e.g., workplaces, pharmacies).

### Conclusion

This study updates the performance characteristics from our previous study,2 with a much larger sample size, more recent iterations of the Canadian Community Health Survey and a far more extensive list of risk factors for serious influenza infection defined using validated methods. We quantified the sensitivity, specificity, positive predictive value and negative predictive value across a variety of variables, including multiple high-risk influenza groups. These results can be used to correct for underdetermination of vaccine coverage levels at the aggregate level, and to account for misclassification bias of vaccination status at the individual level (e.g., in studies of influenza vaccine effectiveness). We have shown the importance of quantifying misclassification bias, with substantial underestimation of influenza vaccine effectiveness when using OHIP physician billing claims data to determine vaccination status. Nondifferential misclassification is generally expected to bias results toward the null hypothesis and thus underestimate effect sizes. However, this may not always be true; therefore it is important to quantify the degree of systematic error in observational studies.30 In addition, the high positive predictive value and specificity suggests that the database can accurately identify patients who have truly undergone vaccination, allowing these data to be used to study influenza vaccine safety using self-controlled study designs.32

In the absence of a vaccination registry in Ontario, administrative data represent the best available data source to study influenza vaccines at a population level. However, we fully support the creation of a registry in Ontario to permit optimal evaluations of our publicly funded vaccination programs, particularly since vaccinations given at public health and workplace clinics are not captured by physician billing claims data. Despite the limitations of administrative data, the results of this study will enable adjustments for systematic error in future studies.

### Supplemental information

For reviewer comments and the original submission of this manuscript, please see [www.cmajopen.ca/content/4/3/E463/suppl/DC1](http://www.cmajopen.ca/content/4/3/E463/suppl/DC1)

## Footnotes

*   **Competing interests:** None declared.

*   **Contributors:** All of the authors contributed substantially to the conception, design, data analysis plan and interpretation of the data. In addition, Nathaniel Jembere performed the data analysis and Kevin Schwartz drafted the manuscript. All of the authors critically reviewed the manuscript for important intellectual content, approved the final version to be published and agreed to act as guarantors of the work.

*   **Acknowledgements:** This work was supported by the Canadian Institutes of Health Research (CIHR) (grant no. MOP 130568). Kevin Schwartz is supported by a Fellowship Award from CIHR. Jeffrey Kwong is supported by a New Investigator Award from CIHR and a Clinician Scientist Award from the University of Toronto Department of Family and Community Medicine. The study sponsors did not participate in: the design and conduct of the study; collection, management, analysis and interpretation of the data; preparation, review or approval of the manuscript; or the decision to submit the manuscript for publication. 

*   **Disclaimer:** Parts of this material are based on data and information compiled and provided by the Canadian Institute for Health Information (CIHI) and Cancer Care Ontario (CCO). However, the analyses, conclusions, opinions and statements expressed herein are those of the authors, and not necessarily those of CIHI or CCO. This study was supported by the Institute for Clinical Evaluative Sciences (ICES) which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The opinions, results and conclusions reported in this article are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intented or should be inferred.

## References

1.  Kuster SP, Shah PS, Coleman BL, et al. (2011) Incidence of influenza in healthy adults and healthcare workers: a systematic review and meta-analysis. PLoS One 6:e26239.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1371/journal.pone.0026239&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22028840&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 

2.  Kwong JC, Manuel DG (2007) Using OHIP physician billing claims to ascertain individual influenza vaccination status. Vaccine 25:1270–4.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.vaccine.2006.10.004&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=17079057&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000244016600016&link_type=ISI) 

3.  Hutchison BG (1989) Measurement of influenza vaccination status of the elderly by mailed questionnaire: response rate, validity and cost. Can J Public Health 80:271–5.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=2790635&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A1989AQ30100008&link_type=ISI) 

4.  Mac Donald R, Baken L, Nelson A, et al. (1999) Validation of self-report of influenza and pneumococcal vaccination status in elderly outpatients. Am J Prev Med 16:173–7.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S0749-3797(98)00159-7&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=10198654&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000079229400002&link_type=ISI) 

5.  Martin LM, Leff M, Calonge N, et al. (2000) Validation of self-reported chronic conditions and health services in a managed care population. Am J Prev Med 18:215–8.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S0749-3797(99)00158-0&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=10722987&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000086044700004&link_type=ISI) 

6.  Zimmerman RK, Raymund M, Janosky JE, et al. (2003) Sensitivity and specificity of patient self-report of influenza and pneumococcal polysaccharide vaccinations among elderly outpatients in diverse patient care strata. Vaccine 21:1486–91.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S0264-410X(02)00700-4&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=12615445&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000182007800023&link_type=ISI) 

7.  Skull SA, Andrews RM, Byrnes GB, et al. (2007) Validity of self-reported influenza and pneumococcal vaccination status among a cohort of hospitalized elderly inpatients. Vaccine 25:4775–83.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.vaccine.2007.04.015&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=17499402&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000247641000004&link_type=ISI) 

8.  Rolnick SJ, Parker ED, Nordin JD, et al. (2013) Self-report compared to electronic medical record across eight adult vaccines: do results vary by demographic factors? Vaccine 31:3928–35.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.vaccine.2013.06.041&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23806243&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000324444200026&link_type=ISI) 

9.  Mangtani P, Shah A, Roberts JA (2007) Validation of influenza and pneumococcal vaccine status in adults based on self-report. Epidemiol Infect 135:139–43.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1017/S0950268806006479&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=16740194&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 

10. Irving SA, Donahue JG, Shay DK, et al. (2009) Evaluation of self-reported and registry-based influenza vaccination status in a Wisconsin cohort. Vaccine 27:6546–9.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.vaccine.2009.08.050&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19729083&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000272056300008&link_type=ISI) 

11. Béland Y (2002) Canadian community health survey-methodological overview. Health Rep 13:9–14.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=12743953&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 

12. Chan B (1999-2000) Supply of physicians' services in Ontario. Hosp Q 3:17.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=11570359&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 

13. Fleet JL, Dixon SN, Shariff SZ, et al. (2013) Detecting chronic kidney disease in population-based administrative databases using an algorithm of hospital encounter and physician claim codes. BMC Nephrol 14:81.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/1471-2369-14-81&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23560464&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 

14. Quinn RR, Laupacis A, Austin PC, et al. (2010) Using administrative datasets to study outcomes in dialysis patients: a validation study. Med Care 48:745–50.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1097/MLR.0b013e3181e419fd&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20613656&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000280115700012&link_type=ISI) 

15. Lam NN, McArthur E (2015) Kim SJ, et al. Validation of kidney transplantation using administrative data. Can J Kidney Health Dis 2:20.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/s40697-015-0054-9&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=26019887&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 

16. Hux JE, Ivis F, Flintoft V, et al. (2002) Diabetes in Ontario: determination of prevalence and incidence using a validated administrative data algorithm. Diabetes Care 25:512–6.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiZGlhY2FyZSI7czo1OiJyZXNpZCI7czo4OiIyNS8zLzUxMiI7czo0OiJhdG9tIjtzOjIwOiIvY21ham8vNC8zL0U0NjMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

17. Starfield B, Weiner J, Mumford L, et al. (1991) Ambulatory care groups: a categorization of diagnoses for research and management. Health Serv Res 26:53–74.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=1901841&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A1991FK09100004&link_type=ISI) 

18. Hall S, Schulze K, Groome P, et al. (2006) Using cancer registry data for survival studies: the example of the Ontario Cancer Registry. J Clin Epidemiol 59:67–76.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.jclinepi.2005.05.001&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=16360563&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000234451600010&link_type=ISI) 

19. Austin PC, Daly PA, Tu JV (2002) A multicenter study of the coding accuracy of hospital discharge administrative data for patients admitted to cardiac care units in Ontario. Am Heart J 144:290–6.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1067/mhj.2002.123839&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=12177647&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000177501400017&link_type=ISI) 

20. Schultz SE, Rothwell DM, Chen Z, et al. (2013) Identifying cases of congestive heart failure from administrative data: a validation study using primary care patient records. Chronic Dis Inj Can 33:160–6.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23735455&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 

21. Gill SS, Anderson GM, Fischer HD, et al. (2009) Syncope and its consequences in patients with dementia receiving cholinesterase inhibitors: a population-based cohort study. Arch Intern Med 169:867–73.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1001/archinternmed.2009.43&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19433698&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000265994000007&link_type=ISI) 

22. Juurlink D, Preyra C, Croxford R, et al. (2006) Canadian Institute for Health Information Discharge Abstract Database: a validation study. Toronto: Institute for Clinical Evaluative Sciences.
    
    

23. Gurevich Y, McFarlane A, Morris K, et al. (2010) Estimating the number of coronary artery bypass graft and percutaneous coronary intervention procedures in Canada: a comparison of cardiac registry and Canadian Institute for Health Information data sources. Can J Cardiol 26:e249–53.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S0828-282X(10)70416-5&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20847972&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000277667400004&link_type=ISI) 

24. Lee DS, Tu JV, Austin PC, et al. (2007) Effect of cardiac and noncardiac conditions on survival after defibrillator implantation. J Am Coll Cardiol 49:2408–15.
    
    [FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo0OiJhY2NqIjtzOjU6InJlc2lkIjtzOjEwOiI0OS8yNS8yNDA4IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby80LzMvRTQ2My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

25. Gershon AS, Wang C, Guan J, et al. (2009) Identifying patients with physician-diagnosed asthma in health administrative databases. Can Respir J 16:183–8.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1155/2009/963098&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20011725&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000272936200004&link_type=ISI) 

26. Gershon AS, Wang C, Guan J, et al. (2009) Identifying individuals with physcian diagnosed COPD in health administrative databases. COPD 6:388–94.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1080/15412550903140865&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19863368&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000278276700009&link_type=ISI) 

27. J aakkimainen RL, , Bronskill SE, , Tierney MC, , , et al, . , . , . . Identification of physician-diagnosed Alzheimer's disease and related dementias in population-based administrative data: a validation study using family physicians' electronic medical records. J Alzheimer Dis.
    
    

28. Antoniou T, Zagorski B, Loutfy MR, et al. (2011) Validation of case-finding algorithms derived from administrative data for identifying adults living with human immunodeficiency virus infection. PLoS One 6:e21748.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1371/journal.pone.0021748&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=21738786&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 

29. Tu K, Campbell NR, Chen ZL, et al. (2007) Accuracy of administrative databases in identifying patients with hypertension. Open Med 1:e18–26.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20101286&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 

30. Fox MP, Lash TL, Greenland S (2005) A method to automate probabilistic sensitivity analyses of misclassified binary variables. Int J Epidemiol 34:1370–6.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiaW50amVwaWQiO3M6NToicmVzaWQiO3M6OToiMzQvNi8xMzcwIjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby80LzMvRTQ2My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

31. Kwong JC, Campitelli MA, Gubbay JB, et al. (2013) Vaccine effectiveness against laboratory-confirmed influenza hospitalizations among elderly adults during the 2010-2011 season. Clin Infect Dis 57:820–7.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiY2lkIjtzOjU6InJlc2lkIjtzOjg6IjU3LzYvODIwIjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby80LzMvRTQ2My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

32. Kwong JC, Vasa PP, Campitelli MA, et al. (2013) Risk of Guillain-Barré syndrome after seasonal influenza vaccination and influenza health-care encounters: a self-controlled study. Lancet Infect Dis 13:769–76.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S1473-3099(13)70104-X&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23810252&link_type=MED&atom=%2Fcmajo%2F4%2F3%2FE463.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000323465900022&link_type=ISI) 

*   Copyright 2016, Joule Inc. or its licensors