Pelvic floor disorders in women who consult primary care clinics: development and validation of case definitions using primary care electronic medical records
==============================================================================================================================================================

* Sue Ross
* Hilary Fast
* Stephanie Garies
* Deb Slade
* Dave Jackson
* Meghan Doraty
* Rebecca Miyagishima
* Boglarka Soos
* Matt Taylor
* Tyler Williamson
* Neil Drummond

## Abstract

**Background:** To date, there has been no validated method to identify cases of pelvic floor disorders in primary care electronic medical record (EMR) data. We aimed to develop and validate symptom-based case definitions for urinary incontinence, fecal incontinence and pelvic organ prolapse in women, for use in primary care epidemiologic or clinical research.

**Methods:** Our retrospective study used EMR data from the Southern Alberta Primary Care Research Network (SAPCReN) and the Canadian Primary Care Sentinel Surveillance Network (CPCSSN) in southern Alberta. Trained researchers remotely reviewed a random sample of EMR charts of women aged 18 years or older from 6 rural and urban clinics to validate case definitions for urinary incontinence, fecal incontinence and pelvic organ prolapse. We calculated sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV), and estimated SAPCReN prevalence as appropriate.

**Results:** Charts of 900 women were included. Sensitivity was 81.9% (95% confidence interval [CI] 75.1–87.2) for urinary incontinence, 61.2% (95% CI 46.2–74.5) for fecal incontinence, and 51.8% (95% CI 40.6–62.8) for pelvic organ prolapse. Corresponding specificity values were 71.9% (95% CI 68.4–75.1), 99.2% (95% CI 98.2–99.6) and 98.8% (95% CI 97.7–99.4), PPVs 40.6% (95% CI 35.4–46.0), 81.1% (95% CI 64.3–91.4) and 81.1% (95% CI 67.6–90.1), and NPVs 94.4% (95% CI 92.1–96.1), 97.8% (95% CI 96.5–98.6) and 95.3% (95% CI 93.6–96.6). The SAPCReN-observed prevalence for urinary incontinence was 29.7% (95% CI 29.3–30.0), but the adjusted prevalence was 2.97%.

**Interpretation:** The case definition for urinary incontinence met our standard for validity (sensitivity and specificity > 70%), and the case definitions for fecal incontinence and pelvic organ prolapse had PPVs greater than 80%. The urinary incontinence definition may be used in epidemiologic research, and those for fecal incontinence and pelvic organ prolapse may be used in quality-improvement studies or creation of disease registries. Our symptom-based case definitions could also be adapted for research in other EMR settings.

Women’s pelvic floor disorders such as urinary or fecal incontinence and prolapse of pelvic organs (bladder, intestines or uterus) into the vagina occur when muscles and connective tissue are unable to provide sufficient support. Pelvic floor disorders are common, being reported to affect as many as 40% of women, with symptoms that may be embarrassing and disabling.1,2 The prevalence of pelvic floor disorders is increasing owing to the aging population with a longer lifespan,1,3,4 and health policy-makers and health care providers must be prepared to manage these conditions more frequently and effectively as budgets and services become more constrained.3,5–8 No previously validated methods were available to us to estimate the current burden of pelvic floor disorders in Canadian primary care that would enable providers or policy-makers to predict the types and extent of future service demand.

We wished to investigate the prevalence of pelvic floor disorders in primary care using data from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN), a national network of primary care clinics. In collaboration with the Southern Alberta Primary Care Research Network (SAPCReN), we then used the CPCSSN methods9 with the aim of developing and validating case definitions for pelvic floor disorders in women who consult primary care providers.

## Methods

This was a 2-part retrospective study. In the first part, we used the regional SAPCReN–CPCSSN database to develop and refine case definition algorithms to identify women with urinary incontinence, fecal incontinence or pelvic organ prolapse.

In the second part, we investigated the diagnostic accuracy of the case definition algorithms compared with the reference standard of detailed review of primary care electronic medical record (EMR) charts. Validated CPCSSN case definitions were already available for 8 chronic conditions: chronic obstructive pulmonary disease, dementia, depression, diabetes, epilepsy, hypertension, osteoarthritis and parkinsonism. 9 The definitions were based on combinations of codes in the EMR data fields: encounter diagnoses, “problem list,” billing, medications prescribed, referrals made and physiologic data (such as height, weight and blood pressure). We also investigated the prevalence of these disorders using the CPSSN data.

### Setting

In Canada’s universal publicly funded health care system, family medicine clinics provide the first point of contact for patients seeking health care, including women concerned about their pelvic floor disorders.2 Many primary care clinics use EMRs to record patient encounters and support all aspects of clinical care.10 The CPCSSN was established in 2005 to provide a data source to support clinical research and national chronic disease surveillance.

### Development of case definition algorithms

Our population for the development phase was 55 335 female patients aged 18 years or older registered in the SAPCReN–CPCSSN database at the time of the initial data extraction for this study, in December 2014.

We based the preliminary case definitions of pelvic floor disorders on previous work that outlined relevant *International Classification of Diseases, 9th Revision* (clinical modification) (ICD-9-CM) diagnostic, procedural codes and textual descriptors.5,7,11 We supplemented these codes with further textual and ICD-9 descriptors and Anatomical Therapeutic Chemical pharmacological codes12 for medications used for “urinary frequency and incontinence.” The preliminary definitions were reviewed for face validity by a urogynecology clinic primary care physician (D.S.) and revised as necessary. The algorithms were applied to the relevant fields within the SAPCReN–CPCSSN database.

The final case definition algorithms incorporated 3 elements: ICD-9-CM diagnostic codes, textual descriptors and relevant drugs (Appendix 1, available at [www.cmajopen.ca/content/8/2/E414/suppl/DC1](http://www.cmajopen.ca/content/8/2/E414/suppl/DC1)). These generated prevalence estimates were similar to the existing evidence and were thus selected for validation.

### Validation study

Validation of the proposed case definition algorithms tested the accuracy of the algorithms compared with detailed review of the primary care EMR chart (reference standard). Consenting SAPCReN physicians working in 6 rural and urban clinics across southern Alberta allowed reviewers (S.R., H.F., D.S., M.D., R.M.) to access their patients’ EMR charts remotely. Two different EMR system vendors were represented. From each physician’s panel, a random sample of eligible women (aged ≥ 18 yr) was identified. Women were excluded if they did not have a valid personal health number (a unique identifier for all residents in Alberta registered with the provincial health care insurance plan), if they had died, or if their care provider no longer participated in SAPCReN–CPCSSN.

We developed a data collection tool to facilitate EMR data abstraction and ensure completeness of each review. The tool included all the specific terms described in Appendix 1, with check boxes for each item. Five reviewers took part in the chart review: 3 principal reviewers (S.R., H.F., R.M.), 1 expert arbitrator (D.S.) and an assistant reviewer (M.D.). Reviewers were blinded to each other’s as well as the algorithms’ designation of charts as cases or noncases. The entire EMR chart was reviewed, including clinic staff notes, copies of laboratory or procedure results, and referrals attached to the patient record. Only information needed to define cases or noncases was derived from the EMR. No personally identifying patient information was recorded. All reviewers attended training sessions to ensure familiarity with the relevant EMRs, the data collection tool and application of the case definitions. Uncertain cases were discussed and resolved at biweekly meetings involving all reviewers, with the expert knowledge of the urogynecology clinic primary care physician (D.S.) guiding consensus.

For each of the 3 conditions (any urinary incontinence, any fecal incontinence and any pelvic organ prolapse), women were classified as cases if any of the descriptor elements were present anywhere in their EMR data, and as noncases if the elements were absent. For each woman, we used the personal health number to link the outcomes of the case definition algorithms to the results of detailed EMR chart review.

We tested interrater reliability among the 3 principal reviewers using a subsample of 10 charts from 1 specific EMR (weighted to include at least 1 case for each definition). We used the Fleiss κ statistic to determine consistency among reviewers.13

#### Sample size for validation

The estimated sample size for the study was 1000 EMR charts. Assuming a sensitivity of 80% for each of the conditions under study and prevalence estimates of 8% for urinary incontinence, 4% for fecal incontinence and 5% for pelvic organ prolapse,1 the sample of 1000 charts would ensure a margin of error for the sensitivity estimate of no more than 15% for any condition and a margin of error of no more than 10% for urinary incontinence.

#### Sampling of electronic medical record charts

Stratification was carried out for the validation study, with 10% of EMR charts selected randomly from among women less than 55 years old and 90% selected randomly from among those aged 55 years or older, in order to intentionally inflate the prevalence of pelvic floor disorders within the sample. 9 Because the expected prevalence of fecal incontinence was lower than that of the other disorders,9 we augmented the sample by selecting all 49 fecal incontinence cases remaining in the participating sites and interspersing them randomly among the sample charts.

After review of the charts at the first 3 clinics, 1 of the 2 EMR vendors unexpectedly removed a patient-specific identifier from the search interface during a system upgrade, which meant we were unable to search for our sample cases in that vendor’s charts. We generated a second sample from sites using the other vendor’s system.

The initial sample of 1050 patients was generated in December 2014, the second sample was generated in February 2015, and the EMR chart review was completed in June 2015.

### Statistical analysis

We analyzed the data using Stata (StataCorp) with the exception of the Fleiss κ statistic, which was calculated with SAS (SAS Institute). We used simple descriptive statistics to characterize the sample of patients. We assessed the validity of the case definition algorithms in terms of sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) with 95% confidence intervals (CIs) for each, according to reporting guidelines.14 Case definitions were considered valid for epidemiologic uses if the sensitivity and specificity were at least 70%. For clinical uses such as case-finding in SAPCReN–CPCSSN data, case definitions were considered valid if PPVs and NPVs were at least 70%.9 To illustrate the use of the case definitions, we estimated the prevalence of the 3 conditions (if appropriate) by applying the validated case definition algorithms to the SAPCReN–CPCSSN data for April–June 2015 for women aged 18 years or older.

### Ethics approval

Ethics approval was provided by the University of Alberta Health Research Ethics Board and University of Calgary Conjoint Health Research Committee.

## Results

We conducted the final analysis using the charts of 900 patients from the 6 primary care medicine clinic sites (Figure 1, Table 1).

![Figure 1:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/8/2/E414/F1.medium.gif)

[Figure 1:](http://www.cmajopen.ca/content/8/2/E414/F1)

Figure 1: 
Flow diagram showing selection of sample of electronic medical record (EMR) patient charts for analysis. Note: CPCSSN = Canadian Primary Care Sentinel Surveillance Network, SAPCReN = Southern Alberta Primary Care Research Network. *Anonymous data from these women were used to develop the case definitions. †Case definition algorithms for pelvic floor disorders were applied to these charts.

View this table:
[Table 1:](http://www.cmajopen.ca/content/8/2/E414/T1)

Table 1: 
Characteristics of sample for detailed review of the primary care electronic medical record chart (*n* = 900)

Interrater reliability testing showed acceptable agreement among the 3 reviewers for all 3 case definitions: there was substantial agreement for urinary incontinence (κ = 0.76, 95% CI 0.42–1.00), and excellent agreement for fecal incontinence (κ = 0.81, 95% CI 0.38–1.00) and pelvic organ prolapse (κ = 1, 95% CI 1.00–1.00).

Table 2 summarizes the diagnostic accuracy of the 3 case definition algorithms. The sensitivity was 81.9% (95% CI 75.1–87.2) for urinary incontinence, 61.2% (95% CI 46.2–74.5) for fecal incontinence and 51.8% (95% CI 40.6–62.8) for pelvic organ prolapse, and the specificity was 71.9% (95% CI 68.4–75.1), 99.2% (95% CI 98.2–99.6) and 98.8% (95% CI 97.7–99.4), respectively. The corresponding PPVs were 40.6% (95% CI 35.4–46.0), 81.1% (95% CI 64.3–91.4) and 81.1% (95% CI 67.6–90.1), and NPVs 94.4% (95% CI 92.1–96.1), 97.8% (95% CI 96.5–98.6) and 95.3% (95% CI 93.6–96.6).

View this table:
[Table 2:](http://www.cmajopen.ca/content/8/2/E414/T2)

Table 2: 
Results of validation of case definition algorithms for urinary incontinence, fecal incontinence and pelvic organ prolapse

The definition of urinary incontinence was found to be valid according to our predefined criteria (> 70% for sensitivity and specificity), so we calculated the prevalence of urinary incontinence in the complete SAPCReN–CPCSSN data for women aged 18 years or older. The observed prevalence was 23 929/80 694 (29.7%, 95% CI 29.3–30.0). Because of concern about levels of misclassification suggested by the sensitivity and specificity values, we calculated the Rogan–Gladen estimator,15 which yielded an adjusted “true” prevalence of 2.97%.

We did not calculate the prevalence of fecal incontinence and pelvic organ prolapse for the complete SAPCReN data because neither of these definitions met our predefined characteristics of validity for epidemiologic purposes. They met the predefined characteristics of validity for clinical uses such as case-finding.

## Interpretation

With random selection of primary care patients from SAPCReN physicians, our results show that the SAPCReN–CPCSSN database may be used to apply complex symptom-based case definitions for pelvic floor disorders (urinary incontinence, fecal incontinence and pelvic organ prolapse), incorporating a variety of textual and coded items to identify cases for epidemiologic research. High PPVs and NPVs for the case definitions for fecal incontinence and pelvic organ prolapse suggest their suitability for clinical applications, such as quality improvement or creation of disease registries, or establishing research cohorts for observational studies.16 Acceptable values for sensitivity and specificity mean that our definition of urinary incontinence may be used to estimate the prevalence of urinary incontinence, bearing in mind that it may be necessary to apply relevant statistical methods (such as the Rogan–Gladen estimator15 or a Bayesian approach17) to adjust for possible misclassification.

In studies of US ambulatory care administrative databases, investigators used case definitions based on ICD-9 diagnoses and billing codes alone, without evidence of definitional validity. 5,7,11 In contrast, our study reports the measurement characteristics of our case definitions.

With our validated case definition for urinary incontinence, the observed prevalence among women was 29.7% (95% CI 29.3–30.0); adjustment for possible misclassification error with the Rogan–Gladen estimator15 yielded a true prevalence as 2.97%. It is likely that the actual prevalence is somewhere between those estimates, which casts doubt on higher published estimates of urinary incontinence prevalence. Both our estimates are below the range of published population prevalence estimates (30%–60%1,3), which often rely on self-reported symptom questionnaires in selected populations. Our findings reflect the situation for primary care patients, who must overcome their discomfort about discussing these topics and consult a primary care provider3,18 in order to be identified as having pelvic floor disorders. The care provider must also judge that the women’s concerns are sufficiently relevant to record them in the EMR. Thus, only women with problematic symptoms would be likely to appear in the SAPCReN–CPCSSN database, fewer than might report symptoms in an anonymous questionnaire.2,3,18

Data from the CPCSSN database have been used to study organization of primary care19–23 and to describe prevalence and risks associated with specific conditions.24,25 Our study provides a foundation for research into the prevalence and treatment of pelvic floor disorders in women and the increasing burden of pelvic floor disorders.1,3,4 This work will be useful for estimating and predicting workload associated with women’s pelvic floor disorders, for example, enabling clinics to predict the need for additional incontinence staff.

Our case definitions may be used to estimate the prevalence of urinary incontinence in other CPCSSN data sets and to develop practice disease registries for managing fecal incontinence and pelvic organ prolapse. These uses are particularly valuable as the population ages and health care planners prepare for growing numbers of older women, with consequent increase in the need for care of pelvic floor disorders. Symptom-based case definitions tend to be complex: our case definitions underwent rigorous review, and our results provide a strong case for validating case definitions before their application in research.

### Limitations

It was difficult to develop case definitions for pelvic floor disorders because EMR data entry is not standardized, and symptoms may be recorded in several parts of the EMR chart. Complex case definitions were needed to capture the variety of descriptions used by different sentinels (Appendix 1). Patient symptoms may be underreported compared to clearly defined physiologic measures, such as blood pressure for hypertension, hemoglobin A1c level for diabetes or use of highly specific drugs.2,26 Our research explored “any” report of a pelvic floor disorder; this definition could be refined to look at other time frames, such as “during the past year,” as necessary for other research purposes.

Technical aspects of clinical EMRs can cause problems. For example, the unannounced system upgrade of 1 EMR vendor resulted in the loss of a large part of our initial sample. We were able to overcome this problem, but other researchers may not have the luxury of working with different EMRs. Another limitation of CPCSSN EMR data is that patients are associated with individual sentinels. If a sentinel leaves CPCSSN for any reason, data from their patients are no longer included in the database. In addition, there is a reported lack of representativeness of CPCSSN providers and patients.27 We needed only a sample of women stratified based on age less than 55 years, or 55 years or more; therefore, this was not a problem for our research.

### Conclusion

Our case definition for urinary incontinence met our standard for validity (sensitivity and specificity > 70%) and can be used for epidemiologic research. High PPVs for case definitions for fecal incontinence and pelvic organ prolapse mean these definitions can be used in applications such as quality-improvement studies, cohort studies and creation of disease registries. With the increasing development of primary care data networks worldwide, our research methods should be useful in studying pelvic floor disorders and in developing other symptom-based algorithms for application to other health conditions encountered in primary care.

## Footnotes

*   **Competing interests:** None declared.

*   This article has been peer reviewed.

*   **Contributors:** Sue Ross, Stephanie Garies, Deb Slade, Tyler Williamson and Neil Drummond supervised the work. Sue Ross, Stephanie Garies, Tyler Williamson and Neil Drummond conceived the study. Neil Drummond and Stephanie Garies acquired the data. Sue Ross, Rebecca Miyagishima, Boglarka Soos and Matt Taylor analyzed the data. Sue Ross and Neil Drummond drafted the manuscript. All of the authors contributed to the study design and data interpretation, revised the manuscript critically for important intellectual content, approved the final version to be published and agreed to be accountable for all aspects of the work.

*   **Data sharing:** The data from this study are held securely by the Southern Alberta Primary Care Research Network/Canadian Primary Care Sentinel Surveillance Network. A deidentified analysis data set from the study may be made available to other researchers who provide a detailed study proposal clearly describing the use of the data. The study proposal must be approved by an independent review committee identified for this purpose. Proposals may be submitted up to 24 months after the date of publication of this paper. Researchers who wish to access the study data may contact the corresponding author, Sue Ross, sue.ross{at}albertahealthservices.ca.

*   **Supplemental information:** For reviewer comments and the original submission of this manuscript, please see [www.cmajopen.ca/content/8/2/E414/suppl/DC1](http://www.cmajopen.ca/content/8/2/E414/suppl/DC1).

## References

1.  1.  Abrams P, 
    2.  Cardozo L
    
    Milsom I, Altman D, Cartwright R, et al. (2017) in Incontinence, Epidemiology of urinary incontinence (UI) and other lower urinary tract symptoms (LUTS), pelvic organ prolapse (POP) and anal incontinence (AI), eds Abrams P, Cardozo L (International Continence Society, Bristol (UK)), 6th ed. pp 4–144.
    
    

2.  Taylor DW, Cahill JJ (2018) From stigma to the spotlight: a need for patient-centred incontinence care. Healthc Manage Forum 31:261–4.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 

3.  Minassian VA, Yan X, Lichtenfeld MJ, et al. (2012) The iceberg of health care utilization in women with urinary incontinence. Int Urogynecol J 23:1087–93.
    
    

4.  Quiroz LH, White DE, Juarez D, et al. (2012) Age effects on pelvic floor symptoms in a cohort of nulliparous patients. Female Pelvic Med Reconstr Surg 18:325–8.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23143423&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 

5.  Sung VW, Hampton B (2009) Epidemiology of pelvic floor dysfunction. Obstet Gynecol Clin North Am 36:421–43.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.ogc.2009.08.002&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19932408&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 

6.  Sung VW, Raker CA, Myers DL, et al. (2009) Ambulatory care related to female pelvic floor disorders in the United States, 1995–2006. Am J Obstet Gynecol 201:508.e1–6.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19683690&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 

7.  Erekson EA, Lopes VV, Raker CA, et al. (2010) Ambulatory procedures for female pelvic floor disorders in the United States. Am J Obstet Gynecol 203:497.e1–5.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20739015&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 

8.  Cundiff GW (2011) On the brink: the future of female pelvic medicine and reconstructive surgery in Canada. J Obstet Gynaecol Can 33:1253–5.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22166280&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 

9.  Williamson T, Green ME, Birtwhistle R, et al. (2014) Validating the 8 CPCSSN case definitions for chronic disease surveillance in a primary care database of electronic health records. Ann Fam Med 12:367–72.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMTIvNC8zNjciO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzgvMi9FNDE0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

10. Garies S, Birtwhistle R, Drummond N, et al. (2017) Data Resource Profile: national electronic medical record data from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN). Int J Epidemiol 46:1091–1092f.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=28338877&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 

11. Sung VW, Washington B, Raker CA (2010) Costs of ambulatory care related to female pelvic floor disorders in the United States. Am J Obstet Gynecol 202:483.e1–4.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.ajog.2010.01.015&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20227673&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 

12. ATC/DDD Index 2016 (WHO Collaborating Centre for Drug Statistics Methodology, Oslo (Norway)) updated 2019 Dec. 16. Available: [https://www.whocc.no/atc\_ddd_index/](https://www.whocc.no/atc_ddd_index/). accessed 2019 June 13.
    
    

13. Landis JR, Koch G (1977) Measurement of observer agreement for categorical data. Biometrics 33:159–74.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.2307/2529310&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=843571&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A1977CY39700012&link_type=ISI) 

14. Benchimol EI, Manuel DG, To T, et al. (2011) Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. J Clin Epidemiol 64:821–9.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.jclinepi.2010.10.006&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=21194889&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 

15. Rogan WJ, Gladen B (1978) Estimating prevalence from the results of a screening test. Am J Epidemiol 107:71–6.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1093/oxfordjournals.aje.a112510&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=623091&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A1978EJ19400010&link_type=ISI) 

16. Lalkhen AG, McCluskey A (2008) Clinical tests: sensitivity and specificity. Contin Educ Anaesth Crit Care Pain 8:221–3.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1093/bjaceaccp/mkn041&link_type=DOI) 

17. Speybroeck N, Devleesschauwer B, Joseph L, et al. (2013) Misclassification errors in prevalence estimation: Bayesian handling with care. Int J Public Health 58:791–5.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1007/s00038-012-0439-9&link_type=DOI) 

18. Shaw C, Tansey R, Jackson C, et al. (2001) Barriers to help seeking in people with urinary symptoms. Fam Pract 18:48–52.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1093/fampra/18.1.48&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=11145628&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000167146700009&link_type=ISI) 

19. Reyes RR, Parker G, Garies S, et al. (2018) Team-based comanagement of diabetes in rural primary care. Can Fam Physician 64:e346–53.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiY2ZwIjtzOjU6InJlc2lkIjtzOjk6IjY0LzgvZTM0NiI7czo0OiJhdG9tIjtzOjIwOiIvY21ham8vOC8yL0U0MTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

20. Ryan BL, Shadd J, Maddocks H, et al. (2017) Methods to describe referral patterns in a Canadian primary care electronic medical record database: modelling multilevel count data. J Innov Health Inform 24:888.
    
    

21. Gagnon J, Lussier MT, MacGibbon B, et al. (2018) The impact of antidepressant therapy on glycemic control in Canadian primary care patients with diabetes mellitus. Front Nutr 5:4.
    
    

22. Drummond N, McCleary L, Freiheit E, et al. (2018) Antidepressant and antipsychotic prescribing in primary care for people with dementia. Can Fam Physician 64:e488–97.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiY2ZwIjtzOjU6InJlc2lkIjtzOjEwOiI2NC8xMS9lNDg4IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqby84LzIvRTQxNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

23. Ehsani-Moghaddam B, Queenan JA, MacKenzie J, et al. (2018) Mucopolysaccharidosis type II detection by Naïve Bayes Classifier: an example of patient classification for a rare disease using electronic medical records from the Canadian Primary Care Sentinel Surveillance Network. PLoS One 13:e0209018.
    
    

24. Abu-Ashour W, Twells LK, Valcour JE, et al. (2018) Diabetes and the occurrence of infection in primary care: a matched cohort study. BMC Infect Dis 18:67.
    
    

25. Queenan JA, Farahani P, Ehsani-Moghadam B, et al. (2018) The prevalence and risk for herpes zoster infection in adult patients with diabetes mellitus in the Canadian Primary Care Sentinel Surveillance Network. Can J Diabetes 42:465–9.
    
    

26. Williamson T, Miyagishima RC, Derochie JD, et al. (2017) Manual review of electronic medical records as a reference standard for case definition development: a validation study. CMAJ Open 5:E830–3.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiY21ham8iO3M6NToicmVzaWQiO3M6ODoiNS80L0U4MzAiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWpvLzgvMi9FNDE0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

27. Queenan JA, Williamson T, Khan S, et al. (2016) Representativeness of patients and providers in the Canadian Primary Care Sentinel Surveillance Network: a cross-sectional study. CMAJ Open 4:E28–32.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiY21ham8iO3M6NToicmVzaWQiO3M6NzoiNC8xL0UyOCI7czo0OiJhdG9tIjtzOjIwOiIvY21ham8vOC8yL0U0MTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

28. Gentil ML, Cuggia M, Fiquet L, et al. (2017) Factors influencing the development of primary care data collection projects from electronic health records: a systematic review of the literature. BMC Med Inform Decis Mak 17:139.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F8%2F2%2FE414.atom) 

*   Copyright 2020, Joule Inc. or its licensors