Developing a patient-oriented realist evaluation for COVID-19 vaccine implementation in Saskatchewan: a methodologic framework
==============================================================================================================================

* Amir Reza Azizian
* Tracey Carr
* Nazeem Muhajarine
* Tanya Verrall
* Collin Hartness
* Jason Vanstone
* Maryam Yasinian
* Candace Skrapek
* Brenda Andreas
* Gerald Farthing
* Gary Groot

## Abstract

**Background:** There is an urgent need to inform decision-making and safe delivery of vaccines in a timely manner. Our objective is to describe the methods we used to perform a patient-oriented realist evaluation of COVID-19 vaccination implementation in Saskatchewan, Canada, in order to understand the underlying mechanisms and contexts of vaccination implementation and vaccine uptake.

**Methods:** This methodology paper describes a patient-oriented, realist, mixed-method evaluation to assess COVID-19 vaccination implementation in Regina, Saskatoon and Prince Albert, Saskatchewan. The study comprised 3 iterative phases guided by Realist And Meta-narrative Evidence Synthesis: Evolving Standards II (RAMESES II). In phase 1 (January–February 2021), we developed the initial program theory, in phase 2 (March–May 2021), we tested and refined the initial program theory, and in phase 3 (June–July 2021), we established the final program theory. Three patient and family partners with different backgrounds and experiences were selected purposively from various locations (urban and rural) in Saskatchewan to engage collaboratively in the evaluation. Data analysis and synthesis occurred at all 3 phases of the project. We analysed qualitative data from phases 2 and 3 using a “retroductive” approach. We used quantitative data to compare outcomes from the 3 sites.

**Interpretation:** This protocol describes how we developed a final program theory for COVID-19 vaccination implementation with patient and family partners to show for whom, under what circumstances, how and why Saskatchewan’s COVID-19 vaccination program has led to vaccine uptake. With patient and family partners’ engagement, the evaluation findings will be shared with the Saskatchewan Health Authority and provincial government policy-makers and communications departments, published in peer-reviewed journals, presented at provincial or national conferences, and disseminated through any additional media identified by the patient and family partners.

There is an urgent need to inform decision-making and safe delivery of vaccines in a timely manner. In the study described here, we aimed to understand why and how COVID-19 vaccination plans used in 3 different sites in Saskatchewan (Regina, Saskatoon and Prince Albert) have led to vaccine uptake. The study team included 3 patient and family partners with different backgrounds from various locations in Saskatchewan. We used a theory-driven approach — realist evaluation — to assess the experiences of people who received vaccines or not, or were involved in the planning and delivery of COVID-19 vaccines in the province. We explored who did or did not participate, in what circumstances and how the COVID-19 vaccination program was or was not implemented, and why the vaccination program has led to vaccine uptake. With patient and family partner engagement, the evaluation findings will be shared with the Saskatchewan Health Authority and provincial government policy-makers and communications departments, published in peer-reviewed journals, presented at provincial or national conferences, and disseminated through any additional media identified by the patient and family partners.

In December 2020, almost a year into the COVID-19 pandemic, Canada implemented a very large vaccination program to prevent further spread of the disease. However, accelerated vaccine development left little time to ground the implementation plans in evidence-based practices, which has contributed to many logistic and ethical challenges. 1 Because vaccine development is typically a lengthy and complex process,2 moving quickly can result in mistrust regarding vaccine safety and effectiveness,3,4 access and equity issues,4,5 lack of community engagement,6 insufficient supplies7 and delays in delivery,8 among other challenges.

Although COVID-19 vaccine programs required implementation plans involving integration of various sectors in the health care system and across sectors, we do not know whether the current COVID-19 vaccination processes in 3 urban areas of Saskatchewan (Regina, Saskatoon and Prince Albert) have been effective and how they will work in other Saskatchewan contexts. Given that there is an urgent need to inform decision-making and safe delivery of vaccines in a timely manner, we performed a patient-oriented, realist evaluation to develop a program theory of the underlying contexts and mechanisms of implementation and vaccine uptake. The primary research question was “How, why, for whom and under what circumstances will the COVID-19 vaccination program lead to vaccine uptake in 3 Saskatchewan sites?” Our objective in this report is to describe the methods used for the evaluation.

## Methods

### Study design

Through engaging those with lived experience of the health care system (patient and family partners) and key stakeholder perspectives, we established a program theory of vaccine implementation that can be adapted to multiple contexts across Saskatchewan and other jurisdictions. Our research team includes 3 patient and family partners (C.S., B.A., G.F.), 4 realist evaluators (T.C., N.M., T.V., G.G.), 2 Saskatchewan Health Authority employees (A.R.A., J.V.), 1 Saskatchewan Health Authority policy-maker (C.H.) and 1 research assistant (M.Y.).

We used realist evaluation, a relatively new approach in health care research, to explain why relations exist in complex systems and interventions.9–11 Since theories depict the essential processes that cause behaviour and system change, theoretically based programs allow researchers to test hypotheses and show program impact and effectiveness.12,13 In developing, testing and refining a program theory, realist evaluators establish explanatory pathways linking how certain contexts (C) evoke underlying mechanisms (M) to generate outcomes (O).14,15 These causal relations, referred to as CMO configurations (CMOCs), are the building blocks of program theory.16

This realist evaluation assessed COVID-19 vaccination implementation in Regina, Saskatoon and Prince Albert using a multilevel sequential exploratory strategy to capture the perspectives of the people who did or did not receive the vaccine or were involved in vaccine delivery. We used the Realist And Meta-narrative Evidence Syntheses: Evolving Standards II (RAMESES II)17 to report the methods and data analysis.

### Setting

Saskatchewan is a Canadian province with a population of 1 179 154.18 More than half of the population lives in the 3 largest cities: Saskatoon (population 336 614), Regina (population 263 184) and Prince Albert (population 46 609).19 Because the Saskatchewan government chose these 3 sites for the COVID-19 vaccination pilot phase, we selected them for the present evaluation. As of February 2021, 2 mRNA vaccines authorized by Health Canada, produced by Pfizer–BioNTech and Moderna,20 had been distributed to the provinces, including Saskatchewan.21 (The Janssen [Johnson & Johnson] and CoviShield [AstraZeneca] COVID-1920 vaccines were granted national approval after the pilot phase.) The provincial government, in partnership with the Saskatchewan Health Authority and the Public Health Agency of Canada, outlined a COVID-19 Vaccine Delivery Plan, a phased approach to delivering the COVID-19 vaccines to residents.21 The first COVID-19 vaccination was delivered in Regina on Dec. 15, 2020, followed by Saskatoon (Dec. 22, 2020) and Prince Albert (Jan. 7, 2021). The plan targeted health care workers in intensive care units, emergency departments and hospital COVID-19 units, staff at testing and assessment centres, older care home residents, adults older than 80 years and residents in northern remote communities.22–24

### Study phases

Our study comprised 3 iterative phases and engaged patient and family partners at each step (Figure 1).

![Figure 1:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/9/4/E1034/F1.medium.gif)

[Figure 1:](http://www.cmajopen.ca/content/9/4/E1034/F1)

Figure 1: 
Patient and family partner activities by realist evaluation phase.

#### Phase 1: development of initial program theory

An initial program theory forms the basis for a realist evaluation, narrows the focus of the evaluation activities, and guides the selection of study methods.16 The goal of this phase, carried out in January–February 2021, was to identify the program’s underlying assumptions, outcomes of interest, proposed mechanisms of achieving targeted outcomes and planned activities — the CMOCs that constitute the initial program theory. The first step in this phase was to review each site’s COVID-19 vaccination program documents and communications. The 2 members of the research team who are Saskatchewan Health Authority employees attended each site’s implementation meetings; therefore, their field notes and observations constituted part of the data collection for this phase.

The second step was to review the literature on implementation of the COVID-19 vaccine and other vaccines to find resources from similar contexts (e.g., theories on implementation of the COVID-19 vaccine or other vaccines, vaccine hesitancy or uptake in various subgroups). We drew on Saskatchewan’s COVID-19 Evidence Support Team rapid review reports and evidence search reports,25 and the Saskatchewan Social Contours and COVID-19 survey findings regarding vaccine hesitancy.26 We also reviewed grey literature sources27,28 through a realist lens to understand why there is hesitancy or resistance to getting vaccinated.

Aside from these systematic sources, in accordance with realist methodology,29 we purposively and iteratively searched the literature on vaccine implementation (e.g., H1N1) and grey sources (e.g., media reports) as the development of the initial program theory required.

The patient and family partners were actively engaged in document review, including presentations, field notes and observations. In multiple virtual meetings with patient and family partners, we used MURAL (a digital workspace for visual collaboration, [https://www.mural.co/](https://www.mural.co/)) to identify CMOCs in each source, and analyze and synthesize them into the initial program theory (see Data analysis section for details).

Based on the aggregated initial program theory (CMOCs) at the end of phase 1, the patient and family partners and researchers prepared an open-ended realist interview guide30 that presented each CMOC. Following a “teacher–learner” stance,14,30 whereby the interviewers “teach” the interviewees about the initial program theory, the interviewees were asked to confirm, refute or refine each initial program theory element.

#### Phase 2: testing of initial program theory

To test and refine the initial program theory, we conducted interviews with 14 purposively recruited eligible participants in March–May 2021. Our inclusion criteria focused on 8 key Saskatchewan Health Authority stakeholders who had planned and implemented the COVID-19 vaccination pilot phase (i.e., stakeholders from Clinical Excellence, Public Health, Protective Services, Human Resources, and Communications, and each site’s clinic managers) and 6 people who were eligible to receive the vaccine in the pilot phase, 2 from each site (i.e., a vaccine recipient and a person who was eligible to be vaccinated but did not receive the vaccine). Following realist sampling strategy,30,31 the patient and family partners (C.S., B.A., G.F.) and A.R.A. interviewed people who were eligible to receive the vaccine in the pilot phase, and A.R.A conducted the key stakeholder interviews. Purposive participant recruitment ensured diverse inclusion of health care workers and non–health care workers.

To identify eligible interviewees, we benefited from the patient and family partners’ experiences and contacts. Using the same recruitment and interview procedures for the 3 locations, we sent email invitations to each eligible potential participant to participate in a 20- to 30-minute interview. Written informed consent was collected before the interview. The interviews were performed via Webex or telephone, and were audiotaped, transcribed and analyzed to build final CMOCs (see Data analysis section), which will be the substance of the program theory for vaccination implementation.16

Quantitative data, such as the number and proportion of people vaccinated per week, and the number of vaccine doses delivered per week, were requested from each clinic at the 3 sites. Aggregated and de-identified data about the number and proportion of people vaccinated are also available from the Government of Saskatchewan website32 and other websites, such as the COVID-19 Vaccination Tracker ([https://covid19tracker.ca/vaccinationtracker.html](https://covid19tracker.ca/vaccinationtracker.html)). Quantitative data allows comparisons between the 3 sites regarding outcomes. For example, if the number of vaccine doses delivered at each site for the first month was less than 100% of the projected number, or if the proportion of people who received a vaccine did not meet the target, we can compare the context and mechanisms (e.g., damaged or wasted doses, not including groups that may be inappropriate to vaccinate in the target calculation) that led to disparate outcomes. As per the COVID-19 Vaccine Delivery Plan, the provincial target was 10 825 Pfizer–BioNTech vaccine doses delivered per week, and 191 426 doses of both available vaccines were delivered in the first quarter of 2021.32

#### Phase 3: development of final program theory

In June–July 2021, we synthesized and consolidated the CMOCs that emerged in phase 2 to construct a final program theory (see Data analysis section). Subsequently, in an online meeting with 2 Saskatchewan Health Authority directors who were involved in the strategic planning and implementation of the COVID-19 vaccination program in the 3 sites, we presented the final program theory using the MURAL platform to receive the directors’ final feedback. The patient and family partners and research team then finalized the final program theory in a series of team meetings.

### Patient engagement

The 3 patient and family partners have been engaged in patient-oriented realist research, are considered at high risk for contracting severe COVID-19, and have extensive lived experience with the health care system or care of family members with chronic health conditions. All were identified purposively from various locations (urban and rural) in Saskatchewan. We collaborated actively with the patient and family partners throughout the study. To maximize our engagement with them, we followed the Saskatchewan Centre for Patient-Oriented Research’s Patient-Oriented Research Level of Engagement Tool (PORLET) to direct our evaluation.33

The patient and family partners’ activities encompassed review of literature related to COVID-19 and non–COVID-19 vaccination programs, and codevelopment of the initial program theory, interview guide and interview questions. They also conducted interviews with people who were eligible to receive the vaccine in the pilot phase, assisted with the analysis and interpretation of qualitative data, codeveloped and refined the final program theory, and planned knowledge translation.

### Data analysis

Data analysis and synthesis occurred at all 3 phases of the project to develop and refine CMOC elements. The study outcome (O) was the degree of vaccine uptake for all 3 phases, and contexts (C) and mechanisms (M) were identified in the literature (phase 1) and refined based on the data collected in phases 2 and 3. After training the patient and family partners using the resources and training materials for realist evaluation,34 the research team iteratively distinguished contexts and mechanisms as they related to the study outcome. A CMO example might be communication (C) leads to perception of mistrust (M), which results in increased or decreased uptake (O).

Quantitative data collected from the Government of Saskatchewan’s COVID-19 Vaccine Dashboard32 as well as daily reports from the 3 sites were imported in Excel (Microsoft 365 Apps for enterprise version). Then, we used the daily vaccination rates to monitor what the 3 sites’ vaccine uptake rates were. These rates represented the outcomes in the initial program theories.

We analyzed the qualitative data from phases 2 and 3 using a “retroductive” approach common in realist research.35,36 In this approach, both inductive and deductive analyses are used, along with the researchers’ insights, to understand generative causation.35 The main stages of inductive analysis are developing a code, identifying initial themes from data sources (e.g., interviews) and coding initial themes. For the deductive analysis, we applied the 3 broad concepts of context, mechanism and outcome to the codes identified in the inductive stage. Two independent researchers (a patient or family partner and a team member who is familiar with realist evaluation [A.R.A.]) analyzed the data by selecting the appropriate segments of text and coding them.

All interview transcripts were imported to NVivo 12 Plus software (QSR International). Thorough iterative steps to analyze interview transcripts in NVivo are described by Gilmore and colleagues.36 In brief, any CMO in a data source (e.g., an interview transcript) was recorded as a code, linked to an appropriate initial program theory (node), and added to the memo that is linked to the initial program theory. We then reviewed all CMOs in each memo using a memo template (Appendix 1, available at [www.cmajopen.ca/content/9/4/E1034/suppl/DC1](http://www.cmajopen.ca/content/9/4/E1034/suppl/DC1)) to develop CMOCs and refine the initial program theory (node). Subsequently, from the new sources (i.e., interview data), we coded CMOCs directly to the most relevant refined initial program theories of former sources.

After we retroductively compared and contrasted each element of the initial program theory (phase 1) with the CMOCs in phase 2, we presented the refined program theory to the phase 3 stakeholder group for final refinement. We collated the refined initial program theories for similarity and overlap to synthesize a final program theory. The patient and family partners along with the researchers synthesized the final program theory in a series of team meetings.

### Ethics approval

The study received letters of exemption from the University of Saskatchewan Behavioural Research Ethics Board and the Saskatchewan Health Authority Research Ethics Board because of its program-evaluation status. The study consent forms reflected the exemption status from the research ethics boards.

## Interpretation

Although our research does not offer direct benefits to individual participants, the findings will have important practical and research implications. With patient and family partner engagement, we codeveloped the final program theory to share with Saskatchewan Health Authority and provincial government policy-makers and communications departments. By including Saskatchewan Health Authority directors in phases 2 and 3 of the study, we provided them with the initial and the final program theory to inform system learning for the current and future large-scale vaccination programs. We have met with the coleads of the COVID-19 vaccination program at the Saskatchewan Emergency Operations Center to notify them of this study. The final program theory can be used to inform current or future COVID-19 vaccination programs.

Our findings will also have important academic and research implications. Results will be disseminated at provincial or national conferences, and articles will be prepared to be published in peer-reviewed journals. The patient and family partners will provide guidance on dissemination through any additional media and will be invited to copresent the findings.

### Limitations

Because of the nature of COVID-19 vaccination plans, the documentations or presentations used as data sources may be limited by changes over time. This may have affected the development of the initial program theory. However, the refinement of the initial program theory in various steps as well as the iterative design of the study may reduce the effect of outdated data sources.

The research team’s alignment with the vaccination program may have introduced potential bias. We intentionally engaged patient and family partners who were in contact with people unwilling to receive the COVID-19 vaccine to mitigate this bias.

The grey literature and media were used to assist in understanding vaccination opposition. The initial program theory relied on limited literature with few peer-reviewed sources. However, we made the initial program theory more robust by benefiting from the expertise of patient and family partners.

Quantitative data were also limited by the accessibility of the Saskatchewan vaccination administrative database (i.e., Panorama). Therefore, we were unable to access demographic data that were related to our program theories. Instead, we were limited to publicly available aggregated data on vaccine rates.

### Conclusion

We codeveloped a final program theory for COVID-19 vaccination implementation with patient and family partners to show for whom, under what circumstances, how and why Saskatchewan’s COVID-19 vaccination program has led to vaccine uptake. We expect that the findings will inform various stakeholders about the current processes embedded in COVID-19 vaccine delivery in Regina, Saskatoon and Prince Albert, and how these processes can guide further vaccination programs in other Saskatchewan contexts.

## Footnotes

*   **Competing interests:** None declared.

*   This article has been peer reviewed.

*   **Contributors:** Gary Groot supervised the work. Amir Azizian, Tracey Carr, Nazeem Muhajarine, Tanya Verrall, Collin Hartness, Jason Vanstone, Candace Skrapek, Brenda Andreas, Gerald Farthing and Gary Groot contributed to the study conception. Amir Azizian, Tracey Carr, Nazeem Muhajarine, Tanya Verrall, Collin Hartness, Jason Vanstone and Gary Groot contributed to the study design. Amir Azizian, Jason Vanstone, Candace Skrapek, Brenda Andreas and Gerald Farthing acquired the data. Amir Azizian, Tracey Carr, Maryam Yasinian, Candace Skrapek, Brenda Andreas and Gerald Farthing contributed to the data analysis. Amir Azizian drafted the manuscript. All of the authors revised the manuscript critically for important intellectual content, approved the final version to be published and agreed to be accountable for all aspects of the work.

*   **Funding:** This work was supported by the 2020 College of Medicine Research Awards (CoMRAD) program, University of Saskatchewan.

*   **Data sharing:** No data are associated with this protocol. It is expected that data generated from the realist evaluation will be made available in response to a reasonable request sent to the corresponding author.

*   **Supplemental information:** For reviewer comments and the original submission of this manuscript, please see [www.cmajopen.ca/content/9/4/E1034/suppl/DC1](http://www.cmajopen.ca/content/9/4/E1034/suppl/DC1).

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY-NC-ND 4.0) licence, which permits use, distribution and reproduction in any medium, provided that the original publication is properly cited, the use is noncommercial (i.e., research or educational use), and no modifications or adaptations are made. See: [https://creativecommons.org/licenses/by-nc-nd/4.0/](https://creativecommons.org/licenses/by-nc-nd/4.0/).

## References

1.  Dodd RH, Pickles K, Nickel B, et al. (2021) Concerns and motivations about COVID-19 vaccination. Lancet Infect Dis 21:161–3.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S1473-3099(20)30926-9&link_type=DOI) 

2.  Thompson SA (Apr 30, 2020) The New York Times, How long will a vaccine really take? Available[https://www.nytimes.com/interactive/2020/04/30/opinion/coronavirus-covid-vaccine.html?auth=login-google](https://www.nytimes.com/interactive/2020/04/30/opinion/coronavirus-covid-vaccine.html?auth=login-google). accessed 2021 Jan. 21.
    
    

3.  Bloom BR, Nowak GJ, Orenstein W (2020) “When will we have a vaccine?” — understanding questions and answers about COVID-19 vaccination” N Engl J Med 383:2202–4.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1056/NEJMp2025331&link_type=DOI) 

4.  National Academies of Sciences, Engineering, and Medicine (2020) Framework for equitable allocation of COVID-19 vaccine (National Academies Press, Washington).
    
    

5.  Burgess RA, Osborne RH, Yongabi KA, et al. (2021) The COVID-19 vaccines rush: participatory community engagement matters more than ever. Lancet 397:8–10.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S0140-6736(20)32642-8&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=33308484&link_type=MED&atom=%2Fcmajo%2F9%2F4%2FE1034.atom) 

6.  Mattoli S (2021) Filling the gap until full vaccine deployment in the war on coronavirus disease-19. Infect Dis Ther 10:27–34.
    
    

7.  Jones RP, Zimonjic P (Jan 7, 2021) CBC News, Premiers give Trudeau a friendly push to increase vaccine supply. updated 2021 Jan. 8. Available: [https://www.cbc.ca/news/politics/first-ministers-january-7-1.5864408](https://www.cbc.ca/news/politics/first-ministers-january-7-1.5864408). accessed 2021 Jan. 21.
    
    

8.  Nagurney A (Jan 8, 2021) Quartz, COVID-19 vaccine delays reveal a shortage of healthcare workers. Available: [https://qz.com/1954675/covid-19-vaccine-delays-reveal-a-shortage-of-healthcare-workers/](https://qz.com/1954675/covid-19-vaccine-delays-reveal-a-shortage-of-healthcare-workers/). accessed 2021 Jan. 21.
    
    

9.  Zibrowski E, McDonald S, Thiessen H, et al. (2020) Developing a program theory of patient engagement in patient-oriented research and the impacts on the health care system: protocol for a rapid realist review. CMAJ Open 8:E530–4.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiY21ham8iO3M6NToicmVzaWQiO3M6ODoiOC8zL0U1MzAiO3M6NDoiYXRvbSI7czoyMToiL2NtYWpvLzkvNC9FMTAzNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

10. Groot G, Waldron T, Carr T, et al. (2017) Development of a program theory for shared decision-making: a realist review protocol. Syst Rev 6:114.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F9%2F4%2FE1034.atom) 

11. Salter KL, Kothari A (2014) Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review. Implement Sci 9:115.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/s13012-014-0115-y&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25190100&link_type=MED&atom=%2Fcmajo%2F9%2F4%2FE1034.atom) 

12. Raine R, Fitzpatrick R, Barratt H, et al. (2016) Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. Health Serv Deliv Res 4, doi:10.3310/hsdr04160.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.3310/hsdr04160&link_type=DOI) 

13. Eccles M, Grimshaw J, Walker A, et al. (2005) Changing the behavior of healthcare professionals: the use of theory in promoting the uptake of research findings. J Clin Epidemiol 58:107–12.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.jclinepi.2004.09.002&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=15680740&link_type=MED&atom=%2Fcmajo%2F9%2F4%2FE1034.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000227083600001&link_type=ISI) 

14. Pawson R, Tilley N (2004) Realist evaluation (Community Matters, Mount Torrens (South Australia)) Available: [http://www.communitymatters.com.au/RE_chapter.pdf](http://www.communitymatters.com.au/RE_chapter.pdf). accessed 2021 Aug. 27.
    
    

15. Reddy S, Wakerman J, Westhorp G, et al. (2015) Evaluating impact of clinical guidelines using a realist evaluation framework. J Eval Clin Pract 21:1114–20.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1111/jep.12482&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=26640246&link_type=MED&atom=%2Fcmajo%2F9%2F4%2FE1034.atom) 

16. Nurjono M, Shrestha P, Lee A, et al. (2018) Realist evaluation of a complex integrated care programme: protocol for a mixed methods study. BMJ Open 8:e017111.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiOC8zL2UwMTcxMTEiO3M6NDoiYXRvbSI7czoyMToiL2NtYWpvLzkvNC9FMTAzNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

17. Wong G, Westhorp G, Manzano A, et al. (2016) RAMESES II reporting standards for realist evaluations. BMC Med 14:96.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F9%2F4%2FE1034.atom) 

18. (2021) Saskatchewan population (Government of Saskatchewan), Available: [https://dashboard.saskatchewan.ca/people-community/people/population](https://dashboard.saskatchewan.ca/people-community/people/population). accessed 2021 June 9.
    
    

19. Table 17-10-0135-01: Population estimates, July 1, by census metropolitan area and census agglomeration, 2016 boundaries (Statistics Canada, Ottawa) Available[https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1710013501](https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1710013501). accessed 2021 July 5.
    
    

20. COVID-19 drugs and vaccines (Health Canada, Ottawa) modified 2021 June 2. Available: [https://www.canada.ca/en/health-canada/services/drugs-health-products/covid19-industry/drugs-vaccines-treatments/vaccines.html](https://www.canada.ca/en/health-canada/services/drugs-health-products/covid19-industry/drugs-vaccines-treatments/vaccines.html). accessed 2020 Dec. 25.
    
    

21. (Dec 9, 2020) Saskatchewan releases COVID-19 vaccine delivery plan [news release] (Government of Saskatchewan), Available: [https://www.saskatchewan.ca/government/news-and-media/2020/december/09/saskatchewan-releases-covid-19-vaccine-delivery-plan](https://www.saskatchewan.ca/government/news-and-media/2020/december/09/saskatchewan-releases-covid-19-vaccine-delivery-plan). accessed 2020 Dec. 28.
    
    

22. (Dec 9, 2020) CBC News, ‘Today is one of the most positive days we’ve had,’ says SHA CEO as vaccine rollout plan announced. Available: [https://www.cbc.ca/news/canada/saskatchewan/sask-premier-covid-19-pfizer-vaccine-arriving-soon-1.5833354](https://www.cbc.ca/news/canada/saskatchewan/sask-premier-covid-19-pfizer-vaccine-arriving-soon-1.5833354). accessed 2021 Aug. 27.
    
    

23. Peterson J (Dec 22, 2020) CBC News, First health-care workers in Saskatoon vaccinated for COVID-19. Available: [https://www.cbc.ca/news/canada/saskatoon/saskatoon-first-covid-vaccines-1.5852045](https://www.cbc.ca/news/canada/saskatoon/saskatoon-first-covid-vaccines-1.5852045). accessed 2020 Dec. 25.
    
    

24. Hicks G (Jan 7, 2021) paNOW, First COVID vaccines given in Prince Albert. Available: [https://panow.com/2021/01/07/first-covid-vaccines-given-in-prince-albert/](https://panow.com/2021/01/07/first-covid-vaccines-given-in-prince-albert/). accessed 2021 Jan. 14.
    
    

25. COVID-19 Dashboard (Saskatchewan Health Authority Library), updated 2021 Aug. 18. Available: [https://saskhealthauthority.libguides.com/covid-19/repository/home](https://saskhealthauthority.libguides.com/covid-19/repository/home). accessed 2021 June 10.
    
    

26. Social Contours & COVID-19 Saskatchewan survey (Saskatchewan Population Health and Evaluation Research Unit (SPHERU)), Available: [https://spheru.ca/covid-19/socialcontours/covid-19.php](https://spheru.ca/covid-19/socialcontours/covid-19.php). accessed 2021 June 10.
    
    

27. Goldman B (Jan 16, 2021) CBC Radio, White Coat, Black Art: Some health-care workers still hesitant to get COVID-19 vaccine. Available: [https://www.cbc.ca/radio/whitecoat/some-health-care-workers-still-hesitant-to-get-covid-19-vaccine-1.5872004](https://www.cbc.ca/radio/whitecoat/some-health-care-workers-still-hesitant-to-get-covid-19-vaccine-1.5872004). accessed 2021 Jan. 18.
    
    

28. Rubin EJ, Baden LR, Lee TH, et al. (2021) Audio interview: planning for the SARS-CoV-2 vaccine rollout [editorial]. N Engl J Med 384:e13.
    
    

29. Carr T, Quinlan E, Robertson S, et al. (2017) Adapting realist synthesis methodology: the case of workplace harassment interventions. Res Synth Methods 8:496–505.
    
    

30. Manzano A (2016) The craft of interviewing in realist evaluation. Evaluation 22:342–60.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1177/1356389016638615&link_type=DOI) 

31. Emmel N (2013) Sampling and choosing cases in qualitative research: a realist approach (SAGE Publications, London (UK)).
    
    

32. COVID-19 vaccines: total doses (Government of Saskatchewan), Available: [https://dashboard.saskatchewan.ca/health-wellness/covid-19-vaccines/vaccines#cumulative-doses-tab](https://dashboard.saskatchewan.ca/health-wellness/covid-19-vaccines/vaccines#cumulative-doses-tab). accessed 2021 Jan. 21.
    
    

33. SCPOR Patient-Oriented Research Level of Engagement Tool (PORLET) (Saskatchewan Centre for Patient-Oriented Research, Saskatoon) Available: [https://www.scpor.ca/porlet](https://www.scpor.ca/porlet). accessed 2021 Jan. 14.
    
    

34. The RAMESES projects (University of Oxford, Oxford (UK)) Available[https://www.ramesesproject.org/Standards\_and\_Training\_materials.php#re\_training](https://www.ramesesproject.org/Standards\_and_Training_materials.php#re_training). accessed 2021 June 28.
    
    

35. Wong G, Westhorp G, Greenhalgh J, et al. (2017) Quality and reporting standards, resources, training materials and information for realist evaluation: the RAMESES II project. Health Serv Deliv Res 5, doi:10.3310/hsdr05280.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.3310/hsdr05280&link_type=DOI) 

36. Gilmore B, McAuliffe E, Power J, et al. (2019) Data analysis and synthesis within a realist evaluation: toward more transparent methodological approaches. Int J Qual Methods 18, doi:10.1177/1609406919859754.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1177/1609406919859754&link_type=DOI) 

*   © 2021 CMA Joule Inc. or its licensors