Abstract
Background: There are few data on patient and public involvement (PPI) in pragmatic trials. We aimed to describe the prevalence and nature of PPI within pragmatic trials, describe variation in prevalence of PPI by trial characteristics and compare prevalence of PPI reported by trial authors to that reported in trial publications.
Methods: We applied a search filter to identify pragmatic trials published from 2014 to 2019 in MEDLINE. We invited the corresponding authors of pragmatic trials to participate in an online survey about their specific trial.
Results: Of 3163 authors invited, 2585 invitations were delivered, 710 (27.5%) reported on 710 unique trials and completed the survey; 334 (47.0%) conducted PPI. Among those who conducted PPI, for many the aim was to increase the research relevance (86.3%) or quality (76.5%). Most PPI partners were engaged at protocol development stages (79.1%) and contributed to the co-design of interventions (70.9%) or recruitment or retention strategies (60.5%). Patient and public involvement was more common among trials involving children, trials conducted in the United Kingdom, cluster randomized trials, those explicitly labelled as “pragmatic” in the study manuscript, and more recent trials. Less than one-quarter of trials (22.8%) that reported PPI in the survey also reported PPI in the trial manuscript.
Interpretation: Nearly half of trialists in this survey reported conducting PPI and listed several benefits of doing so, but researchers who did not conduct PPI often cited a lack of requirement for it. Patient and public involvement appears to be significantly underreported in trial publications. Consistent and standardized reporting is needed to promote transparency about PPI methods, outcomes, challenges and benefits.
In recent years, there has been increasing interest in conducting randomized controlled trials (RCTs) that are pragmatic in orientation.1,2 Pragmatic trials use study settings and methods similar to usual care, as well as taking into account the perspectives of patients.3,4 Some of the design features of pragmatic trial design — for example, selecting outcomes that are important to patients5 — overlap with objectives of patient-oriented research.6–8 Patient and public involvement (PPI) in research provides an opportunity for patients and families to share their lived experiences and better align research with patient values and priorities.6 For the purpose of this research and in alignment with the Canadian Institutes of Health Research (CIHR) Strategy for Patient-Oriented Research (SPOR),6 we define PPI as “researchers consulting or working with members of the public, patients, service users, and carers in any or all part(s) of the research process, including the choice of research topic, design, planning, conduct or dissemination of research.”9 We refer to these individuals as “PPI partners”; partners could include, for example, grant co-applicants, members of a Trial Steering Committee, or members of a patient or public advisory panel. In health research, PPI is strongly encouraged by international funding bodies,6,10,11 institutions12 and stakeholder groups.13,14
Previous reviews of RCTs found low prevalence of PPI, with inconsistent reporting.15–21 However, no trial-specific reporting guidelines require authors to indicate whether their trial included PPI (although the generic Guidance for Reporting Involvement of Patients and the Public [GRIPP2]22 checklist may be used when PPI was part of an intervention), and journals rarely require PPI to be reported.17 Relying solely on information provided in published trial reports would likely lead to an underestimation of the true prevalence of PPI. To address this gap in PPI reporting, we conducted a survey of authors of pragmatic trials. The main objectives of the survey were:
To describe the prevalence and nature of PPI within these trials; in particular:
when and how PPI was sought (e.g., integrated in trial design, conduct, analysis, dissemination)
who was involved (e.g., patients [including children and youth], parents or caregivers, family members, patient group representatives)
justification for involving or not involving PPI partners
perceived benefits or not of involving PPI partners
challenges faced in engaging PPI partners
To describe variation in the prevalence of PPI by trial characteristics including trial population (pediatric v. older adults v. neither), country of study author, trial design (individual v. cluster randomization), pragmatism (explicitly labelled as “pragmatic” in the study manuscript v. not), and year of publication
To compare the prevalence of PPI reported by trial authors in the survey to that reported in study publications.
Methods
Study design
This is a substudy of a study23 evaluating methodological and ethical issues in pragmatic trials. The main study involved the development and validation of an electronic search filter24 (Appendix 1, available at www.cmajopen.ca/content/11/5/E826/suppl/DC1) to efficiently find health-focused trials that were more likely to be pragmatic, because relying on authors to use the words “pragmatic,” “naturalistic” or “real-world” to describe their trials is inadequate for identifying pragmatic trials. The filter used terms related to common designs, settings and data sources of pragmatic trials, such as those captured in the Pragmatic Explanatory Continuum Indicator Summary (PRECIS-2) tool,5 and had an estimated sensitivity of 46% and estimated specificity of 98%.24 Of note, the search filter identifies trials clearly labelled by trial authors as “pragmatic” with 100% sensitivity. The filter was used in MEDLINE to identify primary trial reports published between Jan. 1, 2014, and Apr. 3, 2019. A descriptive analysis of trials identified from the search has been previously published.25 Here we describe the results from an online survey completed by the corresponding authors of the published primary reports, reported in accordance with the Checklist for Reporting Results of Internet E-Surveys (CHERRIES).
Setting and participants
To facilitate conduct of the survey in English and to focus on the jurisdictions in which our investigator team has experience, we targeted corresponding authors in the following countries: Canada, United States, United Kingdom, Australia and New Zealand, South Africa, France, Belgium, Denmark, Finland, Germany, Italy, the Netherlands, Norway, Spain, Sweden and Switzerland. We selected only the most recently published report, even if the individual was corresponding author on multiple eligible studies. For a small fraction of corresponding authors with no email addresses listed, we used online searches to attempt to identify an email address.
Data sources
We used 2 data sources: a survey questionnaire and a database from the broader study23 of previously extracted or downloaded trial characteristics. The survey questionnaire was developed based on previous surveys about PPI,20,26–28 the research team’s own experience, and the perspectives of 2 patient partners (M.S., A.H.) with expertise in patient engagement in research. We included these 2 patient partners from the study conceptualization phase, and they reviewed and contributed to the study protocol, survey content and manuscript, in addition to helping interpret study findings. Members of the international study team and colleagues with experience in pragmatic trials reviewed an initial draft of the survey questionnaire, to verify its usability and confirm that completion time would be about 10 minutes. The questionnaire consisted of 27 open- and closed-ended items pertaining to PPI in the published trial, including whether PPI had taken place, reasons for or against engaging PPI partners in the trial, characteristics of PPI partners, methods used to involve PPI partners, the stage of research when PPI partners were first engaged, and outcomes, benefits and challenges of PPI, followed by a series of demographic questions (see Appendix 2, available at www.cmajopen.ca/content/11/5/E826/suppl/DC1, for the survey questionnaire). The questionnaire included a definition of PPI and respondents had to indicate that they had read and understood the definition before completing the survey. To encourage respondents to provide as much information as possible, the only mandatory question was whether they had used PPI in the trial.
To assess potential nonresponse bias, we compared trial characteristics of respondents and nonrespondents using previously extracted or downloaded information available within the larger database:29,30 country of corresponding author, year of publication, clinical or disease area (obtained from Web of Science), unit of randomization (cluster v. individual randomization), journal impact factor, and age of trial participants (as described in Appendix 3, available at www.cmajopen.ca/content/11/5/E826/suppl/DC1). For trials registered in ClinicalTrials.gov, we examined primary trial purpose (classified as treatment, prevention, diagnostic, supportive care, screening, health services research or other) and type of experimental intervention (classified as drug, device, biological or vaccine, procedure or surgery, radiation, genetic, dietary supplement, educational or behavioural or other). Finally, we also obtained information about PPI reporting in the manuscript from a smaller subset of trials included in a previous substudy.31,32
Survey administration
We administered the survey via SurveyMonkey. We modified Dillman’s Tailored Design Method, which included personalized invitations, a visually appealing survey, incentives and a reminder schedule. Invitation emails (Appendix 4, available at www.cmajopen.ca/content/11/5/E826/suppl/DC1) were personalized with the name of the corresponding author, title, year of publication and journal, as well as a unique link to the survey that corresponded to their trial. This allowed for survey responses to be analyzed in combination with trial characteristics already in our database and prevent more than 1 survey response for each trial. Authors were permitted to forward the survey for completion to another member of the research team. The survey was open; i.e., no password was required to complete it. Survey participants were given the option to be entered into a draw for one of 5 Can$100 Amazon gift cards. We piloted the survey administration with a random sample of 100 corresponding authors, to identify and resolve any technical issues with automated invitations. The initial pilot invitation was sent via SurveyMonkey on Nov. 26, 2021, followed by the first reminder 10 days later and second reminder 36 days later (to accommodate the end-of-year holiday period). No content changes were considered necessary after the pilot, and pilot data were therefore included in the final analysis. Subsequently, we sent personalized invitation emails to all remaining eligible corresponding authors in our sampling frame, via SurveyMonkey, on Feb. 8, 2022, followed by the first reminder 2 weeks later and second reminder another week after that. We sent the third (final) reminder between Mar. 15 and Mar. 18, 2022, from the senior author’s institutional email address rather than SurveyMonkey, in an attempt to bypass spam filters that might have prevented the first 2 invitations from reaching potential participants. This invitation included a unique identifier to be entered by each respondent to allow linkage with their trial. The survey was closed on Apr. 5, 2022.
All participants were prompted to review a participant information sheet at the start of the survey (Appendix 2) and notified that the survey was voluntary and continuing implied consent.
Statistical analysis
We exported survey data from SurveyMonkey into a spreadsheet via Airtable. We summarized participant characteristics using descriptive statistics. Using search filters applied to our database (Appendix 4), we categorized trials for descriptive purposes into 1 of 3 groups: pediatric trials (trial participants were children aged 0–18 yr, or the primary outcome indicated that the study was child focused, or both), older adult trials (mean or median age of trial participants was ≥ 65 yr), or neither.
We analyzed survey data using descriptive statistics (frequencies and percentages). We analyzed all survey responses regardless of completeness. Most survey items were closed ended, but many allowed respondents to select “Other” and elaborate in a text box. Responses to “Other” were collated and reviewed by 2 study team members independently to determine whether the response could be reclassified into an existing category or warranted creation of a new category. Disagreements were resolved through discussion.
To describe variation in the prevalence and nature of PPI, we cross-tabulated trial characteristics — including trial population (pediatric v. older adults v. neither), country of corresponding author, trial design (individual v. cluster randomization), whether the trial was explicitly labelled as “pragmatic” in the study manuscript versus not, and year of publication — against self-reported PPI in the trial.
To compare the prevalence of PPI reported in the survey to that reported in the trial publication, one of 3 authors (S.V., K.C., P.N.) reviewed final trial reports to assess whether PPI was reported or acknowledged, and this was compared with survey responses. Patient and public involvement in the publication was defined as explicit reporting on patient or public involvement, elicitation of patient or public perspectives, or mention of PPI partners in author lists or in the manuscript “Acknowledgements” section.
Ethics approval
This study was approved by the Ottawa Health Science Network Research Ethics Board.
Results
The parent study identified 4336 eligible primary trial reports, of which we excluded 835 (19%) as ineligible because of the country of the corresponding author; a further 335 (7.7%) were excluded as they were multiple publications by the same trial author, and a further 3 (0.1%) for which the corresponding author email address could not be identified. Thus, we invited 3163 unique corresponding authors to complete the survey (Figure 1). Of these, 710 ultimately reported on unique trials and completed the survey for an overall response rate (among all invited authors) of 710/3163 (22.4%) or 710/2585 (27.5%) after we removed undeliverable email addresses. We classified authors as respondents if they completed at least the primary question of interest (whether they used PPI in the trial).
Respondent characteristics are shown in Table 1. Respondents mostly resided in the US (247 [37.7%]), Europe — excluding the UK (166 [25.3%]) — and the UK (110 [16.8%]). Most respondents were females (364 [55.6%]). Respondent ages were most often 46–55 years (203 [31.0%]) or 56–65 years (185 [28.3%]), with the majority indicating they were late career researchers (> 15 yr since first academic appointment; 402 [61.5%]). Most respondents reported “White” as their racial or ethnic group (550, 84.2%). Of respondents, 54% (353) reported having more than 10 years’ experience in conducting PPI in research, followed by 29.1% (190) reporting 4–10 years’ experience; 47 (7.2%) respondents reported less than 1 year of experience with PPI. We observed no major differences between survey respondents and nonrespondents based on the available characteristics, except that the prevalence of reporting PPI in trial reports was slightly higher among respondents (12.4%) than among nonrespondents (7.8%) (Appendix 5, available at www.cmajopen.ca/content/11/5/E826/suppl/DC1).
Table 2 shows the proportion of survey respondents who reported involving PPI partners in their trial, and stated reasons for involving or not involving PPI partners. Of all 710 respondents, 334 (47.0%) reported that they involved PPI partners, 333 (46.9%) reported that they did not involve PPI partners, and 43 (6.1%) stated that they didn’t know whether PPI partners were involved. Among the 334 who reported involving PPI partners, reasons for PPI were reported by 315 (94.3%). Most commonly reported reasons were increased applicability or relevance of research (272 [86.3%]), increased quality of research (241 [76.5%]) and it being the morally or ethically right thing to do (204 [64.8%]). Ten (3.2%) survey respondents cited other reasons for engaging PPI partners, such as increased pragmatism or because the research question originated from a patient. Among the 333 who reported that they did not involve PPI partners, reasons were selected by 307 (92.2%). Most commonly reported reasons were that there was no requirement to do so (118 [38.4%]), it did not seem relevant (99 [32.3%]), and there was a lack of resources or funds (79 [25.7%]). Twelve (3.9%) respondents indicated other reasons for not involving PPI partners, such as conducting a replication study that was based on a trial that had already involved PPI partners, or lack of available flexibility to modify study designs or materials.
Table 3 summarizes the reported characteristics of PPI partners and methods used to involve PPI partners. Most PPI partners were adult patients (189 [59.0%]), followed by members of the public (107 [34.0%]), patient advocacy group members (95 [30.2%]) and older adult patients (80 [25.4%]). Most respondents reported first involving PPI partners at the pre-protocol or protocol development stage (246 [79.1%]). The most common aspects of trials in which PPI partners were involved were designing or developing interventions (217 [70.9%]), developing recruitment or retention strategies (185 [60.5%]) and designing recruitment materials (163 [53.3%]). Most often, PPI partners were involved in in-person meetings (295 [95.5%]) and email or online forums (128 [41.4%]).
Table 4 shows the reported perceived benefits and challenges of involving PPI partners. Of 334 respondents who included PPI partners in their research, 259 (77.8%) indicated at least 1 perceived benefit of doing so. Most often, respondents reported improved or more feasible interventions (184 [71.0%]), increased applicability or relevance of findings (153 [59.1%]), and improved recruitment or retention and higher-quality research (each 152 [58.7%]). Of the respondents, 142 (47.5%) reported that they experienced challenges with involving PPI partners. Among these 142 respondents, the most common challenges were identifying or recruiting PPI partners (64 [45.4%]); communicating about trial design, methods or results (57 [40.4%]); scheduling meetings (53 [37.6%]); and sustaining involvement of PPI partners throughout the study (51 [36.2%]). Eleven (5.7%) respondents cited other challenges, such as travel constraints, ill health or challenges arising from sensitive topics. Of the respondents who reported challenges with PPI, 109 (77.9%) strongly agreed that involving PPI partners was a positive experience and only 5 (3.5%) reported that they somewhat or strongly disagreed.
Table 5 shows the variation in prevalence of PPI across trial characteristics. Respondents who indicated that they did not know whether PPI had occurred in their trial were excluded from this analysis. Patient and public involvement was more common among trials including only children, trials conducted in the UK, cluster randomized trials, and trials explicitly using the term “pragmatic” to describe the trial in study manuscripts. Prevalence of PPI increased over time, from 46.4% in reports published in 2014 to 55.8% in 2019.
Table 6 shows the prevalence of PPI reported in the survey compared with that reported in study manuscripts. Although 334 (47.0%) of survey respondents stated that they used PPI, only 102 (14.4%) of manuscripts reported PPI. The overall agreement (i.e., those who reported PPI consistently across the survey and their manuscript) was 54.9%. However, of 334 survey respondents who said they used PPI (and could be matched to a trial publication), 76 (22.8%) clearly indicated PPI in the corresponding manuscripts and 75.4% did not. This suggests considerable under-reporting of PPI in trial manuscripts, or over-reporting by researchers in the survey.
Overall, missing data were not common: of 334 participants who indicated they conducted PPI, the highest survey item nonresponse rate was 10%; of 333 participants who did not conduct PPI, the highest survey item nonresponse rate was 7.8%.
Interpretation
In this descriptive study, we surveyed authors of trials deemed to be pragmatic and published between 2014 and 2019, to identify the prevalence and nature of PPI in these trials. Among 710 respondents, nearly half reported PPI. Respondents most frequently engaged adult patients via in-person meetings, email or online forums in the planning or design phases of research. Of those who conducted PPI, many cited higher applicability or relevance of research as motivation for doing so, whereas a lack of requirement to conduct PPI was the most common reason for not engaging PPI partners. Reported perceived benefits of PPI included improved or more feasible interventions, increased applicability or relevance of findings, and improved participant recruitment or retention. Common challenges included identifying or recruiting patient partners; communicating about trial design, methods and results; and clarifying roles and expectations. Trials more often involved PPI partners when they were pediatric trials, corresponding authors resided in the UK, used cluster randomization, or had been explicitly described as “pragmatic” in study manuscripts. Of the survey respondents who reported PPI in their trial, only one-quarter of these trial publications described PPI.
Reasons to engage PPI partners, methods to recruit PPI partners and responsibilities of PPI partners identified in our survey are consistent with other literature describing a range of research designs, including RCTs.33–37 The benefits of PPI reported in our survey have also been reported elsewhere, in addition to helping researchers secure funding when PPI partners are engaged at the grant-writing stage.38,39 In our findings, benefits of PPI were more frequently instrumental to research than intrinsic to researchers themselves, which may have implications for future evaluations of PPI.40 In addition to the limitations of PPI identified in our survey, others have reported lack of support,41 risks of tokenism and power imbalances, and scientific and ethical challenges when designing research alongside PPI partners.38,39,41–43 The higher prevalence of PPI reported in our survey, compared with information in the trial manuscripts, is consistent with other published studies. For example, a previous study describing a review of PPI in the UK (not limited to RCTs), which was followed by a survey of authors, found that 51% of research reports and manuscripts described PPI; however, 79% of authors surveyed reported PPI.27
Given that PPI aligns with the intention of pragmatic trials to produce patient-relevant evidence,7,8 it is not surprising that trials self-identifying as pragmatic (i.e., trials in which authors were so confident about the degree of pragmatism in their trial that they were willing to explicitly claim the label in the report) had a higher prevalence of reported PPI than those not using this label. It is also not surprising to find that researchers in the UK reported conducting PPI much more frequently than those in other countries, which may be related to the longstanding promotion of PPI by the National Institute for Health and Care Research Involve Patients and Be Part of Research.11 Neither is it expected that PPI increased over time, as other countries and funding agencies have encouraged researchers to engage PPI partners in recent years, such as the CIHR SPOR, which requires researchers to involve PPI partners. 6,10 To our knowledge, the finding that PPI was more prevalent among cluster randomized trials is novel. The reasons for this may need exploring but may be attributable to the fact that cluster trials often involve entire communities and test behavioural interventions, which may require a higher degree of stakeholder engagement to be successful.
Limitations
First, our response rate was just under 30%, which is similar or better than that achieved in previous PPI surveys.27,44,45 Given that this survey was administered to health researchers during the COVID-19 pandemic, many of whom are likely clinician scientists, usual expectations for survey response rates may need to be tempered, and so the generalizability of our findings may have been limited. As respondents conducted trials in 16 different countries, international differences in PPI policies and regulations, as well as varying emphases on PPI, may have contributed to heterogeneity, which was not explored in our analysis. We assessed the risk of selection bias by comparing trial characteristics of respondents and nonrespondents, and we were unable to identify substantial differences, except that respondents were more likely to have reported PPI in their trial. Self-selection bias in favour of those more likely to report PPI or who had experience with PPI, however, cannot be ruled out, as the prevalence of reported PPI was relatively high and only 7% of respondents reported less than 1 year of experience with PPI. This may have led to an overestimate of the prevalence of PPI but would likely not have affected other results regarding the nature of PPI conducted. Second, it is possible that some respondents did not have the same understanding of PPI as ours, which may have led to misclassification. We attempted to limit this risk by providing a clear definition of PPI and requiring authors to confirm that they had read and understood our definition. Misclassification owing to poor recall is also a possibility, as many trials were published several years before the survey. Fourth, our survey used a large database previously established as part of the larger project and was limited to trials published up to April 2019. If the use of PPI has been increasing in recent years, our estimate of the prevalence of PPI is likely an underestimate. Nonetheless, efforts and legislation to encourage researchers to engage PPI partners were well under way before our search in 2014 (for instance, since 2003 in the UK).
The search filter used to create the database of trials was designed to efficiently identify trials that were more likely to be pragmatic; thus, some potentially relevant trials (e.g., those not using terms such as “pragmatic” to describe their trial) may have been excluded. To the extent that the trials identified by our search are different from the trials not identified by our search, our results may be biased. However, when developing the search filter, a word-frequency analysis of trials excluded by the filter identified no systematic differences between trials captured or missed by the search.
Finally, although the perspectives we gathered in this survey are informative, they represent only those of researchers; soliciting PPI partner feedback would have enriched our findings.
Conclusion
Widespread education for trialists about PPI and standardized reporting guidelines about PPI in RCTs may address a number of gaps identified in this study. We were surprised to observe a relatively high prevalence of PPI reported in the survey results, in comparison with data on PPI reported in trial manuscripts.15 Very few peer-reviewed journals require authors to report PPI in research articles, and reporting checklists such as the Consolidated Standards of Reporting Trials (CONSORT) statement46 for RCTs and pragmatic trials do not require authors to indicate that PPI had been used, which makes it difficult to efficiently and systematically identify PPI in manuscripts. The most commonly reported reasons for electing not to conduct PPI were lack of a requirement to do so, belief that PPI was not relevant to the study, and insufficient researcher knowledge about PPI. Although PPI may not be suitable for every trial, it would be worthwhile to confirm whether PPI is relevant by asking PPI partners themselves. Journal requirements and ample space for trialists to describe their approach to PPI, even if they elect not to pursue it, may allow for greater transparency about gaps in PPI uptake and identify opportunities for guidance. Future qualitative work exploring these and other reasons why authors do not describe PPI in trial manuscripts may help inform strategies to improve reporting. Similarly, one of the common challenges trialists faced was communicating to PPI partners about their research, which further suggests that researchers involving PPI partners may benefit from support to do so. Together, these findings underscore the importance of education about PPI, not only to improve researchers’ awareness of involving patients and members of the public, but also to strengthen the methods used to do so and reduce the risk of challenges known to the field of PPI, such as tokenism, communication issues and power imbalances. Manuscripts substantially under-report PPI compared with investigator survey responses, suggesting the need for better reporting, but PPI did seem to increase over time. Requiring PPI as a condition of funding, providing resources to conduct PPI, and educating trialists about how to meaningfully involve patients and members of the public in RCTs would promote uptake of PPI, strengthen current approaches to engagement, and identify areas of challenge.
Acknowledgements
The authors acknowledge Dr. Paxton Montgomery Moon, Alison Howie, Hayden Nix and Dr. Merrick Zwarenstein for their contributions to the data extraction. They also thank Drs. Bruno Giraudeau and Agnes Caille (University of Tours), Dr. Laura Hanson (University of North Carolina School of Medicine) and Dr. Jill Harrison (Brown University) for assistance with pilot testing of the survey questionnaire.
Footnotes
Competing interests: Charles Weijer reports receiving consulting fees from Cardialen and Eli Lilly & Company. Shelley Vanderhout reports receiving a Postdoctoral Fellowship Award and a System Impact Fellowship Award from the Canadian Institutes of Health Research (in support of the current manuscript). Anne Spinewine reports receiving a Mobility Grant from Fondation Mont-Godinne. No other competing interests were declared.
This article has been peer reviewed.
Contributors: Monica Taljaard, Stuart Nicholls and Shelley Vanderhout contributed to the conception and design of the work. Alicia Hilderley, Maureen Smith, Pascale Nevins, Shelley Vanderhout, Monica Taljaard, Stuart Nicholls and Kelly Carroll contributed to the acquisition of data. Pascale Nevins, Monica Taljaard, Stuart Nicholls and Shelley Vanderhout contributed to the analysis and interpretation of data. Shelley Vanderhout and Monica Taljaard drafted the manuscript. All of the authors revised the manuscript critically for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work.
Funding: This work was supported by the Canadian Institutes of Health Research through the Project Grant competition (competitive, peer-reviewed), award number PJT-153045, and the National Institute of Aging ( NIA) of the National Institutes of Health under Award Number U54AG063546, which funds NIA Imbedded Pragmatic Alzheimer’s Disease and Related Dementias Clinical Trials Collaboratory ( NIA IMPACT Collaboratory). The funders had no role in the study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.
Data sharing: Data are available upon request to the corresponding author.
Supplemental information: For reviewer comments and the original submission of this manuscript, please see www.cmajopen.ca/content/11/5/E826/suppl/DC1.
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY-NC-ND 4.0) licence, which permits use, distribution and reproduction in any medium, provided that the original publication is properly cited, the use is noncommercial (i.e., research or educational use), and no modifications or adaptations are made. See: https://creativecommons.org/licenses/by-nc-nd/4.0/
References
- © 2023 CMA Impact Inc. or its licensors