Patient and public involvement in pragmatic trials: online survey of corresponding authors of published trials
==============================================================================================================

* Shelley Vanderhout
* Pascale Nevins
* Stuart G. Nicholls
* Colin Macarthur
* Jamie C. Brehaut
* Beth K. Potter
* Kate Gillies
* Beatriz Goulao
* Maureen Smith
* Alicia Hilderley
* Kelly Carroll
* Anne Spinewine
* Charles Weijer
* Dean A. Fergusson
* Monica Taljaard

## Abstract

**Background:** There are few data on patient and public involvement (PPI) in pragmatic trials. We aimed to describe the prevalence and nature of PPI within pragmatic trials, describe variation in prevalence of PPI by trial characteristics and compare prevalence of PPI reported by trial authors to that reported in trial publications.

**Methods:** We applied a search filter to identify pragmatic trials published from 2014 to 2019 in MEDLINE. We invited the corresponding authors of pragmatic trials to participate in an online survey about their specific trial.

**Results:** Of 3163 authors invited, 2585 invitations were delivered, 710 (27.5%) reported on 710 unique trials and completed the survey; 334 (47.0%) conducted PPI. Among those who conducted PPI, for many the aim was to increase the research relevance (86.3%) or quality (76.5%). Most PPI partners were engaged at protocol development stages (79.1%) and contributed to the co-design of interventions (70.9%) or recruitment or retention strategies (60.5%). Patient and public involvement was more common among trials involving children, trials conducted in the United Kingdom, cluster randomized trials, those explicitly labelled as “pragmatic” in the study manuscript, and more recent trials. Less than one-quarter of trials (22.8%) that reported PPI in the survey also reported PPI in the trial manuscript.

**Interpretation:** Nearly half of trialists in this survey reported conducting PPI and listed several benefits of doing so, but researchers who did not conduct PPI often cited a lack of requirement for it. Patient and public involvement appears to be significantly underreported in trial publications. Consistent and standardized reporting is needed to promote transparency about PPI methods, outcomes, challenges and benefits.

In recent years, there has been increasing interest in conducting randomized controlled trials (RCTs) that are pragmatic in orientation.1,2 Pragmatic trials use study settings and methods similar to usual care, as well as taking into account the perspectives of patients.3,4 Some of the design features of pragmatic trial design — for example, selecting outcomes that are important to patients5 — overlap with objectives of patient-oriented research.6–8 Patient and public involvement (PPI) in research provides an opportunity for patients and families to share their lived experiences and better align research with patient values and priorities.6 For the purpose of this research and in alignment with the Canadian Institutes of Health Research (CIHR) Strategy for Patient-Oriented Research (SPOR),6 we define PPI as “researchers consulting or working with members of the public, patients, service users, and carers in any or all part(s) of the research process, including the choice of research topic, design, planning, conduct or dissemination of research.”9 We refer to these individuals as “PPI partners”; partners could include, for example, grant co-applicants, members of a Trial Steering Committee, or members of a patient or public advisory panel. In health research, PPI is strongly encouraged by international funding bodies,6,10,11 institutions12 and stakeholder groups.13,14

Previous reviews of RCTs found low prevalence of PPI, with inconsistent reporting.15–21 However, no trial-specific reporting guidelines require authors to indicate whether their trial included PPI (although the generic Guidance for Reporting Involvement of Patients and the Public [GRIPP2]22 checklist may be used when PPI was part of an intervention), and journals rarely require PPI to be reported.17 Relying solely on information provided in published trial reports would likely lead to an underestimation of the true prevalence of PPI. To address this gap in PPI reporting, we conducted a survey of authors of pragmatic trials. The main objectives of the survey were:

1.  To describe the prevalence and nature of PPI within these trials; in particular:
    
    1.  when and how PPI was sought (e.g., integrated in trial design, conduct, analysis, dissemination)
    
    2.  who was involved (e.g., patients [including children and youth], parents or caregivers, family members, patient group representatives)
    
    3.  justification for involving or not involving PPI partners
    
    4.  perceived benefits or not of involving PPI partners
    
    5.  challenges faced in engaging PPI partners

2.  To describe variation in the prevalence of PPI by trial characteristics including trial population (pediatric v. older adults v. neither), country of study author, trial design (individual v. cluster randomization), pragmatism (explicitly labelled as “pragmatic” in the study manuscript v. not), and year of publication

3.  To compare the prevalence of PPI reported by trial authors in the survey to that reported in study publications.

## Methods

### Study design

This is a substudy of a study23 evaluating methodological and ethical issues in pragmatic trials. The main study involved the development and validation of an electronic search filter24 (Appendix 1, available at [www.cmajopen.ca/content/11/5/E826/suppl/DC1](http://www.cmajopen.ca/content/11/5/E826/suppl/DC1)) to efficiently find health-focused trials that were more likely to be pragmatic, because relying on authors to use the words “pragmatic,” “naturalistic” or “real-world” to describe their trials is inadequate for identifying pragmatic trials. The filter used terms related to common designs, settings and data sources of pragmatic trials, such as those captured in the Pragmatic Explanatory Continuum Indicator Summary (PRECIS-2) tool,5 and had an estimated sensitivity of 46% and estimated specificity of 98%.24 Of note, the search filter identifies trials clearly labelled by trial authors as “pragmatic” with 100% sensitivity. The filter was used in MEDLINE to identify primary trial reports published between Jan. 1, 2014, and Apr. 3, 2019. A descriptive analysis of trials identified from the search has been previously published.25 Here we describe the results from an online survey completed by the corresponding authors of the published primary reports, reported in accordance with the Checklist for Reporting Results of Internet E-Surveys (CHERRIES).

### Setting and participants

To facilitate conduct of the survey in English and to focus on the jurisdictions in which our investigator team has experience, we targeted corresponding authors in the following countries: Canada, United States, United Kingdom, Australia and New Zealand, South Africa, France, Belgium, Denmark, Finland, Germany, Italy, the Netherlands, Norway, Spain, Sweden and Switzerland. We selected only the most recently published report, even if the individual was corresponding author on multiple eligible studies. For a small fraction of corresponding authors with no email addresses listed, we used online searches to attempt to identify an email address.

### Data sources

We used 2 data sources: a survey questionnaire and a database from the broader study23 of previously extracted or downloaded trial characteristics. The survey questionnaire was developed based on previous surveys about PPI,20,26–28 the research team’s own experience, and the perspectives of 2 patient partners (M.S., A.H.) with expertise in patient engagement in research. We included these 2 patient partners from the study conceptualization phase, and they reviewed and contributed to the study protocol, survey content and manuscript, in addition to helping interpret study findings. Members of the international study team and colleagues with experience in pragmatic trials reviewed an initial draft of the survey questionnaire, to verify its usability and confirm that completion time would be about 10 minutes. The questionnaire consisted of 27 open- and closed-ended items pertaining to PPI in the published trial, including whether PPI had taken place, reasons for or against engaging PPI partners in the trial, characteristics of PPI partners, methods used to involve PPI partners, the stage of research when PPI partners were first engaged, and outcomes, benefits and challenges of PPI, followed by a series of demographic questions (see Appendix 2, available at [www.cmajopen.ca/content/11/5/E826/suppl/DC1](http://www.cmajopen.ca/content/11/5/E826/suppl/DC1), for the survey questionnaire). The questionnaire included a definition of PPI and respondents had to indicate that they had read and understood the definition before completing the survey. To encourage respondents to provide as much information as possible, the only mandatory question was whether they had used PPI in the trial.

To assess potential nonresponse bias, we compared trial characteristics of respondents and nonrespondents using previously extracted or downloaded information available within the larger database:29,30 country of corresponding author, year of publication, clinical or disease area (obtained from Web of Science), unit of randomization (cluster v. individual randomization), journal impact factor, and age of trial participants (as described in Appendix 3, available at [www.cmajopen.ca/content/11/5/E826/suppl/DC1](http://www.cmajopen.ca/content/11/5/E826/suppl/DC1)). For trials registered in [ClinicalTrials.gov](http://ClinicalTrials.gov), we examined primary trial purpose (classified as treatment, prevention, diagnostic, supportive care, screening, health services research or other) and type of experimental intervention (classified as drug, device, biological or vaccine, procedure or surgery, radiation, genetic, dietary supplement, educational or behavioural or other). Finally, we also obtained information about PPI reporting in the manuscript from a smaller subset of trials included in a previous substudy.31,32

### Survey administration

We administered the survey via SurveyMonkey. We modified Dillman’s Tailored Design Method, which included personalized invitations, a visually appealing survey, incentives and a reminder schedule. Invitation emails (Appendix 4, available at [www.cmajopen.ca/content/11/5/E826/suppl/DC1](http://www.cmajopen.ca/content/11/5/E826/suppl/DC1)) were personalized with the name of the corresponding author, title, year of publication and journal, as well as a unique link to the survey that corresponded to their trial. This allowed for survey responses to be analyzed in combination with trial characteristics already in our database and prevent more than 1 survey response for each trial. Authors were permitted to forward the survey for completion to another member of the research team. The survey was open; i.e., no password was required to complete it. Survey participants were given the option to be entered into a draw for one of 5 Can$100 Amazon gift cards. We piloted the survey administration with a random sample of 100 corresponding authors, to identify and resolve any technical issues with automated invitations. The initial pilot invitation was sent via SurveyMonkey on Nov. 26, 2021, followed by the first reminder 10 days later and second reminder 36 days later (to accommodate the end-of-year holiday period). No content changes were considered necessary after the pilot, and pilot data were therefore included in the final analysis. Subsequently, we sent personalized invitation emails to all remaining eligible corresponding authors in our sampling frame, via SurveyMonkey, on Feb. 8, 2022, followed by the first reminder 2 weeks later and second reminder another week after that. We sent the third (final) reminder between Mar. 15 and Mar. 18, 2022, from the senior author’s institutional email address rather than SurveyMonkey, in an attempt to bypass spam filters that might have prevented the first 2 invitations from reaching potential participants. This invitation included a unique identifier to be entered by each respondent to allow linkage with their trial. The survey was closed on Apr. 5, 2022.

All participants were prompted to review a participant information sheet at the start of the survey (Appendix 2) and notified that the survey was voluntary and continuing implied consent.

### Statistical analysis

We exported survey data from SurveyMonkey into a spreadsheet via Airtable. We summarized participant characteristics using descriptive statistics. Using search filters applied to our database (Appendix 4), we categorized trials for descriptive purposes into 1 of 3 groups: pediatric trials (trial participants were children aged 0–18 yr, or the primary outcome indicated that the study was child focused, or both), older adult trials (mean or median age of trial participants was ≥ 65 yr), or neither.

We analyzed survey data using descriptive statistics (frequencies and percentages). We analyzed all survey responses regardless of completeness. Most survey items were closed ended, but many allowed respondents to select “Other” and elaborate in a text box. Responses to “Other” were collated and reviewed by 2 study team members independently to determine whether the response could be reclassified into an existing category or warranted creation of a new category. Disagreements were resolved through discussion.

To describe variation in the prevalence and nature of PPI, we cross-tabulated trial characteristics — including trial population (pediatric v. older adults v. neither), country of corresponding author, trial design (individual v. cluster randomization), whether the trial was explicitly labelled as “pragmatic” in the study manuscript versus not, and year of publication — against self-reported PPI in the trial.

To compare the prevalence of PPI reported in the survey to that reported in the trial publication, one of 3 authors (S.V., K.C., P.N.) reviewed final trial reports to assess whether PPI was reported or acknowledged, and this was compared with survey responses. Patient and public involvement in the publication was defined as explicit reporting on patient or public involvement, elicitation of patient or public perspectives, or mention of PPI partners in author lists or in the manuscript “Acknowledgements” section.

### Ethics approval

This study was approved by the Ottawa Health Science Network Research Ethics Board.

## Results

The parent study identified 4336 eligible primary trial reports, of which we excluded 835 (19%) as ineligible because of the country of the corresponding author; a further 335 (7.7%) were excluded as they were multiple publications by the same trial author, and a further 3 (0.1%) for which the corresponding author email address could not be identified. Thus, we invited 3163 unique corresponding authors to complete the survey (Figure 1). Of these, 710 ultimately reported on unique trials and completed the survey for an overall response rate (among all invited authors) of 710/3163 (22.4%) or 710/2585 (27.5%) after we removed undeliverable email addresses. We classified authors as respondents if they completed at least the primary question of interest (whether they used PPI in the trial).

![Figure 1:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/11/5/E826/F1.medium.gif)

[Figure 1:](http://www.cmajopen.ca/content/11/5/E826/F1)

Figure 1: 
Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) diagram showing exclusions from the study. Note: OHRI = Ottawa Hospital Research Institute.

Respondent characteristics are shown in Table 1. Respondents mostly resided in the US (247 [37.7%]), Europe — excluding the UK (166 [25.3%]) — and the UK (110 [16.8%]). Most respondents were females (364 [55.6%]). Respondent ages were most often 46–55 years (203 [31.0%]) or 56–65 years (185 [28.3%]), with the majority indicating they were late career researchers (> 15 yr since first academic appointment; 402 [61.5%]). Most respondents reported “White” as their racial or ethnic group (550, 84.2%). Of respondents, 54% (353) reported having more than 10 years’ experience in conducting PPI in research, followed by 29.1% (190) reporting 4–10 years’ experience; 47 (7.2%) respondents reported less than 1 year of experience with PPI. We observed no major differences between survey respondents and nonrespondents based on the available characteristics, except that the prevalence of reporting PPI in trial reports was slightly higher among respondents (12.4%) than among nonrespondents (7.8%) (Appendix 5, available at [www.cmajopen.ca/content/11/5/E826/suppl/DC1](http://www.cmajopen.ca/content/11/5/E826/suppl/DC1)).

View this table:
[Table 1:](http://www.cmajopen.ca/content/11/5/E826/T1)

Table 1: 
Survey respondent characteristics (only authors who provided demographic information)

Table 2 shows the proportion of survey respondents who reported involving PPI partners in their trial, and stated reasons for involving or not involving PPI partners. Of all 710 respondents, 334 (47.0%) reported that they involved PPI partners, 333 (46.9%) reported that they did not involve PPI partners, and 43 (6.1%) stated that they didn’t know whether PPI partners were involved. Among the 334 who reported involving PPI partners, reasons for PPI were reported by 315 (94.3%). Most commonly reported reasons were increased applicability or relevance of research (272 [86.3%]), increased quality of research (241 [76.5%]) and it being the morally or ethically right thing to do (204 [64.8%]). Ten (3.2%) survey respondents cited other reasons for engaging PPI partners, such as increased pragmatism or because the research question originated from a patient. Among the 333 who reported that they did not involve PPI partners, reasons were selected by 307 (92.2%). Most commonly reported reasons were that there was no requirement to do so (118 [38.4%]), it did not seem relevant (99 [32.3%]), and there was a lack of resources or funds (79 [25.7%]). Twelve (3.9%) respondents indicated other reasons for not involving PPI partners, such as conducting a replication study that was based on a trial that had already involved PPI partners, or lack of available flexibility to modify study designs or materials.

View this table:
[Table 2:](http://www.cmajopen.ca/content/11/5/E826/T2)

Table 2: 
Frequency of PPI reported in the survey, and rationale for involving or not involving PPI partners

Table 3 summarizes the reported characteristics of PPI partners and methods used to involve PPI partners. Most PPI partners were adult patients (189 [59.0%]), followed by members of the public (107 [34.0%]), patient advocacy group members (95 [30.2%]) and older adult patients (80 [25.4%]). Most respondents reported first involving PPI partners at the pre-protocol or protocol development stage (246 [79.1%]). The most common aspects of trials in which PPI partners were involved were designing or developing interventions (217 [70.9%]), developing recruitment or retention strategies (185 [60.5%]) and designing recruitment materials (163 [53.3%]). Most often, PPI partners were involved in in-person meetings (295 [95.5%]) and email or online forums (128 [41.4%]).

View this table:
[Table 3:](http://www.cmajopen.ca/content/11/5/E826/T3)

Table 3: 
Characteristics of PPI partners and methods used to involve PPI partners among those reporting PPI in the survey

Table 4 shows the reported perceived benefits and challenges of involving PPI partners. Of 334 respondents who included PPI partners in their research, 259 (77.8%) indicated at least 1 perceived benefit of doing so. Most often, respondents reported improved or more feasible interventions (184 [71.0%]), increased applicability or relevance of findings (153 [59.1%]), and improved recruitment or retention and higher-quality research (each 152 [58.7%]). Of the respondents, 142 (47.5%) reported that they experienced challenges with involving PPI partners. Among these 142 respondents, the most common challenges were identifying or recruiting PPI partners (64 [45.4%]); communicating about trial design, methods or results (57 [40.4%]); scheduling meetings (53 [37.6%]); and sustaining involvement of PPI partners throughout the study (51 [36.2%]). Eleven (5.7%) respondents cited other challenges, such as travel constraints, ill health or challenges arising from sensitive topics. Of the respondents who reported challenges with PPI, 109 (77.9%) strongly agreed that involving PPI partners was a positive experience and only 5 (3.5%) reported that they somewhat or strongly disagreed.

View this table:
[Table 4:](http://www.cmajopen.ca/content/11/5/E826/T4)

Table 4: 
Outcomes of PPI among those reporting PPI in the survey

Table 5 shows the variation in prevalence of PPI across trial characteristics. Respondents who indicated that they did not know whether PPI had occurred in their trial were excluded from this analysis. Patient and public involvement was more common among trials including only children, trials conducted in the UK, cluster randomized trials, and trials explicitly using the term “pragmatic” to describe the trial in study manuscripts. Prevalence of PPI increased over time, from 46.4% in reports published in 2014 to 55.8% in 2019.

View this table:
[Table 5:](http://www.cmajopen.ca/content/11/5/E826/T5)

Table 5: 
Variation in prevalence and nature of PPI across trial characteristics among those who did and did not report PPI

Table 6 shows the prevalence of PPI reported in the survey compared with that reported in study manuscripts. Although 334 (47.0%) of survey respondents stated that they used PPI, only 102 (14.4%) of manuscripts reported PPI. The overall agreement (i.e., those who reported PPI consistently across the survey and their manuscript) was 54.9%. However, of 334 survey respondents who said they used PPI (and could be matched to a trial publication), 76 (22.8%) clearly indicated PPI in the corresponding manuscripts and 75.4% did not. This suggests considerable under-reporting of PPI in trial manuscripts, or over-reporting by researchers in the survey.

View this table:
[Table 6:](http://www.cmajopen.ca/content/11/5/E826/T6)

Table 6: 
Patient and public involvement (PPI) reported in trial manuscripts compared with PPI reported in survey

Overall, missing data were not common: of 334 participants who indicated they conducted PPI, the highest survey item nonresponse rate was 10%; of 333 participants who did not conduct PPI, the highest survey item nonresponse rate was 7.8%.

## Interpretation

In this descriptive study, we surveyed authors of trials deemed to be pragmatic and published between 2014 and 2019, to identify the prevalence and nature of PPI in these trials. Among 710 respondents, nearly half reported PPI. Respondents most frequently engaged adult patients via in-person meetings, email or online forums in the planning or design phases of research. Of those who conducted PPI, many cited higher applicability or relevance of research as motivation for doing so, whereas a lack of requirement to conduct PPI was the most common reason for not engaging PPI partners. Reported perceived benefits of PPI included improved or more feasible interventions, increased applicability or relevance of findings, and improved participant recruitment or retention. Common challenges included identifying or recruiting patient partners; communicating about trial design, methods and results; and clarifying roles and expectations. Trials more often involved PPI partners when they were pediatric trials, corresponding authors resided in the UK, used cluster randomization, or had been explicitly described as “pragmatic” in study manuscripts. Of the survey respondents who reported PPI in their trial, only one-quarter of these trial publications described PPI.

Reasons to engage PPI partners, methods to recruit PPI partners and responsibilities of PPI partners identified in our survey are consistent with other literature describing a range of research designs, including RCTs.33–37 The benefits of PPI reported in our survey have also been reported elsewhere, in addition to helping researchers secure funding when PPI partners are engaged at the grant-writing stage.38,39 In our findings, benefits of PPI were more frequently instrumental to research than intrinsic to researchers themselves, which may have implications for future evaluations of PPI.40 In addition to the limitations of PPI identified in our survey, others have reported lack of support,41 risks of tokenism and power imbalances, and scientific and ethical challenges when designing research alongside PPI partners.38,39,41–43 The higher prevalence of PPI reported in our survey, compared with information in the trial manuscripts, is consistent with other published studies. For example, a previous study describing a review of PPI in the UK (not limited to RCTs), which was followed by a survey of authors, found that 51% of research reports and manuscripts described PPI; however, 79% of authors surveyed reported PPI.27

Given that PPI aligns with the intention of pragmatic trials to produce patient-relevant evidence,7,8 it is not surprising that trials self-identifying as pragmatic (i.e., trials in which authors were so confident about the degree of pragmatism in their trial that they were willing to explicitly claim the label in the report) had a higher prevalence of reported PPI than those not using this label. It is also not surprising to find that researchers in the UK reported conducting PPI much more frequently than those in other countries, which may be related to the longstanding promotion of PPI by the National Institute for Health and Care Research Involve Patients and Be Part of Research.11 Neither is it expected that PPI increased over time, as other countries and funding agencies have encouraged researchers to engage PPI partners in recent years, such as the CIHR SPOR, which requires researchers to involve PPI partners. 6,10 To our knowledge, the finding that PPI was more prevalent among cluster randomized trials is novel. The reasons for this may need exploring but may be attributable to the fact that cluster trials often involve entire communities and test behavioural interventions, which may require a higher degree of stakeholder engagement to be successful.

### Limitations

First, our response rate was just under 30%, which is similar or better than that achieved in previous PPI surveys.27,44,45 Given that this survey was administered to health researchers during the COVID-19 pandemic, many of whom are likely clinician scientists, usual expectations for survey response rates may need to be tempered, and so the generalizability of our findings may have been limited. As respondents conducted trials in 16 different countries, international differences in PPI policies and regulations, as well as varying emphases on PPI, may have contributed to heterogeneity, which was not explored in our analysis. We assessed the risk of selection bias by comparing trial characteristics of respondents and nonrespondents, and we were unable to identify substantial differences, except that respondents were more likely to have reported PPI in their trial. Self-selection bias in favour of those more likely to report PPI or who had experience with PPI, however, cannot be ruled out, as the prevalence of reported PPI was relatively high and only 7% of respondents reported less than 1 year of experience with PPI. This may have led to an overestimate of the prevalence of PPI but would likely not have affected other results regarding the nature of PPI conducted. Second, it is possible that some respondents did not have the same understanding of PPI as ours, which may have led to misclassification. We attempted to limit this risk by providing a clear definition of PPI and requiring authors to confirm that they had read and understood our definition. Misclassification owing to poor recall is also a possibility, as many trials were published several years before the survey. Fourth, our survey used a large database previously established as part of the larger project and was limited to trials published up to April 2019. If the use of PPI has been increasing in recent years, our estimate of the prevalence of PPI is likely an underestimate. Nonetheless, efforts and legislation to encourage researchers to engage PPI partners were well under way before our search in 2014 (for instance, since 2003 in the UK).

The search filter used to create the database of trials was designed to efficiently identify trials that were more likely to be pragmatic; thus, some potentially relevant trials (e.g., those not using terms such as “pragmatic” to describe their trial) may have been excluded. To the extent that the trials identified by our search are different from the trials not identified by our search, our results may be biased. However, when developing the search filter, a word-frequency analysis of trials excluded by the filter identified no systematic differences between trials captured or missed by the search.

Finally, although the perspectives we gathered in this survey are informative, they represent only those of researchers; soliciting PPI partner feedback would have enriched our findings.

### Conclusion

Widespread education for trialists about PPI and standardized reporting guidelines about PPI in RCTs may address a number of gaps identified in this study. We were surprised to observe a relatively high prevalence of PPI reported in the survey results, in comparison with data on PPI reported in trial manuscripts.15 Very few peer-reviewed journals require authors to report PPI in research articles, and reporting checklists such as the Consolidated Standards of Reporting Trials (CONSORT) statement46 for RCTs and pragmatic trials do not require authors to indicate that PPI had been used, which makes it difficult to efficiently and systematically identify PPI in manuscripts. The most commonly reported reasons for electing not to conduct PPI were lack of a requirement to do so, belief that PPI was not relevant to the study, and insufficient researcher knowledge about PPI. Although PPI may not be suitable for every trial, it would be worthwhile to confirm whether PPI is relevant by asking PPI partners themselves. Journal requirements and ample space for trialists to describe their approach to PPI, even if they elect not to pursue it, may allow for greater transparency about gaps in PPI uptake and identify opportunities for guidance. Future qualitative work exploring these and other reasons why authors do not describe PPI in trial manuscripts may help inform strategies to improve reporting. Similarly, one of the common challenges trialists faced was communicating to PPI partners about their research, which further suggests that researchers involving PPI partners may benefit from support to do so. Together, these findings underscore the importance of education about PPI, not only to improve researchers’ awareness of involving patients and members of the public, but also to strengthen the methods used to do so and reduce the risk of challenges known to the field of PPI, such as tokenism, communication issues and power imbalances. Manuscripts substantially under-report PPI compared with investigator survey responses, suggesting the need for better reporting, but PPI did seem to increase over time. Requiring PPI as a condition of funding, providing resources to conduct PPI, and educating trialists about how to meaningfully involve patients and members of the public in RCTs would promote uptake of PPI, strengthen current approaches to engagement, and identify areas of challenge.

## Acknowledgements

The authors acknowledge Dr. Paxton Montgomery Moon, Alison Howie, Hayden Nix and Dr. Merrick Zwarenstein for their contributions to the data extraction. They also thank Drs. Bruno Giraudeau and Agnes Caille (University of Tours), Dr. Laura Hanson (University of North Carolina School of Medicine) and Dr. Jill Harrison (Brown University) for assistance with pilot testing of the survey questionnaire.

## Footnotes

*   **Competing interests:** Charles Weijer reports receiving consulting fees from Cardialen and Eli Lilly & Company. Shelley Vanderhout reports receiving a Postdoctoral Fellowship Award and a System Impact Fellowship Award from the Canadian Institutes of Health Research (in support of the current manuscript). Anne Spinewine reports receiving a Mobility Grant from Fondation Mont-Godinne. No other competing interests were declared.

*   This article has been peer reviewed.

*   **Contributors:** Monica Taljaard, Stuart Nicholls and Shelley Vanderhout contributed to the conception and design of the work. Alicia Hilderley, Maureen Smith, Pascale Nevins, Shelley Vanderhout, Monica Taljaard, Stuart Nicholls and Kelly Carroll contributed to the acquisition of data. Pascale Nevins, Monica Taljaard, Stuart Nicholls and Shelley Vanderhout contributed to the analysis and interpretation of data. Shelley Vanderhout and Monica Taljaard drafted the manuscript. All of the authors revised the manuscript critically for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work.

*   **Funding:** This work was supported by the Canadian Institutes of Health Research through the Project Grant competition (competitive, peer-reviewed), award number PJT-153045, and the National Institute of Aging ( NIA) of the National Institutes of Health under Award Number U54AG063546, which funds NIA Imbedded Pragmatic Alzheimer’s Disease and Related Dementias Clinical Trials Collaboratory ( NIA IMPACT Collaboratory). The funders had no role in the study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.

*   **Data sharing:** Data are available upon request to the corresponding author.

*   **Supplemental information:** For reviewer comments and the original submission of this manuscript, please see [www.cmajopen.ca/content/11/5/E826/suppl/DC1](http://www.cmajopen.ca/content/11/5/E826/suppl/DC1).

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY-NC-ND 4.0) licence, which permits use, distribution and reproduction in any medium, provided that the original publication is properly cited, the use is noncommercial (i.e., research or educational use), and no modifications or adaptations are made. See: [https://creativecommons.org/licenses/by-nc-nd/4.0/](https://creativecommons.org/licenses/by-nc-nd/4.0/)

## References

1.  Patsopoulos NA (2011) A pragmatic view on pragmatic trials. Dialogues Clin Neurosci 13:217–24.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.31887/DCNS.2011.13.2/npatsopoulos&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=21842619&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

2.  Zwarenstein M (2017) ‘Pragmatic’ and ‘explanatory’ attitudes to randomised trials. J R Soc Med 110:208–18.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1177/0141076817706303&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=28504072&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

3.  Nicholls SG, Carroll K, Zwarenstein M, et al., Ethics of Pragmatic Trials project (2019) The ethical challenges raised in the design and conduct of pragmatic trials: an interview study with key stakeholders. Trials 20:765.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

4.  Zvonareva O (2021) The pragmatic turn in clinical research: in search for the real world. Sci Technol Stud 34:6–24.
    
    

5.  Loudon K, Treweek S, Sullivan F, et al. (2015) The PRECIS-2 tool: designing trials that are fit for purpose. BMJ 350:h2147.
    
    [FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNTAvbWF5MDhfMS9oMjE0NyI7czo0OiJhdG9tIjtzOjIxOiIvY21ham8vMTEvNS9FODI2LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

6.  Strategy for Patient-Oriented Research (Canadian Institutes of Health Research, Ottawa) Available: [https://cihr-irsc.gc.ca/e/41204.html](https://cihr-irsc.gc.ca/e/41204.html). accessed 2021 Feb. 25.
    
    

7.  Schwartz D, Lellouch J (1967) Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis 20:637–48.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/0021-9681(67)90041-0&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=4860352&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 
    
    [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A19679847800009&link_type=ISI) 

8.  Heneghan C, Goldacre B, Mahtani KR (2017) Why clinical trial outcomes fail to translate into benefits for patients. Trials 18:122.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/s13063- 017-1870-2&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

9.  Hoddinott P, Pollock A, O’Cathain A, et al. (2018) How to incorporate patient and public perspectives into the design and conduct of research. F1000Res 7:752.
    
    

10. Patient-Centered Outcomes Research Institute (PCORI) [home page], Available: [https://www.pcori.org/](https://www.pcori.org/). accessed 2021 Feb. 25.
    
    

11. Involve patients (National Institute for Health and Care Research), Available: [https://www.nihr.ac.uk/health-and-care-professionals/engagement-and-participation-in-research/involve-patients.htm](https://www.nihr.ac.uk/health-and-care-professionals/engagement-and-participation-in-research/involve-patients.htm). accessed 2021 Aug. 24.
    
    

12. Research Family Engagement Committee (Holland Bloorview Kids Rehabilitation Hospital, Toronto) Available: [https://hollandbloorview.ca/get-involved-give/family-leadership-program/research-family-engagement-committee](https://hollandbloorview.ca/get-involved-give/family-leadership-program/research-family-engagement-committee). accessed 2021 Aug. 24.
    
    

13. CHILD-BRIGHT Network [home page], Available: [https://www.child-bright.ca](https://www.child-bright.ca). accessed 2021 Aug. 24.
    
    

14. Patient engagement (Can-SOLVE CKD Network, Vancouver) Available: [https://cansolveckd.ca/patient-engagement/](https://cansolveckd.ca/patient-engagement/). accessed 2022 June 3.
    
    

15. Fergusson D, Monfaredi Z, Pussegoda K, et al. (2018) The prevalence of patient engagement in published trials: a systematic review. Res Involv Engagem 4:17.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

16. Owyang D, Bakhsh A, Brewer D, et al. (2021) Patient and public involvement within orthopaedic research: a systematic review. J Bone Joint Surg Am 103:e51.
    
    

17. Gray R, Brasier C, Zirnsak T-M, et al. (2021) Reporting of patient and public involvement and engagement (PPIE) in clinical trials published in nursing science journals: a descriptive study. Res Involv Engagem 7:88.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

18. Benizri N, Hallot S, Burns K, et al. (2022) Patient and family representation in randomized clinical trials published in 3 medical and surgical journals: a systematic review. JAMA Netw Open 5:e2230858.
    
    

19. Musbahi A, Clyde D, Small P, et al. (2022) A systematic review of patient and public involvement (PPI) in bariatric research trials: the need for more work. Obes Surg 32:3740–51.
    
    

20. Gamble C, Dudley L, Allam A, et al. (2014) Patient and public involvement in the early stages of clinical trial development: a systematic cohort investigation. BMJ Open 4:e005234.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiNC83L2UwMDUyMzQiO3M6NDoiYXRvbSI7czoyMToiL2NtYWpvLzExLzUvRTgyNi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

21. Wallace SKA, Goulding KR, Myles PS (2022) Consumer engagement and patient reported outcomes in perioperative clinical trials in Australia: a systematic review. ANZ J Surg 92:2464–73.
    
    

22. Staniszewska S, Brett J, Simera I, et al. (2017) GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ 358:j3453.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNTgvYXVnMDJfNS9qMzQ1MyI7czo0OiJhdG9tIjtzOjIxOiIvY21ham8vMTEvNS9FODI2LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

23. Taljaard M, Weijer C, Grimshaw JM, et al. (2018) Developing a framework for the ethical design and conduct of pragmatic trials in healthcare: a mixed methods research protocol. Trials 19:525.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

24. Taljaard M, McDonald S, Nicholls SG, et al. (2020) A search filter to identify pragmatic trials in MEDLINE was highly specific but lacked sensitivity. J Clin Epidemiol 124:75–84.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

25. Nicholls SG, Carroll K, Phillips Hey S, et al. (2021) A review of pragmatic trials found a high degree of diversity in design and scope, deficiencies in reporting and trial registry data, and poor indexing. J Clin Epidemiol 137:45–57.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

26. Crocker JC, Pratt-Boyden K, Hislop J, et al. (2019) Patient and public involvement (PPI) in UK surgical trials: a survey and focus groups with stakeholders to identify practices, views, and experiences. Trials 20:119.
    
    

27. Wilson P, Mathie E, Keenan J, et al. (2015) Health Services and Delivery Research No 3.38, ReseArch with Patient and Public invOlvement: a RealisT evaluation — the RAPPORT study (NIHR Journals Library, Southampton (UK)) Available: [http://www.ncbi.nlm.nih.gov/books/NBK315995/](http://www.ncbi.nlm.nih.gov/books/NBK315995/). accessed 2021 June 6.
    
    

28. Ellis LE, Kass NE (2017) How are PCORI-funded researchers engaging patients in research and what are the ethical implications? AJOB Empir Bioeth 8:1–10.
    
    

29. Zhang JZ, Nicholls SG, Carroll K, et al. (Nov 15, 2021) Informed consent in pragmatic trials: results from a survey of trials published 2014–2019. J Med Ethics doi:10.1136/medethics-2021-107765, [Epub ahead of print].
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1136/medethics-2021-107765&link_type=DOI) 

30. Nicholls SG, Carroll K, Phillips Hey S, et al. (2021) A review of pragmatic trials found a high degree of diversity in design and scope, deficiencies in reporting and registry data, and poor indexing. J Clin Epidemiol 137:45–57.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

31. Vanderhout S, Fergusson DA, Cook JA, et al. (2022) Patient-reported outcomes and target effect sizes in pragmatic randomized trials in ClinicalTrials.gov: a cross-sectional analysis. PLoS Med 19:e1003896.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

32. Taljaard M, Nicholls SG, Howie AH, et al. (2022) An analysis of published trials found that current use of pragmatic trial labels is uninformative. J Clin Epidemiol 151:113–21.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

33. Blackburn S, McLachlan S, Jowett S, et al. (2018) The extent, quality and impact of patient and public involvement in primary care research: a mixed methods study. Res Involv Engagem 4:16.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/s40900-018-0100-8&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

34. Goulao B, Bruhn H, Campbell M, et al. (2021) Patient and public involvement in numerical aspects of trials (PoINT): exploring patient and public partners experiences and identifying stakeholder priorities. Trials 22:499.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

35. Smith SK, Selig W, Harker M, et al. (2015) Patient engagement practices in clinical research among patient groups, industry, and academia in the United States: a survey. PLoS One 10:e0140232.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1371/journal.pone.0140232&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=26465328&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

36. Chudyk AM, Stoddard R, McCleary N, et al. (2022) Activities and impacts of patient engagement in CIHR SPOR funded research: a cross-sectional survey of academic researcher and patient partner experiences. Res Involv Engagem 8:44.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/s40900-022-00376-4&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=36038887&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

37. Selman LE, Clement C, Douglas M, et al. (2021) Patient and public involvement in randomised clinical trials: a mixed-methods study of a clinical trials unit to identify good practice, barriers and facilitators. Trials 22:735.
    
    

38. Vat LE, Finlay T, Jan Schuitmaker-Warnaar T, et al. (2020) Evaluating the “return on patient engagement initiatives” in medicines research and development: a literature review. Health Expect 23:5–18.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1111/hex.12951&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

39. Domecq JP, Prutsky G, Elraiyah T, et al. (2014) Patient engagement in research: a systematic review. BMC Health Serv Res 14:89.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/1472-6963-14-89&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=24568690&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

40. Concannon TW, Grant S, Welch V, et al., Multi Stakeholder Engagement (MuSE) Consortium (2019) Practical guidance for involving stakeholders in health research. J Gen Intern Med 34:458–63.
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

41. Ocloo J, Matthews R (2016) From tokenism to empowerment: progressing patient and public involvement in healthcare improvement. BMJ Qual Saf 25:626–32.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI1LzgvNjI2IjtzOjQ6ImF0b20iO3M6MjE6Ii9jbWFqby8xMS81L0U4MjYuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

42. McKenzie A, Bowden J, Zalcberg JR, et al. (2022) A snapshot of consumer engagement in clinical trials in Australia: results of a national survey of clinical trial networks and research organisations. Res Involv Engagem 8:3.
    
    

43. Martineau JT, Minyaoui A, Boivin A (2020) Partnering with patients in healthcare research: a scoping review of ethical issues, challenges, and recommendations for practice. BMC Med Ethics 21:34.
    
    

44. Weeks L, Polisena J, Scott AM, et al. (2017) Evaluation of patient and public involvement initiatives in health technology assessment: a survey of international agencies. Int J Technol Assess Health Care 33:715–23.
    
    

45. Patel VA, Shelswell J, Hillyard N, et al. (2021) A study of the reporting of patient and public involvement and engagement (PPIE) in orthodontic research. J Orthod 48:42–51.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1177/1465312520968574&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

46. Zwarenstein M, Treweek S, Gagnier JJ, et al., CONSORT group; Pragmatic Trials in Healthcare (Practihc) group (2008) Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 337:a2390.
    
    [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzMzcvbm92MTFfMi9hMjM5MCI7czo0OiJhdG9tIjtzOjIxOiIvY21ham8vMTEvNS9FODI2LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

47. Schulz KF, Altman DG, Moher D, Consort Group (2011) CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. Int J Surg 9:672–7.
    
    [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.ijsu.2011.09.004&link_type=DOI) 
    
    [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=22019563&link_type=MED&atom=%2Fcmajo%2F11%2F5%2FE826.atom) 

*   © 2023 CMA Impact Inc. or its licensors