Toolkit
What is patient and public involvement?
This section explains the difference between patient and public involvement and patient participation in COS studies
Patient and public involvement (PPI) in your COS study – PPI is where research is “carried out ‘with’ or ‘by’ members of the public” not just “‘to’, ‘about’ or ‘for’ them”. For example, patients may help you to design, oversee and disseminate your COS study. You may hear different terms used in different countries to describe the same activity, for example the terms consumer involvement, patient or public engagement.
Patient participation in research – PPI is different to patient / public participation in core outcome set (COS) development. When patients / the public take part in COS development they are giving their views about outcomes and how important they consider them to be. Participation in COS studies generally requires ethical approval, whereas in many countries PPI does not.
Click the image to watch a video about patient and public involvement in core outcome set development.
Planning for patient & public involvement
This toolkit section helps you plan for patient and public involvement (PPI) in your COS study.
Potential benefits of PPI in research might include:
- Defining a more relevant scope for your COS study
- Designing more accessible research materials (e.g. refining interview topic guides or the wording on Delphi surveys, so it is clearer)
- Using a broader range of patient recruitment and retention approaches
- Understanding potential ethical considerations
- Interpreting your research findings from alternative viewpoints, for example, involving patients in interpreting qualitative interview transcripts to explore which outcomes should be included in the consensus aspects of your COS.
- Distributing your research findings to research participants and the general public using accessible approaches.
Relevant PPI resources
- Brief videos explaining the potential impact of PPI in wider research are available.
Planning for PPI takes time and resource. To maximise the potential benefits of PPI and avoid tokenism, we recommend that you involve patients / the public from the outset of your study. This will help you design your COS study through all stages, from the initial idea to disseminating your results and ultimately promoting COS uptake.
When planning for patient involvement in your COS study, consider questions like:
- Who are our patients/public – for example, might they be carers, children and young people, the general public etc.?
- What might the public research partner(s) (PRP) bring to our COS study?
- What impact could the PPI experience have on the PRPs in our COS study?
- What funding is likely to be available to support PPI in our COS study and how can we access this?
- What model of PPI are we hoping for? For example, some studies might have a separate PPI panel, others might have PRPs as team members on a study management group, others have a mixture of such approaches – or do you have a different model in mind?
- What skills/experience will be important for the public research partners to have?
- How will you and the rest of your research team support the PRP(s) throughout their involvement?
Relevant PPI resources
- Guidance on co-production is available from the NIHR in the UK
- The International PPI Network hosted a webinar on Co-production in action
- This publication explores ‘who’ to involve in your research.
- The Standards for Public Involvement in Research have been developed in the UK and are designed to improve the quality and consistency of PPI in research.
- General guidance on PPI – Examples of general guidance on PPI in research include that developed by the NIHR in the UK, by SPOR in Canada and by PCORI in the USA. Other countries may well have other guidance documents. The International PPI Network hosted a webinar entitled Patient and public involvement in research: What, why and how?
The following issues will need considering when planning for PPI in your COS study:
- Costing for PPI
- Finding PRPs
- Explaining your COS work to potential PRPs
- Explaining PPI in research to potential PRPs
- Recruiting PRPs
- Supporting PRPs throughout your COS study
- Evaluating PRPs’ experiences of the study and its impact on them
- Involving special groups
Relevant PPI resources to help you address can be accessed using the links above.
Costing for PPI – relevant PPI resources
- Every country is different in terms of the rules for compensating public research partners, for example, compensation may not be allowed in certain countries and benefits and tax systems. Please explore the rules within the context of your own country. Three examples of payment guidance are from the NIHR in the UK, from SPOR Evidence Alliance in Canada and from PCORI in the USA, which has two documents: Budgeting for Engagement and the financial compensation resource.
Finding PRPs – relevant PPI resources
- Countries may have developed specific ways of promoting opportunities for involvement / engagement, for example the People in Research Network in the UK.
- Cochrane Engage is an international volunteer hub that aims to bring together people across the globe who need assistance with health research. The hub can be used by researchers who need help finding public research partners for their studies. You can post a public involvement / engagement opportunity as a task in the hub. You will be asked to complete a proforma explaining more about the opportunity and detailing the kind of experience your public research partner needs to have. Cochrane Engage is open to all health research organisations with a free Cochrane account.
Explaining your COS work to potential PRPs – relevant PPI resources
- Animation explaining what core outcome sets are (with subtitles in various languages)
- Plain language summary explaining what core outcome sets are
- Glossaries relating to health research may be useful for explaining general aspects of research, for example SPOR, the NIHR and the Global Health Research Network.
Explaining what PPI is – relevant PPI resources
- This animation explains PPI in COS studies
Recruiting PRPs – relevant PPI resources
- A potential template of a PPI in COS advert that you can adapt – Coming soon
- PPI in COS Study – a remit document – Coming soon
- A potential template PPI in COS expression of interest form – Coming soon
Supporting PRPs throughout your COS study – relevant PPI resources
- The Centre for Research in Public Health and Community Care provides a resource to help researchers provide feedback to public research partners.
Evaluating PRPs’ experience of PPI and the impact of PPI – relevant PPI resources
- The University of West England have produced guidance on evaluating PPI
- McMaster University, Canada have produced the following Patient and Public Evaluation Toolkit
- The Public Impact Assessment Framework (PiiAF) has been produced to help researchers assess the impacts of involving / engaging members of the public in their research
- The International PPI Network hosted a webinar on impact in PPI/E.
Involving special groups – relevant PPI resources
- A Top Tips document about involving / engaging children and young people has been produced by the NIHR in the UK.
- This Wellcome Trust study explores the role, benefits and potential of young people’s involvement / engagement in health research, providing recommendations on how to meaningfully involve young people in research, and what it means to do that well.
Involving under-served groups – relevant PPI resources
- The NIHR have produced guidance on Being inclusive in public involvement in health and care research
Reporting PPI in your COS study – relevant PPI resources
- Reporting Guidelines – GRIPP2 provides researchers with reporting guidelines for describing PPI/E in their research.
- Two examples of COS studies that have used the GRIPP2 checklist are: Crudgington et al, 2018 and Smith et al, 2018.
This section of the toolkit was informed by the following collection of resources on the NIHR website “PPI (Patient and Public Involvement) resources for applicants to NIHR research programmes”.
Planning the overall study design
What’s involved in planning a core outcome set study?
A number of stages are involved in planning the core outcome set (COS) study, from the initial idea to develop a COS, through to applying for funding (or for a PhD studentship) and if successful, adding the finer detail of the study plans to a study protocol (manual) and seeking ethical approval. Public research partners (PRP) can help throughout these stages, including:
When agreeing the scope and setting of the COS study, PRPs can collaborate with the research team to consider:
- Which patients / members of the public should this COS be for – for example, only patients with a particular type of epilepsy, or all patients with any type of epilepsy.
- What type of treatment / intervention the COS will be for – for example, only surgical treatment or only physiotherapy treatment for a slipped disc.
- Whether patients / the public from particular countries are important to involve and why – for example in some countries particular diseases are more common.
The research proposal forms the basis of the research protocol, so the research team should think through key aspects of the COS study at this point. When developing the research proposal, PRPs can collaborate with the research team to consider:
What methods should we use to ensure the long list of outcomes includes patient important outcomes?
- What methods should we use to gain consensus about the core outcomes with patients as key participants?
- Will the methods need adapting for the patient population(s) to be targeted? For example, if children and young people are to participate or if the condition that the COS is being developed for means additional support is needed to facilitate people’s participation?
- Which approaches to recruiting patients should we use and why?
- How can we ensure diversity in our study participants?
- If this study is being conducted internationally what should we consider (note, some countries don’t have patient organisations)
- What retention challenges might we face in this study (e.g. asking patients to complete a similar survey on more than one occasion, or inviting survey participants to a final consensus meeting)?
- How might these challenges be addressed?
- How best can patients be involved in the design, delivery and dissemination of this COS study if funded?
- If plans have already been made for PPI are they achievable, inclusive and appropriate?
- Has appropriate support and compensation been factored in to the proposal and do the plans take into account any differences in different countries – for example, any specific consideration of making payments for PPI?
- Are patients appropriately represented on the oversight committee?
- How might the oversight committee membership be more inclusive?
When developing the COS study research protocol, PRPs can collaborate with the research team to consider:
- How can the methods be planned to best suit the needs of patient / public participants in your COS study? (e.g. will any physical needs related to the health condition have an impact on taking part, such as cognitive issues / pain when sitting for long periods, etc)
- What ethical issues might there be and how might these be addressed?
- How appropriate and understandable are your participant information sheets and consent forms?
- How is the information presented in these resources – is it accessible for patients / the public?
- What other information might someone find helpful in deciding whether to take part?
- What should our key recruitment message be for potential patient participants?
- How can we best recruit patients to the COS study?
- If the study is international how can we recruit in countries without patient organisations?
- How should we address the need for translation of the resources into different languages?
- (if using emailed / mailed invitations)
- What should our research invitations say and what email header should be used?
- How should the invitations be designed to be accessible and engaging to potential participants?
- (If using social media) How can we best access patients through social media (which channels and what times) and what key messages should we give?
- (If in-person approaches are to be used) who should invite patients and what should they say about the study?
- (If developing a website) How should our COS website be designed so that it is accessible and easy to navigate?
- What information should be given to participants between rounds of the Delphi to ensure that they know what to expect with further survey rounds?
- What information should be given in further survey rounds to ensure the participants understand the difference between previous rounds and the current rounds and what they are required to do?
- How best can survey participants be encouraged to participate in any later consensus meetings?
- What should we plan to do in terms of disseminating the results to patient participants?
- What should we plan to do in terms of disseminating the results to the wider patient / public community?
Access a PDF of the questions on this page here.
Relevant PPI resources
Resources for public research partners
- NIHR Learning for Involvement – Module 3 – How to review research documents from a patient and public point of view – this resource has been developed for patients and the public sitting on research funding committees and largely focus on clinical trials but it may be useful to share elements of this with your own Public Research Partners to help them consider general research design issues.
- How does a research team know if a COS study is needed – A plain language summary?
Resources for COS researchers
- Designing and planning – Resource about designing and planning research with public research partners from the Imperial College London Public Involvement Research Hub
- COMET PPI reflection log
Consensus processes in core outcome set studies
COS developers use different methods to gain consensus on what outcomes are most important:
When preparing for the survey, public research partners (PRPs) can collaborate with the research team to consider the following:
Who should take part i.e. the eligibility criteria for the survey?
- What information patients might like to see on your website / newsletters?
- How easy is it to find information on the website (if your COS has one) and is this participant information useful and easy to understand?
- How readable and understandable the newsletters are (if these are to be used in your COS study)?
- How readable and understandable the patient information and consent forms are for your COS study?
- How understandable are the outcome definitions?
- How will patients react to the outcomes?
- What ethical issues there may be with patient participation in your COS survey?
- How accessible is the COS survey (for example, is the font size adequate, are the colours suitable for someone with colour blindness, is the scoring system suitable for the target population, for example if children are participating etc)?
- How understandable the COS instructions are for each round of the survey?
- Will patients know which perspective they are to use when completing each round of the Delphi survey (for example, are they to vote from their personal perspective or considering the perspectives of the wider patient group)?
- How understandable is the feedback that we are providing between rounds in a Delphi survey, including any graphs etc.
- Is there likely to be a large patient research burden in the intended patient community at the time we plan to run the COS study (e.g. large-scale, clinical trial / research prioritisation study etc. running at the same time as your COS study)
- How will we reach a diverse range of patients to participate in the COS study?
- Whether there are specific issues to consider when recruiting particular patient groups in the COS study e.g. patients with cognitive / additional learning needs / visual or auditory impairment.
- How best to raise awareness about the COS study to the appropriate patient group(s)?
- How best to maintain patient interest in the COS study over time?
- How to address potential recruitment challenges from a patient perspective?
When preparing for a consensus meeting, public research partners (PRPs) can collaborate with the research team to consider:
- How readable and understandable the participant information and consent forms are for the COS consensus meeting, e.g. what to expect in the meeting?
- What ethical issues there may be with patient participation in the COS meeting? E.g. in some conditions, especially in rare disease, the patient or caregiver’s clinician may be in the room, have you considered with patients how this may make participants feel?
- How understandable the information is about how to get to the venue / join the meeting remotely?
- How accessible the COS meeting will be? (e.g. lifts available, prayer room, breastfeeding facilities, suitable seating, appropriate number and length of breaks, clinical room available, accessible toilets)
- How well is the information on accessibility of the COS meeting communicated to patients in the participant information sheet?
- How understandable the intended patient pre-meeting instructions are? (including for those with additional needs, as relevant)
- How best to cater for any dietary needs throughout the meeting, especially any issues relevant to the condition?
- Is the order of the outcomes appropriate?
- How can the order of the outcomes best be explained to participants?
- How can all stakeholders be prepared for such situations?
- What can be done during the meeting if disagreements arise?
- What information patient participants will need after the meeting.
Access a PDF of the questions on this page here.
Relevant PPI resources
Resources for PRPs
- The Delphi survey – a plain-language summary – developed by the COMET Initiative
- Consensus meeting animation developed by Bristol Centre for Surgical Research
- Nominal Group Technique – a plain-language summary – developed by the COMET Initiative
- Populations and sampling – This resource has been developed by Understanding Health Research to explain populations and sampling in plain-language.
Resources for COS developers involving patients / the public in their work
- The Delphi survey – a plain-language summary – developed by the COMET Initiative
- Consensus meeting animation developed by Bristol Centre for Surgical Research
- Nominal Group Technique – a plain-language summary – developed by the COMET Initiative
- Populations and sampling – This resource has been developed by Understanding Health Research to explain populations and sampling in plain-language.
- Patient and public involvement in research groups – Guidance for Chairs.
- NIHR – A brief guide to patient and public involvement and qualitative methods within health and social care research.
- COMET PoPPIE guidance on Ethics and COS studies https://www.comet-initiative.org/Patients/Ethics
Core outcome set study reporting
After undertaking the core outcome set (COS) study, research teams need to report their approach and findings. These are usually published in study reports, PhD theses and academic research articles but patients and the public are a key audience and increasingly there is an expectation that such publications include a plain language summary. COS research can also be reported in academic conferences where there may be representation from patients, the public or patient communities.
COS developers also have a responsibility to inform their study participants about the results of their work and to engage with the wider patient community. Additionally, COS developers are beginning to report their Patient and Public Involvement (PPI) activities in separate publications. Public research partners (PRPs) have an important role in COS study reporting.
Reporting COS study – findings and PPI
- Is the information understandable and comprehensive?
- Does it adequately and clearly explain the methods and findings?
- Are there any limitations that you feel have not been highlighted about the work?
- Do you think the conclusions are justified?
- Have we adequately highlighted the importance of this work from a patient community perspective?
- What do you think should be the key messages about the COS work?
- Have any differences in opinion between stakeholder groups been transparently reported?
- How have patient participants been involved in producing the report?
- Is the information understandable and comprehensive?
- Does it adequately and clearly explain how PPI was undertaken in the study?
- Are there any limitations to the PPI that you feel haven’t been highlighted about the work?
- Do you think the conclusions are justified?
- What do you think should be the key messages about PPI in the COS work?
- Who are the potential patient / public target audiences?
- What is the best way to share the study results with the target audience?
- What methods add value to communicating the study findings (e.g. podcasts / social media / video etc)
Access a PDF of the questions on this page here.
Relevant PPI resources
The following resources may be useful when involving patients / the public in COS study reporting?
Resources for PRPs
- How to read a scientific paper? Resource produced by Understanding Health Research
- How do I publish a paper? Animation for publication teams including public partners, by Equator Canada.
- Reporting about PPI in a research study – what is the GRIPP2 reporting guidance? Staniszewska et al, 2017 (Lay summary)
Resources for COS developers involving patients / the public in their work
- COS-STAR reporting guidance – General reporting standards for COS studies
- Public involvement in disseminating study results – A resource in the Imperial College London Public Involvement Resource Hub
- Guidance on authorship with and acknowledgement of patient partners in patient-oriented research – Richards et al, 2020
- Examples of two research summaries for COS study participants (designed with patient involvement) are available here)
- Reporting about PPI in a research study – GRIPP2 reporting guidance
- An innovative toolkit: increasing the role and value of patient and public involvement in the dissemination of research findings – McNichol E & Grimshaw P (2014)
- PPI reflection log
This part of the COMET PPI toolkit has been informed by resources in other toolkits / resource hubs: Imperial College London Public Involvement Resource Hub
Outcome List Generation
Identifying outcomes
One of the first stages of COS development is to identify an initial long list of outcomes for use in further consensus work (we will refer to this stage as the outcomes scoping stage). Generating this list might involve one or more methods. For example:
- Systematic reviews of clinical trials and reviews of qualitative literature
- Qualitative interviews and focus groups
Systematic reviews are used to identify which outcomes researchers have used in previous studies of the health condition/topic. Qualitative reviews discover what outcomes have been identified as important in previous qualitative studies in this health condition/topic. Qualitative interviews and focus groups are used to explore what relevant stakeholders feel are important outcomes in the health condition/topic.
When undertaking the systematic review of outcomes or a qualitative literature review (also known as qualitative evidence syntheses), Public Research Partners (PRPs) can collaborate with the research team to consider:
- The eligibility criteria and scope for the review
- Should the review exclude or include specific types of studies?
- Is the scope of the review too broad or narrow?
When planning and analysing qualitative interviews/focus groups with patients about outcomes of importance to them, PRPs can collaborate with the research team to consider:
- What questions do you think should be used to find out what outcomes are important to patients / the public and why they are important?
- Are the intended questions in the topic guide understandable – could anything be misinterpreted?
- What information needs to go into the participant information sheets to help patients understand the purpose of the study, what will happen in the study and any ethical considerations?
- Do the participant information sheets cover everything that you think should be covered, without being overly detailed?
- How readable and understandable are the participant information sheets and consent forms for your COS interview / focus group study? Does anything confuse you? Could anything be misinterpreted?
- How can we ensure the sampling strategy is appropriate and inclusive for the COS interview / focus group study?
- What should we pay particular attention to in terms of the sampling participants for the study?
- What are the best ways of accessing relevant patients / potential public participants for participation in the COS interview / focus group study?
- What should the recruitment strategy be?
- How should the study should be promoted to patients / the public?
- What ethical issues might there may be with patient participation in the COS interview / focus group study (for example, any risks from participation)?
- Is there anything about the interview / focus group design that you feel could cause distress to potential patient / public participants? If so, what could be done to address this?
- Is there anything about the interview /focus group study design that you feel could impact the confidentiality or data protection of the study participants? If so, what could be done to address this?
- Consider whether the interviews/focus groups would best be held in-person, online, by telephone or hybrid?
- If in-person, how can we ensure the venue is accessible for patients / the public?
- What adaptations might be needed to make sure patients / the public can participate in the in-person interview / focus groups (e.g. room layout, breaks, venue accessibility, baby changing / breast feeding facilities, prayer room, timing and day of the interviews / focus group, interpreters etc)?
- What adaptations might be needed to make sure patients / the public can participate in the online interview / focus group to ensure (e.g. instructions for joining, breaks, timing and day of the interviews / focus group, interpreters etc)?
- Do you feel the intended coding and analysis of the data is appropriate from a patient perspective or are important outcomes potentially going to be missed?
Reporting and merging outcomes from different sources
When reporting and merging the outcomes from different sources (e.g. from a systematic review and an interview study), PRPs can collaborate with the research team to consider:
- Which outcomes could be merged from the various sources?
- Why those outcomes should / shouldn’t be merged
- Is the list of outcomes still too long – is further merging needed?
- Are the merged outcomes appropriate from a patient perspective or have important distinct outcomes been lost in the merging?
- Are patient important outcomes included in the final list? (For example, less frequently reported outcomes could be really important outcomes from a patient perspective)
Reporting and merging outcomes from different sources
When reporting and merging the outcomes from different sources (e.g. from a systematic review and an interview study), PRPs can collaborate with the research team to consider:
- Does the reporting of outcomes adequately highlight any differences found between the different sources?
- Which outcomes could be merged from the various sources?
- Why those outcomes should / shouldn’t be merged
- Is the list of outcomes still too long – is further merging needed?
- Are the merged outcomes appropriate from a patient perspective or have important distinct outcomes been lost in the merging?
- Are patient important outcomes included in the final list? (For example, less frequently reported outcomes could be really important outcomes from a patient perspective)
Defining the outcomes
When defining the outcomes, PRPs can collaborate with the research team to consider:
- Is the list missing any outcomes?
- Why do you think these outcomes should be included?
- What plain language descriptions of each outcome should be used?
- Are the plain language descriptions understandable and do they adequately cover what the outcome is about?
- Are the grouped headings appropriate / do you feel they will make sense to patients/the public?
- Would any of the groupings confuse you?
- Is the order in which the outcomes presented important? (Typically, the groups of outcomes are randomised but there might be reasons not to do this).
Access a PDF of the questions on this page.
Relevant PPI resources
The following resources may be useful when involving patients / the public in outcome list generation
Resources for PRPs
Populations and sampling– This resource has been developed by Understanding Health research to explain populations and sampling in plain language.
Resources for COS developers involving patients / the public in their work
- The following article provides an example of PPI in outcome list generation: https://researchinvolvement.biomedcentral.com/track/pdf/10.1186/s40900-018-0091-5.pdf)
- NIHR – A brief guide to patient and public involvement and qualitative methods within health and social care research.
- Piloting a qualitative topic guide with PPI partners (Ryan Combs)
- PPI reflection log
- COMET PoPPIE guidance on Ethics and COS studies https://www.comet-initiative.org/Patients/Ethics
COS uptake
Once a high-quality core outcome set has been developed it is vital that it is used to improve the quality of research and avoid future research waste. Public research partners (PRPs) have the potential to bring a powerful voice to the promotion of COS uptake.
Promoting COS uptake
When promoting COS uptake, PRPs can collaborate with the research team to consider:
- How can patients’ voices help promote the use of the COS?
- What key messages could be used by public research partners to encourage COS uptake?
- Patient communities can help promote COS uptake – how could this happen? Are there any existing links with patient organisations that could be used?
- If barriers to COS uptake are anticipated or develop during the study, is there anything that the patient voice can do to address these challenges?
Access a PDF of the questions on this page here.
Relevant PPI resources
The following resources may be useful when involving patients / the public in promoting COS uptake.