Developing valid patient reported outcome (PRO) content – Is co-creation a viable methodology?

Establishing content validity of a patient reported outcome measure (PRO) to ensure  it measures the concept(s) of interest and that the items and domains are both relevant,  traditionally focuses on a combination of a review of the relevant literature and input from patients and clinicians through in–depth interviews and focus groups.

In the FDA Final Guidance to Industry in 2009 the importance of establishing the content validity of a PRO was stressed. More recently members of the Study Endpoints & Label Development (SEALD) division of the FDA presenting at the 45th annual meeting of the Drug Information Association emphasised that the  content validity is an important qualitative measure.

While there is a range of qualitative methodologies to derive PRO content including, Grounded theory, case study and ethnography, we believe there is scope to look at the more innovative approaches in identifying relevant PRO content. Indeed the FDA guidance document pointed out that it’s critical for instrument developers and reviewers to use and understand state-of-the-art qualitative research methods.

In their paper Eilene et al (2010) put forward a rigorous framework for collecting and analysing  qualitative data for PRO content which we believe is a significant move forward by making explicit the data collection and analysis methodology. However, the authors remain within the traditional approach of data collection using either focus group methods or in-depth individual interviews – ‘…the development of a truly open-ended discussion guide is essential to ensure that qualitative interviews yield data that reflects the subject’s experience…’

Here, the emphasis remains very much based around patients responding to the issues previously identified by the researcher as relevant and which are included in the interview discussion guide as an  aide memoire. Pretty much the same can be said for using focus groups for generating valid PRO content.

Although both in-depth and focus groups can be considered as  interactive as well as it being argued that we are identifying key issues, themes and concepts relevant from the perspective of the patient as a result, the nature of the interaction is one-directional and based on the researcher’s agenda.

Once the data is obtained it’s the researcher that undertakes the thematic analysis, identifies the key concepts and generates the draft items. The role of the patient is then to judge the relevance and comprehensibility of the derived content.

At DHP Research we are currently looking at a number of innovative techniques for PRO content generation that truly reflects patient input beyond that of the in-depth interview and focus group of which co-creation workshops is one.

PRO content generation through co-creation workshops brings patients and key stakeholders (e.g. researchers, clinicians, pharmaceutical advisors)  together and who are tasked to collaboratively generate ideas, concepts and content for the PRO through a range of stimulating activities.

Although these techniques or methodologies include a synthesis of existing information as a precursor to the qualitative phase of content identification and generation, one important aspect is the methodology differs from the traditional use of in-depth interviews and focus groups which is  that concept identification and content generation is a collaborative process.

In contrast to focus groups and in-depth interviews, co-creation workshops have a multi-faceted structure to them where a large group of creative, tuned-in patients became smaller groups who are joined by experts tasked with collaboratively to identify the relevant concepts and content.

These smaller groups act as condensed, more salient focus groups that share and build on each others’ ideas, producing much more direct and actionable results than a standard focus group could ever hope to provide in the same time.

The emphasis is very much on interaction between patients and stakeholders and the sharing and building on ideas. Importantly however, it is for the patient to own the brief and to show us what they think, feel and experience, from which triggers, trends and actionable insight can be identified.

The triggers and trends provide the stimulus for patients and stakeholders to collaboratively generate ideas, concepts and content through small groups and team play which are then further refined and evaluated by patients before more formalised evaluation and psychometric testing.

What is co-creation?

The term co-creation is used fairly indiscriminately and with a lack of conceptual clarity. It should however, not be confused with concepts such as ‘crowdsourcing’ or ‘user generated content’. 

Co-creation is creative as a form of collaborative creativity.’ It’s a rich mix drawing on a combination of knowledge, and group decision-making. It’s the process of involving the client/customer/patient in the creation of meaning and value.

Co-creation is an active, creative and social process based around collaboration rather than just involvement. The patient is no longer passive, but a co-producer or co-creator.

As an iterative process and as a consequence, co-creation involves both construction and de-construction of experiences and as a result will go through cycles of content and value development.

The benefits for the patients is greater personalisation, where for the PRO developer it is about turning real-time knowledge and experience into a meaningful and content valid measure.

Within the context of deriving PRO content there are a variety of emerging qualitative methodologies  that go beyond our existing reliance of in-depth interviews and focus groups. with co-creation being just one approach.

Of course we still need to face the challenges of  establishing reliability and validity that have plagued the value of qualitative research in the past. However, through a paradigm shift from the understanding of the meaning of reliability and validity within a quantitative research context to the application of concepts such as credibility, transferability, dependability and confirmability in understanding the usefulness of qualitative data, this should not be insurmountable if combined with such approaches discussed by Eilene et al.

For more information on the role of co-creation in the development of PRO content contact us at

We also like to hear from you regarding your views on this post.

Why not share this post with someone who might find it important.

Other refs


Categories: Patient reported outcomes

Tags: , , , , ,

2 replies

  1. Keith: I especially like the idea of PRO methodology. Research is always moving from point A to. B rather than reflective or retrospective analysis. Drugs are a perfect example. Once approved and on the market that is the end, period. No further analysis required and therefore no ” patient reported outcome.” The problem, under consideration might be, for example, to lower cholesterol. If a statin drug is given and the outcome is that the cholesterol is lowered it is “end of story.” The problem begins with no PRO of the side effects because the drug system is not set up for post market ongoing analysis. So to me, there is an absolute need to have in place the type of methodology to which you allude.
    Is this the sort of comment you were considering?
    Elizabeth Rankin BScN

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: