Interpretation of data derived from a patient reported outcome (PRO) measure can be challenging, particularly with regard to understanding the meaning of what a change or difference in a score means clinically.
Interpretation of PRO data is linked with both the aims of the study and the constructs measured by the PRO and as a result should be a key factor in the development of any measurement strategy.
Remembering that PROs are assessing outcomes from the perspective of the patient, any change in the score may or may not be related to a clinical outcome. As an example, patients with diabetes who have changed to a different insulin treatment considered more convenient – but, in all other respects is the same – are unlikely to report any improvement in clinical outcomes but, may well report that their QoL has been enhanced as a result of the greater flexibility in their lives.
A range of techniques exist for interpreting PRO data which are:
- Minimal Important Difference (MID) – the smallest difference in a score that is considered to be worthwhile or important.
- Known groups – the mean scores underlying particular clinical groups or clinical indicators which give rise to them and which can be used as a clinically based benchmark to compare other groups.
- Normative and reference groups – mean scores from defined large populations to provide normative data – typical scores – called norms. Mean scores from a particular study can be compared with the population norms.
- Statistical significance – The statistical significance of the probability of treatment (A) is better than treatment (B).
- Effect size – A way of quantifying the difference between two groups of patients that has many advantages over the use of statistical significance alone and emphasises the size of the difference rather than confounding this with sample size.
- Cumulative distribution functions (CDF) – The CDF shows a continuous plot of the proportion of patients at each point along the continuum of the scale score continuum experiencing change at that level or lower levels.
Other approaches include, reference to the PRO content when interpreting its score as well as comparing them with known clinical parameters such as days in hospital and illness severity or the proportion of patients whose PRO score improve or get worse after intervention.
McLeod et al, significantly point out that the final FDA PRO guidance admits that judgement is still required when evaluating whether individual’s responses are meaningful. McLeod et al go on to briefly outline patient-based methods such as the concept of ‘symptom-free days’ as well as consensus-based methods.
The challenges however, when bringing together both quantitative and qualitative data includes:
- Developing an actionable framework to enable the combining of qualitative and quantitative evidence
- Developing frameworks to enable the incorporation of expert intuition into the data analysis process.
- Bringing the analyst’s knowledge and experience of the research craft – what works – what does not – that can enrich the interpretation of the data.
This issue was briefly discussed in a recent blog ‘New thinking in interpreting the patient’s experience’ where we put forward the notion that the understanding of patient reported outcomes can be enhanced by bringing a methodological approach, a ‘holistic’ analysis’ that provides a rounded view of what all the qualitative and quantitative evidence is saying whilst retaining the integrity of the data. The proposed model for this is shown below.
STARTING AT THE END It is essential that every analysis starts with making sure the right problem is being analysed by using a rigorous problem-identification process. This involves getting a full understanding of the desired outcomes and whether our primary and secondary endpoints and the conceptual model are relevant to the questions asked of the data?
COMPENSATING FOR IMPERFECT DATA While there is a wealth of potential contextualising information available to enable a deeper interpretation of pro results such as surveys, interviews with clinicians and patients etc. it is likely that some of this data may fail the criteria of the hypothetico – deductive model of enquiry and classical statistics. However, for the holistic analysis is none the less important in the interpretation of the total picture. Therefore, The first requirement is deciding on the level of compensation required in dealing with less than perfect data and how this might impact on the quality of the results.
This will include asking question of the data such as:
- Did the context in which the research was conducted effect the responses?
- Was the sample representative?
- Were the questions asked neutral and unbiased?
DEVELOPING AN ANALYTICAL STRATEGY Here our focus is on developing a detailed analysis strategy that specifically demonstrates how the various stakeholders decisions to be made will be addressed.Our approach is first to get a picture of the total sample, followed by looking at subgroup variations and differences.
ESTABLISHING THE INTERPRETATION BOUNDARY OF THE DATA This is the process of establishing first, the more statistically derived constraints e.g.
- Study design
- Statistical significance of the quantitative data
- Sampling error,
in which we must work with and then to apply the ‘enablers’ to determine just how far the statistically driven assessments can be legitimately stretched in the interpretation of the data through the application of a set of user-friendly general principles drawn from Bayesian thinking, where significance testing becomes one of a simple probability that any differences are likely to be significant at a given probability.
INTEGRATING ALL THE EVIDENCE We are now at the stage of combining the qualitative and quantitative data sets and in doing we need to examine the evidence within a framework of weight, power and direction.
- Balance of opinion – The proportion of the data set numerically in favour for example of an improvement in symptoms score, number of symptom free days etc.
- Depth of feeling – This is the depth or strength of feeling about the topic and can be based on either quantitative or qualitative assessments.
Power of evidence
- Prior knowledge – How does the data fit into the wider context of what we know e.g. contraindications, ad-hoc evidence.
- Integrity of the data – What we know about the data e.g. that we know can it be a good predictor of outcome or is it need of careful interpretation?
Direction of evidence
- Internal consistency – The level of consistency within each data set.
- External consistency – The level of consistency across the data sets. SEE THE DIAGRAM BELOW
While checks and balances will need to be applied to ensure the integrity of the data however, presentation of research findings through a compelling narrative can:
- Help communicate the complex whole – linking together key relationships and experiences.
- Aid comprehension – Presenting data in a coherent narrative form is more likely to be understood, absorbed and recalled.
- Enhance action – Reduces barriers to change, brings potential actions to life.
- Facilitate buy-in – If the narrative is seen as relevant and timely it’s more likely to be communicated, absorbed and recalled.
If you found this blog helpful why not share it with someone who might find it interesting.You might also find the following blogs of interest
Categories: Patient reported outcomes