Look at any paper or presentation reporting the development or use of a patient reported outcome (PRO) measure and without doubt there will be an array of statistical significance levels, standard deviations, standard errors and correlation coefficients in an attempt to help us understand what the data is telling us. But, is the application of classic statistical methods really telling us what we want to know? Interpretation of data derived from a patient reported outcome (PRO)and experience (PRE) measure can be challenging and even the final FDA PRO guidance admits that judgement is still required when evaluating whether individual’s responses are meaningful. So we ask the question “Do we need a rethink about interpreting patient reported outcomes?”
Breaking away from our methodological straightjacket
As researchers we more often than not believe that to be meaningful our research evidence must meet the criteria of classic statistical significance. Instead we need to start thinking about what the data actually means in the total. So rather than looking at the evidence through a narrow statistical lens, we need to build a case that’s both compelling and insight driven.
By viewing everything through that narrow statistical lens we will miss those vital connections that’s essential for identifying insight. In doing so we need to be able to blend both creativity and technical robustness and be prepared to integrate multiple but, often imperfect data sets.
Rather than look for one “silver bullet” in understanding information we are following De Bono’s idea of breaking down the task into individual activities.This approach has been expounded extensively (1) which includes:
- Developing an actionable framework to enable the combining the different elements of data evidence
- Developing frameworks to enable the incorporation of expert intuition into the data analysis process.
- Bringing the analyst’s knowledge and experience of the research craft – what works – what does not – that can enrich the interpretation of the data.
This involve first, looking at the data through an evaluation frame by applying classic statistical theory, then an experience frame which is an understanding of how research really works which is followed by embracing a holistic analysis incorporating analytical concepts such as the weight, power and direction of all the evidence which for example can be combined and modelled using fuzzy logic.
This approach was briefly discussed in my blog ‘New thinking in interpreting the patient’s experience’ where we put forward the notion that the understanding of patient reported outcomes and experience can be enhanced by bringing a ‘holistic’ approach’ that provides a rounded view of what all the evidence is saying whilst retaining the integrity of the data. The proposed model for this is shown below.
Creating an insight culture
It’s our belief that there’s the need for a paradigm shift from being purveyors of data to one in which we deliver real insight into patient reported outcomes and experience. To achieve this we need to move from a “box ticking” analysis culture as to whether or not a piece of data is statistically significant which stands in the way of that “Aha” Moment, to one that is a blend of creativity and analysis and draws on the integrating of multiple data sets and in doing so accepting that often these data sets – both quantitative and qualitative – will be imperfect.
1 – The Seven Frames Approach to Data Integration. Research world. September 2009
Categories: Patient reported outcomes