Insights

Tangible return-on-engagement tools poised for pharma validation

Recent years have seen great strides in the availability of tools for measuring returns on patient engagement – promising to benefit all with an interest in patient-centric medicines development. Leaders in pharma should now pick up the baton and demonstrate the utility of these tools in real-world industry settings. 

Problems with measuring value from patient engagement  

“Despite [the] perceived importance of assessing the return on patient engagement, little is known about ‘what’ to evaluate, and even less about ‘how.’” In just one sentence, a peer-reviewed research paper summed up a problem that has perplexed the pharmaceutical industry ever since patient-centricity became a trend. 

The many reasons this thorny issue persists to this day are widely reported, but we’ll mention a few. There is a natural delay between any engagement program and the impact. What constitutes value or success is highly subjective. There has historically been a lack of reliable measurement tools and well-defined endpoints. Qualitative measures are much easier to identify than quantitative, tangible ones. And pharma companies don’t apply existing metrics consistently. 

Over the years, many frameworks have sprung up from different sources, guiding industry on how to establish patient involvement in pharmaceutical R&D. This has been hugely beneficial. And yet a criticism often leveled against these instruments is that they are conceptual rather than useful from an operational perspective. For example, they fail to detail how to actually implement patient-centric work in a pharma company, or how to measure impact. 

As a result, while all pharma companies would benefit from reliable, industry-wide data on the prevalence and impact of patient engagement, nobody has stepped up to gather this information in a systematic way

Informal data sources can nevertheless give an indication of the way the wind is blowing. The Drug Information Association (DIA) 2023 Global Annual Meeting included a session titled “Measuring Impact of Patient Engagement Across Research and Development“, where a poll of the audience showed that “68% of attendees’ companies are currently engaging patients systematically across their development programs, yet nearly two-thirds of that percentage aren’t measuring the impact”. 

It’s unsurprising, then, that much of the current evidence showing the value of patient-centricity is made up of “case studies, anecdotes, and cautionary tales”. 

Box 1: Definitions of terminology 

Terms like “return on engagement” mean different things to different people, and they may even be controversial for some in pharma. That’s perhaps to be expected for any hot-topic issue. While A Life in a Day takes a neutral position in this respect, we choose to use widely acknowledged terminology in order to be part of this important conversation. Here we outline some of the language used in this report and our understanding of them, with source links where available.  
 
Patient-centricity: Putting the patient first in an open and sustained engagement of the patient to respectfully and compassionately achieve the best experience and outcome for that person and their family 
 
Patient engagement: The effective and active collaboration of patients, patient advocates, patient representatives and/or carers in the processes and decisions within the medicines lifecycle, along with all other relevant stakeholders when appropriate 
 
Return on engagement: Return on investment when applied specifically in the area of patient engagement. Also referred to as ‘return on patient engagement’; some stakeholders choose to simply say ‘impact’ 
 
Value: The benefits of patient engagement (in relation to the direct and indirect costs) for individuals and organizations 
 
Value-impact: Positive, tangible, and useful outcomes, achieving a deep sense of accomplishment and ownership for those patients and representatives involved 

Deep dive into the literature

It might be tempting to take a dim view of the current landscape. But dig deeper and it quickly becomes clear that much promising work has taken place in just the last few years. The sentence opening this report came from a 2019 research article published by a team working under the umbrella of PARADIGM, a public‐private partnership that wants to see meaningful patient involvement in medicines development. 

At the time of their 2019 article, the researchers had carried out a comprehensive literature review into the monitoring and evaluation of patient engagement. Specifically they were trying to identify indicators of impact, and methods for measuring this. 

Wading through a stack of papers published between 2000 and 2018, PARADIGM found some potential indicators for evaluating the benefits and costs of patient engagement. But they concluded that the indicators weren’t up to scratch for illuminating pathways to impact, or possible changes in the external environment. 

Novel monitoring and evaluation framework  

The work just described wasn’t merely an academic exercise but rather an initial scoping of the environment as a precursor to a much more pivotal PARADIGM project. Having concluded that there was no framework capable of showing the return on patient engagement in a way that would satisfy all stakeholders, the group set out to fill that gap.  

The necessary brain power came from a collective of 15 pharma companies, five patient associations, and two academic institutions. They were organized into a multi-stakeholder ‘working group’, a common practice for pooling expertise in a non-competitive, collaborative setting. As well as insights, working group participants contributed case studies of patient engagement to inform the new framework. 

The PARADIGM team published their “Patient Engagement Metrics Framework” in 2022. With it, they wanted to “facilitate understanding of how patient engagement translates into impact and why initiatives fail or succeed”. PARADIGM had also gone to great lengths to address the weaknesses of previous (concept-led) frameworks. In all, their offering comprised 87 metrics “to operationalize measurement of the different components”. 

Those metrics came in four flavors (see Table 1). ‘Input’ metrics show whether the conditions for meaningful and sustainable patient engagement exist. ‘Activities/process’ metrics focus on progress in the implementation of patient engagement. ‘Learning and change’ metrics measure the short-term, direct results of patient engagement. And ‘impact’ metrics shed light on long-term outcomes. 

Mindful that not all 87 metrics will be relevant in every patient engagement scenario, the PARADIGM framework includes a “Metrics Selector” tool allowing users to choose the ones most applicable to them. 

Table 1: Snapshot of PARADIGM Patient Engagement Metrics Framework* 

Framework component Description of metrics 
Input Metrics that assess the accessibility and preparedness for patient engagement  Metrics that assess the material, human and financial resources used to carry out a patient engagement initiative  Metrics that assess the expertise and diversity of individuals involved 
Activities/process Metrics that assess how engagement activities are facilitated  Metrics that assess the quality of interactions during the process  Metrics that assess the overall experience of those involved with the engagement process 
Learnings and Changes Metrics that capture the insights and recommendations people take away from the patient engagement activities  Metrics that capture changes to research (such as altered research questions, design, processes, outcome measures), or changes to the individual (such as changes in attitudes) 
Impacts Metrics that assess the speed of studies and influencing factors such as study participants’ experiences in trials  Metrics that assess the quality and availability [of] information used for decision-making by regulatory bodies and patient communities, as well as the accessibility and usability of products  Metrics that capture wider consideration of patient engagement in R&D and organizational decision-making 

*Selected extracts of framework shown for illustrative purposes 

Toolkit with “standardized metrics” 

Interestingly, around the same time the PARADIGM team were going about their work, a parallel and highly complementary effort was taking place elsewhere. This one, co-led by the DIA and the Tufts Center for the Study of Drug Development, involved 17 industry stakeholders made up of pharma companies, contract research organizations, and other vendors providing services in patient engagement or patient-centric medicines development. 

Organized in a similar working group arrangement, this collective’s analysis surfaced hundreds of metrics covering costs, quantitative and qualitative impact measures, as well as lessons learned and challenges.  

Again keeping in mind what would be useful operationally, the team used their discoveries to co-design “a toolkit that industry and patient advocacy groups can use to assess the impact of patient engagement and [patient-centric initiatives] PCIs” (see Table 2). Their hope is that management decision-making will become more informed and that industry will adopt the “standardized metrics” in the toolkit to assess return on engagement. 

To give a flavor of what the DIA/Tufts project created, one group of metrics, organized into a category headed “Long-term drug development portfolio and enterprise”, looks at the economics of an initiative, its impact on a drug program and product portfolio, and whether it has influenced business decisions and strategy at corporate level. 

Table 2: Snapshot of DIA/Tufts metrics toolkit* 

Dimension Metrics 
Long-term drug development portfolio and enterprise (program and corporate level) Cost of the patient-centric initiative (PCI)  Changes to development timelines, ie: time to go/no-go decision (compared to other programs)  Program success rates relative to portfolio benchmark 
Prevalence (ie: internal and external reach) Number of patients and/or patient advocacy groups involved in PCI  Total number of PCIs implemented  Number of trials using PCI 
Patient feedback Quality-of-life measures  Total number of changes proposed by patients to: drug program; clinical trial protocol; communication  Patient feedback to regulatory agencies 
Trial performance metrics Number of endpoints relevant to patients and patient advocacy groups  Changes in protocol complexity and patient burden  Recruitment rates; retention rates; adherence rates (to study drug, to patient tool) 

*Selected extracts shown for illustrative purposes 

Now for pharma’s part 

All credit to PARADIGM and DIA/Tufts for their efforts. They’ve taken the initiative to create novel tools promising to benefit all with an interest in broader adoption of patient engagement in pharma. But both teams have made it clear it’s not yet a case of job done. 

PARADIGM researchers are keenly aware that their metrics have “hardly been tested in practice”. So beyond publishing them, they haven’t been able to say which methods work best in a given scenario. The DIA/Tufts group called their offering “an initial set of standardized return-on-engagement metrics”: a clear indication they expect to see further iterations and refinements over time. Crucially, these new tools have a distinct emphasis on measuring return on engagement in R&D. In other words, they leave an expanse of white space for complementary future work in the setting of commercialized therapies. 

While public-private collaborations were first off the mark, pharma companies should now be the ones to pick up the baton. Encouragingly, our separate report “Evaluation of return on patient engagement: How pharma leadership can create a rising tide to lift all boats”, based on unique insights from recent client conversations, points to a growing appetite within pharma to take up the challenge. The report also outlines A Life in a Day’s thoughts on what needs to happen next. Among our recommendations, we would like to see pharma validation of the new measurement tools described above – ideally longitudinally in real-world settings. Pharma should also take the lead in measuring the creation of value from patient engagement after R&D, when products are on the market and reaching patients. 

As in a race, getting a good start can be the hardest part. But leveraging the current momentum in measuring returns on patient engagement, bold leadership in pharma can surely get these business-critical efforts over the finish line. 

FURTHER READING 

Gorbenko, O. et al. (2022) ‘Co-creating with patients an impact framework across the medicine’s life cycle: A qualitative study exploring patients’ experiences of involvement in and perceptions of impact measures’, Research Involvement and Engagement, 8(1). doi:10.1186/s40900-022-00334-0.  

Kim, J.Y. et al. (2023) ‘Benchmarking patient engagement capabilities and preparedness of drug development sponsors’, Therapeutic Innovation & Regulatory Science, 57(5), pp. 1040–1049. doi:10.1007/s43441-023-00545-x

Stergiopoulos, S. et al. (2020) ‘Measuring the impact of patient engagement and patient Centricity in clinical research and development’, Therapeutic Innovation & Regulatory Science, 54(1), pp. 103–116. doi:10.1007/s43441-019-00034-0.  

Vat, L.E. et al. (2019) ‘Evaluating the “Return on patient engagement initiatives” in Medicines Research and Development: A Literature Review’, Health Expectations, 23(1), pp. 5–18. doi:10.1111/hex.12951

Vat, L.E. et al. (2021) ‘Evaluation of patient engagement in Medicine Development: A multi‐stakeholder framework with metrics’, Health Expectations, 24(2), pp. 491–506. doi:10.1111/hex.13191.  

Get in Touch

If you are interested or have questions about A Life in A Day, please reach out and contact us.


    You can select more than one by holding down the Ctrl key on selection