Finding clinical assertions in the world of real-world evidence
Real world evidence (RWE) is often a term that goes hand-in-hand with real-world data (RWD) and has taps into clinical trial design and protocol. However, within the past year, RWE has been used in the research industry more often and in a broader application.
The US Food and Drug Administration advanced a project to determine the scientific validity and regulatory application of RWE in its collaborative project with Brigham and Women’s Hospital – RCT Duplicate.
Harvard Medical School tapped the RWE Aetion Evidence Platform to support its reproducible evidence practices to enhance and achieve transparency (REPEAT) program to evaluate RWE in health care data. Increasing investments into artificial intelligence (AI) to analyze, sort, and optimize electronic health records (EHRs) have also been widely seen in the industry. All of which bolstered the role RWE is playing in the field.
Dan Riskin (DR), founder of Verantos, a digital contract research organization (CRO), tells Outsourcing-Pharma (OSP) that RWE has its own place in the clinical and regulatory world, but it’s use still has room for improvement.
To Riskin, there is more than one type of RWE and in its variances, there are additional aspects of applications. He said that the most commonly seen type of RWE is one that is used to find patterns. However, the use of RWE has greater bandwidth, according to Riskin, and can be used to take the place of control arms in clinical trials, bolster regulatory filings, and give a deeper look into the lives of patients.
OSP: What is your definition of RWE?
DR: A different definition than most. The definition I’ll use has two components: one is evidence which means you are asserting something, the second is real-world which means your assertion is about what’s happening in the real world with real patients.
OSP: What are some of the best ways to collect RWE?
DR: The first question is why, meaning the ways to collect it follow along with what you're trying to collect – evidence, real-world evidence, means showing something. Are you just trying to show patterns and pairs, or are you trying to show what works better than something else? It’s very dependent on what you’re trying to show and there are a lot of techniques being used to collect information.
We break RWE into what we call traditional and advanced. Traditional RWE you might use for trial recruitment, trial design, or marketing insights.
Advanced RWE is different. In advanced RWE what you’re trying to show is that one thing works better than another. Now, it might be that your med works better than no treatment at all, or that your drug works better than another drug. You’re trying to make a clinical assertion in advanced real-world evidence, and that’s a big difference.
OSP: What would a clinical assertion be?
DR: The distinction of RWE is that you’re trying to show a path in the real world. You’re not putting people on a research protocol, you’re not monitoring them, you’re not yelling at them to get their prescription refilled, you’re simply doing a certain type of care. It’s the best type of evidence to understand what will happen in the real world with other patients.
OSP: How can it frame clinical trial design?
DR: Traditional RWE might well support trial design, but advanced RWE might be regulatory support.
Trial design is a very traditional use of RWE -- there’s no clinical assertion being made, there’s just trying to understand how many patients need to enroll and how many sites are needed. This type of real-world evidence tries to see patterns, to understand the number of sites, number of patients and what might need to be checked.
Advanced RWE creates clinical assertions which you can use if you have a drug that you want a label expansion on; a drug that’s been in use for five years. That’s a situation where you may want to use real-world evidence to understand what is actually happening. Another example of a time to use advanced RWE is if we have a drug that’s being used in a rare disease and we want to do a randomized trial but quite frankly the disease is too rare – we can’t enroll the right number of patients and we want to use a synthetic control arm and non-enrolled patients as a comparison. Synthetic control arms are examples of advanced real-world evidence.
OSP: What are the challenges you face when transitioning from traditional RWE to advanced?
DR: This gets real hard real fast – the reason being, if you are not making a clinical assertion, if you’re doing traditional RWE, the data validity doesn’t matter.
If you’re doing traditional real-world evidence like trial recruitment, it doesn’t matter if you’re missing most of your patients. What matters is you’re getting some patients and putting them in a randomized trial. You’ll get good answers because the randomization takes care of the biases in collection.
Traditional real-world evidence doesn’t matter if you have bias selection, it doesn’t matter if you have inaccurate cohorts, it just matters that it's just better than what you did before. It's better to know the patterns.
During the advanced RWE accuracy matters – in fact, accuracy is everything.
The challenge that we face that now that we’re doing advanced real-world evidence we’re starting to dig into data. The frightening thing that we’re finding is that often half the information or more is missing. For example, if you thought a patient was treated for a certain drug and wanted to see under what indications, their claims will often not even show the diseases that are relevant.
When we look at the accuracy numbers for the electronic health records (EHR) the structured data of things that doctors list; the medications, and the procedures list, that information tends to be wrong also.
For example, if a patient has cancer, half the time it doesn’t even show up there. If a patient had a heart attack or has in the past, more than half the time it doesn’t show up there.
You get in a situation where you stop believing in the data, and at that point it's fine to use traditional RWE in which you don’t make assertions, but it starts to feel very unsafe to use it in advanced applications.
OSP: What are some innovations that could enable solutions to the challenges advanced RWE bring up?
DR: One is better data, two is better technology, and three is better expertise. It all comes down to that.
OSP: How can better data be collected?
DR: Better data is enriched data. You need to start with sources of information whether it’s the doctor or the patient and you need to enrich the document of information.
The dataset we need to work with is often called narrative or unstructured data and that’s where the clinical information resides. Studies have shown that about 80% of the information from the electronic health record is in the unstructured data.
OSP: Are there any AI solutions that can enable better use of the RWE in data?
DR: When we talk about better technology, data, and expertise – the better technology is going to be different forms of AI.
One form of that AI is natural language processing. It can understand what was said, for example, if a doctor says this patient is being evaluated to rule out heart attacks, NLP would be able to say this patient may have had a heart attack. But it’s not a be all and end all.
Doctor language is inherently conservative and often ambiguous ‘a patient may have this’, it often also uses a lot of shorthand – acronyms, abbreviations, etc. A doctor may say ‘CP’, well, is that chest pain or cerebral palsy?
Other forms of AI can detect patterns beyond NLP to understand what is actually happening with a patient. In an ideal world, the computer would be the diagnostician or the study itself.
AI should be able to see the patterns and endorse what’s actually happening and it will require better accuracy for data and structured data.
OSP: Are there any other steps that need to be taken for better RWE use?
DR: Right now, when real-world evidence is done we often do not do a protocol. There’s no other form of research where we don’t have a protocol, so that seems crazy. The protocol should include things that make it relevant enough to make a clinical assertion.
What do I need to do to make the clinical assertion believable? I would argue I’d need a cohort, inclusion criteria, and outcomes. The expertise to be able to design real-world evidence studies that are entirely accurate and believable – is limited.
In health care, if you don’t check it's almost always a problem, and we’re doing studies today without even checking.