It is often said that randomized controlled studies (RCTs) are the gold standard in healthcare.
As David Shaywitz explains in a piece in Forbes Will Real World Performance Replace RCTs As Healthcare’s Most Important Standard?:
The value of RCTs lies in the random, generally blinded, allocation of patients to treatment or control group, an approach that when properly executed minimizes confounders (based on the presumption that any significant confounder would be randomly allocated as well), and enables researchers to discern the efficacy of the intervention (does it work better – or worse – than controls) and begin to evaluate the safety and side-effects.
But sometimes there is a mismatch between the results of clinical trials and actual real world experience. The intervention performs much better in scientific studies than in the general public. Why?
The subjects who enroll in clinical trials … may not be representative of either the larger population or of the patients who are likely to receive the intervention currently under study; groups underrepresented in clinical trials include the elderly, minorities, and those with poor performance status (the most debilitated).
This begins to get at what may be the most significant limitations of clinical trials: the ability to generalize results. The issue is that clinical trials, by design, are experiments, often high-stakes experiments from perspective of the subjects … as well as the sponsors, who often invest considerable time and capital in the trial. Clinical trial subjects tend to be showered with attention and followed with exceptional care, and study investigators generally do everything in their power to make sure subjects receive their therapy (whether experimental or control) and show up for their follow-up evaluations. Study personnel strive to be extremely responsive to questions and concerns raised by subjects.
But in real practice, YMMV, as they say on the interwebs — your mileage may vary; adherence is less certain, evaluation can be less systematic, and follow-up more sporadic…
That’s why real world evidence (RWE) is becoming increasingly important in assessing healthcare interventions.
Consider our experience with birth control methods. Early on it became apparent that birth control methods performed much better in scientific studies than in real world use. That’s because, depending on the difficulty of using the method, people don’t always use it properly or regularly or both.
This slide demonstrates the difference between the two:
How each method works in practice is much more important to the individual than how it works in theory. As a result, when providers discuss the risks and benefits of various forms of contraception, we use real world experience as the basis of our discussions.
The same principle applies even when randomization is not possible because it is unethical. Scientific studies are very valuable, but real world experience is possibly even more important.
Consider the case of homebirth. One of the most widely quoted papers on the topic is the Birthplace Study that found in a carefully selected population, subjected to much more stringent eligibility requirements than those in the real world, and followed much more carefully than real world patients, homebirth increased the risk of poor outcomes compared to hospital birth for first time mothers and was essentially equal for mothers having a second, third or higher order child. That’s good to know, but just as in the case of contraception, real world experience is more important than theoretical results.
So how do the outcome of homebirth and hospital birth compare in the real world? The British National Health Service almost certainly knows since they collect the data, but they won’t tell the public. I’m willing to bet that if the real world data showed homebirth in the UK to be as safe as hospital birth it would have been released years ago. The fact that it is not available suggests that in the real world homebirth, which the government has been aggressively supporting as a way of saving money, is significantly more dangerous than hospital birth.
Just as in the case of contraception, real world data is far more relevant and far more accurate than data from the best studies. We would never counsel patients on the risks and benefits various forms of birth control using theoretical effectiveness. Therefore, no one should be counseling women on the risks of homebirth using the Birthplace Study.
When it comes to breastfeeding, the dichotomy between scientific study results and real world experience is massive. The theoretical benefits of breastfeeding — based on extrapolation of small studies that assume causality — are large. The real world benefits of breastfeeding are almost non-existent. There is no real world evidence of which I am aware that shows that increasing breastfeeding rates saves any term babies or any money in actual practice. For example, there seems to have been no impact on infant mortality or healthcare expenditure despite the fact that the US breastfeeding rate has tripled in the past 45 years. Countries with the highest breastfeeding rates have the highest infant mortality rates while countries with the lowest infant mortality rates have the lowest breastfeeding rates.
What are we to think of situations in which the interested parties suppress real world evidence while aggressively promoting scientific studies? The drug Vioxx offers an instructive example. In scientific studies, Vioxx had tremendous benefits and few risks. But then reports began to come in to the manufacturer that in real world experience Vioxx increased the risk of heart attack and stroke. The manufacturer tried to suppress that evidence and many people were harmed as a result.
Homebirth is treated by its advocates the same way as Vioxx was treated by its manufacturer. Real world evidence of harm isn’t released to the public. Breastfeeding is treated by its advocates the same way Vioxx was treated by its manufacturer. Although the information on neonatal hypernatremic dehydration, kernicterus, infants smothering in or falling from their mothers’ hospital beds in the wake of closure of well baby nurseries is available from a variety of databases and published in scientific papers, lactation professionals simply ignore it. Over and over again they cite theoretical mathematical models created by Drs. Melissa Bartick and Alison Stuebe, and completely ignore both real world harms and lack of real world benefits.
So which is more important in health care: scientific studies or real world evidence?
Shaywitz offers his view:
…[A]t its best, real world evidence provides an opportunity to evaluate medical interventions on what arguably matters most – real world performance …
Real world evidence is not always available, but when it is — as in the case of homebirth and breastfeeding — it must take a central place, equal or superior to evidence from scientific studies. Partisans are going to resist, but that’s all the more reason to make sure that individual women have access to it. As with birth control methods, women can not make informed medical decisions in the absence of real world evidence.
