A new study claiming to show that female doctors provide better care than male doctors is a paradigmatic example of the way that scientific research has been perverted by contemporary journalism. In an effort to provide click-bait headlines, scientists are grossly exaggerating the clinical significance of their findings and thereby misleading the public.
[pullquote align=”right” cite=”” link=”” color=”” class=”” size=””]Is a difference in mortality rate of 0.7% clinically meaningful?[/pullquote]
The paper is Comparison of Hospital Mortality and Readmission Rates for Medicare Patients Treated by Male vs Female Physicians published in this week’s edition of the Journal of the American Medical Association (JAMA).
NPR is typical of news outlets hyping the findings:
In a study that is sure to rile male doctors, Harvard researchers have found that female doctors who care for elderly hospitalized patients get better results. Patients cared for by women were less likely to die or return to the hospital after discharge.
Previous research has shown that female doctors are more likely to follow recommendations about prevention counseling and to order preventive tests like Pap smears and mammograms.
But the latest work, published Monday in JAMA Internal Medicine, is the first to show a big difference in the result that matters most to patients: life or death.
According to the paper:
Elderly hospitalized patients treated by female internists have lower mortality and readmissions compared with those cared for by male internists. These findings suggest that the differences in practice patterns between male and female physicians, as suggested in previous studies, may have important clinical implications for patient outcomes.
What did the researchers do?
We analyzed a 20% random sample of Medicare fee-for-service beneficiaries 65 years or older hospitalized with a medical condition and treated by general internists from January 1, 2011, to December 31, 2014. We examined the association between physician sex and 30-day mortality and readmission rates, adjusted for patient and physician characteristics and hospital fixed effects …
So far, so good.
What did they find?
… Patients treated by female physicians had lower 30-day mortality (adjusted mortality, 11.07% vs 11.49%; adjusted risk difference, –0.43%; 95% CI, –0.57% to –0.28%; P < .001; number needed to treat to prevent 1 death, 233) and lower 30-day readmissions (adjusted readmissions, 15.02% vs 15.57%; adjusted risk difference, –0.55%; 95% CI, –0.71% to –0.39%; P < .001; number needed to treat to prevent 1 readmission, 182) than patients cared for by male physicians, after accounting for potential confounders…
The study is methodologically sound. The authors carefully chose the subjects and carefully corrected for a large number of confounding variables. The differences are statistically significant but the authors fail to pay careful attention to the most important question: Are the results clinically significant?
Put another way, is a difference in adjusted mortality rates of 0.47% clinically meaningful?
I doubt it.
What’s the difference between statistical significance and clinical significance?
A biostatistician explains:
In clinical research is not only important to assess the significance of the differences between the evaluated groups but also it is recommended, if possible, to measure how meaningful the outcome is (for instance, to evaluate the effectiveness and efficacy of an intervention). Statistical significance does not provide information about the effect size or the clinical relevance. Because of that, researchers often misinterpret statistically significance as clinical one…
That is what has happened here. The authors have claimed that there is a statistically significant difference in death rates and readmission rates between female and male physicians; that’s true. But they’ve gone on to imply that this finding is the same as a clinically significant difference and that’s false.
What is the clinical significance of these findings?
The authors haven’t done all the appropriate calculations to determine clinical significance and have not provided the information we need to calculate them.
They do provide the number needed to treat (i.e. 233 to prevent one death), but they don’t provide the Cohen’s effect size, nor the standard deviations that we could use to calculate the Cohen’s effect size for ourselves.
There’s another aspect of clinical significance that hasn’t been considered: are 30 day mortality and readmission rates a good measure of effect? Most people do not choose their doctor by calculating the odds of surviving 30 days after admission to the hospital, nor should they.
The authors hype their findings by extrapolating them. According to NPR:
The study’s authors estimate “that approximately 32,000 fewer patients would die if male physicians could achieve the same outcomes as female physicians every year.”
But the authors provide no evidence that their findings in Medicare patients admitted to the hospital could be extrapolated to other elderly, hospitalized patients, let alone younger patients, outpatients or doctors from specialities other than internal medicine.
The claim is an excellent example of the way that scientific research is perverted by the need to publish and to publicize.
The research paper yielded an interesting finding whose clinical significance is uncertain, has yet to be replicated, and should not be extrapolated. Then the authors proceeded to hype their findings in order to get attention for them. In the process, they have made claims that don’t bear scrutiny and essentially mislead the American public.
That behavior may be good for getting tenure, but it’s not good for patients.
