Years ago, when I was working for a large health maintenance organization, I received a curious letter. It had come to their attention that my forceps rate (0%) was well below the average for obstetricians in our institution. I pointed out that my C-section rate was only 16% and I hadn’t left a single baby inside a mother. I asked if this were a problem? No one seemed to know.
At another point I got a notification that I had ordered more ultrasounds for my OB and GYN patients than average for that month. This time I asked whether any of the ultrasounds had been unnecessary? No one seemed to know … or care. It was just something they were required to measure.
[pullquote align=”right” cite=”” link=”” color=”” class=”” size=””]The composite index DOESN’T tell us whether an individual hospital is a safe place to give birth.[/pullquote]
I mention these anecdotes because they illustrate the dangers of simply compiling statistics and comparing them. My employers wanted to know if I was delivering patients appropriately, but my forceps rate couldn’t tell them that. They wanted to know if I was ordering unnecessary ultrasounds but comparing my ordering rate to my colleagues in a given month couldn’t tell them that, either.
It is important to measure, but you have to be sure you are measuring the right thing in the right way.
That was my concern when I read the breathless report in USA Today, The secret number maternity hospitals don’t want you to know, and why we’re revealing it.
It is extremely important to measure maternal complication rates. But are the people at USA Today measuring the right thing in the right way?
It’s not clear to me that they are.
According to the article:
[I]n the United States – the most dangerous country in the developed world to give birth – maternity hospitals’ childbirth complication rates are a well-guarded secret.
Many hospitals know them. So do many state health agencies, insurance companies and researchers. But they fear the complication rates are too complex for regular folks to understand.
In truth, no one knows what they mean.
There are certainly questions about the data. Some hospitals are more likely to treat patients with health problems, so childbirth complication rates are difficult to compare. Some of the best-equipped hospitals in the country may have higher complication rates because mothers who are very sick get referred there.
But there’s more to it than that: How useful is the method they used, a “composite index”? How valuable is it to compare one hospital to another or any hospital to a mean value? What is it we really want to know and do the measurements chosen by USA Today tell us what we want to know?
We want to know several things that are extremely difficult to measure:
Is the complication rate for a given mix of patients appropriate?
Has every complication that could have been avoided been avoided?
Has the hospital staff caused complications?
And were the complications that occurred appropriately treated?
Where did the USA Today “secret number” come from?
More than a decade ago, the U.S. Centers for Disease Control and Prevention created a method for calculating how often women giving birth endure severe complications using diagnosis and procedure codes that hospitals record in patient billing records.
The resulting “severe maternal morbidity rate” is like a composite score of things that can go wrong at the hospital before, during or after delivery – heart attacks, strokes, blood transfusions, hysterectomies and other emergencies that can permanently harm or even kill a new mother.
The first problem is the inclusion of blood transfusions (often a minor complication) with far more serious complications.
[T]he CDC method uses blood transfusions as an indicator that a woman may have hemorrhaged. But it’s impossible from billing data to know whether the woman received one unit or many units of blood.
Some experts say that could inflate the rate. Others note that transfusions are counted for all hospitals and a blood transfusion is not part of a routine childbirth.
No it’s not routine, but it’s also not an indicator of severe complications. It would be far better to look at the rate for women who got 3 units of blood or more. A transfusion that large is a good indicator of a severe complication. So right off the bat, the composite index is almost certainly inaccurate in modeling severe complications.
The next problem: in order for any measurement to have meaning, we must adjust case mix so we are comparing like to like:
[S]ome hospitals have higher rates because they are specialty facilities where doctors send the riskiest cases. Some serve more poor mothers, who often get less consistent prenatal care. Others serve larger numbers of black mothers, who tend to have higher rates of certain health problems, such as hypertension, that can lead to serious complications.
What about just looking at the outliers? The article is accompanied by this graph:
Complication rates are distributed in a bell curve (a “normal distribution”) with a very long tail. (You can’t really see that because of the editorial decision to lump all hospitals over 5% together creating a spike at the end.) A normal distribution with a long tail has particular implications in public health.
As the authors of The “Long Tail” and Public Health: New Thinking for Addressing Health Disparities explain:
The prevailing approaches to improving population health emphasize “shifting the mean” through prevention efforts that target large groups at high risk or through mass environmental control interventions that encourage small but universal changes in individual behavior. This approach has led to the search for “blockbuster” public health interventions that can have the largest effects on determinants of population health and individual behavior…
In a compelling critique, Frohlich and Potvin argue that the prevailing population approach may have the unintended consequence of exacerbating health disparities. Disease risk, they point out, varies not just by behavioral risk factors but also by socially defined groups that vary in their exposure to fundamental risks, for example, low education and low socioeconomic status. Broadly targeted population interventions that focus primarily on behavioral determinants may not be as effective under these conditions or with these groups…
And that’s why simply comparing the composite index of one hospital to the composite index of another hospitals is not particularly helpful. Any hospital that is within the bell curve has an acceptable complication rate. Most hospitals in the long tail probably serve vulnerable populations and their complication rates may be more closely tied to socio-economic factors than to hospital competence.
But that’s not particularly eye catching when compared to a “secret number that hospitals don’t want you to know.”
So what does the USA Today composite index tell readers?
Not much that is useful for them.
No one should decide where to give birth based on the composite index since direct comparisons aren’t really possible and all hospitals in the normal distribution are probably as safe as the others in the normal distribution.
The hospitals in the long tail very likely serve high risk populations and the problem is not necessarily that they are providing poor care; they may be providing exactly the same care as the hospitals in the normal distribution but that isn’t enough for vulnerable populations.
The USA Today composite index can highlight facilities that deserve additional scrutiny but unfortunately it the CAN’T tell us whether an individual hospital is a safe place to give birth.
