The journal Nature published an outstanding piece on one of the most serious problems in scientific research today: a great deal of research is so flawed that it cannot be reproduced. It’s a problem that strikes at the heart of science, since the gold standard for establishing the truth of research results is that other scientists using the same methods will find the same results. Irreproducible research is research that is worthless; it proves nothing and often misleads.
How scientists fool themselves – and how they can stop by Regina Nuzzo offers a comprehensive explanation of why so much of today’s research is not reproducible: simply put, scientists have a great personal stake in the outcome of research, and this personal bias leads to shoddy science.
[pullquote align=”right” color=”” class=”” cite=”” link=””]The bias is simple, pervasive and distorts a great deal of breastfeeding research.[/pullquote]
This personal bias differs in important ways from classic financial conflicts of interest. No money changes hands; there is no quid pro quo, and there are no university or journal rules to protect against such personal bias. Indeed, the researchers themselves are often unaware of the bias because is subconcious.
Breastfeeding research, though not mentioned in the piece, is a classic example of the personal bias that renders much of the reasearch in the field misleading and deceptive. Breastfeeding reseachers believe deeply and fervently that breastfeeding, being natural, must be better than any substitutes. Therefore, they slice and dice the data until it supports their bias. They fall prey to the errors that Nuzzo describes in her piece.
1. Hypothesis myopia:
One trap that awaits during the early stages of research is what might be called hypothesis myopia: investigators fixate on collecting evidence to support just one hypothesis; neglect to look for evidence against it; and fail to consider other explanations…
This probably the most serious problem in breastfeeding research and distorts most of the existing research that claims to show important health benefits. The conclusions are predetermined and the data are arranged to support the conclusion. Critically, the researchers fail to consider alternative explanations for observed outcomes. In the case of breastfeeding research, it is typically manifested as a failure to correct for confounding variables.
We know that women who choose to breastfeed exclusively differ in important ways from women who do not. Any “benefits” of breastfeeding may reflect those differences, not breastfeeding itself. For example, women who choose to breastfeed exclusively are, on average, wealthier, better educated, and have better access to health insurance. Each of these three variables have been shown to lead to improved health outcomes for their children. The alternative explanation for most of the research that purports to show major health benefits of breastfeeding is that those benefits aren’t caused by breastfeeding, but are the inevitable result of the relatively privileged status of the mothers.
2. The Texas sharpshooter fallacy:
Seizing on random patterns in the data and mistaking them for interesting findings.
This is also known as “p hacking”:
“You just get some encouragement from the data and then think, well, this is the path to go down,” says Pashler. “You don’t realize you had 27 different options and you picked the one that gave you the most agreeable or interesting results, and now you’re engaged in something that’s not at all an unbiased representation of the data.”
In 2012, a study of more than 2,000 US psychologists suggested how common p-hacking is. Half had selectively reported only studies that ‘worked’, 58% had peeked at the results and then decided whether to collect more data, 43% had decided to throw out data only after checking its impact on the p-value and 35% had reported unexpected findings as having been predicted from the start, a practice that psychologist Norbert Kerr of Michigan State University in East Lansing has called HARKing, or hypothesizing after results are known.
In the case of breastfeeding studies, researchers often analyze large datasets looking at multiple outcomes. Then they pick the outcomes that have statistically significant differences and announce them as “findings” without acknowledging that any large dataset looking at multiple outcomes is bound to have random statistically significant differences that are coincidental and don’t represent real outcomes.
3. Asymmetric attention:
The data-checking phase holds another trap: asymmetric attention to detail. Sometimes known as disconfirmation bias, this happens when we give expected results a relatively free pass, but we rigorously check non-intuitive results…
This happens all the time in breastfeeding research and especially in its analysis. Professional breastfeeding advocates report findings on the benefits of breastfeeding without analyzing the data. In contrast, when a study is published that does not support a cherished tenet of lactivism, such as the belief that breastfeeding raises IQ, professional breastfeeding advocates immediately try to tear it apart.
How can we avoid falling prey to these cognitive biases?
The most important corrective to cognitive biases is recognizing that they exist. We must recognize that most scientists who do breastfeeding research believe that breastfeeding must be superior. They often fail to consider alternative explanations for their findings, but we don’t have to fall into the same trap. The first question to ask of any breastfeeding study is whether it accounted for confounding variables. If it didn’t, then the results are meaningless.
Second, we must analyze the data in the study ourselves to see if it justifies the conclusions. We need to ask whether the authors’ conclusions relate to the subject they intended to investigate or are just a random finding. For example, researchers may set out to determine if there is a difference in IQ between breastfed and non-breastfed babies, fail to find one and then write a paper about a random difference in fine motor coordination. That suggests p hacking, desperately searching for any difference, not the one that was supposed to be under study.
Finally, we must pay close attention to the results of studies that support our pre-existing biases. We must analyze them with the exact same rigor that we would bring to analyzing studies that don’t support what we believe.
Contemporary breastfeeding researchers often fool themselves into finding “benefits” of breastfeeding but that doesn’t mean that we have to let them fool us, too.
