There is increasing concern that most current published research findings are false.…Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.

-Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. (HT: Public Library of Science)

Modern American Science may no longer be reliable or believable. There have and always will be charlatans and nasty academic cliques. Some claim this negativity started as early as when Sir Isaac Newton was a practicing Physicist. But the heavy involvement of politics, government and cutthroat economic competition present in scientific circles today has led to a greater problem with study reliability.

The professional competition to be a published researcher has enhanced the likelihood that scientists will become exaggerators. A number of academic researchers, who seek to honorably police their own, describe this as a version of what economic researchers refer to as The Winner’s Curse. In order to make it into print, the researcher exaggerates an effect or a phenomena in order to grab attention. By so doing, that researcher invites debunking because a check has been written that the Physics or Chemistry in question simple cannot cash.

So in order to get into ink, researchers claim to have found the Holy Grail. Results that actually comport to reality, but just aren’t as “sexy”, get pushed out of the information space. Good science still occurs everywhere, but it gets drowned out by people who play to what the boss pays for or the politics of a given scientific journal. Dr John Ioannidis describes the factors that should be considered before you actually believe a published scientific work.

the probability that a research finding is indeed true depends on the prior probability of it being true (before doing the study), the statistical power of the study, and the level of statistical significance

(Ioannidis, Ob. Cit.)

Ioannidis than applies the three variables to assign a series of Bayesian probabilities that the study in question is actually factually accurate. Well-done and meticulous scientific studies (large sample sizes, reasonable a priori hypothesis, and a believably low level of effect) have an historical trend to be 85% accurate. Those less well-constructed, using smaller samples or making egregiously ridiculous claims are well less than 1% likely to be accurate. His criteria and his list of probabilities can be accessed here.

A different, but related problem involves the prevalence of publishing positive rather than negative results. Hugh Pickens describes this process in The Economist.

There also seems to be a bias towards publishing positive results. For instance, a study earlier this year found that among the studies submitted to America’s Food and Drug Administration about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results were.

(HT: The Economist)

Pickens argues that this does not prove scientific cheating occurs. He posits a possible reality. Theoretically, the scientists could write up positive and negative results with equal élan, watch only the positive results get selected by Discover and Nature and then go back to writing up positive and negative results with equal élan. This would involve either remarkable tone-deafness or impeccable honesty on the part of the scientific research community.

Human nature contraindicates such a positive interpretation. The entire UN IPCC got seduced by having their name in lights and their achievements lauded by powerful politicians throughout the world. Once the “Hockey Stick Graph” went viral, there was no way Michael Mann or James Hansen COULD have showed up to work one day and announced Anthropogenic Climate Change was amusing, fun, but not quite credible. They were on that ride for the duration, regardless of what evidence came out to challenge their beliefs.

It was American President, Dwight David Eisenhower, who perhaps predicted where all of this would eventually go. He said the following towards the end of his famous Farewell Address.

The prospect of domination of the nation’s scholars by Federal employment, project allocations, and the power of money is ever present and is gravely to be regarded. Yet, in holding scientific research and discovery in respect, as we should, we must also be alert to the equal and opposite danger that public policy could itself become the captive of a scientific-technological elite.

-(HT: Our Documents.gov)

Dwight Eisenhower said it so eloquently, but he still coined a fundamental colloquialism. “He who pays the piper calls the tune.” In Modern America, that may well come to read “He who pays the researcher calls the result.” It doesn’t existentially threaten us the way cities with 47% illiteracy rates do, but this tendency in Modern American Science to succumb to The Winner’s Curse could make our nation more a loser in the end.