Recently, I was listening to NPR Morning Edition as Steve Inskeep and Shankar Vedantam were doing a segment titled “A lively Mind: Your Brain on Jane Austin.”[1] It was an interesting discussion of how a researcher, Natalie Phillips, decided to do a study that used functional magnetic resonance imaging to measure brain activity while subjects were reading in an environment with distractions or in an environment that allowed them to focus on their reading task. So far the lead in to the story was setting the stage for an interesting conclusion. It had my attention.
The next statement maintained my attention but it was no longer based on the theme of the story:
…Phillips said preliminary results showed otherwise: “What’s been taking us by surprise in our early data analysis is how much the whole brain — global activations across a number of different regions — seems to be transforming and shifting between the pleasure and the close reading.”
This admission shifted my attention from the discussion of the preliminary results to the fact that Inskeep, Vedantam and Phillips apparently saw nothing wrong with discussion partial results from a study before they study had been validated by the peer review process and accepted for publication in a credible journal.
This approach to science is wrong on many levels. Without a peer group approving the design of a study including its methods and procedures there can be no certainty that the study shows anything other than the variability that might exist between subjects based on other factors. Maybe there is a meaning difference between groups and maybe there isn’t. The reality is that no one will know until the study has been completed, gone through the peer review process and then published.
As the results were presented on the show they mean absolutely nothing, at least not scientifically. I was offended when I heard the results were preliminary and almost switched stations. I decided to listen to the rest of the story to try to discern why a credible news organization, NPR, was enthusiastically discussing a study that in the end may be utterly meaningless. Such a decision is not without consequences.
Years ago it would have been considered an ethical violation for a researcher to discuss any of their work before it had been published in a peer reviewed journal. But recently there has been a trend for those same journals to produce press releases touting upcoming research. We understand this is due to market pressure. Studies have shown that press releases to the lay press gin up excitement about the article that result in more citations of the article. Publications, and citations of those publications, are manna in the academic world.[2],[3]
The consequence of hyping research before it is published, or worse, before it is actually completed, is that once the public hears a slick story on NPR they tend not to follow-up to see if it actually turns out to be true. The original story, right or wrong, is what the listener will remember and that’s just wrong. In the story, Mr. Vedantam noted the results were preliminary but offered no further caveat that they may turn out to show nothing of importance. Of course, such a statement would be counterproductive if your goal is to make the story seem new, fresh, exciting and relevant. After all, the real goal is to capture listeners for as long as possible and hope they will listen again and ultimately become NPR donors.
Is it too cynical to be suspicious of the validity of preliminary study results discussed on NPR? I don’t think so. The idea of peer review is to weed out studies that are flawed so that readers can be assured that minimal standards have been met. Having had no such review, the preliminary results so enthusiastically discussed on NPR mean nothing to me other than I should look for the results of the completed study when and if they’re ever published in a reputable journal. And even after publication it is necessary to apply one’s own standards to ascertain validity of the study’s conclusions.
I understand that Steve Inskeep and Shankar Vedantam have a job to do and that job is to consistently come up with stories they think will be of interest to their readers. Maybe it’s because I hold NPR to a slightly higher standard than other media entities that I expect a more critical assessment of what is appropriate to be aired. NPR by itself will never be able to influence the entire industry to take a more responsible approach to disseminating scientific or medical information that is only half baked, but NPR can at least be more circumspect of what it considers to be a credible story.
Confronting Mediocrity 