When I taught a science for non-science majors at Cornell University, my favorite lecture was on science in the popular media.  My learning objective was straight forward: I wanted students to pick up a newspaper, flip to the science section, and detect bullshit.

This is such an important life skill because at this point in time, blogs, newspapers, and magazines print unscientific bullshit.  They do it because it sells. ** **I have never seen an article so neatly package all of the problems that lead to this state of affairs than John Bohannon’s recent “Chocolate Diet” charade.  If you haven’t read it yet, please do.  I’ll wait.

Amazing, right?

The study

Here’s the breakdown: Diana Löbl and Peter Onneken are working on a “diet science” documentary, and want to show how diet news can get reported.  So they enlist Johannes to conduct a shoddy study.  Participants will either make no dietary changes, go on a diet, or go on a diet with a bar of chocolate.  By only including 15 people in their study, and measuring a wealth of factors, they were guaranteed to find that the chocolate dieters fared better in some way than the other groups.  The finding would be meaningless, but it would be “statistically significant” – within the confines of their prohibitively small study.  They lucked out, and the 5 chocolate eaters lost weight.

How to Lie With Statistics, by Daryl Huff.

Next, they needed to publish their study somewhere.  Fortunately for Johannes, it’s getting harder and harder to distinguish “fake” publishers from real ones.  Real journals will conduct real peer review, where third-party scientists must evaluate the article before it can be published.  Due to the intentionally poor design of this study, it (hopefully) wouldn’t pass peer review in any credible journal.

A fake publisher, on the other hand, will not conduct review.  Instead, it will collect a publication fee, then immediately accept and publish the article without review.  Bohannon’s paper was easily accepted into the International Archives of Medicine without any peer review.

Once the paper was published, it took off in the science and diet sections of many, many magazines, blogs, and newspapers.  You can easily understand why- “Chocolate helps you lose weight” is the sort of headline The Daily Mail and its ilk dream about.

The Assignment

Back to my students.  To consider a news piece credible, students first needed to ask three questions.

1.  Is the finding reported by a scientist?  Who?  Does the scientist work for a biased source?

2.  Was the finding published in a peer-reviewed journal?

3.  Can you find the article?  Do the conclusions the authors draw in the abstract reflect what is reported in the newspaper?

How would my students fare if they were given this article?  It turns out, not so well.  Here are my answers, working from Bohannon’s press release.

**Question **1:  Is the finding reported by a scientist?  Does the scientist work for a biased source?

Johannes Bohannon is presented as the research director of the nonprofit Institute of Diet and Health, a fictional institute.  They have a website that is intentionally minimalist.  It doesn’t look incredibly legitimate, but it also doesn’t scream crazy.  I imagine most students would give it a pass, despite the possibility that, say, this fictional institute is funded by Nestle.

Question 2:**  Was the finding published in a peer-reviewed journal?**

Yes of course it was!  Here’s the real problem predatory journals present to the common reader going forward.  The International Archives of Medicine (IAM) is a legitimate enough sounding journal.  The Journal’s website has an ISSN, citation metrics, and assurances that it is included in the JournalGuide whitelist of reputable titles.  As Bohannon reports, this journal used to be reputable, as it was published under BioMed Central, a trustworthy open-access publisher.

In short, one would need to do some serious digging to realize the journal is bogus.

Question 3.  Can you find the article?  Do the conclusions the authors draw in the abstract reflect what is reported in the newspaper?

Yes and yes.  This is an easy one, because Bohannon’s wrote both the press release and the article.

The solution

It’s clear that we have a problem here.  What prevents someone from doing poor work (or fabricating work) and “publishing” the work in predatory journals?  Nothing.  And unless some sort of watchdog agency is set up to accredit peer-reviewed journals, this will not change any time soon.  If real scientists can’t distinguish a quality journal from a predatory one, what chance do non-scientists have?  As it is, I find Beall's list of questionable publishers to be the most useful resource for investigating journals. edit: Beal's list has been taken down.  But this is a list built by a science librarian who saw a problem, and can’t be expected to be definitive or exhaustive.  The JournalGuide, cited on IAM’s website, appears to be a reasonable resource in its own right (it’s listed in this Nature article on resources for evaluating journals), but obviously it’s not perfect if IAM is listed.

If policing publishers isn’t the answer, then what can be done?  Bohannon’s piece makes one thing abundantly clear: the popular media did not do their homework on this piece.  Many magazines ran the story without contacting Bohannon at all.  If he was contacted, he was never contacted by someone who knew enough about science to ask the right questions.  In short, science journalism lack skepticism.  It seems that most magazines will run an article without asking the three questions I listed above.  And even if they do, it’s clear that a more careful tact must be taken in the future to prevent spreading misinformation.

Until that day arrives, I urge you, the reader, to bring a healthy dose of skepticism to every new miracle finding you read, especially related to human health, sexuality, and diet.  Do your homework, because no one else is doing it for you.