Sunday, May 31, 2015

Chocolate study sting: Where are these millions of fools, anyway?

On May 27, John Bohannon published a self-congratulatory piece at io9 about how he'd "fooled millions" into thinking that chocolate helps with weight loss. How? He conducted a human subjects study of 15 people, ventured on a statistical fishing expedition with the results, got a vanity journal to publish the paper, and then slapped up a news release that was carefully crafted to sell the hell out of his dubious findings.

I'm sure he was hoping for more in his duplicitous effort to fool millions, which, by the way, is not a typical goal of journalism. Bohannon must have been enormously disappointed to find that in reality, the number of outlets that fell for his silly stunt may have been smaller than his tiny study population. He seems to have wanted to expose science and health journalists, diet studies in particular (although why those and not, say, heart studies, I'm not sure), and the problem of vanity publications for what they are, as though people had not been discussing these problems ad-goddamned-nauseam for ... ever.

His hypothesis appears to have been that if you dangle something sexy and scientific-seeming (e.g., chocolate) before a pack of hungry journalists, they'll bite every time, the suckers, no matter how crappy said sexy science really is. Sadly for Bohannon, his results failed to support his hypothesis. But that hasn't stopped him from trying to manipulate his outcomes with an odd mix of braggadocio and patronizing finger-wagging about p value hacking to justify the compromise of research and journalistic ethics his little caper involved.

What Bohannon's results really show is ... drumroll please ... that some websites recycle news releases. It's exactly the kind of lame finding a desperate researcher would be forced to publish in a vanity journal because it has all the novelty of hot weather during the Texas summer. What he "discovered" isn't a problem with science journalism, diet studies, or vanity journals. It's a problem with churnalism sites trying to make shit in general look like journalism (ETA: one that pre-dates the Internet). Yet even his data for that are weak--people don't seem to buy so easily as perhaps they once did into regurgalism as journalism. As sexy as Bohannon tried to make his chocolate study seem, the churnalism didn't churn up much online interest in it.

First, let's take Bohannon's impossible-to-prove (and indefensible) claim that he "fooled millions" into thinking that chocolate causes weight loss. Below is a breakdown of the handful of outlets that even mentioned this study and also gave numbers of social media shares. Note that of these nine seven, eight just regurgitated the news release. In other words, no journalist had a hand in what was posted, and no reporting took place. Like many churnalism sites, these sites--Times of India, anyone?--simply harvested and packaged a news release.

Social Media Shares
Daily Star: 249 shares; news release regurgitation
Irish Examiner: 10,820; news release regurgitation
Cosmopolitan, German: 48; news release regurgitation+redaction and explanation
Times of India: 15; notorious news release regurgitator
German HuffPo: unknown; news release regurgitation
HuffPo India: 4; news release regurgitation+retraction
Brigitte (German): 64 shares, also a regurgitation
Focus (German): no shares given, but the video has 23 ratings; another news release regurgitation
Tyler, Tex, news station: Shares unclear. Population of Tyler, Tex.: 100,223. This one was reported, with outside comment from a dietician.

Some silly math to illustrate the silliness of this claim: Let's say, generously, that each social media share reached 100 people and that they were all uncritically gullible consumers. Based on the above, that translates into 11,170-ish shares x 100 for a total reach of 1,117,000-ish people plus the entire city of Tyler, Tex. (requiring that all of Tyler had watched that one segment on the local news station) for a grand total of 1,217,000-ish. That's 1.2 million people, perhaps, who might have possibly seen a reference to chocolate and weight loss on somebody's Facebook wall or Twitter feed and then, presumbly having crawled out from under a rock where they had resided since the pre-Internet era, immediately consumed a pound of chocolate right after trying this one weird trick they'd read about for losing belly fat. Those fools.

And that's going on the assumption that every person who read about the chocolate thing was fooled by it instead of doing what any sensible consumer of any such information over the last half century or so would do, which is to think, "That is total horseshit. I've been eating chocolate every effing day for 20 years with nary an ounce lost." (Note: As pointed out to me on Twitter, this list does not include the print outlet Bild and the millions who read it; Bild has retracted what they printed). Bohannon has not shown that millions read and believed that chocolate would help them lose weight.

The only ones who could demonstrably have been fooled are journalists who earnestly reported the story, and they add up to single digits. Churnalist sites were probably just happy to have something to churn that had the word "chocolate" in it.

How important were Bohannon and his chocolate study to social media users during the time period it washed over the global news media and duped all of those millions? Let's ask Topsy. Topsy--who has been talking on Twitter about Bohannon and chocolate lately?
How about Google trends on searches of the 'first author'?
  • John Bohannon, searched by name, last 90 days: not enough volume to show
  • Johannes Bohannon, searched by name, last 90 days: not enough volume to show
No one on social media, much less journalists, seems to have cared much about that chocolate study or about John/Johannes Bohannon until Bohannon wrote his io9 piece.

What about the story when it "broke" a couple of months ago? Looks like Bohannon himself did this search when he was scraping around the web for examples of journalists he'd duped. That list he cited in his io9 piece? That's the entire list. Here's the (English-language) search of Google news on terms "Bohannon" and "chocolate," covering dates from March 27 through April 4. There are literally seven hits, one of which is the site where Bohannon posted the news release. Another report says that at most, 17 outlets seem to have picked up the story (probably including this handful of German outlets, all of which regurgitate the news release).

We can all interpret squishy data in different ways, but whether it's 7 or 17, somewhere in between, or slightly more, we're talking about a global population here of thousands of journalists who somehow managed not to write about Bohannon's super-sexy duplicitous chocolate study. Bohannon may intepret that finding of 7 to 17 as proof that journalists are suckers who need to attend better to their jobs and double check the legitimacy of a scientific journal. I say that it shows that actual journalists aren't quite the suckers Bohannon hypothesized them to be and that churnalist sites gonna churn. In reality, he fed the latter exactly what they need to do what they do.

A couple of outlets published briefs a little later, including Prevention and Shape. Prevention has an editor's note at the top of original article, which again was little more than a regurgitation of the news release. They say:
Editor's Note: This article reports on a study later revealed to have been conducted by an unqualified researcher who manipulated the statistics in order to produce a significant result. It was published in a journal that doesn't peer-review its research. We regret the error. There is legitimate research on the benefits of chocolate, but this study doesn't qualify.
Bohannon did a lot of things with his "sting." He dunked the ethics of two professions into dirty waters. He failed to support his hypothesis and failed to recognize that failure. He showed us things we already know and talk about all the time. And he self promoted (which is now getting us into the real problem with journalism, as Hilda Bastian writes). The one thing he doesn't seem to have demonstrated? That he fooled much of anyone, much less science and health journalists, into thinking that eating chocolate will help you lose weight.

******Last update: Monday, June 1 10:11 a.m. ET******
Origins of churnalism, FYI, dating back almost a quarter century.

No comments:

Post a Comment