Posted on 22 May 2017

Over the last few days, I have seen a lot of journalists sharing stories reporting on a study purporting to have found that journalist’s drink too much, are bad at managing emotions, and have lower-than-average brain function:

Mostly it’s self-deprecating, with people adding little comments like “explains a lot” or “I didn’t know they were studying me!” But just because it rings true and makes you laugh, doesn’t mean it should be reported on as science. And that’s exactly what’s happening when a bunch of journalists with blue checkmarks start sharing the study uncritically and adding words like “science” and “can’t argue with facts!”

The study is published on a personal website

So let’s take a look at where this study came from. Is it a peer-reviewed journal? A well-respected academic institution?
No, it’s a joint release from the London Press Club and “Tara Stewart: Neuroscience. Leadership.” The actual study is in on her personal website.

It is based on a tiny sample size, virtually no control group and old wive’s tales

When you get into the study, you find it was originally going to be on 90 members of the press club. But:

“Ultimately, failure to complete all the elements in the required time limit meant that a total of 21 participants completed every element, and a further 10 completed some elements of the study.”

I’m no neuroscientist, but 21 self-selected individuals does not seem like a great sample size to be drawing wide-ranging conclusion on journalists around the world who have different work environments, cultural norms and backgrounds that can also affect your resiliency, sleeping patterns and “CEO part of the brain” (a phrase that comes up in this study a few times).
A little further on, we learn journalists are dehydrated because in this self-reporting study, most didn’t drink at least 8 glasses of water a day. Never mind that there have been numerous scientific studies indicating that benchmark is an old wive’s tale.
As for the control group placing journalists “below average”? Turns out it’s bankers, telecom groups and sales teams that the author has run as part of the paid services she offers to corporate groups hoping to “to achieve a competitive edge by understanding and improving the physical condition of their brains.”.

There is no indication of the study’s weaknesses- standard for actual science publications

I looked for the section indicating the weaknesses in the methodology, areas for further review and comments about how we shouldn’t take this as gospel– standard fare for any actual scientific piece. There was none. No warnings, no cautions, just a nicely packaged piece ready for distribution to media.

What’s being sold?

A standard question I’ve been taught to ask whenever presented with a study, survey or research is who stands to gain from this? So let’s ask a few questions about this one.
The study came from the personal website of it’s author. What’s on offer there?

And what does the press release about the study say about her?

If I were to be cynical, I’d point out that by putting out a study on the minds of journalists, Swart has managed to get her name in numerous media organizations with a story that is being shared onto the personal Facebook and Twitter pages of countless journalists around the world.

Why does this matter?

Again, haha, I get it, we drink too much coffee and alcohol and are stressed out! And now science proves it!
Except it doesn’t.
Journalism is the primary way most scientific information gets disseminated to the general population, be it through quick news articles recounting study highlights or more in-depth areas like Popular Science or Quirks and Quarks.
Even when it’s for something ~fun~ and ~silly~ we should turn on our skeptical minds and try to educate both ourselves and the general public about how to tell the difference between good and bad science and studies, their shortcomings, and ways to properly evaluate information. That includes asking questions about where information is coming from, whether it was peer-reviewed, sample size and whether something is being sold.
As journalists, we need to avoid confirmation bias- even if it’s about ourselves.
 

Filed under: journalism