The evidence-based community isn’t without flaws. Here’s what they are, and how we can fix them.

PART 1

It’s 2018. The fitness industry, once dominated by the six foods that work, bro splits, and relentless BCAA consumption, has now moved on to flexible dieting, full-body workouts, and, most interestingly, mental health. The words “science” and “evidence”, once either ridiculed as irrelevant or occasionally thrown in for good measure, are now approaching booty selfies in their popularity. And, to be sure, this is an overwhelmingly positive change; it represents a shift in the zeitgeist from an intense focus on that which doesn’t matter, or doesn’t matter much, to that which does – and the result is that people make better gains, more efficiently, and are much happier for it. But this movement isn’t without weaknesses of its own. Even putting aside the people, beliefs, and claims which are obviously not evidence-based (after all, ‘sciency’ justifications for bullshit have always existed) – because going after all that would be too easy and too devoid of originality or substance – it’s been my perception that there are some real, problematic currents within the evidence-based community that need to be directly addressed for the betterment of all of us. And I figure since I have this perception, I might as well be the person to do that.

A quick word on the structure of this piece: it’s actually the first in a series of as-yet-unknown length. I wanted to make a point of breaking things up as I want this to be accessible (I fear the possibility of putting everything in one article and it subsequently flying under the radar). Thus, I’m going to spend the remainder of this piece talking about just one problem in the evidence-based community:

OVER-RELIANCE ON EXPERT OPINION

At the risk of unnecessarily playing down my own work, everything I’m about to say can be summarized in a single statement: expert-based is not evidence-based. With that now in the air for you to consider, here comes the nuance: this is not a critique of experts. Indeed, I have tremendous respect for a number of figures in the community (who I’m deliberately not going to name, to remain narrowly focused on the issue at hand), and there are even a handful of people whose word I do very nearly take as gospel within their area of expertise. However, even if there are 10, or 100, or 1000, or even 10000 people who are very reliably correct in their assessment of the scientific literature, the fact remains that as long as being evidence-based requires you to reasonably account for all evidence (which it does), basing your views on what experts say about evidence does not make you evidence-based. Thus, when I have a question in an area outside my own expertise and I contact a friend or colleague I trust to get their opinion, I might get a workable answer, but I’m not being evidence-based.

The obvious objection to this – and one which I’m happy to acknowledge I almost believe – is that as long as someone I perceive as an expert is actually an expert – in other words, as long as their beliefs are evidence-based, basing my beliefs on their beliefs makes me evidence-based by proxy. This is wrong. No matter how objective, rational, intelligent, and knowledgeable the expert, their interpretation of the available evidence is nonetheless necessarily an interpretation.

As an analogy, consider the case of the telephone game, in which a number of people, sitting in a circle, whisper something from one person to the next, with each person doing their absolute best to whisper to the next person exactly what they heard themselves. In every case, by the time the message returns to the first person, it’s massively distorted. In my view, this is genuinely comparable to much of what we see in the evidence-based community, a community in which, for example, it remains common to see the claim that nutrient timing doesn’t matter, despite the fact a substantial body of scientific research has undermined this view for at least five years. So where does this claim come from, and why does it remain popular? The telephone game, combined with a failure on the part of the lay member of the community to engage with the literature they think supports a view it actually undermines. Now, one might remark that this is an exaggeration; if you trust the judgment of an expert, who might have actually done the research about which they’re speaking, you’re only one step away from the source, and this isn’t the same as hearing a message that’s been filtered through ten people. And of course that’s right – but only as a matter of degree. In other words, the fact remains that in the context of being expert-based, the source of one’s beliefs is, well, what someone else believes. If you don’t buy this argument, consider the following question: is there a genuine distinction – a real difference – between the message whispered by the very first person in the telephone game, and the message whispered by the next? Frankly, I don’t see how one could reasonably argue “No.” without assuming an inhuman level of retention and recall, and as long as the answer is “Yes.”, this implies there is also a real difference between scientific evidence itself and what experts think and say about that evidence.

And while I don’t want to claim that exhausts this line of discussion, that’s the crux of the argument. Next time, I’ll dig in to how we can fix this.

Yours truly in EBP,

Ian McCarthy