How Can We Tell When a Weight Loss study is Unreliable?

In my job as a fat activist, people like to fling links at me that they think disprove the things I am saying.

When the planets align in just the right way and the whim strikes me, I’ll dig into their links and see if there’s anything there. I haven’t been flung a worthwhile one yet, but I’m open to new experiences. And I want to teach you how to do it too so you can have a better idea of how to sort out the good information from the bad.

When we all have so much information at our fingertips, how can we know what’s accurate? Anyone can put up a website or make a video. How can we know who is a real expert and who is playing fast and loose with the facts?

These questions are especially important for information about health and weight. Diet culture has pretty thoroughly broken the system of experts that we would normally use to tell us who to listen to.

People who are opposed to fat activism often make blanket appeals to authority—to doctors, to scientific journals, to government agencies—without understanding or caring that these systems of authority have failed us.

This is not just a problem when it comes to diet culture and health information either. When industries want to shield themselves from responsibility for the harms they cause, they go fund their own research and studies to assure everyone that the industry is blameless.

We’ve seen it with cigarette manufacturers. We’ve seen it with the fossil fuel industry and climate change. We’ve also seen the hollowing out of government regulatory agencies that are supposed to protect the public, replacing career experts with industry people.

Agencies like the Centers for Disease Control and the Food and Drug Administration have had their budgets threatened by members of Congress who are only concerned with making their industry donors happy. In some cases—like tracking deaths from guns or making sure herbal supplements are safe—these agencies have had their regulatory power removed entirely.

When even Young Earth creationists have their own “scientific” journal, who can you trust?

In the long term, we desperately need to fix the way scientific research is funded and restore the regulatory framework that protects the public. But what can we do in the short term to protect ourselves from being taken in by fake experts and misleading studies?

We can learn how to recognize the common red flags of bad information. We can learn how to be skeptical and how to trust ourselves. We can actively evaluate the information we take in, instead of passively accepting it.

And while I can’t exactly tell you how to tell who is an expert in weight and its potential relationship to health, I can definitely tell you who isn’t.

• Anyone who thinks BMI is a telling measurement of health isn’t an expert.
• Anyone who thinks that they can tell how healthy someone is just by looking at their size isn’t an expert.
• Anyone who still believes in the disproven “calories in, calories out” model isn’t an expert.
• And anyone who tells you that there is a guaranteed way to make all fat people into thin people definitely doesn’t know what they’re talking about and definitely isn’t an expert.

But what about all these screaming headlines about Brand! New! Studies! about weight loss and how bad bad bad it is to be fat?

How can we tell the clickbait nonsense from the real articles?

Well, one red flag is an article that talks about a major new study about Fat People BAD that doesn’t actually give the name of the study or where it was published.

I always want to see the original paper, because it’s pretty common for a study to not say what the article claims it does. Not naming the study is also a pretty good indication that the “article” is a barely reworked press release, which is an additional sign that the study probably isn’t very good.

The way I work around this is to google the name of any scientist quoted in the article. Usually whatever scientist these glorified press releases are quoting is one of the study authors. Once you have their name, you can google it along with the topic of the study, and the actual study itself will usually pop right up.

Paywalls on academic journals really cramp my style, but you can often tell a lot just from the abstract. A good study abstract should describe how the experiment was designed and should clearly lay out what the results were.

A lot of studies are available online and are already accessible. For the ones that aren’t, if you live near a university, you may be able to use their library to access studies that would otherwise be paywall locked.

As always, when in doubt, ask a librarian. That’s just good overall life advice.

So, what am I looking for to determine when I don’t think a study is credible?

How many people were included in the study?  

I’ve seen really alarming fatphobic headlines based on a case study on literally one person. The more people that are included in the study group, the more likely it is that the study is actually meaningful.

How long were people studied? Where there any follow-ups with the subjects? How long were the follow-ups continued for? 

This is important in any study of health, but it is especially important for studies of weight. We know that weight regain occurs over time, so a study that doesn’t follow up with people over an extended period of time—5 years or more—is not going to give the full picture of how weight loss and regain acts on the body.

Are the study participants allowed to self-report their results?

It should (but apparently doesn’t) go without saying that allowing people to self-report their own weight loss amounts is not going to give accurate results. So this is me saying it.

How are people who dropped out of the study treated?

People who have dropped out of a weight loss study must be treated as having failed the weight loss program and included within the final results in order to generate credible results.

How broadly or narrowly is the study targeted? 

A study that looks at whether the stress hormone cortisol is elevated in people engaging in weight loss is going to give results that are way more trustworthy than a study into how many fat people “die too soon.”

In the first example, the scientist can test the subjects’ cortisol level prior to a weight loss attempt and then test it after (assuming no other factor might have affected cortisol levels), giving clear results. In the second example, the question is so broad that it makes me wonder how you would even design a study to look into it at all.

What does “too soon” even mean? How could this even be measured, since as far as I know, everyone is going to die eventually? What if almost no one dies in the time period you’re looking at? Is the study going to count every fat person who dies in any way as having died from Fat? (Yes, that has happened.)

Without fail, the studies that claim that they show that “permanent” weight loss is always possible are relying on self-reporting, not doing follow up work, and ignoring their dropouts in order to mess around with the data.

I am really not here for any scientist who complains that actually weighing people or following up with them over time or not using discredited measurements like BMI is “too hard.”

Historically, the only reason people started recording weight as a measure of health in the first place was because it’s really easy to track and record. And I’m not the one telling fat people to magic themselves into thin people permanently with some kind of wizardry that no one has ever once EVER been able to demonstrate under proper scientific conditions.

Not being able to access health care because of bad studies is too hard. Fatphobic discrimination is too hard. Don’t tell me it’s “too hard” to do your job properly, scientists, because when you don’t, you are putting people’s lives in danger.

I wish more scientists cared about that. But until they do, it is possible to teach yourself how to sort out a lot of the bad information from the good. Adopting a mindset of skepticism toward claims of permanent weight loss is a good place to start.

In all the years since the first diet manual was published in 1863, no one has been able to show a diet that produces permanent weight loss for the overwhelming majority of people. It seems like at this point, we should consider that any study claiming to do what no one else can is making an extraordinary claim. And extraordinary claims require extraordinary evidence.

Ali Thompson is the Bill Nye of fat girls. She is the creator of Ok2BeFat, which by an amazing coincidence, is the name of her YouTube channel. She’s a fat activist, writer, YouTuber, and collage artist. She is a bisexual queer who lives in Philadelphia with her husband Josh and their many cats. You can find her on Twitter at @Artists_Ali, where she probably just said something weird. Find out more at alithompson.net.