, , , , , ,

how good is your evidence

If there’s one thing you can’t take away from CF patients and families, it’s their tenacity. A lot of us are opinionated, strong-willed, and enthusiastically outspoken.  Some have tattoos that say things like, Never Give Up, some can rock a fundraising campaign that could rival a corporate-sponsored event, and some of us write blogs. 🙂 So it’s of no surprise that many of us pay eagle-eyed attention to news stories that feature cystic fibrosis and potential new treatment options. Unfortunately, that drug pipeline moves frustratingly slow, especially when every day we’re still breathing is precious. It’s to be expected that many would turn to “natural alternatives” while waiting for that big-time science stuff to catch up. But how do you know that story you just saw in GreenMedInfo about the Top 10 herbs for lung health is for real?

Sure, the article mentions research studies, or they say, “scientists found that…” but is that enough to go on? I would argue no. Sources matter. When you are up for trying anything that helps, or just want to cover all your bases, you need to be sure that the source from which you get your information is trustworthy. Whether it’s from that natural living blog you follow, or a tip from a friend, take the time to investigate it. This is your health. You may think, “yeah, but, it couldn’t hurt!” To you I say: Bollocks. Natural doesn’t always make it safe. I cannot stress that enough. In fact, I’m going to say it again. Natural does not equal safe!

It does take time to dig into the claims you hear, and I realize that despite your disease, you really do keep yourself pretty busy, (it’s that tenacity again). But I promise the more you do it, the faster you’ll get. You’ll start learning the patterns of good source material and bad source material. Soon you’ll be able to perform a quick sort of litmus test to verify what makes for good information.

To start, if you’re reading a news article, is it a reputable site? Consider the experience of the author and decide what level of credibility to assign to them. Science journalism has really gotten bad, as the need for ever streaming information and sensationalized click-bait takes priority, but there are still a few good publications that haven’t gone the route of the Daily Mail. Also, there are a lot of “news” sites that are run by conspiracy theorist nut jobs, or Alt-Med swindlers. Check some of their other postings, does there appear to be an agenda? Is there a link to their store? People who just want to give honest information will not try to sell you anything.

So, you’ve passed that step and things look still good, now check their references. They should be listed. If not, take it as a red flag. Tedious though it may be, click on each one of them. A lot of sites will try to bulk up their source list to appear more legitimate. But it’s a façade. I once read an article that listed dozens and dozens of sources; so many that I had to download it as a PDF. But when I started going through, I realized close to half the studies listed were not even on the subject of the article. Or they would contain one of the keywords, but offer zero contributing information that the author could have pulled from. Still yet, some studies cited will actually not support the author’s argument at all. If you read them, they would sometimes conclude the opposite of what the author claimed was true. This is usually as a result of cherry-picking. An author will pull a sentence out of the abstract of a study that appears to support their claim, but ignore or even hide the rest of the data in hopes you won’t notice. And it works. A lot. Because people don’t check.

**ADDENDUM** July 28, 2014. While reading a like-minded blogger’s post at Fighting Pseudoscience And Promoting Critical Thinking, I was reminded of another tactic that dubious sites will often use to trick people: citing themselves! Many false news websites will claim there is a lot of research to back up their statements, but look closely, they may be written by authors of the same site, perhaps under an alias, or at a sister site, which will in-turn link back to the original article. It’s a cyclical tactic that many people fall for if they don’t do some digging.

Now you’ve got your list of studies and supporting evidence, it’s time to vet them. On the hierarchy of clinical research studies, the gold standard would be a large scale, randomized, double-blind placebo controlled study. Good luck. But it’s something to aim for. Check the methodology of the study. Does it seem fair? Is there anywhere that a researcher’s bias could distort the findings? How many subjects were involved? Could the test be repeated and get the same results? Is it a human, animal, or in vitro study? Results in a petri dish are not on the same level as results in the human body. Promising though it may be that peppermint oil killed pseudomonas in a cell culture, it would be highly irresponsible to recommend nebulizing peppermint oil as part of a daily treatment.

In cases where a topic is well researched, it may be better to look for systematic reviews or meta-analyses. These are when a group of peer-reviewed studies on the same topic are gathered together and carefully reviewed by independent scientists to track trends in results and come to a more conclusive report about the topic as a whole. The best place to find these are Cochrane Reviews. Additionally, learn to familiarize yourself with the names of high quality medical journals. While it’s possible that some junky studies slip through, you’re likely pretty solid when you see a study published in those journals. There are such things as pay-to-publish journals, which really don’t care what you write or how accurate your results, if you have a study to submit, and the funds to submit it, they’ll publish you. Beware of this.

At this point you may be thinking, “That is a lot of fact checking for one little claim.” Yes. Yes it is. Welcome to the life of a skeptic. You may think you don’t have the energy, nor the desire to face that sort of pressure, but don’t you think it’s important? I’m a science advocate, so I may be biased when I say that critical thinking is the number one most important skill to have. Even if you don’t understand much about scientific concepts, you can still train yourself to be a better judge of what’s good for you and what’s just hype. Use that inherent tenacity in the most effective way possible, by not settling for bad source information. And if you still can’t be bothered to do the leg work, you can always ask me and I’ll do it. 🙂 Or check with your doctor. They’re usually pretty good about that stuff.