Facts? What Facts?

If you like my blog, please tell others.
If you don’t like it, please tell me.

"It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so." (Attributed to Mark Twain)

Your neighbor tells you how a certain over-the-counter product greatly improved his health. He had only a couple of mild colds last winter instead of the four or five bad ones he usually gets. Or you read a similar recommendation in somebody’s blog. They claim it’s wonderful and you really should try it. Do you rush right out and buy the stuff? I would wary, for several reasons:

a. The proponents of the product might be seeing just the placebo effect. People tend to get better when they think a medication is helping, even if that medication was in fact just a sugar pill.

b. Some bloggers are being paid to push certain products – and at present they are not required disclose that fact. This can even enhance the placebo effect since the blogger wants it to work so that he can legitimately get paid. He may not be deliberately dishonest, but he can easily fool himself.

c. Your informant may occasionally be deliberately dishonest. That is especially true if he is being paid to advertise that product.

d. People recover from most diseases on their own. The medication may have had nothing to do with the recovery.

e. Random chance plays a big part in things like sickness, as do changes in our bodies. Your neighbor may have just not been exposed to as many colds as usual, or his aging, more experienced body may have already learned to deal with the viruses he happened to meet last winter.

So how do you decide if you should try the medication or not? Or if you should accept some other product recommendation? There is no easy way, but some things can help. The first step is gathering information, that’s kind of obvious. The problem is, what information do you need and how do you know what to trust? The following questions are helpful. They are based on suggestions from the Center for Evaluative Clinical Studies (CECS) of the Dartmouth Medical School:

First, exactly what is the assertion? What does it claim and not claim? It is all too easy to unconsciously extrapolate to other situations where that assertion does not necessarily apply.

Second, would you care if the assertion were true? If you live in Hawaii you probably won’t care much about a product claiming to make a home heating system more efficient.

Third, who stands to benefit? The blogger or other advertiser is likely to be biased in favor of the product, consciously or unconsciously. Likewise the manufacturer or salesperson.

Forth, how good is the evidence? Does it come from multiple, independent studies? How good are those studies?

Now that last question can be difficult for a layperson to answer. Most people lack the background and training to evaluate such studies. However again we can build on CECS suggestions. We should ask ourselves:

What are the key elements of any study? What population? (A study showing a medication safe and effective in adults may not be meaningful to children, for example.) What exposure? What was the outcome in terms of both nature of effect and size of that effect?

Are the data relevant? Did researchers use the right study population and exposure? (Studies exposing rats to many times the equivalent dose human users might take should be considered suspect.) Is the effect large enough to be both statistically significant and meaningful to you?

Are the data valid? Is there a properly chosen comparison group? Were measurements done properly? Might the effect be due to chance?

Finally, are there other studies with similar findings to support the assertion? Are there studies that contradict the assertion?

That’s a lot of questions but they are all useful. If the stakes are low you might not want to pursue answers to all of them. However it is worthwhile to understand each question and know how it applies.

Of course most people don’t know how to evaluate statistical significance. That’s a big subject but I’ll try to make a start on it next time.