In my previous post I said anecdotes can be used as evidence, if the conditions are correct. That is a controversial statement, so does need to be explained. And, here I plan to do that by giving you one example of when anecdotes do work and when they don’t. In typical-teacher fashion, I am going to let you think about what the differences are. But I am going to illuminate the difference.
Imagine there is an illness that has a 1% survival rate untreated and a 5% survival rate treated (both over 5 years); that is a pretty horrible illness. If you get that illness, the bet to make at the bookies is against you. But assume this illness exists and a series of “alternative healers” (spiritual, Biblical and Raiki healers) have thousands of patients who have overcome the illness while visiting them. Thousands! There must be something in it, therefore God (or something). If I had to conclude from that evidence, I would conclude something in line with Occam’s razor: the power of psychology over the body. There is already evidence for that.
But instead of jumping to conclusions, we should be asking questions. How many people had the illness? How many people went to see an alternative healer? How many of them were still receiving medical treatment*? What are the false-positive diagnosis rates for this illness? Once we have the answer to these questions, we can start to look into whether the alternative medicine anecdotes really are evidence.
I’ll have to assume some data for the illness, because I’m making it up. Lets say 15 million people got the illness in the last 5 years, in the English-speaking world. 5 million of them do not get medical treatments because they can’t afford it (shame on you, USA), or for religious or personal reasons. Assume 80% of those people go to an alternative healer; that’s 4 million people. Untreated, they have a 1% survival rate. 1% of 4 million is 40,000. So, there’s 40,000 anecdotes. And 10 million people left to go.
The remaining 10 million people are receiving treatment, and have a 5% survival rate. Still, the illness means falling on pretty desperate times. Assume 10% of all treated patients are seek alternative healers. That’s 1 million people seeking alternative healers. If alternative healing is of no benefit, expect a 5% survival rate, 50,000 people. There’s a further 50,000 people who survived after seeking alternative healers.
That’s a total of 90,000 anecdotes, and yet the alternative healers don’t have a success rate higher than if they didn’t exist. There’s nothing in it.
*When I was teaching in Austria I remember meeting this one particular hippy Teacher who had some low-lying illness that she couldn’t seem to shift. She went to a doctor and the doctor gave her some antibiotics. After she read the possible side effects she decided they couldn’t be good for her, so sought an alternative healer. They gave her homeopathic medicine. And, voilà, her illness was gone. Evidence for homeopathy. Except, she was taking the antibiotics as well.
Public health investigations
Imagine you work in public health, and your associate NHS Hospital reports an increase in severe stomach upsets. How do you go about pinpointing the cause of this increase? First, you make some informed guesses: if the number has gone up from about 2 per day to about 16 per day, we’re probably not looking at a contaminated water supply. You would expect people to be coming in by their hundreds if that was the case. You have 16 people to start asking questions: where have they bought their food over the last week? What restaurants have they visited over the last week? Any overlap helps you find the cause of the illness. That is anecdotal evidence slowly becoming evidence.