From Think, by Simon Blackburn
Suppose you decide to check yourself out for some disease. Suppose that this disease is quite rare in the population: only about one in a thousand people suffer from it. But you go to your doctor, who says he has a good test for it. The test is in fact over 99 per cent reliable!
Faced with this, you take the test. Then — horrors! — you test positive. You have tested positive, and the test is better than 99 per cent reliable. How bad is your situation, or in other words, what is the chance you have the disease?
Most people say, it’s terrible: you are virtually certain to have the disease.
But suppose, being a thinker, you ask the doctor a bit more about this 99 per cent reliability. Suppose you get this information:
(1) If you have the disease, the test will say you have it.
(2) The test sometimes, but very rarely, gives ‘false positives’. In only a very few cases — around 1 per cent — does it say that someone has the disease when they do not. These two together make up the better than 99 per cent reliability. You might think that you are still virtually certain to have the disease. But in fact this is entirely wrong. Given the facts, your chance of having the disease is a little less than 10 per cent.
Why? Well, suppose 1,000 people take the test. Given the general incidence of the disease (the ‘base rate’), one of them might be expected to have it. The test will say he has it. It will also say that one per cent of the rest of those tested, i.e. roughly ten people, have it. So eleven people might be expected to test positive, of whom only one will have the disease. It is true the news was bad — you have gone from a 1 in 1,000 chance of disease to a 1 in 11 chance — but it is still far more probable that you are healthy than not. Getting this answer wrong is called the fallacy of ignoring the base rate.
Two comments.
(a) Surely all this is taken into account when the “reliability” of a test is calculated. ie what are my chances of having the disease based on positive or negative results. Do people in their right mind work differently?
(b) There is the further issue of false negatives to be taken into account. (But you know that)
/davblo
A good doctor will take it into account, but a bad doctor won’t take it into account. That is why patients should themselves go to the trouble of taking it into account. As far as I know, doctors have no special training in interpreting statistics. And even if they had, statisticians themselves routinely misinterpret statistics, because they are mostly epistemologically uninformed.
For example, statisticians routinely confuse relative frequency and credibility (10% of Irish people have red hair, therefore – they wrongly think – our confidence in believing that an Irish person has red hair is 10%). They confuse truth and certainty (we mustn’t think “probability of 1 = true” or “probability of 0 = false”). They routinely extrapolate in unreliable ways (10% of Irish people have red hair, therefore – they wrongly think – 10% of all people have red hair).
Patients tend to “put themselves in the hands” of a doctor, letting the doctor make decisions on their behalf. It takes some effort or even tough-mindedness to resist what the expert says. In Ireland, celebrities on the radio exhort men to take the prostate-specific antigen test for prostate cancer, and women to have mammograms for breast cancer. Yet we may reasonably doubt that either of these screening programmes does more good than harm:
http://www.newscientist.com/article/dn21041-prostate-screening-does-more-harm-than-good-in-us.html
http://www.telegraph.co.uk/health/healthnews/8943479/Breast-cancer-screening-could-cause-more-harm-than-good.html
The manufacturers of tests cannot take the base rate into account if there are significant differences between groups being tested, as there usually are. For example, AIDS is much commoner among gay men than in the population at large, and breast cancer tends to run in families. An intelligent person taking an AIDS test will reflect on his own lifestyle, and an intelligent woman will think back on her family history when interpreting results. These involve personal details that no doctor can reasonably know about. As long as someone or other takes account of the base rate, and we don’t confuse relative frequency with credibility, the “99% reliable” description is an honest and straightforward way of putting it.
Most tests err on one side or the other, that is they yield many false negatives and very few false positives, or else many false positives and very few false negatives. With those that yield many false negatives, the situation is similar to the one described by Blackburn above, except in mirror-image. In that situation, the patient who leaves the doctor’s office relieved that he has “a clean bill of health” should feel less warranted in his optimism. Either way, the rational position is one of mitigated scepticism, or at least greater scepticism that we usually meet. We should not trust experts as much as we do, and think for ourselves more.