Wednesday, 18 March 2009

The Dangers of Self-Selecting Samples

Now first up, you will have noticed a new addition to the blogroll on the left, with the addition of the excellent JustLife, "a space dedicated to exploring the connections between faith, money, justice and lifestyle". So if you would like to explore the connections between faith, money, justice and lifestyle, then do follow the link and discover.

Secondly, you may have been following, with interest or otherwise, the tales of St Patrick's Day riots in the Holylands area of Belfast. Now, there have been a number of calls for the university to suspend students who were involved, but frankly I wonder why this should be necessary. All I know is that, when I was a student, if I had been seen on national television, being drunk in the street and throwing things at people, my parents would have been hot-footing it up to Belfast and there wouldn't have been enough pieces of me left for the university to bother suspending me. In any case I would probably not have been seen doing any of those things at all, for I would have been in the library gaining both an Education and Curvature of the Spine.

But to get to today's point, which is the latest in our series on Dodgy Mathematics. Our topic today is 'The danger of the self-selecting sample', and I think we shall use our second point from above as an illustration, although it was not, in fact, the inspiration behind the post.

You will be aware that if you wish to find out something about people (let us say, their behaviour when drunk), it is not normally practical to survey everyone; instead one selects a sample of people (let us say 400 persons), and aims to ensure that one's sample is representative (so one selects 200 men and 200 women, all of a range of ages and so on, covering all pertinent factors). One carries out an experiment (plying the people with alcohol) on the sample of people, and then we use a process called extrapolation, which essentially says that if half the people in our sample get drunk and see pink rabbits, then half of all people everywhere will get drunk and see pink rabbits.

However. This is well and good in a laboratory, but it is a little too scientific for the modern media, who are less concerned with truth and more with selling papers, and who therefore do not take time to select a careful sample, but just listen to whoever shouts loudest. Hence we see that all students shown on the news last night were rioting, and we are led to conclude that 'Students Riot'; this ignores the students like WhyNotSmile who are not drunk in the street because they are in the library reading books and not being drunk at all.

I mention this because I came across someone today who had been given new medication and was concerned for the side-effects. She promptly did what one should never do with any sort of medical condition, and Googled it. Now we all know that if you type any random selection of symptoms into Google, and click for long enough, you will sooner or later be told you are going to die, which is not entirely helpful and may not even be true; likewise, if you type in a drug name followed by "side effects", it is unlikely that you will be taken to a message board which consists of messages saying "I took this drug and it was fine", because, frankly, if you take a drug and nothing much goes wrong, you are not terribly likely to tell the internet (unless it asks); if, on the other hand, it turns you into a wibbly mess then you are highly likely to decide that the world needs to know.

So, to summarise, typing drug name + "side effects" into Google is not likely to be comforting, but fortunately is also unlikely to be representative, and is therefore A Bad Idea. Like rioting.


Anonymous said...

Came across an anomaly on Lisinopril (high blood pressure tablets) recently. Read the small print that came with packet and thought the side effects, though many, were not too serious. Three weeks later after getting chest pains for several nights, googled Lisinopril and chest pains and what do you know. Do not pass go, contact doctor immediately! Questioned doctor as to why chest pains were not on packet small print, but on internet big print. No answer. Now off Lisinopril and seeing hospital tomorrow about alternative medication.

whynotsmile said...

Yikes! Scary!

I once had a thing of tablets that said 'If you take too many, restful sleep would be expected. In the event of an overdose, see a doctor if awake'.

nellyandi said...

2 points

1) if you take lisinopril it's generally to stop you dying from things like heart attacks (which cause chest pain) and therefore if you need to be on lisinopril then you are already at risk of heart attacks (and chest pain) and maybe the lisinopril did not cause your chest pain. correlation and causality and all that.

best example - people who receive oxygen in hospital are far more likely to die. Does oxygen kill these people? No.
The more straightforward way to look at it is that sick people receive oxygen in hospital and sick people have this annoying tendency of dying

2) if you developed left ear lobe pain with this medication you would be annoyed but not concerned that the medicine is killing you. the chest pain you have may not do you any harm whatsoever (however annoying and painful it may be) - it is not chest pain that will do you harm but a heart attack. thankfully lisinopril doesn't cause heart attacks... even though it may cause chest pain

PS my best quote on oxygen is - that it's a toxic substance that brings about a mixture of euphoria and depression, ultimately resulting in death over a period of roughly 70-80 years