In his first Thursday Night Insight, Kyle Cockett examines the dangers of taking statistics at face value.
Although I have only been working in the field of market research for a relatively short length of time, I have quickly realised the value of a well executed piece of quantitative research. When correctly designed, using a valid, reliable sample, quantitative research can be used to provide clients with strong conclusive findings, often enhanced by the use of inferential statistical techniques. Correlation analysis, CHAID analysis and factor analysis are all examples of such techniques that can add value to the overall conclusion, for example, to prove or disprove a prior hypothesis the client holds, such as ‘is group x significantly more satisfied than group y’ etc.
Almost every day we come across the findings of various types of research reported in the media, often on a range of weird and wonderful topics – but how many of us actually question the research method utilised and where the findings have come from?
Recently a story made the national news concerning a Plymouth school teacher – Richard Gribble – who found that pupils were finding it increasingly difficult to concentrate in class. The headline read:
Children addicted to computer games ‘unfit for school’
The headline was augmented by a passage of text which stated that ‘games addicted children are missing meals, talking about computer games during lesson times, tired and show poor concentration according to new research’. Upon investigating the story further, I found that the same damning verdict on our nation’s children was delivered by several other media outlets, one, reporting that ‘primary pupils are falling asleep at their desks after playing computer games until 4am’. It was only on delving deeper into the article that it revealed that the sample size of the research was in fact Mr Gribble’s own primary school class of 26 pupils – hardly representative of the population sample, especially when the National Office of Statistics estimates there are around 11.5 million under sixteen year olds! Unfortunately this point appeared to be lost to the majority of online commentators, for whom the power of the headline overshadowed the reliability and validity of the methodology used to arrive at the finding.
Indeed, it seems that the famous Albert Einstein quote ‘if the facts don’t fit the theory, change the facts’ has become all too literal when it comes to general media reporting of research findings. In fact, the National Health Service has now dedicated the news section of their website to investigating and reporting the statistical significance and causality between many of the tenuous links often reported in the media – can saucepans really cause early menopause? Can a pill cure your fear of heights? Is breastfeeding linked to school grades?
Michael Blastland, writing in his ‘Go Figure’ column for the BBC, perfectly sums up the problem of taking numbers at face value without reading between the lines. Take a look at the picture above. The picture was taken just prior to the general election. It shows a well known potato crisp company handing out free packets of crisps displaying the image of the three prospective prime ministers. Ostensibly, on face value the image implies that Nick Clegg is the most popular of the three candidates; based on the seeming popularity of his bags of crisps, depicted by the emptiest bin. We might be surprised by this, but at the same time, question why the image would lie? However, further thought raises a number of competing explanations; has the Gordon Brown bin just been refilled? Are the crisps different flavours, with Nick Clegg’s the most popular flavour? Which candidate was supplied with the most boxes of crisps initially?
I guess the point I am trying to make is that anybody can take numbers at face value. Blastland himself professes that ‘clever people – and newspapers and politicians – say outrageously daft things, often, with them and about them’. However, without some form of added insight, figures in their own right are often of little value. Click here to find out how you can use statistical techniques to add value to your research project at B2B International.
