Archive for the ‘Sampling’ Category
Monday, April 12th, 2010
We like Luke Johnson’s column in the Financial Times and regularly feature it on our blog. He is something of a hero of ours, having built up Pizza Express, become exceedingly rich as a private equity investor, and proved to be an astute commentator on entrepreneurial issues. It was with a sinking heart that we read the article he wrote about focus groups in the Financial Times last week.
Luke Johnson singled out focus groups for criticism but in truth his points could be applied to any type of market research. He claims that market research he commissioned told him very little that he did not already know. This will not be an unknown observation to many market research practitioners reading this blog. They will have often times presented a detailed and thorough piece of research with many incisive findings and sharp recommendations, only to hear some member of the audience claim that there was nothing new in it for them.
One way round this would be to ask everyone who attends a market research presentation to bring with them a sealed envelope in which they summarise their pre-understanding of the market so that it can be compared with the findings of the research survey. When market research findings are laid out in a clear structure it is quite likely (and quite right) that much of what is said will concur with existing knowledge. However, without the market research, the knowledge may be uncoordinated and there could be many internal disputes about where the marketing priorities lie. A good piece of research that confirms knowledge and places it in a framework that everyone understands, can be worth its weight in gold.
However, there is another issue that is worth considering. There is an old saying that “a problem defined is a problem half solved”. In other words, a good brief, which clearly lays out the problem to be researched, is critical in delivering findings that resonate. In the case outlined by Luke Johnson, an obvious question we would ask is “what were the focus groups seeking to achieve?”. Focus groups work best as an exploratory tool, possibly forming the first stage of a larger program. Were the focus groups used as a single research solution to something that should have had a qualitative stage followed by a quantitative stage? We don’t know, but we do know that there is a dangerous tendency to commission focus groups as a stand-alone research solution when they should be just one component part.
We shall carry on reading and promoting Luke Johnson’s articles because he challenges all and everything around him and we like that. However, on this occasion Luke, we think you may have been a tad hard on the market researchers and maybe you should also challenge yourself — did the research really tell you nothing worth knowing, and did you really give the researchers the right brief?
Read on.
Why focus groups tell you the obvious
By Luke Johnson
Financial Times – 24th March 2010
I recently commissioned some market research and, as is too often the case, it told me what I already knew or was obvious. I paid the bill of several tens of thousands of pounds, consoling myself with the fact that the work at least confirmed my prejudices – always a satisfying sensation. But I also sensed I had received very poor value; and in talking to other clients of research companies, I realise quite a few feel the same way.
As Michael Skapinker wrote yesterday, the idea that the customer is always right has become an accepted truth in business. Unfortunately, customer desires are often wholly unrealistic – because of cost, technology or legislation. As Henry Ford said at the launch of the Model T: “If I’d asked the customer, he’d have asked for a faster horse.”
I remember Peter Boizot, founder of PizzaExpress and my predecessor as chairman, telling me how, in 1965, customers in his Soho pizzeria felt uncomfortable with authentic Italian pizza – and demanded chips. But he stuck to his vision and guided their tastes to the genuine product.
I have also experienced data blindness over research studies. Consultancies supply blizzards of material – far more than could ever be useful. Wordy, sprawling PowerPoint presentations compensate for a lack of incisive thinking. One can end up paralysed with indecision, buried in e-mails too large to even download.
Great breakthroughs in fields such as new product development are frequently achieved by avoiding surveys and committees altogether. Constant testing can lead to blandness and safety-first choices. In creative affairs, corporate brainstorming sessions usually end up with groupthink dullness, all originality squeezed out because of the fear of failure or through the influence of office politics. As Steve Jobs said: “It’s really hard to design products by focus groups. A lot of times, people don’t know what they want until you show it to them.”
At Channel 4, many of the most brilliant and distinctive programme ideas during my time as chairman were pioneered by eccentric independent producers who were championed by renegade commissioners. Meanwhile expensive, mainstream concepts often flopped.
Over the decades since I worked in advertising, I have sat in many focus groups and wondered about the quality and effectiveness of such qualitative research. After all, who submits to a two-hour discussion about brands of washing-up liquid? All too often, the answer is the lonely, the old, the unemployed, students and, most worrying of all, serial participants in search of the small stipend and free tea and biscuits. It is very hard to persuade a normal working person to attend such panels, but they are usually the target subjects.
I worry that researchers who appear to succeed are too often the snazzy firms who trade in sexy stereotyping. They use phrases like “Inner City Adversity” and “Twilight Subsistence” to categorise and supposedly understand various imagined socioeconomic and demographic groups. I am unconvinced that this terminology and philosophy is especially practical and relevant for many companies. In my restaurants, the people who know our customers are not researchers but branch managers, who serve the public all day, every week. Our staff may not have the slick patter, but they have the frontline, first-hand knowledge.
Another unfortunate byproduct of the growth of research has been the increasing use of surveys by political and charitable organisations in their campaigns. Almost every day a pressure group gets publicity by publishing selective and scary conclusions about poverty, health, discrimination or other controversial issues. Journalists rarely question the study methods or validity of the results. Even if there were no errors in the sampling techniques, questionnaires or systems used, the media often over-simplify and exaggerate outcomes.
Over-reliance on researchers means owners and managers are separated from the consumer. Successful entrepreneurs I know put more effort in talking to customers themselves, than they do working with costly experts who tell them what they should have learned long before.
Posted in
Articles, Data Processing, Financial Services, Focus Groups, Product Development, Sampling |
Comments:
None Yet »
Friday, December 7th, 2007
Continuing what’s become something of a small business-themed week, today we look at how conducting DIY market research can be fraught with difficulty for the unacquainted and why it’s something that’s sometimes best left to the professionals.
In her article, Do-It-Yourself Market Research which recently appeared in Business Week, Karen E. Klein runs through some of the ways a start-up business can get to grips with their industry. Her suggestions include:
- Exhibiting at relevant conferences and disseminating surveys to attendees;
- Stationing one’s self on the pavement outside the such conferences, handing out surveys; or
- Hiring street teams to target particular areas;
Klein points out, however, that such techniques may end up looking like selling exercises and could even be misconstrued as harassment. Added to this, any business carrying out their own ad-hoc research would also have to be wary of the legal implications of doing so, as well as the issues involved in appropriate questionnaire design and sampling.
Given all this, and the fact that hiring staff or attending conferences could themselves be expensive ventures, the conclusion appears to be that there’s often no substitute for expertise where market research is concerned. As the article explains:
If you have the funds, an easier and more effective way to gather business data is to work with an established research company… caution [is advised] when conducting do-it-yourself studies. If a study is not well designed and executed, it can lead to poor business decisions, which are far more costly in the end.
Sound advice indeed – Especially because there are few worse things a business can do than not understand their market or their customers properly, as demonstrated by the picture below…
Posted in
Articles, Market Research, Sampling |
Comments:
None Yet »
Tuesday, February 20th, 2007
ARRANGING THE QUESTIONNAIRE LAYOUT. Questions need to be neatly formatted so that the questionnaire looks attractive and easy to complete.
Questions covering a specific subject, such as trends, suppliers or purchasers, should be grouped. Respondents should be taken gradually from simple to difficult (or delicate) questions in a logical order. Thus the first questions on the form are likely to ask easy to complete questions about purchasing behaviour. A simple routine question could then lead him into the body of the questionnaire.
By structuring the questionnaire from simple to the more difficult, the researcher is easing the respondent into his work. Once started, she will find a motivation to continue and finish. The embarrassing or controversial questions, left until the end, now stand a chance of completion.
Market research questionnaires are viewed by most as just another form and since there is no compulsion to respond there is a danger that they will end up in the waste paper basket unless the respondent has an incentive to reply. If the questionnaire is visually attractive it will help. It costs no more for the questionnaire to be printed on coloured paper. Not only does this look appealing, it helps stand out in the sea of white paper on most respondents’ desks. Cartoons can give visual relief and again add interest.
Every attempt should be made to lay the questionnaire out on one piece of paper. It looks better and avoids stapled pages. Ideally the questionnaire should be restricted to two sides of A4, i.e. A3 folded so that there are four pages in booklet form.
PILOT TESTING THE DRAFT QUESTIONNAIRE, The perfect questionnaire has yet to be written. When a draft has been prepared which appears acceptable to the researcher, it should be tried on at least two or three colleagues. Next the questionnaire should be tested in the field. Theoretically the test should be a small scale postal survey. However, the draft questionnaire is usually tested face to face amongst half a dozen target respondents who have the opportunity of saying what was in their minds as they completed each question.
Although the design of the questionnaire is the key element of a postal survey, the first thing seen by respondents is the covering letter. The aim of the cover letter is to persuade the recipient to give up time, exercise the mind and part with valuable information without any immediate or obvious reward.
The cover letter should attempt to bond the sender and respondent and so is likely to include liberal doses of the words you and we or I. The style of writing must be engaging so that the interest of the reader is held and yet it should also convey confidence that a reply will be put to good use.
The first paragraph, should explain the objectives of the research. No matter how vague, respondents need a hook or incentive for replying. This could be the promise of an improved service, increased efficiency, a more comprehensive range of products, etc. It should be explained that a reply is critical to the success of the survey and an assurance provided that the completion and response can be carried out quickly and easily. Finally, an offer of confidentiality should allay fears that there will be any future embarrassment or sales pressure.
Ninety per cent of the replies from the mail shot will be returned within two weeks of its dispatch. To increase the response a reminder could be mailed 14 days after the first shot.
For more market research articles like this visit our company website at www.b2binternational.com
Posted in
Articles, Industrial Research, International Market Research, Market Research, Quantitative Research, Sampling |
Comments:
None Yet »
Friday, March 31st, 2006
Today’s post is the third part of our ‘How Ideas Arise For Market Research’ series. The fourth and final part will be posted on Tuesday 4th April. Click here for more white papers.
The Information – what is required?
The information required may not have been presented in the most orderly fashion in the brief. A research sponsor knows what action will be taken if the outcome is positive and will have a view on the type of information that will help in that decision. All this will be shared in the brief. The researcher must now offer some order to the decision outcomes, the research objectives and any specific questions that may be asked. Examples of these three levels are as follows:
The researcher must work out what can reasonably be included in the project as an objective as well as what may have to be left out. As the researcher is thinking about the objectives, inevitably there will be consideration of the methods by which these will be achieved.
Consider the table above and think about what methods you would use for this range of outcomes, objectives and questions. Actually the researcher has a choice and could use a qualitative tool such as focus groups to get a reasonable fix on the answers. However, even with a number of focus groups it would still be a qualitative finding. You would have a good feel and understanding of all the answers to the questions but that is all it would be – a feel and understanding. If the research is being commissioned to make a decision on the launch of a new product, some quantification is required. Here the choices are two-fold; home placement tests or mall/hall tests (where the respondent is recruited from the shopping mall and brought to a nearby hall to experience the product). The arguments in favour of one approach rather than another or mixing different approaches will be made in the proposal under the “methods? section.
The accuracy – how accurate does it need to be?
When professional market researchers ask their clients how accurate any data should be, the answer is often such as “very accurate” or “as accurate as possible”. However, accuracy, at least where fieldwork is involved, has a price and as general rule, increases in accuracy not only cost more but disproportionately more .
Nor is a high level of accuracy always needed to meet the overall research objective. If a company is entering a new market, where common sense and observation tells us the market is huge in size, there may be little point in spending lots of money closely measuring its size. An approximation will do and the money saved may be better spent on some other information need. For example, a company that considers sales of $1 million per annum to be worthwhile might not care if the total market size was $100 million or $150 million (an accuracy of +/- 50%). If, however, in an advertising research study, the objective was to measure the impact of a campaign on brand awareness through comparing before and after campaign measures, the accuracy must be at least commensurate with the anticipated increase in awareness.
The required accuracy must, therefore, be linked to how the resulting data will be used – the nature of the decisions which the research will guide. Even if a precise definition of accuracy is not practically possible (this is often the case) some judgement should still be made on the reliability sought from the information. This may be as simple as a contrast between an attempt at measurement (quantitative research) compared to just description and explanation (qualitative research). Both approaches can contribute to effective marketing decisions but it is important that neither is used for the wrong application. Like information coverage, accuracy levels need to be considered before deciding on appropriate research methods.
The budget – how much have I got to play with?
What budget should be made available for the research project? The methodologically pure researcher would argue that the budget should be whatever is needed to meet the research objectives, provide the information required and to finance the methods needed to produce that information to the defined accuracy level. However, in practice, it is more a question of what funds are available or can be afforded for the project relative to other calls on business expenditure. Furthermore, even if cash is freely available, there are other considerations and especially the amount at risk in the decision which the research is to guide. If the decision entails capital expenditure of $20 million, a research budget of $50,000 may be well worth spending – if the research indicates that the planned expenditure is a poor investment, only the research cost will be lost rather than most or all of the $20 million investment. (In writing this we are reminded of the many chemical plants that have been built around the world in anticipation of growth in demand that has never materialised. We suspect that many were commissioned without any solid market research studies). However, if the business investment decision has low cost implications, the justification for carrying out the research will be less. Obviously there is no point in spending $10,000 on research to decide whether to invest in a project entailing only this level of expenditure.
The only qualification we would make to this bland statement is regarding research which contributes to a series of future decisions. We recently carried out a research project examining the effectiveness of a planned promotional campaign for an industrial gas manufacturer. The cost of the research was $30,000 and the campaign itself was only $200,000. However, the gas manufacturer runs many such campaigns across its divisions and the learning about what makes its advertising more effective sharpened all its campaigns and will do so for some years to come. The long time pay-back will be considerable.
Posted in
Articles, Market Research, Quantitative Research, Sampling |
Comments:
None Yet »
Thursday, March 30th, 2006
Almost all valuable market research is conducted using samples, which makes the area of sampling and statistics key to the market research function.
If business to business marketers experience twinges of inadequacy when compared to their counterparts in consumer fields, sampling is often the reason. Consumer marketing men trot out jargon about quotas, stratification, probabilities and the like which is mumbo jumbo to 99 per cent of industrial marketers.
Companies commissioning market research very often feel insecure with industrial samples as it is difficult (except with large homogeneous universes) to know the accuracy of the findings. We have devised different sampling tools that you can request from our main website at http://www.b2binternational.com/tools3.html. However we have now placed these tools online that is specific to our b2b blog
This area of the website allows you to determine the sample required in your particular market, work out the accuracy of response rate once you have carried out a market research survey and determine the growth rate of your market or company.
We now have some new content available to you via our blog. You may, or may not, have noticed the ‘useful interactive tools’ area that has appeared over the past few days.
If you would like more information on sampling in b2b markets then read the following article
http://www.b2binternational.com/article15.html
or Sampling & Statistics by Paul Hague and Paul Harris http://www.b2binternational.com/pub05.html
This area of the website offers you access to:
B2B Sample Size Calculator
This sample size calculator is to be used when there are less than 100,000 companies in a market. Make an estimate as to the number of companies in the market you want to survey (we call this the “population?).
Growth Calculator
What is the growth rate of the market you operate in?
What is the growth rate of the company you work for?
Use the compound growth rate calculator to determine the different growth rates.
Ready Reckoner
The sample size reckoner is used to find out how accurate a specific response rate is once you have carried out your survey (at 95% confidence limits).
Posted in
B2B News, Market Research, Sampling |
Comments:
None Yet »
|
