Why We Still Didn’t Trust the Data That Passed Every Check

B2B brand trackers are a core part of what we do, and many of these trackers have been running for years. The core questionnaires tend to remain the same, the methodology is proven, and we apply a robust set of quality controls tailored to each study throughout fieldwork. Completion times are checked, locations are verified, open-ended responses are reviewed, and respondents are screened using specialist fraud detection tools.

For a long time, that worked well.

Recently, however, we encountered a situation where something didn’t feel right. The first warning sign was a significant increase in spontaneous brand awareness in one market. On paper, it looked like a positive result. The problem was that it simply didn’t make sense. When we compared it with previous waves and other markets, there was no obvious commercial, competitive, or marketing explanation for such a substantial shift.

As we looked more closely, other things started to stand out. Open-ended responses were unusually polished, respondents consistently referred to brands by their full names, and there were notable similarities in the structure and style of answers. None of these indicators would necessarily be enough to raise concerns on their own. Taken together, however, they suggested something wasn’t quite right.

What made the situation particularly interesting was that the data had already passed our standard quality checks, including:

• Completion time checks
• Attention and consistency questions
• Straight-lining and flatlining analysis
• Location verification
• Brand mention reviews
• Specialist fraud detection tools

By traditional measures, the dataset looked good. Yet despite this, we remained uncomfortable with what we were seeing. Assessing data quality isn’t simply about applying a set of predetermined rules. It requires judgement, context, and an understanding of what constitutes a credible result. Sometimes the most important question a researcher can ask is simply: does this really make sense?

Why This Matters in B2B Research

This challenge feels particularly relevant within B2B research. Unlike many consumer studies, we are often trying to reach highly specialized audiences drawn from relatively small populations.

These same audiences also create additional challenges. Higher incentives, smaller target populations, and increasingly sophisticated technology mean that poor-quality responses are becoming harder to identify.

Historically, researchers were often looking for respondents who rushed through surveys, failed attention checks, or submitted nonsensical answers. Increasingly, the challenge is the opposite. Some responses appear highly credible, well written, and thoughtfully constructed. In some cases, they look exactly like the sort of responses we might hope to receive from an engaged B2B respondent.

As researchers, we naturally launched a detailed investigation. We spoke with fieldwork partners, reviewed sourcing routes, and recontacted suspicious respondents. This did not point to any single supplier, panel provider, or recruitment source. Instead, it highlighted another important lesson about data quality: warning signs rarely appear in isolation.

A significant movement in a key metric can, of course, be genuine. Similar language patterns may occur naturally. An individual respondent may simply be more articulate than average. Concerns tend to emerge when multiple signals align to create a pattern that warrants closer scrutiny.

Some of the signs we are now paying closer attention to include:

• Significant changes in key metrics without an obvious explanation
• Respondents consistently mentioning the same combinations of brands
• Blocks of responses within a market or audience profile
• Open-ended responses that follow a very similar structure or style
• Answers that seem unusually perfect relative to the audience being surveyed
• Responses that feel more like prepared copy than a genuine point of view

How Our Approach Is Evolving

As a result, our approach to quality assurance continues to evolve. Alongside traditional checks, we are increasingly examining behavioral indicators that help us understand how responses are being generated. These include:

• How long it takes a respondent to begin typing
• Time spent answering individual questions
• Whether content appears to have been copy and pasted into the survey
• Typing patterns and pauses
• Whether responses appear naturally generated or entered all at once

None of these indicators provides definitive proof on its own. However, when combined with traditional quality controls and a broader review of the data, they can provide a much more complete picture of respondent behavior.

Human Judgement Still Matters

Perhaps the biggest lesson from this experience is that automated checks alone are unlikely to solve the problem.

New tools will undoubtedly play an important role as AI continues to evolve and the B2B panel market becomes more complex. However, the original concern in this project did not arise because a respondent failed a quality check or because an algorithm highlighted an issue. It arose because an experienced researcher looked at the aggregated findings and questioned whether they were believable.

Recognizing that required knowledge of the category, familiarity with previous waves of data, and an understanding of how brands typically perform across different markets. In short, it required judgement.

The reality is that data quality challenges are changing, and our approaches need to change with them. Better technology, smarter quality controls, and more sophisticated methods of respondent validation will all play an important role.

This experience suggests that the most valuable quality check may still be the simplest one: an experienced researcher who is prepared to challenge the findings, even when the data appears to have passed every test.