As a leading online data collection agency, STRAT7 Researchbods manage and deliver multiple market research studies for research and media agencies across many regions.
It’s been almost a year since we launched our Data Integrity Unit and during that time a lot has happened. We’ve partnered with the Market Research Society to support their Global Data Quality initiative; our Data Integrity Team has expanded, and we’ve run a vast range of projects for new and existing clients, which has helped us build a norms database looking at market level and sample type trends.
The industry is pushing to drive an enhanced quality initiative, but bot completes, duplications and professional respondents are still as prevalent as ever. There’s clearly no silver bullet to tackle the issues and it’s naïve to think Google reCAPTCHA or Double Opting-In respondents will do the trick. Granted they help, but malicious technology has moved way beyond these deterrents.
So what are the best practises? In our experience, and where we’ve had the most success over the past year in identifying and mitigating fraudsters, is by incorporating a framework that leverages tech and human intervention and works at a project/ respondent level.
Our team have developed a range of bespoke question types that challenge bots and screen reading AI. Some of the question types require human interaction that bots simply cannot perform. These include button hold tests, layered image tests, noise and image association tests, and hidden honey pot questions. Quite quickly with these type of questions within the screening section, we can identify bot-like responses failing what a human respondent would easily complete.
Even if you’re not going to use the open-ended data in your analysis, it’s a great tool for us to vet respondents. We can review open responses across various metrics including plausibility, general quality and duplication/ AI risk.
Save your team time and automate standard checks across speeding and straight lining. Within our survey tool environment, we’ve created a syntax that automatically flags speeders and straight liners based on the parameters we set. This allows the team to swiftly remove poor respondents and pay more attention to high level manual checks around plausibility or open responses.
There’s a few solutions available such as Survalidate and Survey Defender and they really do help from a Data checking point of view. We’ve incorporated Survalidate which runs in the background as soon as a respondent enters a survey and looks more at their device behaviour. The tool tracks various parameters such as IP, email score, time of survey completion, location access etc to give a general score indicator across fraud and duplication.
Over the past year our Data Integrity Team have been compiling a norms database which details project metrics at a market and respondent level. This helps us tactically inform clients around certain market trends or nuances which can help inform project setup and areas to pay particular attention to for validation checks.
Our Data Integrity Managers apply these solutions and are solely responsible for validating every single survey respondent at a project level before handing that data over to the client.
From a respondent point of view, each person is given an overall Integrity Score following the checks. Those below a certain threshold are instantly disqualified, others are validated by our Data Integrity Team and those with a high score we’re happy to accept.
On average we see a 32% removal of respondents across standard consumer studies, with higher prevalence in certain markets. This spikes again in certain areas of sample type such as B2B with close to 45% of respondents being removed. Compared to general industry stats, anecdotally removal rates are somewhere around 15-20%.
We have had push back from partners, but we transparently share the reasons for removals, which has helped them better maintain their audiences.
We’re proud to do the heavy lifting for our clients, so they can get back to doing what researchers do best – engaging with clients, critical thinking and delivering impactful insight
