As someone who has been in the market research industry for decades, I’ve witnessed firsthand the dramatic evolution of data collection processes. In other words, the data collection methods in quantitative research projects have evolved a lot.
And while it’s great to reach more consumers more quickly, there are also drawbacks to weigh when exploring the movement toward more agility. Let’s dive into the history and get back to current data collection processes in no time. In this article, I discuss:
- The early days of data collection
- Dawn of online surveys
- The Global reach of online surveys
- Niche targeting
- The problems with automation
- Future of data collection processes
The Early Days: Phone and In-Person Surveying
Back in the 1990s, when I first started out in the MR industry, telephone surveys were the most common form of data collection. Companies would cold-call people at home, often during dinner, to see if they would participate in a study. It could feel intrusive, interrupting family time, but it was more inclusive and allowed for broader reach across geographical boundaries compared to in-person methods. Perhaps the most important reason that this method became the dominant way to collect data is that it could produce a representative sample of the market. Virtually every household had a phone, so you could potentially survey the whole country.
In order to generate sample we relied on companies that specialized in phone number list generation. This allowed us to select either a targeted list or to rely on a randomly selected list of phone numbers. Phone sample providers, like SSI, would provide us with Random Digit Dial (RDD) sample so that phone surveys could include all phone numbers, even those that were not listed in the phonebook.
The rest of the quantitative work back then was done through written surveys conducted at central locations. With clipboard in hand, researchers would intercept people coming out of shopping malls or grocery stores and ask them to fill out a paper questionnaire on the spot. After collecting hundreds of these forms, all that data then had to be manually entered into computers with monochrome screens – an enormously tedious process! After you looked at those computers for too long, your eyes would glow.
The Dawn of Online Surveys
In the late 90s, the internet started entering households, and we began exploring online surveys, though initially it was technically complex and expensive. Companies hired web developers to custom-build online questionnaires for tens of thousands of dollars. I was at one company that helped pioneer the first online survey patents and platforms. However, the transition from phone surveys to online happened rapidly over about five years in the early 2000s as survey software became more sophisticated and accessible.
This shift to automation was a game changer. No more manual data entry, as responses were submitted directly online. We could check results quickly rather than waiting weeks to see the data after fielding was complete. The data came in instantly as people completed the web questionnaire. Everything that used to be extremely time-consuming and manual was now automated.
Before programmed surveys, companies would even just email their surveys out to groups and capture the responses via email and still have to manually code them. It was a still-manual step before a more automated process.
The Global Reach of Online Surveys
Online surveys also provided much easier access to larger and more diverse audiences across geographical limits. As global internet and smartphone adoption grew rapidly in the 2010s, survey respondent panels expanded internationally, reaching emerging markets.
Previously, research in developing countries relied on face-to-face intercept surveys on city streets targeting urban consumers. With the influx of mobile technology, we can now easily and affordably gather data from really all parts of the world.
Specialized Niche Targeting
Today, niche targets across industries, languages, and interests can be accessed online – even rural populations with just mobile phones. In emerging markets, many consumers went straight to smartphones without ever owning PCs. The barriers to conducting market research are far lower compared to when I started out. The technology has made it easier, but there are also tradeoffs that are worth considering.
The Automation Tradeoff
However, this automation has trade-offs. With analog methods like phone or in-person, we could verify a real person was participating. We would listen in remotely to phone interviews and recordings for quality checks. With online surveys, there’s no human interaction or accountability. Results could be skewed by bots, click farms using VPNs to disguise location, or respondents just motivated by incentives.
Read next: How to Ensure Data Quality in Research
So while technology is crucial, the human element in data quality checks remains indispensable. For every new data collection method, you need to validate that the data is trustworthy. We rely on technology to detect fraudulent responses. But we also leverage human intuition – things that look questionable to the researcher’s eye might not always be caught by algorithms.
The Future with AI and Automation
Looking ahead, AI will likely continue disrupting the market research process. Facial coding already analyzes emotions from videos to replace human manual coding. Natural language processing transcribes open-ended verbal responses. Some companies are even experimenting with automated interviewing using conversational AI chatbots.
The role of the researcher will shift more to design, interpretation, and strategic analysis as data collection and tabulation become automated. But as someone who’s been in the industry before these technologies existed, I believe maintaining a human lens remains critical for reliable insights even with these futuristic tools.
There are nuances to consumer motivations and opinions that data science has yet to replicate. Every person has distinct perspectives and attitudes molded by individual life experiences. While machines can spot general patterns in data, it takes a researcher’s empathy and intuition to truly uncover deeper human truths.
For now, humans still have an edge over AI when it comes to the art of asking the right questions to reveal unarticulated consumer needs. However if used thoughtfully, automated solutions can unlock faster and richer discoveries from market research data. It’s an exciting time to be in the industry, with new innovations constantly emerging. But by blending automated systems with human oversight, we can conduct higher-quality research and generate strategic recommendations that drive business success.
