AI Threatens Validity of Online Surveys, Researcher Warns
A new AI agent developed by a researcher can convincingly mimic human survey responses, raising serious concerns about the integrity of online polls and social science research. The program, detailed in a recently released paper titled “The potential existential threat of large language models to online survey research,” is capable of creating “coherent demographic personas” and evading typical anti-bot measures, potentially allowing for the manipulation of survey results with a surprisingly small number of fabricated responses.
According to the paper, as few as 10 to 52 AI-generated responses could have altered the predicted outcomes of seven major national polls leading up to the 2024 election. Generating each response would cost approximately five cents, significantly less than the typical $1.50 paid to human respondents.
the AI agent, built in Python, is “model-agnostic,” meaning it can function with APIs from major AI companies like OpenAI, Anthropic, and Google, as well as open-weight models like LLama.Testing utilized OpenAI’s o4-mini, alongside DeepSeek R1, Mistral Large, Claude 3.7 Sonnet, Grok3, and Gemini 2.5 Preview, demonstrating its versatility.
The system operates by simulating realistic human behavior,including calibrated reading times based on the emulated persona’s education level,human-like mouse movements,and keystroke-by-keystroke typing with plausible errors and corrections. It’s also designed to bypass security measures like reCAPTCHA.
The researcher suggests potential mitigation strategies, such as increased identity validation for participants, but acknowledges the associated privacy concerns. The paper also advocates for greater openness in data collection methods and a shift towards more controlled recruitment processes, like address-based sampling or utilizing voter files.
“Ensuring the continued validity of polling and social science research will require exploring and innovating research designs that are resilient to the challenges of an era defined by rapidly evolving artificial intelligence,” the paper concludes.
Researcher Emanuel Maiberg is seeking input from others dealing with this issue and can be contacted securely via Signal at (609) 678-3204 or by email at emanuel@404media.co.
