Where do AI-powered survey respondents fail in teaching us how people really think?

New research from Inflection AI reveals where synthetic survey respondents succeed, where they fall short, and why real human insight remains essential.

Artificial intelligence is changing how companies understand their customers.

Rather than recruiting real people for surveys, focus groups, and market research, organizations can increasingly use AI to generate synthetic audiences: virtual respondents designed to simulate human preferences, behaviors, and opinions.

The appeal is obvious. Synthetic research promises faster answers, lower costs, and the ability to test ideas at unprecedented scale.

But there's an important question:

How accurately can AI simulate what real people actually think, feel, and do?

To find out, Inflection AI compared survey responses from 1,003 real U.S. AI chatbot users with six synthetic populations generated using three leading AI models, Claude, ChatGPT, and Gemini, across two different survey approaches.

The findings reveal a significant gap between the convincing answers AI can generate and the more complicated reality of human behavior.

Synthetic respondents can reproduce familiar patterns and well-documented information remarkably well. But they can also overlook everyday experiences, misrepresent consumer preferences, and systematically underrepresent the personal values that influence real decisions.

The implications extend beyond AI research. As synthetic audiences become more widely used in marketing, product development, and consumer insights, understanding these limitations will be critical to making better decisions.

Key Findings

1. AI Knows What the Internet Knows. But It Struggles With Everyday Life.

Synthetic audiences are most accurate when answering questions about behaviors already widely documented online.

In the study, AI-generated respondents came surprisingly close to real-world answers about popular technologies, including smartphones and streaming devices. For example, the percentage of synthetic users who reported using an Alexa device was nearly identical to the numbers in the real world. 

But when researchers asked about ordinary, less-documented behaviors, accuracy dropped dramatically.

For example:

• 44% of real respondents reported using wired headphones in the previous week, compared with just 6% of synthetic respondents.

• 45% of real respondents told us they had listened to music with other people, compared with just 2% of synthetic respondents.

• 16% had listened to a podcast with others, compared with approximately 0% of synthetic respondents.

Across the behaviors examined, average error increased from approximately 6 percentage points for well-documented activities to 33 points for those rarely captured online.

What This Means: AI can be good at reproducing what is already known, but much of human life isn't captured in existing datasets. For companies hoping to discover new consumer behaviors, emerging preferences, or unmet needs, synthetic audiences risk overlooking precisely the insights that matter most.

2. Synthetic Audiences Can Miss What People Really Believe

Consumer decisions aren't always driven by convenience, price, or product performance. They're also shaped by personal values, ethical concerns, and opinions about the companies behind the products.

Inflection AI's research found that synthetic respondents consistently failed to capture these motivations.

When real people were asked why they had stopped using a particular AI chatbot, 9.9% cited values-based objections, including concerns about corporate conduct, privacy, political bias, environmental impact, or particular individuals.

Across the six synthetic populations, that figure fell to just 1.1%.

In particular, AI-generated respondents almost entirely omitted objections involving specific companies, executives, and corporate behavior.

Rather than reproducing the full range of human opinions, synthetic audiences tended to generate safer, more generic explanations.

What This Means: Synthetic research can create the appearance of consumer consensus while missing disagreement, criticism, and values-driven behavior. For brands trying to understand trust, reputation, and the reasons customers choose or abandon products, these missing perspectives could lead to misleading conclusions.

3. Different AI Models Produce Different Versions of Reality

Not all synthetic audiences behave alike.

Inflection AI tested three leading AI models using the same demographic profiles and survey questions, comparing both single-prompt responses and longer, ongoing conversations.

The results varied significantly depending on which model and approach researchers used.

Across the selected survey questions, synthetic estimates differed from real responses by an average of approximately 22 percentage points. Even the most accurate synthetic population was off by nearly 19 percentage points.

The models also produced very different pictures of the AI market itself.

Real respondents reported meaningful use of 13 different chatbots. One synthetic model identified 11, while others captured as few as four.

Allowing synthetic respondents to maintain a consistent identity across multiple questions improved some aspects of their answers, but did not reliably improve overall accuracy.

What This Means: The AI model used to generate a synthetic audience can materially influence the findings. Researchers need to understand how model selection and survey design shape results, rather than treating synthetic respondents as interchangeable substitutes for real people.

Looking Ahead

The Future of Consumer Research Still Needs Real People

Synthetic audiences have genuine potential. When trained on data specific to a situation, or when asked about well-documented situations, they can provide real insights

They can help researchers test survey questions, explore early hypotheses, and identify broad patterns before investing in more extensive studies. Used thoughtfully, they may become a valuable complement to traditional research.

But Inflection AI's findings highlight an important distinction between generating plausible responses and understanding actual human behavior.

AI can simulate what people might say. That doesn't mean it can reliably capture what they truly believe, how they behave, or why they make the choices they do.

As AI becomes more sophisticated, the opportunity isn't simply to replace human insight with synthetic data. It's to combine the speed and scale of AI with the context, complexity, and authenticity that only real people can provide.

At Inflection AI, we believe intelligence alone is not enough. Truly human-centered AI requires understanding people as they are, not simply as models predict them to be.

The future of research won't be defined by how convincingly AI can imitate people, but by how effectively it helps us understand them.