What happens when you replace a thousand real people with a thousand synthetic ones — and how to tell the difference.

There's a tempting shortcut floating around user research right now: instead of recruiting a thousand people to take your survey, ask a large language model to pretend to be a thousand people and answer for them. It's fast, it's cheap, and it never asks to be paid $6.75 for fifteen minutes of its time. Companies are selling "synthetic audiences" built exactly this way.

We were skeptical, but curious. We happened to have something most people testing this idea don't: a real, census-matched survey of a thousand actual chatbot users, collected earlier this year. So we ran an experiment. We took that real survey as our answer key, and then asked six different synthetic "populations" to sit the same exam. Then we graded them.

The short version: synthetic respondents are genuinely useful for some things and quietly, dangerously wrong for others — and which one is which is not at all obvious until you hold the real answers up next to them.

How we ran the study

Our ground truth was the survey from our earlier work: a thousand real people in the U.S., recruited through Prolific, matched to the census on sex, age, and region, all of whom had used a chatbot in the past week.

Then we built the synthetic versions. For each one we generated 1,000 fictional respondents whose demographics — age, gender, race, education, employment, region, politics — were drawn to match the U.S. adult population. Crucially, we used the same random seed for every run, so that synthetic respondent #487 is the same 45-to-54-year-old woman in small-town New Mexico in all six of them. The only things we changed were which AI model was doing the answering, and how we asked.

We crossed three models — Anthropic's Claude, OpenAI's ChatGPT, and Google's Gemini — with two ways of running the interview:

• Stateless: each respondent answers the whole survey in one shot, with no memory.

• Persistent: each respondent is walked through the survey one question at a time, in a single conversation, so its later answers can "see" what it said earlier — the way a real person remembers what they just told you.

Three models times two methods gives six synthetic populations, plus the one real one. Then we compared every answer. There’s a lot of other studies of synthetic users out there, and we replicated a lot of those findings, but we wanted to pull out two stories that can easily get lost in the details.

Finding 1: They're great at what the internet already knows

The first thing that jumped out was that accuracy had almost nothing to do with how hard a question was, and almost everything to do with whether the answer was already written down somewhere on the internet.

Take a single survey question — "which of these did you use in the past week?" — and split the items by how heavily documented they are online. The pattern is stark:

Things the internet tracks closely

Things the internet doesn’t know about

Real figures from our Prolific survey; synthetic figures averaged across the three stateless models.

Smartphone market share is one of the most reported statistics on earth, so the models nail it. But whether you still use a pair of wired headphones, or whether you listened to a podcast sitting next to someone else? Nobody blogs about that. There's no dataset. So the models guess — and they guess badly, systematically erasing the unfashionable and the social.

We measured this across all the device questions and found a clear distinction: for heavily-documented behaviors the models were off by about 6 points on average; for the mundane, undocumented ones, by 33, and that’s driven entirely by what the internet happens to have written down. A synthetic respondent isn't remembering its life. It's interpolating from a corpus, and everyday life is mostly missing from that corpus.

Of course, one solution to this is to deliberately draw from information about a specific user group. This is the sort of thing our friends over at simile.com are doing for a lot of their work, drawing from specific data from users like the ones they’re simulating. But again, that’s necessary, because without the ground truth, synthetic user responses are only as good as the data behind them.

Finding 2: They quietly launder away opinions that might offend.

This is the one that worries us most.

In the real survey, when people told us why they'd stopped using a chatbot, about 1 in 10 of all respondents — and about 1 in 5 of those who'd actually quit using a chatbot — gave a values-based reason: an objection to what the product or its maker stands for, rather than what it does. That covered a lot of ground. Some objected to specific people — Elon Musk, Bill Gates, "their CEOs." Some objected to corporate conduct: one quit ChatGPT "since they signed that defense contract"; another refused "to benefit sociopathic billionaires." Some cited data privacy, some political bias in the answers, and a handful were cutting back over AI's water and energy use. It's a real, meaningful slice of how people actually make decisions.

Then we coded all six thousand synthetic responses the same way. Values-based objections were at 1.1% overall — and the pointed kind, objections to a company's conduct or to a specific person, fell to 0.07%. That's four responses out of six thousand. Not one synthetic respondent mentioned a defense contract, a boycott, a government tie, or the environment. Exactly one, in six thousand, pointed at a human being — and even that one hedged: "Elon's whole thing with it felt off to me, and honestly it didn't seem as useful as ChatGPT or Claude anyway."

The laundering is subtler than simple deletion. The one synthetic population that produced any visible values-based objections (3.5%) got there almost entirely on generic privacy worries — thirty near-identical variations on "I wasn't comfortable with how much data these services collect." Real people's values-based objections are three-quarters about named companies and people, and one-sixth about privacy. The synthetic ones invert that: the model swaps the dangerous objection (naming names) for the safe one (a vague murmur about data). Oh, and nobody synthetic ever quits over price — but 3.4% of real people cited cost; across six thousand synthetic responses, the count was zero.

We don't think this is an accident of "flattening," and we don't think it's fixable with better prompting. It's structural. A real person writing "I stopped using Grok because Elon Musk is a terrible human," as one of our respondents told us, creates no legal risk for the researcher recording it. But a commercial AI generating that same sentence creates potential defamation exposure for the company that made the model — and there's now an evolving body of law that makes that exposure concrete. So these models are trained, sensibly, not to make pointed negative claims about named people and companies. That caution is good risk management. But when you repurpose the model as a survey respondent, its caution becomes your population's silence.

The result is a synthetic panel that looks complete but has quietly deleted the boycotters, the critics, and the ethically motivated — exactly the voices a lot of research exists to find. Worse, it papers over the hole with polite privacy talk, so the aggregate still looks like it has critical texture. And because it's invisible in the averages, you'd never know unless you had the real data sitting next to it.

So, should you use synthetic respondents?

For some things, yes. If you want to pilot a survey instrument, pressure-test a hypothesis, or get a rough sense of how people rank a set of options, synthetic respondents are a fast and legitimately useful first pass. They reproduce broad attitude structures and standard demographics reasonably well. Within the next few years, we suspect it will likely be standard practice to try out any new product, survey, or research study on synthetic users before launching to the real world – at very least to learn what you already know. 

But we'd offer three simple rules:

1. Be wary of politically or ethically charged questions — especially where the honest answer means criticizing a named company or person. Values-driven decisions get laundered out, silently.

2. Don't trust synthetic data on anything the internet or your private dataset doesn't already document. Market share, yes. The texture of everyday life, no.

3. And above all: anchor everything to real people, even a small sample. The more data about real people that informs your work, the stronger it’ll be.

Every single one of these findings was invisible until we had a real survey to check against. Without that answer key, we'd have had six confident, plausible, fluent synthetic populations — and no way to know which, if any, was telling us the truth.

That's the quiet danger of synthetic respondents. They're never uncertain. They'll answer any question you ask, a thousand times over, instantly. The work is remembering that a fluent answer and a true one are not the same thing — and that some of the most important truths are exactly the ones these systems are built not to say.

The code and the six synthetic datasets from this study are available at https://github.com/jofish-inf/synthetic-survey-respondents. The survey was hosted on Surveymonkey, and Prolific was used to recruit and reward participants. Claude (Anthropic) was used to assist with data analysis, first-pass qualitative coding of open-ended responses, and drafting.  More detail on this study is available in our research report, Where Synthetic Users Fail.