Services Checking AI synthetic users
AI synthetic users, checked against real people
Whether you use synthetic users, build them or train the models behind them, we put your questions to real people in qualitative conversations, and show you where the synthetic answers hold, where they do not, and why.
Prefer to write? Start the conversation.
60+ countries · 400+ projects delivered · 50+ global clients
MindMarket checks AI synthetic users against real people in qualitative research. We ask real people the questions a team asked its synthetic users, in the markets you choose across 60+ countries. Moderators who live there run the conversations and follow up to understand why and how people answer as they do. Brands check before a launch; companies that build synthetic users check to calibrate them.
Who it is for
Who asks us to check synthetic users?
-
Brands and their insights teams.
You test ideas with synthetic users and have a launch or a big decision ahead. We put your questions to real people in the markets you are launching to, and show you where the synthetic answers hold, where they do not, and why, as we did for a fintech in Seoul.
-
Companies that build synthetic users.
You build synthetic users. We put the questions they answer to real people in the markets you sell into, at the rhythm we agree on the call. You calibrate your synthetic users on what real people said, and you can say they have been checked against real people.
-
AI companies.
You train or run the model itself. We put the questions it answers to real people in the countries and languages it knows least, and show you why and how their answers differ, so your team has real conversations to calibrate against.
The question
Do your synthetic users sound like the people they stand for?
More and more companies test their ideas with synthetic users, sometimes called synthetic personas: customers generated by AI. They are quick to run and useful for exploring early, and we use AI ourselves where it helps, in transcription, translation drafts and first-pass coding.
Before a launch, the question changes: does your synthetic user say what the people in your market would say? The one way to find out is to ask them.
So that is what we do. We ask real people the same questions, in the markets you choose, and show you where your synthetic users were right, where they were wrong, and why. Your team keeps what held up and fixes what did not, before it costs you a launch.
How we work
How do you check synthetic users against real people?
-
You share your questions and what your synthetic users answered.
It starts with a 15-minute call. We look at what your synthetic users told you, and agree the markets and the questions to check. Send them in whatever form they exist: the guide your team used, the answers that came back, a deck or a document. We do not need access to your tool, only what it produced.
-
We ask real people the same questions.
The questions stay word for word, in the language people speak at home, and researchers who live in the market moderate the conversations. These are in-depth conversations with real people. The moderator then follows up to understand why people answer as they do, and how they would use what you are testing. We use in-depth interviews by default, and focus groups when they suit the question better.
-
We compare the answers, question by question.
We set what your synthetic users answered next to what real people answered, one question at a time, and note whether the two agree: yes, in part or no. Where they differ, we go back to the conversations to find what explains the gap, in people's own words. You see where your synthetic users matched real people, and where they did not.
-
You get a written comparison and a debrief call.
The written comparison is short. It sets out what to change before launch, and which answers held up, so your team knows where it can keep relying on its synthetic users in that market, for now. On the debrief call, the team that ran the conversations takes you through it and answers your questions.
Qualitative
Why is the check qualitative?
Because a synthetic user gets the what wrong less often than the why. Only a conversation with a real person tells you why people answer as they do, and how they would live with your idea: in their family, their budget, their day.
A survey can count how many people say yes. The why and the how, the two things a synthetic user can only estimate, come out when a moderator asks a second question, and a third.
Before a launch
What can checking your synthetic users change before a launch?
Think of a check as insurance for your launch: a small study before you commit the budget, so the decision rests on what real people told you, not only on what a model predicted. Sometimes it confirms the plan, and you launch with more confidence. Sometimes it finds the one thing the synthetic users could not know.
Fintech · in-depth interviews
An example from Seoul
A fintech client had tested its new app with synthetic users, who showed strong interest among young professionals. The client wanted to challenge those results with real people through a small qualitative study. Our researchers in Seoul ran in-depth interviews. People liked the app, but deep concerns about financial privacy, rooted in how money is managed within the family, stopped them from adopting it. That finding became the basis for a complete overhaul of the client's messaging.
What the synthetic users said
Strong interest in the new app among young professionals.
What real people said
They liked the app, but concerns about financial privacy within the family stopped them from adopting it.
What changed
A complete overhaul of the client's messaging.
The deliverable
What do you get at the end of a check?
- A short written comparison, question by question: what your synthetic users answered, what real people answered, whether the two agree (yes, in part, no) and why.
- A few sentences from participants, anonymised, in their own words, so your team hears how people in the market talk about your idea.
- A one-page conclusion: what to change before the launch, and where your team can keep relying on its synthetic users in that market, for now.
- A debrief call with the team that ran the conversations, to go through the comparison and what it means for your launch.
Every six months
Why check your synthetic users again every six months?
People change. Family life, society and the wider context move, and the answers move with them. Even if your synthetic users were right six months ago, that may no longer be true today. So we ask the same questions in the same markets again, and show you what has moved since the last check.
Synthetic Users, one of the companies that build them, makes a similar point in a piece on modelling public opinion:
“Static calibration degrades.”
“Synthetic Users without calibration are individually believable, but collectively wrong.”
There is a second reason: the AI behind your synthetic users can change too. A 2024 study in Nature found that when AI models are trained indiscriminately on content generated by other models, the rare cases in the original data disappear. The study is about how AI models are trained, not about any synthetic user product, and its authors expect data collected about genuine human interactions to become increasingly valuable. A regular check with real people tells you whether your synthetic users still sound like the people they stand for.
Two ways
How can we work together?
-
One check
We run a single check before a launch or a big decision, in one market or several. It works like a First Listen: a focused study with real conversations and a debrief call.
-
The twelve-month check
We run two checks a year for twelve months, with the same questions in the same markets, so each check can be compared with the last. You can add a check before a launch, or when your synthetic users change, at the same rate.
Frequently asked questions
Questions about checking synthetic users, answered straight
Should we stop using synthetic users?
No, you do not need to stop using synthetic users: they are a quick way to explore early, and the companies that build them say the same about their limits. Synthetic Users writes that a calibrated synthetic population “should never replace direct consultation with the communities it claims to represent.” Keep using them for the early work, and check them against real people before the decisions that cost the most.
Do you need access to our synthetic user tool?
No, we do not need access to your synthetic user tool: we work from the questions you asked and the answers it gave. Share them in whatever form they exist: the guide your team used, the answers that came back, a deck or a document. We keep the questions word for word so that the conversations can be compared, and we tell you if one needs rewording to make sense to a person in that market.
Our synthetic users were a focus group. Can you check that too?
Yes, we can check synthetic users that were run as a focus group. Some tools run a group of synthetic personas that react to one another. We put the same questions to a real focus group in your market, moderated by a researcher who lives there, and compare what the two groups said. Where the subject is personal, such as money or health, we may suggest in-depth interviews instead, and we tell you why on the call.
Which markets can you check synthetic users in?
Any of the 60+ countries where MindMarket works, with researchers who live in the market and moderate in the language people speak at home. You choose the markets, and a check can cover one or several. See the countries and cities where we run fieldwork.
How long does a check take?
How long a check takes depends on your markets and the method, and we confirm the timing on the 15-minute call, before the quote. Here is what happens in that time. You send us your questions and what your synthetic users answered. We recruit participants who match the people your synthetic users stand for, and run the conversations. We compare the answers, write the comparison and go through it with you on a debrief call. The twelve-month check does this twice a year. Read how we recruit participants for a study like this.
What if our synthetic users turn out to be right?
Then you launch knowing the answers held up with real people. A check that confirms the answers is as useful as one that overturns them: your team knows where it can keep relying on its synthetic users in that market, and for which kind of question. The written comparison shows which answers held up and which did not, question by question, so the next decision starts from what the synthetic users got right.
Can synthetic users drift over time?
Yes, synthetic users can drift over time. In machine learning, drift means a model doing its job less well because the data, or what the data means, has changed since it was built. People's lives move too, so synthetic users that matched real people six months ago may not match them today. We do not measure a model's performance or give it a score: we ask real people the same questions again and show you what has moved.
How much does it cost to check synthetic users with real people?
Each check is priced per market, and you receive an itemised quote within 48 hours of a 15-minute call. The price depends on the number of markets, the number of questions and the method, in-depth interviews by default or focus groups when they suit the question. On the call we look at what your synthetic users told you, agree the markets and the questions, and tell you how we would check them. The quote then sets out each market line by line, with no commitment until you confirm. Book the 15-minute call or write to us.
Already using synthetic users?
Tell us what they told you, and in a 15-minute call we will show you how we would check it.
60+ countries · 400+ projects delivered · 50+ global clients