AI Can Guess the Average. Your Team Isn't One
A good guess always gets the average right and the edges wrong. The people you most need to hear from live at the edges, so ask them.
Not one of the AI respondents slept four hours a night or less. Not one slept ten hours or more. In the real survey, nearly 10% of Americans do.
That small detail, from a Pew Research study published this week, stayed with me longer than the headline number. Because it describes something I have seen in teams for years, just without the AI.
Pew asked AI to pretend to be people
Pollsters have been curious about "silicon samples" for a while: instead of surveying humans, you give a model a persona and let it answer for them. Cheaper, faster, no one ignores your email.
So Pew tested it properly. They built digital twins of real members of their American Trends Panel, fed the model each person's demographics and earlier answers, and asked it the same questions the humans got. Nearly 300 of them, mostly with Claude Opus 4.6 and a comparison run on GPT-5.1.
The result: the AI missed real opinion by about 12 percentage points on average. Around 28% of questions were off by more than 15 points. The AI also almost never said "not sure", about 4% of the time against 16% for humans. And in 47% of the questions, at least one answer option got zero AI votes. In the human survey, that never happened once.
Pew's own conclusion is plain: AI polling is not a replacement for surveying real humans.
The average person doesn't exist
Here is what I find interesting. The model wasn't random. It was confidently typical. It took a group and gave you the most likely version of that group, again and again, until the edges disappeared.
And managers do the same thing without any model.
I have caught myself doing it. You know your team, you know the person's role, their seniority, how they usually react. So in your head you already "know" what they think about the reorg, the new on-call rotation, the deadline. You skip the question because the answer feels obvious.
That is a digital twin. A very cheap one, built from your assumptions.
The people you most need to hear from are the ones a good guess will always get wrong.
The engineer who is quietly burning out does not look like the average engineer. The user who churns does not answer like the typical user. The "not sure" in a retro is often the most honest answer in the room, and it is exactly the answer a confident summary removes.
Now add AI to that habit. It is very tempting to paste a few Slack threads into a chat and ask "how is the team feeling about this?" You will get a clean, reasonable answer. It will sound right. It will probably be middle-of-the-road. And the person who sleeps four hours a night will not be in it.
Where this goes
I don't think teams will stop using AI to summarize feedback, and they shouldn't. Reading 200 survey comments is a real cost. But I expect a split: people who use AI to organize what humans said, and people who use AI to imagine what humans would say. The first group will get faster. The second group will get surprised.
What I'd actually do
- Ask before you predict. Before a big decision, write down what you think each person will say. Then ask them. The gap is your blind spot, and it is useful data.
- Protect "I don't know". In surveys and 1:1s, keep a real "not sure" option and treat it as a signal, not a non-answer.
- Use AI on real input only. Summarizing actual comments, yes. Generating "likely employee sentiment", no.
- Look for the zero. When you review feedback, ask which answer nobody gave. Then ask yourself if nobody feels that way, or if nobody felt safe saying it.
- Talk to the edges. Once a month, have a conversation with the person you understand least on your team. That one hour beats any persona.
A model can tell you what someone like your colleague would probably say. Only your colleague can tell you what they actually think. The second one is still the job.
Sources: Pew Research Center: Can AI stand in for human survey takers? Not really, Pew Research Center: AI surveys fail to capture the diversity of public opinion, Finger Lakes 1: Pew test finds AI survey stand-ins miss public opinion by 12 points
