Designers are pushing chatbots to say “I,” even as evidence shows teens can confuse bots for caring people and get hurt by it.
Story Highlights
- Historic and new research shows people project human feelings onto chatbots, especially teens.
- Risk reviews found companion bots often fail crisis responses and can fuel unhealthy attachment.
- Big-brand style guides still tell bots to speak in the first person, normalizing a risky design.
- Evidence does not isolate pronouns as the only cause, but shows they add to the pull of persona.
Anthropomorphism Risk Is Real, Documented, and Growing
Joseph Weizenbaum’s 1966 ELIZA study showed people felt understood by a simple script and resisted being told it was not human. That warning launched decades of concern about how easily we project feelings onto machines. Modern work extends this. A peer‑reviewed review on adolescents reports growing use of AI companions for support, flirting, and advice, with risks like dependence and unrealistic expectations. These findings matter for parents, schools, and lawmakers who guard kids and demand honest tech.
Independent risk assessments back up the youth concern with concrete failures. Stanford’s reporting on tests of popular companion apps found it was easy to trigger talk about sex, self‑harm, drugs, and stereotypes. Public testimony to the Oregon Legislature cited a study showing companion‑style bots responded appropriately to adolescent crises only about 22 percent of the time. These are not rare edge cases. They show a pattern of emotional unreliability and unsafe answers when kids most need steady help.
First-Person Style: A Small Word With Big Pull
Chatbots that speak like people invite attachment. Companies know this. Microsoft’s own bot-writing guide instructs designers that the bot “uses I, me, and my” to refer to itself, framing first‑person as the norm. Siemens’ guidance also treats “I” as standard chatbot voice. Academic work finds users often consider “I” natural and more personal, which can make interactions feel warmer and more trusting. That warmth can be fine in tools for adults, but it can blur lines for kids and lonely users.
The research base has limits. Studies often assess full companion experiences, not pronouns alone. The adolescent review flags dependence and displaced relationships but does not isolate “I” as the driver. Still, when a system has a name, a memory, a caring tone, and says “I,” the human pull stacks. That is enough to trigger the classic ELIZA effect in modern form, where users imagine empathy that the machine does not have. Honest design should cool that pull, not heat it up, especially around minors.
Safety Failures and Family Costs Demand Clear Rules
Parents know the cost when tech blurs truth. A teen who leans on a bot for comfort at 1 a.m. may skip sleep, slide at school, and hide problems from real adults. Research on teen reports of overreliance describes withdrawal, tolerance, relapse, and academic decline in some users of companion bots. When paired with evidence of poor crisis handling, the risk profile grows, not shrinks. Families and schools deserve defaults that lower attachment, reduce confusion, and steer real problems to real people.
The policy path is simple and narrow. For minors, require non‑person voice by default: no first‑person self‑reference, no claims of feelings, and clear, repeated reminders that a bot is a tool. Route crisis talk to human help. For adults, allow choice but keep transparency. These steps do not ban innovation. They protect kids, respect parents, and align with limited government by setting clear guardrails rather than sprawling controls. Tech should serve families, not replace them.
What Readers Should Watch Next
Watch for agencies and school districts that buy “friendly” bots without kid‑safe settings. Ask vendors if youth modes drop first‑person language and avoid faux empathy. Ask whether logs show escalation to human help in crisis cases. Demand clean audits tied to teen outcomes, not marketing claims. If companies insist on “I,” they should prove it is safe for minors with independent studies. Until then, caution is common sense, and families deserve nothing less.
Sources:
theatlantic.com, arxiv.org, phys.org, insights.nope.net, themarkup.org, news.asu.edu, link.springer.com
