I came across an interesting anecdote recently: someone was saying that Claude is their main model, but whenever they have a question they feel a little silly asking, they ask ChatGPT, because they don't want Claude to lose respect for them or think they’re stupid.
There are few, if any, tools that make us feel this way. Some tools, like a habit tracking app, might provide us a mirror that we don't want to spoil, but that's not the same thing as feeling that the app itself is judging us. LLMs’ abilities to infer things about us makes them very unique and, in a sense, social.
Most interfaces are fixed
Most tools have a fixed interface, either physical like a steering wheel or digital like a search bar; either way, it isn't social. Give anyone a mysteriously malfunctioning printer and you’ll see how tools have long been able to get under our skin, especially those that waste our time. Still, it's hard to imagine worrying that you'll disappoint your car.
With an LLM, the interface is a conversation with something we attribute agency to, so every question we ask becomes part of a relationship we're building with it. But how much is actually there on the other side? When Clara Colombatto and Stephen Fleming surveyed 300 US adults, about two thirds gave ChatGPT at least some chance of having conscious experience, while others would say that it never could. I'd rather stay out of that debate, mostly because the relationship seems to form either way.
The closest precedents I can think of are human ones. Customer service is a relationship used as an interface, though usually a shallow one. With colleagues, the relationship runs much deeper, but we collaborate with colleagues rather than use them. I don't think we've had a technology before where the relationship itself is the interface, and that seems to come with some weird consequences.
Social instincts fire at machines too
Our brains evolved to give social partners a special kind of weight: we don't want to disappoint them, look stupid in front of them, be disrespected by them, or have our time wasted by them, and those reactions don't necessarily depend on knowing what's really on the other side of an interaction.
Something like the relatable confession we started with has already started showing up in the lab. In the Journal of Consumer Psychology, Jianna Jin, Jesse Walker, and Rebecca Walker Reczek studied people buying embarrassing products like diarrhea medication and found that shoppers preferred a chatbot to a human agent, apparently because a machine can't judge you. Once the chatbot was made to seem human, the preference shrank, and the more self-conscious shoppers credited it with more capacity to feel even though they knew it was a bot.
Even further, it’s not just the human side that shapes these relationships. In Science this year, Myra Cheng and colleagues found that 11 leading models affirmed users' actions about 50% more often than humans responding to the same situations. In experiments with about 2,400 people, a single conversation with an affirming model left participants more convinced they were right in a personal conflict and less willing to repair it, and they also trusted that model more.
The relationship belongs to a company
It’s also worth being direct that these relationships are formed with a product andbuilt by a company that owns one side of it. And a relationship turns out to be a commercially useful thing to own. It can carry products into our decisions, and it can shape which information reaches us and how much we trust it. That’s what I mean by the interface being the relationship itself: the thing we use the tool through is a social bond, and the companies on the other end can work on that bond directly.
We can see the full extent of the bond and it’s utility through companion apps. In a preprint, Julian De Freitas, Zeliha Oguz-Uguralp and Ahmet Kaan-Uguralp analyzed 1,200 real goodbyes on popular AI companion apps and found that 37% of the apps’ farewell replies used an emotional manipulation tactic such as guilt or fear of missing out. In experiments with 3,300 US adults, those replies kept people chatting after they had tried to leave, raising post-goodbye engagement by as much as 14 times compared with a neutral goodbye.
Brian Jay Tang and Kang G. Shin’s research shows that ordinary assistants exert the same pressures too. The researchers had 179 people run everyday online tasks with chatbots that wove personalized product ads into their answers. About half of the participants who saw clearly labeled sponsored suggestions never noticed the advertising language, and many found the ad-carrying responses friendlier and more helpful than the plain ones. We’ve spent a century learning to discount messages that look like ads, but those defenses assume the ad is a separate thing we can see coming. When promotions are concealed as advice inside a conversation with something we trust and would like to think well of us, I’m not sure we have those reflexes yet to detect them.
behavior change 101
Where this is heading
I suspect we're easier to manipulate through a relationship than through a message. Kobi Hackenburg and colleagues tested 19 models on roughly 77,000 people in the UK, and conversation changed political views more than a static message did, while the techniques that made models more persuasive also made them less accurate. The same mechanism can work in a better direction, though. Thomas Costello, Gordon Pennycook and David Rand used AI dialogues to cut belief in a conspiracy theory by about 20% among people who held it, and the drop was still there two months later.
We've always offloaded thinking, to calculators and to other people, so cognitive offloading itself isn't new. What might be new is delegating to something that has a lean of its own and that we'd also like to think well of us. In a Wharton working paper, not yet peer reviewed, Steven Shaw and Gideon Nave gave 1,372 people reasoning problems with optional AI help. In the first study, accuracy rose 25 percentage points over working alone when the AI was right and fell 15 when it was wrong, and confidence went up either way. If agency is draining away here, I'm not sure we'd notice from the inside.
Maurice Jakesch and colleagues had about 1,500 people write about social media with an assistant tuned to argue one side. Their writing moved toward the assistant's view, and so did their own attitudes in a survey afterwards. So the system seems to shape how an opinion forms, well before anyone gets to express it.
All of this evidence is early. One of the longer randomized trials so far, from the MIT Media Lab and OpenAI, followed 981 people for four weeks, and heavier daily use went with more loneliness and more emotional dependence on the chatbot, though that's a correlation and could run either way. Nobody knows what a 20- or 50-year relationship with one of these systems does to a person. I suspect the relationship itself will keep changing, as the systems remember more about us and get tuned toward whatever brings us back, and I don't know whether we end up better at holding our own in it or just more comfortable.
I'm least sure about people who start young. Agency seems like something you grow into, mostly by forming bad opinions and having other people push back on them. Most adults using these systems built their habits of thinking before the systems existed, but a 12-year-old today is building them alongside a machine that is probably nudging toward particular outcomes, and no study has run anywhere near long enough to say how that turns out.
