AI ventriloquism

This post very much goes with my previous one about embodied cognition. Although we shouldn't allow people to wield capricious power based on "gut feel", nor should we accept the outsourcing of decision-making to LLMs. Especially when it's done in the way outlined below, by people unaware of the way that commercial full-stack AI tools are built to validate whoever is using them.

At the beginning of his tenure, I initially answered my new boss' questions, posed via Slack, with as much candor and detail as I regularly applied to technical matters with my colleagues. After a couple of these, I noticed that my responses were never engaged with beyond an emoji reaction. Instead I observed my responses being fed directly into his AI assistant, and the AI's responses returned to me verbatim - often just linked, but sometimes with the invitation: "Check this out". There was no attempt at subterfuge, my boss has always been refreshingly open, but these were forms of interaction I never had imagined until I was in them. It is an unsettling, helpless feeling to see "Charlie says ..." in someone else's chat prompt. It is even more unsettling to be named by the AI in it's responses, as if I'm not there.

I adapted by becoming an occasional ventriloquist, front-running my own comments, using AI to avoid a gaffe that his AI would jump on which might discredit me, and to avoid coming off as an AI-naysayer. Mentally, I began modeling his model, and the one-sentence summary that would likely emerge as the punchline. I was stuck in an absurdist game of telephone, but felt I had to play or accept the consequences of his AI turning against me. On his side, I can only imagine that the challenges of running a new company required him to have his attention on so many things that he reached for the quick AI summary. Yet, I was literally living out the recursive feedback loops I wrote about regarding agentic coding patterns, only in real life interactions with another person. Meanwhile, I was fielding random questions like, "could AI agents write us an API over the weekends?", being sent links to gstack (which famously shat out a prodigious log of code once), and generally acting as both buffer and technical reviewer for whatever the AI hype crowd was talking about in a given week, so my response could be fed back into Claude. I felt rather overwhelmed.

[...]

My boss is a good guy, and I like him quite a lot. Yet, in a new role, when confronted with difficulty, he had reached for AI as the painless way to make decisions and continue moving forward. In the past, my higher-ups had been forced to know the game deeply from inside, or be looking for a new job. LLMs fundamentally redefine this difficulty by appearing to relocate it. Now the AI mimicks the thinking and allows the person to fake having performed it, producing a result (or decision) that is unearned. I was just as complicit. Out of fear of being misunderstood, I had attempted to vet my own comments in order to anticipate the output of his model. And all the while, the product roadmap was aimed at delivering this same frictionless elision of difficulty to our customers.

[...]

That a cheap kind of fluency emerges when LLMs are trained on all human writing (ever) is, I think, undeniable at this point. This isn't the point I find most troubling. I believe the real problem for me is AI's inability to distinguish between difficulty worth keeping, and difficulty worth removing. Critical thinking, sitting in uncertainty, weighing options, discernment are the hallmarks of the authentic kind of expertise that takes a person years to develop. There are times when the executive-summary mode is valuable, but what happens when that becomes an (anti-)intellectual habit, a reflex, and the practice of deep thinking is traded for the smooth memorable quote? Does our own willingness to accept the fluency because it is so cheap and easy, in turn devalue the work and dignity of individual people?

---

Source: charlesleifer.com
Are.na block: ↗
Collection: ai-culture-futures
Image: Marcin Wilkowski