Smuggling in a "Self"

All novel understandings in life come from bundling or unbundling concepts. - Jim Barksdale, basically

Every intelligent thing humans have ever needed words up to the last few years for came with a self attached. If it was smart, the lights were on and someone was inside, thinking, intending, desiring, etc. So the very words we use for intelligence are person-verbs: wants, knows, tries, refuses, understands, lies, decides. There is no vernacular, neutral register to describe something as smart but not conscious/aware/etc. You cannot describe what a language model did without conjugating it into a subject, and the moment you do, you have asserted something you did not intend to assert and cannot defend if pressed.

All the AI debates get tied in this knot, usually without anyone noticing.

You can see this whenever major AI labs or news articles talk about something an AI did. You get stuck in this passive language, but I’d wager 99% of people reading that anthropomorphize the AI. Hell, we do it with clouds and trees and the natural world all the time, and those things don’t speak to us without psychadelics!

You can see evidence for how we bundle intelligence and a self in law, where intent is load-bearing rather than decorative. Intent separates murder from manslaughter. Fraud requires that the speaker knew the statement was false. Now read a sentence from a safety paper: “the model attempted to deceive its evaluators.” That is not equivalent to “outputs in condition B were scored as deceptive by graders.” The first summons a courtroom. It implies a knower, a state of belief, a preference the knower held about what the evaluator would come to believe. Nobody has said what the model knew, or what knowing would consist of here, or how you would check.

The deeper cost is that our language settles the hard problem of consciouss by grandfathering in a self. Whether producing tokens that look like reasoning brings awareness along with it is open — genuinely open, with no agreed method for resolving it. But “the model wants to avoid being shut down” answers the question in the syntax before anyone has made the case. The deflationary move is the same error running backwards. “It’s just predicting the next token” claims equally confident knowledge that there is nothing inside. Both camps are helping themselves to an answer nobody has.

Where this gets extra tricky is accountability. When a model does something, is the model owner accountable? For the act? Or to have insurance like we do with cars?

I don’t know if the fix is to be a language lawyer. First: you won’t get someone to change language around something they are unintentionally doing anyway (the accidental personification). Second: there’s something powerful in re-using all our intuition about an active intelligence: you predict a chess engine far better by asking what it is trying to do than by reasoning up from the computer chips.

Perhaps we can learn from the law, which built the workaround once: Corporate personhood invented a legal person for a thing with no inside. The law handles a thing that is smart and acts, without necessarily having a self. We have not done that labeling for AI.

However, noticing how much most people do not reason well about corporate personhood makes me think this will not practically help.

We have unbundled personhood from intelligence. But our language hasn’t, so debates go haywire from the start.

—————-

PS: Ade Oshineye noted to me that technically AIs are articulate, and not 100% necessarily intelligent. I think I agree, though articulate in embedding space, which can be combined and output into language that seems an awful lot like intelligence, so I’ve left the article alone since I want it to focus on bundling personhood with our language tying is all in knots.