Everyone's a consequentialist until they get punched in the face

An intentionalist is someone who believes that moral judgement should depend on the intentions behind the action. A consequentialist is someone who believes that moral judgement should depend on the consequences of the action.

Most people, I'd suggest, are consequentialists when thinking about other people but intentionalists about things that affect themselves. (Me? I'm a virtue ethicist, but that's a different story...)

So there are cuts to social services that you never used, jobs automated in sectors you don't work in, and customer service failings by companies you've never heard of. You hear about it on the news, but go about your day. The argument is always the same: although there will be costs, the overall outcome will be better (faster, cheaper, more productive/convenience). Ah well, you think, that's the cost of progress. You're a consequentialist about things that don't affect you.

But what happens when the consequences arrive at your own door?

Your benefits claim is rejected, your face is misidentified, your jobs is redesigned out of existence, your work is in the training data, the customer service representative says there's no-one who can override the decision?

All of a sudden, you think, "people cannot treat me like this!" You're no longer a consequentialist, but employing the moral vocabulary of an intentionalist. Distance makes things easier to ignore, but proximity makes relationships, duties, and rights harder to ignore.

I think this is why arguments about AI go round in circles, as the disagreement isn't about facts, per se. I think people are asking different moral questions without realising they're doing so. Some are asking "what will produce the best overall outcome?" while others are asking "did people act responsibly?" or "was the process legitimate?" or even "has someone's rights been breached?"

While most of us can ask all of these questions, we tend to give one of them more priority, especially when the costs and benefits affect various groups in society in different ways. That's the thought about this grid that I asked my little robot friend to create:

(click here or on image to enlarge)

It's probably worth saying that this isn't a test of personality or morality. And also, nobody is simply an intentionalist or consequentialist; there is no "ethical calculator" where you feed a problem into the above boxes, cross your fingers, and hope for a "correct" answer to emerge.

No, this is just a way of noticing which questions people are actually asking, and which have (either purposely or accidentally) faded into the background.

For example, let's imagine a university that's decided to introduce genAI to speed-up the feedback given to students. It's easy enough to think of the institutional rationale for this: it reduces turnaround time for students, and staff have more time for other things. There's a benefit there, but that's only one quadrant.

What happens to the student who needs a lecturer to recognise that they are confused, or struggling, or taking a risk with an idea? I remember writing one of my "essays" as a Socratic dialogue at university, which was very much taking a risk. But I received the best mark I ever received as an undergraduate for that piece of work. What I have done that if genAI was giving me feedback? Probably not.

Who decided that speed was a problem worth solving? Did the institution allow students and staff to shape the decision, or were they just informed about it? And who takes responsibility when feedback is inadequate or inappropriate?

That's just over the short term, as there are longer-term issues as well. Given that we usually differentiate between formative and summative feedback what happens to teaching as a practice, when formative feedback is treated in a summative way? Also, what kinds of student writing therefore become easier to assess? Importantly, given that universities tend not to be in the software business any more, which companies become embedded in educational infrastructure? On what terms?

While none of those questions refute the claim that genAI could "save time", nor does "time-saving" settle the argument. And the same applies to almost ever deployment of AI, such as local authorities using automated triage to reduce caseloads, or health services using predictive tools to direct limited resources. Even in the private sector, a platform might train a model on creative work and then argue that it "expands access" to creative tools.

There can be tangible benefits to all of this, but they may also distribute harms in ways that are difficult to see from the points of view of the organising doing the measuring. We usually refer to these as "externalities".

People, on the whole, usually have good intentions. So when they are building, procuring, or using a system, they are trying to solve a real problem within constraints. But good intentions do not remove responsibility for foreseeable harms. That's why we're usually intentionalists about our own actions, but consequentalists about other people's.

Going back to the 2x2 grid in the diagram above, I think it's useful because it allows us to ask questions to help us view things more holistically:

• Top-left quadrant: duty and responsibility. Who acted, what did they know, what did they owe, and who can be held accountable? This quadrant is focused on things like consent, privacy, professional judgement, explanation, and the right to challenge a decision.
• Top-right quadrant: legitimacy and governance. Who has the authority to set the purpose of the system? Who decides what data can be used, which risks are "tolerable", and whether a system should be deployed at all? Who is able to say no? Who has access to the evidence, and who has the power to change the arrangement?
• Bottom-left quadrant: direct harm and benefit. What happens to people in practice? This includes such things as errors, discrimination, workload, access, dignity, safety, and the uneven distribution of benefits. A system that is useful to an organisation, for example, might be extremely costly to the people whose lives it reorganises.
• Bottom-right quadrant: systemic consequences. What happens if this becomes normal/ordinary? Does the system concentrate power? Does it make people dependent on a small number of vendors? Does it deskill a profession or weaken public capability? Does it normalise surveillance, create new environmental costs, or shift the burden of dealing with a system onto those least able to do so?

The point is that we need to think more carefully, and ask better questions. When people say things like an AI system "works" we need to ask for who? At whose expense? According to whose definition of success? Also, what other options have been considered? If there are mistakes made, who has to "absorb" those, and who gets to call such absorption "acceptable"?

All of this usually gets labelled as "AI ethics" which makes things sound philosophical and abstract. It all seems very ivory tower, with people in flowing gowns thinking about the future. But ethical questions are usually much more mundane. They include figuring out how someone can get a decision changed, whether a child is treated as a person rather than a data point, or whether a worker gets to have a say in how their work is redesigned.

This grid is a heuristic, and is not a substitute for regulation, organising, or professional judgement. I'm just saying that we need to orient ourselves before we start arguing, and a "good" decision about AI should be able to answer questions in all four quadrants. Were people and institutions responsible in how they acted? Was the process legitimate and open to challenge? What are the actual benefits and harms? What wider patterns does this make more likely?

If the answer is only "it works", it's not a justified decision.

As I say in the title, everyone's a consequentialist until they get punched in the face. The harder task is to notice the consequences that land on other people before they realise that for themselves.