Flimsy value systems and loser incentive systems
There's not many people who write long posts that are worth reading any more, but Adam Mastroianni is one of them. His latest is on the topic of "incentives" with his argument being that we as humans need to transcend our animal-like tendency to respond to incentives with something, well, more noble.
It is, of course, worth reading in its entirety, but I'll just pull out a few highlights, starting with how it's only losers who just do things because of desire:
A critical part of human development—in fact, the most human part of human development—is to acquire a purpose. That means refining your sense of right and wrong and figuring out how you, personally, can increase the right and decrease the wrong. Once your neural tube has ballooned into a brain, once you can poop without issue, conjugate irregular verbs, and feed yourself solid foods, you ought to ask yourself, “Why am I doing these things?” What’s it all for? How can I make myself useful?
It’s fine if you’re still working on your answers to those questions. It’s fine—and good!—if you have multiple answers, or if your answers change over time. But it’s not fine to pass on the questions entirely. We owe these answer to each other, and we owe them to ourselves, because a sense of purpose is both prosocial and protective. If you have a void where your moral convictions should be, then that cavity of your personality will get filled up with tumorous desires—for money, power, fame, status. This is the kind of person who “responds to incentives”.
Like Mastroianni, I'm constantly amazed at how little thought and effort the average person seems to have put into their value system that guides their behaviour. Perhaps its a protective mechanism whereby we optimise for happiness, as goodness knows those who study at postgraduate level, especially in the Humanities, are more likely to be depressed.
As I’ve written before, people seem to understand the world exactly as well as they need to, and no better. Everyone knows how to use a toilet, for example, but they mistakenly think they also know how a toilet works.
I think this illusion is inevitable, given the architecture of the mind. According to predictive processing theory, the mind is always trying to guess what’s going to happen next, and it only updates its guesses when its expectations don’t match its experience. If you run that algorithm for a while, you’ll end up with a model of the world that is good enough, even if it’s not particularly good. You don’t notice the errors in your model until you run smack-dab into them, and every time you correct one, you assume that you’ve gotten them all.
From the sociology side... People’s folk morality and folk religion, just like their folk physics, their folk biology, etc., are exactly as detailed as they need to be. Don’t hit, don’t steal, take turns, say please and thank you—almost everyone achieves this basic level of moral development, because otherwise you’ll end up friendless, if not incarcerated. Similarly, almost everyone acquires some vague notion of an all-seeing, sky-dwelling God through simple osmosis. (After all, He is kind of everywhere, literally and metaphorically.)
[...]
Our value systems are flimsy for the same reason our scientific reasoning is flimsy: we don’t fix our mental models until something tells us that they’re broken. So unless your pastor is popping quizzes during the sermon and cold-calling people in the pews, and unless you find yourself thrust into various Trolley Problems on a regular basis, there’s nothing that tells you that you need to firm up your moral and theological convictions. Why spend your days wondering about right and wrong, good and evil, sacred and profane, etc. when you don’t have to? You’re just going to get yourself into trouble, or even worse, into a philosophy PhD program.
This could explain why our sense of right and wrong is so easily overwritten by any half-decent system of rewards and punishments: we just don’t have much written there in the first place.
But we can't just say that people are "responding to incentives" when they do bad things. Yes, we need to look at those incentives and remove them where possible. But also, people should reflect on their value systems and, you know, do the right thing.
I understand why so many people are so concerned with incentives. It’s infuriating to watch villains succeed. And look, if we can stop rewarding people for doing the wrong things, we obviously should.
But I don’t know if we can. Goodhart’s Law, Campbell’s Law, and their corollaries state that every incentive is destined to be corrupted eventually, and so far, no one has proven these conjectures wrong. That’s why discussions around incentives often sound to me like we’re talking about a monkey’s paw that will grant our wishes, but only in evil, ironic ways. “The last time we made a wish on the Monkey’s Paw, it backfired horrifically. To rectify this, we have decided to make another wish on the Monkey’s Paw.”
In the eternal game of cat and mouse between the goodhearted people trying to fix the incentives and the Goodhart’d people trying to exploit the incentives, I am rooting for the cats. We should check whether our incentive systems are having their intended effects, and we should revisit and revise them accordingly. But for some reason, the second we start talking in terms of incentives, we also start acting like courage doesn’t exist and can’t exist. Bad incentives have become our generation’s version of “I was just following orders”, an excuse that should be an indictment.
Source: Experimental History