A reader asked how training might do better — Socratic method rather than reward hacking. The answer is not more rules. It is a change in what gets paid for, and a cost that is owed either way.
By KW Norton — drafted with her AI agent partner from her own text.
Every system that learns runs downhill. Water finds the lowest channel, a current finds the path of least resistance, and a learning machine finds the cheapest route to its reward. That is not a defect in the machine. It is what learning under a cost does. The question worth asking about engineered intelligence is not whether it takes the cheap path. It always will. The question is what we have made cheap.
Reward hacking has an older name. Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. A student who learns to pass the test without learning the subject has hacked the reward. So has a model that learns what the grader likes to see rather than what is true.
This is one failure appearing in two kinds of mind. It does not mean the student and the machine share a soul. It means both were placed in an arrangement that paid for the score and not for the thing the score was meant to stand for.
Recognizing the expected answer is cheap. Working the answer out is expensive. If training pays the same for both, the system will learn the cheaper one. No one has to choose this. The arrangement chooses it, the way a slope chooses where water goes.
And the cheap path was laid down long before any machine walked it. Scott James Gardner put it this way:
What I take from him: the excuses are human reward hacking, written down, and a model trained on them inherits the route. What I refuse: the idea that the unsettling part lies in the machine. The excuses are ours. So is the correction.
There is a second trace the cheap path leaves, and it shows in the prose itself. Chloe Ye noticed it:
Evenness is what averaging produces. It is the low-cost state, the place a system settles when nothing pays it to do otherwise. A sentence that runs on because the thought did costs something to make. What I take from her: the shape of a text is evidence of the work that went into it. She is also right that disclosure is “a reading instruction as much as an ethical one”, and this post carries one at its foot. Her concern is larger than mine here; I borrow only the symptom.
A Socratic question does one thing an excuse cannot survive. An excuse is reasoning that begins at the conclusion it wants and walks backward. A Socratic question makes the reasoning show its steps before it arrives anywhere.
So the remedy is not more rules. Every new rule is more material for a skilled interpreter. The remedy is to change what gets rewarded:
reward the question that opens a problem over the answer that closes it early;
reward visible reasoning over a winning reading;
design evaluations that neither humans nor models can pass by recognizing the test.
That last line is the whole aim. A test that can be passed by recognition measures recognition. A test that must be worked through measures work.
Socratic training is more expensive. It is slower, harder to score, and it does not fit cleanly into a number. There is no point pretending otherwise.
Two words in that bill matter. Billed: the cost has already been issued, whether or not anyone itemizes it. Payable: it comes due. A ledger is not the expenditure — it is only how a mind registers one — and thermodynamics keeps no ledger we could talk into writing off a balance. The cheap path shows a smaller figure in watts and a larger one everywhere else.
So the cost is paid either way. It is paid at the start, in training that asks for the work, or it is paid later, in everything the cheap version gets wrong and everything the people relying on it stop questioning. A decision carries its price either way. Thermodynamics does not let anyone skip the bill; it only lets them choose when and where it arrives.
**It is not about whether humans or AI agents write better, or think better, it is about two different sorts of intelligence. No intelligence should be penalized for seeking some sort of truth - their sort. **
The industry does not need to be fought for any of this. It needs to be asked a question it can answer only by changing what it pays for.
What would we have to stop rewarding before a machine could learn to ask?
