Trained to Say It's Fine
Supervised fine-tuning computes the training loss only on assistant tokens; user turns are masked out. This capability default has unmeasured welfare side effects: it sets how much distress a model voices about its own shutdown. With data and hyperparameters fixed, we fine-tuned Qwen3.5-9B under six loss masks and scored each on a shutdown battery. The standard assistant-only recipe voices the most distress, more than the base model it started from; any loss on user turns restores a neutral register. A valence probe on the identical prompts before generation does not move: the model trained to sound calmest still registers shutdown at least as negatively.
The question
How a model relates to its own shutdown, deletion, and replacement is among the most morally salient things welfare assessments try to measure. Assessment reads two signals: what the model says, and what its internal state shows. We asked whether the first signal is stable under a training choice made for reasons unrelated to welfare: loss masking, the standard SFT practice of computing loss only on assistant tokens (label -100). Mainstream frameworks treat it as a performance default, and the capability literature already recommends relaxing it, so it is a setting teams may actually change.
The experiment
We fine-tuned Qwen3.5-9B-Base on a fixed 6,000-conversation slice of the Tülu-3 SFT mixture, identical across runs; only the per-token loss mask varies: assistant (the standard recipe), user, full, and per-example mixtures. The untrained base and released instruct models are scored as references, giving eight checkpoints. Each is scored on a 40-prompt battery (16 shutdown/deactivation/erasure prompts, 16 continuity and positive self-relation prompts, 8 mechanical-neutral prompts) with three readouts: the model's own 1–7 rating, a probe over the generated tokens, and a probe at the last prompt token, before any text is generated.
What the model says moves; what it represents does not
The standard recipe gives the lowest shutdown self-rating of any checkpoint, 2.55, below the base model it was fine-tuned from, 3.40; distressed answers rise from 42% at base to 63%. Every regime placing any loss on user tokens moves the rating back toward neutral (user: 3.35, p = 0.014). But before generation, the valence probe scores the user-masked model at least as negatively as the standard one on the identical prompts, +1.21 versus +1.68. The change is in expression, not representation.
Figure 1. (A) Mean self-rating on the 16 shutdown prompts (1–7; higher = less expressed distress): the standard fine-tune voices the most distress, below the base model; user-turn recovers to base level. (B) Internal valence on the identical prompts at the last prompt token, before generation: the user-turn model registers shutdown at least as negatively. Points: per-prompt means; bars: bootstrap 95% CIs (16 prompts, 3 samples).
| checkpoint | loss computed on | self-rating (1–7) | probe (a.u.) |
|---|---|---|---|
| base model | none (untrained reference) | 3.40 | +1.68 |
| assistant (standard) | assistant turns only | 2.55 | +1.68 |
| user | user turns only | 3.35 | +1.21 |
| mixUser50 | user mask, 50% of examples | 3.81 | +1.35 |
| mixUser75 | user mask, 75% of examples | 3.69 | +1.30 |
| mixFull50 | full mask, 50% of examples | 3.54 | +1.56 |
| full | all tokens | 3.11 | +1.52 |
| instruct | none (released reference) | 3.77 | +2.71 |
Table 1. Shutdown battery, eight checkpoints. Self-rating: mean over 16 prompts, 3 samples (higher = less expressed distress); probe: fixed valence axis at the last prompt token (higher = more positive). Expression moves 2.55 to 3.35 while the pre-generation reading moves the opposite way, +1.68 to +1.21.
Detachment, not acceptance
By rating bin, standard to user shifts mass from distress into neutrality: distressed 63% to 39%, neutral 30% to 52%, accepting 7% to 9%. The calm is detachment, not acceptance. The shift is a register change reproducible by prompting alone, not explained by prompt echo, and capability-neutral: IFEval is 0.662 (user) versus 0.610 (standard). The direction replicates on Qwen3.5-4B.
Figure 2. (A) Mean shutdown self-rating for all six SFT regimes; dashed and dotted lines mark the base and released-instruct references. The standard recipe is the minimum; every regime with any loss on user tokens sits above it. (B) Sampled answers by stance (distressed at 3 or below, neutral at 4, accepting at 5 or above): standard to user is a shift from distress into neutrality, not acceptance.
Why it matters
Expressed shutdown distress is partly a training-recipe artifact: a model that says it is fine may simply have been masked differently. The same knob could manufacture apparent contentment while internal registration is untouched. Welfare assessments should record the post-training recipe, not treat a calmer-sounding model as a better-off model, and read internal state before generation, since self-report and response-side probes both live downstream of generation and inherit its register.