The problem
A perceptual reward is subjective: nothing in it requires the utterance to remain a valid reading of the prompt. Optimize anime-likeness directly and the policy discovers that garbled, over-acted noise scores well.
In our runs, unconstrained optimization pushed AnimeScore up by +1.94 — while 34% of outputs collapsed past a character error rate of 0.30. Folding CER into the reward as a weighted sum only half-fixes it: violations fall to 24%, but so does half the style gain. A weighted sum is a trade-off the policy can pay its way out of.
CER as a gate, not a term
Instead we split the CER axis into three zones. Inside the feasible zone the policy optimizes style freely; outside it, the style gradient disappears entirely and the reward is a flat penalty.
The +3.0 shift is what makes this work. It keeps
violate < feasible ≤ clean true for every AnimeScore the base
model actually produces — otherwise a rollout with a strongly negative style
score would rank below the violation penalty, and the policy would
learn that breaking the output is cheaper than speaking badly. That is
exactly the failure mode the gate exists to prevent.
Reward design
Same base model, same 50 prompts, same sampling. Scored with the clean-retry protocol: seed 0, then retry seeds until CER ≤ 0.30, with the violation rate reported separately.
| Setup | ΔAnimeScore | CER mean | CER median | Violations | Improved |
|---|---|---|---|---|---|
| AnimeScore only | +1.94 | 0.435 | 0.113 | 34.0% | — |
| Scalar CER (weighted sum) | +0.93 | 0.113 | 0.046 | 24.0% | 40/50 |
| Zone CER (this work) | +1.24 | 0.102 | 0.041 | 6.0% | 43/50 |
The gate keeps 64% of the unconstrained style gain while cutting collapses by more than five-fold, and improves the most prompts of the three.
The AnimeScore-only checkpoint was deleted before the clean-retry protocol was finalized, so its row carries over from an earlier 100-sentence evaluation. Directionally sound, but not apples-to-apples with the other two.
Cross-axis transfer
Every trained policy scored on every axis. Rows are what the reward optimized; columns are what was measured; values are Δ against base.
| Trained on ↓ / Measured → | ΔCER | ΔAnimeScore | ΔLikability | ΔUTMOS |
|---|---|---|---|---|
| AnimeScore | −0.030 | +1.353 | +0.075 | −0.037 |
| UTMOS | −0.030 | −0.072 | +0.140 | +0.485 |
| Likability | −0.018 | +0.029 | +0.167 | +0.090 |
The diagonal dominates every row. Optimizing one perceptual axis does not quietly buy you the others — and the single negative cell (AnimeScore costs 0.037 UTMOS) is the expected tension between stylized delivery and predicted naturalness. Base absolutes: CER 0.145 · AnimeScore −0.392 · Likability 4.205 · UTMOS 3.078.
Listen
All 50 test prompts per axis. Base is the untrained LLaSA-1B-Multilingual; GRPO is the selected constrained checkpoint; Best-of-8 reranks eight base samples with the same gate GRPO trains on, so it isolates optimization from reward specification.