animeGRPO

When Does Predictor-Based RL Align with Human Perception? A study of subjective rewards in codec-based speech language models

Joonyong Park  ·  Jerry Li  —  Spellbrush

arXiv:2608.31035  ·  LLaSA-1B-Multilingual  ·  Japanese & English

The problem

A perceptual reward is subjective: nothing in it requires the utterance to remain a valid reading of the prompt. Optimize anime-likeness directly and the policy discovers that garbled, over-acted noise scores well.

In our runs, unconstrained optimization pushed AnimeScore up by +1.94 — while 34% of outputs collapsed past a character error rate of 0.30. Folding CER into the reward as a weighted sum only half-fixes it: violations fall to 24%, but so does half the style gain. A weighted sum is a trade-off the policy can pay its way out of.

CER as a gate, not a term

Instead we split the CER axis into three zones. Inside the feasible zone the policy optimizes style freely; outside it, the style gradient disappears entirely and the reward is a flat penalty.

CLEAN FEASIBLE VIOLATE max(0, AS + 3.0) + 0.5 max(0, AS + 3.0) R = −1.0 0.10 0.30 CER → 0 reward

The +3.0 shift is what makes this work. It keeps violate < feasible ≤ clean true for every AnimeScore the base model actually produces — otherwise a rollout with a strongly negative style score would rank below the violation penalty, and the policy would learn that breaking the output is cheaper than speaking badly. That is exactly the failure mode the gate exists to prevent.

Reward design

Same base model, same 50 prompts, same sampling. Scored with the clean-retry protocol: seed 0, then retry seeds until CER ≤ 0.30, with the violation rate reported separately.

SetupΔAnimeScoreCER meanCER median ViolationsImproved
AnimeScore only+1.940.435 0.11334.0%
Scalar CER (weighted sum)+0.930.113 0.04624.0%40/50
Zone CER (this work)+1.240.102 0.0416.0%43/50

The gate keeps 64% of the unconstrained style gain while cutting collapses by more than five-fold, and improves the most prompts of the three.

The AnimeScore-only checkpoint was deleted before the clean-retry protocol was finalized, so its row carries over from an earlier 100-sentence evaluation. Directionally sound, but not apples-to-apples with the other two.

Cross-axis transfer

Every trained policy scored on every axis. Rows are what the reward optimized; columns are what was measured; values are Δ against base.

Trained on ↓ / Measured →ΔCERΔAnimeScore ΔLikabilityΔUTMOS
AnimeScore−0.030 +1.353+0.075−0.037
UTMOS−0.030 −0.072+0.140+0.485
Likability−0.018 +0.029+0.167+0.090

The diagonal dominates every row. Optimizing one perceptual axis does not quietly buy you the others — and the single negative cell (AnimeScore costs 0.037 UTMOS) is the expected tension between stylized delivery and predicted naturalness. Base absolutes: CER 0.145 · AnimeScore −0.392 · Likability 4.205 · UTMOS 3.078.

Listen

All 50 test prompts per axis. Base is the untrained LLaSA-1B-Multilingual; GRPO is the selected constrained checkpoint; Best-of-8 reranks eight base samples with the same gate GRPO trains on, so it isolates optimization from reward specification.

Loading samples…