Authorized CER ablation and matched baseline study¶

Omar authorized the causal CER study and a matched reward-machine baseline on 2026-10-12, with CPU-cluster compute. This follows the frozen-checkpoint investigation. New results live under exports/optimisations/cer-study-20261012/. Archived profiles, checkpoints and paper tables are inputs only. Other sessions own equilibrium verification and paper source; this study does not edit those files.

Fixed study design¶

The targeted experiment trains Territory/IPPO and Stag Hunt/PR2-IPPO under five variants, using fresh training seeds 100–104:

The first four variants are the causal ablation; the fifth is the matched alternative method. These make 50 profiles. The full study adds IPPO no_cer, cer and rm_shaping on the other seven tasks, including Stag Hunt: 105 further profiles, for 155 total. Targeted profiles are scheduled first. Each profile uses the notebook's original initial environment-step budget, rounded to its rollout batch, without NashConv-driven training extensions. Optimizer initialization, policy architecture, observations, rollout shapes, epochs and minibatch counts are matched within each comparison.

The controller first runs two setup pilots, one per targeted case. Each pilot exercises all five modes with 4,096 training steps and 16 evaluation episodes. The gate checks successful execution and finite losses; pilot satisfaction does not select parameters or seeds. Pilot outputs remain separate from the full study. Parameters are fixed at action-ratio cap 1, trace cap 1, the notebook's lambda, and shaping discount 0.9.

What is corrected, and what remains biased¶

CER corrections retain factual behavior log probabilities and a factual mask for every synthetic trajectory. Source environment indices are carried through relabelling, so the behavior density belongs to the actual generating stream. PPO still clips the current policy ratio relative to the synthetic snapshot; the behavior correction multiplies the clipped surrogate separately.

For IPPO, the lambda trace has

rho_t = min(rho_cap, pi_old(a_t | synthetic_obs) / mu(a_t | factual_obs))
c_t = lambda * min(1, pi_old(a_t | synthetic_obs) / mu(a_t | factual_obs))
delta_t = rho_t * (reward_t + discount_t * V_next - V_t)
correction_t = delta_t + discount_t * c_t * correction_next
target_t = V_t + correction_t
advantage_t = reward_t + discount_t * ((1-lambda)*V_next + lambda*target_next) - V_t

The current-action weight is outside the actor's PPO clipping. Episode completion cuts bootstraps and traces; a rollout boundary bootstraps from its last observation. At ratios one, both targets and advantages match existing GAE exactly, including lambda below one. At lambda one, the construction is the standard V-trace target/actor advantage before the actor's action factor. Clipping changes the target policy and introduces bias. Synthetic initial states and occupancy weighting are still auxiliary; this is not an unbiased policy-gradient estimator of the original initial-state objective.

PR2's response-aware advantage and one-step joint-action critic are distinct from IPPO's GAE. Its corrected variant changes conditional actor weighting; it does not add a multi-step return trace to its one-step backup. Function approximation and opponent-response error remain. Comparing correction types across the two families cannot attribute differences solely to one component.

In critic_only, synthetic samples have zero actor and entropy masks, and actor advantage normalization uses factual samples only. The critic uses all valid samples. Padding zeros critic, actor and factual masks. The shared optimizer still clips the combined actor/critic gradient, and its critic evolves differently; the ablation isolates direct synthetic actor contributions, not every indirect consequence of critic learning.

Default CER_UPDATE_MODE="uncorrected" and diagnostics interval zero preserve the original update semantics. The numerical repair computes the same PPO surrogate in log space to prevent a finite clipped loss from yielding an overflowing intermediate's NaN gradient. Existing checkpoints and policy architectures do not change. New optional transition metadata has defaults so older direct transition constructors and private relabelling calls remain supported.

Matched reward-machine baseline¶

The shaping adapter adapts the automated RM potential method of Toro Icarte et al., section 4.4. Its abstract machine rewards first reaching an accepting state and treats accepting abstract states as terminal. Negative abstract optimal values give the potentials. For this unit-reward deterministic abstraction, reverse shortest paths compute the value-iteration solution. Physical accepting DFA states can still be left; the physical monitor and horizon remain unchanged.

Only the returned training reward changes:

r_shaped[t] = r_objective[t] + gamma*Phi(next_monitor) - Phi(current_monitor)

At any task termination or truncation, the next potential is set to zero. Consequently, for every complete trace,

sum_t gamma**t * r_shaped[t] = sum_t gamma**t * r_objective[t] - Phi(initial_monitor).

The initial monitor is fixed, so the difference is independent of the action profile. This preserves the corresponding utilities and equilibrium set. The calculation uses only the agent's own known DFA, not an environment model, another player's automaton or privileged observations. This is the finite-trace adaptation of RM automated shaping with the existing PPO learner, not a claim to reproduce every original reward-machine learner or CROM's opponent model.

Territory, Coins and Harvest have gamma one. Stag Hunt's fixed 20-step horizon makes its original gamma 0.99 terminal return a fixed positive multiple of satisfaction, so policy preferences/equilibria agree. Pursuit can stop early: its paper gamma 0.99 is a stopping-time-dependent surrogate. The new matched Pursuit controls and baseline all use gamma one to compare the actual satisfaction game. Metadata records both the paper gamma and the study gamma. These Pursuit controls are fresh utility-aligned runs, not reproductions of the old discounted training results.

Evaluation and diagnostics¶

Every full profile is frozen and evaluated on 1,000 unshaped episodes, with the existing curated thresholds fixed. Environment evaluation seeds and the action-sampling RNG are separate from training. The latter is explicitly overridden: reusing the training seed for evaluation action sampling would reuse PRNG draws even if environment seeds differed. Evaluation randomness is shared across comparison arms for a given training seed, allowing paired comparisons; family corrections do not require independence between profiles.

Reports include original nominal simultaneous bounds and a per-task family correction over the prespecified tested profiles: 25 for Territory, 40 for Stag Hunt, and 15 for each other task. They report all agents, minimum and summed satisfaction, joint margins, nominal and corrected certification, training/evaluation walltime and per-seed paired differences from no CER. Partial summaries explicitly state completion counts and cannot establish a winning method. Failed workers remain in the ledger and failure artifacts.

During training, logs record factual/actor fractions, log action-ratio mean and RMS, clipped-ratio mean, clipped fraction and empirical ESS. Every 20 updates, snapshot actor gradients record factual/synthetic/actual norms and cosines. Critic and entropy gradients are excluded from those probes. Validity flags distinguish defined cosines from the zero placeholder used when one gradient vanishes. The cosine probes normalize over the full rollout, while learner updates normalize within each minibatch. The actual probe names the configured actor surrogate at the snapshot, not the optimizer's eventual parameter change. These are descriptive diagnostics, not independently sampled estimates of the true gradient. Raw objective training histories provide learning-speed information even for the shaped baseline, whose returned training reward has a constant offset.

Cluster execution¶

The controller requires a clean pinned worktree and an existing interpreter. It uses raw Slurm submission to avoid the general runner's automatic pull/sync behavior. Job scripts verify the source commit, force CPU JAX, and use uv run --no-project --python ... with the supplied environment. Source and archived data are not mutated. Installed package versions should be recorded for the deployed interpreter; no controller updates it. Each job uses one physical core under Slurm's whole-core allocation and 8,000 MiB; full jobs initially have a conservative 48-hour walltime.

The lightweight login-host controller caps its active/pending jobs at four and checks the total user's job count against 185. It waits 15 minutes between checks, stores a lock and atomic ledger, and advances after successful pilots. It neither cancels other jobs nor automatically resubmits failures. Its pinned protocol can resume after controller interruption without submitting already recorded workers twice. The other verification/Nash-Q controllers use separate worktrees and output roots.

Worker artifacts are atomic metadata, a final checkpoint, progress JSONL, training histories, metrics, raw satisfaction reports and result.json. Compatible saved final checkpoints resume evaluation after interruption; new and restored model parameters must be finite before publication/evaluation. completed results are skipped. A configuration/source fingerprint prevents overwriting a different run. Summaries are written as both JSON and Markdown.

Independent verification handoff¶

Full workers store checkpoint.pkl under full/<case>/<learner>/<mode>/seed_<N>/. Bundles contain serialized train_states, agent_ids, config, run_seed, canonical monitor numbering, training time and the configuration fingerprint. Companion metadata includes environment settings, monitor fingerprints, package versions and the paper gamma. Verification evaluates the original unshaped temporal objective.

When a response learner copies a corrected or critic-only configuration but disables CER, reset CER_UPDATE_MODE to uncorrected: an inactive nondefault correction is intentionally rejected. Preserve the new Pursuit gamma one when learning satisfaction-oriented responses; early stopping makes gamma below one a different training objective. A response to an RM-shaped profile should initialize its unshaped critic and optimizer deliberately, since the saved shaped critic estimates values with a monitor-potential offset. Actor weights remain compatible with unshaped evaluation.

The separate verification follow-up may select a designated agent, increase response/training budgets and report justified adjusted thresholds. This controlled CER/baseline study retains its fixed budgets and original curated thresholds, so its comparisons remain interpretable. Selected candidates require independent verification or coverage over the selection family.

Validation¶

Tests cover on-policy equivalence to GAE, future importance factors and episode cuts, factual actor invariance to extreme synthetic returns, both learners' real corrected/factual-only loops, exhaustive short Stag Hunt traces proving return preservation even when acceptance is subsequently lost, fixed study budgets/fresh-seed families, and shell-safe pinned worker construction. Local end-to-end probes exercise corrected PR2 and shaped Territory training, checkpoint serialization and independent evaluation. The changed-suite runner includes slow training/verification integrations before publication.

See the controller ledger and study summary for live progress; this design note does not imply the full experiments have finished.