CER bias investigation, 2026-10-12¶

The W6 review of pltlf-paper accurately identifies mixed CER results and the uncorrected actor-critic replay distribution. The current evidence does not establish that missing importance weights caused the observed failures, or that adding them will improve every task. The initial investigation audits archived evaluations and frozen-checkpoint distribution mismatch. The approved follow-up adds isolated controlled training, numerical repairs and matched reward-machine shaping; the dated sections below distinguish those results from the initial archive audit.

Review assessment¶

• The handcrafted baselines generally change utilities, hence potentially equilibria. They cannot alone establish an advantage over a method solving the same game. However, the paper also includes objective-trained no-CER variants. Those are matched ablations of CER's incremental contribution.
• Table app_certification.tex reports Territory IPPO at four certified seeds without CER and zero with CER, and Stag Hunt PR2-IPPO at four versus two. Both differences persist in the confirmatory evaluations.
• Appendix D's actor-critic integration explicitly acknowledges that the relabelled update is biased and applies no importance correction.
• Appendix E chooses thresholds using initial evaluation results, and selects tasks where at least one agent reaches estimated satisfaction 0.5. Fresh evaluations with thresholds fixed in advance can support subsequent certificates, with the stated multiple-profile correction. They do not make the benchmark's threshold or task selection independent of the pilot data. Threshold adaptation alone does not invalidate valid simultaneous lower bounds: on the coverage event p_i >= L_i, certification L_i >= q_i implies p_i >= q_i even if q_i was selected using the same data. Selecting which policies, objectives or tasks to report requires coverage for the selected family or an appropriate independent evaluation. Benchmark independence and false-certificate control are separate questions.

Archived evaluations¶

The input artifacts are the working archive copy under /vol/gpudata/oja24-pltlf/exports/. Temporary copies of four profiles, their reports, and all five final checkpoints per profile were read locally. No archived artifacts were modified. See archive provenance.

Confirmatory results use 1,000 episodes per seed. The minimum and sum columns below average per-seed quantities across five independently trained profiles; the minimum is an estimate, not a confidence bound.

The archived thresholds differ between arms. Territory/IPPO uses (0.25, 0.2) without CER and (0.5, 0.5) with CER; Stag Hunt/PR2-IPPO uses (0.9, 0.9) and (0.5, 0.5), respectively. Certificate counts therefore need an explicit common-threshold comparison. Reapplying the stored nominal joint lower bounds at the no-CER thresholds gives Territory 4/5 versus 0/5 and Stag Hunt 4/5 versus 2/5. At the Territory CER thresholds both arms certify zero seeds; Stag Hunt remains 4/5 versus 2/5 at either threshold pair. These are descriptive recomputations of archived bounds, without adding family coverage for post hoc profile selection. The raw probabilities above are independent of the threshold choice. The new training study fixes curated task thresholds across all comparison arms.

Territory is a race with mutually exclusive objectives. CER retains substantial success probability but distributes it more unevenly between roles. For example, seed 3 changes from (0.427, 0.550) to (0.924, 0.039). Improving one player's utility can defeat the other player's threshold. Joint certification is consequently different from maximizing an individual agent's probability or finding a stable strategy profile. Stag Hunt's unsuccessful PR2 seeds have zero satisfaction, rather than narrowly missing a confidence threshold.

These five-seed comparisons describe the existing profiles. They are not a causal experiment isolating importance weighting, or a precise estimate of performance across all possible training seeds.

What the PPO ratio corrects¶

For a factual history, the generating policy is mu_i(a_t | x_t) = pi_old_i(a_t | o_t, m_t). On a relabelled history, the snapshot policy instead has density pi_old_i(a_t | o_t, m_t_tilde).

Both actor-critic trainers recompute snapshot log probabilities at the relabelled observation. Their PPO ratio is therefore

r_t(theta) = pi_theta_i(a_t | x_t_tilde) / pi_old_i(a_t | x_t_tilde).

At the snapshot it is exactly one. It tracks subsequent parameter changes; it does not correct the discrepancy with the factual generating policy. The missing conditional action factor is

rho_t = pi_old_i(a_t | x_t_tilde) / mu_i(a_t | x_t).

Keeping the snapshot denominator for PPO clipping and applying a separate correction is different from replacing the denominator and clipping the combined ratio. The latter also clips the factual-to-synthetic discrepancy.

Even a separate action factor does not generally recover an on-policy update:

• IPPO returns: GAE uses future actions collected under the factual policy. Recomputing values does not change that continuation policy. Correcting only the current action leaves future-return bias.
• Occupancy and objective: arbitrary synthetic monitor starts change the training distribution. Conditional importance weighting within a chunk does not restore the original task's initial-state distribution, or make uniformly weighted alternative monitor states its policy-gradient objective.
• PR2-IPPO: its advantage is response-aware and critic-based, and its critic has one-step targets. The IPPO GAE argument does not directly apply to it. Action/occupancy mismatch and errors in the response-aware critic still matter.

For a fixed physical start distribution and fixed opponents, whole-trajectory likelihood ratios correct the changed learner's conditional trajectory law when dynamics and labels are unchanged and support holds. The other players' factors cancel only when their conditional policies and factual monitor histories are unchanged. A target that relabels multiple players requires the corresponding joint policy factors. This is not a general correction for arbitrary hidden-state aliasing or a new initial-state distribution.

Exactly enumerated examples¶

The diagnostic script enumerates a two-step terminal Boolean-reward example, without Monte Carlo error or a learned critic. At the first step, both policies choose between two branches with probability 0.5. One branch succeeds with probability 0.5 independent of the second action. In the other, the behavior policy chooses the successful second action with probability 0.1 and the target with probability 0.9.

This establishes that continuation mismatch can reverse a policy-gradient direction, even with the exact target value used as baseline. It does not establish that this mechanism occurred in the archived training runs.

A second example uses an action-independent reward and a nonzero baseline error. The on-policy expected score contribution is zero, whereas a behavior success probability 0.1 and synthetic probability 0.9 produce a spurious logit gradient of -0.8 without weighting. An exact baseline removes this term; an imperfect critic or normalization across heterogeneous states can expose it.

For independent Bernoulli decisions, the population effective sample size fraction of trajectory weights is asymptotically

ESS / N -> [p^2/q + (1-p)^2/(1-q)]^(-H),

where q is the behavior probability, p the target probability and H the horizon. With q=0.5, p=0.55, it is about 0.8195 at horizon 20 and 0.0783 at horizon 256. With q=0.6, p=0.8, it falls to 0.0458 and 7.27e-18. These are illustrative independent-decision calculations, not estimates for the actual benchmarks. Exact IS can introduce severe variance. Clipping, self-normalization and truncated trace corrections trade bias against variance.

Frozen-policy measurements¶

The script ran eight new stochastic episodes for each of 20 final checkpoints (160 factual rollouts), at seeds 812000000 + 10000 * training_seed + episode. It uses the actual relabeler, measures total variation and KL between factual and synthetic action distributions, and reports action and full-episode likelihood-weight diagnostics. It restores the legacy automaton permutation selected for the paper's re-evaluation from reevaluation.json; default canonical ids can evaluate a different policy. Checkpoint hashes are in the recorded output.

Whole-episode relabelling matches Territory's aligned 256-step training chunks. For Stag Hunt it is an episode diagnostic; training used 32-step chunks across 20-step episodes. The diagnostic does not reproduce that training minibatch composition, GAE, or the PR2 critic update.

For Territory IPPO-CER, the largest average action-distribution TV across the sampled agent/seed/synthetic-start groups is 0.0005254. Its smallest empirical episode ESS fraction is 0.9995008. For base IPPO those numbers are 0.00003755 and 0.9999797. The final actors are almost insensitive to the monitor id on these trajectories. Missing action weights consequently have little effect at these final snapshots; this does not constrain mismatch earlier in training.

Territory's DFA has one unresolved state and absorbing success/failure states. From an episode-aligned factual start, CER yields the factual trajectory and one trajectory starting in each absorbing state. All have equal loss weight: exactly two thirds of the samples are synthetic absorbing-state trajectories. Their terminal rewards are fixed independently of actions. They can train values, but they offer no new action-dependent information about winning the race. With exact values and suitable normalization their expected actor contribution can vanish; finite sampling, critic error, cross-state advantage normalization and entropy still change the update. This is a concrete distribution/variance alternative to the action-weight hypothesis, not a demonstrated cause of the performance drop.

In Stag Hunt PR2-IPPO-CER, the strongest measured action mismatch occurs in a successful seed (seed 3, player 1), rather than in the three failed seeds. Its synthetic start 2 has average TV 0.1 and log mean episode weight about -1684.24. All eight observed trajectories carry essentially the same tiny weight, so empirical ESS is 8/8 despite terrible importance coverage. Reporting ESS without weight scale or coverage diagnostics would be misleading. No final-checkpoint measurement establishes what happened during training.

Controlled training study to isolate the cause¶

Start with Territory/IPPO and Stag Hunt/PR2-IPPO, keeping the temporal reward, observations, rollout shape, environment budget, optimizer initialization and update count matched. Keep the existing thresholds fixed; use new training seeds and disjoint evaluation seeds. The minimum useful arms are:

1. Objective-trained no CER.
2. Current uniformly weighted CER.
3. CER for critic updates, with factual actor/entropy samples and factual-only actor advantage normalization.
4. CER with a separate clipped behavior-to-synthetic action correction and, for IPPO, off-policy return targets such as V-trace. Label this as a truncated, biased correction rather than unbiased PPO.

Add an actor arm that excludes absorbing-state synthetic samples, or preserves the factual actor's total weight while bounding the synthetic contribution, to separate replay dilution from importance mismatch. One-step IS alone is a useful diagnostic arm, not a complete off-policy correction. Monitor-state encoding changes should be a separate ablation.

Log factual and synthetic actor-gradient norms/cosines, per-monitor-state values and advantage distributions, action-distribution KL/TV, action and trace weight scales, clipped fractions, and ESS during training. Compare per-agent satisfaction, minimum satisfaction, fixed-threshold certification, learning speed and walltime. Synthetic samples reuse physical evidence and are not independent new environment interactions. More useful value backups do not imply universal improvement of approximate policy optimization, general-sum game outcomes, or joint threshold satisfaction.

This training study remains unrun. The investigation supports doing it without claiming in advance that corrected CER will win on every environment.

Reproduction and validation¶

# Exact examples without checkpoints.
uv run python scripts/investigate_cer_bias.py

# Trusted archived working-copy inputs; write new outputs separately.
uv run python scripts/investigate_cer_bias.py \
  --exports /path/to/exports --episodes 8 \
  --output exports/optimisations/cer-bias/diagnostics.json

# Read archived evaluations without running policy rollouts.
uv run python scripts/investigate_cer_bias.py \
  --exports /path/to/exports --episodes 0 \
  --output exports/optimisations/cer-bias/reports.json

Tests check the corrected gradient against an independent finite-difference derivative, the IS second moment, stable ESS for extreme log weights, and the legacy-numbering guard. The real checkpoint runs exercise environment construction, parameter restoration, proposition extraction and relabelling.

Primary references: LCER, PPO, reward-machine counterfactual learning, and V-trace/IMPALA. LCER itself describes counterfactual trajectories as off-policy and distinguishes the policy-gradient integration from the Q-learning construction. Its empirical successes are not a universal-dominance guarantee for this multi-agent finite-trace adaptation.

Preliminary controlled-training results¶

A dated interim snapshot records the first 16 completed full profiles from the frozen 612145f campaign. All 155 planned profiles and scientific settings remain unchanged. The snapshot is incomplete and cannot establish method rankings.

For Territory seed 100, uncorrected CER gives probabilities (.924, .064), while corrected CER gives (.229, .733); neither jointly certifies the fixed (.25, .20) thresholds. For seed 101, these are (.054, .707) and (.923, .031), respectively. Thus the weaker-agent improvement on seed 100 reverses on seed 101. The no-CER controls are (.807, .178) and (.274, .709).

The same-seed Stag Hunt PR2 seed 100 results are (0, 0) for uncorrected, corrected and critic-only CER, versus (.024, .024) without CER and (1, 1) with utility-preserving RM shaping. These single-seed outcomes are descriptive; the remaining fixed seeds are needed before any comparative conclusion.

Seed 100 training diagnostics continue to show very small sampled action importance mismatch in Territory (synthetic ESS fractions essentially one). Stag Hunt's uncorrected CER factual/synthetic actor-gradient cosine is negative for player 0 in 95 of 98 valid diagnostic windows, and remains negative in 94 of 98 windows under clipped correction. Sampled importance weights mostly remain near one. Gradient magnitude matters: these cosines are actor-term diagnostics, can include near-collapsed policies, and neither measure the true gradient nor establish that missing importance sampling caused failure. The snapshot retains median gradient norms for interpretation.

Complete targeted full-budget ablation, 13 October¶

All fifty targeted profiles completed: five matched seeds, five modes, for Territory/IPPO and Stag Hunt/PR2-IPPO. The complete report retains raw success counts, thresholds, finite-parameter validation, source/configuration hashes, artifact paths, and independently recomputed payoff-ceiling certificates. All profiles use source 612145f, 1,000 fresh unshaped evaluation episodes, and the original fixed scientific settings. Territory trains for 200,192 steps and Stag Hunt for 500,224. This table excludes half-budget screens.

The mean minimum averages each seed's weaker-agent probability estimate. Certificate counts here use the original task-level Hoeffding correction: family 25 for Territory and 40 for Stag Hunt, with fixed thresholds (.25, .20) and (.90, .90), respectively.

Original CER has a lower estimated minimum than no CER on every Territory seed. Correction improves three seeds relative to original CER, but its mean is slightly lower, and it certifies no Territory profile. In Stag Hunt, correction and critic-only replay recover the no-CER certification count; they do not exceed it. RM shaping succeeds on all five Stag Hunt seeds but trails no CER in Territory. These are five-seed comparisons, not precise population rankings or proof that one bias mechanism caused the differences. Clipped correction still leaves state-distribution and approximation effects.

Payoff-ceiling equilibrium evidence¶

The existing certify_payoff_ceiling verifier was applied without changes to these raw counts, using one-sided exact binomial bounds and correction over all 155 planned CER-study profiles at delta .05. Both agents of every Stag Hunt RM profile have 1,000/1,000 successes. Their simultaneous exploitability upper bound is .0086943, so all five certify joint satisfaction and initial state epsilon-.05 NE in the original satisfaction-utility game. This follows from bounded utilities (gain_i <= 1 - p_i), not from an unsuccessful finite response search. Other Stag Hunt modes certify respectively 2/5 (no CER), 1/5 (CER), 2/5 (corrected), and 2/5 (critic-only).

The payoff-ceiling bound is inconclusive for all Territory profiles. Broader paper-wide selection, archived candidates and future full confirmations need a recomputed combined comparison family; the current report's scope is the 155-profile study. Neither the half-budget screens nor separate PR2 discount reruns are pooled into this full-budget table.

Remaining work and numerical failure¶

At the dated 07:58 UTC check, the staged study has 100/155 completed results: these fifty full-budget profiles and fifty of 105 half-budget screens. The approved staging manifests preserve whole matched comparisons, all five seeds, full schedule horizons, original evaluation budgets and pinned worker sources. See staging protocol. CER's admission maximum is now 14, with unchanged global ceiling 185. Static full-budget summarizers cannot describe the new staged paths.

The separate PR2 gamma extension is paused for numerical diagnosis. Three PR2-IPPO-CER trials trained nonfinite parameter states: Classic2/.99/full (job 61924), Classic3/1/half (61996), and Capture3/.99/half (62019). The verifier correctly rejected their profiles before reporting satisfaction or stability. An independent checkpoint audit of 61924 found all 51,998 player-1 parameters nonfinite and player 0 entirely finite. Its SHA256 is 83432430c17049f7643f712c8582981125eaab810c94fe8b14a80d7acd17e790. The precise failing numerical operation remains unresolved; these are not queue or pre-training import errors. Failed artifacts remain preserved.