CER checkpoint numerical audit and PPO overflow repair¶

On 2026-10-12, the independent equilibrium session found corrupt archived IPPO-CER guards in two-player Classic and Capture Pursuit, both seed zero. This session independently inspected the copied final checkpoints. Each guard (player_0) has 12/12 parameter leaves and 50,438/50,438 entries nonfinite, including both actor and critic. Each intruder has 12 finite leaves. The full cohort audit belongs to the equilibrium session, which reports 18/20 profiles finite; its numerically invalid profiles cannot certify a normalized learned policy.

The 20 final checkpoints copied for the original CER investigation all pass the same parameter check: 10 Territory/IPPO profiles and 10 Stag Hunt/PR2-IPPO profiles, covering both CER arms and seeds zero through four. This establishes finiteness only for those cohorts, not every Stag Hunt learner or Commons.

Original paths relative to /vol/gpudata/oja24-pltlf/exports/ are gridworlds/pursuit/{classic_2,capture_2}/ippo_cer/checkpoints/run_1_seed_0.pkl. The read-only archive and all working-copy inputs are preserved. Local input copies are under the equilibrium session's /tmp/pltlf-rational-refinement/.cache/pursuit-original-ppo/.

Both saved configurations use four environments, 64 rollout steps, four minibatches/epochs, learning rate 0.0003, no LR annealing, maximum gradient norm 0.5, tanh, gamma 0.99, lambda 0.95, clipping epsilon 0.2, value coefficient 0.5 and entropy coefficient 0.01. Capture records 249,856 extra auto-resume steps. Neither checkpoint declares a CER update mode or monitor numbering; legacy monitor permutations come from companion reevaluation metadata.

Reproduced numerical defect¶

The original PPO actor calculation forms ratio = exp(log_ratio) before min(ratio*A, clip(ratio)*A). With float32, A=1, epsilon 0.2 and log ratios 90 or 100, JAX returns the finite clipped loss -1.2 but a NaN gradient. The unused overflowing exponential's derivative is multiplied by zero. Ratios 10 and 80 produce the same finite loss and zero gradient.

This provides a plausible mechanism: CER includes actions generated under a different monitor-conditioned policy, so synthetic actions can be unlikely under their relabelled observations. A NaN actor gradient also contaminates the shared actor/critic gradient-norm clipping. It is not proof of the archived failure's cause. The final bundles do not contain earlier finite models, per-minibatch gradients or the first invalid update. The original experiment writer saves episode histories and final checkpoints rather than the trainer's complete per-update loss array. No claim that importance correction would have prevented these historical failures follows from this audit.

Mathematically equivalent actor calculation¶

The IPPO and PR2-IPPO actor surrogate now selects the clipping branch in log space before exponentiation:

A >= 0: selected_log_ratio = min(log_ratio, log(1+epsilon))
A < 0:  selected_log_ratio = max(log_ratio, log(max(1-epsilon, 0)))
surrogate = exp(selected_log_ratio + log_importance) * A

This equals the original clipped PPO surrogate. Epsilon values at or above one remain supported: their lower ratio bound does not restrict positive ratios. In corrected CER, stopped, clipped log importance weights are added before exponentiating, allowing a small importance factor and large policy ratio to have a representable product. Inactive actor samples use zero inputs before exponentiation, ensuring extreme synthetic ratios cannot poison a factual-only actor update.

The repair applies to all modes and changes no reward, utility, observation, information, importance cap, clipping rule or study hyperparameter. Ordinary values and gradients agree within floating-point tolerance. A genuinely unrepresentable active negative-advantage objective remains nonfinite and must be rejected; this implementation does not silently add a new ratio cap. CER's occupancy, trace and importance-clipping bias remain as documented in the study protocol.

Prevent invalid checkpoint evidence¶

scripts/run_cer_study.py:validate_model_parameters inspects every agent's actor/critic/response parameter leaves in live or serialized states. New models are checked before checkpoint publication; resumed final checkpoints are checked before satisfaction evaluation. Failures name the agent, parameter path and nonfinite count, and follow the study's explicit failure-artifact path. Valid model leaf/entry counts appear in metadata and result reports. This is needed even with the existing per-update loss check: the final optimizer update can corrupt parameters after its reported forward loss was evaluated.

The original zero-job controller was stopped before any study submission so the entire fresh cohort can use the repaired, validated source consistently. The deployment record identifies the replacement pin. Scientific budgets, seeds, thresholds, pilot gate, output isolation and admission limits stay fixed; the archived checkpoints are not retrained or overwritten.

Regression checks cover ordinary PPO values/gradients for both advantage signs, overflowing positive-advantage ratios, correction-weight cancellation, inactive synthetic gradients, live/serialized parameter inspection, and a finite-loss fake final update that corrupts its actor. The pipeline test confirms that such a model is neither published nor evaluated, and an already-saved corrupt model is also rejected before evaluation. Both learners' actual corrected and factual-only training loops remain covered by their existing tests.

Validation before publication: 82 focused tests, 472 routine tests selected from the working-tree changes, all 11 slow integrations, Ruff and strict documentation builds passed. The three extreme-ratio regressions also pass under JIT compilation, matching the learner's execution path.

Replay reset warning follow-up¶

A completed IQL calibration job reported invalid value encountered in cast when SequenceReplayBuffer converted its reset storage to Boolean. The buffer allocates capacity with np.empty; before it fills, entries after _size have never been initialized. The candidate builder cast the whole allocation before selecting valid indices. Signaling NaN bits in those unused entries reproduce the warning deterministically, even with clean inserted transitions. This warning alone is not evidence of nonfinite trained parameters or sampled data.

The candidate builder now casts only reset[:_size], matching the initialized prefix used by _valid_indices. When the ring is full, _size == capacity and every slot has been initialized, so wraparound indexing remains valid. The change preserves the candidate set, sampled data and RNG draws. It does not initialize the whole allocation or suppress errors from actual stored data. The regression poisons unused reset slots with float32 signaling NaNs, raises on invalid casts, and verifies successful sampling with the expected recurrent reset, padding and learning masks. Existing coverage also checks full-buffer wraparound and stream isolation.

This is a separate published-source follow-up. It does not change the frozen CER study pin 612145f, calibration artifacts, other campaigns or scientific settings.

Validation: all 32 trajectory tests and the 229 RL/verification tests selected by scripts/run_tests.py changed --include-slow pass, including slow learner integration checks. Ruff and the strict documentation build also pass.