Equilibrium evidence and independent response searches¶

The W1/W3 follow-up distinguishes three outputs: a true initial-state epsilon-Nash certificate, a statistically supported profitable deviation, and confidence bounds for the finite set of responses actually evaluated. A small learned-response score cannot upper-bound gains from unsearched policies. The experiment tolerance stays epsilon=0.05.

PPO optimizer repair¶

The former PPO/PR2-PPO response search restored the profile's Adam moments and optimizer counter, then replaced the rollout shape and total training budget. On Commons, the ordinary 782 trainer updates accumulate 12,512 optimizer updates. Interpreting that counter in a three-update BR annealing schedule sets the learning rate to zero. This was reproduced with a real Flax/Optax restore and gradient update.

fresh_ppo_best_response_states in src/verify/nash_conv/common.py retains all policy parameters and creates fresh Adam state for all agents. The BR trainer uses the original configured learning rate without annealing; only the target's parameters are updated. Resetting every optimizer keeps frozen opponents' serialized state compatible with the new optimizer structure. Ordinary training, checkpoint restoration and auto-resume are unchanged. Existing checkpoints remain valid, but archived annealed-PPO deviation scores must be treated as legacy diagnostics. Re-evaluation cannot replay original auto-resume decisions.

Payoff-ceiling certificate¶

For satisfaction utilities in [0,1], every unilateral gain is at most 1 - u_i(profile). Therefore

Exploit(profile) <= max_i(1 - utility_lower_i).

src/verify/equilibrium.py::certify_payoff_ceiling computes one-sided Clopper–Pearson bounds from each seed's episode counts. For upper-bound requirements, utility successes are the complement of satisfaction bits. The error budget is allocated over agents and every tested seed profile on the task. The same simultaneous event supports both threshold and equilibrium claims, without splitting its error budget. Strict thresholds remain strict. The claim concerns initial-state epsilon-NE, not subgame perfection.

For 1,000 successes in 1,000 episodes and 150 comparisons, the bound is 1 - (0.05 / 150)**(1 / 1000) = 0.007974.... This is below 0.05 and does not require best-response training. The method is uninformative for many otherwise stable profiles whose utilities are far from one; failure to obtain this certificate is not evidence of instability.

Cross-family search protocol¶

src/verify/deviation_search.py provides PolicySpec, PolicyActor, FrozenOpponentEnv, and mixed-profile rollouts. A single-agent trainer sees only the deviator, while opponents execute their original PPO or IQL policies. Each opponent retains its architecture, observations, reward-monitor context and recurrent episode memory. PPO samples its deployed policy; IQL deploys greedily. Opponents do not explore when the deviator explores. Explicit episode seeds reset independent per-agent action streams; unseeded training resets advance a separate seed generator instead of replaying the same action stream.

scripts/equilibrium_followup.py deviations searches with both PPO and IQL. The native family has a warm policy start with fresh optimizer state; both families also have three fresh restarts. Search rewards are terminal temporal acceptance with gamma=1 and CER disabled. Original architectures and learning rates are retained for matching-family starts. The original profile itself is an admissible candidate. Responses continue through 50k and 200k total search steps, rounded to complete 256-step batches, with exploration scheduled over the declared 200k horizon. Environments/replay restart at the budget boundary. The candidate pool for the larger budget also includes earlier candidates. Fresh responses additionally observe elapsed actions divided by the horizon. This clock is available from the agent's own action history and adds no privileged environmental information. Warm starts and frozen opponents retain their original input dimensions. The exact calibration uses Markov IQL and this public clock, so the search can represent time-dependent responses rather than being structurally limited to a stationary policy. The clock goes immediately before the final monitor-state coordinate. Replay and counterfactual utilities require that coordinate to remain last; appending a clock after it would silently round the clock as a monitor id in IQL replay.

Candidates are selected using 512 validation episodes. All selection finishes before the 8,192 held-out paired episodes are generated. Validation and test blocks start at run_seed * 1_000_000 + 700503 and +800503, disjoint from the original primary, confirmatory and NashConv evaluation blocks. Every profile/seed writes independent outputs and records the input checkpoint hash, monitor permutation, configuration, candidates, selection and paired bits.

Held-out paired differences lie in {-1,0,1}. Exact one-sided binomial bounds on the positive and negative probabilities give a two-sided interval for their difference, with four tail budgets per agent. This preserves pairing and can be much tighter than range-only Hoeffding bounds. Confidence is corrected over all selected original seed profiles and both reported BR budgets. Half the task error budget supports satisfaction and half supports deviation assessment. Legacy reports without paired counts retain their existing Hoeffding bounds.

A lower gain bound above epsilon supplies an instability witness. An upper gain bound below epsilon passes the empirical search assessment, conditional on the selected learned responses. It is explicitly not a true NE certificate or an upper bound over every trained/unsearched policy.

Exact calibration and reporting¶

src/verify/exact_pursuit.py enumerates the actual independent slippery moves, wall blocking, simultaneous collisions, event-delayed termination and finite horizon of two-player Classic Pursuit. It uses backward induction against public, time-dependent Markov opponents. Recurrent opponents with additional history are rejected by the calibration driver. The reduced 5×5, 12-step task retains the original Classic objectives; training remains model-free, while the evaluator knows the model. Both PPO and MLP-IQL original profiles and selected responses are evaluated exactly, exposing the search gap separately from rollout error. Terminal utilities are checked against the real wrapper for every action pair in a deterministic small instance.

Evaluator version 2 retains a goal-ever/collision-ever context and the actual final action after the first goal/collision, except when the horizon already truncates the episode. Final monitor labels can invalidate a Classic intruder's earlier goal success or a guard's earlier collision success. Scores from version 1 at source 300a7eb must be recomputed. The dense implementation in src/verify/pursuit_oracle.py agrees with the recursive reference on small stochastic games and audits the original 9×9 two-player Classic and Capture layouts over all 100 actions. It supports deployed PPO and MLP-IQL policies, including their DFA context and optional public clock. Exact responses cover all admissible history-dependent strategies against these public Markov opponents, up to floating-point arithmetic. Recurrent opponents are excluded.

All neural NashConv rollout entry points reject nonfinite model parameters before constructing an evaluation environment. This checks both original profiles and newly trained responses; heterogeneous PolicyActor deployment uses the same guard. PPO/IQL sampling or argmax on NaN logits can otherwise silently produce actions, and warm response training from a corrupt model can incorrectly appear to find zero improvement. Reject and preserve the artifact for diagnosis rather than reporting a small exploitability estimate. The full Pursuit audit records invalid archive profiles explicitly; continuation cannot warm-start them. These checks do not turn finite learned-response searches into unrestricted equilibrium certificates.

scripts/equilibrium_summary.py reads preserved reports only, prefers the confirmatory counts, and corrects across the full task family. The optional --replacement-exports overlay substitutes profiles by algorithm name without double-counting displayed denominators or changing the original exports. Superseded original profiles remain in the selection correction: replacing three five-seed Stag Hunt algorithms leaves 75 displayed profiles including handcrafted baselines, but corrects across all 90 examined profiles. Supply the complete original export root plus complete replacement reports when refreshing the paper. scripts/render_equilibrium_results.py produces the primary coverage table and satisfaction/certified-synthesis heatmaps. The older per-agent Hoeffding tables retain their original meaning; their counts need not equal the tighter exact-binomial counts.

The baseline audit is in ../experiments/results/equilibrium-baseline-2026-10-12.json. It finds 55/75 joint satisfaction-and-NE certificates on Stag Hunt including handcrafted profiles, or 46/50 among objective-trained profiles. The payoff-ceiling method certifies none on the other tasks. These baseline numbers must be refreshed when the independently authorized Nash-Q replacement campaign finishes.

Alternatives include empirical Bernstein bounds for paired gains, exact responses on larger enumerable games, and structured navigation/search responses on spatial tasks. These may improve precision or search coverage; none turns an unrestricted finite neural-policy search into an NE certificate.

Native response configuration provenance, 14 October 2026¶

Current native PPO response copies preserve COUNTERFACTUAL_REPLAY together with CER_UPDATE_MODE, the importance cap and diagnostic options. Published sources 57d6729, c2dbf27, de942e5 and ddb8ddf have this behavior. Frozen sources c84a075 and 612145f instead set COUNTERFACTUAL_REPLAY=False and CER_UPDATE_MODE="uncorrected" in both PPO response factories. Their recorded native-search scores therefore describe CER-free responses, including responses to CER-trained profiles. Preserve their scientific source pins and identify these scores separately; never describe them as variant-preserving response training.

The c84 matched discount controls disable auto-resume, so this response-copy distinction cannot alter their trained checkpoints, primary/confirmatory objective estimates or discount selection. The fifteen selected scalar-PPO profiles already have direct exact negative equilibrium results; another native response search cannot overturn those negatives and is unnecessary under the selective-search scope. Deliberately independent stronger-response protocols also disable CER, but have separate declared gamma-one/true-utility protocols and must not be conflated with native searches.