Skip to content

Checkpoint Re-evaluation

Re-scores saved checkpoints with the current evaluation code without training. It exists to repair evaluation bugs (the 2026-09-30 Pursuit horizon fix in final-satisfaction.md), to add fields to old reports (per-rollout bits, certification), and to draw a fresh confirmatory rollout batch for selection-free certification.

Switches

Shared config keys, all False by default and declared in the eight curated notebooks so they can be passed on the command line:

  • reevaluate: skip training; re-score every completed run in the manifest.
  • reevaluate_nashconv: also recompute ApxNashConv (retrains the finite-budget best responses, so it costs about as much as the original NashConv stage).
  • confirmatory_satisfaction: also evaluate a second, disjoint batch of satisfaction_eval_episodes rollouts per seed.

  • reevaluation_tag (string, default empty): write every output under <algorithm>/reevaluation/<tag>/ and leave the main artifacts untouched, so several candidate re-evaluations of one profile can run concurrently.

  • reevaluate_satisfaction (default True): False re-scores only NashConv (requires reevaluate_nashconv=True, rejects confirmatory_satisfaction). Satisfaction artifacts and reevaluation.json are left untouched, a legacy automaton numbering is reused from reevaluation.json's numbering_identification.permutation (no probes), and the outcome goes to reevaluation_nashconv.json.

  • objective_nashconv (default False, only affects hc_* profiles): score ApxNashConv in temporal satisfaction like the objective-trained profiles. Baseline and deviation rollouts use the objective-reward environment (episode success = automaton acceptance) and the deviating agent trains on that reward with COUNTERFACTUAL_REPLAY=True. The previous handcrafted-return scores are kept once as nash_handcrafted_reward.npy/nash_handcrafted_reward_report.json, and nash_report.json metadata records nashconv_score (objective_satisfaction or handcrafted_return). Training-time NashConv, which drives auto-resume, is unchanged: handcrafted profiles still use their own reward there.

These keys are listed in EVALUATION_ONLY_CONFIG_KEYS and excluded from the partial-resume fingerprint. The fingerprint hashes the whole resolved config, so without the exclusion adding the keys would have invalidated every existing export's manifest and the next resume would have wiped it.

Behaviour

_run_temporal_game_algorithm_experiment branches to _reevaluate_algorithm_checkpoints before the partial-resume loader, so the loader's artifact clearing never runs in this mode. The branch:

  1. Requires partial_resume_manifest.json and a fingerprint equal to the one computed from the current config. Pass the same notebook arguments used for training (for the paper campaign: num_runs=5 calculate_satisfaction=True; the launcher adds resume_partial_runs=True and just_vis=False). A mismatch raises before any rollout and leaves artifacts untouched.
  2. Copies the first evaluation's satisfaction.npy/satisfaction_report.json (and NashConv files with reevaluate_nashconv) into legacy_evaluation/ once; later re-evaluations never overwrite that record.
  3. For every completed run, loads checkpoints/<label>.pkl, rebuilds the run config as training did (algo_config with SEED=run_seed) and re-runs satisfaction on the original seeds (run_seed * 10^6 + 503 + i). Where the rollout semantics did not change, the counts reproduce exactly.
  4. Confirmatory rollouts use run_seed * 10^6 + 500503 + i (CONFIRMATORY_SATISFACTION_SEED_OFFSET), disjoint from the primary satisfaction (+503), NashConv (+7) and AlphaRank (+701) blocks.
  5. NashConv, when requested, reuses the original seeds (+7) and replaces the entries at the manifest's nash_index positions.
  6. Writes satisfaction.npy, satisfaction_report.json, optional nash.npy/nash_report.json, confirmatory_satisfaction.npy, confirmatory_satisfaction_report.json, and reevaluation.json (legacy vs new success counts and NashConv per run, a satisfaction_changed flag, and the confirmatory joint verdict). The manifest, runs.npy, checkpoints and timings are not modified, so ordinary resumes keep reusing the runs.

Checkpoint train states are Flax state dicts; rollouts read params through train_state_params, and best-response trainers restore them with restore_train_state, so no TrainState reconstruction is needed.

Automaton numbering and identification

Legacy checkpoints were trained under a hash-seed-dependent automaton state numbering (automaton-state-numbering.md), and the same checkpoint behaves differently under another numbering. Bundles that record monitor_state_numbering are evaluated under it. For legacy bundles (no record) the branch recovers the training numbering automatically (_identify_legacy_numbering):

  1. Read the objective formulas from one objective env and compute, for hash seeds 0..REEVALUATION_PROBE_HASH_SEEDS-1 (160), the flloat permutation of canonical state ids each seed would produce (flloat_numbering_permutations, one short subprocess per seed). Distinct permutations are the task's numbering classes (Pursuit Capture 1, Pursuit Classic and Stag Hunt 2, Coins and Territory 4, Harvest 16).
  2. With more than one class, replay each class in-process (use_monitor_state_numbering(permutation)) on the training job's own NashConv baseline rollouts (same rollout function, env and seeds run_seed * 10^6 + 7 + i, eval_episodes episodes per run) and count per-seed, per-agent baseline success counts that equal the legacy baseline_p * eval_episodes. A class matching every count stops the scan; otherwise the class with the most matches wins (earliest on ties, which only occur when the policies ignore the differing ids). For Pursuit Classic only the guard's counts can match, because the horizon bug inflated the intruders' legacy baselines; the guard's G !goal_reached bit is unaffected by it.
  3. Evaluate everything under the winning permutation and record numbering_identification (number of classes, chosen hash seed, matches, every probe's counts), monitor_numbering fingerprints and python_hash_seed in reevaluation.json.

The 64-episode baseline probe cannot separate classes that differ only in rarely visited automaton states, and the default scan stops at the first class matching every count. reevaluation_primary_tiebreak=True (evaluation-only, default off) probes every class and, when several share the best score, re-scores each tied class on the full primary satisfaction rollouts against the original evaluation's per-seed, per-agent counts; the identification records these under tiebreak. It costs one full primary evaluation per tied class, so it is used only where the integrity audit shows a profile that does not reproduce its original counts. In the campaign that happened for Coins PR2-IPPO-CER (seed 4, player 1: 374 vs 381) and Pursuit Classic 3 hc-IQL (guard off by one on two seeds), both of which had stopped at the first full baseline match.

legacy_successes and the legacy NashConv baselines always come from the first evaluation (legacy_evaluation/ when present).

Protocol caveat

Auto-resume decisions made during the original campaign used the NashConv estimates available at the time. For Pursuit those estimates were inflated by the horizon bug, so the saved final checkpoints may have received (or missed) the 25% extra budget for a different reason than the corrected diagnostic would give. Re-evaluation certifies these frozen profiles as they are; it does not replay the auto-resume rule.

Launching

Use the ordinary launchers with the training arguments plus the switches, e.g. on cpucluster:

bash scripts/pltlf/env.sh classic_2 amd96 bundles=8 feasible=True \
  walltime=24:00:00 num_runs=5 calculate_satisfaction=True \
  reevaluate=True reevaluate_nashconv=True confirmatory_satisfaction=True

Then aggregate with uv run python scripts/certification_summary.py --exports exports --json <out.json>; it prints per-profile joint-certification counts, task- and campaign-wide family-wise counts, confirmatory counts, worst-agent joint lower bounds, ApxNashConv and exclusivity violations.

2026-09-30 campaign re-evaluation

All 66 curated profiles (5 seeds each) were re-scored on cpucluster with reevaluate=True confirmatory_satisfaction=True, plus reevaluate_nashconv=True for Pursuit, after the horizon fix and canonical numbering (commits 3c9126f to 52bac3c). The 200-job per-user submit cap was full with another project's jobs, so a login-node script (~/logs/pltlf_reeval/trickle_v*.sh) submitted one bundled job at a time as slots freed. A first round without numbering identification was discarded.

  • Numbering identification: Pursuit Capture tasks have a unique numbering; every other profile's chosen class matched all legacy NashConv baseline counts except where only the Pursuit Classic guard could match (intruder baselines were inflated by the horizon bug). No best-scoring tie disagreed on counts (certification_summary.py column "Numbering").
  • Stag Hunt, Coins: primary satisfaction reproduced the original counts (Coins pr2_ippo_cer seed 4 differed by one rollout). Stag Hunt IPPO-CER, IQL-CER, PR2-IQL-CER, Nash-Q-CER and hc-PR2-IQL are jointly certified on 5/5 seeds on both batches and after task- and campaign-wide Bonferroni corrections; hc-Nash-Q 3/5, PR2-IPPO-CER 2/5, hc-PR2-IPPO 1/5.
  • Pursuit: no seed profile is jointly certified (guard and intruder objectives exclude each other); the largest worst-agent joint lower bound is 0.339 (Capture 2 IQL-CER). The legacy "certified" Pursuit seeds were horizon-bug artifacts. No rollout in any task violates a declared exclusivity group.
  • Coins: all eight profiles identified with 10/10 matches over four classes; none is certified on either batch (largest worst-agent bound 0.474).
  • Territory (jobs 9117-9120, finished 2026-10-01): every profile identified its numbering with 10/10 matches over four classes, reproduced the original counts exactly, and certifies no seed on either batch (largest worst-agent bound 0.325).
  • Harvest (jobs 9079-9086, finished 2026-10-02): every job's early-stopped identification matched 20/20 baseline counts; seven profiles reproduce their original counts exactly. IQL-CER did not (1-6 rollouts per seed) and was re-evaluated with reevaluation_primary_tiebreak=True (job 9659, 47 h, finished 2026-10-04): the tie-break chose hash seed 11 (20/20 baseline matches over 16 classes), every seed reproduces its original counts exactly, and no seed is jointly certified on the confirmatory batch. The paper's tables were regenerated without overrides (commit b19e393).
  • Integrity audits (check_reevaluation_integrity.py, jobs 9594, 9621 and 9660): the first flagged Coins PR2-IPPO-CER, and a guard-count cross-check flagged Pursuit Classic 3 hc-IQL. Both had stopped at the first full baseline match and were fixed by tie-break reruns (jobs 9603 and 9604): the chosen classes, from hash seeds 3 and 2, reproduce all original counts. The final audit (job 9660, exports/integrity_audit_final2.json) passes all 66 CER and handcrafted profiles. Its 33 FAILs are exactly the no-CER ablation profiles, for reasons that follow from training them after the campaign start (artifact times, no legacy numbering to identify, backups of their own fresh evaluation, NashConv computed in training, eight still training); the job therefore exits non-zero by design.

scripts/regenerate_paper_figures.py (job pltlf_figs) rebuilt the Pursuit paper panels into exports/paper_figures/, because handcrafted-baseline markers move with the corrected satisfaction reports. Job 10856 (2026-10-03) then rebuilt all eight suites with the 3% display smoothing (package.md); the paper (commit 13c175d) uses those panels.

No-CER ablation campaign (2026-10-03)

The paper claims every learner family with and without CER, but the campaign had trained only the -CER and hc- profiles. The 33 missing no-CER profiles (ippo, pr2_ippo, iql, pr2_iql on all eight tasks, plus nashq on Stag Hunt) were queued on cpucluster by ~/logs/pltlf_ablation/trickle.sh. For each profile it submits a training job (env.sh <task> amd96 algorithms=<algo> bundles=1 ablations=True num_runs=5 calculate_satisfaction=True, walltime 7 days for Harvest, 2 for Stag Hunt, 4 otherwise) and a re-evaluation job with the same arguments plus reevaluate=True confirmatory_satisfaction=True, made dependent on training (--dependency=afterok) through the SUBMIT_CMD wrapper ~/logs/pltlf_ablation/submit_after.sh. Job pairs are logged in ~/logs/pltlf_ablation/submitted.txt. New checkpoints record canonical automaton numbering, so the re-evaluation needs no identification; it only adds the confirmatory batch the -CER profiles have. The integrity audit's --reeval-start check flags these profiles' training artifacts as newer than the re-evaluation campaign; audit them with a later start time or skip that check for them.

All 33 profiles finished on 2026-10-05 (the four Harvest confirmatory re-evaluations last, at about 20:00); every primary batch reproduces its training-time counts. Under the curated thresholds the base variants are close to their -CER counterparts overall, with notable differences in both directions: base IPPO certifies 4/5 Territory seeds (CER 0/5), base PR2-IPPO 4/5 Stag Hunt seeds (CER 2/5), and base IQL reaches 0.919 in Territory (CER 0.696), while CER is clearly stronger for Harvest PR2-IPPO, Territory PR2-IQL and Pursuit Capture 2 IPPO. The paper (commit 2cf8af3) now picks each task's selected profile from base and -CER together (Classic 3 and Territory switch to base IQL) and lays the certification table out as one row per reward set; including the base variants raises the campaign family size from 330 to 495.

A notebook kernel can die after a job has saved its results and leave the Slurm job RUNNING with nothing left to do: Harvest PR2-IQL training (job 11167) died with signal 7 (SIGBUS) while plotting on 2026-10-04, after writing all five runs, and blocked its afterok re-evaluation. Check the log for "kernel ... died", confirm the manifest lists five completed runs with satisfaction and NashConv evaluated, then scontrol update jobid=<reeval> dependency=afterany:<train> and scancel <train>.

Handcrafted ApxNashConv in temporal satisfaction (2026-10-04)

All 33 hc_* profiles were re-scored with reevaluate=True reevaluate_nashconv=True reevaluate_satisfaction=False objective_nashconv=True (~/logs/pltlf_hc_nashconv/, jobs from 11829; all completed the same day, Harvest in about 2 h each). Every directory has reevaluation_nashconv.json with objective_nashconv: true, a nash_report.json labelled objective_satisfaction, and its handcrafted-return scores in nash_handcrafted_reward_report.json; every legacy profile reused the numbering recorded in reevaluation.json. Mean ApxNashConv stays small (at most 0.059): it rose for Pursuit Capture 2 hc-PR2-IQL (0.016 to 0.059), Capture 3 hc-PR2-IQL (0.010 to 0.046) and Coins hc-IPPO (0.013 to 0.050), fell to 0 for Pursuit Classic 2 hc-IPPO, Capture 3 hc-IQL and Territory hc-IPPO, and is 0 for every Stag Hunt profile. The paper quotes no handcrafted ApxNashConv value; its method text was updated in paper commit 51b2728. Integrity audit job 12011 (exports/integrity_audit_final3.json) passes all 33 CER and all 33 handcrafted profiles, the latter including the re-scored NashConv checks; its 33 FAILs are exactly the no-CER ablation profiles, for the same expected reasons as job 9660.

On 2026-10-10 the curated profiles (including legacy_evaluation/), the paper panels and integrity_audit_final3.json were archived; the earlier audit JSONs and every non-paper export were deleted. See paper data archive.

Integrity audit

scripts/check_reevaluation_integrity.py --exports exports --json <out> audits every curated algorithm directory against the campaign's failure modes. It checks that training artifacts are untouched (manifest, runs.npy and checkpoints older than --reeval-start; checkpoints load with their seeds) and that numbering identification was recorded with no ambiguous tie. It checks that primary and confirmatory reports are internally consistent (seed blocks, bit counts, Hoeffding intervals, joint outcome counts, zero exclusivity violations, recomputed certificates, .npy against JSON), and that reevaluation.json agrees with the reports and the original evaluation in legacy_evaluation/. It also checks that NashConv files are re-scored only for Pursuit and that no tagged reevaluation/<tag>/ directories remain. Unfinished re-evaluations are reported as pending; the exit code is non-zero on any failure. It loads every checkpoint, so run it as a cluster job.