Skip to content

Algorithm Artifact Persistence

Experiment result artifacts are persisted by src/experiments/temporal_game_experiment.py inside _run_temporal_game_algorithm_experiment(...).

The helper writes each algorithm's artifacts immediately after that algorithm finishes all configured seeds and optional verification stages. In a multi-algorithm notebook, this means save_dir / "ippo" / "runs.npy" and the matching NashConv/satisfaction reports exist before the suite starts the next selected algorithm.

Per-algorithm artifacts written by the helper:

  • runs.npy
  • nash.npy
  • nash_report.json
  • satisfaction.npy
  • satisfaction_report.json
  • alpharank.npy and alpharank_report.json when AlphaRank is enabled

The notebook save cells should not duplicate these writes. Their responsibility is only to display returned artifact paths or produce plotting artifacts. This keeps interrupted multi-algorithm notebook runs recoverable: completed algorithms keep their result arrays and JSON reports even if a later algorithm fails or is stopped.

Incremental per-run persistence and resume

The shared temporal-game runner supports resume_partial_runs. The library default is False, while runnable experiment notebooks set it to True so a stopped multi-seed job can restart at the first incomplete seed.

The cluster training launchers also inject resume_partial_runs=True unless a caller explicitly disables it. The env_resume*.sh and all_envs_resume*.sh wrappers force it on and reject an override, providing a clear operational entry point for recovering interrupted scheduler jobs.

When enabled, each completed run immediately rewrites the cumulative runs.npy, nash.npy, nash_report.json, satisfaction.npy, and satisfaction_report.json artifacts, then records a validated prefix in partial_resume_manifest.json. Checkpoints continue to be written per run under checkpoints/run_<n>_seed_<seed>.pkl.

On the next invocation the runner reuses only a contiguous prefix whose manifest metadata, config fingerprint, expected seeds, checkpoint bundles, and required result artifacts match the current request. Config or metadata mismatches start fresh in the same save_dir; missing or corrupt artifacts keep the valid prefix and resume from the first incomplete run. AlphaRank remains an end-of-algorithm artifact, but skipped checkpoint bundles are loaded back as checkpoint-style train-state mappings so AlphaRank can evaluate the complete strategy set after the remaining runs finish.

Do not split one algorithm into concurrent per-seed jobs with the current artifact format. All seeds replace the same cumulative arrays and manifest; the manifest represents one contiguous run prefix. Safe scheduler-level seed fanout would require per-seed result files and an atomic merge/index layer.