Skip to content

Artifacts

Experiment artifacts are written under each notebook's configured save_dir. The repository ignores exports/, so generated experiment outputs do not need to be committed.

Directory Layout

Suite runs write one subdirectory per selected algorithm:

save_dir/
  ippo/
  ippo_cer/
  pr2_ippo/
  pr2_ippo_cer/
  iql/
  iql_cer/
  pr2_iql/
  pr2_iql_cer/
  nashq/
  nashq_cer/
  hc_ippo/
  hc_pr2_ippo/
  hc_iql/
  hc_pr2_iql/
  hc_nashq/

Only selected algorithms are created. Environment-specific helpers use the same convention.

Per-Algorithm Files

Each algorithm directory can contain:

  • runs.npy: raw run histories.
  • nash.npy: raw NashConv payloads.
  • nash_report.json: NashConv aggregate report.
  • satisfaction.npy: final satisfaction payloads.
  • satisfaction_report.json: final satisfaction aggregate report.
  • alpharank.npy: AlphaRank payloads when enabled.
  • alpharank_report.json: AlphaRank report when enabled.
  • checkpoints/: per-seed train-state bundles.
  • partial_resume_manifest.json: validated contiguous-prefix metadata when partial resume is enabled.
  • timings.json: stage durations and per-run hardware references.
  • timings_hardware.json with deduplicated CPU, GPU, memory-allocation, host, and scheduler descriptions.
  • objective satisfaction plots and legends.

The shared experiment helper writes algorithm artifacts as soon as that algorithm finishes. If a later algorithm fails, completed algorithm outputs remain usable.

Comparison Plots

The suite root is reserved for cross-algorithm comparison outputs, such as:

  • average-agent plots,
  • best-agent plots,
  • per-agent summary plots,
  • exported legends,
  • handcrafted baseline comparison tables when handcrafted runs are enabled.

The table is saved as handcrafted_baseline_comparison.txt. Handcrafted learning curves remain in their own algorithm directories; suite plots use their final satisfaction summaries only.

These plots are generated from the returned histories and saved artifacts.

Partial Resume

When resume_partial_runs is enabled, each completed seed updates cumulative artifacts and a partial_resume_manifest.json.

On restart, the runner reuses only a valid contiguous prefix. It checks:

  • config fingerprint,
  • expected seeds,
  • manifest version,
  • required artifacts,
  • checkpoint bundles,
  • completed record metadata.

If the config changed or artifacts are inconsistent, the runner starts fresh or resumes from the first incomplete valid seed.

All seeds for one algorithm update the same arrays and manifest. Running seeds concurrently in separate scheduler jobs is unsafe with the current format.

NashConv Auto-Resume

nashconv_auto_resume is different from partial resume. After a completed run is evaluated, the helper may continue that run once when its NashConv point estimate exceeds the configured threshold. The extra budget is capped by nashconv_auto_resume_step_fraction; the final checkpoint replaces the pre-continuation checkpoint, while auto_resume_events records the before/after values and actual extra steps.

Generated Docs Output

The documentation build writes static HTML to:

site/

This directory is ignored by git. Rebuild it with:

uv run zensical build --strict