Skip to content

Artifacts

Experiment artifacts are written under each notebook's configured save_dir. The repository ignores exports/, so generated experiment outputs do not need to be committed.

Directory Layout

Suite runs write one subdirectory per selected algorithm:

save_dir/
  ippo/
  ippo_cer/
  pr2_ippo/
  pr2_ippo_cer/
  iql/
  iql_cer/
  pr2_iql/
  pr2_iql_cer/
  nashq/
  nashq_cer/
  hc_ippo/
  hc_pr2_ippo/
  hc_iql/
  hc_pr2_iql/
  hc_nashq/

Only selected algorithms are created. Environment-specific helpers use the same convention.

Per-Algorithm Files

Each algorithm directory can contain:

  • runs.npy: raw run histories.
  • nash.npy: raw NashConv payloads.
  • nash_report.json: NashConv aggregate report.
  • satisfaction.npy: final satisfaction payloads.
  • satisfaction_report.json: final satisfaction aggregate report.
  • alpharank.npy: AlphaRank payloads when enabled.
  • alpharank_report.json: AlphaRank report when enabled.
  • checkpoints/: per-seed train-state bundles.
  • timing sidecars produced by src/experiments/experiment_timings.py.
  • objective satisfaction plots and legends.

The shared experiment helper writes algorithm artifacts as soon as that algorithm finishes. If a later algorithm fails, completed algorithm outputs remain usable.

Comparison Plots

The suite root is reserved for cross-algorithm comparison outputs, such as:

  • average-agent plots,
  • best-agent plots,
  • per-agent summary plots,
  • exported legends,
  • handcrafted baseline comparison tables when handcrafted runs are enabled.

These plots are generated from the returned histories and saved artifacts.

Partial Resume

When resume_partial_runs is enabled, each completed seed updates cumulative artifacts and a partial_resume_manifest.json.

On restart, the runner reuses only a valid contiguous prefix. It checks:

  • config fingerprint,
  • expected seeds,
  • manifest version,
  • required artifacts,
  • checkpoint bundles,
  • completed record metadata.

If the config changed or artifacts are inconsistent, the runner starts fresh or resumes from the first incomplete valid seed.

Generated Docs Output

The documentation build writes static HTML to:

site/

This directory is ignored by git. Rebuild it with:

uv run zensical build --strict