Skip to content

Workflow

Most experiments follow the same loop whether they run in matrix games, Pursuit, or native PettingZoo gridworlds.

1. Pick A Benchmark

Choose an environment and objective pair from the notebook tree:

  • notebooks/matrix/stag_hunt/ for the repeated matrix game.
  • notebooks/gridworlds/pursuit/ for guard and intruder pursuit tasks.
  • notebooks/gridworlds/coins/ for the two-player own-coin-first task.
  • notebooks/gridworlds/chemistry/ for native chemistry tasks.
  • notebooks/gridworlds/harvest/ for commons harvest tasks.
  • notebooks/gridworlds/territory/ for resource-claim tasks.
  • notebooks/gridworlds/gift_refinements/ for gift and consume tasks.

Each training notebook constructs a temporal-game experiment spec and calls a shared suite helper.

The curated Python entrypoints are listed in the Experiments API and Environments API.

2. Configure Algorithms

The notebook config has top-level switches such as:

{
    "run_ippo": False,
    "run_ippo_cer": False,
    "run_iql": False,
    "run_nashq": False,
    "run_hc_ippo": False,
    "calculate_conv": True,
    "calculate_satisfaction": False,
    "resume_partial_runs": True,
    "nashconv_delta": 0.05,
    "satisfaction_delta": 0.05,
}

All algorithm switches default to False. Enable only the algorithms intended for the current local invocation; cluster launchers set only their selected run_* switches to True.

Algorithm-specific hyperparameters live in nested blocks such as ippo, iql, and nashq.

3. Train

The suite helper resolves config, creates the wrapped environment, and runs selected algorithms across configured seeds. Each algorithm writes its own subdirectory under save_dir, for example:

exports/gridworlds/territory/role_claim/2/ippo/
exports/gridworlds/territory/role_claim/2/ippo_cer/

Long multi-seed runs can opt into partial resume. When enabled, a contiguous completed prefix is reused only after its config fingerprint, expected seeds, artifacts, manifest, and checkpoint bundles validate. Do not split one algorithm's seeds across concurrent scheduler jobs; they share cumulative files.

4. Verify

Verification can run as part of training or in later analysis.

  • NashConv estimates unilateral deviation incentive and reports simultaneous Bonferroni-Hoeffding bounds across agents.
  • AlphaRank ranks learned policies through response graph dynamics.
  • Final satisfaction estimates each agent's objective satisfaction probability with marginal and simultaneous confidence intervals.

An optional NashConv auto-resume pass can continue a run once with a bounded fraction of its original training budget when the measured NashConv exceeds a configured threshold. This is separate from partial resume after interruption.

Verification is always performed against the objective reward path, even when a handcrafted baseline trained on shaped rewards.

See the Verification API for the report and estimator helpers.

5. Compare

The experiment helpers return per-algorithm results and cross-algorithm comparison histories. Notebook plotting cells write:

  • per-algorithm satisfaction curves,
  • cross-algorithm average-agent summaries,
  • best-agent summaries,
  • per-agent summaries,
  • legends for exported plots.

Use the generated JSON reports for exact numbers and the plots for trend inspection.

Handcrafted variants are evaluated against the same temporal objective, but their shaped-reward learning curves are excluded from the ordinary suite comparison. The best handcrafted final satisfaction appears as an endpoint marker instead.

6. Diagnose

Low NashConv does not automatically mean the project succeeded. A profile where everyone has zero satisfaction can also have low measured deviation incentive if short best-response training cannot find a better policy.

For each run, inspect:

  • objective satisfaction curves,
  • final satisfaction reports,
  • NashConv baseline and best-response probabilities,
  • per-agent formulas and labels,
  • whether episode horizons make the finite trace too sparse.