Skip to content

Pursuit Multi-Algorithm Notebooks

The four pursuit Marimo notebooks call src/experiments/pursuit_experiment.py:run_pursuit_experiments(...) instead of directly calling the IPPO-only helper.

Notebook config shape

Each notebook keeps one top-level config passed through apply_cli_overrides(...).

Top-level shared keys:

  • num_agents
  • num_env_steps
  • seed
  • num_runs
  • progress_chunks
  • run_ippo
  • run_ippo_cer
  • run_pr2_ippo
  • run_pr2_ippo_cer
  • run_iql
  • run_iql_cer
  • run_pr2_iql
  • run_pr2_iql_cer
  • run_nashq
  • run_nashq_cer
  • run_hc_ippo
  • run_hc_pr2_ippo
  • run_hc_iql
  • run_hc_pr2_iql
  • run_hc_nashq
  • calculate_conv
  • calculate_alpharank
  • calculate_satisfaction
  • calculate_conv_quick
  • eval_episodes
  • br_train_iters
  • br_batch_steps
  • satisfaction_eval_episodes
  • satisfaction_delta
  • nashconv_delta
  • nashconv_auto_resume
  • nashconv_auto_resume_threshold
  • nashconv_auto_resume_step_fraction
  • resume_partial_runs
  • just_vis
  • save_dir

Algorithm-specific settings live in nested visible blocks:

  • ippo = {...}
  • iql = {...}
  • nashq = {...}

PR2_IPPO reuses the visible ippo block and PR2_IQL reuses the visible iql block. There are no extra notebook-facing pr2_* hyperparameter blocks.

Only the top-level keys are CLI-overridable. Nested ippo / iql / nashq entries remain notebook-edited defaults.

Every supported run_* toggle must be present in the notebook config, even if it defaults to False, because apply_cli_overrides(...) rejects unknown top-level keys.

calculate_alpharank is wired through the notebooks but defaults to False; users must explicitly enable the potentially expensive cross-play stage.

Artifact layout

save_dir is the root comparison directory.

Pursuit notebooks use the gridworld export namespace:

  • exports/gridworlds/pursuit/classic_2
  • exports/gridworlds/pursuit/classic_3
  • exports/gridworlds/pursuit/capture_2
  • exports/gridworlds/pursuit/capture_3

Automaton diagram exports follow the same namespace under exports/diagrams/gridworlds/pursuit/..., with the same flattened suite names.

Run ../scripts/migrate_pursuit_export_paths.sh from inside exports/ to copy artifacts from the pre-flattened pursuit directories into the current locations. Runtime code does not look in the legacy paths.

Per-algorithm outputs are written beneath it:

  • save_dir / "ippo"
  • save_dir / "ippo_cer"
  • save_dir / "pr2_ippo"
  • save_dir / "pr2_ippo_cer"
  • save_dir / "iql"
  • save_dir / "iql_cer"
  • save_dir / "pr2_iql"
  • save_dir / "pr2_iql_cer"
  • save_dir / "nashq"
  • save_dir / "nashq_cer"
  • save_dir / "hc_ippo"
  • save_dir / "hc_pr2_ippo"
  • save_dir / "hc_iql"
  • save_dir / "hc_pr2_iql"
  • save_dir / "hc_nashq"

Each algorithm subdir owns its own:

  • checkpoints
  • timings sidecar
  • runs.npy
  • nash.npy
  • nash_report.json
  • satisfaction.npy
  • satisfaction_report.json
  • optional alpharank.npy
  • optional alpharank_report.json
  • objective_satisfaction_by_agent.png
  • objective_satisfaction_by_agent_legend.png

The shared experiment helper writes the array/report artifacts immediately after each algorithm finishes all configured runs. Notebook cells should not re-save those artifacts after the full suite completes.

The root directory is reserved for cross-algorithm comparison artifacts:

  • avg_agent_best_algos.png
  • avg_agent_best_algos_legend.png
  • avg_agent_all_algos.png
  • avg_agent_all_algos_legend.png
  • best_agent_best_algos.png
  • best_agent_best_algos_legend.png
  • best_agent_all_algos.png
  • best_agent_all_algos_legend.png
  • summary/agent_1_best_algos.png, summary/agent_1_all_algos.png, etc.
  • a *_no_hc.png and *_no_hc_legend.png sibling for every suite-root and per-agent summary comparison above, excluding the Best Handcrafted marker
  • a *_tightened.png and *_tightened_legend.png sibling for each full-scale comparison, using an adaptive probability axis that retains the 0.5 reference

Suite return shape

run_pursuit_experiments(...) returns:

  • config
  • selected_algorithms
  • experiments
  • comparison_runs
  • save_dir
  • num_agents

experiments is keyed by algorithm label:

  • "IPPO"
  • "IPPO_CER"
  • "PR2_IPPO"
  • "PR2_IPPO_CER"
  • "IQL"
  • "IQL_CER"
  • "PR2_IQL"
  • "PR2_IQL_CER"
  • "NASHQ"
  • "NASHQ_CER"
  • "hc_IPPO"
  • "hc_PR2_IPPO"
  • "hc_IQL"
  • "hc_PR2_IQL"
  • "hc_NASHQ"

Each value keeps the same shape as the single-algorithm helper returns.

When AlphaRank is enabled, each per-algorithm result also includes:

  • alpharank_result
  • alpharank_summary

Combined comparison runs

comparison_runs is a relabeled history structure for the configured objective-trained algorithms. Suite plots use handcrafted satisfaction reports to add the strongest result as a single Best Handcrafted endpoint marker; *_no_hc variants exclude that marker.

Each run dict uses keys like:

  • IPPO/player_0
  • IPPO/player_1
  • IPPO_CER/player_0
  • IPPO_CER/player_1
  • PR2_IPPO/player_0
  • PR2_IPPO/player_1
  • PR2_IPPO_CER/player_0
  • PR2_IPPO_CER/player_1
  • IQL/player_0
  • IQL/player_1
  • IQL_CER/player_0
  • IQL_CER/player_1
  • PR2_IQL/player_0
  • PR2_IQL/player_1
  • PR2_IQL_CER/player_0
  • PR2_IQL_CER/player_1
  • NASHQ/player_0
  • NASHQ/player_1
  • NASHQ_CER/player_0
  • NASHQ_CER/player_1

The suite notebooks use plot_suite_satisfaction_comparisons(...) to write average-agent, best-agent, and per-agent summary plots. Its best-algorithm panels use plot_best_ewma_success_by_algorithm(...), which:

  • averages EWMA satisfaction across agents within each run,
  • compares base vs CER variants inside each algorithm family,
  • keeps the better variant for display,
  • reuses the base algorithm colour for CER winners.

The per-algorithm plots are still one curve per agent within one algorithm. Pursuit role objectives usually make these curves distinct.

The suite notebooks also use plot_all_ewma_success_by_algorithm(...) for the all-algorithm panels. It shows base and CER variants together, using dashed lines for CER variants.

Helper split

The module exposes 16 notebook-relevant helpers: one for each selectable algorithm plus the suite orchestrator.

  • run_pursuit_ippo_experiment(...)
  • run_pursuit_ippo_cer_experiment(...)
  • run_pursuit_pr2_ippo_experiment(...)
  • run_pursuit_pr2_ippo_cer_experiment(...)
  • run_pursuit_iql_experiment(...)
  • run_pursuit_iql_cer_experiment(...)
  • run_pursuit_pr2_iql_experiment(...)
  • run_pursuit_pr2_iql_cer_experiment(...)
  • run_pursuit_nashq_experiment(...)
  • run_pursuit_nashq_cer_experiment(...)
  • run_pursuit_hc_ippo_experiment(...)
  • run_pursuit_hc_pr2_ippo_experiment(...)
  • run_pursuit_hc_iql_experiment(...)
  • run_pursuit_hc_pr2_iql_experiment(...)
  • run_pursuit_hc_nashq_experiment(...)
  • run_pursuit_experiments(...)

The suite helper is only orchestration. Each algorithm still uses its own trainer and its own NashConv routine, so approximate best-response search stays algorithm-aware.

AlphaRank follows the same pattern: each algorithm reuses its own rollout implementation while building the cross-play empirical game from the saved run snapshots.

Comparison title

The comparison EWMA plot title is derived from " vs ".join(selected_algorithms) instead of being hardcoded to IPPO vs IQL. That keeps the combined chart correct when only one algorithm is enabled or when any mix of the ordinary PLTLf-trained variants joins the comparison run. Handcrafted baselines are summarized by the Best Handcrafted endpoint marker.