Skip to content

Experiment Hyperparameter Tuning

The training notebooks use visible apply_cli_overrides({...}) config cells as their source of truth. The April 2026 tuning pass keeps that public shape, but moves away from leaving rollout length implicit through num_steps=None.

Shared Default

src/experiments/temporal_game_experiment.py still derives num_steps from num_env_steps, progress_chunks, and num_envs when a notebook leaves it as None, but the cap is now 256 instead of 512.

This shared fallback is intentionally conservative. Current training notebooks set explicit num_steps values because each environment family has a different episode horizon and runtime profile.

progress_chunks also controls tqdm redraw cadence for shared temporal-game training runs. The callback still receives every trainer update, but the notebook progress bar only calls tqdm.update(...) at configured milestone boundaries, for example 10% increments when progress_chunks=10. This avoids Marimo/Jupyter output spam and truncation during long PPO runs.

Notebook Presets

The tuned defaults target a balanced faster workflow:

  • Matrix Stag Hunt uses 500_000 steps, three runs, short 32-step rollouts, and eight cheap parallel envs.
  • Pursuit uses 1_000_000 steps for two-player notebooks, 1_500_000 for three-player notebooks, 64-step rollouts, and four cheap parallel envs.
  • Coins uses 200_000 steps, one environment, and 256-step rollouts.
  • Chemistry uses 200_000 steps and 128-step rollouts.
  • Territory and Harvest keep single-env native-gridworld rollouts, use 200_000 steps, and use 256-step rollouts.
  • Gift Refinements keeps one environment and 256-step rollouts. Consume uses a 512-step horizon and 400_000 training steps; the harder RefinedThenConsume objective uses a 1000-step horizon and 200_000 steps.

The IPPO minibatch choices keep the rollout batch divisible by num_minibatches. For IQL and PR2-IQL, the replay sample size is kept near 1024 transitions by pairing each family-specific num_steps value with a matching buffer_batch_size.

Parameters Left Stable

Core optimizer settings remain consistent across the notebook presets:

  • PPO learning rate, clipping, value/entropy coefficients, and GAE lambda, except the validated Safe Harvest profile described below.
  • IQL and Nash-Q learning rate, epsilon endpoints/decay fraction, and Nash-Q minimum alpha.
  • Most suites use gamma = 0.99; OwnCoinFirst_2, RoleClaim_2, and Consume_6 use gamma = 1.0 for their finite-horizon terminal objectives.

Safe Harvest defaults to its agent-local handcrafted reward, gamma=gae_lambda=1, and linear IPPO learning-rate annealing. Its three-seed 1M-step validation reached fixed player-1 mean satisfaction 0.5987 with minimum seed 0.585, versus the same role's 0.582 random baseline. These settings address long-horizon reward credit and late-update stability; frozen evaluation still uses the unchanged objective formula.

The other values were not the source of the runtime mismatch. The main issue was that copied notebook configs used long training budgets and a generic num_steps=None fallback that resolved to 512 even for short-horizon tasks.

RoleClaim_2 and OwnCoinFirst_2 do not discount terminal satisfaction across their complete 256-step traces. RoleClaim_2 originally combined a 1000-cycle horizon with gamma = 0.99, which made its terminal-only satisfaction reward too sparse. Its notebook now uses a 256-cycle finite horizon. Consume_6 also uses gamma = 1.0 across its 512-step horizon.

Guardrails

tests/test_notebook_experiment_configs.py parses the training notebooks without executing them and checks that the visible defaults remain explicit and batch-compatible. If a future notebook intentionally changes these presets, update that test and this note together.