Skip to content

Algorithms

The project compares independent learners, opponent-aware variants, a tabular game-theoretic baseline, handcrafted-reward baselines, and counterfactual replay variants.

IPPO

Independent Proximal Policy Optimization trains one PPO learner per agent. Each learner treats the other agents as part of the environment. IPPO is a common deep multi-agent baseline and is available for matrix and gridworld experiments.

Useful config keys include:

  • num_envs
  • num_steps
  • num_minibatches
  • update_epochs
  • lr
  • gamma
  • gae_lambda
  • clip_eps
  • ent_coef

IQL

Independent Q-Learning trains one value function per agent. The repo supports MLP and recurrent network variants through the IQL config surface.

Useful config keys include:

  • network_kind
  • num_envs
  • num_steps
  • buffer_size
  • buffer_batch_size
  • learning_starts
  • target_update_interval
  • eps_start
  • eps_finish
  • eps_decay

PR2 Variants

PR2 variants model or condition on opponent behavior when choosing actions. The repo exposes:

  • PR2_IPPO
  • PR2_IPPO_CER
  • PR2_IQL
  • PR2_IQL_CER

Notebook configs usually reuse the visible ippo or iql block for PR2 runs instead of introducing separate PR2-only blocks.

Nash-Q

Nash-Q is the tabular stage-game baseline. It stores Q-values by joint observation key and solves stage games to select equilibrium policies.

Important config keys include:

  • eps_start
  • eps_finish
  • eps_decay
  • min_alpha
  • equilibrium_selection
  • counterfactual_max_joint_variants

Nash-Q is most natural for small discrete spaces. It is intentionally more constrained than the deep algorithms.

Counterfactual Experience Replay

Counterfactual experience replay, abbreviated CER, is enabled by algorithm variants with _CER in their names:

  • IPPO_CER
  • PR2_IPPO_CER
  • IQL_CER
  • PR2_IQL_CER
  • NASHQ_CER

CER uses the LTLf automaton to relabel a factual trace from alternative monitor states. The environment observations and actions stay fixed, while the appended monitor-state component and terminal objective reward are recomputed.

CER requires temporally extended monitor observations. It is specific to finite-trace LTLf monitors in this repo; it is not a Buchi jump-transition construction.

Handcrafted Baselines

Handcrafted baselines use names such as:

  • hc_IPPO
  • hc_PR2_IPPO
  • hc_IQL
  • hc_PR2_IQL
  • hc_NASHQ

These runs train on native or custom shaped rewards when available, but still evaluate the temporal objective through the same monitor and satisfaction reporting path. They are useful for comparing PLTLf terminal reward training against more direct task rewards.

Algorithm Selection

For a first pass:

  • Use IPPO for the simplest deep baseline.
  • Add IPPO_CER or IQL_CER when the objective is sparse and the monitor has useful alternative states.
  • Use final satisfaction alongside NashConv when diagnosing whether low deviation incentive is meaningful.
  • Treat Nash-Q as a small-state baseline rather than a default for image-based Melting Pot experiments.