Skip to content

Algorithms

The project compares independent learners, opponent-aware variants, a tabular game-theoretic baseline, handcrafted-reward baselines, and counterfactual replay variants.

IPPO

Independent Proximal Policy Optimization trains one PPO learner per agent. Each learner treats the other agents as part of the environment. IPPO is a common deep multi-agent baseline and is available for matrix and gridworld experiments.

Useful config keys include:

  • num_envs
  • num_steps
  • num_minibatches
  • update_epochs
  • lr
  • gamma
  • gae_lambda
  • clip_eps
  • ent_coef

IQL

Independent Q-Learning trains one value function per agent. The repo supports MLP and recurrent network variants through the IQL config surface.

Useful config keys include:

  • network_kind
  • num_envs
  • num_steps
  • buffer_size
  • buffer_batch_size
  • learning_starts
  • target_update_interval
  • eps_start
  • eps_finish
  • eps_decay

PR2 Variants

PR2 variants model or condition on opponent behavior when choosing actions. The repo exposes:

  • PR2_IPPO
  • PR2_IPPO_CER
  • PR2_IQL
  • PR2_IQL_CER

Notebook configs usually reuse the visible ippo or iql block for PR2 runs instead of introducing separate PR2-only blocks.

The discrete PR2 critics enumerate opponent joint actions. Their output grows exponentially with the number of agents: for action size A and N agents, each critic head contains A**N values. PR2-IQL is therefore unsuitable for the six-player, nine-action Gift Refinements benchmark without a factorized or sampled redesign.

Nash-Q

Nash-Q is the tabular stage-game baseline. It stores Q-values by joint observation key and solves stage games to select equilibrium policies.

Important config keys include:

  • eps_start
  • eps_finish
  • eps_decay
  • min_alpha
  • equilibrium_selection
  • counterfactual_max_joint_variants

Nash-Q is most natural for small discrete spaces. It is intentionally more constrained than the deep algorithms.

Stage games are solved with pygambit. The default payoff_dominant selection ranks returned equilibria by total expected Q value and then the minimum per-agent expected Q value; first keeps solver order. Solver failure raises instead of silently falling back to a uniform policy.

Counterfactual Experience Replay

Counterfactual experience replay, abbreviated CER, is enabled by algorithm variants with _CER in their names:

  • IPPO_CER
  • PR2_IPPO_CER
  • IQL_CER
  • PR2_IQL_CER
  • NASHQ_CER

CER uses the LTLf automaton to relabel a factual trace from alternative monitor states. The environment observations and actions stay fixed, while the appended monitor-state component and terminal objective reward are recomputed.

For IPPO and PR2-IPPO, CER remains a biased heuristic: actions were sampled under the factual monitor history, and recomputing their probabilities under a relabeled history does not recover an on-policy occupancy distribution. Use the non-CER variants as the canonical PPO baselines; there is currently no importance-weight correction for the PPO CER variants.

CER requires temporally extended monitor observations. It is specific to finite-trace LTLf monitors in this repo; it is not a Buchi jump-transition construction.

Handcrafted Baselines

Handcrafted baselines use names such as:

  • hc_IPPO
  • hc_PR2_IPPO
  • hc_IQL
  • hc_PR2_IQL
  • hc_NASHQ

These runs train on native or custom shaped rewards when available, but still evaluate the temporal objective through the same monitor and satisfaction reporting path. They are useful for comparing PLTLf terminal reward training against more direct task rewards.

Handcrafted variants always run final objective-satisfaction evaluation. Their suite comparisons report only the strongest handcrafted-reward result as a Best Handcrafted endpoint marker. The corresponding *_no_hc exports omit that marker.

Algorithm Selection

For a first pass:

  • Use IPPO for the simplest deep baseline.
  • Add IPPO_CER or IQL_CER when the objective is sparse and the monitor has useful alternative states.
  • Use final satisfaction alongside NashConv when diagnosing whether low deviation incentive is meaningful.
  • Treat Nash-Q as a small-state baseline rather than a default for high-dimensional native object-observation vectors.