Skip to content

Coins Environment

Coins is a native two-player PettingZoo gridworld implemented under src/environments/substrates/coins/. Its stable factory is make_coins_env(...) in src/environments/gridworlds/coins/env.py.

Dynamics And Observations

Each episode procedurally samples a map between min_map_size and max_map_size (defaults 10 and 15), assigns each player a distinct coin colour, and regrows empty coin locations with regrow_rate (default 0.0005). Seven actions cover no-op, four relative moves, and two turns.

Matching and mismatched collection both give the collector +1 by default. Mismatched collection also gives the partner -2, creating the social dilemma. The native event payload records the collector, player colour, coin colour, and whether they matched.

Like the other native factories, Coins supports raw structured observations, normalized flat object vectors without rendering, and global/local RGB modes. The temporal-game experiment adapter uses the flat object path.

Labels And Objective

CoinsLabelMixin exposes aggregate and per-agent collection atoms:

  • coin_collected
  • matched_coin_collected
  • mismatched_coin_collected
  • player_i_coin_collected
  • player_i_matched_coin_collected
  • player_i_mismatched_coin_collected
  • player_i_harmed_by_partner

It also follows the shared timeout, terminal, and stochastic_termination lifecycle convention.

CoinsOwnCoinFirstConstraint gives each player:

(!player_i_mismatched_coin_collected) U player_i_matched_coin_collected

The finite trace must avoid taking the partner's coin until the player takes its own. CoinsProsocialConstraint and CoinsClassicConstraint are compatibility aliases for the same selected objective. Their handcrafted baseline teaches the underlying colour preference at every collection: the collector receives +1 for its own colour or -1 for the other colour. Partner-harm labels never affect learner reward. Repeating the collector-local signal after the first decisive event gives PPO denser colour-selection credit; frozen success evaluation still uses only the first-collection formula.

This co-safe ordering objective was selected because its accepting state can be reached early. It gives exploration a realistic chance of producing positive terminal objective rewards, while still making the first collection decision strategic in the underlying general-sum game. Both agents receive the same formula schema with their own agent-indexed atoms.

Alternatives Considered

  • (F player_i_matched_coin_collected) & (G !player_i_mismatched_coin_collected) expresses stronger, episode-long prosocial behavior. It was not selected because one exploratory mismatch irreversibly rejects an entire 256-step trace, which is likely to produce a nearly all-zero objective signal early in training.
  • F player_i_matched_coin_collected is an easier reachability objective, but it does not use temporal ordering and gives no reason to distinguish a matching-first policy from an initially harmful policy.
  • Retaliation and repair formulas based on player_i_harmed_by_partner are strategically richer, but they encourage lower-welfare trajectories and make satisfaction depend heavily on whether enough trace remains for a response.
  • A shared no-harm objective would turn the monitored game toward identical interests. The selected agent-indexed objective keeps satisfaction payoffs separate: movement, coin consumption, and access to matching coins couple whether each player's distinct formula can be satisfied. In objective mode, the monitor replaces native rewards with terminal satisfaction rewards; the native +1/-2 mismatch externality remains part of the environment dynamics but is not copied into the objective-aligned handcrafted reward.

Port Note

Runtime code was copied from meltingpot-pz commit 7d7af52. The structured observation mapping was corrected during integration to distinguish all five possible coin_* colours; the source helper only assigned explicit categories to red and blue, which would hide some sampled player and coin types from the no-render training policy.

Experiment Surface

CoinsExperimentSpec and run_coins_experiments(...) expose the complete 15 algorithm suite, including CER and handcrafted variants.

notebooks/gridworlds/coins/OwnCoinFirst_2_mo.py uses a 256-step horizon, regrow_rate=0.005, 200,000 environment steps, and gamma=1.0 for IPPO, IQL, and Nash-Q. It defaults to the validated local-observation, collector-local handcrafted reward profile with IPPO lr=1e-4. Across three 1M-step seeds, fixed player 1 reached mean satisfaction 0.510 with minimum seed 0.502, versus its 0.498 random baseline. This is a narrow point-estimate pass; the formula and objective-mode frozen evaluator remain unchanged. Artifacts live under exports/gridworlds/coins/own_coin_first/2/<algorithm>/; the curated launcher ID is own_coin_first_2.