Skip to content

Gift Refinements Experiments

Gift Refinements is a fixed six-player native PettingZoo substrate under src/environments/substrates/gift_refinements/. The factory rejects other player counts and exposes the historical substrate name gift_refinements.

Observation And Label Contract

Structured mode returns categorical grids plus inventory/readiness fields. Experiment adapters flatten and normalize these fields without rendering. Visual mode uses GiftRefinementsRGBObservationWrapper; in that mode the policy sees normalized RGB, while infos retain namespaced inventory, readiness, and collective reward.

GiftRefinementsLabelMixin provides:

  • lifecycle atoms;
  • aggregate and per-agent token-consumption atoms with sticky forms;
  • aggregate and per-agent refined-gift-receipt atoms with sticky forms;
  • gift-given and gift-received event atoms;
  • inventory and beam-readiness atoms.

Consumption is positive native reward. Refined receipt is detected from native gift events with an inventory-delta fallback. Events are read from native info["events"] dictionaries.

Objectives

  • GiftRefinementsConsumeConstraint: F player_i_tokens_consumed
  • GiftRefinementsRefinedThenConsumeConstraint: F (player_i_refined_gift_received_ever & player_i_tokens_consumed)

Consume is an agent-specific reachability smoke benchmark. Do not replace it with aggregate F tokens_consumed, which credits all agents for one player's consumption. It is no longer part of the curated all-environment launcher: uniform random actions already satisfy it for 0.912--0.948 of 512-step episodes across the six roles, so learned success above 0.5 is not meaningful reward-optimisation evidence. The notebook remains available by direct path for environment smoke checks. RefinedThenConsume is a temporal and strategic direct-run task whose corresponding 1,000-step random baseline is only 0.084--0.102 across agents. It is also no longer curated: every completed 250k IPPO profile failed, and the strongest 1M profile stayed at or below 0.176 for every fixed role over its first two seeds. Even a perfect third seed could not lift any three-seed fixed-role mean to 0.5. Its notebook and APIs remain available for diagnostics.

The temporal formula never contains equilibrium atoms. NashConv and AlphaRank remain post-training policy-profile analyses.

Scaling Constraint

The environment has nine actions. A six-player PR2-IQL critic therefore has 9**6 = 531,441 joint outputs per agent and can be impractical even though the ordinary IPPO/IQL paths are supported. See PR2-IQL Joint-Action Scaling.

Direct-Run Notebooks And Defaults

  • Consume_6_mo.py: 512-step horizon, 400,000 environment steps, gamma=1.0.
  • RefinedThenConsume_6_mo.py: 1000-step horizon, 200,000 environment steps.
  • play_mo.py, play_labelled_mo.py, and test_mo.py cover interaction, labels, and spaces.

The nine-action play mapping includes movement, turning, refineAndGift, and consumeTokens. Artifacts live under exports/gridworlds/gift_refinements/<objective>/6/<algorithm>/.