Skip to content

Handcrafted Rewards By Environment

This note records the handcrafted learner rewards used by the hc_* baselines. reward_mode="objective" keeps the original PLTLf terminal reward. In reward_mode="handcrafted", BoolRewardWrapper still advances the PLTLf monitors and logs cum_reward as objective satisfaction, but returns the handcrafted/native reward to the learning algorithm as learner_reward.

When a constraint does not provide a custom handcrafted function, handcrafted mode falls back to the native environment rewards returned by env.step().

Matrix Stag Hunt

  • SHSustainedStagConstraint: native Stag Hunt reward. This matches the temporal intent of repeatedly coordinating on stag while avoiding unmirrored play.
  • SHHandshakeConstraint: custom stateful reward. If actions are unmirrored, both agents get -1.0. Each both_hare event gives both agents +0.5 and records that the hare phase happened. The first later both_stag after hare gives both agents +1.0.
  • SHBothModesConstraint: custom stateful reward. If actions are unmirrored, both agents get -1.0. The first both_hare gives both agents +1.0, and the first both_stag gives both agents +1.0.

Coins

  • CoinsOwnCoinFirstConstraint, CoinsProsocialConstraint, and CoinsClassicConstraint reward the collector's colour preference at every collection: matched gives +1.0, mismatched gives -1.0. The signal is agent-local and deliberately ignores partner-harm labels. Repetition makes colour selection learnable while the objective evaluator still resolves the unchanged until formula at the first decisive collection.

Pursuit

  • PEClassicConstraint: custom role reward. The guard, player_0, gets -1.0 if goal_reached occurs and +1.0 at episode end if the goal was prevented. Intruders get +1.0 when goal_reached occurs and -1.0 on any collision.
  • PECaptureVsStealth: custom role reward. The guard gets +1.0 on guard_intruder_collision and -0.5 at episode end if no capture happened. Intruders get -1.0 on guard capture or any recorded collision, and +1.0 at episode end if they survived without collision/capture.
  • PEIntruderCoordination and PEGapStrategy: no custom handcrafted function is attached, so handcrafted mode uses the native environment reward.

Harvest

  • CHHarvestConstraint: native reward. The native apple collection reward matches the F {agent_id}_apple_collected objective.
  • CHSafeHarvestConstraint and CHClassicConstraint: custom agent-local safe- apple reward. The first apple while still unzapped pays +1; the first zap pays -1 and cancels an earlier apple payment. Later apples cannot restore a failed (G !player_i_zapped) & (F player_i_apple_collected) trace.

Territory

  • TerritoryClaimConstraint: native reward for resource claim.
  • TerritoryPeacefulClaimConstraint: custom reward equal to native reward plus a shared -1.0 penalty whenever all_zappers_ready is false.
  • TerritoryRoleClaimConstraint: custom competitive role reward. The first sole claim gives that player +1.0 and the opponent -1.0; repeat labels are not paid. A simultaneous first claim is a zero-reward tie. This matches the mutually exclusive first-sole-claim race without a shared team signal.

Chemistry

  • ChemFoodRewardConstraint: bounded custom per-agent reward. Each of the six own-agent X/Y cycle vesicle reactions pays +0.1 at most once per episode; the agent's own food metabolism pays +1.0 once. Unowned ground reactions, other agents' events, and the unrelated native XY payoff are ignored.
  • ChemXYOnlyConstraint: native reward for XY metabolism.
  • ChemFoodThenXYConstraint and ChemClassicConstraint: custom stateful reward per agent. They use the same six bounded own-agent reaction-progress payments, the first food metabolism gives max(1.0, native_reward) and marks food as seen, and the first later XY metabolism after food gives max(1.0, native_reward).

Gift Refinements

  • GiftRefinementsConsumeConstraint: native reward for token consumption.
  • GiftRefinementsRefinedThenConsumeConstraint and GiftRefinementsClassicConstraint: custom stateful reward per agent. The first refined gift receipt gives +1.0. The first token consumption after a refined gift receipt gives max(1.0, native_reward).

Regraph Awareness

notebooks/generic/regraph_mo.py loads suites through notebooks/utils/regraph.py. The regraph utility maps hc_* export directories back to mixed-case algorithm labels, reloads satisfaction_report.json into satisfaction_summary, skips per-algorithm objective-reward plot exports for handcrafted algorithms, and passes their reports and histories to the shared suite plotters. Full suite comparisons show only the strongest handcrafted result as a Best Handcrafted endpoint marker, while *_no_hc variants omit it.

When a suite contains handcrafted_baseline_comparison.txt, the generic regraph notebook displays that saved Rich table text after the satisfaction-summary table.