Skip to content

Handcrafted Baselines

Handcrafted baselines are implemented as hc_* algorithm variants in the shared temporal-game experiment runner. The supported labels are:

  • hc_IPPO
  • hc_PR2_IPPO
  • hc_IQL
  • hc_PR2_IQL
  • hc_NASHQ

Suite configs enable them with run_hc_ippo, run_hc_pr2_ippo, run_hc_iql, run_hc_pr2_iql, and run_hc_nashq. Defaults are False, so existing notebooks keep their previous behavior unless a baseline is selected.

Reward Streams

BoolRewardWrapper now supports reward_mode="objective" and reward_mode="handcrafted".

  • objective is the previous behavior: learners receive terminal PLTLf satisfaction rewards.
  • handcrafted returns native environment rewards or a constraint-provided handcrafted reward function to the learner.
  • In both modes, temporal monitors still update every step and episode logs keep cum_reward as PLTLf objective satisfaction. This preserves the meaning of existing satisfaction plots.

Handcrafted runs additionally record learner_cum_reward and handcrafted_cum_reward in episode logs, and step infos include objective_reward, learner_reward, and handcrafted_reward.

Evaluation

Handcrafted algorithms always force calculate_satisfaction=True, even when the suite config disables satisfaction for PLTLf-trained algorithms. The runner uses the handcrafted reward mode for training and NashConv/AlphaRank, but switches the evaluation environment back to reward_mode="objective" for final PLTLf satisfaction verification.

Suite plot helpers use the available hc_* satisfaction summaries to add only the strongest handcrafted-reward result as a Best Handcrafted endpoint marker. Handcrafted runs are not drawn as individual learning curves or included in best-family selection. The *_no_hc siblings omit the marker.

Artifacts

When a suite includes at least one hc_* run, it writes:

handcrafted_baseline_comparison.txt

in the suite export directory. The Rich table compares available final satisfaction scores, highlights the best PLTLf algorithm when PLTLf satisfaction was calculated, and highlights the best handcrafted baseline.

Reward Defaults

Native rewards are used when they naturally match the temporal intent, such as Harvest collection, Territory claim, sustained Stag Hunt, and simple Chemistry or Gift consumption goals.

Custom handcrafted reward functions are attached to constraints whose temporal intent differs from native rewards:

  • Coins OwnCoinFirst/Prosocial
  • Pursuit Classic and Capture
  • Stag Hunt Handshake and BothModes
  • Safe Harvest
  • Territory PeacefulClaim and RoleClaim
  • Chemistry FoodThenXY
  • GiftRefinements RefinedThenConsume

These functions are intentionally small, local to each constraint module, and stateful only where ordering matters.

See Handcrafted Rewards By Environment for the concrete reward values used by each current environment constraint.