Reward optimisation change log¶

This is the maintained list of repository changes made for the reward- optimisation campaign that began at commit 912bf1a. It records durable changes rather than every submitted job or intermediate documentation update. Items are ordered by scientific and implementation impact, not chronology. Failed experimental profiles that changed no retained code are documented in reward-screen-2026-08-10.md and are intentionally not repeated here.

1. Repaired or retired benchmark tasks¶

Safe Harvest now has an attainable adversarial safety objective¶

• Replaced the inert per-agent punishment condition with (G !player_i_zapped) & (F player_i_apple_collected). The native benchmark roles never activated the old hidden-tile punishment, which made the former task almost automatic under random play.
• Extended HarvestLabelMixin with aggregate, per-agent, and sticky zap atoms decoded from native zap-target events.
• Added an agent-local handcrafted reward: first safe apple +1, first zap -1, and cancellation of an earlier apple payment after a later zap. Other agents' events are never copied into a learner's reward.
• Updated the notebook, environment notes, assumptions, and tests for the revised formula. Historical artifacts from the punishment formula are not comparable with the new task.

Territory Role Claim is now a competitive first-claim race¶

• Replaced the historical semantically shared role formulas with mutually exclusive player objectives: each player must make the first sole claim; simultaneous first claims are ties.
• Changed player_i_resource_claimed to use the substrate's explicit claimed_resource ownership event instead of a delayed stochastic native payout.
• Replaced shared shaping with an equal-and-opposite first-resolution reward: winner +1, loser -1, tie 0, and no later reward changes.
• Updated the active notebook, environment documentation, assumptions, and label/reward tests. Pre-change Role Claim artifacts are invalid for the revised benchmark.

The curated launcher suite dropped from 12 tasks to 8¶

• Removed consume_6: its fixed random baseline was already 0.912--0.948, so it was not a meaningful optimisation target.
• Removed food_4 and food_then_xy_4 after objective, shaped-reward, discount, algorithm, observation, and exploration screens all failed the p >= 0.5 gate. The opposed-cycle replacement candidate also failed.
• Removed refined_then_consume_6 after all completed 250k IPPO profiles failed and the strongest profile stayed below 0.176 for every fixed role over two 1M seeds. A perfect third seed could not raise any three-seed mean to 0.5. Three extremely slow PR2 seed-0 runs were stopped at 70% after they showed no sustained terminal-return signal; they are not counted as frozen results.
• Updated the canonical environment list, historical partition launcher, all-environment and resume launchers, job-count documentation, and launcher coverage tests. The removed notebooks remain runnable by direct path; no environment implementation or prior artifact was deleted.

2. Encoded validated optimisation profiles¶

Four-player Safe Harvest¶

• Set SafeHarvest_4_mo.py to default to the agent-local safe-apple reward, gamma=gae_lambda=1, and linear IPPO learning-rate annealing.
• The selected IPPO-CER validation reached fixed player-1 mean satisfaction 0.5987 across three 1,000,000-step seeds, with minimum seed 0.585; all three seeds beat the same role's 0.582 random baseline. The marginal mean interval is [0.5557, 0.6416].
• Kept global observations, the four agent-indexed safety/collection formulas, and objective-mode frozen evaluation unchanged. Fixed lr=1e-4 was rejected on lower best-agent mean (0.587) and lower worst seed (0.510).

Three-player Capture Pursuit¶

• Set Capture_3_mo.py to default to the existing formula-aligned handcrafted role reward. IPPO-CER validation reached fixed intruder means 0.883 across three 1,000,000-step seeds, with worst seed 0.822.
• Kept the opposed guard/intruder PLTLF formulas unchanged. Frozen satisfaction and equilibrium checks still reconstruct an objective-mode environment.
• Corrected a misleading source comment: the intruder task is trace-wide stealth/safety, not goal reachability.

Two-player Own Coin First¶

• Set OwnCoinFirst_2_mo.py to the validated local-observation profile with reward_mode="handcrafted" and IPPO learning rate 1e-4.
• Changed Coin shaping to repeat the collector-local colour preference on every collection: own colour +1, other colour -1. Removed partner-harm credit so one learner never trains directly on another learner's event.
• Kept the first-own-coin PLTLF formulas, 256-step trace, and regrowth dynamics unchanged. The selected player's three-seed 1,000,000-step mean is 0.510 with minimum seed 0.502; this is a deliberately narrow pass over the 0.498 same-role random baseline.

3. Added controlled optimisation surfaces¶

Reward-mode and hyperparameter overrides¶

• Added an explicit reward_mode="objective" visible default to every training notebook so objective and handcrafted runs can be selected through the shared CLI without editing notebooks.
• Extended apply_cli_overrides to support one-level nested fields such as ippo_gamma, ippo_lr, ippo_num_steps, and ippo_ent_coef, including collision/unknown-key validation and tests. This made one-factor cluster controls reproducible from launcher arguments.

Global/local observation controls¶

• Exposed each substrate's existing fully_observable setting through the Coins, Chemistry, Harvest, and Gift Refinements experiment configs and notebooks.
• Added tests that exercise native global and rotated local flat observation paths. This changes policy representation only; it does not add privileged state, centralise execution, or alter the objective evaluator.

Chemistry diagnostic rewards and candidate task¶

• Extended Chemistry labels with agent-owned vesicle-reaction atoms for the six X/Y cycle stages. Ground reactions are deliberately not assigned to an agent.
• Added bounded per-agent progress shaping (0.1 once per reaction stage, target milestone +1) for Food and Food-Then-XY wrappers.
• Added the standalone OpposedCycles_4_mo.py diagnostic and its constraint: alternating agents own X/Y completion goals whose products inhibit the opposing cycle, with bounded branch-local shaping.
• Retained these Python APIs and the direct notebook for diagnostics, but did not add the failed Chemistry tasks or candidate back to the curated suite.

4. Verification and durable records¶

• Added or expanded targeted tests for Harvest zap labels/rewards, Territory ownership events and race semantics, Coin shaping, Chemistry progress ownership, observation-mode configuration, nested CLI overrides, notebook presets, and curated launcher coverage.
• Added docs/AI/optimisations/assumptions.md for human-review decisions and docs/AI/optimisations/reward-screen-2026-08-10.md for job manifests, baselines, per-seed results, confidence intervals, rejected profiles, and validation decisions.
• Updated the relevant environment, handcrafted-reward, CLI, launcher, and user-facing environment notes to match the retained behavior.