Skip to content

Reward optimisation assumptions requiring review

This file records interpretation choices that can materially change which experiments count as successful. Move an item into the relevant campaign note once it has been resolved.

Success aggregation

  • The initial screen treats an environment as promising when the mean frozen- policy estimate across seeds is at least 0.5 for at least one agent. A single lucky seed is not sufficient.
  • The final validation should use three seeds, 1,000 satisfaction rollouts per seed, and report the estimate and Hoeffding interval for every agent. The requested p >= 0.5 gate is currently interpreted as a gate on the point estimate; interval-based certification is reported separately because the existing verifier does not emit threshold certificates.
  • The experiment-spec success_threshold remains 0.5. Lower thresholds are not introduced to make an environment pass. A domain-specific exception, such as a stochastic pursuit role with a genuinely low attainable probability, must be justified explicitly and does not relax the separate requirement that the best agent reaches p >= 0.5.

Scope

  • "All environments" now means the 8 curated launcher IDs in notebooks/experiments/scripts/support.bash. Plain Gift Refinements Consume was removed after a predeclared random baseline reached 0.912--0.948. Food and Food-Then-XY were removed after all declared objective, reward, algorithm, discount, representation, and exploration controls failed the 0.5 gate; the separate opposed-cycle replacement failed too. Refined-Then-Consume was removed after every completed 250k IPPO profile failed and its strongest long profile stayed at or below 0.176 for every fixed agent over two 1M seeds; even a perfect third seed could not lift any three-seed mean to 0.5. These notebooks remain runnable by direct path but are not meaningful optimisation targets. Alternative notebooks are not additional required environments unless one replaces another curated task that is not viable.
  • The primary optimisation algorithms are IPPO_CER and PR2_IPPO_CER. PR2_IPPO is an optional diagnostic. Nash-Q is reserved for Matrix Stag Hunt because the other environments are not practical tabular problems. The cluster launcher's feasible=True profile encodes this scope by omitting all tabular Nash-Q variants from spatial tasks while retaining them for sustained_stag_2; it does not change task objectives or algorithm logic.

Objective design

  • Per-agent instances of the same formula schema are treated as distinct when they use agent-indexed propositions, for example F player_i_food_metabolized. An aggregate proposition that rewards every learner for one agent's event is not acceptable.
  • The retained-suite general-sum audit uses objective ownership and substrate externalities together. Pursuit has an opposed guard/intruder split (two intruders may share a role objective, but the guard does not); Coins has collector-owned colour goals over consumable coins; Safe Harvest has agent- owned collection/safety goals with finite apples and strategic zapping; and Territory is a mutually exclusive first-claim race. None of these credits every learner when one learner succeeds. The removed Chemistry and Refined- Then-Consume tasks also used agent-owned goals and genuine externalities; they were removed for failed trainability, not for being cooperative games.
  • Matrix Stag Hunt retains its established shared Sustained Stag objective for the first screen because it is a pre-existing paper benchmark with known Nash-Q and IPPO-CER results. Human review is requested on whether this shared objective should remain an explicit exception to the preference for non-identical satisfaction payoffs.
  • PECaptureVsStealth currently gives intruders a safety/stealth objective, G !guard_intruder_collision & G !collision_ever. A former source comment incorrectly said "reach goal" and was corrected during this audit; the paper, class name, handcrafted reward, and reported results consistently describe stealth without a reachability conjunct. Human review is requested before changing the scientific task.
  • Historical Territory Role Claim gave both agents semantically identical satisfaction conditions even though their source strings were reordered. The optimisation pass replaces this with a first-sole-claim race: each agent wins by claiming before the opponent, while a simultaneous first claim is a tie. Shaping is equal-and-opposite and resolves at that first claim step. This is a constant-sum competitive task, not a shared-payoff construction. All earlier Role Claim artifacts are invalid for the revised task; only the fresh race screen can count. Human review is requested on whether the retained role_claim_2 launcher name is sufficiently clear or should be renamed in a later compatibility migration.
  • The Territory proposition player_i_resource_claimed now denotes the substrate's explicit ownership-change event, not a later stochastic native payout to the owner. This matches the proposition name and removes an accidental reward-rate dependency from the temporal objective. Human review is requested if any historical paper result intentionally treated delayed payout as the claim timestamp; those artifacts must not be compared as if they used the revised atom.
  • Safe Harvest now treats an agent's native zap target event as the safety violation. The historical punishment_received atom cannot occur under the benchmark's default partnership roles because only the unassigned putative_cooperator role receives the hidden-tile penalty. Human review is requested before comparing the revised zap-avoidance task with paper numbers produced from the inert punishment condition.
  • OpposedCycles_4_mo.py was evaluated as a replacement candidate, not an easier interpretation of Food-Then-XY: it assigns agent-owned X and Y cycle targets whose products inhibit the opposing cycle. Default IPPO, higher-entropy IPPO, and PR2 all failed their random baselines and the 0.5 gate, so neither candidate is retained. Human review is requested before re-adding either Chemistry task or substituting a different objective.

Experiment isolation

  • Screening and validation outputs use pass-specific directories. Early jobs used exports/optimisations/; later cluster jobs use short names under exports/o/ to keep generated Marimo artifact paths below filesystem limits. Neither location overwrites the canonical paper exports.
  • Seeds remain the ordinary contiguous sequence starting at zero. They are not selected based on performance, and all submitted seeds are included in the aggregate.