Skip to content

Objective Thresholds

Each agent's objective is P_{>= q_i}[phi_i]. Until 2026-10-03 every curated task used q_i = 0.5; the paper now gives each agent a role-specific q_i. src/experiments/objective_thresholds.py is the single source: CURATED_PROBABILITY_THRESHOLDS maps a task's export path (relative to exports/) to {agent: q_i}, and curated_probability_thresholds(task) returns the {agent: (">=", q_i)} form the certification code takes.

Task Thresholds Intent
Pursuit Classic 2 guard 0.1, intruder 0.75 guard rarely denies the goal
Pursuit Capture 2 guard 0.3, intruder 0.5 balanced
Pursuit Classic 3 guard 0.5, intruders 0.1 guard-favoured; the guard over-reaches
Pursuit Capture 3 guard 0.05, intruders 0.8 intruders expect to evade
Coins 0.4, 0.4 slightly below an even share
Stag Hunt 0.9, 0.9 sustained cooperation demanded
Commons Harvest 0.8, 0.7, 0.3, 0.2 players 2-3 over-reach (they rarely harvest safely)
Territory 0.25, 0.2 each wants a share of a winner-take-all race

The Pursuit tasks deliberately lean different ways: Classic 2 and Capture 3 favour the intruders, Capture 2 is balanced and Classic 3 favours the guard (revised on 2026-10-03 from an all-intruder-favoured first draft, which had guard 0.15/intruder 0.7 in Capture 2 and guard 0.3/intruders 0.6 in Classic 3). Guard and intruder (and the two Territory) objectives exclude each other, so a profile can only be jointly certified when the paired thresholds leave room for both sampling margins (about 0.05 each at 1000 rollouts).

Where thresholds apply

  • Task specs: PursuitExperimentSpec, CoinsExperimentSpec, HarvestExperimentSpec and TerritoryExperimentSpec gained a probability_thresholds field that is passed to TemporalGameExperimentSpec; the eight curated notebooks set it from the registry (cell-private _curated_thresholds import). New satisfaction reports therefore record and certify against these thresholds. The spec thresholds are not part of the partial-resume fingerprint, so existing manifests and checkpoints stay valid.
  • scripts/certification_summary.py re-certifies every report against the registry by default, recomputing intervals from the stored counts, so reports written under 0.5 and new ones are treated identically. --reported-thresholds uses each report's recorded thresholds instead. The summary records threshold_source, per-profile probability_thresholds, and primary and confirmatory joint_margin_by_run.
  • The paper's scripts/make_tables.py writes q_i into the per-agent tables and reports the joint margin min_i (L_i - q_i) (non-negative exactly when a seed is jointly certified) in place of the uniform-threshold bound min_i L_i, which loses its meaning when thresholds differ.
  • The integrity audit (check_reevaluation_integrity.py) re-checks each report's certificates against the thresholds that report recorded; it does not apply the registry.

Selection caveat

The thresholds were chosen after the primary evaluation, from its per-seed estimates, to span attainable and over-reaching requirements, so primary-batch certification under them is descriptive. The confirmatory batch uses disjoint rollouts and was not the basis of the choice (its averages, nearly identical to the primary ones, had been seen); the paper states the choice in the appendix threshold paragraph. Changing a value only needs a registry edit and a summary/table rerun; no retraining or re-evaluation is required.