Skip to content

Experiment Timing Sidecars

src/experiments/temporal_game_experiment.py records the wall-clock duration of user-visible experiment stages. Environment-specific adapters, including Pursuit and Coins, reuse this orchestration rather than implementing their own timing schema.

Sidecar Location

Each selected algorithm owns two canonical files inside its artifact directory:

  • timings.json
  • timings_hardware.json

Examples:

  • exports/gridworlds/pursuit/classic_3/ippo/timings.json
  • exports/gridworlds/coins/own_coin_first/2/iql_cer/timings.json

The loader reads only the canonical location. Pursuit exports created before the flattened layout can be copied once by running ../scripts/migrate_pursuit_export_paths.sh from inside exports/; runtime code does not fall back to legacy paths.

Stored Structure

timings.json is a mapping keyed by stage name. Each stage contains total_seconds and runs; every run records seconds, a hardware_id, and optional metadata such as run_idx or label. The sibling timings_hardware.json is schema-versioned and deduplicates full hardware descriptions in its hardware map.

Historical timing files with embedded system_ram_gb and gpu_ram_gb remain readable. The timings report identifies those values as legacy metadata and does not attribute them to the machine that later reads the file.

Stage Names

The per-run training stage identifies the algorithm:

  • ippo_train, ippo_cer_train
  • pr2_ippo_train, pr2_ippo_cer_train
  • iql_train, iql_cer_train
  • pr2_iql_train, pr2_iql_cer_train
  • nashq_train, nashq_cer_train
  • hc_ippo_train, hc_pr2_ippo_train, hc_iql_train, hc_pr2_iql_train, hc_nashq_train

Other stages are checkpoint_save, optional nashconv, optional satisfaction, and optional one-shot alpharank. A NashConv-triggered continuation adds <training-stage>_auto_resume, checkpoint_save_auto_resume, and nashconv_auto_resume for that run. run_idx values are zero-based and align with checkpoint and report records.

Resume And Replacement Semantics

The runner saves a sidecar immediately after each measured stage, so an interrupted experiment retains completed timing records. The generic timing helper applies these replacement rules:

  • an indexed stage resets when record_timing(...) receives run_idx=0;
  • later indices append within the current invocation;
  • a stage without run_idx, such as alpharank, replaces its previous value;
  • with resume_partial_runs=True, only timings belonging to the validated completed prefix survive before remaining seeds continue.

These rules prevent stale tail entries from being mixed into a restarted or partially resumed experiment. Each algorithm has its own directory, so running several algorithms in one suite does not merge their timing records.

Hardware Allocation Semantics

Hardware records describe resources visible to the timed process:

  • RAM is the smallest finite cgroup v1/v2 limit. Slurm allocation variables are used only when no finite cgroup limit is visible; total node RAM is not substituted.
  • CPU availability comes from process affinity, preserving accessible physical cores versus logical threads. Slurm and PBS job metadata is retained.
  • CPU clock ranges normally come from Linux cpufreq. ACPI CPPC may extend the maximum to nominal_freq * highest_perf / nominal_perf.
  • KVM guests without physical clock interfaces can use a fixed guest-visible /proc/cpuinfo frequency only when every visible vCPU advertises hypervisor and tsc_known_freq and readings agree within 0.1 MHz.
  • The processor name is guest-visible and may not identify the physical host SKU.
  • Visible NVIDIA GPUs record model, usable VRAM, memory-bus width, maximum memory clock, and derived peak memory bandwidth when NVML exposes them.

Timing belongs at the generic orchestration boundary because it corresponds to the stages users run and resume: train, save, verify, optionally continue, and optionally construct the cross-play empirical game. The sidecar intentionally does not promise a finer per-agent or verifier-substage breakdown.

Persistence

The files are human-readable JSON. Saves use per-file advisory locks, merge distinct stage keys, and atomically replace a file only after flushing it. The hardware sidecar uses the same locking and atomic-write protocol, preventing concurrent experiment processes from dropping records or exposing partial JSON.