Cluster Submission Scripts¶

The launchers in notebooks/experiments/scripts/ share their environment, algorithm, path, validation, grouping, and runner policy through support.bash. The user-facing quick reference is the README beside the scripts.

Launcher topology¶

Additional Slurm tools are hyper.sh, which fans one environment over two partitions, hyper_cc.sh, which pins both slots to the CPU cluster, vis.sh, which performs plot-only reruns, and pltlf.sh, which retains the historical training/a30/a40/a100 environment assignment.

Commands use the Topaz runner's shared ~/Projects convention and submit pltlf/<notebook-path>. The first path component must be the remote project directory, not this checkout's local quartz/ parent: both gpucluster and CX3 normally clone the repository at ~/Projects/pltlf. PROJECT_PREFIX can override pltlf for a nonstandard remote clone. All runners add venv=pltlf unless the caller supplies venv=....

Curated environments¶

The canonical order in support.bash is:

1. classic_2
2. capture_2
3. classic_3
4. capture_3
5. own_coin_first_2
6. sustained_stag_2
7. safe_harvest_4
8. role_claim_2

env.sh and env_hpc.sh also accept a direct notebook path. The all-suite and visualization launchers deliberately use only the curated keys; archived comparison notebooks under alt/ are excluded.

Algorithm selection and isolation¶

The policy contains 15 algorithm IDs: ippo, ippo_cer, pr2_ippo, pr2_ippo_cer, iql, iql_cer, pr2_iql, pr2_iql_cer, nashq, nashq_cer, hc_ippo, hc_pr2_ippo, hc_iql, hc_pr2_iql, and hc_nashq.

They are split into a ten-algorithm default and five no-CER ablations. The default contains ippo_cer, pr2_ippo_cer, iql_cer, pr2_iql_cer, nashq_cer, and all five hc_* baselines. The ablation set contains ippo, pr2_ippo, iql, pr2_iql, and nashq. With no selector, only the default is active. ablations=True adds the ablation set, while all=True selects the complete known list. Both are launcher-owned boolean arguments and are not forwarded to notebooks. An explicit algorithms= or PLTLF_ALGORITHMS selector remains exact, but selecting any no-CER variant without ablations=True fails before submission. all=True takes precedence over an explicit selector and always expands to the complete known list.

Every checked-in training notebook defaults all 15 run_* settings to false. Training commands therefore carry only the selected algorithms as true; omitted flags remain disabled. Callers select a subset with either:

bash notebooks/experiments/scripts/env.sh classic_2 a16 algorithms=ippo_cer,iql_cer
PLTLF_ALGORITHMS="ippo_cer iql_cer" bash notebooks/experiments/scripts/env_hpc.sh classic_2

Selectors are validated and de-duplicated before submission. Direct run_* arguments and just_vis are rejected because the launcher owns them.

GPU launchers balance the selected order over three jobs by default and four on the training partition. The launcher-owned bundles=<positive integer> argument overrides this per invocation and is consumed before notebook submission. PLTLF_ALGORITHM_GROUPS provides the environment-level default when bundles= is absent. A count larger than the selection is capped so empty jobs are not created. Both the CER-first default and an all-algorithm run create 24 jobs with all_envs.sh a16, or 32 with all_envs.sh training; the profile changes the algorithms within those jobs, not the default bundle count.

HPC launchers default to one CPU-only job per selected algorithm, with ngpus=0 and walltime=40:00:00. They accept the same bundles= argument; PLTLF_HPC_GROUP_COUNT provides its environment-level default. An optional positional queue is emitted as queue=<name>; omission preserves the HPC routing default. A default all_envs_hpc.sh run creates 80 jobs; passing ablations=True or all=True creates 120.

The DoC Slurm CPU host (ssh cpucluster) uses all_envs.sh, not the Imperial HPC launcher. On an amd* partition, env.sh automatically selects ~/py_cpu.sh (or py_cpu.sh when the home runner is absent); this is required because the generic py.sh requests a GPU in its Slurm header. Passing bundles=15 is the stable one-job-per-algorithm setting: grouping caps the requested count to the number of selected algorithms, producing 80 jobs for the unfiltered default suite or 120 for all 15 algorithms.

Pass feasible=True to remove the three tabular Nash-Q IDs (nashq, nashq_cer, and hc_nashq) from every spatial task while retaining them for sustained_stag_2. This implements the optimisation assumption that the image/grid state spaces are not practical tabular problems without changing any objective or learner. The 10-algorithm default suite then produces 66 jobs: eight algorithms for each of seven spatial instances and ten for Matrix Stag Hunt. An explicitly selected Nash-Q-only spatial invocation exits successfully with a skip message.

Slurm launchers consume walltime=[days-]hours:minutes[:seconds] and emit it as sbatch --time; it is not forwarded as a Marimo parameter. For example, the five-seed paper campaign can be previewed with:

DRY_RUN=1 SUBMIT_DELAY_SECONDS=0 \
  bash scripts/pltlf/all_envs.sh amd96 bundles=15 feasible=True \
  walltime=96:00:00 num_runs=5 calculate_satisfaction=True

Resume policy¶

All training launchers add resume_partial_runs=True unless the caller passes an explicit value. The four resume-named wrappers make intent visible in job logs, force the value to true, and reject caller overrides:

bash notebooks/experiments/scripts/env_resume.sh classic_2 a16 num_runs=5
bash notebooks/experiments/scripts/all_envs_resume.sh a16 algorithms=ippo_cer,iql_cer num_runs=5
bash notebooks/experiments/scripts/env_resume_hpc.sh classic_2 cpu72 num_runs=5
bash notebooks/experiments/scripts/all_envs_resume_hpc.sh cpu72 num_runs=5

The wrapper does not guess completion from file counts. The temporal-game runtime is the authority: it validates the manifest schema, algorithm/config fingerprint, expected seed sequence, cumulative result artifacts, and every checkpoint bundle before reusing a contiguous completed prefix. A mismatch starts the algorithm at run zero; missing or corrupt tail state resumes from the first invalid run.

Per-seed scheduler fanout was intentionally not ported from EqEx/COM. PLTLF currently writes cumulative runs.npy, evaluation arrays/reports, and one partial_resume_manifest.json in a shared algorithm directory. Concurrent seed jobs would race while replacing those files and could invalidate the contiguous-prefix contract. Supporting safe seed fanout would first require independent per-seed artifacts plus an atomic merge/index design.

Two-partition and CPU-cluster fanout¶

hyper.sh <environment-or-path> [key=value ...] balances algorithms over two Slurm partition capacities. Defaults are p1=training, p2=a16, p1_jobs=4, and p2_jobs=3. The partition list alternates between p1 and p2 until each capacity is filled, so a small algorithm subset uses both when possible. Environment controls PLTLF_HYPER_P1_JOBS and PLTLF_HYPER_P2_JOBS provide the default capacities.

bundles= is deliberately rejected by hyper.sh: the total number of hyper jobs follows the two partition capacities, so callers should use p1_jobs= and p2_jobs= instead.

hyper_cc.sh calls the same scheduler with both partitions set to CC_PARTITION (default amd48), selects py_cpu.sh when JOB_SCRIPT is not set, disables visible CUDA devices, and requests the JAX CPU platform.

Visualization behavior¶

vis.sh submits CPU jobs with just_vis=True. It defaults to amd48 and checks every curated export directory for at least one <algorithm>/runs.npy. Empty environments are reported and skipped; if all are empty, the command exits successfully without submitting. Use ignore_empty=False to submit every notebook, PLTLF_VIS_EXPORT_BASE to change the local root used for the preflight, or save_dir=... when all jobs use an explicit artifact directory.

Algorithm selectors are rejected in visualization mode because the notebook runtime discovers and loads all available algorithm artifacts.

Validation and operational controls¶

The all-environment wrappers validate partition/queue names, algorithm selectors, and owned arguments before their first submission. Shell commands are constructed as arrays and printed with shell escaping. Common controls are:

• DRY_RUN=1: print without submitting or sleeping.
• SUBMIT_DELAY_SECONDS=<seconds>: pacing between jobs and environments.
• SUBMIT_CMD=<command>: override scheduler submission command.
• JOB_SCRIPT=<script>: override py.sh / py_cpu.sh selection.
• PROJECT_PREFIX=<prefix>: override the remote project directory pltlf.
• DEFAULT_VENV=<name>: override the injected venv value.

Before a large CX3 fanout, pre-update the shared checkout and synchronize its virtual environment once. This avoids concurrent git pull/uv sync races in the jobs. The pinned pygambit version also requires an ABI-safe CX3 build; see HPC container toolchain for the failure signatures, rebuild command, and validation procedure.

Script integration behavior is covered in tests/test_cluster_submission_scripts.py; it runs every path in dry-run mode with scheduler stubs.