HPC Container Toolchain¶

submit python PROJECT/PATH --container bookworm resolves through $HOME/container/hpc.sh. The container is used to provide a newer Bookworm glibc than the base HPC OS while still using project files, virtual environments, uv caches, and other user-space paths from the host via Apptainer bind mounts.

The old default image, docker://debian:bookworm-slim, fixed the glibc layer but did not include a compiler. uv sync --all-groups can still need a native C++ build. In particular, the pinned pygambit==16.7.0 ships Linux source but no Linux wheel in uv.lock, so it invokes c++ during wheel creation. With the slim image this fails as:

error: command 'c++' failed: No such file or directory

The default image was changed to docker://buildpack-deps:bookworm, which keeps the newer glibc and includes common build tools such as c++. Explicit container=slim, container=bookworm-slim, or container=debian-slim still selects the minimal image for jobs that do not need native builds.

Avoid trying to use the host compiler from inside the container by binding individual host binaries. GCC and G++ depend on absolute paths for compiler backends, headers, startup objects, and linker components; partial host binds are fragile and can accidentally mix the old host userspace with the newer container glibc.

CX3 Host Builds Of Pygambit¶

The non-container CX3 runner uses the host's glibc 2.28 and purges environment modules before synchronizing the shared virtual environment. The default host compiler is GCC 8.5. This combination needs special handling for pygambit==16.7.0:

A pre-existing pygambit==16.6.0 extension built on a newer system failed to import because it required GLIBC_2.29.
Building 16.7.0 with the default compiler mode failed on C++17 language features such as std::optional.
Adding only CXXFLAGS=-std=c++17 was insufficient because CX3's GCC 8 standard library does not implement std::transform_reduce.
Dynamically linking an extension built with a newer compiler module is also unsafe: $HOME/py.sh calls module purge, so that compiler's libstdc++ is absent when the notebook imports the extension.

The validated repair is to compile with CX3's GCC 9.3 module, request C++17, and statically link the GCC C++ runtimes while retaining the host glibc ABI:

module load tools/prod GCC/9.3.0
source "$HOME/bucket/.venv-pltlf/bin/activate"
cd "$HOME/Projects/pltlf"
CXXFLAGS="-std=c++17" \
LDFLAGS="-static-libstdc++ -static-libgcc" \
  uv sync --active --all-groups --frozen

Validate the result under the same module-purged conditions used by a job:

module purge
source "$HOME/bucket/.venv-pltlf/bin/activate"
cd "$HOME/Projects/pltlf"
uv sync --active --all-groups
uv run --active --no-sync python -c \
  'import marimo, pygambit, src; print("CX3 environment OK")'

ldd on site-packages/pygambit/gambit*.so should not list libstdc++ or libgcc; it should resolve its glibc dependencies from /lib64. If it reports a missing GLIBC_* or GLIBCXX_* version, rebuild rather than reusing that extension.

Before a large launcher fanout, fast-forward the clean remote checkout and run the environment sync once from the login node. The CX3 job runner performs a git pull and uv sync in every job. If the shared checkout or environment is stale, simultaneous first-job updates can race on Git refs and can all repeat the same expensive native build. A prepared checkout and environment make both operations no-ops in the jobs.

This failure mode was confirmed on 2026-08-09: a 120-job PLTLF submission had two early jobs collide during git pull, while the remaining jobs failed before notebook execution while building pygambit. The jobs used effectively no training CPU time and produced no new run state.

Marimo Batch Helper Compatibility¶

The shared CX3 $HOME/marimo_batch.py helper deliberately uses private Marimo session and export APIs so it can execute a notebook once while writing periodic live IPYNB/HTML snapshots. Marimo 0.23.16 removed the marimo._server.export package used by that helper. Once the pygambit build was repaired, a smoke job exposed the next pre-training failure as:

ModuleNotFoundError: No module named 'marimo._server.export'

That failure originally required a marimo==0.23.15 pin. On 2026-10-02, PLTLF upgraded to marimo==0.25.0 after the maintained Topaz helper was ported to the modern marimo._export request/snapshot APIs and asynchronous session startup. The shared helper in ~/Projects/topaz/setup supports 0.24 and 0.25; its implementation and compatibility tests live in Server/jobs/marimo_batch.py and Server/jobs/tests/marimo_batch_test.py.

Deploy this helper to $HOME/marimo_batch.py on DoC and CX3 before upgraded project runs start. The two DoC clusters share that file; CX3 has a separate deployment. In 0.25, session creation both requires a SessionStartup and must be awaited. Merely updating the import locations is insufficient.

Any future upgrade must run a real batch-helper smoke notebook, not merely import marimo, because private session/export behavior is reached only by batch execution. Include CLI overrides, live/final IPYNB and HTML, layout inlining, successful cleanup, and preserved failure artifacts. The current helper passed all nine scenarios on DoC amd24 job 9736 across 0.24.0, 0.24.2, and 0.25.0.

For the PLTLF upgrade, CX3 PBS job 4248232.pbs-7 passed the real helper's regression suite on 0.25.0 in an isolated node-local environment on cx3-3-12 (2026-10-02). It tested successful execution, live/final exports, layout inlining, cleanup, and error snapshots. The validated helper was deployed to CX3; its SHA-256 matches the maintained Topaz source. The log and previous helper are retained in $HOME/bucket/marimo-upgrade-20261002-ZX87q1/. The project environment also executed notebooks/test.py with populated IPYNB/HTML exports, and the representative experiment/report notebooks passed marimo check --strict.

Version Policy¶

pygambit is exactly pinned to 16.7.0 and Marimo to 0.25.0 in pyproject.toml; Dependabot is configured to ignore both. Do not relax these pins or update them as routine dependency maintenance. A proposed pygambit upgrade must first be built and imported on CX3 using the module-purged validation above. A proposed Marimo upgrade must pass the real batch-helper smoke described above. Update a pin and its Dependabot policy deliberately only after the corresponding check passes.

The Marimo upgrade uses uv lock --upgrade-package marimo. Only Marimo's package version changes; the resolver may also normalize platform markers in the generated lockfile. Retaining an exact pin and the Dependabot exclusion keeps future private-API migrations deliberate rather than allowing an untested minor release to reach scheduled jobs automatically.

Existing cluster runs continue using their current process/environment. Future runner setup pulls the upgraded project lockfile and syncs Marimo to 0.25.0. Avoid manually synchronizing a shared venv while older jobs are still using it; this upgrade does not require cancelling or restarting those jobs.