HPC Container Toolchain¶

$HOME/submit.sh ... container=latest resolves through $HOME/container/hpc.sh. The container is used to provide a newer Bookworm glibc than the base HPC OS while still using project files, virtual environments, uv caches, and other user-space paths from the host via Apptainer bind mounts.

The old default image, docker://debian:bookworm-slim, fixed the glibc layer but did not include a compiler. uv sync --all-groups can still need a native C++ build. In particular, the pinned pygambit==16.7.0 ships Linux source but no Linux wheel in uv.lock, so it invokes c++ during wheel creation. With the slim image this fails as:

error: command 'c++' failed: No such file or directory

The default image was changed to docker://buildpack-deps:bookworm, which keeps the newer glibc and includes common build tools such as c++. Explicit container=slim, container=bookworm-slim, or container=debian-slim still selects the minimal image for jobs that do not need native builds.

Avoid trying to use the host compiler from inside the container by binding individual host binaries. GCC and G++ depend on absolute paths for compiler backends, headers, startup objects, and linker components; partial host binds are fragile and can accidentally mix the old host userspace with the newer container glibc.

CX3 Host Builds Of Pygambit¶

The non-container CX3 runner uses the host's glibc 2.28 and purges environment modules before synchronizing the shared virtual environment. The default host compiler is GCC 8.5. This combination needs special handling for pygambit==16.7.0:

A pre-existing pygambit==16.6.0 extension built on a newer system failed to import because it required GLIBC_2.29.
Building 16.7.0 with the default compiler mode failed on C++17 language features such as std::optional.
Adding only CXXFLAGS=-std=c++17 was insufficient because CX3's GCC 8 standard library does not implement std::transform_reduce.
Dynamically linking an extension built with a newer compiler module is also unsafe: $HOME/py.sh calls module purge, so that compiler's libstdc++ is absent when the notebook imports the extension.

The validated repair is to compile with CX3's GCC 9.3 module, request C++17, and statically link the GCC C++ runtimes while retaining the host glibc ABI:

module load tools/prod GCC/9.3.0
source "$HOME/bucket/.venv-pltlf/bin/activate"
cd "$HOME/Projects/pltlf"
CXXFLAGS="-std=c++17" \
LDFLAGS="-static-libstdc++ -static-libgcc" \
  uv sync --active --all-groups --frozen

Validate the result under the same module-purged conditions used by a job:

module purge
source "$HOME/bucket/.venv-pltlf/bin/activate"
cd "$HOME/Projects/pltlf"
uv sync --active --all-groups
uv run --active --no-sync python -c \
  'import marimo, pygambit, src; print("CX3 environment OK")'

ldd on site-packages/pygambit/gambit*.so should not list libstdc++ or libgcc; it should resolve its glibc dependencies from /lib64. If it reports a missing GLIBC_* or GLIBCXX_* version, rebuild rather than reusing that extension.

Before a large launcher fanout, fast-forward the clean remote checkout and run the environment sync once from the login node. The CX3 job runner performs a git pull and uv sync in every job. If the shared checkout or environment is stale, simultaneous first-job updates can race on Git refs and can all repeat the same expensive native build. A prepared checkout and environment make both operations no-ops in the jobs.

This failure mode was confirmed on 2026-08-09: a 120-job PLTLF submission had two early jobs collide during git pull, while the remaining jobs failed before notebook execution while building pygambit. The jobs used effectively no training CPU time and produced no new run state.

Marimo Batch Helper Compatibility¶

The shared CX3 $HOME/marimo_batch.py helper deliberately uses private Marimo session and export APIs so it can execute a notebook once while writing periodic live IPYNB/HTML snapshots. Marimo 0.23.16 removed the marimo._server.export package used by that helper. Once the pygambit build was repaired, a smoke job exposed the next pre-training failure as:

ModuleNotFoundError: No module named 'marimo._server.export'

PLTLF therefore pins marimo==0.23.15, the last version validated with the deployed helper, and excludes Marimo from routine Dependabot updates. Keep the pin until $HOME/marimo_batch.py and its maintained Topaz source are ported to the newer marimo._export request/snapshot APIs. Any upgrade must run a real batch-helper smoke notebook, not merely import marimo, because the break is in private export behavior reached only by batch execution.

Version Policy¶

pygambit is exactly pinned to 16.7.0 and Marimo to 0.23.15 in pyproject.toml; Dependabot is configured to ignore both. Do not relax these pins or update them as routine dependency maintenance. A proposed pygambit upgrade must first be built and imported on CX3 using the module-purged validation above. A proposed Marimo upgrade must pass the real batch-helper smoke described above. Update a pin and its Dependabot policy deliberately only after the corresponding check passes.