‘It Works on My Machine’: The Ladder of Numerical Dependencies

A fix at one rung doesn’t fix the rungs around it

“It works on my machine” is a joke with a precise technical meaning: the program you ran and the program someone else runs are different programs, and neither of you can see the difference. Your colleague clones the repo and the import fails. The cluster job dies in a way it never dies locally. Production returns a number four decimal places off from your notebook — which is worse, because nothing looks broken.

This keeps happening because a running computation sits on a ladder of dependencies, and every rung is a place where two machines can differ:

Chipset and instruction set — x86_64, arm64/Apple Silicon, and the accelerator underneath: NVIDIA with CUDA, AMD with ROCm, Apple’s Metal, or nothing at all.
Operating system — Linux (and which distro, and which glibc), macOS, Windows; kernel, C library, system compilers.
Language runtime — CPython 3.11 vs 3.13, R 4.3 vs 4.4, Julia 1.10 vs 1.11, Node vs Bun.
Packages and libraries — NumPy, dplyr, PyTorch, and the several hundred transitive dependencies you never chose.
Your application code — the only rung most people think about, and the only one git tracks.

The critical property is that the rungs don’t insulate each other. Pinning numpy==2.1.3 guarantees nothing if one machine gets a prebuilt wheel linked against Intel MKL and the other compiles from source against OpenBLAS. Pinning the OS in a container guarantees nothing about the GPU driver on the host, which the container does not own. A fix at one rung leaves everything above and below it free to vary, and almost every tool in this space really owns only one or two rungs while implying it owns the whole ladder.

That mismatch — between how much of the ladder a tool covers and how much of the ladder you need covered — is the measuring stick for everything that follows. The first place to apply it is the community with the most resources to throw at the problem.

Developers pinned the ladder by owning the whole machine

Production engineering solved the ladder problem with a strategy that looks like cheating: stop trying to make heterogeneous machines agree, and instead ship the machine along with the code. A container image is the bottom rungs — OS, libc, system libraries, runtime — frozen into an artifact and moved around as one unit. Whatever your laptop is, the image is Debian 12 with Python 3.12, in CI and in production alike.

Above the image, lockfiles pin the package rung. package-lock.json, pnpm-lock.yaml, uv.lock, Cargo.lock, and go.sum all do the same job: record the resolved version and content hash of every transitive dependency, so an install is a replay rather than a fresh solve. Around them, devcontainers push the same image onto the developer’s laptop, CI/CD rebuilds the artifact on every commit, and infrastructure-as-code pins the instance type it lands on.

Stacked up, that covers the ladder end to end. But look at what it costs: a container registry, a CI budget, a staging environment, and — critically — the freedom to declare that production is linux/amd64 and nothing else. That declaration is the load-bearing move. Developers don’t so much solve chipset and OS variance as abolish it by fiat, standardizing on one or two targets they control completely, usually with a platform team whose full-time job is keeping the images and pipelines healthy.

Take away the control over the target and the budget for the team, and every one of these tools becomes harder to use — which is precisely the situation the scientific computing world works in.

Scientists have to pin the same ladder without owning any of it

The tools scientists reach for are recognisably the same shape as the developer stack, but each is bent around a constraint that a platform team would simply have removed. conda and its faster reimplementation mamba handle the package rung and the compiled-library rung below it, because scientific packages drag along C, C++, and Fortran libraries that pip historically could not install. pixi is the modern version of that idea: conda-forge packages plus a lockfile that pins several platforms at once, so linux-64, osx-arm64, and win-64 resolve into one committed file rather than three environments that drifted apart. R has renv for per-project snapshots, and Julia’s Project.toml and Manifest.toml are the cleanest version in any language — an exact record of every resolved dependency plus the Julia version itself, replayed by Pkg.instantiate().

Containers show up too, but in mutated form. HPC clusters generally will not run Docker, because its daemon runs as root and hands users an effective route to root on a shared machine. So the HPC world uses Apptainer (formerly Singularity), which is rootless by design: no persistent daemon, containers run with the calling user’s privileges, and since v1.1.0 no setuid helper by default. At the far end of the rigour spectrum, Nix and Guix rebuild the whole dependency graph from source with hashed inputs, while Snakemake and Nextflow pin reproducibility one level up, wiring per-step containers into a graph of analysis stages.

The resource asymmetry is the whole story

That toolkit is not weaker than the DevOps one. It is being asked to do a harder job with less. A typical lab runs on three or four personal laptops of different vintages and operating systems, a shared university cluster whose OS image was chosen years ago and where nobody has root, and maybe one GPU box under a desk. There is no standard target to declare, and no platform team — there is a graduate student who has inherited 4,000 lines of undocumented analysis code from someone who left in 2023, with no lockfile and no record of which package versions it ever worked with.

The objective is different in kind, too. Production software needs to build today, on maintained infrastructure; if a dependency breaks, someone fixes it that afternoon. A paper’s code needs to run in five to ten years, on hardware that doesn’t exist yet, for a replicator with no access to the author and nobody maintaining anything in between.

That horizon is where things measurably fall apart. A large-scale study of research code took over 2,000 replication datasets from the Harvard Dataverse — around 9,000 R files published between 2010 and 2020 — and simply tried to run them. Seventy-four percent failed to execute without error; automated code cleaning brought that down to 56%, an improvement and still a majority.

Those failures are not mostly exotic. A large share are the bottom rungs of the ladder giving way — and those rungs deserve their own look, because they’re the ones both communities misdiagnose first.

The rung below the language is the one everyone gets wrong first

Package versions get the blame for those failures because packages are the rung people can see. The rung underneath is hardware and OS, and it varies along two axes at once.

The first axis is where the code runs: a laptop (macOS, Windows, or Linux, each with its own toolchain), a cloud VM (possibly Graviton, and therefore arm64), an on-prem server, an HPC cluster with a scheduler and an ancient glibc, or an edge device with a fixed vendor image.

The second axis is what the silicon is: x86_64 or arm64 for the CPU, and CUDA, ROCm, Metal/MLX, or CPU-only for the numerics. These are orthogonal — an arm64 Mac and an arm64 Graviton instance share an instruction set and nothing else.

Three failures make the point concrete:

The CUDA wheel that doesn’t match the driver. You install a PyTorch build compiled against CUDA 12.4; the cluster node’s driver only supports 12.1. Nothing about your Python environment is wrong — the package rung is pinned exactly. It fails at import, against a rung the lockfile has no vocabulary for.

The Mac that has to build from source. A package ships x86_64 wheels only. On an M-series Mac, pip finds no matching wheel and falls back to the source distribution, which needs a Fortran compiler and BLAS headers macOS doesn’t ship. The install dies with a compiler error from a language the user has never written, in a package they wanted for one function.

The answer that quietly changes. This is the dangerous one. Compile with -ffast-math, or let the compiler contract a multiply-add into a fused FMA instruction, and the operations get reassociated — and floating-point addition is not associative. The same effect appears with no compiler flags at all: two BLAS implementations, or the same one on a different core count, sum a reduction in a different order and return different last digits. Same source, same package versions, different answer. In an iterative solver or a chaotic simulation those last digits are amplified, not averaged away.

Note that only one of those three is fixed by a lockfile. The other two need something that reaches below the language layer. Which raises the obvious question: how far down does each language’s tooling actually reach?

Every language leaves a different rung unpinned

Answering that — how far down each ecosystem reaches — is the fastest way to see why the same class of failure looks different in every language. Each has staked out a different portion of the ladder, and the rungs it leaves uncovered are where its characteristic breakage lives.

Python has the most tools and the most confusion. pip pins packages but not the interpreter or anything native; conda and pixi reach below the language into compiled libraries and the CUDA toolkit. uv pins packages and the Python version, and its lockfile is universal — a forking resolver records the resolution for every combination of OS, architecture, and Python version in one committed file. Its --exclude-newer flag is a useful trick for the long-tail case: resolve as though it were an earlier date.

R has renv for per-project snapshots, but its harder problem is the repository model. CRAN serves only current versions, so a snapshot is only as good as an archive that still holds the old tarballs — which is why Posit Package Manager offers dated CRAN snapshots. Bioconductor moves in release trains tied to specific R versions, coupling the runtime rung to the package rung.

Julia comes closest to right by default: Project.toml declares, Manifest.toml records the resolved graph including the Julia version, and instantiate replays both. It also ships binary dependencies as versioned JLL packages, putting part of the native-library rung inside the manifest rather than outside it.

C, C++, and Fortran share the deepest problem, because they are the rung everything else sits on — Fortran is not a historical curiosity but sits under climate models, CFD, and much of LAPACK, and therefore under NumPy and R. CMake describes C/C++ builds, vcpkg and Conan manage their dependencies, and Fortran has barely even that (fpm is young and not where legacy code lives). None of it pins the compiler, which is what actually matters: link a library built with one GCC version against code built with another and it can break on ABI differences, while optimization level and flags alone move the last digits. The package rung is pinnable here; the toolchain rung mostly isn’t.

Shell is the least reproducible rung of all, and the one nobody thinks to lock. The run_analysis.sh gluing everything together depends on whichever awk, sed, and grep the OS provides — and macOS ships BSD versions whose flags differ from Linux’s GNU ones. There is no lockfile for that; it is pure ambient dependency on the machine.

JavaScript and TypeScript matter here because the dashboard and visualization layer usually are JS. npm, pnpm, and yarn all lock packages. Bun collapses runtime, bundler, test runner, and package manager into one binary, and compiles a project into a self-contained executable — pinning the runtime rung into the artifact rather than requiring it be installed.

Read down that list and one pattern stands out: the ecosystems that reach furthest down the ladder are the ones that treat the lockfile as mandatory and fast rather than optional and slow. That preference used to be a matter of taste. Something has recently made it a matter of economics.

AI didn’t add a rung — it raised the price of a loose one

The tooling preference just described — fast, locked, unambiguous — has been pushed from nice-to-have to load-bearing by AI coding agents, and it happened through two separate mechanisms worth keeping apart.

The first is a change in how often environments get exercised. A human sets up an environment once and works in it for weeks; a two-minute conda solve is an annoyance amortized over a month. An agent runs a write–test–observe loop dozens or hundreds of times per session, paying that cost every iteration, so a slow resolver becomes a throughput ceiling. Worse, ambiguity becomes a correctness problem: if an environment resolves differently between two runs, the agent gets an inconsistent signal about whether its last edit helped, and will happily “fix” what was actually dependency drift. Agents need environments fast to construct and deterministic to reconstruct — the two properties reproducibility always wanted, now with a much higher price on failure.

The second mechanism is that this demand has pulled the AI labs directly into the tooling layer. In December 2025, Anthropic acquired Bun, framing it as “essential infrastructure for AI-led software engineering”; Claude Code ships as a Bun-compiled single-file executable, and Bun’s announcement confirmed it stays MIT-licensed and developed in the open. In March 2026, OpenAI agreed to acquire Astral — the company behind uv, Ruff, and ty — bringing the fastest Python packaging tool into the Codex ecosystem with a similar open-source commitment. In June 2026 it agreed to acquire Ona, formerly Gitpod, explicitly for secure persistent cloud environments so Codex agents can work unattended for hours or days.

Read together, those three moves are one bet: that the environment layer is strategic infrastructure for agents rather than plumbing. Two things are worth saying plainly about it. They concentrate widely-depended-on open-source tooling inside companies that compete with each other, which is a live governance question for tools the scientific Python world now leans on. And none of it removes a rung — a CUDA driver mismatch is exactly as broken with an agent as without one. What has changed is the cost of leaving a rung unpinned, and the speed at which tools that pin rungs quickly are displacing tools that pin them slowly.

Which rungs have you actually pinned?

Everything above is one question asked repeatedly: which rungs of the ladder have you pinned, and which have you left to chance? Application code, always — that’s git. Packages, if you committed a lockfile. Runtime version, if your tool records it. OS and libc, only if you containerized. Chipset and driver, almost never, which is why that rung produces the failures that feel like haunting rather than debugging.

The two threads in this post are answers to that question under different constraints. Developers pin the low rungs by controlling the target and paying a platform team to keep it controlled. Scientists must pin the same rungs across borrowed, heterogeneous machines they don’t administer, for a result that has to survive a decade rather than a sprint — and the 74% figure is what that difference costs. The gap between those situations is resources, not competence, and tools designed for the first situation quietly assume away the constraints of the second.

AI shifts who feels this and how urgently. An agent iterating a hundred times a session is an unusually sensitive instrument for detecting a loose rung, and the money now flowing into uv, pixi, Bun, and persistent sandboxes is a response to that sensitivity. But the ladder is unchanged. The useful habit is still to name your rungs explicitly — write down the architecture, the driver, the runtime, and the resolved packages, and commit that record next to the code — because the rung you never wrote down is the one that will differ.