060c8f20

By: lcnbr <im@lcnbr.ch>

prevent cold NixCI artifact cache stampedes

User prompt:

No—not end-to-end. Artifact reuse is good inside each Cargo/Crane lane, but poor across the full cold NixCI fan-out.

### The main problem: a cross-job cache stampede

Across all 75 jobs:

- 70 unique compile-bearing Nix derivations were executed 110 times.
- 35 exact `.drv` paths ran more than once.
- That produced 40 redundant executions and at least 880 repeated `Compiling` lines.

Because the full `.drv` paths match, these are genuinely identical inputs, features, profiles, and sources—not merely crates with similar names.

Examples:

| Exact derivation | Executions | Work per execution |
|---|---:|---:|
| Global `cargoArtifacts` | 3× | 201 compile units |
| Workspace-hack dependencies | 2× | 214 units |
| Linnet-Python test dependencies | 3× | 33 units |
| Spenso test dependencies | 3× | 14 units |
| Clinnet dependencies | 2× | 77–82 compile lines |

The run used 5h28m of worker time for 1h13m elapsed. Parallelism explains part of that ratio, but the identical derivations confirm substantial duplication. [NixCI timing](https://nix-ci.com/gh:alphal00p:gammaloop/disconnected/4a787c2c8be301971150dd708e4e9435676148f5/timing)

### What is reusing correctly

Within one dependency chain, the design generally works:

- Final package phases normally compile only their own crate.
- `gammaloop-api`’s final Cargo phase took 4.03 seconds.
- The final GammaLoop package was effectively copy-only.
- The `gammalooprs` test-binary phase compiled only `gammalooprs`.
- Integration, Python API, Vakint, and Clinnet nextest archives compiled nothing.
- Every nextest execution job ran its archive without compiling.
- Python ABI artifacts are correctly separate because their features and PyO3 environment differ.

That structure lives in the package-local test artifact and archive construction in [flake.nix](/home/lcnbr/dev/gammaloop/flake.nix:2308) and [flake.nix](/home/lcnbr/dev/gammaloop/flake.nix:2646).

### Why NixCI still duplicates work

The explicit scheduling graph removes all edges involving `doNotBuild` nodes instead of preserving the effective dependency relationship through them. See [nix-ci.nix](/home/lcnbr/dev/gammaloop/nix-ci.nix:921).

Consequently, separate workers can start consumers whose Nix derivations contain the same hidden dependency. Both workers build it before either has published it to the shared cache.

There are also some incorrect or missing manual edges:

- The Spenso nextest scheduling group omits `symbolica-utils`, although the actual archive includes it.
- Python nextest waits for the ordinary integration dependency artifact rather than its distinct `python-api-tests` binary context.
- `spynso3` test artifacts are scheduled despite being excluded from nextest and having no consumer.

### Remaining Cargo-context misses

Grouped nextest archives combine packages under a different Cargo root/feature-unification context than the individual package prebuilds:

| Archive | Recompiled workspace units | Cargo time |
|---|---:|---:|
| Core | 2 | 1m48s |
| Linnet | 1 | 6.29s |
| Spenso | 7 | 58.95s |

This is not arbitrary third-party rebuilding, but it is still missed reuse. Singleton archives avoid it because their build and archive contexts match.

Check, Clippy, docs, and doctests also perform separate large passes. That is mostly intentional: their Cargo/rustc artifact kinds are incompatible, and earlier attempts to persist mode-specific trees created large, low-value cache objects.

### There is also major non-compilation overhead

Some “long builds” are mostly artifact movement:

- `gammaloopApiPackageArtifacts`: final Cargo build 4.03s, but approximately 40m54s across 104 cache uploads in a 54m05s job. [Log](https://nix-ci.com/gh:alphal00p/gammaloop/disconnected/4a787c2c8be301971150dd708e4e9435676148f5/3a98d2d9-d888-4cbb-895a-e6992771efa3/raw-logs)
- Python module: final Cargo build 0.42s, but approximately 30m22s of uploads in a 40m49s job. [Log](https://nix-ci.com/gh:alphal00p/gammaloop/disconnected/4a787c2c8be301971150dd708e4e9435676148f5/aa5d1c3e-3fea-4805-8dc8-addd69d62ab3/raw-logs)

`doNotLinkInheritedArtifacts = true` and artifact fan-in cause large target trees to be copied and republished, as already noted in the [architecture audit](/home/lcnbr/dev/gammaloop/docs/architecture/nix-crane-cache-reuse.md:1369).

So my fix order would be:

1. Make shared artifact roots explicit NixCI producer/barrier jobs and preserve scheduling through hidden nodes.
2. Correct the Spenso and Python nextest dependencies and remove unused `spynso3` test jobs.
3. Align grouped nextest prebuild contexts—or archive per package—to eliminate those final workspace rebuilds.
4. Reduce deep-copying and repeated publication of inherited target trees.

Also, `72 successful / 3 cached` is misleading here: it describes top-level attributes, not whether nested Cargo artifacts were reused.

Follow-up prompt:

do this on top of main

Total

12m51s

Timeline