Skip to content

Make the dev image usable as a VS Code devcontainer, with Rust - #80

Open
alycda wants to merge 8 commits into
mainfrom
feat/rust-in-dev-profile
Open

Make the dev image usable as a VS Code devcontainer, with Rust#80
alycda wants to merge 8 commits into
mainfrom
feat/rust-in-dev-profile

Conversation

@alycda

@alycda alycda commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Makes the dev image actually work as a VS Code devcontainer, and gives it a Rust toolchain. The two are independent — the Rust commits are the original goal, the Docker commits are what it took to get a devcontainer to open at all.

Two defects, one root cause

The image is a Nix rootfs; Dev Containers and the VS Code Server assume FHS. Both failures reported something other than themselves, which is most of why they took a while:

docker exec was broken. The base image ships /etc/{passwd,group,shadow} as absolute symlinks into the store. They resolve fine inside the container, so docker run works — but docker exec resolves the user entry host-side with openat2 RESOLVE_BENEATH semantics, which rejects absolute symlinks as escaping their parent. Dev Containers execs a shell server as its first act, so it died before any lifecycle command ran:

Shell server terminated (code: 126, signal: null)
openat etc/passwd: path escapes from parent

The server's node couldn't run. It ships a 121 MB glibc-linked binary wanting /lib/ld-linux-aarch64.so.1; this rootfs has no /lib. The visible error was check-requirements.sh: getconf: command not found — a symptom, not the disease. Fixed with the mechanism Microsoft documents under "Can I run VS Code Server on older Linux distributions?": given VSCODE_SERVER_CUSTOM_GLIBC_LINKER, VSCODE_SERVER_CUSTOM_GLIBC_PATH and VSCODE_SERVER_PATCHELF_PATH, the server patchelfs its own node on first launch and the requirements check short-circuits.

--inputs-from resolves nixpkgs through this flake's lock rather than the ambient registry, so it reuses the glibc the image already links against — marginal cost is patchelf and gcc-lib, not a second libc.

Rust in the dev profile

alyssa@dev had no rustup, cargo or bacon, so the image that exists to be a devcontainer couldn't build a Rust project. modules/dev/rust.nix had been there all along with work.nix as its only importer — because importing it cost 1.6 GiB of closure for lldb (measured; rustup is 94 MiB, bacon 63 MiB). That would have landed on alyssa@dev-x86 too, on the 2012 MBP, which is the disk headroom PR #34 exhausted. lldb isn't a Rust dependency and CodeLLDB bundles its own, so it moves to work.nix and the shared module becomes importable.

Also replaces programs.zsh.initExtra = "rustup update" with an activation-time rustup default stable. The old form ran a network call on every shell — miserable when every VS Code terminal pays it — and was the wrong command besides: rustup update installs nothing when no toolchain exists, leaving cargo as a shim that errors. Guarded with || true, because activation runs at container start and a failed run aborts it after dotfiles are linked, which is the documented route to a container with a perfect prompt and no packages.

Verification

  • alyssa@dev and alyssa@dev-x86 evaluate; dev gains exactly rustup + bacon, work keeps rustup + bacon + lldb.
  • The exec fix was bisected against the plain image before rebuilding: docker exec on dev returned 126, and 0 once the three files were materialized.
  • The glibc fix was proven end-to-end against the exact server build VS Code 1.131.0 pulls — unpatched node fails with a bare "no such file or directory"; patched through these paths, node -e prints v24.18.0.
  • Confirmed working: a devcontainer now opens against this image.

Notes

  • alyssa@work-dev does not evaluate on aarch64-linux — work.nix carries cocoapods, which is aarch64-darwin only. Verified pre-existing on main; left alone as unrelated.
  • The Docker commits are independent of the Rust ones and cherry-pick cleanly to main on their own. Worth landing first if the Rust work needs discussion — until they're in, every fresh dev build breaks Dev Containers for any folder.
  • Consumers need "postStartCommand": "/opt/dotfiles/docker/entrypoint.sh true" in their devcontainer config: the CLI starts containers with --entrypoint /bin/sh, so the image ENTRYPOINT and its home-manager activation never run on their own.

Full write-up in docs/solutions/integration-issues/vscode-dev-containers-on-a-nix-rootfs.md, including the two wrong turns.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KBhs11SMRHwp2CGkoGRzRj

alycda and others added 7 commits August 5, 2026 02:32
modules/dev/rust.nix carried lldb, so importing it cost 1.6 GiB of closure
(measured: lldb 1.6 GiB, rustup 94 MiB, bacon 63 MiB). That priced the
container profiles out of importing it at all - which is why alyssa@dev has
no Rust today despite the module existing since forever.

lldb is not a Rust dependency: cargo and bacon never invoke it, and VS Code's
CodeLLDB extension bundles its own. It belongs to the one profile that wants
a system debugger, so it moves to work.nix rather than being dropped.

No profile's package set changes: work.nix was the only importer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`programs.zsh.initExtra = "rustup update"` ran a network call on every shell
start. Tolerable with one terminal open; in a devcontainer every VS Code
terminal pays it before showing a prompt.

It was also the wrong command. `rustup update` refreshes installed toolchains
and installs nothing when there are none, so a fresh profile got rustup's
shims with nothing behind them - `cargo` on PATH, failing with "no default
toolchain configured". `rustup default stable` is what makes the profile
usable and no-ops afterwards. Same pattern profiles/code.nix already uses.

Guarded with `|| true`: this is the only network call in activation, and
activation runs at container start. A failed `run` aborts activation after
linkGeneration has already written the dotfiles - the documented route to a
container with a perfect prompt and no packages. Starting offline should mean
"no toolchain yet", not a broken profile.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
alyssa@dev shipped no rustup, cargo or bacon, so the image that exists to be
a devcontainer could not build a Rust project. modules/dev/rust.nix has been
there all along; work.nix was its only importer.

Affordable now that lldb's 1.6 GiB lives in work.nix. rustup is 94 MiB and
fetches toolchains at runtime into ~/.rustup - the devhome volume, not an
image layer - so alyssa@dev-x86 on the 2012 MBP gains ~160 MiB, well inside
the headroom PR #34 exhausted.

Verified: alyssa@dev and alyssa@dev-x86 evaluate, and dev's package set gains
exactly rustup + bacon while work-dev keeps rustup + bacon + lldb.

Note: alyssa@work-dev does not evaluate on aarch64-linux, and did not before
this branch either - work.nix carries cocoapods, which is aarch64-darwin
only. Left alone as an unrelated bug.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The nixos/nix base image ships /etc/{passwd,group,shadow} as absolute symlinks
into the store. Resolvable from inside the container, so `docker run` and every
shell in it are fine - but `docker exec` under OrbStack fails:

  Shell server terminated (code: 126, signal: null)
  openat etc/passwd: path escapes from parent

That takes VS Code Dev Containers with it, which is what this image exists for:
the CLI's first act after starting the container is to exec a shell server, and
it dies before any lifecycle command runs.

The exec path resolves the user entry host-side, rooted at the container
rootfs, with openat2 RESOLVE_BENEATH semantics - absolute symlinks are rejected
as escaping their parent rather than re-rooted as RESOLVE_IN_ROOT would.
`docker run` uses a different path, hence the split behaviour and the
misleading error.

Copying the content into place is inert: same bytes, same lookups, store paths
untouched, and nothing in this image adds users afterwards.

Independent of the Rust commits on this branch - touches only the Dockerfile,
so it cherry-picks to main on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With docker exec working, the Dev Containers flow got as far as installing the
server and then died:

  check-requirements.sh: line 48: getconf: command not found

getconf is the symptom, not the disease. The server ships a 121 MB
glibc-linked node built for FHS - it wants /lib/ld-linux-aarch64.so.1 as its
ELF interpreter, and this image has no /lib at all. Verified against the real
artifact: the unpatched binary fails with a bare "no such file or directory"
naming the binary rather than the missing loader.

Microsoft ships an escape hatch for exactly this. Given all three variables,
bin/code-server patchelfs its own node on first launch, and
check-requirements.sh exits 0 the moment it sees the LINKER one - before any
FHS probing. Verified end to end against the exact server build VS Code
1.131.0 pulls: patch via these paths, then `node -e` prints v24.18.0.

--inputs-from resolves nixpkgs through the flake's lock rather than the
ambient registry, so this reuses the glibc the image already links against;
marginal cost is patchelf and gcc-lib. -o roots them for the GC, as
/opt/hm-activation already does.

The unsuffixed symlinks exist so ENV encodes no store path, architecture, or
nix output-naming rule - `-o foo` on a non-default output lands at foo-lib,
which cost a debugging round here. The tests turn that class of mistake into a
build failure.

Not fixed via /etc/os-release ID=nixos, the other bypass in that script: it
parses the file with sed, which is absent from a docker exec PATH here, under
set -e. That trades missing-getconf for missing-sed. The env-var check runs
first and shells out to nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ning

Two defects, one root: the image is a Nix rootfs while Dev Containers and the
VS Code Server assume FHS. Both reported something other than themselves - an
exec failure that reads as a volume bug, and a missing getconf that is really
a missing dynamic linker - so the write-up leads with the misdirection and
records both wrong turns rather than only the fix.

Also defines "Dev image" in CONCEPTS.md. Three entries in the Dev container
cluster already leaned on the term without it being defined anywhere, and the
non-FHS property is what makes this whole class of failure predictable rather
than surprising.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The VSCODE_SERVER_CUSTOM_GLIBC_* mechanism was reverse-engineered from the
server's own shell scripts, but it is documented - under "older Linux
distributions", which is why it does not surface when searching for NixOS or
FHS. Same mechanism: "older distro" and "no FHS at all" fail its glibc
assumptions identically. Recording the link so the next person reads the docs
instead of the tarball.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

⊕ Entity-level changes

CONCEPTS.md

Status Type Name
+ heading Dev image

Dockerfile

Status Type Name
Δ chunk lines 61-80
Δ chunk lines 81-100
Δ chunk lines 101-120
- chunk lines 121-130
+ chunk lines 121-140
+ chunk lines 141-160
+ chunk lines 161-180
+ chunk lines 181-200
+ chunk lines 201-220
+ chunk lines 221-223

docs/solutions/integration-issues/vscode-dev-containers-on-a-nix-rootfs.md

Status Type Name
+ preamble (preamble)
+ heading VS Code Dev Containers fails against the Nix-based dev image
+ heading Problem
+ heading Symptoms
+ heading What Didn't Work
+ heading Solution
+ heading Why This Works
+ heading Prevention
+ heading Related Issues

home-manager/modules/dev/rust.nix

Status Type Name
Δ orphan module-level
Δ binding home.packages
- binding programs.zsh.initExtra
~ orphan module-level
+ binding home.activation.rustupDefaultToolchain

home-manager/profiles/dev.nix

Status Type Name
- orphan module-level
+ orphan module-level
+ binding imports

home-manager/profiles/work.nix

Status Type Name
Δ binding home

Summary: 19 added, 7 modified, 3 deleted across 6 files (1 added orphan, 2 modified orphans, 1 deleted orphan)

functions and classes, not lines · sem

Two gaps that only show up once someone actually builds in the container.

`linker \`cc\` not found` on the first dependency with a build script. rustup
ships rustc and cargo and deliberately stops there - linking is the system's
job - so a profile with no C toolchain cannot compile anything with a -sys
crate or a build.rs. stdenv.cc provides cc, ld and binutils wired to this
nixpkgs. pkg-config rides along because the -sys crates shell out to it and
its absence fails just as indirectly; it does not supply the libraries, so a
crate needing openssl still needs openssl in the profile or its vendored
feature.

rust-analyzer as a rustup component, plus an arch-independent symlink for
editors to name. VS Code's extension ships its own prebuilt server, which is
an unpatched FHS binary that cannot run here - it fails as `spawn ... ENOENT`,
which reads as "file missing" when the file is present and only its ELF
interpreter is absent. Same root cause as the VS Code Server node binary, and
the same reason the standing "rustup provides rust-analyzer" rule is load
bearing rather than stylistic: rustup's copy is patched by nixpkgs and runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@alycda
alycda marked this pull request as ready for review August 5, 2026 04:34
@alycda

alycda commented Aug 5, 2026

Copy link
Copy Markdown
Owner Author

@alycda

alycda commented Aug 5, 2026

Copy link
Copy Markdown
Owner Author

home-manager/modules/dev/rust.nix:30

P1 — stdenv.cc here reaches ditto, not just the container

work.nix:7 imports this module, so this line puts a C toolchain on the work MacBook's PATH too. On aarch64-darwin it resolves to clang-wrapper-21.1.8 (verified by eval), which means nixpkgs' cc, clang, ld, as and ar land ahead of Apple's — on a profile that also carries cmake, cocoapods, openjdk and flutter-via-puro, all of which shell out to cc/ld and expect Apple clang plus the macOS SDK.

That's outside what this PR set out to do: the commit fixes linker `cc` not found in the container, and as a side effect changes which compiler a laptop build picks up. pkg-config has the same shape, shadowing Homebrew's.

Suggested fix — scope both to Linux:

home.packages = with pkgs; [
  rustup
  bacon
] ++ lib.optionals pkgs.stdenv.hostPlatform.isLinux [
  stdenv.cc
  pkg-config
];

The mechanism is verified; the practical breakage on macOS is inferred — I couldn't test it on the actual machine.

@alycda

alycda commented Aug 5, 2026

Copy link
Copy Markdown
Owner Author

home-manager/profiles/dev.nix:5

P2 — this size justification is now off by ~4x, on the machine it names

The comment says rustup is 94 MiB and that the import "stays cheap for alyssa@dev-x86 on the 2012 MBP, whose disk headroom is the constraint PR #34 blew through." That was accurate when written, before stdenv.cc joined the module in ff59da3.

Measured against cache.nixos.org, deduplicated:

closure
rustup + bacon 100.9 MiB
+ stdenv.cc 408.0 MiB
marginal +307.1 MiB

Both container images carry it, alyssa@dev-x86 included. Either the comment gets corrected or the x86 image shouldn't carry the C toolchain — but a comment asserting a constraint the code no longer respects is the worse of the two, because it's what the next reader will trust.

If the stdenv.cc finding lands as lib.optionals … isLinux, this stays true for both Linux images and the number still needs updating.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant