Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,9 @@ Icon
Network Trash Folder
Temporary Items
.apdisk

# cf-now writes per-repo publish state (slug -> key) into ./.cfnow/state.json
# wherever it is run, so running the skill inside any checkout drops an
# untracked directory in it. No secret - it deliberately never persists a
# pre-signed URL - but it is machine state, not repo content.
.cfnow/
7 changes: 7 additions & 0 deletions CONCEPTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,13 @@ The merged symlink tree that becomes the active set of installed programs, assem
### Activation
The process that makes a generation live. It runs in ordered stages, and in practice the stage that links files runs before the stage that installs packages — so a failed activation can leave a machine with correct dotfiles and none of its tools. Activation is non-fatal by design in the container: the entrypoint reports the failure and still starts a shell.

## Sharing

### Slug
The opaque random identifier that names one published upload and stands in for access control. Note this inverts the usual meaning: a slug here is deliberately *unreadable*, because unguessability is the only thing keeping a private object private — the bucket is never public and content is reached solely through time-limited signed URLs.

A slug is stable and reusable: republishing to an existing slug replaces its content in place, so a link already shared keeps working. Uploads are ephemeral by default, expiring through a storage lifecycle rule unless explicitly published as permanent; permanence governs how long the *object* survives, which is independent of how long any signed URL for it remains valid.

## Flagged ambiguities

- *Profile* had been used for both a home-manager profile (a composition of modules, as in this repo's profiles) and a Nix profile (the installed-package tree, as in the base profile). These are distinct, and conflating them hides the fact that packages from both a Nix profile and a home-manager profile land in the same *user environment*.
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
---
title: "R2 reports AccessDenied for a working token: the auth probe needed wider scope than the tool"
date: 2026-08-05
category: integration-issues
module: tools/agents/skills/cf-now/scripts/publish.sh
problem_type: integration_issue
component: tooling
symptoms:
- "publish.sh dies with \"not authenticated. Configure the R2 token creds for profile 'alyssa-r2'\" while upload, presign and delete all work fine with that same token"
- "`aws s3 ls` returns \"An error occurred (AccessDenied) when calling the ListBuckets operation\" even though object operations on the target bucket succeed"
- "`aws s3 ls s3://<name>` returns AccessDenied rather than NoSuchBucket when the bucket name is simply wrong"
- "GetBucketLifecycleConfiguration returns AccessDenied, so the script's \"storage expires in ~7 days\" claim cannot be confirmed from the client"
root_cause: wrong_api
resolution_type: code_fix
severity: medium
tags:
- cloudflare-r2
- s3-api
- aws-cli
- least-privilege
- auth-probe
- cf-now
- presigned-urls
related_components:
- development_workflow
- documentation
---

# R2 reports AccessDenied for a working token: the auth probe needed wider scope than the tool

## Problem

`publish.sh` refused to run against a freshly minted, entirely functional R2 token, reporting that it was not authenticated. Every operation the script actually performs — upload, list objects, delete, pre-sign — worked when run by hand with the same profile. The auth probe, not the credentials, was wrong.

## Symptoms

- `error: not authenticated. Configure the R2 token creds for profile 'alyssa-r2'` on a token that could read, write, delete and pre-sign in the target bucket
- `aws --profile alyssa-r2 s3 ls` → `An error occurred (AccessDenied) when calling the ListBuckets operation`
- `aws --profile alyssa-r2 s3 ls s3://cfnow` → `AccessDenied` on ListObjectsV2, when the bucket was really named `cf-now` — a *name* error presenting as a *permission* error
- `s3api get-bucket-lifecycle-configuration` → `AccessDenied`

## What Didn't Work

- **Assuming the credentials were wrong.** The obvious reading of "not authenticated" is a bad key. Re-minting the token would have produced an identical failure, since the token was never the problem.
- **Reading `AccessDenied` on the bucket as a scope problem.** It was a typo — `cfnow` vs `cf-now`. R2 does not distinguish the two cases for the caller (see *Why This Works*), so no amount of staring at the token's bucket scoping would have revealed it. What actually found it was listing buckets through a *different* credential path (the Cloudflare MCP connector, which authenticates as the account) and seeing the real name.
- **Trusting a shell harness over a direct run.** A loop that captured `2>&1 >/dev/null` reported all three candidate probes as denied, including two that worked. Re-running each command directly gave the true result. When a probe matrix disagrees with a single manual run, believe the manual run.

## Solution

Probe with the narrowest operation the tool itself requires. `publish.sh` knows its bucket, so `HeadBucket` on that bucket tests exactly what the script goes on to use:

```bash
# publish.sh:98 — was: s3api list-buckets
if ! "${AWSP[@]}" s3api head-bucket --bucket "$BUCKET" >/dev/null 2>&1; then
die "cannot reach bucket '$BUCKET' with profile '$PROFILE' — check the bucket name, and that the R2 token is scoped to it (see SKILL.md → Authentication)"
fi
```

Measured against a live Object Read & Write token scoped to one bucket:

| Probe | Result |
|---|---|
| `s3api list-buckets` | AccessDenied |
| `s3api head-bucket --bucket <bucket>` | exit 0 |
| `s3api list-objects-v2 --bucket <bucket>` | exit 0 |

`setup.sh:46` deliberately **keeps** its account-level `list-buckets` probe: that script creates the bucket and writes its lifecycle rule, so it requires Admin Read & Write regardless, and probing account-level fails early with a clear message instead of dying at `create-bucket`. Its error now names the permission level and points at the alternative (create the bucket and rule in the dashboard, use an Object-scoped token, skip `setup.sh`).

The bucket default was corrected to the real name at `setup.sh:17`.

Fix opened in PR #47; unmerged as of this writing.

## Why This Works

Two independent facts combine into one misleading error.

**R2 returns `AccessDenied` rather than `NoSuchBucket` for any bucket outside the credential's reach.** This is deliberate — `NoSuchBucket` would let a caller enumerate which buckets exist by probing names. The cost is that a wrong bucket name and an out-of-scope bucket are indistinguishable from the client. No client-side check can tell them apart, so the error text must name both possibilities.

**An auth probe that needs broader rights than the tool encodes the wrong permission model.** `ListBuckets` is account-level; the least privilege the script needs is object access to one bucket. The probe was asserting a permission the tool never uses, so it failed exactly for correctly-scoped credentials — the tighter the credential, the more likely the false alarm. Tight scoping matters here beyond principle: an account-wide Admin token could delete every unrelated bucket in the account.

## Prevention

- **Probe with the narrowest operation the tool itself performs.** If a script only ever touches one bucket, probe that bucket. A liveness check that demands more privilege than the work is a false negative waiting for the first least-privilege credential.
- **When a platform collapses two failures into one error, say both in the message.** "cannot reach bucket X — check the name, and that the token is scoped to it" costs nothing and removes the entire misdiagnosis.
- **Distinguish setup-time from steady-state privilege.** Provisioning (create bucket, write lifecycle) legitimately needs admin; daily use does not. Two scripts, two permission levels, two probes — and say so in the error when the admin one fails.
- **Don't let a script assert what it cannot verify.** `publish.sh` prints `storage_expires=~7 days (bucket lifecycle rule)` as a flat claim, but reading the lifecycle config is an admin operation that returns AccessDenied under least privilege. If the rule were missing or mis-prefixed the script would still promise expiry. Either soften the wording or let the provisioning script be the only place that claims it.
- **Verify a probe matrix against direct runs.** Redirection order in a test harness silently inverted three results here.

## Related Issues

- PR #47 — the cf-now skill and these fixes
- [`../build-errors/home-manager-bash-collides-with-base-image-profile.md`](../build-errors/home-manager-bash-collides-with-base-image-profile.md) — the other case in this repo where a green-looking signal hid the real failure
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
---
title: "agenix never prunes, so a secret with no ciphertext behind it looks perfectly managed"
date: 2026-09-03
category: runtime-errors
module: home-manager/modules/agenix-activation.nix
problem_type: config_error
component: tooling
severity: high
symptoms:
- "`~/.aws/config` and `~/.aws/credentials` are symlinks into `~/.local/share/agenix/`, exactly like every real managed secret, but no `.age` file backs them and no `age.secrets` entry declares them"
- "`grep -r r2-credentials` over the whole repo — every branch — returns nothing, while the decrypted plaintext sits in the volume working fine"
- "The active generation's closure contains no reference to the secret name, yet the secret is present on disk"
- "Everything works, indefinitely, so nothing ever prompts you to look"
root_cause: config_error
resolution_type: config_change
related_components:
- authentication
- development_workflow
- documentation
tags:
- agenix
- ragenix
- secrets
- home-manager
- activation
- docker-volume
- orphaned-state
- credential-loss
- cloudflare-r2
---

# agenix never prunes, so a secret with no ciphertext behind it looks perfectly managed

## Problem

The R2 API token that `cf-now` depends on existed in exactly one place on
earth: a plaintext file in the `devhome` Docker volume. There was no
ciphertext in `secrets/`, no entry in `secrets/secrets.nix`, and no
`age.secrets` declaration in any module on any branch.

It had been that way for a month, and nothing was wrong. `~/.aws/credentials`
was a symlink into `~/.local/share/agenix/`, the AWS CLI authenticated, the
skill published files. From the outside it was indistinguishable from a
correctly managed secret.

The hazard is what that implies. `docker/CLAUDE-arm64.md` names
`docker volume rm devhome` as the recovery path for a failed home-manager
activation — so the documented fix for a broken container was also an
unrecoverable loss of a credential, requiring a re-mint in the Cloudflare
dashboard. **The only copy of a secret was in the thing the troubleshooting
guide tells you to delete.**

## Root cause

`home-manager/modules/agenix-activation.nix` installs secrets and never
removes them. `installOne` decrypts to a canonical path and symlinks any
`path` override at it:

```nix
run mv -f "$_canon.tmp" "$_canon"
run chmod "$_mode" "$_canon"

if [ "$_dest" != "$_canon" ]; then
run mkdir -p "$(dirname "$_dest")"
run ln -sfn "$_canon" "$_dest"
fi
```

then the module folds that over `config.age.secrets`:

```nix
${lib.concatMapStrings installOne (lib.attrValues config.age.secrets)}
```

It is a pure write loop over the *declared* set. Nothing enumerates what is
already in `~/.local/share/agenix` and nothing deletes. So the directory is
append-only across every generation the volume has ever seen, and it accretes
two kinds of junk:

1. **Removed secrets.** Drop an `age.secrets` entry and its decrypted
plaintext stays on disk forever. You did not un-deploy the credential; you
only stopped refreshing it.
2. **Hand-placed files.** Anything written into that directory by hand is
adopted by every subsequent activation, because activation never looks.

Both are invisible, because the *consumer* keeps working. A `path` override
is what makes this dangerous rather than merely untidy: it puts the symlink at
a real, load-bearing location like `~/.aws/credentials`, so the orphan is
wired into the tool it serves.

## How it surfaced

Not from a failure — from timestamps. Managed secrets are re-decrypted on
every activation and all carry the current generation's mtime. Orphans keep
the mtime of whenever they were written:

```
-r-------- Sep 3 20:19 agent-instructions git-config hackmd-api-token linear-api-key-{work,personal}
-r-------- Aug 5 07:06 cloudflare-api-token r2-config r2-credentials
```

Three files a month stale in a directory whose whole purpose is to be
rewritten at every container start. That skew is the tell, and it is the only
one — there is no error, no warning, and no missing file.

Confirm by asking the active generation what it actually installs. The
activation script renders one `_agenix_install` line per declared secret, so
that list *is* the declared set:

```bash
GEN=$(readlink -f ~/.local/state/home-manager/gcroots/current-home)
grep -oE '_agenix_install [^ ]+' "$GEN"/activate | sort -u
```

Anything in `~/.local/share/agenix` that this does not name is an orphan. On
the container that prompted this note it printed five names — exactly the five
carrying the current mtime.

Two traps in getting that command right, both of which produced a wrong answer
first:

- **`/nix/var/nix/profiles/per-user/$USER/home-manager` and
`~/.local/state/home-manager/gcroots/current-home` are the generation; a
`result` symlink in a checkout is not.** A stale `./result` from some earlier
`nix build` points at a generation that may never have been activated, and
grepping *that* answers a question about a build nobody switched to.
`home-manager generations` prints the real one.
- **Do not `grep -r` the closure for the secret name.** The activation script
embeds the module's comments, and `agenix-activation.nix` happens to use
`~/.local/share/agenix/cloudflare-api-token` as a worked example in its
header — so a recursive grep reports that secret as present when the only
match is prose. Match on `_agenix_install` instead.

## Solution

Capture the plaintext as a real secret before anything can destroy it, then
declare it. For the R2 credentials that meant encrypting both halves to the
repo's age recipient, armored (a binary `.age` blob does not reliably survive
every path into a commit):

```bash
PUB=age1mxz3lqtpxg35s2cct2gex76l66wrw9xpv5v8tk340gqxsdzxh5msq8vp09
rage -a -r "$PUB" -o secrets/personal/r2-credentials.age ~/.local/share/agenix/r2-credentials
```

then round-tripping it against the live file *before* trusting it — comparing
hashes rather than printing either:

```bash
a=$(rage -d -i ~/.age/personal-key.txt secrets/personal/r2-credentials.age | sha256sum)
b=$(sha256sum ~/.local/share/agenix/r2-credentials)
[ "${a%% *}" = "${b%% *}" ] && echo "round-trip OK"
```

then registering it in `secrets/secrets.nix` and declaring it in a module
(`home-manager/modules/tools/cf-now.nix`), with the `path` overrides that
reproduce the symlinks that were already there.

The alternative resolution is equally valid and should be the default for
anything unclaimed: **delete the orphan.** If nothing in the repo consumes it,
a stale plaintext credential in a volume is a liability, not an asset.

## Why this works

The point is not that the file moved — it is already on disk either way. The
point is that the credential is now *derivable*. Before, the volume was the
source of truth and the repo knew nothing; after, the repo is the source of
truth and the volume is a cache that any activation can rebuild. A
`docker volume rm devhome` becomes what the troubleshooting guide assumes it
is: an inconvenience.

This also fixes rotation, which was quietly broken in the same way.
`SKILL.md` told the user to rotate with `aws configure set`, but
`~/.aws/credentials` is a symlink into the agenix runtime dir — so that either
writes through the symlink or replaces it, and the next activation silently
reverts either way. A rotated token would appear to work until the next
switch. With the secret declared, rotation is
`just edit-secret personal/r2-credentials.age`.

## Prevention

- **Audit by mtime, not by presence.** `ls -la ~/.local/share/agenix` and
compare against the last activation. Anything older is undeclared. This is
a five-second check and it is the only signal available.
- **Treat a `path` override as a claim that must have ciphertext behind it.**
If a tool reads a credential from a stable home path, grep the repo for it.
A working tool proves nothing about where its config came from.
- **When removing an `age.secrets` entry, delete the decrypted file too.**
Removing the declaration does not un-deploy the secret; on every machine
that ever activated that generation, the plaintext is still sitting there.
- **`cloudflare-api-token` is a known remaining orphan** — same Aug 5 mtime,
and absent from the `_agenix_install` list. It is the case the comment trap
above was hiding: a recursive grep says it is declared, and it is not.
Nothing consumes it, so it wants deleting or capturing — deliberately left
alone rather than silently swept up with the R2 work.
- Adding a prune step to `agenix-activation.nix` would fix the class outright,
but it is not obviously safe: the activation script cannot distinguish an
orphan from a secret belonging to a *different* home-manager generation
sharing the same home, and deleting credentials is not a good place for a
heuristic. Documented rather than automated, on purpose.

## Related

- `docs/solutions/integration-issues/r2-auth-probe-fails-on-least-privilege-token.md`
— the other cf-now failure whose error text is equally uninformative; R2
answers `AccessDenied` for a missing identity, a revoked token and a wrong
bucket name alike
- `docs/solutions/runtime-errors/ragenix-edit-fails-on-identity-without-trailing-newline.md`
— the other way this secrets pipeline fails without saying so
- `home-manager/modules/agenix-activation.nix` — the module header explains why
activation-time decryption exists at all (ragenix installs via a systemd user
service; the container has no user systemd daemon)
1 change: 1 addition & 0 deletions home-manager/modules/common.nix
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ in
./dev/nix-lang.nix
./tools/agent-skills.nix
./tools/agents.nix
./tools/cf-now.nix
./tools/cheat.nix
./tools/claude-code.nix
./tools/crush.nix
Expand Down
2 changes: 2 additions & 0 deletions home-manager/modules/tools/agent-skills.nix
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
#
# Repo skills:
# brag-doc - promo-packet impact entries from raw work notes
# cf-now - private file sharing from Cloudflare R2 (pre-signed URLs)
# commit-craft - commit-message craft + jj describe/push workflow
# failure-doc - failures as deliberate-learning records
# html-deck - self-contained single-file HTML slide decks
Expand Down Expand Up @@ -77,6 +78,7 @@ let

repoSkills = [
"brag-doc"
"cf-now"
"commit-craft"
"failure-doc"
"hackmd-cli"
Expand Down
Loading
Loading