Skip to content

ci(tests): stop reporting success for tests that never ran - #193

Merged
NathaelB merged 1 commit into
mainfrom
chantier/v12-ci-services
Sep 12, 2026
Merged

ci(tests): stop reporting success for tests that never ran#193
NathaelB merged 1 commit into
mainfrom
chantier/v12-ci-services

Conversation

@NathaelB

Copy link
Copy Markdown
Owner

Part of #177.

The bug

Four test suites landed in #188 and #189 that report success without running.

aether-s3::object_store, aether-api::archive_bucket, aether-transit::key_provider and aether-api::wrapping_key all skip themselves when their endpoint is unset, which is what lets a developer run cargo test without a store or a key manager. Nothing in the workflow ever set one, so in CI they skipped, and a skip reads as a pass.

This is the same shape as the bug integration-postgres already carries a comment about:

Every rule that lives in SQL was untested: the repository tests skip themselves when DATABASE_URL is unset, and it was unset here, so they reported success without running.

I reintroduced it two chantiers later, which is a reasonable argument that the guard belongs beside every test that can skip rather than in one job's comment.

The fix

  • Service containers for RustFS and OpenBao on a new integration-services job.
  • REQUIRE_OBJECT_STORE_ENDPOINT and REQUIRE_KEY_MANAGER_ADDRESS, the same mechanism REQUIRE_DATABASE_URL uses: set in CI, they turn a skip into a failure, so losing a service container fails the job instead of quietly emptying it.

Verified in both directions rather than assumed. With the services up, all four suites run. With REQUIRE_OBJECT_STORE_ENDPOINT=1 and no endpoint:

panicked at libs/aether-s3/tests/object_store.rs:39:
REQUIRE_OBJECT_STORE_ENDPOINT is set but OBJECT_STORE_ENDPOINT is not:
these tests would have skipped and reported success

Two details worth the lines they cost

RustFS needs RUSTFS_SSE_S3_MASTER_KEY here too. Without one it refuses every encrypted write rather than storing plaintext. That is the right failure, and exercising it is more useful than configuring around it. The value is the same local stand-in docker-compose.yaml uses and is not a secret.

Neither image ships a health check, and an S3 endpoint answers an unauthenticated request with 403, so the wait loop treats answering at all as ready rather than a successful status. This is the same mistake that made the compose health check fail earlier in this chantier: curl -f turns a perfectly healthy 403 into a failure.

build does not depend on this job, matching how integration-postgres is already wired.

What it is worth

The verification sections of #188 through #192 claim these suites cover real infrastructure. That was true on my machine and false in CI. It is true in both now.

cargo clippy --workspace --all-targets    clean
cargo nextest run --workspace             988 passed, 1 skipped
                                          with every REQUIRE_ guard armed

The object store and key manager tests skip themselves when their endpoint is
unset, and nothing in the workflow set one. So four suites reported success
having exercised nothing, in the same shape as the bug integration-postgres
already carries a comment about: five repository tests went a whole chantier
that way.

Service containers for RustFS and OpenBao, and the same REQUIRE_ guard
integration-postgres uses, so losing a container fails the job instead of
quietly emptying it.

RustFS needs its master key here too. Without one it refuses every encrypted
write rather than storing plaintext, which is the right failure and one worth
exercising rather than configuring around.

What these cover is what a mock cannot answer: path style addressing, absence
of a key, prefix listing, whether the store applies the rules it was asked for,
and whether a rotation leaves an archive taken before it still readable.
@NathaelB NathaelB added the enhancement New feature or request label Sep 12, 2026
@NathaelB NathaelB self-assigned this Sep 12, 2026
@NathaelB NathaelB added the enhancement New feature or request label Sep 12, 2026
@NathaelB
NathaelB merged commit 8dffd43 into main Sep 12, 2026
17 checks passed
@NathaelB
NathaelB deleted the chantier/v12-ci-services branch September 12, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant