Skip to content

fix: take the verdict from the test service instead of up - #39

Merged
fey merged 2 commits into
masterfrom
fix/verdict-from-test-service
Aug 25, 2026
Merged

fey merged 2 commits into
masterfrom
fix/verdict-from-test-service

Conversation

@fey

@fey fey commented Aug 17, 2026 •

Copy link
Copy Markdown
Collaborator

Hexlet ticket: DEVELOPMENT-432 (FEEDBACK-299)

What

-docker compose run app make setup
-docker compose -f docker-compose.yml up --abort-on-container-exit
+docker compose -f docker-compose.yml run --rm app make setup
+docker compose -f docker-compose.yml run --rm test   # up, if the image has no test service

Plus: dependencies' logs are dumped when the test run fails, and the test fixture moves to hexletprojects/hexlet-project-source-ci_en:latest.

Why

Projects are being accepted with failing tests right now. The rollout of the test service (FEEDBACK-299) moved command: make test off app, so a console project's app has no command, falls back to the base image CMD, exits 0 within a second and triggers the abort. Without --exit-code-from, up reports the exit code of whichever container stopped the run — so the verdict is 0 while the tests are killed mid-flight. Reproduced on ru/js_l2_differ3_project:

$ docker compose -f docker-compose.yml up --abort-on-container-exit; echo $?
app-1 exited with code 0
test-1 exited with code 2
0

$ docker compose -f docker-compose.yml run --rm test; echo $?
2

run returns the test container's own exit code and still waits for depends_on with condition: service_healthy. --exit-code-from is not the answer: it implies --abort-on-container-exit, which kills any service that exits before the tests, so a console project would have to keep app alive artificially.

-f docker-compose.yml on both commands. The setup step lacked it, which is already a bug: docker-compose.override.yml ships inside the project image and switches app to its dev command. With run --rm test the price goes up — app comes up as a dependency, and its healthcheck may never pass under the dev command.

Logs on failure. up --abort-on-container-exit interleaved app/db/caddy output into the Actions log; run --rm test attaches to the test container only. That is exactly the diagnostics the server projects need.

The fallback, and why it is not the reverted variant B

The action is pinned by every student workflow as hexlet/project-action@release, and the monolith validates that literal string (app/schemas/github_action_project_check.json). Merging to master rebuilds release and switches everyone at once — there is no gradual rollout and no per-project revert.

An unconditional run --rm test would therefore be a flag day: a project whose published image has no test service answers no such service: test with exit code 1, indistinguishable from failing tests. Those projects still carry command: make test on app, so up reports their verdict correctly today. Detecting the service keeps them where they are and moves only the projects that are broken:

Published image Today After this PR
has test falsely accepts verdict from test
no test correct via up unchanged, up

The check reads docker compose config --services from the compose file unpacked out of the image, i.e. the same source run will use — so image-vs-git divergence cannot fool it.

This is not variant B, reversed on 2026-08-17: that proposed auto-detect as the permanent contract, feeding --exit-code-from. This is a temporary bridge with its removal condition written next to it, and the sibling runner already ships exactly this shape — hexlethq/hexlet-project-source-ci, gitlab-ci-template.yml on release, merged 2026-08-24 (!9). The up branch goes away once every published image carries test.

State of the rollout (2026-08-25)

  • hexlethq/hexlet-project-source-ci!9 merged; hexlet-project-source-ci_en:latest republished 2026-08-24, verified to carry the test service — the fixture here is green again.
  • Image sweep (bin/agent/image-sweep.ts in hexlet-exercise-kit): 0 projects with an image older than the rollout commit.
  • Direct scan of origin/HEAD: test present in 166 of 178 project repos; missing in 3 es clones parked on branches with open MRs (data_analytics_charts, devops_bulletin_board_observability, php_testing_page_downloader); 8 have no docker-compose.yml at all.
  • 38 projects have no published image whatsoever (36 of them es) — those already fail at docker pull, before any of this.

With the fallback, none of the above blocks the merge: it is a strict improvement over the current behaviour for every project.

Campaign state lives in hexlet-exercise-kit, docs/project-test-service.md. Supersedes #36.

The exit code of `up` comes from whichever container stops first, so app and
db shut down after successful tests turned a green run red. `run --rm test`
returns the test container's own exit code and still waits for depends_on
with condition: service_healthy.

Both commands now pass -f docker-compose.yml: the project image also carries
docker-compose.override.yml, which switches app to its dev command.

On failure the dependencies' logs are dumped, since `run` attaches to the
test container only and server projects would otherwise lose their only clue.

The test fixture moves to hexletprojects/hexlet-project-source-ci_en:latest —
the bare tag is frozen at 2022 and has no test service.

Hexlet ticket: FEEDBACK-299

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An unconditional `run --rm test` would be a flag day: the action is pinned by
every student workflow as `hexlet/project-action@release`, and a project whose
published image has no `test` service answers `no such service: test` with exit
code 1 — indistinguishable from failing tests.

Those projects still carry `command: make test` on `app`, so `up` reports their
verdict correctly today. Detecting the service keeps them on `up` and moves only
the projects that already have `test` — the ones falsely accepting right now,
since their `app` has no command, exits 0 first and aborts the run before the
tests finish.

The same bridge is already live in the sibling runner
(hexlethq/hexlet-project-source-ci, `gitlab-ci-template.yml` on `release`). The
`up` branch goes away once every published image carries `test`.

Hexlet ticket: DEVELOPMENT-432 (FEEDBACK-299)
@fey
fey marked this pull request as ready for review August 25, 2026 18:44
@fey
fey merged commit 4db0e11 into master Aug 25, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant