Repository navigation
Attest the state file a failed apply rewrote - #36
Conversation
terraform apply writes the state file to S3 even when it fails part-way, but apply.yml only attests the state file in reset-drift-detection, which needs the apply job to succeed. A failed CI apply therefore leaves a state file Kosli has no provenance for, and drift detection reports it exactly as if terraform had been run from a laptop. It stays that way until the same component next applies successfully. This happened on 2026-10-06 to the cyber-dojo prod apply of custom-start-points: the apply timed out waiting for the ECS service to stabilise after writing the state file, and aws-prod-terraform-drift-detection stayed non-compliant until a forced redeployment the next day. https://github.com/cyber-dojo/aws-prod-co-promotion/actions/runs/37458113081/job/112253462088 base.yml now records the state file's ETag before Apply and, only if Apply fails, attests the state file when its ETag has changed. A state file the failed apply did not change is left unattested, since it may have been written outside CI. The bucket and key come from the backend config tf plan initialised, so they are the ones terraform used. The ETag step is continue-on-error so it can never block an apply. A cancelled apply, or one whose AWS credentials have expired, attests nothing, so it is still reported as drift. A failed apply also does not reset the drift plan, so infrastructure it changed is reported as drift by the next detect-drift run until an apply succeeds. The README says so. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
On 2026-10-06 this happened to the prod apply of cyber-dojo's custom-start-points,
|
|
Is this new behaviour desirable? As If an Therefore, the environment in Kosli goes red and everyone runs around flapping their arms. And someone (an engineer, an auditor) asks -- why did the environment go non-compliant and the answer is "because a deploy failed part way through leaving us in an inconsistent state". It is inconsistent precisely because merged code is not fully deployed. With the change in this PR, the partially successful Perhaps this is a philosophical question -- if we only apply 50% of a change that's on |
|
With the specific CyberDojo case you described, it looks like the Attesting the statefile after that timeout is problematic because we don't know what the state of the infrastructure is -- will the deployment succeed in AWS in a few minutes? Or will AWS eventually decide that the ECS Service couldn't be stabilised and roll it back to the previous release? At the time at which I would argue that the problem is in step 3 of your description -- that you were unable to run |
gsavage
left a comment
gsavage
left a comment
There was a problem hiding this comment.
I'm marking as "Request changes" so that we don't accidentally merge this without discussing the comments I've posted on the PR.
I don't think the change here is the right change, but I can be persuaded to view it differently
Attesting the state file a failed apply rewrote would report a partly-applied commit as compliant. Terraform writes the state file as it goes, so an apply that fails part-way leaves infrastructure only partly at the merged commit. That is an inconsistency worth seeing, and drift detection reporting it is the right outcome, not a false positive. So this drops the base.yml change from the previous commit, and the README now says that a failed apply attests nothing on purpose, and that the environment stays non-compliant until an apply of the same component succeeds, which is the remedy once the failure's cause is fixed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2 participants
terraform apply writes the state file to S3 even when it fails
part-way, but apply.yml only attests the state file in
reset-drift-detection, which needs the apply job to succeed. A failed
CI apply therefore leaves a state file Kosli has no provenance for,
and drift detection reports it exactly as if terraform had been run
from a laptop. It stays that way until the same component next
applies successfully.
base.yml now records the state file's ETag before Apply and, only if
Apply fails, attests the state file when its ETag has changed. A state
file the failed apply did not change is left unattested, since it may
have been written outside CI. The bucket and key come from the backend
config tf plan initialised, so they are the ones terraform used. The
ETag step is continue-on-error so it can never block an apply.
A cancelled apply, or one whose AWS credentials have expired, attests
nothing, so it is still reported as drift. A failed apply also does
not reset the drift plan, so infrastructure it changed is reported as
drift by the next detect-drift run until an apply succeeds. The README
says so.
Co-Authored-By: Claude Opus 5.5 (1M context) noreply@anthropic.com