Skip to content

fix(indexer): mainnet snapshot upload bucket + scope task S3 policy - #172

Open
tqin7 wants to merge 4 commits into
mainfrom
tq/fix-mainnet-snapshot-bucket
Open

tqin7 wants to merge 4 commits into
mainfrom
tq/fix-mainnet-snapshot-bucket

Conversation

@tqin7

@tqin7 tqin7 commented Sep 18, 2026

Copy link
Copy Markdown

Summary

Two coupled fixes for the mainnet snapshot full node (mainnet-indexer-full-node-snapshot-service, ap-northeast-1):

  1. snapshot_full_node_ap_northeast_1.tf — the entry_point passed var.s3_snapshot_bucket (mainnet-full-node-snapshots), but for mainnet the bucket resource is created as ${local.account_id}-${var.s3_snapshot_bucket} (332066407361-mainnet-full-node-snapshots) per the TODO(IND-457) in s3_bucket.tf. So the task uploaded to a bucket that doesn't exist and failed every cycle with NoSuchBucket (silently — the script's aws s3 cp has || true). Now references aws_s3_bucket.indexer_full_node_snapshots.bucket so the task and the bucket can't drift apart in any env.

  2. snapshot_iam.tf — the task's S3 policy was resources = ["*"] (every bucket in the account). Scoped to the snapshot bucket resource ARN (object RW + bucket list).

Impact / background

  • Mainnet snapshot uploads had been failing with NoSuchBucket for 8+ days (surfaced while fixing an unrelated credential-leak issue on the same box — see v4-chain#3395, which removed aws s3 cp --debug that was logging STS session tokens into Datadog).
  • Stopgap already applied manually so uploads resume now: ECS task def mainnet-indexer-full-node-snapshot-task:81 points at 332066407361-mainnet-full-node-snapshots, and the live IAM policy mainnet-indexer-apne1-ecs_task_s3_policy was scoped to that bucket (version v5). This PR makes both durable — the next terraform apply will reconcile to the same end state instead of reverting them.

Notes

  • The bucket has an expire-old-snapshots lifecycle rule gated on var.enable_s3_snapshot_lifecycle — worth confirming it's enabled for mainnet so snapshots don't accumulate.
  • Snapshots will be of an old height until the node finishes replaying (separate issue: it was crash-looping on a missing cosmovisor v9.5 binary and is catching up).

🤖 Generated with Claude Code

@tqin7 tqin7 changed the title fix(indexer): mainnet snapshot upload bucket + scope task S3 policy (IND-457) fix(indexer): mainnet snapshot upload bucket + scope task S3 policy Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant