feat: tag E2E job infrastructure resource groups with jobID by galchammat · Pull Request #7436 · Azure/ARO-HCP

Ghazi Alchammat (galchammat) · 2026-10-08T21:25:38Z

What

Tags the infrastructure resource groups a DEV E2E Prow job creates with jobID.aro-hcp-ci.redhat.com=<BUILD_ID>, the same key the e2e framework already sets on test resource groups. A job opts in by setting RUN_COST_TAG_VALUE in its openshift/release configuration; the literal BUILD_ID resolves to this job's Prow build ID, because ci-operator configuration cannot expand $(BUILD_ID) itself:

Resource groups	Mechanism
Service and management AKS node resource groups	the `build-config-override` Go generator, invoked by `hack/ci/build-config-override.sh`, sets `svc.aks.tags` / `mgmt.aks.tags` to the values ci00/ci01 otherwise inherit from `config/config.yaml` plus the run cost tag; AKS copies cluster tags onto node resource groups
Regional, service and management resource groups	`hack/ci/provision-environment.sh` resolves the same `RUN_COST_TAG_VALUE` and passes templatize's `--resource-group-tags`, set in full on resource groups the run creates

Both consumers read RUN_COST_TAG_VALUE from the environment they already share, so no tag handoff file is needed.

Why

Cost Management tag inheritance is enabled on the five E2E subscriptions (AROSLSRE-2276), so a resource group's tags reach the usage records of everything inside it. In a sampled run, AKS node resource groups were about 84% of the cost and templatize-created groups 5%; neither carried a run identifier. With them tagged, one Cost Management filter shows most of a job's spend across subscriptions.

Behavior and safety

Opt-in is per job: jobs that do not set RUN_COST_TAG_VALUE — including long-lived environments such as cspr that source the same override script — are never tagged, so shared clusters never carry the ID of whichever job deployed them last.
A tag value with characters outside [A-Za-z0-9._-], including whitespace, is rejected; RUN_COST_TAG_VALUE=BUILD_ID without BUILD_ID set is an error in both the generator and the provisioning shell.
Templatize sets --resource-group-tags in full only on resource groups the run creates. On an existing group each extra tag only replaces a key the group already carries, in any key casing; a group never gains a tag it did not have. ensureResourceGroupExists creates a group only on 404 and returns other Get errors, so a transient failure can no longer re-create, and re-tag, an existing group.

CI configuration change: this modifies the CI-invoked scripts hack/ci/build-config-override.sh and hack/ci/provision-environment.sh. A companion openshift/release PR (openshift/release#87928) declares RUN_COST_TAG_VALUE in the provision environment step and enables it for the dev e2e-parallel jobs.

Testing

Unit tests: tag validation (options_test.go); extra tags with persist rules on created and existing groups (arm_test.go); ensureResourceGroupExists against a fake ARM server, covering 404 create, existing-group preservation, re-stamping a carried key, untagged groups gaining nothing, and no writes on 401/403/429/500 (arm_resourcegroup_test.go).
Run cost tag resolution (override_test.go, cmd_test.go): sentinel resolution, explicit values, no opt-in, and rejection of invalid values; the sourced build-config-override.sh adapter end-to-end, including that no handoff file is published.
Rendered the ci01 configuration through templatize with the generated override: both svc.aks.tags and mgmt.aks.tags contain the job ID. aks-cluster-create and the Bicep csvTagsToObject parsers accept the dotted key.
go build, go vet, go test -race and golangci-lint on the changed packages.

To verify a run, query Resource Graph for resource groups with tag jobID.aro-hcp-ci.redhat.com equal to the job's build ID in the E2E infrastructure subscription. Usage records pick up inherited tags within about 24 hours.

Non-goals

HCP clusters' RP-managed resource groups (worker VMs and disks, about 9% of the sampled run) stay untagged. Tagging them from the test client costs extra ARM reads on every cluster GET; propagating tags in the RP, as AKS does for node resource groups, would cover them and also benefit customers.
Shared and pooled resources (leased identity containers, global Key Vault, registries, slot certificates) are never tagged with a job ID.
Historical charges are not retagged.

PR Checklist

openshift-merge-bot · 2026-10-08T21:25:40Z

Pipeline controller notification

This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration.

Use /test ? to list jobs and /pipeline help for commands.

openshift-ci · 2026-10-08T21:25:49Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copilot

🟡 Changes recommended

Job-ID validation and case-insensitive tag protection have correctness gaps, and the new resource-group safety branches lack direct coverage.

3 open findings

Whitespace stripping accepts malformed BUILD_ID values · New
Case-insensitive tag lookup can misattribute job ownership · New
Missing tests for safe GET error and tag preservation flows · New

What changed in this PR

Adds Prow job-ID tags to E2E-owned Azure resource groups for cross-subscription cost attribution.

Changes:

Adds creation-only resource-group tags to templatize.
Tags per-run AKS infrastructure from CI scripts.
Tags RP-managed HCP resource groups during creation and cleanup.

File	Description
`tooling/templatize/pkg/pipeline/run.go`	Adds resource-group creation tags to run options.
`tooling/templatize/pkg/pipeline/arm.go`	Applies tags only during resource-group creation.
`tooling/templatize/pkg/pipeline/arm_test.go`	Tests creation-tag composition.
`tooling/templatize/pkg/pipeline/arm_stack.go`	Passes tags to ARM stack resource groups.
`tooling/templatize/cmd/entrypoint/run/options.go`	Adds and validates the tagging CLI flag.
`tooling/templatize/cmd/entrypoint/run/options_test.go`	Tests tag validation.
`test/util/framework/per_test_framework.go`	Discovers and tags managed resource groups.
`test/util/framework/managed_resource_group_tags.go`	Implements managed-group job tagging.
`test/util/framework/managed_resource_group_tags_test.go`	Tests tagging helpers and ownership matching.
`test/util/framework/helpers_v20261001preview.go`	Tags managed groups after cluster creation.
`test/util/framework/helpers_v20260901preview.go`	Tags managed groups after cluster creation.
`test/util/framework/helpers_v20260630preview.go`	Tags managed groups after cluster creation.
`test/util/framework/helpers_v20251223preview.go`	Tags managed groups after cluster creation.
`test/util/framework/helpers_v20240610preview.go`	Tags managed groups after cluster creation.
`test/util/framework/deployment_helper.go`	Tags groups created by HCP Bicep deployments.
`hack/ci/provision-environment.sh`	Passes job tags to templatize.
`hack/ci/build-config-override.sh`	Adds job tags to per-run AKS configuration.
`docs/ci/execution.md`	Documents cost-attribution behavior.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

Direct and asynchronous cluster-creation paths remain untagged until teardown, leaving their lifetime costs unattributed.

1 open finding

Tag all HCP cluster creation paths after managed group creation · New

3 resolved since last review

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

The AKS tag replacement is case-sensitive and can retain a stale differently-cased job-ID tag.

1 open finding

Normalize Azure tag key comparison to prevent duplicate case variants · New

1 resolved since last review

Tag all HCP cluster creation paths after managed group creation

🧠 Review effort: Balanced

Copilot

🔵 Needs a closer look

Cross-cutting CI and resource-cleanup behavior requires human approval after the draft is finalized and pending checks complete.

0 open findings

1 resolved since last review

Normalize Azure tag key comparison to prevent duplicate case variants

🧠 Review effort: Balanced

Sajeel Irkal (sajeelirkal) · 2026-10-15T11:06:57Z

/test all

openshift-merge-bot · 2026-10-15T11:50:22Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-parallel

roi (roivaz) · 2026-10-15T18:54:39Z

Ghazi Alchammat (@galchammat) I would drop the HCP managed resource group tagging from this PR and merge the rest, given that managed group spend is a tiny fraction of the cost compared with AKS spend.
The approach of tagging HCP MRGs from the test client side requires extra ARM calls on every cluster GET, plus listing every RG in the subscription during cleanup and Bicep hooks. That adds unnecessary latency, read pressure on shared subscriptions, and is ultimately best-effort.

Instead, what we could do is implement tag propagation directly in the RP, same way AKS does. This would also benefit our customers, who would be able to apply tags and have them propagated to their HCP managed resource groups. I would check with the 1p/CS teams about the feasibility of implementing this.

What to drop from the PR:

The RP client policy (managedResourceGroupJobIDPolicy)
The post-Bicep hook in CreateBicepTemplateAndWait and pre-cleanup hook in deleteCreatedResources
Supporting test framework code and tests for managed-RG tagging

There's another issue with the PR as it is: it conflicts with #7502, which moves the build-config-override.sh shell script into go code in the test binary. You will need to rework this PR to apply the override logic to the new go code and wait until #7502 merges.

roi (roivaz) · 2026-10-15T18:55:37Z

/hold

Changes requested. Also conflicts with #7502.

Copilot

🔵 Needs a closer look

Required CI results remain unconfirmed for the provisioning-script and resource-group creation changes.

0 open findings

🧠 Review effort: Balanced

Ghazi Alchammat (galchammat) · 2026-10-15T20:00:06Z

roi (@roivaz) Dropped the managed resource group tagging and its supporting code. The port onto #7502's Go generator is ready (diff on top of #7502); I'll swap it into this PR once #7502 merges.

openshift-merge-bot · 2026-10-15T20:12:11Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-parallel

Copilot

🔵 Needs a closer look

CI provisioning and shared-resource tagging need human validation, alongside fixes to publication ordering and its regression test.

2 open issues

_{Review effort: Balanced}

openshift-ci · 2026-10-21T15:44:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: galchammat
Once this PR has been reviewed and has the lgtm label, please assign geoberle for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sajeel Irkal (sajeelirkal) · 2026-10-22T06:58:44Z

roi (roi (@roivaz)) Dropped the managed resource group tagging and its supporting code. The port onto #7502's Go generator is ready (diff on top of #7502); I'll swap it into this PR once #7502 merges.

#7502 is now merged

Ghazi Alchammat (galchammat) · 2026-10-24T03:42:28Z

/test integration

openshift-merge-bot · 2026-10-27T15:42:31Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-parallel

openshift-merge-bot · 2026-10-27T18:59:27Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-parallel

Ghazi Alchammat (galchammat) · 2026-10-27T23:01:36Z

/retest unrelated failure

Ghazi Alchammat (galchammat) · 2026-10-27T23:02:38Z

/retest

roi (roivaz) · 2026-10-28T10:52:32Z

+func newResourceGroupTags(creationTags map[string]string, persist bool) map[string]*string {
+	tags := make(map[string]*string, len(creationTags))
+	for key, value := range creationTags {
+		tags[key] = to.Ptr(value)
+	}
+	return computeResourceGroupTags(tags, persist)
+}


Don't wrap the function. Extend computeResourceGroupTags to adapt it to the new behavior instead. Also, any particular reason we don't want these cost tags reconciled instead of just applied during resource group creation?

roi (roivaz) · 2026-10-28T11:24:31Z

 	flags.StringVar(&opts.MSIMockCatalog, "msi-mock-pool-catalog", opts.MSIMockCatalog, "MSI mock catalog (default source-dir/dev-infrastructure/openshift-ci/msi-mock-pool.yaml)")
 	flags.StringVar(&opts.ARMHelperCatalog, "arm-helper-pool-catalog", opts.ARMHelperCatalog, "ARM helper catalog (default source-dir/dev-infrastructure/openshift-ci/arm-helper-pool.yaml)")
 	flags.StringVar(&opts.HypershiftOverlay, "hypershift-override-file", opts.HypershiftOverlay, "Hypershift overlay (default shared-dir/hypershift-image-overrides.yaml)")
+	flags.StringVar(&opts.TemplatizeSettings, "templatize-settings", opts.TemplatizeSettings, "Templatize settings deciding per-run environments (default source-dir/tooling/templatize/settings.yaml)")


Walking the whole config file to find just the AKS tags that are set looks way too complex. All other overrides blindly overwrite the config key value, let's just do the same with the AKS tags - check what tags are being set today for ci00/ci01 envs and just overwrite the whole config key with those and the new cost tag.

roi (roivaz) · 2026-10-28T11:27:44Z

 	flags.StringVar(&opts.ARMHelperCatalog, "arm-helper-pool-catalog", opts.ARMHelperCatalog, "ARM helper catalog (default source-dir/dev-infrastructure/openshift-ci/arm-helper-pool.yaml)")
 	flags.StringVar(&opts.HypershiftOverlay, "hypershift-override-file", opts.HypershiftOverlay, "Hypershift overlay (default shared-dir/hypershift-image-overrides.yaml)")
+	flags.StringVar(&opts.TemplatizeSettings, "templatize-settings", opts.TemplatizeSettings, "Templatize settings deciding per-run environments (default source-dir/tooling/templatize/settings.yaml)")
+	flags.StringVar(&opts.ServiceConfig, "service-config", opts.ServiceConfig, "Service config providing effective AKS tags (default source-dir/config/config.yaml)")


All other overrides are opt-in via environment variables in the openshift/release configuration - do the same here. Include a RUN_COST_TAG_VALUE env variable in the job configuration and use it, then you don't need to discover it by parsing the settings.yaml file. Keep it simple.

roi (roivaz) · 2026-10-28T11:29:09Z

 	flags.StringVar(&opts.HypershiftOverlay, "hypershift-override-file", opts.HypershiftOverlay, "Hypershift overlay (default shared-dir/hypershift-image-overrides.yaml)")
+	flags.StringVar(&opts.TemplatizeSettings, "templatize-settings", opts.TemplatizeSettings, "Templatize settings deciding per-run environments (default source-dir/tooling/templatize/settings.yaml)")
+	flags.StringVar(&opts.ServiceConfig, "service-config", opts.ServiceConfig, "Service config providing effective AKS tags (default source-dir/config/config.yaml)")
+	flags.StringVar(&opts.RunCostTagFile, "run-cost-tag-file", opts.RunCostTagFile, "File receiving the run cost tag for the provisioning shell (default shared-dir/run-cost-tag.env)")


If the step has an opt-in RUN_COST_TAG_VALUE env variable you won't need this either.

Attribute each DEV E2E Prow job's infrastructure spend by tagging the resource groups it creates with jobID.aro-hcp-ci.redhat.com=<BUILD_ID>. Cost Management tag inheritance on the E2E subscriptions applies the tag to usage records of everything inside them. - jobs opt in by setting RUN_COST_TAG_VALUE; the literal BUILD_ID resolves to this job's Prow build ID, because ci-operator configuration cannot expand $(BUILD_ID) itself - the config override generator adds the tag to the service and management AKS cluster tags, replacing the whole tags key with the values ci00/ci01 otherwise inherit from config/config.yaml; AKS copies them onto each node resource group - templatize's --resource-group-tags set the tags on resource groups a run creates; on an existing group they only replace tag keys the group already carries, so shared groups never gain a run-scoped tag - build-config-override and provision-environment.sh both resolve RUN_COST_TAG_VALUE from the same environment, so no tag handoff file is needed - the offline command test clears RUN_COST_TAG_VALUE and BUILD_ID, which Prow sets, with the other CI inputs it isolates (AROSLSRE-2394) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

openshift-merge-bot · 2026-10-29T03:40:10Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-parallel

openshift-ci · 2026-10-29T03:55:54Z

Ghazi Alchammat (@galchammat): The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-parallel	`3071de3`	link	true	`/test e2e-parallel`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copilot AI balanced review requested due to automatic review settings October 8, 2026 21:25

Copilot started reviewing on behalf of Ghazi Alchammat (galchammat) October 8, 2026 21:26

View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Comment thread hack/ci/build-config-override.sh Outdated

Comment thread test/util/framework/managed_resource_group_tags.go Outdated

Comment thread tooling/templatize/pkg/pipeline/arm.go

Ghazi Alchammat (galchammat) force-pushed the feat/e2e-jobid-cost-tags branch from b8c63e0 to 3cafdf9 Compare October 10, 2026 11:36

Copilot AI balanced review requested due to automatic review settings October 10, 2026 11:36

Copilot started reviewing on behalf of Ghazi Alchammat (galchammat) October 10, 2026 11:37

View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

Comment thread test/util/framework/helpers_v20240610preview.go Outdated

Ghazi Alchammat (galchammat) force-pushed the feat/e2e-jobid-cost-tags branch from 3cafdf9 to 8abe184 Compare October 10, 2026 11:57

Copilot AI balanced review requested due to automatic review settings October 10, 2026 11:57

Copilot started reviewing on behalf of Ghazi Alchammat (galchammat) October 10, 2026 11:57

View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

Comment thread hack/ci/build-config-override.sh Outdated

Copilot AI balanced review requested due to automatic review settings October 10, 2026 12:03

Ghazi Alchammat (galchammat) force-pushed the feat/e2e-jobid-cost-tags branch from 8abe184 to f4e11f4 Compare October 10, 2026 12:03

Copilot started reviewing on behalf of Ghazi Alchammat (galchammat) October 10, 2026 12:03

View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

Ghazi Alchammat (galchammat) marked this pull request as ready for review October 13, 2026 02:07

openshift-ci Bot removed the do-not-merge/work-in-progress label Oct 13, 2026

openshift-ci Bot requested review from David Eads (deads2k) and Gerd Oberlechner (geoberle) October 13, 2026 02:07

Copilot AI reviewed Oct 15, 2026

View reviewed changes

openshift-ci Bot added the needs-rebase label Oct 18, 2026

Copilot AI balanced review requested due to automatic review settings October 21, 2026 15:34

Copilot started reviewing on behalf of Ghazi Alchammat (galchammat) October 21, 2026 15:35

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread test/cmd/aro-hcp-tests/infra/build-config-override/cmd.go Outdated

Comment thread test/cmd/aro-hcp-tests/infra/build-config-override/cmd_test.go Outdated

openshift-ci Bot removed the needs-rebase label Oct 21, 2026

Ghazi Alchammat (galchammat) force-pushed the feat/e2e-jobid-cost-tags branch from 169d7b8 to 85e60ba Compare October 24, 2026 03:12

Ghazi Alchammat (galchammat) requested review from David Eads (deads2k) and Gerd Oberlechner (geoberle) and removed request for David Eads (deads2k) and Gerd Oberlechner (geoberle) October 24, 2026 03:40

Ghazi Alchammat (galchammat) force-pushed the feat/e2e-jobid-cost-tags branch from a4c25f2 to 8759c9f Compare October 27, 2026 18:21

roi (roivaz) reviewed Oct 28, 2026

View reviewed changes

Ghazi Alchammat (galchammat) force-pushed the feat/e2e-jobid-cost-tags branch from 8759c9f to 3071de3 Compare October 29, 2026 02:53

Ghazi Alchammat (galchammat) mentioned this pull request Oct 29, 2026

AROSLSRE-2394: declare RUN_COST_TAG_VALUE in the provision step and enable it for dev e2e-parallel jobs openshift/release#87928

Open

Conversation

Ghazi Alchammat (galchammat) commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Behavior and safety

Testing

Non-goals

PR Checklist

Uh oh!

openshift-merge-bot Bot commented Oct 8, 2026

Uh oh!

openshift-ci Bot commented Oct 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Sajeel Irkal (sajeelirkal) commented Oct 15, 2026

Uh oh!

openshift-merge-bot Bot commented Oct 15, 2026

Uh oh!

roi (roivaz) commented Oct 15, 2026

Uh oh!

roi (roivaz) commented Oct 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Ghazi Alchammat (galchammat) commented Oct 15, 2026

Uh oh!

openshift-merge-bot Bot commented Oct 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

openshift-ci Bot commented Oct 21, 2026

Uh oh!

Sajeel Irkal (sajeelirkal) commented Oct 22, 2026

Uh oh!

Ghazi Alchammat (galchammat) commented Oct 24, 2026

Uh oh!

openshift-merge-bot Bot commented Oct 27, 2026

Uh oh!

openshift-merge-bot Bot commented Oct 27, 2026

Uh oh!

Ghazi Alchammat (galchammat) commented Oct 27, 2026

Uh oh!

Ghazi Alchammat (galchammat) commented Oct 27, 2026

Uh oh!

roi (roivaz) Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

roi (roivaz) Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

roi (roivaz) Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

Ghazi Alchammat (galchammat) commented Oct 8, 2026 •

edited

Loading