Proposal: hierarchical metric tag maps in standalone.yml

[liaodn]

Background

Currently StandaloneStarter only adds a single env common tag to the global registry. In multi-cluster deployments, it would be nice to have more flexible tagging at different levels.

Rough Idea

Introduce a metricsTags map at different levels, something like:

Process-level — global tags for all metrics in the process:

metricsTags:
  region: gz
  deployment: canary

Cluster-level — tags scoped to a specific host-cluster:

agent:
  hostcluster:
    metricsTags:
      cluster: regional-a
      cluster_scope: regional

Subservice propagation — cluster-level tags could also propagate to base-kv, base-rpc, etc.

Thoughts on Precedence

For the same key: process-level < cluster-level < metric-local tags. But this is open for discussion.

Possible Phases

This is quite a bit of scope, so it might make sense to phase it:

1. Process-level metricsTags
2. Cluster-level metricsTags under agent.hostcluster
3. Propagation to base-cluster subservices

Would love to hear thoughts on whether this direction makes sense, and if the naming / precedence / phasing look reasonable.

[Gujiawei-Edinburgh]

Background

Currently StandaloneStarter only adds a single env common tag to the global registry. In multi-cluster deployments, it would be nice to have more flexible tagging at different levels.

Rough Idea

Introduce a metricsTags map at different levels, something like:

Process-level — global tags for all metrics in the process:

metricsTags:
region: gz
deployment: canary
Cluster-level — tags scoped to a specific host-cluster:

agent:
hostcluster:
metricsTags:
cluster: regional-a
cluster_scope: regional
Subservice propagation — cluster-level tags could also propagate to base-kv, base-rpc, etc.

Thoughts on Precedence

For the same key: process-level < cluster-level < metric-local tags. But this is open for discussion.

Possible Phases

This is quite a bit of scope, so it might make sense to phase it:

1. Process-level metricsTags
2. Cluster-level metricsTags under agent.hostcluster
3. Propagation to base-cluster subservices

Would love to hear thoughts on whether this direction makes sense, and if the naming / precedence / phasing look reasonable.

I think introducing metricsTags would improve observability, especially for multi-cluster and multi-deployment setups.

To avoid tag cardinality explosion, it might be helpful to explicitly scope which tags are expected at each level (e.g., process vs cluster), and encourage low-cardinality dimensions such as region, deployment, and cluster. This would also make precedence rules easier to reason about.

Regarding propagation and conflict resolution, my understanding is that these should ideally be handled at construction time (i.e., one-shot), where tags are merged and injected into each component’s metrics context, rather than resolved dynamically at runtime.

[popduke]

Background

Currently StandaloneStarter only adds a single env common tag to the global registry. In multi-cluster deployments, it would be nice to have more flexible tagging at different levels.

Rough Idea

Introduce a metricsTags map at different levels, something like:

Process-level — global tags for all metrics in the process:

metricsTags:
region: gz
deployment: canary
Cluster-level — tags scoped to a specific host-cluster:

agent:
hostcluster:
metricsTags:
cluster: regional-a
cluster_scope: regional
Subservice propagation — cluster-level tags could also propagate to base-kv, base-rpc, etc.

Thoughts on Precedence

For the same key: process-level < cluster-level < metric-local tags. But this is open for discussion.

Possible Phases

This is quite a bit of scope, so it might make sense to phase it:

1. Process-level metricsTags
2. Cluster-level metricsTags under agent.hostcluster
3. Propagation to base-cluster subservices

Would love to hear thoughts on whether this direction makes sense, and if the naming / precedence / phasing look reasonable.

Thanks for putting this proposal together — this is a good direction.

On the clusterConfig.env field: yeah, today it’s not just used to identify the cluster, it also gets reused as a global tag on all metrics. Having a separate metricTags config makes sense to me. It decouples concerns and gives users more flexibility instead of overloading env.

I think introducing metricsTags would improve observability, especially for multi-cluster and multi-deployment setups.

To avoid tag cardinality explosion, it might be helpful to explicitly scope which tags are expected at each level (e.g., process vs cluster), and encourage low-cardinality dimensions such as region, deployment, and cluster. This would also make precedence rules easier to reason about.

Regarding propagation and conflict resolution, my understanding is that these should ideally be handled at construction time (i.e., one-shot), where tags are merged and injected into each component’s metrics context, rather than resolved dynamically at runtime.

For the tag definitions themselves (labels and value cardinality), I’d suggest we keep the boundary simple: BifroMQ provides the ability to attach custom process-level metric tags, but what tags to use and how to control their cardinality should be up to the user. Different setups have very different needs, so hardcoding an opinionated “best practices” set here probably won’t age well.

As for base-cluster, I think it would help to explicitly add a clusterEnv tag to the metrics it emits. That should make cross-region deployments easier to reason about, since metrics will carry clearer deployment context.

[liaodn]

Thanks for the feedback. I’ll go ahead and submit a PR for this later.