From dcb7fcd76e01d65a66aa7cef7213ca65ee416b15 Mon Sep 17 00:00:00 2001 From: thc1006 <84045975+thc1006@users.noreply.github.com> Date: Wed, 26 Aug 2026 11:23:28 +0000 Subject: [PATCH 1/5] RW 2.0-rc.5: allow exemplar-only time series Prometheus stores exemplars per series, not per sample. populateV2TimeSeries writes one output TimeSeries per queue item, and for an exemplar item it appends only an exemplar, so an exemplar can leave the sender in a TimeSeries of its own. The specification requires every TimeSeries to carry a sample or a histogram, so that shape is not covered. Describe it instead of forbidding it, and keep the expectation that the exemplars travel in the same request as the series they belong to. Related to prometheus/prometheus#17857 and prometheus/prometheus#16944. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> --- docs/specs/prw/remote_write_spec_2_0.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/specs/prw/remote_write_spec_2_0.md b/docs/specs/prw/remote_write_spec_2_0.md index 774d0138a..0f01167dc 100644 --- a/docs/specs/prw/remote_write_spec_2_0.md +++ b/docs/specs/prw/remote_write_spec_2_0.md @@ -362,7 +362,7 @@ For every `TimeSeries` message: -* At least one element in `samples` or in `histograms` MUST be provided. A `TimeSeries` MUST NOT include both `samples` and `histograms`. For series which (rarely) would mix float and histogram samples, a separate `TimeSeries` message MUST be used. +* At least one element in `samples`, in `histograms`, or in `exemplars` MUST be provided. A `TimeSeries` MUST NOT include both `samples` and `histograms`. For series which (rarely) would mix float and histogram samples, a separate `TimeSeries` message MUST be used. A `TimeSeries` carrying exemplars but neither samples nor histograms represents series-level exemplars for the series identified by `labels_refs`. * MAY contain labels e.g. referencing trace or request ID. If the exemplar references a trace it SHOULD use the `trace_id` label name, as a best practice. * MUST contain a timestamp. While exemplar timestamps are optional in Prometheus/Open Metrics exposition formats, the assumption is that a timestamp is assigned at scrape time in the same way a timestamp is assigned to the scrape sample. Receivers require exemplar timestamps to reliably handle (e.g. deduplicate) incoming exemplars. +* MAY be sent in a `TimeSeries` that carries neither samples nor histograms. In that form the exemplars are associated with the series identified by `labels_refs`, rather than with a particular sample or histogram. +* SHOULD be sent in the same request as the samples or histograms of the series they belong to, including when they are sent in a `TimeSeries` of their own. ## Out of Scope From 8f9d1fa291bfd2d77881228cfe976ba6c5ee3417 Mon Sep 17 00:00:00 2001 From: thc1006 <84045975+thc1006@users.noreply.github.com> Date: Wed, 26 Aug 2026 14:43:14 +0000 Subject: [PATCH 2/5] RW 2.0-rc.5: match the embedded protobuf comments The copy of io.prometheus.write.v2 in this document still says a TimeSeries specifies samples or histograms, and that exemplars belong to the series' samples. Neither holds once a TimeSeries can carry exemplars on its own. krajorama spotted this in review. The same three comments live in prompb/io/prometheus/write/v2/types.proto, which is the source of truth, so they are changed there too. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> --- docs/specs/prw/remote_write_spec_2_0.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/specs/prw/remote_write_spec_2_0.md b/docs/specs/prw/remote_write_spec_2_0.md index 0f01167dc..be72a2b28 100644 --- a/docs/specs/prw/remote_write_spec_2_0.md +++ b/docs/specs/prw/remote_write_spec_2_0.md @@ -257,22 +257,23 @@ message TimeSeries { // or start timestamp. repeated uint32 labels_refs = 1; - // Timeseries messages can either specify samples or (native) histogram samples - // (histogram field), but not both. For a typical sender (real-time metric - // streaming), in healthy cases, there will be only one sample or histogram. + // TimeSeries messages must specify at least one float sample, native histogram sample + // or exemplar. Samples and histograms must not be specified at the same time. + // Exemplars may be specified without samples or histograms. For a typical sender + // (real-time metric streaming), in healthy cases, there will be only one sample or histogram. // // Samples and histograms are sorted by timestamp (older first). repeated Sample samples = 2; repeated Histogram histograms = 3; - // exemplars represents an optional set of exemplars attached to this series' samples. + // exemplars represents an optional set of exemplars for this series. repeated Exemplar exemplars = 4; // metadata represents the metadata associated with the given series' samples. Metadata metadata = 5; } -// Exemplar is an additional information attached to some series' samples. +// Exemplar contains additional information associated with a series. // It is typically used to attach an example trace or request ID associated with // the metric changes. message Exemplar { From 81c879afb444f3096f26290b84e538b84ffa508e Mon Sep 17 00:00:00 2001 From: thc1006 <84045975+thc1006@users.noreply.github.com> Date: Fri, 11 Sep 2026 08:34:54 +0000 Subject: [PATCH 3/5] RW 2.0-rc.5: drop the same-request SHOULD What this PR is about is that an exemplar-only TimeSeries is valid and that it identifies its series by labels_refs. Whether a sender keeps a series' exemplars in the same request as its samples is a different question. It is about interoperability rather than about what the wire format allows, and it is the part of this change that is under discussion in review. Taking it out leaves the wire format statement on its own. Co-location is worth writing down, just not in this rule. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> --- docs/specs/prw/remote_write_spec_2_0.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/specs/prw/remote_write_spec_2_0.md b/docs/specs/prw/remote_write_spec_2_0.md index be72a2b28..d123b753b 100644 --- a/docs/specs/prw/remote_write_spec_2_0.md +++ b/docs/specs/prw/remote_write_spec_2_0.md @@ -454,7 +454,6 @@ Rationales: https://github.com/prometheus/proposals/blob/alexg/remote-write-20-p * MAY contain labels e.g. referencing trace or request ID. If the exemplar references a trace it SHOULD use the `trace_id` label name, as a best practice. * MUST contain a timestamp. While exemplar timestamps are optional in Prometheus/Open Metrics exposition formats, the assumption is that a timestamp is assigned at scrape time in the same way a timestamp is assigned to the scrape sample. Receivers require exemplar timestamps to reliably handle (e.g. deduplicate) incoming exemplars. * MAY be sent in a `TimeSeries` that carries neither samples nor histograms. In that form the exemplars are associated with the series identified by `labels_refs`, rather than with a particular sample or histogram. -* SHOULD be sent in the same request as the samples or histograms of the series they belong to, including when they are sent in a `TimeSeries` of their own. ## Out of Scope From 6a3e7c92a63ec10299966f1004ae7c177953fd14 Mon Sep 17 00:00:00 2001 From: thc1006 <84045975+thc1006@users.noreply.github.com> Date: Wed, 16 Sep 2026 18:49:15 +0000 Subject: [PATCH 4/5] RW 2.0-rc.5: rule out metadata-only time series The at-least-one rule names samples, histograms and exemplars, so a TimeSeries carrying only metadata is already invalid. That is left to be inferred from a list rather than stated, which bwplotka flagged in review. State it in the rule. Then say what metadata means on the shape this PR adds, because an exemplar-only TimeSeries still carries the field and the Prometheus sender fills it. Calling it series-level and letting Receivers ignore it means a sender cannot rely on that shape to deliver metadata. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> --- docs/specs/prw/remote_write_spec_2_0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/specs/prw/remote_write_spec_2_0.md b/docs/specs/prw/remote_write_spec_2_0.md index d123b753b..3a651d49b 100644 --- a/docs/specs/prw/remote_write_spec_2_0.md +++ b/docs/specs/prw/remote_write_spec_2_0.md @@ -363,7 +363,7 @@ For every `TimeSeries` message: -* At least one element in `samples`, in `histograms`, or in `exemplars` MUST be provided. A `TimeSeries` MUST NOT include both `samples` and `histograms`. For series which (rarely) would mix float and histogram samples, a separate `TimeSeries` message MUST be used. A `TimeSeries` carrying exemplars but neither samples nor histograms represents series-level exemplars for the series identified by `labels_refs`. +* At least one element in `samples`, in `histograms`, or in `exemplars` MUST be provided; `metadata` alone does not satisfy this requirement. A `TimeSeries` MUST NOT include both `samples` and `histograms`. For series which (rarely) would mix float and histogram samples, a separate `TimeSeries` message MUST be used. A `TimeSeries` carrying exemplars but neither samples nor histograms represents series-level exemplars for the series identified by `labels_refs`. * MAY contain labels e.g. referencing trace or request ID. If the exemplar references a trace it SHOULD use the `trace_id` label name, as a best practice. * MUST contain a timestamp. While exemplar timestamps are optional in Prometheus/Open Metrics exposition formats, the assumption is that a timestamp is assigned at scrape time in the same way a timestamp is assigned to the scrape sample. Receivers require exemplar timestamps to reliably handle (e.g. deduplicate) incoming exemplars. -* MAY be sent in a `TimeSeries` that carries neither samples nor histograms. In that form the exemplars are associated with the series identified by `labels_refs`, rather than with a particular sample or histogram. +* MAY be sent in a `TimeSeries` that carries neither samples nor histograms. In that form the exemplars are associated with the series identified by `labels_refs`, rather than with a particular sample or histogram. `metadata` on such a `TimeSeries` describes that series and Receivers MAY ignore it. ## Out of Scope From f9578cf8b376bac7fd72c645bd4197cac5e01f56 Mon Sep 17 00:00:00 2001 From: thc1006 <84045975+thc1006@users.noreply.github.com> Date: Fri, 25 Sep 2026 10:23:03 +0000 Subject: [PATCH 5/5] RW 2.0-rc.5: apply the review wording bwplotka's suggestions, applied as given. The at-least-one rule splits into two bullets and makes the metadata point an example rather than a second clause. The exemplar bullet loses the sentence about metadata on a sample-less TimeSeries. The two exemplar comments in the embedded protobuf now say the exemplars may belong to the series or to its samples and histograms. Those two comments are the ones prometheus/prometheus#19530 changes in the source proto, so they move together there, which is what krajorama asked for when this pair was opened. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> --- docs/specs/prw/remote_write_spec_2_0.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/specs/prw/remote_write_spec_2_0.md b/docs/specs/prw/remote_write_spec_2_0.md index 3a651d49b..339b306cf 100644 --- a/docs/specs/prw/remote_write_spec_2_0.md +++ b/docs/specs/prw/remote_write_spec_2_0.md @@ -266,14 +266,14 @@ message TimeSeries { repeated Sample samples = 2; repeated Histogram histograms = 3; - // exemplars represents an optional set of exemplars for this series. + // exemplars represents an optional set of exemplars for this series or series's samples/histograms if present. repeated Exemplar exemplars = 4; // metadata represents the metadata associated with the given series' samples. Metadata metadata = 5; } -// Exemplar contains additional information associated with a series. +// Exemplar contains additional information associated with a series or series's samples/histograms if present. // It is typically used to attach an example trace or request ID associated with // the metric changes. message Exemplar { @@ -363,7 +363,8 @@ For every `TimeSeries` message: -* At least one element in `samples`, in `histograms`, or in `exemplars` MUST be provided; `metadata` alone does not satisfy this requirement. A `TimeSeries` MUST NOT include both `samples` and `histograms`. For series which (rarely) would mix float and histogram samples, a separate `TimeSeries` message MUST be used. A `TimeSeries` carrying exemplars but neither samples nor histograms represents series-level exemplars for the series identified by `labels_refs`. +* At least one element in `samples`, in `histograms`, or in `exemplars` MUST be provided. For example, a `TimeSeries` with only `metadata` and `labels_refs` is not valid. +* A `TimeSeries` MUST NOT include both `samples` and `histograms`. For series which (rarely) would mix float and histogram samples, a separate `TimeSeries` message MUST be used. * MAY contain labels e.g. referencing trace or request ID. If the exemplar references a trace it SHOULD use the `trace_id` label name, as a best practice. * MUST contain a timestamp. While exemplar timestamps are optional in Prometheus/Open Metrics exposition formats, the assumption is that a timestamp is assigned at scrape time in the same way a timestamp is assigned to the scrape sample. Receivers require exemplar timestamps to reliably handle (e.g. deduplicate) incoming exemplars. -* MAY be sent in a `TimeSeries` that carries neither samples nor histograms. In that form the exemplars are associated with the series identified by `labels_refs`, rather than with a particular sample or histogram. `metadata` on such a `TimeSeries` describes that series and Receivers MAY ignore it. +* MAY be sent in a `TimeSeries` that carries neither samples nor histograms. In that form the exemplars are associated with the series identified by `labels_refs`, rather than with a particular sample or histogram. ## Out of Scope