RW 2.0-rc.5: allow exemplar-only time series

[thc1006]

This changes the 2.0 specification so that a TimeSeries carrying only exemplars is described rather than forbidden.

Why the current wording does not cover what a sender produces

Prometheus stores exemplars per series, not per sample. populateV2TimeSeries in storage/remote/queue_manager.go writes one output TimeSeries per queue item, and for an exemplar item it appends only an exemplar, having already reset that entry's samples and histograms. So an exemplar can leave the sender in a TimeSeries of its own.

The specification currently says:

At least one element in samples or in histograms MUST be provided. A TimeSeries MUST NOT include both samples and histograms.

That shape therefore has no wording behind it, and a receiver has nothing to rely on when it arrives.

What this changes

Two normative sentences, plus the proto copy this document carries:

• the "at least one element" rule accepts exemplars as well, and says what a TimeSeries carrying only exemplars means;
• the #### Exemplars section says the same thing from the exemplar's side;
• the copied proto comments say the same as the rules above them, so the two do not disagree inside one document.

No field, field number or wire layout changes, and nothing is asked of a sender that was not asked before.

What the first version also had

It added a SHOULD that exemplars travel in the same request as the samples or histograms of their series. That is a rule about how a sender batches rather than a statement about what the wire format allows, and it is the part of this change review is stuck on, so it is out as of 81c879a.

Co-location is still worth writing down somewhere. QueueManager.Append and AppendExemplars both enqueue on the series ref, and shards.enqueue picks the shard with ref % len(queues), so a series' samples and its exemplars go into the same shard queue in order. They are usually in one request and a full batch is what splits them. That belongs wherever the receiver profile question lands, not in this rule.

What this leaves alone

The receiver tests in prometheus/compliance only cover exemplars sent alongside a sample. Covering a TimeSeries that carries only exemplars is a separate change. The canonical proto comments are in prometheus/prometheus#19530, which carries the source version bump to rc.5 as well.

On versioning

Retitled with the version, the way rc.4 was done in #2762, so the title is the record of what went into the release. #3080 still carries the single bump and #3081 is the other change in this one.

Related to prometheus/prometheus#17857 and prometheus/prometheus#16944.

[bwplotka]

This looks what wanted as per https://docs.google.com/document/d/1ZcyKVF87Xf6FhaGkb33LgVLb27FH5NIlRXWEGO2Cc4Q/edit?tab=t.0#bookmark=id.nllt8bbnl0i decision. However we need strict versioning.

I initially planned to pack this and async change under rc.5 for an easy reference -- we don't use git tags so it's a bit challenging to pack two separate changes under same version (:

[thc1006]

Retitled with the version, the way rc.4 was done in #2762, so the title carries the record that git tags would otherwise hold. #3080 keeps the single bump and #3081 is the other change in rc.5, so nothing has to be packed together beyond the two titles saying the same version.

Also added the second half of the decision, which I had left out: exemplars SHOULD be sent in the same request as the samples or histograms of their series, including when they travel in a TimeSeries of their own. Without it the wording read as a licence to detach them freely, which is not what the sender does today.

Out of draft now.

[bwplotka]

I changed base to #3085 so we can squash it all as a rc.5, hope that's ok

[bwplotka]

Ups I think you need to manually rebase, sorry @thc1006 - do you mind doing this?

[krajorama]

This would be updated once the proto is updated then?

[bwplotka]

Let's update it at one go and prep a PR.

[thc1006]

Both done: the copy here in this PR, the original in prometheus/prometheus#19530.

[krajorama]

LGTM, except the question about the copied protobuf

[thc1006]

Rebased onto rw2-spec-rc5, so this is one commit against that branch again. No trouble at all, and moving the base was the right call for keeping rc.5 in one place.

@krajorama good catch on the copied protobuf. The comment you pointed at turned out to be one of three saying the same thing: that one says a TimeSeries specifies samples or histograms, the exemplars field says they belong to the series' samples, and the Exemplar message says it too. All three are in this PR now.

The originals live in prompb/io/prometheus/write/v2/types.proto, so those go in prometheus/prometheus#19530. That one does not depend on this PR: populateV2TimeSeries clears Samples and Histograms and appends only an exemplar when the queue item is a tExemplar, so the comment already disagrees with the sender sitting next to it.

One thing I would rather ask than guess. In prometheus/prometheus#17857 back in January you proposed two halves, "Add SHOULD for per sample, but allow per series by spec". This PR has the second one. The SHOULD I added after your first comment here is about sending exemplars in the same request, not about preferring per sample association. Should the per sample SHOULD go in as well, or does the August decision replace it? Your 26 August comment on that issue reads as moving away from the per sample model for now, which is why I left it out.

Last thing, unrelated to the diff: prometheus/prometheus#18014 has been open since February taking the other route, dropping exemplars that arrive without a matching sample. @AftAb-25 wrote it against the spec as it read then. If this lands, they deserve to hear that the direction moved. The bot has already marked it stale.

[bwplotka]

General thought when I look a the changes: It feels this change is a step back when it comes to compatibility with systems oriented around OpenTelemetry model. As-in, you can use PRW2 to have a reliable transport against those, but it requires being strict on various elements beyond what protocol requires from sender e.g.:

• Avoid unknown metadata.
• Use NHCB instead of classic histograms.

Now with this change it also technically needs:

• Exemplar per sample (or at least within same request).

In the same time this relaxation of rules for exemplars allows (slightly) simpler protocol for existing Prometheus systems, and it's unlikely the exemplar part would change for those given the current APIs.

I wonder how to tackle this redundancy of implementing PRW2 around those intentions to battle ossification/proliferation and being too lean on sending so "I implement PRW2 sending" is not ambiguous.

Perhaps what we need is to introduce some "mode" or consider OTLP export again given advances we have in the storage.

[thc1006]

One piece of data first, since it may change the shape of the question.

Prometheus already sends metadata in a message with no samples. AppendExemplars reads seriesMetadata when it enqueues, and populateV2TimeSeries fills Metadata before it looks at what kind of item it has, so an exemplar-only TimeSeries goes out carrying type, unit and help. That is today's traffic rather than something this change introduces.

The same holds for the OTel side. A receiver reading real Prometheus traffic already had to decide what to do with exemplar-only messages, and the specification told it they were invalid. This does not hand it a new case, it names one it already had.

What the change does add is the SHOULD about the same request, and that one is mine to defend or drop. It is there so a receiver has a bounded window to associate within. If the cost is another implicit requirement on senders, dropping it leaves the MAY describing the wire format and nothing more.

On modes and on OTLP export I do not have a view worth your time. What would help me is knowing whether you want this narrowed to the wire-format statement, held until that larger question settles, or something else.

[bwplotka]

Prometheus already sends metadata in a message with no samples. AppendExemplars reads seriesMetadata when it enqueues, and populateV2TimeSeries fills Metadata before it looks at what kind of item it has, so an exemplar-only TimeSeries goes out carrying type, unit and help. That is today's traffic rather than something this change introduces.

That is true, and it's the tech debt due to abandoning the idea of using prometheus/prometheus#15911 - if we want to bring this back, we need to fix it.
The path using __type__ and __unit__ feature leverages metadata per samples correctly, so we do have path to use it.

I understood it wrongly -- for exemplars sure, but why exemplars even have metadata then? 🙈

[thc1006]

Thanks, that framing helps, and the __type__/__unit__ pointer is one I had not followed.

Correcting something in what I wrote above: I leaned on enableTypeAndUnitLabels as though it described how things run, but it sits behind --enable-feature=type-and-unit-labels, so it is the path you are pointing at rather than the current state. The argument I built on it does not carry the weight I gave it.

What is left is narrower. Exemplars are enqueued separately from their samples whichever way metadata goes, so that part does not resolve with it.

Happy to drop the SHOULD and leave this describing only the wire format, or to park it until the metadata direction settles. Your call.

[thc1006]

I took the same-request SHOULD out in 81c879a. What is left is only that an exemplar-only TimeSeries is valid and that it identifies its series by labels_refs.

@bwplotka that was the same-request half of the third item on your list, and the only one of the three this PR was writing into the spec. Whether a sender keeps a series' exemplars with its samples is about interoperability, not about what the wire format allows, so it belongs with the mode or profile question rather than as a SHOULD in the base spec.

@krajorama flagging this because of prometheus/prometheus#17857 (comment). I read it as asking to soften the per-sample MUST so the mode in that plan would not be a violation, rather than as asking for a new rule about batching. This PR makes the shape valid instead, so there is nothing left to violate. If I have read you wrong, say so and I will put the line back.

Co-location is still worth writing down somewhere. Append and AppendExemplars both enqueue on the series ref, and shards.enqueue picks the shard with ref % len(queues), so a series' samples and its exemplars go into the same shard queue in order. They are usually in one request, and a full batch is what splits them. I can open an issue for that.

Does this settle the protocol format part of your review?

[bwplotka]

Sorry for lag - let's clean this up!

Will you have time to address suggestions?

[thc1006]

All four applied in f9578cf, as written.

The two exemplar comments also live in prompb/io/prometheus/write/v2/types.proto, so prometheus/prometheus#19530 carries the same change now and the two copies match line for line again.

One thing I left alone because it is your wording: the metadata field and the Metadata message say series', while the two exemplar lines now say series's. Happy either way, say the word if you want them matched.

[bwplotka]

LGTM