Repository navigation
[release/17.0] Fix trace attribute filter freezing the traces page - #22048
Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
Mitch Denny (mitchdenny) wants to merge 1 commit into
Conversation
### Description **Problem** On the Traces page, filtering by a span attribute can freeze the page once the dashboard holds tens of thousands of spans. For example, filtering by `work_item.queue_name` contains `update`: - the trace list stays empty, - the footer keeps showing the unfiltered trace count, - the dashboard process stays at ~100% CPU until the page is closed. <img width="1893" height="885" alt="image" src="https://github.com/user-attachments/assets/7038545d-3f86-4efa-b610-3f559bec1937" /> **Cause** For each span, the filter checks: "does **this** span have this attribute with a matching value?" The `span_attributes` table has two indexes that SQLite could use for that check: | Index | Sorted by | Good for | |---|---|---| | `owner_key` (and the primary key) | span first, then attribute | "What attributes does this span have?" | | `key_value_owner` | attribute key, then value, then span | "Which spans have this attribute?" | The filter compares the key with `COLLATE NOCASE`. That matches how `key_value_owner` is defined, so SQLite picks it: ``` SEARCH a USING COVERING INDEX ix_telemetry_span_attributes_key_value_owner (attribute_key=?) ``` That index can't jump to a particular span. The span columns come after the value, and the value filter is `LIKE '%…%'`, which can match anywhere. So **for every span**, SQLite reads through **every row with that key**. With 60k spans, that's about 60k × 60k row reads, and the cost grows with the square of the span count. **Fix** Change `a.attribute_key` to `+a.attribute_key` in `BuildSpanFieldPredicate`. A unary `+` doesn't change the value or the result. It only stops SQLite from using that column to choose an index. SQLite then falls back to looking up the span's own attributes by `(trace_id, span_id)`, which is a few rows per span: ``` SEARCH a USING INDEX sqlite_autoindex_telemetry_span_attributes_1 (trace_id=? AND span_id=?) ``` **Results** (same database, same filter): | Spans | Before | After | |---|---|---| | ~60k | didn't finish after several minutes | ~0.45 s (10,120 of 30,360 traces) | | ~200k | — | ~1.7 s (33,333 traces) | With the fix, the filtered list and the correct count appear, and CPU stays near idle. **Other queries checked** I ran `EXPLAIN QUERY PLAN` on the other attribute queries: trace and log text search, span type filters, the log field filter and the GenAI checks. They already look up attributes per span or per log, so they don't have this problem. The filter value dropdown still uses `key_value_owner`, which is correct there: it asks for all values of one key. ### Checklist - Is this feature complete? - [x] Yes. Ready to ship. - Are you including unit tests for the changes and scenario tests if relevant? - [x] No - Did you add public API? - [x] No - Does the change make any security assumptions or guarantees? - [x] No - Does the change require an update in our Aspire docs? - [x] No --------- Co-authored-by: James Newton-King <james@newtonking.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 91f1a1e)
14 tasks
This comment has been minimized.
This comment has been minimized.
Mitch Denny (mitchdenny)
marked this pull request as ready for review
November 10, 2026 05:15
Mitch Denny (mitchdenny)
requested review from
James Newton-King (JamesNK) and
Adam Ratzman (adamint)
as code owners
November 10, 2026 05:15
Member
Author
|
Approved for inclusion in release/17.0. Do not merge yet. |
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-50-pr21910
branch
from
November 10, 2026 08:06
94954e2 to
b062671
Compare
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22048Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22048" |
This comment has been minimized.
This comment has been minimized.
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-50-pr21910
branch
from
November 10, 2026 08:26
b062671 to
94954e2
Compare
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #21910 to
release/17.0(cherry-picked from 91f1a1e with no modifications).Layer 50 of 55 in the release/17.0 backport stack; stacked on #22047.
Backport recommendation: Yes — Trace attribute filter freezes the dashboard (also #21997).
Backport of #21910
Checklist
<remarks />and<code />elements on your triple slash comments?