Skip to content

[release/17.0] Fix trace attribute filter freezing the traces page - #22048

Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-49-pr21556from
mitchdenny-backport-17-0-50-pr21910
Closed

Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-49-pr21556from
mitchdenny-backport-17-0-50-pr21910

Conversation

@mitchdenny

Copy link
Copy Markdown
Member

Description

Backport of #21910 to release/17.0 (cherry-picked from 91f1a1e with no modifications).

Layer 50 of 55 in the release/17.0 backport stack; stacked on #22047.

Backport recommendation: Yes — Trace attribute filter freezes the dashboard (also #21997).

Backport of #21910

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

### Description

**Problem**

On the Traces page, filtering by a span attribute can freeze the page
once the dashboard holds tens of thousands of spans. For example,
filtering by `work_item.queue_name` contains `update`:
- the trace list stays empty,
- the footer keeps showing the unfiltered trace count,
- the dashboard process stays at ~100% CPU until the page is closed.
<img width="1893" height="885" alt="image"
src="https://github.com/user-attachments/assets/7038545d-3f86-4efa-b610-3f559bec1937"
/>

**Cause**

For each span, the filter checks: "does **this** span have this
attribute with a matching value?"

The `span_attributes` table has two indexes that SQLite could use for
that check:

| Index | Sorted by | Good for |
|---|---|---|
| `owner_key` (and the primary key) | span first, then attribute | "What
attributes does this span have?" |
| `key_value_owner` | attribute key, then value, then span | "Which
spans have this attribute?" |

The filter compares the key with `COLLATE NOCASE`. That matches how
`key_value_owner` is defined, so SQLite picks it:

```
SEARCH a USING COVERING INDEX ix_telemetry_span_attributes_key_value_owner (attribute_key=?)
```

That index can't jump to a particular span. The span columns come after
the value, and the value filter is `LIKE '%…%'`, which can match
anywhere. So **for every span**, SQLite reads through **every row with
that key**. With 60k spans, that's about 60k × 60k row reads, and the
cost grows with the square of the span count.

**Fix**

Change `a.attribute_key` to `+a.attribute_key` in
`BuildSpanFieldPredicate`.

A unary `+` doesn't change the value or the result. It only stops SQLite
from using that column to choose an index. SQLite then falls back to
looking up the span's own attributes by `(trace_id, span_id)`, which is
a few rows per span:

```
SEARCH a USING INDEX sqlite_autoindex_telemetry_span_attributes_1 (trace_id=? AND span_id=?)
```

**Results** (same database, same filter):

| Spans | Before | After |
|---|---|---|
| ~60k | didn't finish after several minutes | ~0.45 s (10,120 of 30,360
traces) |
| ~200k | — | ~1.7 s (33,333 traces) |

With the fix, the filtered list and the correct count appear, and CPU
stays near idle.

**Other queries checked**

I ran `EXPLAIN QUERY PLAN` on the other attribute queries: trace and log
text search, span type filters, the log field filter and the GenAI
checks. They already look up attributes per span or per log, so they
don't have this problem. The filter value dropdown still uses
`key_value_owner`, which is correct there: it asks for all values of one
key.

### Checklist
- Is this feature complete?
  - [x] Yes. Ready to ship.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] No
- Did you add public API?
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [x] No
- Does the change require an update in our Aspire docs?
  - [x] No

---------

Co-authored-by: James Newton-King <james@newtonking.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 91f1a1e)
@github-actions

This comment has been minimized.

@mitchdenny

Copy link
Copy Markdown
Member Author

Approved for inclusion in release/17.0. Do not merge yet.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22048

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22048"

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants