Skip to content

fix(logs): keep compacted child span trees shaped as trees - #8403

Merged
waleedlatif1 merged 7 commits into
stagingfrom
fix/trace-span-tree-compaction
Sep 29, 2026
Merged

waleedlatif1 merged 7 commits into
stagingfrom
fix/trace-span-tree-compaction

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Block output compaction ran the generic payload compactor over childTraceSpans. Compaction works bottom-up, so when a nested span's children list grew past the threshold it became a large-array manifest or large-value reference instead of an array. Trace span building later spread that value (flattenWorkflowChildren) and threw TypeError: directChildren is not iterable. Span building ran outside the finalization try, so the pause was never persisted, the run was never finalized or billed, and the log stayed running until it was swept. Any later resume was then refused. Other readers that walk raw span trees (children.map, for…of) have the same exposure.

  • Serializer: a child span tree is kept whole or not at all. compactBlockOutput splits childTraceSpans off the block output:
    • The output compacts for execution state exactly as before, so an oversized output still spills whole.
    • The spans compact structurally: children and nested output.childTraceSpans stay arrays, and each span's payload fields spill individually, with a cycle guard.
    • A tree still over the threshold as a whole keeps only its skeleton: shape, names, timing, status and cost. This uses summarizeTraceSpansWithoutIo, the same stripping the execution log applies to oversized traces, now shared from trace-spans/summarize.ts. A tree whose skeleton is still over the threshold is dropped with a warning; when large values are rejected, an oversized tree is rejected instead. This bounds the tree as generic compaction did, including in pause snapshots.
    • The executor puts the spans only on the block log; they were always stripped from state. compactBlockLogs applies the same rule to log.childTraceSpans.
  • Span factory: a child span list that is not an array (for example one persisted by an earlier build) is dropped with a warning instead of throwing.
  • Execution core: trace spans are built inside a guarded helper. If building fails, the run is finalized with no spans and the error is logged, so logs, pauses, billing and run counts still settle.

Testing

  • Serializer:
    • a tree with oversized payloads stays whole with each payload spilled;
    • a tree too large as a whole keeps its skeleton, and is dropped only when the skeleton is still too large, or rejected when large values are rejected;
    • an oversized state output still spills whole;
    • block logs follow the same rule;
    • nested child workflow trees keep their shape;
    • a cyclic tree terminates.
  • Trace spans: reference-shaped children and output.childTraceSpans no longer throw.
  • Execution core: pause and failed-run finalization when span building throws.
  • Each guard was reverted individually and its test went red.
  • bun run type-check, bun run lint, bun run check:audits, and the full apps/sim suite pass.

@vercel

vercel Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Sep 29, 2026 5:49am UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 7 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/execution/payloads/serializer.ts">

<violation number="1" location="apps/sim/lib/execution/payloads/serializer.ts:309">
P1: When an output carries `childTraceSpans`, this call can spill the whole `rest` object; the following spread then treats the reference as output and loses its ordinary fields. Compact `rest` with `preserveRoot: true` so only individual payload fields spill.</violation>
</file>

Comment thread apps/sim/lib/execution/payloads/serializer.ts
@greptile-apps

greptile-apps Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Refactors trace span compaction and child span handling.

The PR appears safe to merge; no outstanding blocking finding was established.

Summary

The PR keeps compacted child span trees traversable by separating them from execution-state output, compacting span payloads individually, and retaining a bounded skeleton when necessary. It also makes span building failure non-fatal to run finalization.

  • Block logs retain child-span structure without placing it in resumable block state.
  • Trace construction tolerates older reference-shaped child lists.
  • Pause and failed-run finalization continue when trace construction throws.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Block output] --> B[Separate child spans]
  B --> C[Compact execution-state output]
  B --> D[Compact span payloads]
  D --> E{Tree within limit?}
  E -->|Yes| F[Keep tree in block log]
  E -->|No| G{Skeleton within limit?}
  G -->|Yes| H[Keep skeleton in block log]
  G -->|No| I[Omit child tree]
  F --> J[Build run trace]
  H --> J
  I --> J
  J -->|Build fails| K[Finalize without spans]
  J -->|Build succeeds| L[Finalize with spans]
Loading

Reviews (7) · Last reviewed commit: "fix(logs): skip span fields not in their..."

Comment thread apps/sim/lib/execution/payloads/serializer.ts Outdated
Comment thread apps/sim/lib/workflows/executor/execution-core.test.ts
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

Comment thread apps/sim/lib/execution/payloads/serializer.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread apps/sim/lib/execution/payloads/serializer.ts Outdated
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 7 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@waleedlatif1
waleedlatif1 force-pushed the fix/trace-span-tree-compaction branch from 4ffec75 to cc3ee9c Compare September 29, 2026 04:59
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 7 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

Comment thread apps/sim/lib/execution/payloads/serializer.ts Outdated
Block output compaction spilled oversized child span lists to large-value
references, so a span's children could stop being an array and trace span
building threw while finalizing the run. That left pauses unpersisted and
runs unfinalized.

- Compact child span trees structurally, spilling only each span's payload fields
- Drop non-list child spans with a warning when building trace spans
- Finalize without spans if building them fails, so logs, pauses, and billing settle
A structurally compacted tree had no whole-tree bound, so a large one stayed
inline in block logs and pause snapshots. A tree still over the threshold
after its payloads spill is now dropped (or rejected), as generic compaction
bounded it. Block log outputs compact generically again; new logs never carry
child spans there.
…hole

A tree over the threshold as a whole now keeps its shape, names, timing,
status, and cost instead of disappearing, using the same content stripping
the execution log applies to oversized traces (moved to a shared module). Only
a tree whose skeleton is still over the threshold is dropped.
@waleedlatif1
waleedlatif1 force-pushed the fix/trace-span-tree-compaction branch from cc3ee9c to 5ce7f18 Compare September 29, 2026 05:16
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread apps/sim/lib/execution/payloads/serializer.ts
Comment thread apps/sim/lib/logs/execution/trace-spans/summarize.ts Outdated
…s in the skeleton

The structural walk now keeps only span objects and drops a child list that
is not an array, so building a skeleton can never throw on a malformed entry.
The skeleton keeps a nested child workflow's output.childTraceSpans.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread apps/sim/lib/execution/payloads/serializer.ts
Per-field compaction can spill an oversized modelToolCalls, toolCalls, or
providerTiming to a large-value reference; the skeleton now drops such a field
instead of reading it.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 9 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit c47ed6c into staging Sep 29, 2026
32 checks passed

This branch was previously deployed

1 inactive deployment
Preview — c9212926 Deployed Sep 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant