Skip to content

[api][runtime][python] Add minimal Event lineage - #923

Open
rosemarYuan wants to merge 6 commits into
apache:mainfrom
rosemarYuan:feature/eventlineage
Open

[api][runtime][python] Add minimal Event lineage#923
rosemarYuan wants to merge 6 commits into
apache:mainfrom
rosemarYuan:feature/eventlineage

Conversation

@rosemarYuan

@rosemarYuan rosemarYuan commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Linked issue: Closes #841

Related discussions:

Purpose of change

The Event Log currently records individual Events but does not preserve enough information to reconstruct the direct causal path between an Event, the Action it triggered, and the Events emitted by that Action.

This PR implements the minimal Event-lineage phase discussed in #710.

It adds two framework-managed fields to each Event emitted by an Action:

  • upstreamEventId: the ID of the Event consumed by the emitting Action;
  • upstreamActionName: the name of the Action that emitted the current Event.

Together with the existing per-occurrence Event ID, these fields are sufficient to reconstruct an InputEvent-rooted Event-Action-Event causal tree from the available Event Log records.

This PR deliberately does not implement the complete execution-tracing model proposed in #900. Run identity, concrete Action invocation identity, execution hierarchy, lifecycle, retry/replay semantics, ordering, and failure details remain part of that separate design.

A real Runtime execution was verified with the following causal path:

InputEvent(input=1)
  -> action1
  -> MiddleEvent(num=2)
  -> action2
  -> OutputEvent(output=4)

The File Event Log contained three physical Event records:

Event upstreamEventId upstreamActionName
_input_event absent absent
MiddleEvent InputEvent ID action1
_output_event MiddleEvent ID action2

Running tools/reconstruct_trace_tree.py against that Runtime log produced:

Trace Tree 1
  _input_event
    [Action: action1]
      MiddleEvent
        [Action: action2]
          _output_event

The JSON reconstruction contained one root, all three Event nodes, the two expected virtual Action edges, and no warnings.

Tests

Add e2e test to review Trace Tree python/flink_agents/e2e_tests/e2e_tests_integration/python_event_logging_test.py

API

This PR changes the default Python Event ID generation strategy from content-derived IDs to UUIDv4.

This is also a breaking Python API change: Event.id is now immutable. Custom Event.from_event() implementations that use result.id = event.id must instead return result.reconstruct_from(event).

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

@github-actions github-actions Bot added doc-label-missing The Bot applies this label either because none or multiple labels were provided. fixVersion/0.4.0 priority/major Default priority of the PR or issue. doc-included Your PR already contains the necessary documentation updates. and removed doc-label-missing The Bot applies this label either because none or multiple labels were provided. labels Jul 23, 2026
@joeyutong

Copy link
Copy Markdown
Collaborator

One ActionStateStore replay semantic seems worth making explicit before merging. When a completed action result is reused, the runtime returns the stored output Events and rebinds their lineage to the current triggering Event while preserving the output Event IDs. The resulting behavior is:

first execution: E1 -> A -> O1
reused execution: E2 -> A(reused) -> O1

I think this is a reasonable semantic, but it is currently implicit. Could we document it and add a test that asserts O1.id is preserved while its upstreamEventId changes from E1 to E2? This also establishes that the same logical Event ID may appear in multiple Event Log records across reuse, which reconstruction needs to tolerate.

Comment thread tools/reconstruct_trace_tree.py Outdated
Comment thread api/src/main/java/org/apache/flink/agents/api/Event.java Outdated
Comment thread python/flink_agents/api/events/event.py Outdated
Comment thread python/flink_agents/api/events/event.py Outdated
Comment thread docs/content/docs/operations/monitoring.md Outdated
Comment thread tools/reconstruct_trace_tree.py Outdated
Comment thread api/src/main/java/org/apache/flink/agents/api/Event.java
@rosemarYuan
rosemarYuan force-pushed the feature/eventlineage branch 2 times, most recently from 7d122a1 to d8432ea Compare July 28, 2026 04:10
Comment thread python/flink_agents/cli/trace_tree.py
Comment thread python/flink_agents/api/events/event.py
Comment thread python/flink_agents/cli/trace_tree.py Outdated
f"across {len(matching_records)} records.",
)
)
continue

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

continue skips building a node for the conflicting id, and every descendant leaves the rendered trees with it, not just the conflicting Event.

Six records in (a root, K logged twice with different content, then C <- K, D <- C, E <- D):

warnings: [('EVENT_ID_CONFLICT', 'k'), ('MISSING_PARENT', 'c')]
nodes:    ['c', 'd', 'e', 'r']
text:
Trace Tree 1
  _input_event (r)

C gets a MISSING_PARENT, but D and E get nothing, since they link cleanly to nodes that are themselves unreachable. Four Events drop out of the rendered tree. They stay in nodes, but nothing links them to a root, and two warnings name two of them.

The path I can see reaching this: FileEventLogger truncates at write time from event-log.standard.max-string-length and friends, so a reused Event ID logged before and after a threshold change differs in content and reads as a conflict. That is the cross-restart case the new hint block describes.

Could the conflicting id keep its first observation as the node, so the branch survives and the inconsistency still surfaces as a warning?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @joeyutong and @weiqingy for the review. I have pushed the fixes.

  • The Trace Tree reader can now find multi-node cycles. It reports a CYCLE_DETECTED warning and removes only the edge that creates the cycle. Other valid branches are kept.
  • The document now explains that Python Event IDs cannot be changed. The old result.id = event.id code raises a ValidationError. Please use reconstruct_from(event) instead.
  • For EVENT_ID_CONFLICT, the reader now keeps the first record. Later conflicting records are counted, but their lineage links are not used. Child Events are no longer lost.

I also added tests for these cases. Please take another look when convenient. Thanks!

@github-actions github-actions Bot added doc-included Your PR already contains the necessary documentation updates. and removed doc-included Your PR already contains the necessary documentation updates. labels Jul 29, 2026
@rosemarYuan
rosemarYuan force-pushed the feature/eventlineage branch from d8432ea to e58b907 Compare July 30, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-included Your PR already contains the necessary documentation updates. fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add lineage fields to the event log for per-run trace reconstruction

3 participants