Skip to content

[runtime][python] Release Pemja objects when closing action executor - #944

Open
joeyutong wants to merge 1 commit into
apache:mainfrom
joeyutong:codex/release-pemja-pyobjects
Open

[runtime][python] Release Pemja objects when closing action executor#944
joeyutong wants to merge 1 commit into
apache:mainfrom
joeyutong:codex/release-pemja-pyobjects

Conversation

@joeyutong

Copy link
Copy Markdown
Collaborator

Linked issue: #942

Purpose of change

PythonActionExecutor owns the Pemja PyObject handles for its Python async thread pool and runner context. The existing close path invokes Python-level cleanup but does not call PyObject.close(), which performs Pemja's native decRef. Repeated task restarts can therefore retain old runner contexts and their Flink task graphs through JNI global references.

This change:

  • runs Python-level cleanup before releasing each owned PyObject;
  • releases both handles even if one cleanup operation fails;
  • preserves the first failure and attaches later failures as suppressed exceptions; and
  • clears the fields so repeated close calls are idempotent.

Tests

  • mvn -B --no-transfer-progress -pl runtime -am -Dtest=PythonActionExecutorTest -Dsurefire.failIfNoSpecifiedTests=false test (2 passed)
  • Maven Spotless formatting/checks passed as part of the Java verification
  • Local 20-restart failover A/B: post-Full-GC heap growth changed from 29.2 MiB to 1.7 MiB, and old runner-context/task graphs were no longer retained after termination

API

No public API changes.

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

AI-Contributed/Feature: 0/54
AI-Contributed/UT: 0/149
@joeyutong
joeyutong force-pushed the codex/release-pemja-pyobjects branch from 59a8307 to b761707 Compare July 31, 2026 07:37
@joeyutong
joeyutong marked this pull request as ready for review July 31, 2026 07:48
@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant