Skip to content

[lake/tiering] Add pendingRecordLag metric for lake tiering#3469

Open
beryllw wants to merge 5 commits into
apache:mainfrom
beryllw:tiering-pendingRecordLag
Open

[lake/tiering] Add pendingRecordLag metric for lake tiering#3469
beryllw wants to merge 5 commits into
apache:mainfrom
beryllw:tiering-pendingRecordLag

Conversation

@beryllw

@beryllw beryllw commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #3464

Brief change log

Tests

API and Format

Documentation

@beryllw beryllw marked this pull request as draft June 11, 2026 00:47
@beryllw beryllw force-pushed the tiering-pendingRecordLag branch from 7579425 to 91dce35 Compare June 30, 2026 09:48
@beryllw beryllw marked this pull request as ready for review June 30, 2026 12:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new backlog-aware lake-tiering metric (pendingRecordsLag) to better represent “how long the oldest committed-but-not-tiered record has been waiting”, avoiding false positives caused by idle periods.

Changes:

  • Introduces pendingRecordsLag metric name and documents it in the metrics guide.
  • Adds estimatedPendingStartTimeMs tracking and new lag calculation helpers in LogTablet.
  • Registers the new gauge in Replica and adds tests covering idle-then-write and caught-up reset scenarios.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
website/docs/maintenance/observability/monitor-metrics.md Documents the new pendingRecordsLag gauge under table_bucket_lakeTiering.
fluss-server/src/test/java/org/apache/fluss/server/replica/ReplicaTest.java Adds a test ensuring pendingRecordsLag starts from 0 after an idle period when new backlog appears.
fluss-server/src/test/java/org/apache/fluss/server/replica/NotifyReplicaLakeTableOffsetTest.java Adds a test ensuring pendingRecordsLag is cleared/reset when tiering catches up.
fluss-server/src/main/java/org/apache/fluss/server/replica/Replica.java Registers the new LAKE_PENDING_RECORDS_LAG gauge and refactors timestampLag gauge to use LogTablet#getTimestampLag.
fluss-server/src/main/java/org/apache/fluss/server/log/LogTablet.java Implements backlog detection + best-effort pending-start estimate and exposes getPendingRecordsLag / getTimestampLag.
fluss-common/src/main/java/org/apache/fluss/metrics/MetricNames.java Adds LAKE_PENDING_RECORDS_LAG metric name constant.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@luoyuxia luoyuxia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@beryllw Thanks for the pr. Left some comments. PTAL

Comment thread fluss-server/src/main/java/org/apache/fluss/server/log/LogTablet.java Outdated
Comment thread fluss-server/src/main/java/org/apache/fluss/server/log/LogTablet.java Outdated
Comment thread fluss-server/src/main/java/org/apache/fluss/server/log/LogTablet.java Outdated
Comment thread fluss-server/src/main/java/org/apache/fluss/server/log/LogTablet.java Outdated
Comment thread fluss-server/src/main/java/org/apache/fluss/server/log/LogTablet.java Outdated
Comment thread website/docs/maintenance/observability/monitor-metrics.md Outdated
@beryllw beryllw requested a review from luoyuxia July 8, 2026 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add pendingRecordLag metric for lake tiering

3 participants