Skip to content

Cluster fixes - #789

Open
jdmarshall wants to merge 6 commits into
prometheus:mainfrom
jdmarshall:clusterFixes
Open

Cluster fixes#789
jdmarshall wants to merge 6 commits into
prometheus:mainfrom
jdmarshall:clusterFixes

Conversation

@jdmarshall

@jdmarshall jdmarshall commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Rework of worker and cluster lifecycle management. Also allows the primary cluster thread to report stats, adds debug information to common possible failure paths.

Fixes #155, #181, #183, #280, #501, #563, #788

@jdmarshall jdmarshall modified the milestones: v1, v0.16 Jul 28, 2026
@jdmarshall

jdmarshall commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

This changes unreleased API modifications for AggregatorRegistry, so this will need to be part of v0.16.

Also fixes a memory leak in trunk for broadcastchannels, and improves lifecycle management for dead worker threads (both kinds)

Comment thread lib/cluster.js Outdated
Comment thread lib/registry.js Outdated
Comment thread test/workerTest.js
return { threadId, channel };
});

await delay(5); // Let announcements arrive

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just speaking from past experience, this seems like it could become a flaky test in the future (on slow CI machines).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll check if there's callback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test already includes delay(), btw.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No callback. Might be stuck with this one.

Comment thread lib/cluster.js Outdated
.then(metrics => Registry.aggregate(metrics.flat()).metrics())
.then(result => done(undefined, result), done);
const myMetrics = Promise.all(
registries.map(async r => r.getMetricsAsJSON()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you do something similar in a couple other places in this PR, and those places don't add the extra async to the map callback.

In particular, cluster now uses a similar announcement system to
filter workers.

This fixes prometheus#181.
Signed-off-by: Jason Marshall <jdmarshall@users.noreply.github.com>
Signed-off-by: alencristen <299997878+alencristen@users.noreply.github.com>
99.9% of the time process.send() is going to work. We don't need to
guard it when it's already inside of a try block. Just guard the
retry send.

Also reduces the amount of excessive mocking going on in the tests
by using jest more instead of creating our own mocks.

Signed-off-by: Jason Marshall <jdmarshall@users.noreply.github.com>

@cjihrig cjihrig left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but there are a couple linter issues in lib/worker.js.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cluster event listeners are attached even if not using clusters

3 participants