Repository navigation
Give each Kibana replica enough heap on the docker environment - #136
Merged
Merged
Conversation
Kibana 9.4.6 runs out of JavaScript heap at --max-old-space-size=768: Mark-Compact 643.7 (794.0) -> 629.2 (794.0) MB FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory V8 kills the process, so the container reports OOMKilled=false and the host still has gigabytes free. `restart: always` brings the replica back, and until it serves again the proxy sends half of the suite's requests to a Kibana that has no session. The browser goes back to the login page and the running spec fails. The 9.4.6 docker leg failed this way in runs 34759281391, 34850881267, 34913897924 and 34938018373, about half the time. The ECK legs run one Kibana and never restart, so they keep the 768 MB heap. The heap goes to 1280 MB. The limits overlay follows it: the container holds the heap plus about 270 MB of node runtime, so 1 GB no longer fits.
This comment was marked as outdated.
This comment was marked as outdated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Kibana 9.4.6 runs out of JavaScript heap at
--max-old-space-size=768:V8 kills the process itself. The container reports
OOMKilled=falseand the host still has 5 GB free, so the restart looks like it has no cause.restart: alwaysbrings the replica back. Until it serves again, the proxy sends half of the suite's requests to a Kibana that has no session, the browser goes back to the login page, and the running spec fails.The 9.4.6 docker leg failed this way in runs 34759281391, 34850881267, 34913897924 and 34938018373 — four of the last eight runs. No other leg failed once.
The ECK legs run one Kibana, which never restarted in 124 samples. They keep the 768 MB heap.
The fix
The heap goes to 1280 MB. The replica died with a 640 MB live set, so 1280 leaves real room above it.
The limits overlay follows: the container holds the heap plus about 270 MB of node runtime, so 1 GB no longer fits. It goes to 1792 MB.
Checks
docker compose configmerges all four overlays: heap 1280 MB,mem_limit1879048192.The runner has 15 GiB and 5.2 GiB free at the worst sample, with a peak Kibana RSS of 1.06 GB. The new ceiling adds at most 512 MB per replica, so about 4.2 GiB stays free.
The overlay is off on CI.
APPLY_RESOURCE_LIMITSisauto, which applies limits only below 12 GB. Under limits on a smaller host the total is 6.5 GB.If the leg fails again, Kibana leaks rather than needs room, and 1280 MB only buys time.