Validate Consul lookup-parallelism and tidy the client lifecycle - #956
Open
pjfanning wants to merge 1 commit into
Open
Validate Consul lookup-parallelism and tidy the client lifecycle#956pjfanning wants to merge 1 commit into
pjfanning wants to merge 1 commit into
Conversation
Motivation: `boundedTraverse`, added in apache#906, splits the work by `pekko.discovery.pekko-consul.lookup-parallelism`, and nothing validates that value. `splitAt(n)` with `n <= 0` returns `(empty, remaining)`, so the batch is empty, `Future.traverse` of it completes immediately, and `loop` recurses with an unchanged `remaining`/`acc` pair. Running the merged function verbatim with `lookup-parallelism = 0` and five items spins through 4,861,031 iterations in five seconds without completing - a lookup that never finishes and pins a core, rather than a configuration error. Four smaller problems came in with the same PR. The CA certificate stream in the TLS branch is never closed, which leaks a file descriptor. The TLS branch passes an `SSLContext` but no trust manager, and the Consul builder then falls back to the default JVM trust manager: the built client accepts all 144 CAs in the JDK trust store for chain cleaning instead of only the CA configured via `ca-path`. The `consul-close` shutdown task runs the blocking `consul.destroy()` on `system.dispatcher` even though the same PR introduced `blockingEc` for exactly this kind of work. And the comment above the new `Promise`-based timeout claims it avoids leaking Consul HTTP connections when the timeout fires, which it does not - nothing cancels the requests already in flight. What it does do is cancel the scheduled timeout once the lookup completes. apache#906 also added seven configuration keys without documenting any of them. Modification: Require `lookup-parallelism > 0` in `ConsulSettings`, and guard `boundedTraverse` itself, which now takes the parallelism as a parameter rather than reading the settings directly. Close the CA stream with `Using.resource`, and pass the CA's `X509TrustManager` to the builder alongside the `SSLContext`. Move `destroy()` onto `blockingEc`. Replace the timeout comment with what the code actually does. Build the Consul client lazily behind an `AtomicReference` that is populated only after the client exists, so a configured-but-unused discovery instance creates nothing and shutdown never forces the lazy val - the same shape apache#907 introduced for the AWS clients. Client creation moves behind `private[consul] createConsulClient()` so tests can observe it. Drop the redundant `immutable.Seq(targets: _*)` copy of an already-immutable `Seq`. Document the configuration keys added by apache#906, and note the client lifecycle. Result: A non-positive `lookup-parallelism` fails at startup with a message naming the key, instead of hanging the first lookup. A client configured with `ca-path` trusts only that CA. No file descriptor is leaked, blocking shutdown work stays off the default dispatcher, and the settings are discoverable from the docs. Tests: - sbt "discovery-consul/testOnly *ConsulServiceDiscoveryInternalsSpec" - 9 succeeded, 0 failed (new spec; needs no Consul container) - Same spec with ConsulSettings.scala reverted to main - "should reject a lookup-parallelism of 0" and "should reject a negative lookup-parallelism" both FAILED - Same spec with the withTrustManager call removed - "should trust only the configured CA certificate when ca-path is set" FAILED, 144 accepted issuers instead of 1 - sbt "discovery-consul/mimaReportBinaryIssues" - success - sbt "discovery-consul/scalafmt" "discovery-consul/Test/scalafmt", sbt headerCreateAll - clean - ConsulDiscoverySpec not run - it needs a Consul testcontainer, left to CI References: Refs apache#906
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Follow-up to #906.
1.
lookup-parallelism = 0hangs the lookup forever.boundedTraversesplits bysettings.parallelism, and nothing validates it:splitAt(n)withn <= 0returns(empty, remaining), soFuture.traverse(empty)completes immediately andlooprecurses with identical arguments. Running the merged function verbatim:A user who reads
lookup-parallelism = 0as "unbounded" gets a future that never completes and pins a core, rather than a configuration error.2. The
ca-pathFileInputStreamis never closed (ConsulServiceDiscovery.scala:59) — a file descriptor leak, in the PR whose headline item was fixing a resource leak.3. The TLS branch supplies an
SSLContextbut no trust manager.Consul$Builder.addSslSocketFactoryfalls back toTrustManagerUtils.getDefaultTrustManager()when none is given, so the built client uses the default JVM trust manager for chain cleaning while the handshake uses the custom CA context. I had initially called this latent; it is measurable — the client ends up accepting 144 accepted issuers (the whole JDK trust store) instead of the 1 CA configured viaca-path.4.
consul.destroy()runs onsystem.dispatcher. The PR's own second item moved blocking work offsystem.dispatcherand introducedblockingEc; the shutdown task added by its first item then didn't use it.destroy()shuts down OkHttp's connection pool and executor service, which blocks.5. The timeout comment claims something untrue. Nothing in the new code cancels the in-flight Consul requests when the timeout fires. What the
Promisepattern actually buys is cancelling the scheduled timeout once the lookup completes — a real improvement, described as a different one.6. No tests, and no docs. 133 added lines with zero tests, and seven new configuration keys that never reached
docs/.Modification
ConsulSettingsrequireslookup-parallelism > 0;boundedTraversetakes the parallelism as a parameter and guards it too.Using.resource; the CA'sX509TrustManageris passed to the builder alongside theSSLContext.destroy()moved ontoblockingEc.AtomicReferencepopulated only after it exists, so a configured-but-unused discovery instance creates nothing and shutdown never forces the lazy val — the shape improve discovery-aws-api #907 introduced for the AWS clients. Creation sits behindprivate[consul] createConsulClient()so tests can observe it.immutable.Seq(targets: _*)copy of an already-immutableSeq.Result
A non-positive
lookup-parallelismfails at startup with a message naming the key instead of hanging the first lookup. A client configured withca-pathtrusts only that CA. No file descriptor leak, blocking shutdown work stays off the default dispatcher, and the settings are discoverable from the docs.Tests
New
ConsulServiceDiscoveryInternalsSpec— 9 succeeded, 0 failed. It needs no Consul container (the kiwiproject builder does not ping by default, so the client builds offline), so it runs in ordinary CI alongside the existing container-basedConsulDiscoverySpec.Directional checks:
ConsulSettings.scalareverted tomain— "should reject a lookup-parallelism of 0" and "should reject a negative lookup-parallelism" both FAILED.withTrustManagercall removed — "should trust only the configured CA certificate when ca-path is set" FAILED:had length 144 instead of expected length 1.Also:
sbt "discovery-consul/mimaReportBinaryIssues"— success.sbt "discovery-consul/scalafmt" "discovery-consul/Test/scalafmt"andsbt headerCreateAll— clean.ConsulDiscoverySpecnot run locally — it needs a Consul testcontainer; left to CI.src/test/resources/consul-test-ca.crtis a self-signed certificate generated for this test. Only the certificate is checked in — no private key.References
Refs #906