Skip to content

AMQP reconnections don't clean up sync-out event handlers, causing memory leaks #202

Closed

Description

@JulienZD

Background

I just had a deep dive into some memory issues of our application. It kept growing over time and we couldn't pinpoint what was causing it. Eventually we saw thousands of references of 131 kB to an instance of Mux from amqplib (amounting to ~2.1GB of unnecessary memory). This led led us to feathers-sync, as we use it with the AMQP adapter.

Update 2025-09-25

See my comment below

Issue

The AMQP adapter doesn't properly clean up the registered app.on('sync-out', handlerFn) event handler when a channel is closed. The handlerFn contains a reference to the channel which is never cleaned up, as the event listener stays active even after a connection is closed.

The AMQP adapter uses amqp-connection-manager for automatic reconnections. The setup option of createChannel (related feathers-sync code) is called each time a new connection is made.
feathers-sync sets up an event listener to sync-out to publish the event to the queue within that setup function. However, in the event of a channel closing (due to whatever reason), the event listener is never cleaned up. Upon new events it'll keep executing the now redundant event listener, as the channel can't be published to anymore.

See the below image for an example of the issue. I added some debug logs to illustrate when a channel is reconnected / closed, as well as which events are sent on which pid / channel. For this I also had to make a small change in amqplib to attach a random string to each created channel.

Scenario: I make a change to an item, wait for the event to be sent, and then I restart my AMQP Docker container to trigger a reconnection.

Image

Impact

  • Memory leaks
    • The channel reference (and all other references within it) stay in memory
  • Unnecessary processing, as it'll keep trying to call channel.publish on a closed channel, which will just error

Reproduction

  1. Setup a Docker container with AMQP (or run it locally)
  2. Run a Feathers app with the AMQP adapter for feathers-sync using DEBUG=feathers-sync:amqp for debug logs
  3. Trigger something that causes events to be sent via feathers-sync
  4. Observe how only one "Publish success: |true|" debug is logged
  5. Restart AMQP
  6. Trigger the same thing again
  7. Observe how there is now also a "Publish fail: |Channel closed|" log alongside the "Publish success: |true|" log
  8. Repeat however many times as you want. Each restart of AMQP will add another "Publish fail: |Channel closed|" log

Proposed solution

Unregister the event listener when a channel is closed. From what I can gather, you can listen to close events on the channel itself. The following seems to fix it for me:

-          // Publish the received message to the queue
-          app.on('sync-out', (data) => {
-            try {
-              const publishResponse = channel.publish(
-                key,
-                queue,
-                Buffer.from(data)
-              )
-              debug(`Publish success: |${publishResponse}| APMQ channel`)
-           } catch (error) {
-              debug(`Publish fail: |${error.message}| APMQ channel`)
-           }
-          })

+         function publishToQueue(data) {
+           try {
+              const publishResponse = channel.publish(
+                key,
+                queue,
+                Buffer.from(data)
+              )
+              debug(`Publish success: |${publishResponse}| APMQ channel`)
+            } catch (error) {
+              debug(`Publish fail: |${error.message}| APMQ channel`)
+            }
+          }

+          // Publish the received message to the queue
+          app.on('sync-out', publishToQueue)

+          channel.on('close', () => {
+            debug(Channel closed')
+            app.off('sync-out', publishToQueue)
+          })
+        })

Side note: all the debug logs in the AMQP adapter have a typo: APMQ should be AMQP

Activity

  1. JulienZD commented on Sep 5, 2025

    @JulienZD
    ContributorAuthor

    My suggested fix definitely seems to help. See below for an overview of our server memory over the past month. We deployed the fix on August 29, ever since then the memory has stopped increasing over time. The hard drops before were each time the server was restarted.

    Image

    We found out some more information as to why connections were being recreated often, apparently we occasionally had ~25MB of JSON data in our context.data (hundreds of luxon DateTime objects). This - probably - caused the client from not being able to respond to RabbitMQ heartbeats in time, resulting in the server closing the connection.

  2. changed the title [-]Closed AMQP channels don't clean up existing `sync-out` event handlers[/-] [+]AMQP reconnections don't clean up `sync-out` event handlers, causing memory leaks[/+] on Sep 5, 2025
  3. daffl commented on Sep 5, 2025

    @daffl
    Member

    Thank you for digging into this! That does look like the correct fix. Can you submit it as a pull request? Wondering if this is happening for the other adapters as well.

  4. added a commit that references this issue on Sep 5, 2025
    3d46224
  5. JulienZD commented on Sep 5, 2025

    @JulienZD
    ContributorAuthor

    Here's a PR: #203

    I looked at the others and I don't think so? The sync-out event listeners are setup directly in the returned function from the adapters, so I think they're only registered once. Not 100% sure though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions