Repository navigation
AMQP reconnections don't clean up sync-out event handlers, causing memory leaks #202
Description
Background
I just had a deep dive into some memory issues of our application. It kept growing over time and we couldn't pinpoint what was causing it. Eventually we saw thousands of references of 131 kB to an instance of Mux from amqplib (amounting to ~2.1GB of unnecessary memory). This led led us to feathers-sync, as we use it with the AMQP adapter.
Update 2025-09-25
See my comment below
Issue
The AMQP adapter doesn't properly clean up the registered app.on('sync-out', handlerFn) event handler when a channel is closed. The handlerFn contains a reference to the channel which is never cleaned up, as the event listener stays active even after a connection is closed.
The AMQP adapter uses amqp-connection-manager for automatic reconnections. The setup option of createChannel (related feathers-sync code) is called each time a new connection is made.
feathers-sync sets up an event listener to sync-out to publish the event to the queue within that setup function. However, in the event of a channel closing (due to whatever reason), the event listener is never cleaned up. Upon new events it'll keep executing the now redundant event listener, as the channel can't be published to anymore.
See the below image for an example of the issue. I added some debug logs to illustrate when a channel is reconnected / closed, as well as which events are sent on which pid / channel. For this I also had to make a small change in amqplib to attach a random string to each created channel.
Scenario: I make a change to an item, wait for the event to be sent, and then I restart my AMQP Docker container to trigger a reconnection.
Impact
- Memory leaks
- The
channelreference (and all other references within it) stay in memory
- The
- Unnecessary processing, as it'll keep trying to call
channel.publishon a closed channel, which will just error
Reproduction
- Setup a Docker container with AMQP (or run it locally)
- Run a Feathers app with the AMQP adapter for
feathers-syncusingDEBUG=feathers-sync:amqpfor debug logs - Trigger something that causes events to be sent via
feathers-sync - Observe how only one "Publish success: |true|" debug is logged
- Restart AMQP
- Trigger the same thing again
- Observe how there is now also a "Publish fail: |Channel closed|" log alongside the "Publish success: |true|" log
- Repeat however many times as you want. Each restart of AMQP will add another "Publish fail: |Channel closed|" log
Proposed solution
Unregister the event listener when a channel is closed. From what I can gather, you can listen to close events on the channel itself. The following seems to fix it for me:
- // Publish the received message to the queue
- app.on('sync-out', (data) => {
- try {
- const publishResponse = channel.publish(
- key,
- queue,
- Buffer.from(data)
- )
- debug(`Publish success: |${publishResponse}| APMQ channel`)
- } catch (error) {
- debug(`Publish fail: |${error.message}| APMQ channel`)
- }
- })
+ function publishToQueue(data) {
+ try {
+ const publishResponse = channel.publish(
+ key,
+ queue,
+ Buffer.from(data)
+ )
+ debug(`Publish success: |${publishResponse}| APMQ channel`)
+ } catch (error) {
+ debug(`Publish fail: |${error.message}| APMQ channel`)
+ }
+ }
+ // Publish the received message to the queue
+ app.on('sync-out', publishToQueue)
+ channel.on('close', () => {
+ debug(Channel closed')
+ app.off('sync-out', publishToQueue)
+ })
+ })Side note: all the debug logs in the AMQP adapter have a typo: APMQ should be AMQP
Activity
JulienZD commentedon Sep 5, 2025
My suggested fix definitely seems to help. See below for an overview of our server memory over the past month. We deployed the fix on August 29, ever since then the memory has stopped increasing over time. The hard drops before were each time the server was restarted.
We found out some more information as to why connections were being recreated often, apparently we occasionally had ~25MB of JSON data in our
context.data(hundreds ofluxonDateTimeobjects). This - probably - caused the client from not being able to respond to RabbitMQ heartbeats in time, resulting in the server closing the connection.[-]Closed AMQP channels don't clean up existing `sync-out` event handlers[/-][+]AMQP reconnections don't clean up `sync-out` event handlers, causing memory leaks[/+]daffl commentedon Sep 5, 2025
Thank you for digging into this! That does look like the correct fix. Can you submit it as a pull request? Wondering if this is happening for the other adapters as well.
fix(amqp): remove event listener when channel is closed
JulienZD commentedon Sep 5, 2025
Here's a PR: #203
I looked at the others and I don't think so? The
sync-outevent listeners are setup directly in the returned function from the adapters, so I think they're only registered once. Not 100% sure though