Repository navigation
[Bug]: Lack of logrotate configuration causes disk exhaustion and unit crashes for high-traffic ingresses #638
Description
Activity
Can you please check your logrotate configuration? The apt package deployed by the charm deploys a daily rotation from what I see in our staging environment:
root@juju-cef4f1-stg-haproxy-56:~# cat /etc/logrotate.d/haproxy /var/log/haproxy.log { daily rotate 7 missingok notifempty compress delaycompress postrotate [ ! -x /usr/lib/rsyslog/rsyslog-rotate ] || /usr/lib/rsyslog/rsyslog-rotate endscript }If urgent, please reach out on mattermost as we are currently facing issues with the github2jira synchronization.
Thank you for reporting your feedback to us!
The internal ticket has been created: https://warthogs.atlassian.net/browse/ISD-6548.
This message was autogenerated
We are using 50G root disks on our HAProxy units. The logs are uploaded to Loki running in COS, so we have no need for long-term storage on the charm units themselves. We would want a way to override this log-rotate configuration to enforce a max disk usage, instead of a number of days.
e.g. something like
/var/log/haproxy.log { hourly size 50M # would want this to be configurable rotate 10 # would want this to be configurable compress delaycompress }I assume we even do not need the config option, the charm can come with some sensible defaults and the opinionated deployment to use Loki in production
@alexdlukens-canonical wdyt?
Reacted by Alexandre Gomes and Alex LukensAfter internal discussion, we don't strictly need full logrotate configuration exposed through charm options.
In enterprise environments with COS integration, local disk logs only serve as a temporary buffer while being scraped into Loki (e.g., via the OTEL collector). Charms should remain opinionated, so exposing user-facing configuration knobs isn't necessary here.
Instead, the charm could implement a hardcoded, opinionated capping and retention policy. Enforcing a frequent size-based or short-interval rotation (calibrated to safely buffer log volume during peak traffic bursts between scrape intervals) will prevent disk exhaustion while keeping charm management simple.
Reacted by Maksim BeliaevWe discussed the topic internally with @gregory-schiano.
Our approach was to propose a "logrotate-configurator" charm with the corresponding relation and lib for any machine charm to be able to configure their log rotation.
The lib would expose methods for the charm to init a proper logrotate, and the relation would let the configurator override the default settings when necessary.
This way the charm can remain opinionated and still be adapted to specific cases.The alternative of adopting an opinionated capping and retention policy could also be a cheaper approach. We will discuss it.
The main caveat I see is the risk of not fitting all use cases (e.g. not being a good fit for small deployments).@seb4stien, do we expect the charm to be used without COS in an enterprise setting?
If not, local development could use the same 10% of storage. For example, 1 GB would result in 100 MB of data, which is still a lot to store.
I don't know if we have a policy to develop charms assuming they would always be connected to CoS.
To keep it open, we propose to change the default logrotate configuration to the following:/var/log/haproxy.log { daily rotate 7 maxsize XXG missingok notifempty compress delaycompress postrotate [ ! -x /usr/lib/rsyslog/rsyslog-rotate ] || /usr/lib/rsyslog/rsyslog-rotate endscript }Where XXG will be 10% of the disk size at install time, and we will trigger logrotate hourly to rotate if we hit maxsize.
This way we keep daily rotation for low volume sites, and rotation happens automatically for heavy workloads.@alejdg @alexdlukens-canonical What do you think? Could we saturate within 1 hour? It is still non-deterministic
We can go for 1GB maxsize to make it a bit more deterministic.
- linked a pull request that will close this issueAdd size-based HAProxy log rotation with hourly checks #673
on Oct 6, 2026 1GB could be saturated in 1hr on high throughput services. From a check on an archive VM on PS7, we had a spike in excess of 49GB/day. I would be in favor of the 10% disk usage default
Bug Description
We are experiencing disk exhaustion issues on units running the haproxy charm when handling high-traffic ingresses. Because the charm currently does not expose any configuration for logrotate, it falls back to the default log rotation policy, which only rotates logs once a week.
For high-volume environments, this weekly rotation is far too slow, causing the HAProxy logs to fill up the entire disk before the rotation is ever triggered. When the disk is filled, the charm goes into error and stops serving.
We already expanded the disk size but this is not a sustainable solution, we have and application that stops every 2 days. We need the ability to configure log rotation (e.g., size-based rotation, daily/hourly frequency, or capping retention) directly through the charm.
Impact
Medium (functionality degraded, workaround exists)
Impact Rationale
No response
To Reproduce
Environment
channel=2.8/edge
revision=537
running on canonical-k8s
Relevant log output
Additional context
No response