Site icon New Generation Enterprise Linux

Stop the silent death of your logs when journald hits the rate limit

Observability & Logging (Journald, EBPF Tracing)

Stop the silent death of your logs when journald hits the rate limit

Technical Briefing | 10/1/2026

I was once debugging a service that kept crashing under high load, but the logs simply stopped halfway through the event. I spent hours checking OOM killer logs and kernel panic triggers, thinking the system was dying, but it turned out journald had decided my application was too loud. It silently dropped thousands of lines of output because I hit the default rate-limiting threshold. When your observability stack relies on the system journal, the last thing you want is the daemon policing your verbosity.

Why journald decides you are spamming

By default, systemd restricts how many lines a single unit can log per second. If your process enters a tight loop or starts dumping a massive stack trace to stderr, the journal rate limiter kicks in. It doesn’t tell your application to slow down; it just throws the messages into the bit bucket. You get a log entry saying the limit was hit, but unless you are actively tailing that specific service with a high-alert threshold, you will miss the gap entirely.

journalctl --since '1 hour ago' --priority 3 | grep -i 'suppressed'
  • Check /etc/systemd/journald.conf for RateLimitIntervalSec and RateLimitBurst defaults
  • Set these values to 0 to disable the throttle entirely if you need absolute fidelity
  • Move noisy debug output to a file or sidecar instead of relying on the system log

When you need to look deeper with eBPF

If you really need to capture every event without the kernel-level buffer overhead of journald, look at using an eBPF tracepoint on the writev syscalls your process makes. This lets you inspect the data before it ever touches the system logging pipe. It is more work than a config tweak, but it ensures that your observability tooling is seeing the reality of the process, not just what the logging daemon allows through.

If you find logs missing, start by increasing the burst limit rather than disabling it globally. Logging to disk is expensive and journald exists for a reason, but keep in mind that once your I/O wait climbs, you are already fighting a losing battle. Monitor your disk latency alongside these limits so you know if you are being throttled because of policy or because your storage can’t keep up.

Linux Admin Automation  |  © www.ngelinux.com  |  10/1/2026
0 0 votes
Article Rating
Exit mobile version