Stop Your Self-Hosted Runner from Silently Trashing Your Linux Host

Technical Briefing | 9/23/2026

Self-hosted CI/CD runners are great, aren’t they? You get the control, the custom environment, often better performance than managed alternatives. But that control comes with a catch: they’re running on *your* Linux box. And while they’re typically designed to clean up after themselves, that ideal often clashes with the messy reality of build processes. What you end up with, more often than not, is a slowly accumulating pile of garbage: cached dependencies, build artifacts, orphaned processes, and general cruft that quietly eats away at your host’s resources until something breaks, or at least slows down to a crawl.

It Starts with the Disk: Unseen Caches and Artifacts

The most visible culprit is usually disk space. Every build, every test run, it all generates files. Caches are supposed to make things faster, but if they’re not managed, they just grow. And those big build outputs? Sometimes they’re only needed for a short while, but they get dumped into the runner’s workspace and just sit there. You might have `tmpfs` mounts, which are brilliant for ephemeral data, but if they’re not regularly flushed or properly sized, they can just shift the problem. One day you run `df -h` and wonder why your dedicated build server is nearly full when you just pushed a tiny change.

find /var/lib/gitlab-runner/builds /tmp -type f -size +500M -atime +14 -print0 | xargs -0 du -h

More Than Files: Lingering Processes and Cgroup Headaches

But this isn’t just about disk space. I’ve seen this bite us in prod too many times: builds take longer and longer, or fail with weird memory errors, even when disk looks okay. You might be dealing with orphaned processes from poorly terminated jobs, or container remnants that didn’t get properly pruned. These invisible zombies chew up CPU cycles, memory, and even PID allocations. When your runner host silently runs out of available PIDs because a previous build leaked a hundred processes, new jobs just won’t start. It’s a subtle, frustrating failure mode that’s hard to debug without the right tools.

• Runner-specific cleanup jobs or post-build hooks that silently fail.
• Container runtimes (Docker, Podman) not being pruned often enough.
• Build scripts creating large temporary files outside the main workspace or cleaning them poorly.
• Unbounded tmpfs usage in systemd unit files or fstab entries, allowing temporary files to grow indefinitely.

Setting Proactive Guardrails

Reactively hunting down large files with `find` is a good start, but it’s a firefighter’s job. The right approach here is to be proactive. That means aggressive pruning policies for your container runtime (`docker system prune -a –force –volumes` for instance), proper `systemd` unit file settings for your runner service to apply cgroup limits on CPU, memory, and even I/O, and ensuring your `tmpfs` mounts are appropriately sized and monitored. Don’t rely solely on the build scripts; they often assume a pristine environment and aren’t built for robust cleanup.

This isn’t a set-it-and-forget-it kind of problem. A little proactive monitoring and regular hygiene on your self-hosted runners will save you a lot of head-scratching and late-night paging down the road. Keep an eye on your resource usage, and don’t let those unseen piles of junk accumulate.