Moving to an immutable OS like Fedora CoreOS or a bootc-based image feels like you finally have control. No more configuration drift, no more package updates breaking your production environment at 3 AM. But then you reach for the package manager to add one little utility, and you realize you have stepped into a trap. If you are baking every single tool you need into your base image, you are just building a heavier, harder-to-move monolith that happens to be read-only.

Why you should stop layering everything

Most people assume that because they have a container-based workflow, every binary they need must be part of the OS image. That is a mistake. When you layer packages on top of an immutable base, you increase the surface area for failures and significantly bloat your update payload. If you need a specific tool to debug a performance issue, don’t re-roll your entire base image. Run it as a container with host access instead.

podman run --rm -it --privileged --pid=host --net=host fedora /usr/bin/toolbox

• Keep your base image lean to ensure rapid node provisioning during auto-scaling events
• Use ephemeral containers for ad-hoc debugging to avoid messing with the host OS state
• Separate your operational tooling from your core system dependencies to keep updates predictable

The trap of persistent drift

I have seen teams push custom layer updates to production just to fix a missing grep flag. That defeats the entire purpose of immutable infrastructure. If your base image requires constant modification to be useful, it is not an infrastructure solution, it is just a bloated server you are updating the hard way. Stop fighting the read-only filesystem and start embracing the container runtime that is already running under the hood. If the tool is not essential for the boot process or networking, keep it out of the image.

The next time you feel the urge to add a tool to your bootc build, stop and ask if it actually needs to live on the root partition. If it is only there for when things go south, a containerized toolbox is safer and won’t hold your deployment hostage while you wait for a reboot. Save the image layers for the things that literally cannot run anywhere else.