Don’t Trust Cloud-init to Always Run Once (Or Perfectly)

Technical Briefing | 9/24/2026

Cloud-init is often seen as a set-it-and-forget-it thing. You craft some user data, launch a VM, and magically, your hostname is set, your SSH keys are deployed, and maybe a custom script runs. It just works, right? Well, sometimes. But that happy path can lead you right into a ditch when cloud-init *doesn’t* run perfectly, or when you actually *need* it to re-evaluate its state after the first boot. I’ve seen this silently mess up deployments more than once, and it’s a common trap when you move beyond basic provisioning.

The ‘First Boot’ Fairy Tale

By default, cloud-init runs through a series of stages — init, config, final — during what it considers the ‘initial boot’ of an instance. It sets up users, networking, pulls in packages, and executes your `runcmd` scripts. Once it’s all done, it drops a marker, usually in `/var/lib/cloud/instance/boot-finished`, and declares itself `DONE`. Most documentation, and frankly, most of us, treat it as a one-shot deal. Spin up, configure, move on. And for straightforward tasks, it genuinely is. But that simple model glosses over a ton of edge cases, especially the ones that bite you when you’re trying to debug or build more complex, dynamic systems.

When ‘Done’ Isn’t Really Done

Here’s the rub: if cloud-init encounters an error during a stage, it might retry, or it might just skip that particular task and move on, leaving your VM in a weird, partially configured state. Your `cloud-init status` might still say `DONE`, but critically, the thing you *needed* might not have happened. And even worse, if you clone a VM or revert to a snapshot *after* cloud-init has already run, it won’t re-execute its `init` or `config` stages by default. Why? Because it thinks it’s already finished. Any changes in your user data, or any new services you expected to kick off, just won’t happen. This behavior can be a nightmare to debug when you’re tearing your hair out wondering why a change isn’t applying or why a new instance isn’t behaving like the first one. It absolutely bit me when I was trying to debug a complex custom service startup on instances provisioned from a golden image.

sudo cloud-init clean --logs --seed
sudo reboot
sudo cloud-init status --wait

Crafting Idempotent Cloud-init Scripts

To handle re-runs gracefully, and to generally build more resilient cloud-native systems, you absolutely have to write your `cloud-config` with idempotence in mind. Don’t assume a `runcmd` script will only execute once. Always check for existing state: does the file already exist? Is the service already running? Is the package already installed? For simple package installations, `apt-get install` or `dnf install` are usually idempotent, which is great. But for anything involving custom files, service restarts, or complex logic, you need explicit checks within your scripts. For immutable systems, this means thinking less about *installing* and more about *configuring* already present components or injecting data for containers or local services to pick up.

• Include explicit checks in `runcmd` scripts: Use `if [ ! -f /var/log/my_script_marker ]; then …; touch /var/log/my_script_marker; fi` to prevent re-execution.
• Favor declarative sections: Use `users`, `apt`, `write_files`, `network` where possible; they’re generally idempotent by design. `runcmd` is your last resort for logic that can’t be expressed otherwise.
• Monitor `cloud-init status` and logs religiously: Don’t just `ssh` in and hope. Check `journalctl -u cloud-init` and `tail -f /var/log/cloud-init-output.log` if things aren’t working as expected.
• Test re-runs and failures: Create a snapshot after initial boot, then `cloud-init clean –logs –seed`, reboot, and rigorously verify your expected state and error handling.

Understanding cloud-init’s full lifecycle, its state tracking, and crucially, embracing idempotence isn’t just an academic exercise. It’s the difference between predictable, repeatable deployments and a frantic scramble to understand why half your fleet isn’t doing what it’s supposed to after a ‘minor’ config change or a fresh deploy from an old snapshot. Knowing how to force a re-run and, more importantly, how to build your scripts so they don’t break when that happens, will save you a ton of pain when the easy button stops working.