Stop your ephemeral CI/CD runner from disappearing with the evidence

Technical Briefing | 10/27/2026

You’ve seen it: a CI/CD job fails, the pipeline lights up red, and you scramble to check the logs. And then it hits you – the logs are just… not enough. Some cryptic error, a timeout, a missing file, and the runner itself? Already recycled, vanished into the ether. There’s no machine to SSH into, no live environment to poke around. You’re left staring at a vague error message, forced to blindly add `echo` statements or just hit “retry” hoping for a different outcome. This particular flavor of pain has bitten me in prod too many times when a subtle race condition or an unexpected environmental variable decides to rear its ugly head.

Why your debug session keeps going poof

Modern CI/CD environments thrive on ephemerality. Spinning up a fresh, clean VM or container for each job is great for consistency and security. But it’s also a double-edged sword when things go sideways. The moment that job finishes, successfully or not, the whole environment is often nuked from orbit. Cloud provider VMs terminate, Docker containers stop and are removed. That quick, ‘let me just `ls -la /tmp`’ thought you have? Gone. It’s an intentional design choice, and generally a good one, but it means our old-school debugging instincts of ‘log in and look around’ just don’t apply anymore. And often, simply adding more `set -x` to your bash scripts isn’t going to cut it when the issue is an external dependency or an unexpected filesystem state.

Catching the culprit before it evaporates: the `trap` solution

The key here is to capture diagnostic information *before* the environment gets destroyed. The shell’s `trap` command, specifically `trap “commands” EXIT`, is your secret weapon. This isn’t just for signal handling; an `EXIT` trap executes its commands right before the shell exits, regardless of success or failure. This gives you a golden window to dump system state. Think about what you usually check manually: running processes, disk usage, mounted filesystems, network interfaces, environment variables, exit codes of previous commands, contents of temporary directories. You can package all that into a script that runs automatically on job completion.

#!/bin/bash
set -eo pipefail # Exit on error, pipefail


# Define a temporary log file
DEBUG_LOG_FILE="/tmp/debug_output_$$.log"


# Function to gather debug information
gather_debug_info() {
echo "--- Debug Information ---" >> "$DEBUG_LOG_FILE"
echo "Job failed with exit code: $?" >> "$DEBUG_LOG_FILE"
echo "Timestamp: $(date)" >> "$DEBUG_LOG_FILE"
echo "--- Last 20 commands ---" >> "$DEBUG_LOG_FILE"
history 20 >> "$DEBUG_LOG_FILE" 2>/dev/null || echo "History not available." >> "$DEBUG_LOG_FILE"
echo "--- Running processes (ps aux) ---" >> "$DEBUG_LOG_FILE"
ps aux >> "$DEBUG_LOG_FILE"
echo "--- Disk usage (df -h) ---" >> "$DEBUG_LOG_FILE"
df -h >> "$DEBUG_LOG_FILE"
echo "--- Current directory (pwd) ---" >> "$DEBUG_LOG_FILE"
pwd >> "$DEBUG_LOG_FILE"
echo "--- Files in current directory (ls -la) ---" >> "$DEBUG_LOG_FILE"
ls -la >> "$DEBUG_LOG_FILE"
echo "--- Network interfaces (ip a) ---" >> "$DEBUG_LOG_FILE"
ip a >> "$DEBUG_LOG_FILE"
echo "--- Environment variables (env) ---" >> "$DEBUG_LOG_FILE"
env >> "$DEBUG_LOG_FILE"
echo "--- End Debug Information ---" >> "$DEBUG_LOG_FILE"


# Print the collected info to stderr so it shows up in CI logs
cat "$DEBUG_LOG_FILE" >&2
}


# Set the trap to call our function on EXIT
trap gather_debug_info EXIT


# --- Your actual CI/CD job commands would go here ---
echo "Starting CI/CD job..."
sleep 2
# Example of a command that might fail
# cp /no/such/file /tmp/output.txt
echo "Job finished."


# Optional: You might want to unset the trap if successful runs shouldn't dump full diagnostics
# trap - EXIT

Now, a word of caution: don’t just dump `env` or other sensitive commands blindly if your runner handles secrets directly. Filter that output or pass specific, scrubbed variables. And be mindful of the performance overhead; running a bunch of diagnostic commands on every single job exit, successful or not, can add noticeable latency if your CI loop is tight. You might want to guard the `trap` with a condition, perhaps an environment variable like `CI_DEBUG=true`, so you only enable it when you’re actively troubleshooting. But when you’re staring at an inexplicable failure, that slight overhead is a small price to pay for sanity.

• **Artifact all the things:** If a specific directory (like `/tmp/artifacts` or `/var/log`) might contain important files, ensure your CI platform’s artifact collection is configured to scoop it up on every run, especially failed ones.
• **Interactive debug modes:** Some CI platforms (like GitLab with ‘Debug jobs with a new job in the merge request pipeline’) or container tools allow you to temporarily keep the runner alive or start a shell in the context of a failed job. Get familiar with these — they’re life savers.
• **Pre-flight checks:** Run explicit checks at the start of your job for common environmental issues. Does `kubectl` work? Is `docker` logged in? Is `terraform` installed and the right version? Fail early with a clear message if prerequisites aren’t met.
• **Staged rollbacks:** For complex deployments, break your CI job into smaller, atomic stages. If one fails, you’ve narrowed the scope. This doesn’t help with the ephemeral runner, but it makes the next re-run more focused.

This whole approach is about being proactive. You build your safety net *before* you need it. A properly configured `trap` and sensible artifact collection won’t solve every mystery, but they will drastically reduce the amount of time you spend guessing when a flaky job blows up. It means the next time your CI/CD runner tries to perform its magic trick of vanishing with all the clues, you’ll have already extracted the rabbit from the hat. This won’t eliminate all debugging, but it’ll give you actual data to work with instead of just error codes and vague log lines.