You think your incremental backup job is catching everything because you are looking at mtimes. I have seen this blow up in production more times than I care to admit. If a process rewrites a file in place without changing the timestamp, or if you are dealing with filesystems that suffer from clock skew across nodes, those changes are effectively invisible to standard incremental tools. You end up with a backup that says it finished successfully, but your data is stale.

Why mtime is a liar

The kernel update for your data isn’t always reflected in the inode metadata immediately, and some applications manually preserve the mtime after a write to avoid breaking cache invalidation logic elsewhere. When your backup script looks at the filesystem, it sees the old timestamp and skips the file. This bit me in prod when a key-value store database file was modified but left with an old mtime. We restored the backup only to find a hole where the last four hours of writes should have been.

find /data/db -type f -not -newermt '2023-10-27 12:00:00' -exec rsync -av --checksum {} /backup/dir \;

• Force a checksum pass at least weekly to catch bit rot or metadata anomalies
• Use ctime if you need to detect permission or ownership changes that mtime ignores
• Avoid relying on directory-level metadata as it is often updated inconsistently by different filesystems
• Test your recovery process by actually mounting the backup volume and verifying hashes against the source

The right approach is to integrate block-level snapshots before you even start the file-level copy. If you have an LVM or ZFS snapshot, you get a point-in-time view that prevents mid-stream corruption. Run your integrity checks against the frozen snapshot, not the live mount point. It adds a few extra lines to your script, but you will stop waking up at 3 AM because your restoration failed to find the truth.