Quick one from building cron-for-agents. The first version kept the watermark, the marker of how far a job had gotten, in the running process's memory. It worked fine for a week, right up until I redeployed and the process restarted.
On the next run, the watermark was gone, so the job did what it was told: process everything since the beginning, because as far as it knew, nothing had ever run. It re-summarised a week of old data and re-sent every report through the delivery channel. Nobody was harmed, but it was exactly the kind of bug that looks like a logic error and is actually a storage decision.
The fix was not clever. A single JSON file, .cfa/state.json, written after every successful run, read on startup. A file that survives a restart is worth more than a smarter dedup algorithm running on state that does not. Anything a scheduled job needs to remember between runs has to live somewhere a restart cannot touch, full stop.