Skip to content

A cron job's watermark file is not optional

A short note from building cron-for-agents. The first version kept its watermark in memory, a restart erased it, and an agent reprocessed a week of old data. State belongs on disk.

Quick one from building cron-for-agents. The first version kept the watermark, the marker of how far a job had gotten, in the running process's memory. It worked fine for a week, right up until I redeployed and the process restarted.

On the next run, the watermark was gone, so the job did what it was told: process everything since the beginning, because as far as it knew, nothing had ever run. It re-summarised a week of old data and re-sent every report through the delivery channel. Nobody was harmed, but it was exactly the kind of bug that looks like a logic error and is actually a storage decision.

The fix was not clever. A single JSON file, .cfa/state.json, written after every successful run, read on startup. A file that survives a restart is worth more than a smarter dedup algorithm running on state that does not. Anything a scheduled job needs to remember between runs has to live somewhere a restart cannot touch, full stop.