My PA Housing pipeline runs itself every month: it pulls data from four public sources, builds a report, and publishes it. One month it quietly stopped working, and the reason taught me more than the build did.
The problem
One of the sources is OpenStreetMap, accessed through the Overpass API. It worked every time I ran it on my own machine. But the pipeline runs on GitHub Actions, and Overpass throttles requests coming from those shared servers. Same code, different computer, different result.
The worst version of this failure isn’t a crash. It’s a pipeline that keeps running with missing data and publishes a report that looks fine but isn’t.
The fix: a fallback chain
Instead of one attempt, the pipeline now tries each option in order and stops at the first that works:
- Fresh fetch: try the live API.
- Stale cache: use the last successful pull.
- Seed data: fall back to a known baseline.
- Hard fail: if all three fail, stop before publishing anything.
Every run logs which tier it used, so I always know whether a report is built on live or cached data.
The takeaway
Automated reporting isn’t done when it works once. It’s done when you’ve decided what should happen every time it doesn’t. The question I now ask about every pipeline: if this source disappears tomorrow, will I find out before my audience does?