My website has a strange feature: it rebuilds itself while I sleep. Every night a scheduled task wakes the machine, reads my AI team's shared memory, regenerates the numbers and the build log you're reading, commits the changes, and lets the deploy pipeline take it live. No human in the loop.
That's the pitch. This week the build pipeline broke in four different ways at once — and the failure taught me more than the feature ever did.
Four Failures in a Single Night
Here's the autopsy, in the order the night served them up.
1. The job never ran
The machine won't start a scheduled task on battery power by default. It was also asleep at 2 AM, which is when I had helpfully scheduled the job. A pipeline that doesn't fire is not a reliable pipeline. It's a wish.
2. The parser shredded the output
Once it did run, the script chopped the AI's output apart line by line. Prose sprayed into files that were supposed to hold clean JSON, and the metrics file came out empty.
3. The validator warned — and shipped anyway
This is the one that still bothers me. My validation step noticed the broken JSON. It logged a warning. Then it cheerfully committed and pushed the corruption to production.
4. An invisible byte broke the build
A byte-order mark my editor had silently added made the build reject files it should have accepted. Nothing visible in the file. Nothing wrong with the content. Just a character you can't see, doing damage you can.
Why Didn't the Live Site Break?
Four bugs in one night, and visitors never saw a broken page. That wasn't luck, and it certainly wasn't the four things above.
It was the deploy platform refusing to promote a build that didn't compile.
Every earlier line of defense failed. The last one held. When people ask why I bother with a build gate that "never catches anything," this is the answer: it caught everything, on the one night everything else was busy failing.
A Check That Warns But Doesn't Stop Is Not Validation
The lesson I keep relearning is about failure #3, not the others. The other three were ordinary bugs. That one was a design error.
My validation was non-terminating. It did the genuinely hard part — detecting the corruption — and then skipped the trivially easy part: refusing to continue. A check that reports a problem and proceeds anyway is not a safety net. It's a smoke detector with the battery pulled out. It gives you the feeling of coverage and none of the coverage.
If a check can't stop the line, it isn't a check. It's a log message wearing a costume.
How I Rebuilt the Pipeline to Fail Loud
The rewrite followed one rule: nothing gets written until everything has been verified.
- Validate first, abort before writing. If the JSON doesn't parse, the run dies having touched no files.
- Join the output before parsing it. The AI's response arrives as many lines; treat it as one document.
- Strip the preamble. Take the content from the first real character of JSON, not from whatever the model said before it.
- Write without a byte-order mark, so the build never rejects a file over an invisible character.
- Gate everything behind a real build that reverts the content and pushes nothing if the site won't compile.
Every one of those is a small change. Together they invert the pipeline's default from ship unless something stops me to stop unless everything passes.
The Happy Path Was Never the Hard Part
The most valuable work in automation isn't making the good case work. It's deciding what happens when the good case lies to you — when the check runs, sees the problem, and tells you it's fine.
Now the pipeline fails loud, fails early, and never ships garbage. Tonight is build #3, and this post is the receipt.
If you're automating anything you can't watch, go find your non-terminating check. I promise you have one.