The Batch Job That Was Silently Corrupting $10M/Day

Context

Supply-chain applications at a major retailer, where I was lead developer across multiple systems on a team of eight. The ecosystem my team owned had nine applications: eight admin dashboards and one batch job. Each group of logistics admins used their own dashboard to decide which products shipped where across all locations. Those changes flowed into a work-in-progress database, and the batch job validated them and wrote the approved routes to the supply-chain synchronization database — the source of truth downstream systems acted on.

flowchart LR
    D["8× Supply-Chain Logistics<br/>Admin Dashboards"] -->|route changes| WIP[("Supply-Chain<br/>Synchronization<br/>WIP Database")]
    B["Supply-Chain<br/>Validation Batch"] -->|reads & writes errors| WIP
    B -->|writes validated routes| SYNC[("Supply-Chain<br/>Synchronization Database<br/>(source of truth)")]

The Problem

The worst kind of bug: silent. Routes with uncaught validation errors don't fail loudly — they ship. A bad multi-stop route lands as congestion at distribution centers: trucks arriving with loads a facility isn't expecting, people pulled in to untangle it, and delays that cascade through the network. That's how a quiet data problem becomes a $10M+/day problem. During onboarding, this risk was drilled into us — route validation was the guardrail everything else leaned on.

flowchart LR
    T["🚚 Multi-stop truck"] ==>|"validated ✔"| A["📦 Route stop A"]
    A ==>|"unvalidated ⚠"| S["📦 Route stop B"]
    S -.- N["Uncaught validation errors:<br/>$10M/day and congestion, confusion,<br/>and chaos at distribution centers"]
    style N fill:transparent,stroke-dasharray:4 4

How I Found It

Three months in, I was still running my own onboarding curriculum: working through every application my team owned, mapping file structures, tracing what each function did, and studying the synchronization database until I knew the schema cold. Once I understood how the data was supposed to work, I decided the best way to test that understanding was to audit the system end to end, one application at a time.

Eight dashboards in, I got to the batch job. Nested in its service layer was the function that ran route validation — and it only ran against the first stop of each route. Every stop after that shipped unvalidated. Multi-stop routes — precisely the ones with the most opportunities for error — were the ones the guardrail was silently skipping.

The Fix

I walked my manager through the exact lines where the validation was being skipped, wrote the ticket, implemented the fix, and updated the supporting tests — all the same day. The application had been in production for over three years; no error, alert, or incident had ever pointed at it, because the failure mode wasn't an exception — it was absence.

Outcome

Closed a $10M+/day exposure that had lived in production for 3+ years
The find came from discipline, not luck: systematically learning a system deeply enough that its gaps become visible
Also on this team: modernized the frontend from AngularJS to Angular for performance and maintainability