Your Supply Chain Is a Distributed System (And It's Failing Like One)

A vaccine is left at 4°C and begins its journey. It passes through three trucks, two warehouses, a regional depot, and finally a fridge at a clinic. At some point it sits above 8°C for about forty minutes: maybe on a loading dock in July, in a doorway while someone signs for the pallet, or overnight because a compressor failed and no one noticed. When it arrives, the label looks fine, the barcode scans, and the system shows it in stock. So a nurse takes it and draws a dose. But the vaccine is already spoiled. It might be hours before anyone notices, if anyone notices at all. You may have seen the statistics. The WHO estimates up to half of the world’s vaccines are wasted each year, with cold chain issues a major cause. Some studies estimate about a quarter of doses are already degraded by arrival, and the industry loses tens of billions of dollars annually because of this. The typical response is to buy better containers, thicker insulation, or a smarter freezer with an app. I understand the impulse, but it misses the real issue. The cold chain usually doesn’t fail because refrigeration is difficult. It fails like distributed systems do. If you’ve ever been responsible for one, that first story probably feels uncomfortably familiar. A supply chain is a distributed system with even less visibility Set the vials aside for a moment and look at the structure. Each part, the plant, depots, trucks, fridges, and wards, acts as an independent node with its own local information. They communicate over an unreliable network with unpredictable delays and no shared clock. No one has the full picture. Updates move slowly from place to place, if they arrive at all, and any node can fail while still reporting everything is fine. This isn’t just a metaphor. That description is the definition of a distributed system, only with different terminology. When you look at it this way, the language we use for software fits the physical world surprisingly well. An empty shelf is like a stale read. A spoiled vaccine that still scans as good is like a node passing its health check while actually corrupted. The confusion between procurement’s records and the ward’s log is like two replicas that don’t match. When the forecasting model gives bad results, it’s because it’s asking a system that can’t provide a reliable answer. None of this is new. We just tend to think about these problems when what’s moving is a JSON payload, not a box of insulin. The three-realities problem is really a consistency issue If you spend a day mapping how a hospital tracks a single item, you’ll see it has three separate identities. Purchasing sees it as a case with a buying unit. The EHR sees it as a clinical item, the way a nurse would request it. Billing sees it as an insurance code. These aren’t just different views of one record. They’re three separate copies, each made for a different audience and each convinced it’s correct. Simply put, your data is spread across systems with no shared ID and nothing to connect them. It’s no surprise they drift apart. Writing it out as code makes the problem obvious. # What three "sources of truth" actually return for one physical object erp.get_stock(purchasing_unit="CASE-24") # -> 3 cases (last synced 09:00) ehr.get_stock(clinical_code="SUTURE-3-0") # -> 0 units (updated 14:32) billing.get_stock(insurance_code="A4550") # -> unknown (no event since restock) # The actual item on the shelf: 0. All three systems are reporting what they believe is true. But the shelf is still empty. You can’t solve this with models alone. If you give the problem to an optimizer or an LLM agent and ask it to move inventory, you haven’t given it a tough challenge. You’ve given it an impossible one, because there’s nothing to reconcile underneath. The system will still give you an answer confidently, because that’s what these tools do. But confidence built on bad data isn’t intelligence. It just means being wrong more quickly. The solution isn’t AI. It’s the basic work that rarely gets attention: give each physical object a single identity, ensure every system refers to it, and run a background process that checks for mismatches and alerts you when they happen. It’s not glamorous, but it’s essential. The inventory system is a cache, and caches can be misleading Here’s how I started to see it differently. Your inventory database isn’t a real-time picture of what’s on the shelf. It’s a cache of what was there the last time someone updated it. Caches can quietly become outdated, and a stale cache can return the same answer as a fresh one. The query looks the same, but reality may have changed without you knowing. Anyone who has worked on distributed systems knows this tradeoff well. You can’t have consistency, availability, and partition tolerance all at once. You have to choose, and live with the consequences. Supply chains make this choice too, often without realizing it, and almost always pick availability. When you ask for stock levels, you always get a number, but no one guarantees it’s correct. So you end up with a system that’s only eventually consistent, even though it looks solid. That gap is where the vaccine failed. Here’s what should concern you. In software, eventual consistency means the copies eventually match if you wait long enough. In the physical world, that doesn’t happen. If a fridge fails and no event is triggered, the system never learns about it. Nothing gets updated, because there’s no message coming. The truth just sits on the shelf, unnoticed, until someone finally checks. You can’t fix what you don’t observe That’s the real underlying problem: observability. No one would run a production service the way we run a cold chain and expect to keep their job. Imagine shipping something with no logs, metrics, traces, or health checks, where the first sign of trouble is a customer showing up hours later to report a problem. In software, that would get you fired. In temperature-sensitive logistics, it’s just another day. The issue shows up only when someone uses the product, long after anyone could have fixed it. That’s why IoT sensors matter more than they seem. A probe that streams temperature data from inside a shipment isn’t just a hardware upgrade. It’s telemetry. It’s like adding monitoring to your code: you turn a black box that only reports at the end into something that provides a live stream you can watch and set alarms for. This is the real reason the cold-chain monitoring market keeps growing. You can call it a logistics trend, but it’s really the physical world catching up to the observability software has had for years. But telemetry alone isn’t enough. If you have readings without thresholds, alerts, or anyone responsible for responding, it’s just an expensive log no one checks. Observability isn’t just about the sensor. It’s about the sensor, the SLO, the alert, and the runbook working together. Measuring temperature was never the hard part. The real challenge is deciding in advance what counts as a problem and who should be notified when it occurs. Approach it like SRE, not procurement If this is truly a distributed system, the right playbook already exists, and it’s not supply chain management. It’s SRE. Most of the same principles apply. Choose a single source of truth and protect it. Give each physical item a single identity, ensure everything references it, and establish a reconciliation process that treats mismatches as incidents rather than data-entry mistakes. Monitor the entire journey, not just the start and end points. A good reading at the plant and another at the ward don’t tell you anything about the forty problematic minutes in between. Tracing matters because the most interesting failures happen between nodes, and this situation is no different. Use event sourcing. Track the journey as an append-only log (shipped, scanned, excursion, quarantined) instead of a single status field that gets overwritten. By replaying the log, you can see exactly why the shelf ended up empty. Write SLOs for the real world. For example, the percentage of doses that always stayed between 2 and 8°C is a valid service-level objective. Treat any breach like an error-budget overrun: trigger an alert to the responsible party, not just a note for the next quarterly report. Design for partition, because partition is the default. The truck drops signal. The clinic goes offline. Buffer locally, sync when it comes back, and never read silence as good news. None of this is revolutionary. It’s the basic, reliable work that keeps software running, but here it’s applied to a system where what’s at stake is someone’s health, not just a website visit. An empty shelf is a failed health check Let’s return to the nurse and the spoiled dose. The forecast wasn’t incorrect, and the database wasn’t lying. It just repeated the last information it received. Nothing failed inside any single system. The failure happened in the gaps: no shared ID, no live telemetry, nothing to reconcile the data, and no check to tell whether “it’s here” also meant “it’s still good.” If you’ve ever shipped a real product, you know this kind of failure instinctively. Everything looks good in testing, but then it falls apart at 2am under real conditions, in the gap between what you planned and what actually happens. An empty shelf is like that 2am failure in the real world, only with higher stakes and worse tools. People always want to ask about the next model: which one, which vendor, which algorithm will finally predict the shortage. That’s the wrong question, and it has been for a while. The real question is older, harder, and completely answerable: does the system’s state match reality, and if it drifts, will anyone notice? We already know how to build systems that can answer yes to that question. We do it every day for software that’s far less important than a vaccine. The cold chain doesn’t need a smarter algorithm. It needs us to recognize it’s always been a distributed system, and to start treating it that way.