500,000 Autonomous Decisions Per Day at $10K Per Mistake: How Fabs Run Agentic AI on Streams

The most advanced AI inference systems running in production today aren't at a GenAI startup. They're on semiconductor fab floors. I stumbled onto this realization while building an agentic streaming architecture last year. A former colleague from my semiconductor days watched me sketch out the Kafka-to-agent pipeline and laughed. "We've been running that in production for years. We call it neural APC." Advanced Process Control. Same pattern, same ML frameworks (PyTorch, edge inference, reinforcement learning), same autonomous reasoning loop. And here's why that's exciting: it means there are production-proven patterns we can borrow directly. We don't have to figure this out from scratch. The Pattern Match That Changed How I Think About This Strip away the marketing and an agentic streaming system has four components: A continuous data stream (events arriving in real-time, unbounded) An autonomous reasoning layer (decides what to do without human approval) Action execution (modifies the environment based on decisions) Feedback integration (uses outcomes to improve future decisions) That's it. That's the pattern. Now here's what a semiconductor Advanced Process Control (APC) system looks like: Continuous sensor streams : 500+ parameters per process step, measured every wafer, every 100ms Autonomous reasoning : statistical models + physics models that decide recipe adjustments without operator intervention Action execution : directly modifies equipment settings (gas flow, temperature, power, time) for the next wafer Feedback integration : measures the outcome (film thickness, etch depth, overlay error), feeds it back to update the model for the following run Same. Damn. Pattern. Except the fab version has been in production at scale, processes 50,000 wafers per week, and operates under constraints that make enterprise software look like a homework assignment. Why the Fab Version Is Harder (And Why That's Encouraging) Let me paint the picture of what "agentic streaming" looks like in a semiconductor fab; not to make anyone feel bad, but because when you see what these systems handle, you realize how much headroom our software implementations have to grow. The solutions already exist. Latency budget: 15 milliseconds. Not 15 seconds. Fifteen milliseconds . The time between a wafer finishing one process step and the equipment needing updated parameters for the next wafer. Your Kafka consumer has 200ms to process a message? Luxury. Absolute luxury. Cost of a wrong decision: $10,000+ per wafer. At advanced nodes, a single 300mm wafer carries 50-100 chips worth $100-500 each. If your "agent" makes a bad recipe adjustment and the wafer gets scrapped, that's five figures gone in one action. There's no "oops, let me retry." The silicon is ruined. Zero tolerance for hallucination. In LLM-based agents, a hallucinated tool call might send a weird Slack message or generate a bad report. Annoying. In a fab, a hallucinated recipe adjustment (say, setting plasma power 50% too high) physically destroys the wafer and potentially damages the $5M chamber. The APC system cannot hallucinate. Ever. Volume: 500,000 decisions per day. A single fab makes autonomous, real-time, high-stakes decisions approximately 500,000 times per day (50,000 wafers × ~10 controlled steps per wafer). Your agent handling 1,000 events per day? The fab does that before its first coffee break. And yet. It works. Reliably. For decades. Which means the patterns are proven. They've survived every edge case, every failure mode, every "but what about..." scenario. We get to stand on those shoulders. Three Patterns Worth Borrowing (They Transfer Beautifully) Pattern 1: The Run-to-Run Controller (Not Run-to-Crash) Here's something I wish I'd known earlier: most of us (myself included, embarrassingly) start by building agentic streaming systems that reason from scratch on every event. Agent gets a Kafka message, loads context, reasons, decides, acts. No memory of what happened 10 events ago. No awareness of how its own actions affected the stream. Fab APC uses a fundamentally different pattern called Run-to-Run (R2R) control . The core idea: Maintain a state model of the process (what's the current drift direction? what's the trend over the last N runs?) Each new observation updates the state model incrementally Decisions are made relative to the model , not relative to the raw observation The model includes the effect of your own previous actions That last point is crucial. In a fab, if the R2R controller adjusted the etch time +2 seconds on the last wafer, it accounts for that adjustment when interpreting the next measurement. It knows the difference between "the process drifted" and "my own correction is taking effect." # Run-to-Run controller pattern for agentic streaming class R2RAgentController: """Maintains process state across events. Doesn't reason from scratch every time; updates incrementally like a fab R2R controller.""" def __init__(self): self.state = ProcessState() # Current estimated state self.action_history = deque(maxlen=50) # What I've done recently self.ewma_target = None # Exponentially weighted target def on_event(self, event): # Step 1: Adjust observation for my own previous actions corrected = self.remove_own_effect(event, self.action_history) # Step 2: Update state model incrementally (not from scratch) self.state.update(corrected) # Step 3: Decide based on STATE, not raw event if self.state.drift_exceeds_threshold(): action = self.compute_correction(self.state) self.execute(action) self.action_history.append(action) # else: do nothing. Not every event needs a response. def remove_own_effect(self, observation, history): """Critical: separate process drift from my own corrections. Without this, you get oscillation; the agent fights itself.""" expected_effect = sum(a.predicted_impact for a in history[-5:]) return observation.value - expected_effect Without this pattern, agentic streaming systems oscillate. The agent sees a metric drift up, corrects down, then sees the correction and interprets it as drift in the opposite direction, corrects up, and you get a system bouncing between extremes. This is a well-documented failure mode in control theory; I've seen it firsthand in side projects and community discussions multiple times. Fab engineers have a name for it: controller wind-up . They solved it in the 1990s. Pattern 2: The Dead Zone (When Your Agent Should Do Nothing) One of the most counterintuitive things I learned from studying APC systems: they have a built-in "dead zone"; a range of variation that is intentionally ignored . If the measurement is within ±1σ of target, the controller does nothing. No adjustment. No action. Why? Because not all variation is signal. Some of it is noise. React to noise, and you amplify it. You inject unnecessary variation into a system that was doing just fine. (This is the process control equivalent of "don't touch that, it's working." Which is advice that every on-call engineer wishes they'd followed at 3am at least once.) The first agentic streaming prototype I built on a personal project was hyper-reactive; and I bet yours might be too. Every event triggered reasoning. Every anomaly triggered action. The agent was always doing something . And the result was: Unnecessary LLM calls (cost) Unnecessary actions (risk) Oscillation from over-correction (instability) Alert fatigue downstream (noise) What worked for me: I built a dead zone into the agent's decision layer. A range of "acceptable variation" where the correct decision is explicitly no action . It felt wrong at first; like the system was being lazy. But it's not laziness, it's control theory. The optimal controller often does nothing. # Dead zone implementation: sometimes the best action is inaction class DeadZoneAgent: def __init__(self, target, dead_zone_sigma=1.0): self.target = target self.dead_zone = dead_zone_sigma * self.process_stdev def should_act(self, observation) -> bool: deviation = abs(observation - self.target) if deviation <= self.dead_zone: return False # Within normal variation. Do nothing. return True # Cost savings: in production, 60-70% of events fall within dead zone # That's 60-70% fewer LLM calls, 60-70% fewer actions, 60-70% less risk In my prototype, adding the dead zone reduced agent action frequency by roughly 60-70% while improving outcome quality. The agent had been making things worse by over-reacting to noise. Doing less literally made the system better. Counterintuitive, but once you see it through the SPC lens, it makes total sense. (Published research on APC dead zones in semiconductor contexts reports similar numbers.) Pattern 3: The Fault Detection Gate (Don't Reason About Garbage) This one seems obvious in hindsight but we missed it initially. Before a fab APC controller ever sees data, it passes through an FDC (Fault Detection & Classification) gate. The FDC system asks: "Is this data trustworthy? Did the measurement come from a properly functioning sensor? Is the wafer even in a valid state for this measurement?" If FDC says the data is suspect (sensor malfunction, equipment fault, abnormal process condition) the APC controller never sees it . The data is quarantined. No decision is made. A human is alerted. This is critical because an autonomous system that reasons about corrupt data will make confidently wrong decisions . And in a fab, a confidently wrong recipe adjustment on a fault condition can cascade into scrapping an entire lot of 50 wafers. Here's the question I started asking myself: do I validate input data before my agent reasons about it? Or am I letting the LLM happily consume malformed events, null-filled records, duplicate messages, and stale data; and then make "intelligent" decisions based on garbage? (Spoiler: I was doing the latter. It was humbling.) # FDC-inspired input validation gate class StreamFaultDetector: """Sits BEFORE the agent. Quarantines suspect data. The agent never reasons about garbage; it only sees validated events.""" FAULT_CHECKS = [ lambda e: e.timestamp is not None, # Not null lambda e: e.value != e.previous_value, # Not stuck sensor lambda e: abs(e.value) < 1e6, # Not overflow lambda e: e.source_healthy, # Source system reports OK lambda e: (time.time() - e.timestamp) < 60, # Not stale (>60s old) ] def validate(self, event) -> tuple[bool, str]: for check in self.FAULT_CHECKS: if not check(event): return False, f"Failed: {check.__name__}" return True, "clean" def gate(self, event): valid, reason = self.validate(event) if not valid: self.quarantine(event, reason) # Log, alert, but DO NOT process return None return event # Only clean data reaches the agent What I Took Away From All This The reason fab APC systems work so reliably at such extreme scale isn't because semiconductor engineers are smarter than software engineers. (They'd be the first to tell you that.) It's because their constraints forced good architecture from the beginning. And that's actually liberating for us; because we can adopt the architecture without needing the $10K-per-mistake pain to learn it the hard way. When a wrong decision costs $10,000 and you make 500,000 decisions per day, you have to: Build stateful controllers (reasoning from scratch is too slow and expensive) Build dead zones (over-reaction amplifies variation) Build input validation gates (reasoning about bad data causes cascading failures) Build feedback-aware models (ignoring your own previous actions causes oscillation) Enterprise agentic streaming systems don't have $10K/decision stakes (usually). But they do have: Cost constraints (LLM calls aren't free × 500K events/day = real money) Quality constraints (hallucinated actions erode trust) Latency constraints (customers don't wait 30 seconds) Reliability constraints (the system needs to work at 3am Saturday with no one watching) The constraints map. The solutions transfer. And honestly, once I started seeing my streaming agent through the APC lens, the architecture decisions became clearer; because I wasn't inventing from scratch anymore. I was adapting something that already worked. Where I'd Start (If I Were Doing This Fresh) Add state to your agent. If your streaming agent reasons from scratch on every event, it's a bad architecture. Implement R2R-style incremental state updates. Track your own action history. Separate process drift from correction effects. Build a dead zone. Define "acceptable variation" explicitly. When the stream is within bounds, the correct action is nothing. You'll cut costs 60%+ and improve quality. Gate your inputs. Put a fault detector before your agent, not after. The agent should never see stale, duplicate, null, or malformed data. Quarantine first, reason second. Budget for 500K decisions. If your agent costs $0.01 per decision and you process 500K events/day, that's $5K/day: $1.8M/year. Design for cost at scale, not cost at prototype. The fab mentality: "what does this cost at production volume?" should be your first question, not your last. Measure outcome, not activity. Fabs don't measure "how many recipe adjustments did APC make?" They measure "what's the yield?" If your agent is taking lots of actions but outcomes aren't improving, it's over-acting. Turn up the dead zone. The streaming + agentic pattern is real and it's coming to every enterprise. I genuinely believe the teams that ship it successfully won't be the ones with the fanciest LLM. They'll be the ones who found the right prior art to build on. And modern neural APC (running today in every leading-edge fab on the planet) is the richest source of prior art I've found for this problem. If any of this resonates, I'd love to hear how you're approaching it. These patterns transfer more cleanly than you'd expect, and the teams running them at scale are eager to share what they've learned. The views expressed in this article are my own and do not represent those of my employer. All examples, metrics, and code snippets are illustrative or drawn from personal projects and publicly available research. No proprietary or customer-specific information is disclosed.