Your AI Agent Can Run the Test Just Fine. But Can It Answer the Business Question?

Experiments are systematic tests run on a smaller scale to test a hypothesis. However, in a lot of cases, experiments fail because of the design of the system, constraints, or decision environment, even when they are executed correctly. To put it simply, the experiment was technically sound, statistically powered but the experiment could never have answered the question the business was asking, despite putting enormous efforts to run it. Especially as AI is making strides and experimenting is becoming easier by the day, the question is, “Should I run this test?” \n Before running an experiment, I use a simple decision framework to evaluate whether the test will actually produce a meaningful answer. \n The Framework In my experience, this typically happens due to Identifiability, when multiple explanations for the observed experiment outcome are plausible and the experiment cannot distinguish what caused it. Let's talk about what causes Identifiability and how to solve it. The 4 failure modes are | Failure Modes | What does it mean? | |----|----| | Interference | Treatment affects control | | Coupled Metrics | Optimizing one metric shifts others | | Delayed Effects | Impact shows up later | | Selection Effects | Test group ≠ real-world population | \n Let's discuss each in more detail, Interference is when the treatment applied to one group affects outcomes in the control group, breaking causal isolation. Simply put, when treatment affects non-treated users. For example, a team runs a pricing test where a subset of customers are exposed to a lower price or a discounted bundle. But if the screenshots or pictures get circulated externally, it causes behavior in the control group to change as well. So even if the conversion differs, the experiment is no longer measuring isolated response to price. The way to address this is to randomize variables or measure system level effects. So, instead of asking “what is the individual lift”, we ask “what happens to the system if we deploy this”. There will still be cases where the system is highly coupled, this approach or randomization will not be able to isolate individual effects. Experimentation is the wrong choice for this. \ \n Coupled metrics where metrics are not independent; changing one mechanically or behaviorally shifts others. Simply put, where optimizing one shifts another. For example, when we run an experiment to test additional product touch points, intended to increase revenue per user. This might lead to increased revenue per user, but cause an increase in support tickets and negative discussions externally, which aren't metrics in the experiment. This leads to long term erosion of customer trust and less conversions over time despite success of the experiment. The way to address this is to define the decision metric explicitly “we will measure uplift of X, subject to Y not exceeding Z”. This transforms the experiment to an optimization problem. Additionally, understanding model tradeoffs over declaring winners helps to make the dynamic explicit and legible to decision makers. Delayed responses or time lagged effects is where effects appear after the testing window. For example, a team builds an AI driven lead pipeline experiment, which increases conversions in the experiment duration. But the model reduces pipeline diversity, causing weaker performance in future cycles due to over reliance on a certain kind of customers, impacting LTV of customers and causing business to spin up additional acquisition efforts to make up for lost customer base in certain segments. The way to address this would be to align observation windows to risk, higher risk decisions need longer tests to understand success or to pivot. Understanding leading indicators helps understand test windows. \ Selection effects is where experimentation population differs from the future population For example, an experiment run for SMB where a price adjustment or a feature introduction performs well and leadership wants to extrapolate the results to bigger accounts where price sensitivity and buying dynamics vary fundamentally. The way to address this would be to segment explicitly, limit scope of claims, and simulate rollouts - don't simulate by average user, segment by dimensions that drive behavior, change exposure, and affect risk. Closing Experiments are powerful tools, but they are not universally applicable tools. In environments with interference, delayed effects, or binding decision constraints, experimentation can create false confidence rather than clarity. The real question is, “Will this test actually impact our decision?” And not, “Can we test this?” \ \