What billions of AI predictions taught Expedia before the age of AI agents

There's an important distinction between AI that just works today, and AI that lasts at scale. Many companies optimize hard for the first one without ever asking whether they're building the second. Velocity without discipline and strategic direction is a liability, not an asset. The hardest part of building AI at scale isn't getting a model to work once. It's building systems that continue to work, scale beyond individual teams and use cases, and improve consistently over time. Today's AI systems do more than just predict and optimize. They converse, reason, and increasingly take action. An autonomous system making decisions on a traveler's behalf creates a very different set of expectations around reliability, governance, and accountability. As AI takes on more of those roles, the principles behind how these systems operate matter more than ever. We have spent years applying AI and machine learning (ML) across the traveler journey — from personalization, ranking, and recommendations, to fraud prevention, customer support, and, more recently, generative and agentic AI experiences. That depth of experience is what led us to develop a set of ML and AI principles to guide how we build, deploy, and evolve AI systems across our company. The goal is simple: Make sure the systems we build create real business value, scale, and operate safely. These principles define how we measure, design, govern, and operate our systems. From principles to practice Publishing principles is the easy part. The harder and more important work is turning them into operating mechanisms: Recommendations, requirements, tooling, and release processes that teams actually use. We have begun using 'Agentic Release' tollgates: A set of recommended and, in some cases, required checks before launching agentic AI features. These tollgates translate principles like clear ownership, risk-based governance, evaluation, safe rollout, and monitoring into concrete expectations for teams. Some of these recommendations and requirements are already being automated and integrated into the software development lifecycle (SDLC). Over time, the goal is for these expectations to become embedded in how we design, evaluate, approve, launch, and monitor AI systems from the start. Outcomes: Measuring what actually matters The first test for any model is whether it improves a business outcome and, ultimately, the traveler experience — not whether it just improves a technical metric. Align models to metrics with business impact: Every ML effort must tie directly to a key business outcome or traveler experience metric. Technical optimizations are useful midpoints, not end goals . Optimize for return on cost : The value a model creates has to justify what it costs to develop, train, and monitor, plus the operational complexity it adds. Favor solutions that deliver lasting impact relative to what they cost to run. Justify complexity against strong baselines: Complexity should be earned, not assumed. Start with a strong baseline: An existing general model, a simple heuristic, an off-the-shelf solution. Reach for specialized models or more complex architectures only when simpler options genuinely can't meet the bar. Require both offline and online evaluation : No model goes to broad deployment on offline validation alone or jumps straight to A/B testing. Every model must perform in both offline and online evaluations. Over time, our offline evaluations should reliably predict what we see online. Design: building systems that scale beyond the teams that build them Getting a model to work is one challenge. Making its value extend beyond a single team or use case is the harder one. Build on shared foundations; specialize only when justified: Favor shared, platform-wide foundations for core capabilities, data representations, and model building blocks. Specialization should build on those foundations, not spin up isolated stacks, so when the foundation improves, the gains flow across the organization. Treat data as a first-class product : A model's quality is bounded by the quality of its data. We need to maintain robust pipelines, clear lineage, reproducibility, and reusable features built with documented ownership, clear schemas, and SLAs that other teams can rely on. Prioritize generality over local optimization : When two approaches perform similarly, favor the one whose learnings, assets, and operating patterns can be reused across teams, brands, and use cases. We should optimize not just for local performance, but for how quickly improvements can diffuse across the company and compound over time. Minimize and sunset manual business rules: Manual rules are sometimes necessary for policy, safety, or compliance, but they should be explicit and reviewed regularly, never silent patches for weak models or a source of permanent maintenance debt. Reproducibility and traceability by default : Training data, features, configurations, evaluation results, deployment versions, and key decisions should all be documented and recoverable. That's what lets you debug a production issue months later and hand off ownership without losing institutional knowledge. Trust: ownership, governance, and operating responsibly at scale The bar for deploying AI isn't just "does it work?" It's "can we stand behind it?" Trust isn't something you add at the end; it's earned over time and maintained across the full lifecycle of every model we ship. Assign clear ownership and accountability: Every model needs defined ownership across its lifecycle — a business owner, a product owner, an AI owner, and an operational owner. These don't need to be four people, but the responsibilities must be explicit. Who's accountable for outcomes? Who responds if the model drifts? Who answers the incident at 2 a.m.? Without this in place, models become orphaned and problems surface with no one to own them. Adhere to standards and governance: AI and ML models must use approved platforms and comply with established company standards, release gates, and governance processes. Operating outside these guardrails requires a clear, defined path to remediation or deprecation, rather than an open-ended exception. Govern proportionally to risk : The level of review, evaluation rigor, and human oversight should scale with a model's impact. A customer-facing model that affects pricing or availability for millions of travelers demands a far higher bar than an internal tool used by a small team. For high-impact, safety-sensitive, or highly autonomous systems, human-in-the-loop checkpoints are built in from the start. Design for fairness, privacy, and transparency : We actively test for unintended bias, have strong data guardrails, and favor explainability when decisions meaningfully affect users. These are incorporated from the start, not added on. Design for safe rollout, rollback, and control : Deployments are progressive, with rollback paths, fallback mechanisms, and circuit breakers ready before launch. The ability to safely undo a deployment matters as much as the ability to ship it. Monitor continuously and adapt: Once live, teams must actively monitor quality, drift, latency, cost, and business performance and retrain or recalibrate when the data shifts. A team should always be able to explain how its model is performing now, not just how it performed when it launched. These principles do more than define how we build. They define what we're willing to ship and how we stand behind it. In a world where AI systems are increasingly consequential and make real decisions for real travelers and partners, these standards matter. Applied consistently, they build responsible AI that lasts. Xavi Amatriain is Chief AI and Data Officer at Expedia Group Xavier will share more details about Expedia's architecture during his session at VB Transform on July 14 at 11:10 am PT. He will discuss: "Expedia's blueprint for building autonomous agents for high-stakes transactional systems." Interested in attending VB Transform 2026? Register here . A select number of complimentary passes are also available to senior technology leaders. Contact us to get yours.