AWS IoT Core in Production: How We Built OTA Firmware Pipelines for 300+ Physical EV Chargers

Most articles about AWS IoT Core are written by people who started with it from day one. We didn't. We started with a raw MQTT broker, hit its limits across four dimensions simultaneously, and migrated to IoT Core mid-production. That migration — and everything we learned building OTA infrastructure for 300+ deployed EV chargers afterward — is what this article covers. This is not a tutorial. It is a production architecture guide written by someone who owned this system end-to-end. Two Things You Need to Know Before We Start If you come from a DevOps or cloud background without IoT experience, two terms need a quick definition before this article makes full sense. AWS IoT Core is Amazon's managed cloud service for connecting, managing, and communicating with physical devices at scale. Think of it as the cloud backbone for your device fleet — it handles device registration, MQTT messaging, device state persistence, job scheduling, security policy enforcement, and event routing. It replaces the need to build your own device management layer. OTA (Over-The-Air) updates are firmware updates delivered wirelessly to physical devices — in our case, EV charging hardware deployed across multiple US states. Instead of a field technician visiting each charger to update its software manually, OTA pipelines push firmware updates remotely, automatically, and safely while the devices are in operation. How they work together: AWS IoT Core provides the infrastructure that makes OTA possible at scale — device connectivity, job delivery, state tracking, and status reporting. The OTA pipeline is built on top of IoT Core's primitives. You cannot safely operate OTA for hundreds of physical devices without a managed backbone underneath it. \ 1. Why Updating Physical EV Hardware Is a Different Problem Software services fail gracefully. A crashed container restarts in seconds. A bad deployment rolls back with a single command. The blast radius is contained to compute resources you control entirely. Physical hardware doesn't work this way. A firmware update pushed to an EV charger mid-session — while a vehicle is actively drawing power — can interrupt the charging session, strand the driver, and require a field technician to physically visit the device to recover it. At $150–300 per field visit, across hundreds of geographically distributed chargers, this is not a theoretical concern. It is a real operational and financial risk. The constraints that make EV OTA uniquely challenging: Active session protection: A charger with an active charging session must never receive a firmware update. The update must be held until the session ends naturally. Multi-component firmware: A single charger runs multiple software components — main controller, payment processing module, connectivity stack. These have hard version interdependencies. Updating them in the wrong order produces a charger that powers vehicles but cannot process payment, or vice versa. Hardware generation heterogeneity: A fleet of 300+ chargers deployed over multiple years contains multiple hardware generations, each with different firmware stacks and different valid update sequences. Geographic distribution: Devices are deployed across multiple states with varying connectivity reliability. Your OTA system must handle intermittent connectivity gracefully — resuming updates after reconnection without corrupting device state. These constraints ruled out generic software deployment patterns. They required a purpose-built system. \ 2. Why We Migrated From a Raw MQTT Broker to AWS IoT Core Our initial architecture used a self-managed MQTT broker. This decision made sense at the beginning — MQTT is the right protocol for IoT device communication, and a self-managed broker offered flexibility. We hit its limits across four dimensions as the fleet scaled: Scaling: A self-managed broker requires you to own capacity planning, connection limits, and horizontal scaling. As the charger fleet grew past 100 devices and toward 300+, managing broker capacity became a dedicated operational concern that distracted from building product. Device management features: MQTT is a messaging protocol. It handles publish/subscribe. It does not natively provide device registry, device shadow state persistence, job scheduling and tracking, or fleet-wide targeting. Building these capabilities on top of a raw broker meant building and maintaining our own device management layer — a significant engineering investment with no product differentiation value. Security: Certificate management for IoT devices at scale requires infrastructure for certificate issuance, rotation, revocation, and policy enforcement. A self-managed broker provides no native support for this. We were managing certificates manually, which was not sustainable at fleet scale. Operational overhead: Running, monitoring, patching, and scaling a self-managed broker is undifferentiated heavy lifting. Every hour spent on broker operations was an hour not spent on OTA pipeline reliability, observability, or device security. AWS IoT Core addressed all four simultaneously. The migration cost us time upfront. It paid back that investment quickly once the new architecture was running. \ 3. System Architecture Key IoT Core concepts used in this architecture: In AWS IoT Core, each physical device is registered as a Thing — a logical representation of the device in the cloud. Things can be organized into Thing Groups for batch operations: targeting a firmware update to a Thing Group updates all devices in that group simultaneously, without having to address each device individually. A Device Shadow is a persistent JSON document in IoT Core that stores the last known state of a device — even when the device is offline. If a charger loses network connectivity mid-update, its pending update state is preserved in the shadow and automatically resumes when connectivity returns. This makes Device Shadows the right mechanism for tracking firmware versions and update status across a fleet with intermittent connectivity. The full production architecture: ┌─────────────────────────────────────────────────────────────────┐ │ BACKEND SYSTEMS │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────────────────────┐ │ │ │ Jenkins │───▶│ S3 │───▶│ Presigned URL │ │ │ │ Pipeline │ │ Artifact │ │ Generator │ │ │ └──────────┘ │ Store │ └────────────┬─────────────┘ │ │ └──────────┘ │ │ │ PRE-CHECK (runs before job creation) │ │ ┌──────────────────────────────────────────┐ │ │ │ │ Charger Status API + Metabase (on EKS) │ │ │ │ │ - Port 1 / Port 2 session status │ │ │ │ │ - Deployed component versions │◀┘ │ │ │ - Last heartbeat timestamp │ │ │ └──────────────────┬───────────────────────┘ │ │ No active │ session → proceed │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ AWS IoT Core │ │ │ │ │ │ │ │ ┌─────────────┐ ┌──────────────┐ ┌────────────────┐ │ │ │ │ │Device Shadow│ │ IoT Jobs │ │ IoT Rules │ │ │ │ │ │ (per-device│ │ Engine │ │ Engine │ │ │ │ │ │ state) │ │ │ │ │ │ │ │ │ └─────────────┘ └──────────────┘ └────────────────┘ │ │ │ │ │ │ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ │ │ Device Registry — Thing Groups │ │ │ │ │ │ (grouped by hardware generation / region) │ │ │ │ │ └──────────────────────────────────────────────────┘ │ │ │ └──────────────────────────────────────────────────────────┘ │ │ │ MQTT (TLS) + Job Delivery │ └──────────────────────────────┼──────────────────────────────────┘ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ FIELD DEVICES │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Charger 1 │ │ Charger 2 │ │ Charger N │ (300+) │ │ │ Port1 Port2 │ │ Port1 Port2 │ │ Port1 Port2 │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ └─────────────────────────────────────────────────────────────────┘ Important: The Charger Status check is a pre-step that runs before an IoT Job is created — it is not in the communication path between IoT Core and the devices. If the check finds an active session, no job is created. Only when the check passes does the backend proceed to create and deliver an IoT Job. MQTT over TLS: All device communication uses MQTT with TLS mutual authentication using X.509 certificates. Each device has a unique certificate. IoT policies restrict each device to its own MQTT topics only. \ 4. The OTA Pipeline: From Commit to Charger Why IoT Jobs instead of plain MQTT messages? A common question for engineers new to IoT Core: why not just publish the firmware URL to the device's MQTT topic directly? The answer is delivery guarantees and operational control. A plain MQTT message is fire-and-forget — if the device is offline, the message is lost. IoT Jobs provide guaranteed delivery (the job is delivered when the device reconnects), per-device status tracking (you know exactly which devices succeeded, failed, or are in progress), timeout handling, and the ability to trigger rollback jobs automatically on failure. For a fleet of 300+ physical devices where a failed update requires a field technician, these guarantees are not optional. The complete pipeline flow: Developer commits firmware build │ ▼ Jenkins pipeline triggers │ ▼ Firmware artifact built and versioned (naming: {component}-{version}-{hw-gen}.bin) │ ▼ Artifact uploaded to S3 (versioned bucket with lifecycle policies) │ ▼ Charger Status API queried via Metabase (pre-check: session status + deployed versions) │ ┌────┴────┐ │ │ Active No active session session │ │ ▼ ▼ Hold Create IoT Job update with Job Document until session ends │ ▼ IoT Core delivers Job Document to device via MQTT (guaranteed delivery — queued if device offline) │ ▼ Device downloads firmware from presigned S3 URL │ ▼ Device applies update, validates, reports status via MQTT │ ▼ Job completion tracked in IoT Jobs + CloudWatch metrics The IoT Job Document: When an IoT Job is created, the backend provides a Job Document — a JSON payload that IoT Core delivers to the target device via MQTT when the job is assigned. This is not an AWS standard format; it is a custom JSON structure your backend defines and your device firmware is written to parse and execute. The device receives this document, reads the firmware URL, downloads the binary from S3, validates the checksum, and applies the update. { "operation": "firmware-update", "component": "main-controller", "version": "2.4.1", "hardware_generation": "gen3", "files": [ { "filename": "main-controller-2.4.1-gen3.bin", "url": "${aws:iot:s3-presigned-url:https://s3.amazonaws.com/firmware-bucket/main-controller-2.4.1-gen3.bin}", "checksum": { "algorithm": "SHA256", "value": "a3f5c8d2e1b4..." }, "filesize": 2457600 } ], "update_sequence": 1, "total_components": 3, "pre_conditions": { "min_battery_voltage": 48.0, "require_idle_port": true, "min_connectivity_signal": -80 }, "rollback_version": "2.3.8", "timeout_seconds": 300 } Key fields explained: update_sequence and total_components — the device uses these to understand its position in a multi-component update chain. A device will not apply component 2 until component 1 reports success to IoT Core. pre_conditions — device-side validation before applying the update. The device checks its own state against these conditions before downloading. This is a second layer of protection independent of the backend Metabase check — the device validates for itself that conditions are safe before acting. ${aws:iot:s3-presigned-url:...} — This is an IoT Core-specific substitution syntax. When IoT Core delivers the job document to the device, it replaces this placeholder with a time-limited presigned S3 URL at delivery time — not at job creation time. Why presigned instead of a public S3 URL? Security: only the device that receives the job can download the firmware binary, and only within the validity window. If a job is queued for hours waiting for a session to end, the URL is still fresh when delivered because substitution happens at delivery, not at creation. rollback_version — if the device fails post-update health validation, it uses this version string to revert to the previous known-good firmware. The device handles rollback autonomously — it does not wait for a backend signal. A separate monitoring job detects the rollback and flags the device for investigation. \ 5. Session-Aware Deployment: Using Metabase for Real-Time Charger State This is the component that most articles about IoT OTA skip entirely — and the one that matters most for physical hardware safety. The problem: An EV charger has two physical ports. Port 1 may have an active charging session. Port 2 may be idle. The device as a whole is "in use" even if one port is available. A firmware update cannot begin while any port has an active session — the update process requires a controlled restart of software components that would interrupt power delivery. Our implementation: Before creating any IoT Job for a device, we query our internal Charger Status API. This API reads from Metabase — our open-source analytics platform deployed on EKS — which maintains real-time data for every charger in the fleet. The data Metabase provides per charger: Session status for Port 1 (active/idle/faulted) Session status for Port 2 (active/idle/faulted) Currently deployed component versions (main-controller, payment-module, connectivity-stack) Last heartbeat timestamp The pre-deployment check logic: # Pre-deployment check — runs before IoT Job creation def can_deploy_to_charger(charger_id: str, component: str, target_version: str) -> bool: status = charger_api.get_status(charger_id) # Block if either port has active session if status.port1.session_active or status.port2.session_active: logger.info(f"Charger {charger_id} has active session. Holding deployment.") return False # Skip if already on target version current_version = status.deployed_versions.get(component) if current_version == target_version: logger.info(f"Charger {charger_id} already on {target_version}. Skipping.") return False return True Production deployments to chargers are scheduled during low-traffic windows — typically between midnight and 4am — when active session probability is lowest. The Metabase check runs immediately before job creation to catch any sessions that started after the window opened. This approach — combining a scheduled low-traffic window with a real-time status check immediately before deployment — provides two independent layers of session protection without requiring a complex event-driven scheduler. \ 6. Multi-Component Firmware Ordering: The Problem That Breaks Chargers A physical EV charger is not a single application. It runs multiple software components with hard interdependencies: ┌─────────────────────────────────────────┐ │ EV Charger Software Stack │ │ │ │ ┌─────────────────────────────────┐ │ │ │ Connectivity Stack │ │ │ │ (MQTT client, TLS, networking) │ │ │ └────────────────┬────────────────┘ │ │ │ depends on │ │ ┌────────────────▼────────────────┐ │ │ │ Main Controller │ │ │ │ (power management, session │ │ │ │ control, hardware interface) │ │ │ └────────────────┬────────────────┘ │ │ │ depends on │ │ ┌────────────────▼────────────────┐ │ │ │ Payment Module │ │ │ │ (transaction processing, │ │ │ │ card reader interface) │ │ │ └─────────────────────────────────┘ │ └─────────────────────────────────────────┘ What happens when you update in the wrong order: Update the payment module before the main controller is on a compatible version → payment module initializes but cannot communicate with the controller API it expects → charger powers vehicles but cannot process payment → driver session cannot close cleanly. Update the connectivity stack while a component update is in progress → device loses MQTT connection mid-update → partial firmware state → device requires manual recovery. Our solution: dependency graph per hardware generation Each hardware generation has a defined update sequence stored as configuration: { "hardware_generation": "gen3", "update_sequences": { "full_update": [ { "step": 1, "component": "main-controller", "must_complete_before": ["payment-module", "connectivity-stack"] }, { "step": 2, "component": "payment-module", "depends_on": ["main-controller"], "must_complete_before": ["connectivity-stack"] }, { "step": 3, "component": "connectivity-stack", "depends_on": ["main-controller", "payment-module"] } ] } } The IoT Job document's update_sequence field references this configuration. The device firmware validates its position in the sequence before applying any component update. Out-of-sequence updates are rejected at the device level — not just at the backend level. This dual enforcement — backend respects the sequence when creating jobs, device rejects out-of-sequence updates — provides defense in depth against sequencing errors. \ 7. Observability for a Physical Device Fleet Monitoring 300+ physical devices requires different instrumentation than monitoring software services. Devices go offline. Connectivity is intermittent. A device that stops reporting is not necessarily healthy — it may be offline, or it may be stuck in a failed update state. Firmware version distribution dashboard: The most operationally important metric during a rollout is fleet-wide version distribution — what percentage of devices are on the target version, what percentage are on the previous version, and what percentage are in an unknown or error state. We track this via IoT Core device shadows aggregated into CloudWatch custom metrics, visualized in Grafana: Firmware Version Distribution — Main Controller ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ v2.4.1 (target) ████████████████░░░ 82% (246 devices) v2.3.8 (prev) ████░░░░░░░░░░░░░░░ 16% (48 devices) unknown/error █░░░░░░░░░░░░░░░░░░ 2% (6 devices) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Rollout started: 4h ago | Target completion: ~2h remaining Per-device update tracking: Every job execution event is logged to CloudWatch Logs via IoT Rules Engine. This provides a per-device audit trail: when the job was received, when download started, download duration, when update was applied, success or failure, and failure reason if applicable. Automated rollback triggers: Post-update, devices publish a health report to their MQTT topic within 5 minutes of applying firmware. If health reports show error rates above threshold across a batch, CloudWatch alarms trigger an automated rollback job targeting the affected device group. Field technician integration: Devices that fail to report success within the job timeout window generate alerts that create tickets in our operations system with device ID, last known location, error logs, and recommended recovery steps. This eliminates manual monitoring of the IoT Jobs console. \ 8. Security: IoT Core Certificates and Policies Certificate-based authentication: Every device in our fleet has a unique X.509 certificate issued at provisioning time. Certificates are registered in the IoT Core certificate registry. A device that connects without a valid registered certificate is rejected at the TLS handshake level — it never reaches MQTT. IoT Policies with least privilege: Each device certificate is attached to an IoT policy that restricts the device to its own MQTT topics: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iot:Connect", "Resource": "arn:aws:iot:us-east-1:123456789012:client/${iot:ClientId}" }, { "Effect": "Allow", "Action": "iot:Publish", "Resource": [ "arn:aws:iot:us-east-1:123456789012:topic/chargers/${iot:ClientId}/status", "arn:aws:iot:us-east-1:123456789012:topic/chargers/${iot:ClientId}/session", "arn:aws:iot:us-east-1:123456789012:topic/chargers/${iot:ClientId}/health" ] }, { "Effect": "Allow", "Action": "iot:Subscribe", "Resource": [ "arn:aws:iot:us-east-1:123456789012:topicfilter/$aws/things/${iot:ClientId}/jobs/*", "arn:aws:iot:us-east-1:123456789012:topicfilter/$aws/things/${iot:ClientId}/shadow/*" ] } ] } The ${iot:ClientId} substitution variable is the critical security mechanism. It ensures that at policy evaluation time, the device can only publish and subscribe to topics containing its own client ID. A compromised device cannot read another device's shadow or receive jobs intended for a different device. The certificate rotation gap: We have not implemented automated certificate rotation. This is a known gap. Certificates were provisioned at device manufacturing time with multi-year validity periods. The correct approach — which remains in our backlog — is AWS IoT Core fleet provisioning with certificate rotation using the RegisterThing API. This allows devices to request new certificates programmatically using a provisioning claim certificate, which is then invalidated after first use. For any team implementing this from scratch: build certificate rotation into the initial provisioning flow. Retrofitting it to a deployed fleet of 300+ physical devices requires coordinated firmware updates just to enable the rotation mechanism itself. \ 9. What We Would Do Differently Start with IoT Core, not a raw broker. The MQTT broker migration cost us time and introduced operational risk during the cutover. IoT Core is the right foundation for a managed device fleet from day one. Implement certificate rotation at provisioning time. Retrofitting rotation to a deployed fleet requires coordinated firmware updates just to enable the mechanism. Building it in from the start adds two days of engineering and saves months of operational pain later. Build the dependency graph configuration before the first multi-component update. Component ordering errors produce chargers in inconsistent states that require manual recovery. The dependency graph should be defined, reviewed, and tested before any multi-component update reaches production. Design observability for the fleet before the first rollout. Firmware version distribution tracking, per-device audit logging, and rollback triggers should be in place before the first production OTA job runs. Discovering observability gaps during a live rollout to hundreds of physical devices is the wrong time. Use Metabase — or equivalent — for fleet state visibility from day one. Having a queryable, real-time view of every device's session status and deployed versions is not optional for safe OTA operations. Without it, session-aware deployment requires custom state management that duplicates what a good analytics layer already provides. \ Summary Building production OTA infrastructure for physical EV hardware on AWS IoT Core requires solving problems that don't exist in software deployment: Pre-deployment session checks integrated with real-time device state data Multi-component firmware ordering with hardware-generation-specific dependency graphs Fleet observability designed for intermittent connectivity and physical device failure modes Security that prevents cross-device attacks at the MQTT policy level The architecture described here supports 300+ deployed EV chargers across multiple US states with zero reported charging session interruptions caused by OTA operations. AWS IoT Core — specifically device shadows, IoT Jobs with presigned URL support, thing groups, and the IoT Rules Engine — provides the right primitives for this problem. The engineering effort is in the layers built on top: session awareness, component ordering, observability, and security policy design. \