That Time a Rolling Deployment Almost Corrupted Our Data

During what should have been a routine deployment, our pipeline briefly ran two copies of itself at the same time. Nothing failed. No alerts fired. For that window, though, the system was producing output that could not be trusted. Some…