Rollback
A rollback is the act of reverting a software system to a previous known-good state after a deployment causes problems. When a new release introduces bugs, performance issues, or unexpected behavior, rolling back restores the prior version and stops the bleeding while the team investigates and fixes the underlying issue.
Why it matters
No matter how careful the testing, production deployments sometimes go wrong. Systems behave differently under real load. Edge cases emerge that weren't anticipated. Integration points fail in unexpected ways. When this happens, the ability to quickly restore service is critical.
Without rollback capability, teams face an ugly choice: leave users suffering while frantically debugging, or attempt hot fixes under pressure that might make things worse. With rollback capability, the first response to problems is simple: revert to what worked, then investigate calmly.
This safety net also enables faster iteration. Teams comfortable with their rollback process take more risks - shipping smaller changes more frequently because recovery from failure is straightforward. Fear of deployment is often fear of unrecoverable failure.
How rollbacks work
Rollback mechanisms vary by technology and architecture:
Binary/artifact rollback. Redeploy the previous version's compiled code, container image, or packaged application. Requires keeping previous artifacts available.
Blue-green rollback. Switch traffic back to the previous environment. If green (new) fails, route back to blue (old). Quick but requires running parallel environments.
Database rollback. Revert database schema changes and data migrations. Complex and often the hardest part of rollback.
Configuration rollback. Restore previous configuration values if config changes caused problems.
Feature flag disable. If the new code is behind a feature flag, disable the flag rather than redeploying. Fastest option when available.
When to roll back
The rollback decision involves tradeoffs:
Roll back when:
Consider not rolling back when:
The decision often comes down to: how long until fix vs. how painful is current state vs. how risky is rollback?
Rollback challenges
Several factors complicate rollbacks:
Database migrations. If the new version changed the database schema, the old version might not work with the new schema. Backward-compatible migrations help.
Data format changes. If the new version wrote data in a new format, the old version might not read it correctly.
External dependencies. If third-party APIs or services changed, rolling back your code doesn't roll back their changes.
User state. Users might have taken actions only possible in the new version. Rolling back could leave their data in inconsistent states.
Distributed systems. Rolling back one service while others remain updated can create version mismatches.
Making rollbacks reliable
Several practices improve rollback capability:
Keep previous artifacts. Store multiple versions of deployable artifacts. Automated pipelines should retain prior builds.
Backward-compatible migrations. Database changes should work with both old and new code. Deploy schema changes separately from code changes.
Feature flags. Separate deployment from activation. If new code is flagged off, "rollback" is just flipping a switch.
Automated rollback. Connect monitoring to automated rollback triggers. If error rates spike, revert automatically.
Rollback testing. Periodically practice rollbacks. Discover problems before you need rollbacks in emergencies.
Runbooks. Document rollback procedures. Under stress, people forget steps. Written procedures prevent mistakes.
Rollback vs. fix forward
Two philosophies exist for handling deployment problems:
Rollback first: Restore service immediately, investigate later. Prioritizes user experience and reduces pressure.
Fix forward: Identify and deploy a fix rather than reverting. Avoids rollback complexity and keeps momentum.
Neither is universally right. Rollback makes sense for serious, mysterious problems. Fix forward makes sense for obvious, quick fixes. Many teams default to rollback for production incidents, reserving fix forward for minor issues with clear solutions.
Organizational considerations
Rollback capability has organizational implications:
Blameless culture. If rolling back is seen as failure, people hesitate to trigger it. Rolling back should be a normal operational response, not an admission of defeat.
Clear authority. Someone must be empowered to make the rollback decision. Waiting for approval chains extends outages.
Communication protocols. When rollbacks happen, stakeholders need to know. Establish who communicates what to whom.
Post-incident review. After rolling back, conduct retrospectives. What went wrong? How do we prevent recurrence? What can we learn?
Measuring rollback health
Track rollback-related metrics:
Tools like Klero can help reduce the need for rollbacks by ensuring features address real user needs before development. When you build what customers actually want, you're less likely to ship changes that cause unexpected problems.
