Deployment rollback: how to roll back a bad release and pick a rollback strategy

Every deployment guide spends its time on the happy path. The part you actually care about at 4pm on a Friday is the other one: the release is live, error rates are climbing, and you need the old version back. How long that takes is a property of how you deploy, and most teams only find out during the incident.

This is the practical version. Five rollback methods with their real recovery times, the runbook to follow when it happens, and an honest list of the things a rollback cannot fix.

What is a deployment rollback?

A deployment rollback is the act of reverting a deployed application to a previous version after the current one turns out to be broken. It is a recovery action, not a release: the goal is to get back to a state you already know works, as fast as possible, without introducing anything new. A rollback is judged on one number, which is how long your users spend on the bad version.

Two terms get mixed up here. A rollback puts the previous version back. A roll forward leaves the bad version live and ships a fix on top of it. Roll forward sounds braver and it is often the wrong call, because writing a fix under pressure is how a small outage becomes a long one. The rule most teams settle on: roll back first, diagnose second.

The five ways to roll back a deployment

These are not equivalent. They differ by an order of magnitude in recovery time and in what they leave behind.

The gap between the first row and the fourth is the whole argument for release directories. Redeploying the previous commit sounds like a rollback, but it runs the same slow pipeline that just failed you: a Composer install, an asset build, a cache rebuild. If your rollback takes twelve minutes, you have twelve minutes of downtime, and you will hesitate before pulling the trigger. Hesitation is the real cost.

How do you roll back a deployment?

The sequence matters more than the tooling. Follow it in this order.

1. Confirm the deploy is the cause. Check the timestamp on the first error against the deploy timestamp. If errors started eleven minutes before the release went out, rolling back will waste the only quiet time you have.
2. Roll back the code first, before you investigate. Put the previous release back and stop the bleeding. The broken release directory is still on disk and the logs are still there, so you lose nothing by reverting first.
3. Deal with the schema separately. If the deploy ran a migration, decide whether the old code can live with the new schema. Usually it can, if the migration was additive. If it was not, this is where a rollback becomes an incident.
4. Restart the background workers. Queue workers hold your application code in memory. A rolled back web tier with workers still running the bad release is a split brain that produces the strangest bug reports you will ever get.
5. Verify from outside. Hit the real URL, not localhost. Check the error rate has dropped rather than assuming it has.
6. Write down what happened while it is fresh. Ten minutes of notes now saves the same argument three weeks later.

Rolling back the code is the easy half

Application code is stateless, so reverting it is a file operation. Everything your release touched that is not code is where rollbacks get hard, and the database leads the list.

If your migration only added things, a rollback is safe. The old code ignores the new column and carries on. If your migration renamed or dropped a column, changed a type, or added a NOT NULL constraint, the old code hits a schema it was never written against and you get a second outage on top of the first. That is the argument for the expand and contract pattern, where every breaking change is split into additive steps across several deploys so that at no point are the old code and the current schema incompatible. We covered the mechanics of that in zero downtime database migrations, including which specific operations are safe on live Postgres and MySQL tables.

The practical rule: never ship a destructive migration in the same release as the feature that stops using the column. Make the code change in one deploy and the cleanup in a later one, once you are confident you will not need to go back.

How atomic releases make rollback instant

The release directory model is what turns a rollback from a deploy into a pointer change. Each deploy is checked out into its own timestamped directory, dependencies and assets are built there, and only when everything succeeds does a symlink flip to make it live. The previous release never moved. It is still sitting on disk, dependencies installed, assets compiled, ready to serve.

Rolling back is then a symlink flip in the other direction plus a graceful reload of PHP FPM, which is measured in seconds. This is the same mechanism that gives you zero downtime deployment in the first place, so you get instant rollback as a side effect of deploying atomically. If you keep five releases, you can go back five deploys without touching Git.

Two details people miss. Shared state has to live outside the release directory, so uploads, the environment file and logs are symlinked in from a shared path rather than copied per release. And your rollback needs to reload the process manager, because PHP FPM with OPcache will happily keep serving the compiled bytecode of files that are no longer the live release.

Blue green deployment rollback strategy

In a blue green setup the rollback is a routing decision. Both environments exist, one is taking traffic, and reverting means pointing the load balancer back at the environment that was live five minutes ago. Recovery is as fast as your DNS TTL or load balancer config reload, and the old environment is still warm, so there is no cold start penalty.

What it costs you is a second full environment, and what it complicates is state. Sessions written on green are not on blue unless your session store is shared, so a rollback logs people out if sessions live on local disk. Move sessions and cache to Redis or the database before you rely on this. The tradeoffs between this and the incremental approach are laid out in blue green vs rolling deployment.

Rolling deployments are different again: because instances update in batches, a rollback is a second rolling pass with the old version, so recovery is gradual rather than instant. You are exposed for the length of that pass, which is the price of not paying for double infrastructure.

Automated rollback: health checks and gates

A rollback that waits for a human to notice is a rollback that takes twenty minutes. The fix is a gate in the deploy itself. After the new release is built but before the symlink flips, run a real health check against it: boot the app, hit an endpoint that touches the database and the cache, and check the response. If it fails, the deploy aborts and nothing ever went live. That is the cheapest rollback there is, because it happens before users see anything.

The second gate runs after the flip. Watch error rate and response time for a few minutes and revert automatically if either crosses a threshold. This is where teams tend to have a blind spot, because application logs only tell you about requests that reached the application. A deploy that breaks Nginx, exhausts PHP FPM's process pool, or lets a certificate expire produces silence in your logs and a broken site for your users. An external check that hits the app from outside every thirty seconds is what catches that class of failure, and it is the signal worth wiring to your rollback trigger.

Keep the automation conservative. An automatic rollback on a genuine error spike is a good trade. An automatic rollback that fires on one slow request will flap, and a flapping pipeline gets switched off.

What a deployment rollback cannot undo

This is the part that turns a rollback into a longer conversation. Reverting the code does not revert the world.

• Destructive migrations. A dropped column is gone. Down migrations restore the structure, not the data. Restoring the data means a backup, and that means real downtime.
• Emails, texts and webhooks already sent. The bad release notified 4,000 customers before you reverted. That is now a support problem, not a deploy problem.
• Payments and third party writes. Anything charged, refunded or pushed to an external API happened. Rolling back your code does not roll back Stripe.
• Jobs already processed. Queued jobs picked up by the bad release ran with the bad logic. Some are idempotent and some silently corrupted rows.
• Bad data written by good code. If the release wrote incorrect values for eight minutes, those rows are still incorrect after the rollback and you need a repair script.
• Cache and session poisoning. A release that cached a malformed view or config keeps serving it after the revert until you flush.

This list is why the order of operations in a risky release matters. Ship the schema change and the code that reads it in separate deploys. Put the genuinely irreversible action, the email blast or the batch charge, behind a feature flag so you can turn it off without deploying anything at all.

How fast should a rollback be?

Under a minute from decision to old version live. That is achievable with a symlink flip or a load balancer switch and it should not require a senior engineer's laptop. If your current answer is longer than five minutes, the fix is usually not a faster pipeline. It is keeping previous releases on disk so a rollback stops being a build.

Measure it deliberately. Roll back a staging deploy on a normal Tuesday and time it. Teams that have never practiced a rollback discover during a real incident that the last good release was pruned, or the command needs a key nobody on call has.

Writing a rollback plan your team can run

A rollback plan does not need to be a document nobody opens. It needs to answer five questions, on one page, where the person on call can find it: who decides to roll back, what the exact command is, what happens to the database, which background processes need restarting, and how you confirm it worked.

Add one line per risky release. If a deploy includes a migration, the plan for that release says whether the migration is reversible and what the fallback is if it is not. Written before the deploy, that takes two minutes. Worked out during an incident, it takes an hour you do not have.

The teams that recover fastest are not the ones with the most sophisticated pipeline. They are the ones for whom rolling back is boring: a known command, a known duration, and no debate about whether they are allowed to run it.