Rollback From A Beach
I never get why people panic at any point during a deployment. When you’re doing a deployment you’re in only one of two situations: 1) You’ve never deployed the app before 2) You have. If you never deployed the app, there’s no audience to complain if things go wrong, so take your time and figure them out – no need to panic. If you have deployed the app before, you damn sure better have a rollback rip chord in place. Fact is, the vast majority of deployments are done without even the semblance of a rollback plan. The best they have is to restore the database from dump files and build an earlier source control tag. That’s not a plan, that’s inviting disaster.
The stupidest thing I’ve ever heard as it related to rollbacks was from – off all people – a chief architect. The statement was, “We never rollback, only forward”. Said by a military general in the face of overwhelming odds this may still not be prudent, but at least it’s inspiring. In this particular situation, customers were going berserk, data was being lost, and it was during prime business hours. Why in gods name wouldn’t you roll back? Comes to find out, there wasn’t a rollback plan.
I want you to listen to me when I say this: Whether you are deploying the simplest application in the world, or the world’s largest website, you must have a rollback plan. Murphy’s law works overtime during a deployment, and sh*t will happen that you cannot anticipate. Rather than put the business at risk, you need to pull the plug, call off the deployment, and restore the app to a working state. This is a strategic retreat, not an admittance of defeat. There was a flaw in the deployment plan, and you made a wise choice. Now you can figure out what went wrong without customers losing their minds and the database purging terabytes of data. With the ability to rollback from anything, you’ll never panic during a deployment again. When things go screwy, you just hit the big red rollback button and go back to sipping Mai-Tais, thinking of how they’re just handing out chief architect titles nowadays.