Good Software Deployment Practices, Part IV: “Deploy The Same Bits”

This is part of a series of posts on how to make software deployments boring, because Boring is good!

In earlier posts I’ve said that everything—everything—getting deployed should come from the output directory of the build.  No exceptions; include even environment-specific configuration information.

Doing so gives us another way to reduce risk.  We do this by deploying the same bits in every environment.  Allow me to explain.

 

The Bad?

I once worked at a company where standard practice was to recompile the source code before deploying into each environment.  I was always surprised at this and didn’t understand it…at the time I didn’t challenge the practice because I was still new to the industry at that point.  But I still don’t understand it.  Maybe someday I’ll get an explanation. 

The Worse

I’ve since worked, quite often, in organizations where recompiling was not a formal practice but was often done by default.  (These organizations didn’t have a controlled build environment.)  I think it’s a weak practice, at least for the kind of software and operating environments we tend to have now.

So here’s what to do instead: 

The Good

Every time the software is built, its output should be uniquely identified.  If you have something like Microsoft Team Foundation Server or CruiseControl, this identifier is generated for you.  (If not, you’ll have to do it for yourself.)  Then, keep track of which specific build has been deployed in the test environments.  At my current workplace we have a number of test environments, and we know exactly which build numbers have been deployed in each environment.  This means we know exactly what is being tested…

…which means that if we deploy the same build number into production, then we know that those exact bits have been tested.  We don’t risk introducing errors by recompiling or reassembling the deployment artifacts.

Having confidence that we’ve deployed exactly what we’ve tested requires having confidence in these things:

  • Every artifact is in the build output directory (also frequently called the drop directory).
  • The drop directory is controlled—once the artifacts of a build are put in the drop directory, very few people should be able to alter the directory’s contents.
  • We are properly tracking build numbers so we know that the build number we’re currently deploying into production is the one we deployed into our test environments earlier.

This is one more place in which to reduce risk, as we travel on our journey to make deployments boring.

Good Software Deployment Practices, Part III: More on “The Build”

This is part of an ongoing series of posts dealing with the practices that will make software deployments boring.  Because boring is good!

So far I’ve been saying some pretty standard stuff but this time I want to state a practice that is less-often done well, at least in my opinion.

 

What to do with configuration files?

I’ve noticed that many people generate configuration information during the deploy.  For example, the build output might include a development or master copy of a web.config file, and during deployment the environment-specific stuff gets replaced with the correct information.

I think there’s a better way…

…and that better way is to include the config files for ALL environments in the build.  Either keep them all in source code, or go through the “replace elements in the master with the correct values” process, once for each environment.  I would tend to go for the latter, since most config files have a mix of environment-specific and non-environment-specific information, and we don’t want to have to maintain multiple copies of the non-environment-specific information.  But either way, the build output should contain all the configs.

Why?

Earlier I said that every deployment artifact—every artifact—must be generated or assembled during the build and be placed in the output directory of the build.  Handling config files in the manner I’ve described is just an example of that.  And it comes with real benefits:

  • Risk.  Selecting the right config file for an environment and putting it in place is in my opinion less risky than walking through a config file making changes to it.  So we can do the harder work that may be more error prone during the build, and do the less-risky portion during the deploy.  We push risk into an earlier step than deployment.
  • Testing/inspection.  This is related to managing risk.  We should test both the build process and the deployment process, but if we generate config information during the deploy, then the first time we can verify the generation of correct production values is after production deployment!  If we generate the configuration information during the build, we can inspect the result at our leisure, as often as we want, with any process we want, before the actual deployment takes place.

 

In other words, handle configuration files, and any other environment-specific items, just as you would anything else.

…which means we follow the practice of assembling ALL deployment artifacts during the build.  Everything that will get deployed should be in the build’s output directory.  The fewer exceptions we make to that rule the more we make our deployments boring.  And that’s good!

Good Software Deployment Practices, Part II: “The Build”

I’ve worked in plenty of environments in which code is compiled on a team member’s machine, then pushed to production.  This raises some important questions:

  • What source code was present when the compile happened?  Was it the latest version?  Did it include something that the developer hadn’t yet checked in to the source control system?
  • What is installed on the developer’s machine?  Are all relevant service packs at the same level as production?  Are production servers missing any required frameworks or APIs that exist on the developer’s machine?
  • Does the code used during the compile match the code that has been tested before going to production?

There are other issues, but hopefully you get the idea.  The wrong answer to any of these questions can lead to production problems.  And in this kind of environment I’ve found that no one actually knows the answer to the questions…

…which brings us to the practice that will resolve these issues:  a formal, controlled build process.  Now, by formal I don’t mean loaded with paperwork; rather, I mean that certain rules are established, communicated, and enforced.

The rules (there may be more for your environment; these are a good start):

  • The “build” (compile and assemble the deployment artifacts—executables, web pages, etc.) occurs on a dedicated machine:  the “build server”.  This is not a developer’s box.  It’s locked down, with only some team members having access to it.  Those team members must know exactly what has been installed at any given time and how the configuration compares to all test and production environments.
  • The “build” is automated to the extent that it is likely to be a very consistent process; i.e., not dependent on who or what triggers it, not subject to human errors, etc.
  • All artifacts being assembled for deployment come from source code, either directly (such as a text file) or indirectly (such as source code, which then gets compiled during the build).  No exceptions.
  • The deployment artifacts generated/assembled during the build are placed in a secure location, probably a network share with limited access.
  • Artifacts deployed to any test or production environment come only from the secure location updated by the build process.
  • The development team does not have the network, database, or other permissions necessary to deploy to production, nor to any test environments other than perhaps a low-level shared development environment.  Thus, it’s not possible to deploy anything even by accident.

These rules can be scaled up or down to fit the size and budget of most development teams.  In a complex environment (such as government contracting) there will probably be much more formality included.  In many businesses in the United States, government regulations dictate a clean separation between those who write the code and those who can deploy it.  On the other hand, in a very small team the dedicated build server may be a virtual machine administered by the team.  In this case the above rules still apply but enforcement may depend on team discipline rather than permissions.

Developer tools can help with some of this.  If you have the budget for it, Microsoft Team Foundation Server (TFS) is an excellent system that integrates source control, the build process, and much more.  You can define any number of build servers and have the TFS server(s) communicate with them.  If this is out of your budget, CruiseControl can accomplish much of the same thing.  It doesn’t offer source control or other features, but it integrates well with other source control system and does a good job managing builds—and it’s free.  There are plenty of other tools out there, including those for non-Microsoft environments; these are just the ones I’m familiar with.

Follow these practices and you’ll reduce the number of unpleasant surprises that occur when deploying to production.  We are shooting for boring deployments:  Boring Is Good!