TFS Source Code Permissions Affect Branches


Today I solved a pesky problem with Team Foundation Server:  sometimes, some developers (but not others) could check into a branch even though all permissions had been denied.

My group does all development in the source code trunk, then branches at the end of any iteration that we intend to take up to our production environment. After early bug fix work on the branch we lock the branch down. Thereafter, developers have to ask for permission to check into the branch. This ensures that we know exactly what is being checked in, and when.  And we can decide, on a case-by-case basis, whether to allow it–no one wants a surprise going out to production.


So why could developers sometimes check into the branch even though all permissions had been denied?


As the TFS administrator (for the first time), I had beat my against this without ever getting anywhere. And recently I figured out the problem.

The key was to realize that the problem was described inaccurately.

Incorrect definition: sometimes a developer could get into a branch when I had (or thought I had) denied it.

Correct definition: that developer could ALWAYS get into a certain part of the branch, but not into other parts. Other developers might be denied in some areas but not others, too, but not the same parts as the first developer. Since all developers may sometimes work in any part of the branch the end result seemed somewhat random but that was not the case.


How does it happen?


Our source code tree is rather cluttered, and when I cut the branch intended for production, I end up actually branching several separate parts of the trunk, based on a common label, into a new, single destination root for the branch. I administer permissions to the branch at the root of the branch.

The problem: some subtrees in the trunk had explicit permissions granted to them rather than inheriting from higher up the source tree. These explicit permissions carried over into the branch, overriding the permissions I set at the root.

Very simple, once I understood it.

The solution was to keep an eye on where permissions were explicitly granted in the trunk, and remove those permissions in the branch so that all permissions get inherited from the root of the branch.  It would also be helpful to see where I can eliminate those explicit permissions and allow inheritance to do its work.

This fits the needs for my group, based on our local needs and the structure of our source code (which I inherited and do not like but am probably stuck with for a long time). It may not fit yours–but understanding that explicit permissions in a subtree of the trunk carry over into a branch may help you fix your own problems someday.

Good Software Deployment Practices, Part II: “The Build”

I’ve worked in plenty of environments in which code is compiled on a team member’s machine, then pushed to production.  This raises some important questions:

  • What source code was present when the compile happened?  Was it the latest version?  Did it include something that the developer hadn’t yet checked in to the source control system?
  • What is installed on the developer’s machine?  Are all relevant service packs at the same level as production?  Are production servers missing any required frameworks or APIs that exist on the developer’s machine?
  • Does the code used during the compile match the code that has been tested before going to production?

There are other issues, but hopefully you get the idea.  The wrong answer to any of these questions can lead to production problems.  And in this kind of environment I’ve found that no one actually knows the answer to the questions…

…which brings us to the practice that will resolve these issues:  a formal, controlled build process.  Now, by formal I don’t mean loaded with paperwork; rather, I mean that certain rules are established, communicated, and enforced.

The rules (there may be more for your environment; these are a good start):

  • The “build” (compile and assemble the deployment artifacts—executables, web pages, etc.) occurs on a dedicated machine:  the “build server”.  This is not a developer’s box.  It’s locked down, with only some team members having access to it.  Those team members must know exactly what has been installed at any given time and how the configuration compares to all test and production environments.
  • The “build” is automated to the extent that it is likely to be a very consistent process; i.e., not dependent on who or what triggers it, not subject to human errors, etc.
  • All artifacts being assembled for deployment come from source code, either directly (such as a text file) or indirectly (such as source code, which then gets compiled during the build).  No exceptions.
  • The deployment artifacts generated/assembled during the build are placed in a secure location, probably a network share with limited access.
  • Artifacts deployed to any test or production environment come only from the secure location updated by the build process.
  • The development team does not have the network, database, or other permissions necessary to deploy to production, nor to any test environments other than perhaps a low-level shared development environment.  Thus, it’s not possible to deploy anything even by accident.

These rules can be scaled up or down to fit the size and budget of most development teams.  In a complex environment (such as government contracting) there will probably be much more formality included.  In many businesses in the United States, government regulations dictate a clean separation between those who write the code and those who can deploy it.  On the other hand, in a very small team the dedicated build server may be a virtual machine administered by the team.  In this case the above rules still apply but enforcement may depend on team discipline rather than permissions.

Developer tools can help with some of this.  If you have the budget for it, Microsoft Team Foundation Server (TFS) is an excellent system that integrates source control, the build process, and much more.  You can define any number of build servers and have the TFS server(s) communicate with them.  If this is out of your budget, CruiseControl can accomplish much of the same thing.  It doesn’t offer source control or other features, but it integrates well with other source control system and does a good job managing builds—and it’s free.  There are plenty of other tools out there, including those for non-Microsoft environments; these are just the ones I’m familiar with.

Follow these practices and you’ll reduce the number of unpleasant surprises that occur when deploying to production.  We are shooting for boring deployments:  Boring Is Good!

Don’t Clog The Production Pipeline

Here’s a situation that you don’t want your team to be in:

  1. A low or medium priority (but still significant) bug gets reported in your production environment.
  2. A fix for the bug is checked into the branch of source code that represents production.
  3. The fix is not tracked well, and it languishes in a test environment for a while.
  4. A really hot production bug (undoubtedly written by someone else, not you!) gets reported and must be fixed and placed into production right away.  The new bug is in the same general area of the software as the first bug.



The problem, if you haven’t seen it, is that you can’t easily release the critical fix (bug #2) until you know that the less-critical fix (bug #1) is good.  Has it been tested yet?  Is the problem fixed?  Have regression tests shown that it didn’t break anything?  If not, then what do you do?

The bug fix that sat around in the pipeline to production has become like sludge, potentially blocking the way for something more important.  Either bug #2 has to wait until bug #1 is ready for release, or bug #1 has to be backed out.  Or worse yet, perhaps in the rush no one remembers bug #1, so it gets released in an unknown state—pushing the sludge through the pipeline and right into the machinery.  You may find out too late what you’ve broken.

Keep it moving

Make sure that someone in your organization is monitoring the bug list closely.  Make sure they understand the importance of keeping things moving.  If a fix can’t be put into release fairly soon, once checked into the code branch it should still be worked on until it is fully ready for release.  If there are not resources to get it production-ready quickly, it should not be checked into the production code branch.  Maybe it should be assigned to the next general release instead.

But don’t assume that someone else is responsible for this.  I currently work in a pretty small shop.  Besides the developers we have a handful of Quality Assurance people.  They’re an excellent group, and I mistakenly thought that we wouldn’t have this problem.  But we did, despite good people and good tracking software.

So, as the “keeper” of the source code repository, I instituted a very simple policy that has made a contribution toward keeping the pipeline moving:  I lock down permissions on a branch once it gets close to production, and I don’t open it up for anyone until a member of the Quality Assurance group makes the request (usually just verbally).  That way I know that the developer and a member of QA are talking with one another, and QA understands that when the branch is opened, they’re responsible for staying on top of the fix until it is ready to go out the door.

Such a policy is not for everyone, and wouldn’t scale well to a big team.  But it adds one more opportunity to ensure the we keep the pipeline clear.  The point is to make your own contribution.  What are you doing?


Good Software Deployment Practices

I earlier mentioned my software deployment goal: Boring Is Good!  Over several posts I will cover some of the things that can help accomplish that goal, starting with the very basics.


Basic practice number one, coming up!

Use Source Control

This one seems obvious to many developers but not to all, especially as we change the discussion from standard (Java, c#) code into database schemas and into other technologies where development is done via point-and-click rather than writing code.  There’s often a “publish” button in the Integrated Development Tool that looks so tempting…and it’s great for local testing and for small projects, which is what it was intended for.  But for larger projects, the first rule is unbreakable:  other than on your own machine, the deployment process begins in the source code library.  Nothing goes to production that didn’t originate there.  That means nothing!

“Do you really mean that if Brad, our technical writer, needs to update our release notes (a .pdf file), he has to email it to someone who can check it into source control before you will deploy it?  Isn’t deployment in this case just a file copy to a web server?”


Read the rest of this entry »