Friday, February 15, 2013

Improvements on Continous Integration

Continuous integration allows code on each commit to be directly integrated into the code base and pushed to a production or test environment. This is a great idea in theory and works well on many projects but is it always the right solution?  What alternatives exist with modern build systems and what are the benefits of distributed version control systems(DVCS)?

Common Issues that are introduced by Continuous Integration:

  1. Developers commit changes that break automated tests. Regardless of the reason this should not happen, but when it does the whole team suffers.
  2. Automated testing can start to take a long time as the project grows. Constant effort is needed to keep build time down.  The whole team suffers as the build time increases and builds start to happen concurrently but are deployed out serially.
  3. Only the developer responsible can commit when the build is broken, but not everyone can help in fixing the build.
  4. Broken builds that cannot be fixed within a few minutes are happening too often and the rollback process is affecting other commits.

This results in developers waiting long periods of time to find a "commit window" and waiting to make sure everything is fixed in sequential order when there is a broken build before they can commit again.

What can be done to overcome these types of issues:

The idea that each commit is directly pushed into the main line of development lies at the heart of many of these issues. If a developer makes a mistake, it becomes everyone else’s problem on their commit and the team must work together to resolve the issue quickly. On some projects with many developers this could be a constantly recurring theme that is very time consuming, especially if the team is new and doesn’t have enough discipline to adhere to the demanding continuous integration principles. While it could be argued that marking software the right way isn’t easy, we think that in some cases there is a better way to handle some of these issues.  The root principles of Continuous Integration can still be achieved while protecting the development process.

Our Solution:

  1. Temporarily separate developers code from the development trunk until the code passes all automated testing.
  2. If code passes all tests it is then merged into the development pipeline where all other developers can access the code.
  3. All builds can run concurrently in parallel.

A dirty word in CI community is feature branching. However, when used correctly and with modern build systems you can still have a code base that is integrated and tested often, which is the real essence of what CI is all about.

Recommended Branch Layout:
  • Master
  • Release
  • QA
  • Development

Each of these branches directly gets deployed on successful merge or push into an environment in Heroku or whatever setup you might have.  The development branch is the first real point of integration of various branches. We use branches for major features or epics of any application that we build in this type of setup.  In this environment, testers and developers can review their integrated code and see if it is really ready to be merged into a User Acceptance Testing environment (UAT).  When a tester/developer feels something is ready we merge the development branch into the QA branch.  This essentially creates a snapshot and sends a revision to an environment that the customer uses to verify if a feature is ready for staging and final review.  

We use the next branch, Release, to feeds an environment for any final testing before actually being deployed in a production environment, which is fed by the master branch. Since these branches are automatically pushed into environments on updates and shared by developers, we don’t want any developer to directly make changes to these branches.  We get around this by implementing branches for each epic that a developer is working on.  This means that when a developer starts working on an epic they will create a new branch specifically for that epic/feature (if it has not already been created by another developer).  All of their work will be done solely in this branch.  To handle the actual code merging into the development branch, we use Atlassian’s Bamboo build system.  

As of version 4.0, Bamboo can automatically detect new branches in your repository using regular expressions and generate build plans for them.  Bamboo can be configured to detect any branches that started with a string like “epic-” and create build plans that run all of our automated testing.  While Bamboo has automatic branch merging through their tools named Gatekeeper and Branch Updater, the problem with these tools is that if multiple builds are merging into the same branch in parallel you end up with Git Head errors.  As of writing this article, there is no way to configure either of these tools to merge in such a situation.  This is why we have created an additional step in the Apache Ant build file to do the merge.  
The plan runs the following steps:

  1. Prepare environment
    1. Install any linux libraries required by gems.
    2. Run bundle install.
    3. Configure database.yml to use a unique database name for this specific plan.
  2. Prepare database
    1. Create, run migrations, and seed the database.
    2. Using the parallel_tests gem prepare test databases
  3. Run RSPEC tests in parallel over 8 cores
  4. Run Cucumber tests in parallel over 8 cores
  5. Merge code
    1. Clone a clean copy of the repository and checkout the development branch
    2. Run a merge of the epic branch into the development branch (the name of the epic branch is passed along to the Ant file as a Bamboo environment variable)
    3. Push code back to BitBucket

Each of the main branches have a deployment build plan that push the code in the branch to their respective Heroku environment.  All of the main branches have a merge build plan but only development is automated; the rest are manual.  These builds are used by testers to create the “snapshots” mentioned before to update branches.  Unlike the development merge plan, they only prepare the environment and merge code.  

Here is a diagram showing the development flow:

For developers, Git is a much more powerful tool than traditional central version control systems and strict policies are often necessary to minimize issues.  Developers should be doing commits as often as possible, even if these commits are not immediately pushed.  With central version control systems, a commit is directly associated with a repository change.  With Git, developers can associate commits with check points or the end of a thought process.  Along with these frequent commits, developers should be syncing the changes merged into the development branch into their local branch’s code.  This allows developers to test what the end result of merging into development will be and helps prevent merge conflicts from becoming overwhelming.


Continuous Integration is great, however, you might want to consider alternative solutions when presented with some of the issues we have outlined. While feature branching used to be difficult and not work well with CI, new features of modern build systems liked Bamboo and distributed version control with easy, automatic merging have opened the door to new possibilities while still keeping the overall idea of the values that continuous integration brings to projects.

This article was written by Grant Hudgens, a Build Engineer at Software Allies.


  1. Nowadays, most of the businesses rely on cloud based CRM tool to power their business process. They want to access the business from anywhere and anytime. In such scenarios, salesforce CRM will ensure massive advantage to the business owners. Cloud Computing Training in Chennai

  2. Definitely a great post. Hats off to you! The information that you have provided is very helpful. agen bola

  3. I really enjoy simply reading all of your web. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post taruhan bola

  4. Using a stocks tip list like this is a smart move to make if you are inexperienced or don’t have the time to put towards analytics yourself like hardly any of us do.electrical contractors perth

  5. That post is really informative and interesting, thanks!
    Check football prediction website also!

  6. This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai