Agile Zone is brought to you in partnership with:

Arnon Rotem-Gal-Oz is the director of technology research for Amdocs. Arnon has more than 20 years of experience developing, managing and architecting large distributed systems using varied platforms and technologies. Arnon is the author of SOA Patterns from Manning publications. Arnon is a DZone MVB and is not an employee of DZone and has posted 68 posts at DZone. You can read more from them at their website. View Full User Profile

Continuous Deployment and Continuous Integration

03.19.2009
| 4746 views |
  • submit to reddit
You might have noticed my blogging is slowing down a little,  in case you're wondering the culprit is xsights - As we're getting closer to
production (2-3 weeks if all goes well) everything else slows down...

We all know about continuous integration (CI). We (xsights) are seeing a lot of ROI on the investments we made into signing up to CI esp. when we augmented by automated testing. Everyday we can and do actually release several versions and run load tests on them on different server seeking the best release to  "freeze". Where "freezing" means the  system that will be used in production.  That's the way it is done, right? You work on a release for x months (sometimes years..) and after an integration period at the client's site you go live and forget about them (save for show-stopping bugs) until the next major update several months later. For instance, setting up our "methodology" I figured a 6 months release cycles for major versions plus 1 month releases of minor version would suffice.

These days, however the more I think about it, The more I believe this approach is not the right one for our situation. We are going to go with Continous Deployment. There are many reasons for that  primarily:
  • The requirements stream (new reqs and changes) is still pouring in - and in increased rate.
  • We are building a service platform not an installable product -  This means we get to control where how it is run so there's less overhead in streamlining updates
  • It would be our first public installation - were going to roll out bug fixes anyway (I personally never write bugs... but you know ;) ). We'll need a proper procedure for updates anyway
  • We are doing this anyway - every new demo/internal run we have is using the "stablest" latest and greatest :)


What does  Continuous Deployment means? It means we'll be rolling out new production versions on a daily basis (or even more frequently!)

How that's going to work? Well, First there are the prerequisites:

  • Version control - something that can track and tag versions...
  • Continous integration which is set up and running. You cannot release continously if every new development offer breaks the system. An automated smoke test is
  • Automates tests that provide a good approximation of real usage (.e.g. usage patterns, loads etc.) - You need to be able to have a high onfidence level that whatever you have.
  • Staging environment - An environment as close as possible to the production system (just like you'd have for a web-site)
  • Monitoring and alert system - You need to be able to identify problems quickly - the running

Then there's the process:
  1. identify a baseline Stable release -The "fallback".
  2. pick the most promising build of the day
  3. run it through more extensive tests (at least burn-down and load - which means a release maybe 1-2 days old by the time it hits the production system)
  4. if everyhting is cool deploy to staging - else fix bugs and repeat from 2
  5. repeat tests in staging environment
  6. Update system (This is a point that still needs work so we can support a version hot-swap i.e. without a shutdown)..
  7. If the monitoring system alerts unforeseen bugs (i.e. problems with the release) - roll back to stable release, try to add test for the problematic behavior and repeat from 2
  8. if the release worked good enough - mark it as the next Stable release

That's the plan. However as we all know, talk is cheap so be sure to check in here in 2-3 months to hear how did we fare :)
References
Published at DZone with permission of Arnon Rotem-gal-oz, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Gene Gotimer replied on Fri, 2009/03/20 - 7:57am

One thing to consider in between your steps 5 and 6 is shadow testing. If you can find a way to record your live traffic (to the production instance) and replay it in a staging area, you can see real world activity. If you can do it real-time (or near real-time), you might get a good feel for timing and performance issues. At least you can make sure that it won't crash-and-burn under real world use because you missed some scenario in your smoke tests.

Giridharan Rama... replied on Sun, 2009/03/22 - 9:44pm

We have to be bit cautious about Continous Deployment since it can sometimes set unrealistic expectations regarding Turn-around-time for defect fixes in development cycle.. There are frameworks like Smartfrog that can assist in automated deployments but requires very mature release management process.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.