DevOps Zone is brought to you in partnership with:

Leaving university, I thought I'd be a developer happily knocking out code but was always drawn to tech support. How do we help people use our tools better? Now, I mostly specialize in consulting and introducing new customers to our tools. I am a Tech Evangelist and Lead Consultant at Urbancode. Eric is a DZone MVB and is not an employee of DZone and has posted 82 posts at DZone. You can read more from them at their website. View Full User Profile

Q&A From Building a DevOps Team That Isn't Evil

02.27.2013
| 2590 views |
  • submit to reddit

We had more questions in our live (and now available on-demand) webinar  ”Building a DevOps Team that Isn’t Evil” than I could get to. What follows are answers to the questions we received.

Slides for the event are also available after the jump.

Q: How do you get focus without dedication of DevOps team members? Would like a bit more on incorporating “part-time” members.

Ideally the core of a DevOps enablement team is dedicated to the project. But if some of the experts are borrowed from existing development or operations teams, you do need them to dedicate a clear percentage of their time to the DevOps team and commit to attending set working sessions. The feasibility of this approach is largely dependent on corporate culture.

Q: I have a single Scrum team of 4 people (2 dev, tester, business analyst) with DBA as extended team member. We have to do both development and prod support. How can DevOps idea work with small teams in multiple roles?

In this kind of small team, the cross-functional and anti-silo goals are achieved. You’re instead going to use retrospectives to improve the team’s delivery. For inspiration check out:

Q: You’ve emphasized that DevOps is anti-silo without talking enough about why. Why are silos so evil? 

Fundamentally silos lead to local optimizations rather than focusing on how to improve pace and delivery from idea to customer. If we were manufacturing cars, a local optimization might result in being able to build 30% more car doors a day without actually being able to build more cars. At best, we have people sitting around bored, at worst, we fill a warehouse with doors. In IT terms, the Ops team is incentivized to maximize uptime. An easy optimization to make is to make it painful to get a change to production approved so fewer changes happen. The Ops team gets a bigger bonus, but the business gets less of what it needs and loses money.

Q: If a DevOps team is successful it seems to me that it will eventually become redundant. Is this a fair comment?

If the team’s charter is to enable a shift to DevOps across the enterprise, then it is successful when the enterprise has embraced DevOps. Likewise, a good tutor may not be needed after helping a student catch up to the class and improve her study habits.

Q: How have auditors been ok with a devops model where devs have access to prod? e.g., continuous testing, automation, version control, logging, etc?  is one single dev allowed to push a code chg thru the entire process?

Few public companies allow a developer to go from change to production, or access production databases directly. There may still be roles with responsibility for building new features and others that operate on production. The goal is enhance the flow of knowledge between the people in those roles, and for there to be some joint responsibility. For instance, logs may be exposed to developers without giving access to the production box. I think the team at Stackify is trying to make a business around exactly that.

We also see customers who today don’t care who presses the “deploy to production” button. They setup uDeploysuch that something can only be deployed after it has been approved / certified by other people.

Q: Who pays/sponsors Dev/Ops (advisory team)?  Dev/Ops by def’n will typically not fit clearly in to Dev or Ops budget buckets in a typical organization?  What’s a typical approach here?

Sometimes it falls under an existing group that spans those concerns, such as Release Management. Other times there’s a charter from the CIO getting this going. Otherwise, it’s most likely to be chartered by the operations side of the house when it realizes that a radical change is required to continue to be relevant.

Q: How cost effective is this model as the overall impression I get is DevOps team sits in between Dev and Ops team.. they are only facilitators.. cost over head? 

It’s easy to view most of IT as overhead. This team is an investment in driving waste out of the process. Silo’d, low automation approaches are slow to deliver to market (inventory waste). Writing documents to explain a deployment and then executing that deployment manually is hugely wasteful compared to an automated system. Knowledge of production application behavior not shared to development leads to a range of bad development decisions. This DevOps enablement team is a repair crew. They’re fixing something that’s broken. An engine burning oil still “works” and getting it repaired is “overhead”. Same thing here. Check out our whitepaper on ROI for deployment automation (a slice of DevOps) as an example.

Q: Who creates the automation processes?

This can be tough. This can be a collaboration between development, release engineering and dedicated automation guys who act as a service bureau. If you feel that getting developers who understand the app to collaborate with operations guys who understand the production environment is difficult, you have a Dev / Ops problem. Regardless, it is up to the delivery team (development with some ops support) to streamline their releases and be successful when hitting prod. This can involve changes to the basic application infrastructure to support canary or zero downtime deployments or just updated scripts.

Q: How do you control/mandate compliance with the direction provided by Dev/Ops…I mean DO is tauted as an Advisory Boards of sorts….so its branded as advice…but for it to have legs there has to be some way to incentivizing Dev and Ops people to comply.

Compliance is non-negotiable for many organizations. This is where bringing in the “Grumpy Team” including audit early in the process is important. Typically, we see folks who build automated infrastructure or app deployment processes wrap that with role based security. While everyone on the team can see what’s where, and how it got there, only certain people can release to production.

Q: Would it be fair to combine an automation-focused team with an on-going DevOps feel? As said team doesn’t hit any buttons (low/no ownership), they just build and maintain the buttons

Yep.

Q: Could you give an outline of the 20% (that gives 80% results) when facilitating “softening the silos / softening up ownership that enables faster change”

Facilitating a finger-pointing free post-mortem for outages and releases where:

  1. Practitioners re safe to say what went well and badly
  2. Recommendations are generated that don’t involve “be more careful next time”
  3. The team follows through and changes accordingly
From an automation perspective, automate application deployments first. Environment provisioning, awesome monitoring, etc can come later. An automated deployment that lets everyone see what’s where promotes transparency and shared process. Very nice for forming the right habits.Q: What do you recommend to capture (document?) what is done so it can be repeated elsewhere? Is this cultural? Once handed over to the team’s, can DevOps help support keeping this up to date?To the extent the DevOps enablement team is maintaining an updated SDLC document, that will likely be updated if suggestions to process are made. Updates to automation are ideally captured and either made part of templates that can be shared across the enterprise or at least referenced from knowledge base or wiki.
Published at DZone with permission of Eric Minick, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)