Title Text

Text

General topics of discussion were:

• How IT Development Operations (DevOps) can help improve speed-to-market.

• DevOp challenges and how business and federal agencies are addressing them.

• Good first steps to take on the journey towards full DevOps realization.

• How to break down the opposition to adopting DevOps.

• DevOps inspiration—people and organizations.

• How to integrate DevOps into the ever-present compliance functions.

• About the ACT-IAC Conference #1

• About DevOps #7

• General Topics of Discussion #2 

• DevOps Basics #14

• Release Management #29

• About the Presentation #4

• Questions Raised #5

• Security & Risk #31

• Inspiration & Innovation #38

• Where Do We Go From Here? #40

This presentation is a summary of the ACT-IAC "Achieve Agile Nirvana Through DevOps!" educational and training conference. 

NCI's Agile Technology Sector volunteered the services of Senior Vice President and General Manager Lawrence Fitzpatrick and Corporate Writer Heidi Heider, who wrote and prepared this presentation for ACT-IAC.

The panel discussion has been organized by subject, and contains additional comments and explanations to clarify salient points and make the conversation accessible to a wider audience. The authors have enhanced the presentation with sidebars, graphics, and original drawings by H.A. Raebeck.

• How do I know this isn’t another buzzword?

• How do you apply DevOps to legacy systems?

• How do you unify DevOps, software engineering, and release engineering?

• Does DevOps mean I have to do agile?

• How do you go from a silo culture to a methodology that is the antithesis of that culture; where do I start?

• Can you acquire DevOps?

• What is the first step you take to make sure a cross-functional team works together?

• Is DevOps being accepted by organizations?

• How does DevOps change how you think about change control boards (CCBs)?

• Why should those crazy developers have more control over release?

• How do you apply devops to legacy systems development?

• How do you work with DevOps and the security world?

• Is there still a real need to have policies that contain risk aversion?

• What models have been inspirational?

• What are your thoughts on driving innovation?

• How does DevOps hurt or help security?

• How do you deal with all of the policy hurdles in the IT environment?

• What information should be public and what should stay private?

DevOps is a set of IT principles and practices that aim to reduce the cycle time between dev and ops (development and operations) and improve quality.

To do this, you have to include all of the related IT functions, such as security operations, etc. Ultimately, you want rapid, short feedback cycles from production. This is a standard way of operating; everyone should be doing this.

For example, one purpose of 18F is to improve the way government builds application services; making them better, faster, and more frequent; and this wouldn’t be possible without DevOps.

Leadership matters. To apply DevOps successfully, you need smart folks at the helm who know what they’re doing.

In order to reduce cycle time significantly, compliance people (involved with FISAM security, Privacy Act, etc.) need to engage with the DevOps team and they need to do this early in the process. Do not wait to check compliance until right before releasing to production. Have the compliance staff test and evaluate alongside and in parallel with the dev and ops teams, from the git-go.

Full transparency of all relevant artifacts, at all times, to all stakeholders, makes a big difference in getting everyone on board and it raises productivity levels.

Don't let the tools stand in the way of productivity. If your standard tools are not getting the job done, throw them in the trash. Adopt the tools that help you get going.

DevOps promotes continual improvement. Examine impediments and don't be afraid to jettison baggage that's slowing down the team.

• Always make the work accessible to everyone.

• Create systems that allow work to continually move forward and change.

• Choose a distributed version control system and put everything in it, now.

• If tools are hurting your organization, get rid of them. (Trash your office apps if it will improve products and/or processes.)

The impact of DevOps hasn’t fully been felt because it’s still new. In many cases, it has reduced stress over things that go to production, but people involved have to adjust to operating at a faster pace.

Customers, on the other hand, will easily accept DevOps when they get products and services faster. And if everything is running smoothly, people rarely question why.

To make sure a cross-functional team works together:

• Start small and show that the process has worked really well.

• Identify a line of business that gets little attention from IT and do something awesome for them by promising agile [“work with the willing & needy”].

• Leverage that success to sell to the mainstream. (Sell your strategy to leaders of business units.)

• Then take the big jump by selling to directors and higher up.

Stay in touch with the teams.

Get involved in your vendor’s activities.

• Testing should be built into your process from the jump.

• Work with security early.

• By changing priorities in a backlog, you can start a large project immediately.

• Talk to stakeholders and speak in their voice. Don't make them do all the work. 

• Let the stakeholders know that this is about making them successful.

To regression-test a large complicated legacy system without automated regression testing is very expensive. At some companies, test teams can be three times larger than dev teams. [That is a 3:1 ratio; and while there is no perfect number of testers to developers, a ratio greater than 1:3 is unusual.]

Much of the DevOps focus is on new development, but many legacy systems were constructed without the benefit of a DevOps mindset.

So, how do you apply DevOps to legacy systems?

Testing automation is lacking for legacy systems. So it's harder to make changes quickly and deploy them when you don't have an automated way to tell whether the system works after you've made changes. This is why regression testing can be lengthy and time consuming. However, over time, for some systems, there may be some remediations for the automated testing deficits. [In the meantime, there are other parts of the problem, such as provisioning and deployment that can be successfully addressed using DevOps and automation.]

Back to: Inspiration & Innovation

Technical debt can be a problem if you fail to build or fix something that is required.

But it can be okay not to implement certain features, as long as you're doing it intentionally, in order to build "the minimum viable product."

In other words, intentionally incurring technical debt can be a good way to get minimally sufficient products into the marketplace quickly.

The government tends to deal with technical debt by planning a massive swap of old versus new, usually involving a large legacy modernization project, which is high risk and very costly.

In many cases, a smarter alternative is to remove technical debt incrementally, in small pieces.

Remove Technical Debt Incrementally

Back to: Inspiration & Innovation

Stay in touch with the teams. 

When an environment has one contractor/vendor for ops and another on the dev side; one or the other can start feeling paranoid. [Their missions are often at cross purposes, so there is a natural divide—one to safeguard stability—the other to improve capability.]

Is it wise to have one vendor handle the dev side and the ops side?

But when accountability is properly aligned, it doesn’t matter if there are multiple vendors or one vendor. It’s about skill set.

Get involved in your vendor’s activities.

Being your own systems integrator helps to mitigate the tension and unify the goals.

Organizations often struggle to do releases. Releases can be inconsistent and often are unsuccessful. Some of the challenges faced:

• How fast can you push from one environment to the next, including production?

• How reliably and consistently can you push from one environment to the next?

• How stable is the product that you pushed? [One common source of problems is multiple environments with unique configurations, which cause the software to behave differently.]

• Once diagnosed, how quickly can you push a corrected version of the software; or make an environment modification to deal with the issue (which may involve adding resources, etc.)?

• When you find a problem in an environment, how quickly can you diagnose the problem? Can you get the information you need from logs, etc.?

[DevOps addresses these points by increasing the speed with which you can release software, while increasing the certainty that the release activity actually works as intended. Double win.]

DevOps aims to unify software engineering and release engineering. You can do this by adding better tools to the broader toolset to improve processes and performance.

Traditionally, deployment takes forever. But by rotating developers through the DevOps team they can “walk a mile in the other man’s shoes,” and this gives the developers greater understanding (and empathy) and makes the entire process, (inlcuding release) smoother, faster, and better.

• If you’re not including security in the DevOps process, you’re not doing DevOps.

• Monitor production—and if you find something—the time required to repair it is so fast that it reduces risk. [DevOps automation allows the teams to go through the cycle of software fix, build, regression test, and deploy very quickly—tens or hundreds of times faster than the all-to-common manual practices.]

• DevOps will increase security, if only because of automated test pipelines, static code analysis for every build, and the use of the same diagnostics continually.

• You have to work with security early.

• The policy folks almost always want to state what people should do or should not do to protect the organization, and since they see this as their job, it’s not surprising. But they usually have a hard time describing the actual outcomes they want to achieve. Often the route they suggest is not the most direct path to their desired result, and this is your leverage.

• Use a sales approach. [Emphasize that you agree with the policy makers and enforcers about the fact that security requirements must be met. Show them that you can better meet their needs and achieve the objectives of the organization by doing things in a new way.]

• Recognize that oversight is about reducing risk, and propose that your technique reduces risk as much as possible. Also, you can stay a bit below the radar by continuing to make small incremental changes.

• All DevOps information should be kept in open repositories for sharing, and that includes security documents.

• Try to be proactive and outline the things up front that should not be made public.

• You need to conduct your analysis up front to identify this information and determine how it should be protected. But use restraint—try not to achieve your security goals by relying on "security through obscurity" [a well known tactic, akin to the intellectual property model known as "trade secret"]. 

• If information needs to be protected, be open and straightforward about what and why. There should be no hazy areas.

People frequently hire an outside integrator, believing that it is less risky to make just one person accountable (have "one neck to choke"), but often it increases risk. And this is because it is usually less risky to place key decision-making authority in the hands of someone more closely aligned to the organization’s mission.

It's almost always safer to do less than more. The people closest to the mission—the customer—should be trained to prioritize. When work gets too complicated, sometimes the only viable action is to simplify.

The government, or the customer, is in the best position to correct problems (reduce scope, reprioritize), and this minimizes the biggest risks.

Perhaps the risk aversion that is baked into security policy is not a real need. There is a great deal of oversight in government. The traditional environment tends to be low trust and high criticism, so risk aversion is a natural response. Avoiding risk is now built in to the program, but “at all costs” increases costs, as well as complexity. In the end, it can cost more to avoid all risk than to tolerate a certain amount of “controlled risk.” [Excessive oversight also squelches innovation.]

There is an organization that embeds security engineers in every agile team and this has been so successful that the entire security and compliance function has become agile. They integrate their security requirements as “stories” on the backlog and execute them as part of the two week sprints; anyone can do this.

Agile and Compliance

Back to: Does DevOps Require Agile?

To innovate, create a sandbox where people can experiment and fail, and this will reduce the cost of actual failure. DevOps is perfect for this.

[Because DevOps lets you build, deploy, and test so efficiently and quickly; you can run this cycle with really small changes, which may not even be noticed. This inherently reduces risk of failure and allows for experimentation.]

The Twelve Factor App and Heroku. These principles involve keeping your configuration coupled or decoupled; testing; rapid deployment.

Moving from full builds to packages; Amazon.

Inspiring DevOps Models

Amazon's transformation to service oriented architecture (SOA). They moved away from big monolithic apps to everything as a service. Now, they can enhance, fix, and deploy changes to pieces much easier than to the huge apps they had previously. The new major apps are just assemblies of SOA parts.

Based on this discussion, we can say that:

• DevOps can cover a lot of ground.

• It still has its challenges, but solutions to each problem abound and many companies and organizations are already experiencing great benefits.

• It’s compatible with other popular process improvements frameworks.

• DevOps is not a fad but is here to stay. It’s not a question of whether you should get onboard the DevOps train, but when.

• Resistance is more likley to come from staff members and vendors who may not want to take on more responsibility or work faster (or erronesously believe they are giving up control), than it is from customers who are getting better, faster service.