These are my notes from the DFW DevOps Live 2017 Kickoff meetup. We had the privilege of meeting a prominent advocate within the DevOps community, Derek Weeks. Among other things, you many know Weeks as a co-founder of the free, high-quality, online, world-wide conference, All Day DevOps.

Weeks' work and research at Sonatype gives him a unique insight into the state of DevOps and security at many large enterprises. Many of the statistics and insights given during his presentation came from Sonatype's 2016 State of the Software Supply Chain report, of which Weeks was a primary contributor.

If you are not familiar with Sonatype, they build and support the Nexus Repository Manager product - a self-hosted software artifact repository manager that works with popular open source package managers including Docker, Maven (Java), NPM (JavaScript), PyPi (Python), NuGet (.NET), and RubyGems. As far as I can tell the basic Nexus product is free, with source code available on GitHub, but it does not appear to be a community driven project.

The Software Supply Chain

Before this meetup, I had never heard of the software supply chain concept. The software supply chain represents the sourcing of (mostly) open source components that engineers assemble into commercially viable applications. Throughout the presentation, Weeks drew parallels to the writings of W. Edwards Deming and his book, Out of the Crisis.

Of most interest to DevOps professionals is Deming's insights into the differences between Japanese and American automative supply chains in the mid-twentieth century. Two of the insights we can extrapolate from Deming's work include:

  • Build quality in at every stage - inspection (at the end of the assembly line) is too late!
  • One of the key factors that allowed Japanese auto manufacturers, such as Toyota, to build high-quality automobiles at scale, while their American counterparts struggled with quality issues was that the Japanese manufacturers sourced 80% of their components from third parties, whereas American manufacturers built 70% of their components in-house.

According to Weeks, DevOps gives companies a competitive advantage by employing principles and practices that improve velocity and increase software quality. One of the ways we can achieve higher quality and velocity is by using fewer, high-quality components, and continuously tracking when and where we use those components. It is vital to continuously monitor the origin, usage, and status of your software components in order to be able to quickly and accurately asses and remediate affected applications when a defect, such as a bug our security vulnerability, is discovered. This practice is rooted in one of the DevOps principles, making invisible things visible.

Weeks observed that one company he researched used over 16 million open source software component combinations, of which only 5,000 were unique components when excluding version differences. That's a nightmare supply chain to manage, and as he continued to expound open charts and graphs and statistics from many companies and the industry in aggregate, the problem is a common and often overlooked problem.

Some other useful insights from this part of the presentation included, "Every developer is in procurement" unless you do something about it. Very few companies actually employ processes and tools to control which open source components their engineers are bringing into an organization, or how they are monitored and managed after they are introduced. To emphasize the gravity of the situation, Weeks expressed that

Variability in high velocity supply chains will kill you!

Ok, maybe it won't actually kill you as a person, but it will leave you with applications using old components with known and actively exploitable security holes. Variability in software components will also require re-training headaches as components are not consistent between teams and projects. It also means that introduction of new, high-quality components that can increase velocity, don't spread throughout the organization an instead remain in the silo of the team or project that made the discovery.

Throughout the night, Weeks threw out plenty of interesting stats and quotable phrases like,

  • 43% or organizations have no Open Source policy to vet and approve components for use. #1 Problem: Vetting a component takes longer than a sprint
  • 55% of open source projects weren't fixing defective components
  • If an attacker has an exploit for a vulnerable component, they have a way in anywhere that component is deployed. Example: some of last year's ransomeware attacks were traced back to a vulnerability in JBoss that was discovered and patched in 2010.

Components are like milk, not wine—they spoil with age.

common saying at Sonatype

This picture is starting to sound pretty grim. But don't be too anxious. You can do something about it!

Improving your software sourcing practices

Much of the information that comes next is geared towards the enterprise Java world, like large financial institutions and global technology companies. That doesn't interest a whole lot, so I'm only going to cover that parts that do.

Use in-house repository managers

Don't download your app's dependencies directly from the internet. Use in-house repository managers (Docker registry , RubyGems, NPM) that only contain known, pre-approved packages for use. Usually it's not developers driving this decision, but a mandate from the security team.

Inspect your dependencies

It's not enough to require the use of an in-house package repository. When needing to hit a deadline, or wanting to use the latest and greatest, developers will often bypass the repository and go straight to the internet anyway. So it's important to know if you are using good or bad parts as part of your CI process. Fortunately, there are tools such as the OWASP Dependency Check that will let you know if any of your dependencies have known security vulnerabilities and can easily be integrated into your CI pipeline. You may also want to look for language specific dependency checkers, such as bundler-audit for Ruby projects.

As an aside, Weeks also mentioned that some government contracts now require a software bill of materials detailing your software sourcing processes for software components, and you can even lower your cyber-security insurance premiums by demonstrating good software procurement practices.

On a personal note, this reminded me of a comment a friend of mine in the security industry once made, "Over 90% of the security tools and practices we sell are unnecessary if maintainers keep their systems patched and use strong passwords."

Another thing you can build into your CI process is automatic detection of license violations, but so far the only open source solution for this is Apache Creadur project. I didn't even know CI for legal existed until now! Very DevOps.

Automation

Ok, I've already rehashed this several times now, but by building these checks into your continuous integration pipeline, you can have a Zero TTR (time to remediation) for many security vulnerabilities. What does Zero TTR even mean? It means that you are detecting and remediating unpatched vulnerabilities before they make it to production.

Build quality in

If you rely on (end of the line) inspection for quality control, it's too late! You're going to overlook defects, or it will be too late to fix them. You need to build automatic quality control checks into every stage of your software development process.

Quality components comes from improvement of the process.

I believe that's another Out of the Crisis, but I'm not entirely sure now.

Adoption

When introducing security checks into your CI pipeline, you should start with reporting to start making it visible. As people start noticing improving the situation, you can tighten up the policy by issuing warnings and even failing builds.

The remainder I won't go into in much detail, we looked at Sonatype's Nexus IQ Server and its integration with Jenkins and Java IDEs. I will say, I found their Nexus repository manager really interesting because it supports multiple packaging formats and protocols, such as RubyGems, PyPi, and NPM. I've worked in environments where we were running all of these ourselves, and a tool like Nexus could have saved us a ton of time maintaining these solutions.

Wrap-up

Anyways, Derek Weeks is a great presenter and due to the nature of his work knows some really interesting things about how companies build software. Some of my key thoughts and takeaways from this meetup include:

  • The software development lifecycle (SDLC) is speeding up as development becomes more componentized.
  • The Java community is still huge!
  • You need to care about the quality of your components.

The 800-pound gorilla is back

Ok, this is just a personal trend I've observed over the last couple of years, and I will definitely need to follow-up with a more in-depth blog post on the subject. But for now, I'll just say this.

Enterprises who are practicing DevOps are significantly out-pacing their small businesses competitors in both speed and quality of development.

You can quote me on that. It hurts to say because as someone who prefers working at small companies, that scares the crap out of me! But I'm telling everyone right now, some large corporations are finally figuring out how to move with the same speed and agility as their startup and small business competitors, only corporations have a ton of capital at their disposal, too. It's definitely not a fair fight, and I don't think it's permanent. But the pendulum is definitely swinging in that direction right now.