The Unintended Consequences of the “Infrastructure as Code” Paradigm

This blog covers a contrarian — possibly heretical — view of the “infrastructure as code” paradigm. TL;DR: If left unchecked, the drive to institutionalize “infrastructure as code” concepts can drive teams to make decisions that negatively impact the business.

At DockerCon19, I asked a number of the attendees who professed proficiency in Docker containers and Kubernetes the following question: “How did you end up in your current role?” It’s always interesting to me how engineers make career choices and why. And Kubernetes is anything but simple to understand or implement at scale.

The majority response was as follows:

• I used to be a software developer
• Once the company moved to Kubernetes for cluster management, the operations team had a really tough time maintaining our compute environments and keeping up with all the new tools they needed to learn
• So we decided to just do it ourselves

Kubernetes — and the open-source ecosystem that goes with it — imposes serious specialization requirements on companies that results in them making ongoing and incredibly steep resource and time investments. The creators of Kubernetes likely never intended it to become the YAML-driven complex beast it is now. Joe Beda, one of the founders of Kubernetes agrees and voiced the following opinion on Twitter:

The team at Heptio (now part of VMware) attempted this with their ksonnet project. Unfortunately, they were unable to find a clear path to a user-friendly and simple Kubernetes abstraction layer. At Rafay Systems, we have accomplished this. Reach out to me directly for a demo.

And here’s the most important point: Infrastructure software falls squarely into the “undifferentiated” category that Noah Pepper highlights in his blog. Regardless of the vertical your company falls into, running containers across a number of clusters is not a specialization anyone should be building in-house. Your company and team presumably have more strategic projects to work on. Unless you’re Google or Facebook, any perceived differentiation on the infrastructure front is temporary at best because your competitors will eventually build/buy their way into solving the same challenges you did. Granted some verticals may need deeper solutions in certain areas such as storage security, and the invisible hand will quickly drive startups such as Rafay to fill that gap.

When companies encourage software developers to write complex infrastructure software, they are actively creating barriers to their own growth. This may seem counter-intuitive but in our present-day economy, high-caliber engineering talent is a scarce resource. If your top engineers are learning how container networking works and aren’t developing new capabilities for your products, do you really have your priorities straight? Are you trading short-term wins for long-term losses in the market?

Eventually the intelligentsia will start talking about the next big thing on Twitter, and your top engineers will want to work on that. At that point, who will be in charge of care and feeding for all the in-house software that’s already been written to operate your infrastructure?

Business should continue to maintain a strong focus on their core competitive competencies. Of course your teams can build anything, but should they? If you’re Google — sure. If you’re an insurance or healthcare organization, maybe not. We need to come together as an industry and demand more of our vendors. We can’t keep filling gaps with hard-to-find engineering resources. And we must prioritize looking for external tools and solutions for technology challenges that are not umbilically tied to our core business competencies.

Questions? Comments? Strenuous Objections? Please feel free to get in touch or email me directly.