Part 1 of 5 By: Dan Cohn, Sabre Labs
How do you go about increasing software engineering productivity? How do you deal with the fact that every engineering team has a unique blend of standards, practices, and tools? How do you measure team productivity in the first place?
About two and half years ago, we began an initiative to improve the software developer experience at Sabre while increasing productivity. It started as a Sabre Labs “DevProd” (developer productivity) project and soon became “EngProd,” taking its name from the Engineering Productivity discipline at Google. (Both Sabre Labs President Sundar Narasimhan and Chief Architect Andrew Gasparovic are ex-Googlers.)
The more we analyzed the problem, the more complicated it appeared to be.
The answer for us was to start small and local. By local, I am referring to the Sabre Labs team itself which had just opened its Boston Innovation Lab and was hiring software developers to build prototypes and platforms to nurture innovation at Sabre and in the travel industry at large. As a relatively new team, we had the benefit of a (mostly) blank canvas on which to paint the perfect landscape of software development methods and tools that would accelerate our tech transformation efforts. If the model proved successful, we would then spread the good word to other teams throughout the company.
Of course, we were not the first to look for more efficient and effective ways to develop software. Every enterprising manager or architect is eager to improve the morale and output of their teams. A big part of our approach was to draw on best practices from leaders in the industry, weave them with those of other trailblazing teams at Sabre, and align them with existing standards and tools in wide use across the company.
As those in the software business know, there is a lot more to software engineering than writing code. To enhance productivity, we would have to consider many facets of the work from onboarding and training new engineers to collaboration tools, documentation, build tools, coding standards, source code management, testing, continuous integration, and more. We looked at the software engineering process from end to end to ensure we had clear and effective practices at every stage. What I mean by “clear” is that the process is easy to understand and follow. Nothing stifles productivity (and morale) more than not knowing what to do and having to figure it out for yourself. I cannot overstate the importance of having the right set of tools for each job and leveraging those consistently across the team.
Before describing what we developed, it is worth mentioning that this initiative began as Sabre formed a 10-year partnership with Google to migrate its IT infrastructure to Google Cloud and combine the talent, tools, and other assets of both companies to accelerate Sabre’s digital transformation and innovation program. The timing is largely coincidental but extremely relevant because Google’s engineering practices were a source of inspiration and influenced some of our decisions. We wanted to replicate the success that Google has had in tackling many of the same problems, albeit at a somewhat larger scale.
At the core of every modern software engineering project is a source code repository. Each line of code you write goes into the repository for safe-keeping, version control, and collaboration. We decided to experiment with the concept of a monorepo, or single shared repository, for all new Sabre Labs projects.
Before the era of GitHub (a popular service that helps developers store and manage their code) and “microservices” (an approach to software development where software is composed of small independent components), it was common for multiple applications to keep their source code in one large repository. After all, if a repository is already available and in wide use, why go through the trouble of creating a new one and all the infrastructure that goes along with it? At the start of my career (in the mid-90s), I worked for a large technology company that had its own operating system and a common software platform that powered most of its major products at the time. It was only natural to house this code in the one big repository. Otherwise, it would have been much more difficult to import base libraries and other shared resources like type definitions.
This mode of software development became less popular with the rise of web development, open-source libraries, and the distributed version control system known as Git. In the mid-2000s, the world was moving toward smaller, independent software components with external interfaces (often HTTP-based). Why? Mainly because they were easier to maintain and evolve than their monolithic predecessors. Monolith became a bad word in software development circles, and monorepos were guilty by association.
Monorepos still have their detractors. At the time, my first reaction to our foray into monorepos was to share a recent post titled “Monorepos: Please don’t!” with our chief architect Andrew to get his response. It turns out there are many misconceptions about monorepos and quite a few advantages. Shunning monorepos may have been a true case of throwing out the baby with the bathwater. While monorepos do have challenges, as I will explain in my next post, we found that a single monorepo shared by many project teams can serve as the foundation for a variety of productivity improvements. It simplifies collaboration and code sharing, encourages consistency, and fosters transparency and openness.
Imagine trying to improve the transportation system used by a group of neighboring towns. You want to expand two-lane streets into highways, add road signs for safety, repair potholes, and manufacture more fuel-efficient vehicles. However, the citizens of each town insist on building their own roads and using only these local roads. Moreover, each town has its own favorite model of car and will not drive anything else. How do you make meaningful improvements under such conditions? This is the predicament of an engineering productivity initiative in a multi-repo environment.
Our solution was to roll out a new software development platform at Sabre. As you might expect, the platform centers around a monorepo. In this series, we will dig into the following four key elements of the platform:
• Git-based monorepos • Bazel (the build tool) • Containers (as in the Docker variety) • Centralized CI (continuous integration)
Git is a version control system that tends to perform best with a multitude of smaller repos. Is it practical to use Git to house a large and growing monorepo? What obstacles did we have to overcome? What challenges do we still face? Our next post will answer these questions and describe how to make Git work well in a monorepo environment.
