Journey to Monorepo and Bazel

Around the middle of this year, we noticed the number of repositories in our Github account had reached 3 digits. Just understanding what was deployed to each environment (production vs staging) or who was working on what and in which branch was getting tougher and tougher as the number of repositories continued to grow. Updating dependencies across all repos seemed like a full-time job in itself. All this resulted in an increasing slowdown in the pace of iteration of the product. We decided we had to move to a monorepo.

The Astradot monorepo contains 100% of the code for Astradot. Moving to a monorepo would fundamentally change how we develop our software. Right from the outset, we knew we would need to switch to Bazel for building our monorepo. However switching to both a monorepo and Bazel, a fairly undocumented build tool (compared to other popular build tools) in one go would be too much to handle all at once.

CircleCI Path Filtering

Fortunately, we noticed that CircleCI, the CI service we use, had recently added support for ‘path-filtering’. We found path-filtering to be a good ‘poor man’s way of building a monorepo. Path-filtering allows running build jobs based on changes since the last commit. This meant that you are only building a subset of your repo each time. Thus looking at the green status of the most recent build doesn’t fully tell you if your entire repo is actually green or not. Maybe that recent commit broke some other service that didn’t even get compiled on this build. However, we found this to be a good intermediate way to migrate our existing CI jobs and code to the monorepo as we figured out Bazel.

Bazel

After migrating all code to monorepo, we started the journey to Bazel. We currently use Bazel for all code in Go, Rust, and C++. Bazel is not a good fit for Javascript. However, we use Javascript only for our frontend, which is in a single folder in the monorepo anyway. Thus we decided to continue using npm for our javascript codebase. The use of remote dev codeboxes that all run Linux, allowed us to target only 1 platform (Linux) for our builds thus further simplifying our Bazel configuration.

Bazel forces the codebase to have only 1 version of a dependency. This radically new way of thinking about dependencies was a knee-jerk at first but now we get it and boy it’s amazing! Dependency management is a non-issue now which used to be a massive headache before.

Bazel runs on every push to the monorepo and builds the entire repo. A big part of Bazel’s performance comes from its build cache. However the cache can get very large and on container-based cloud CI environments like CircleCI, sometimes just uploading and downloading the cache for each build from storage can negate any performance gains. To solve this, we used self-hosted runners connected to CircleCI. This way the Bazel builds would execute on our boxes which would have the build cache locally stored already. It does mean that these runner boxes would have to stay up at all times and we cannot take advantage of the dynamic scaling features of CircleCI but that was a tradeoff we were willing to make.

The switch to monorepo and Bazel has been a game-changer for us. We cannot imagine coding without a monorepo now.