At the heart of every CI/CD process there is a build that can either make or break it. Your CI/CD process will be only as good as the build it support and any improvements that you want to add to it will have to begin at the build level. Many times the build processes is handed down from generation to generation and people continue using it without a second thought. People also try to build their process on top of existing build process without thinking of improving the current builds. Very rarely do you get the opportunity or the luxury of creating a build process from scratch. Here are some do-s and don’t-s that can help you engineer a good build process.
Build a Framework
Instead of creating builds, plan a build framework right from the start. Even if you don’t plan to build all the components of the framework right from the start, have a clear idea of what you want to do. Then start building a framework for the same instead of configuring individual builds. Your typical build will have one or more build file, one or more kind of automated tests, configuration for one or more environment, configuration for continuous integration and possibly continuous deployment. To do all of this, the more manual steps you have to take, the more lacking your framework is. Instead create a framework that will bootstrap and on board any project and any new development effortlessly. The framework should generate a build file, configuration, tests mock ups etc for your project. Start with just the build file generation first then continue to add the bells and whistles. This will also standardize all your builds, their output, their execution and save you a lot of time and effort in the long run.
Modularize
Avoid big monolithic builds at all cost. Break down your builds into small bite sized modules that are easy to build quickly and often. Arrange your modules in meaningful hierarchy and according to their dependencies. Build all your dependencies within your project at the start of the build and look out for circular dependencies. Ensure your dependencies are built only once and right at the start. Make sure your build can be called with and without the clean parameter for quicker feedback. Know every path through every module that your build can possibly take and support documentation on them.
Create a Binary Repository
For all your external dependencies, create a binary repository. Ensure all your developers are pulling binaries from a single authoritative source to avoid surprises. Also check in all your output binaries into your binary repository. Never check in binaries into your Version Control System. That is just bad practice. Also make sure any addition to the binary repository goes through a review and approval process. That means only an automated build process should be able to check in binaries to the binary repository and all binaries in it should be certified.
Embrace Duplication
Don’t be afraid to have multiple builds for multiple purposes, specially if you have continuous integration in place. You can have a build for each module that runs compile plus unit tests for every check in for every module without running clean just to catch compile errors. One that runs clean plus compile plus unit plus integration tests every hour on a different server. And one that runs clean plus compile plus integration tests plus code quality plus code coverage every night which would take hours to run on all modules. That is my current setup BTW. You can customize to your needs.
Builds Must be Reproducible
What that means is, give just a build number, you should be able to reproduce any build on demand without doing anything special. What this also means is your whole process in general should be well documented, well established and reproducible. This means your build environment, build process, binary repository and any other meta data should be well documented.
Have Absolute Tractability
All components of your build should be traceable back to their source at any point. Depending on your framework, language, type of build etc. you may have to insert metadata in your binaries at build time. You will probably also have to insert metadata in your Version Control System in form of tags to establish links between code and builds. Make sure these links are built and maintained. Make sure you insert other metadata into the build, perhaps in form of a manifest, such as versions of binary dependencies, compiler version etc.
Gather Build Metrics
Builds are not just about compile and packaging. I have seen most people stop at writing unit test. Go beyond writing Unit Test. Write as many meaningful metric gathering tasks in your build as you can. Again if you are hooking up your build to a CI, have your CI gather these metrics and track them over time. Pay close attention to these metrics and make use of them.
Build Once, Deploy Many
Ensure all your builds come from a single authoritative source, hopefully from a CI and are getting deployed in a good fashion hopefully by a CD system. The first step to build once, deploy many is to separate out your configuration and code. Never do a build in every environment you wish to deploy. As a good practice servers that run binaries should not have compile tools installed at all. Decouple your configuration all together from your code, even from your Version Control System. This way a configuration change won’t trigger a build and you won’t have to push a new binary for a configuration change. Have a separate pipeline for your configuration changes.
Document your build process
There is nothing like gazing at someone else’s build file and wondering what was going on in their mind when they wrote it. So for the love of Zeus, please document your build process.
So how do your builds work ? Are there any best practices you follow ? Let me know in the comments below.