Optimizing remote testing
The last article described how to optimize the prepare phase of kitchen testing. This post will describe methods to optimize the kitchen converge step.
The current duration for each stage.
|Stage||Duration (seconds)||% Total|
|Converge||10 minutes 14 seconds||95%|
- Remote testing in Bluebox with Chef
- Optimization overview of Bluebox with Chef
- Optimizing boot
- Optimizing create
- Optimizing prepare
The converge stage of test kitchen executes the specified chef recipes on the test node. This stage should, in most instances, take the longest. Converge has two distinct segments:
- Cookbook resolution
- Chef run
This stage resolves all of the cookbooks and their dependencies. Depending on the number of dependencies this may consume 60 seconds or ore of the converge stage. I may attempt to optimize the segment at a later time, but for now I have not performed any optimization.
Chef run optimization strategy
It is not feasible to cover every potential optimization path for all cookbooks as those optimizations are heavily dependent on the cookbook’s goals. This post will provide generic optimization recommendations.
I recommend using fpm-cookery to create system native packages for everything installed by Chef. This includes repackaging of complex binary installers, arbitrary small binary files like security keys, and system packages. Basically, anything that is not a configuration file should be contained in a system package. I recommend this approach multiple reasons:
- A system package is often faster than the a binary installer or shell installer, which sometimes need a JVM to start, etc.
- System packages prevent having to write complex custom guards to determine if actions are necessary.
- On certain platforms, after the initial caching of binary package data, Chef can quickly make decisions if a package is installed at the desired level.
FPM Cookery Example
Below is a quick example of using FPM cookery to build a system package for a product that initially required a complex Chef resource to manage. The examples consists of two parts, a vagrant file for creating a virtual machine and the fpm-cookery recipe.
Below is a sample Vagrant File to use with FPM Cookery
The vagrant file creates a new RPM for both RHEL 6 and 7 platforms. The script definition at the top calls FPM cookery to create the RPM. The vagrant configuration iterates over both the RHEL 6 and 7 platform, creates a new virtual machine, links the appropriate directories, executes two sets of chef recipes. The ei-build cookbook recipe that installs most dependencies for building images and a java installation recipe which is required for this particular build. After the dependencies are installed, the fpm_script defined above is invoked.
FPM Cookery Recipe
Below is an example FPM Cookery recipe
This is a multi-platform recipe used to create the RPM for the Urbancode Deploy Agent. FPM Cookery, and the underlying FPM application abstract away the usually painful process of creating packages. Above is all the instructions necessary to build the package. The recipe has a few distinct sections. The first section is a DSL in FPM-Cookery that describes information about how to make the package and metadata about the package. The source section, tells FPM-Cookery to grab the file in that path and uncompress it. The omnibus directives tell FPM-Cookery to scoop up the contents of the specified directory and put them into the package. The name through depends identifiers are metadata that is passed onto the package itself. Next, FPM-Cookery calls the build and install methods. The build methods changes the permissions on files. The install method prepares a properties file to be used with the installer and then executes the package installer. When complete, FPM-Cookery calls FPM to package up the contents of /opt/ibm-ucd-agent and the package is created.
It only takes a handful of lines of code to repeatably create packages.
What are the benefits of this method? Our environment uses UrbanCode deploy on every node. Every Chef role converge would need to install this package. Running the install above via a chef resource took about 30 seconds, the equivalent yum install takes less than 2 seconds. Some of larger packages like IBM Tivoli Monitoring went from 5+ minutes to under 50 seconds.
Many recipes will install packages, for RedHat based system Chef will use YUM to install the requested packages. The first thing YUM will do is download all YUM repo data and create a cache.
That operation can take almost to 15 seconds to complete.
The method below optimizes cache creation by taking advantage of the fact that the system is running while test-kitchen is uploading cookbooks in the prepare stage. This can be performed using the
at command inside of cloud-init. This method results the command immediately exiting and running the job in the background. If
at was not used, cloud-init would delay the boot process until the YUM cache was created, negating any potential gains.
The cloud init configuration looks like this:
This update to the cloud-init file saves about 15 seconds for every converge.
Package optimization and yum-cache enhancements provide a measurable performance improvement in the converge phase:
|Stage||Original Duration||Original % Total||Optimized Duration||% Improvement||Optimized % total|
|Kitchen Converge||10 minutes 14 seconds||95%||6 minutes 59 seconds||31.76%||92%|
15 seconds is removed from every converge by using the cloud-init optimization for YUM cache creation. Using packages with the fpm-cookery method saves the remainder of the time.
Current stage times after converge optimization:
|Stage||Duration (seconds)||% Total|
|Converge||6 minutes 59 seconds||92%|
Next up for optimization? Verify.