A previous article described all the reasons to test remotely using Bluebox. This post is going to focus on how to create a systematic approach to optimizing the test times for remote testing in bluebox.
When forming an optimization plan it is important to have an end goal. Determine, the acceptable time for this operation to complete. The goal time will impact the major focus areas for optimization and how the optimization is approached.
After discussion with a major stakeholder of our project, the following non-functional requirement for performance was selected.
I need to test across all sponsorship event websites across all suites in 5 minutes.
There are 6 sponsorship websites, with 3 suites (test, pre-production and production) each. Therefore, the system must support running 18 tests in under 5 minutes without compromising test integrity.
From the previous blog post, the time with no optimization for a single test was 18 minutes and 8 seconds.
Taking 18 minutes 9 seconds to 5 minutes, is a 72% reduction in test time. This is a very aggressive target, will require some thorough optimzation.
Bluebox/Openstack can run tests concurrently, however I anticipate hitting some scaling issues with the tooling when attempting to run 18 concurrently.
With a goal set, the next step in optimization is to break the problem down into different chunks that can be independenatly optimized.
There are 4 main stages to optimize during a kitchen test
Create - Creation and startup of the image to test against
Prepare - The setup duration between create and start of the converge
Converge - This is the cookbook converge time
Verify - This is how long it takes to run the tests
Because the optimization goal is so aggressive, iteration of improvements over these stages may be necessary. Tackling the lowest hanging furit from each stage until the goal is met or it is determined that no further optimizations may occur without impacting test validity.
Before starting an optimization journey, it is important to measure the starting point for each area to keep track of improvements.
Creating the image takes 1 minutes and 22 seconds.
Prepare is part of the converge process in test kitchen and is measured as the time between the start of the kitchen converge command and the execution of the first cookbook. There is not a distinct command for test kitchen, therefore it will be measured by looking at the log.
Prepare takes 7 minutes and 36 seconds.
The converge is measured by looking in the log and finding the log statement that records the duration of the chef client run.
Converge takes 10 minutes and 14 seconds.
Verify takes 11 seconds.
This table summarize the test kitchen stages, we will refer to it throughout this series as we optimize our test times.
1 minute 22 seconds
7 minutes 36 seconds
10 minutes 14 seconds
The approach we will take is to optimize the testing process in the order of the kitchen stages. Next up, will be create optimization.