The scenario is very well known. You've prepare your load and performance tests for several days, execute the test, and realize that you've tested an old version of the application under test.
Such tests on outdated applications or configurations are a frustrating experience for both you as the consultant and the client for wasting his money.
How do you manage and control your load and performance testing environment?
There are hundreds of changes in such complex systems, and you might have no chance to control all the layers manually. Complexity increases if different teams are configuring, deploying, and changing their services on shared testing environments. They deploy services in an automated fashion and have a change record in their deployment solution. Are they also tracking configuration changes such as JVM arguments, Thread pool sizes, or Database connection pool configurations?
# Infrastructure. Load and performance test results will be different if your infrastructure has changed. It would be best if you consistently track the hardware and network configuration of all your tests. In complex environments, such tracking won't work, and an automated solution is needed.
# Application. Shared testing environments are standard these days. On the development and testing stages, multiple engineers have access. How do you control what version they've deployed and was up & running during your load and performance test?
# Configuration. During software development and testing, configurations of your applications under test change frequently. For example, your developers might enable some debug log levels, and your testers check how the services are working with a smaller heap size. All of these changes will impact your load and performance tests. So how do you track and report which configuration was in place during your test?
According to research, an average application consists of 80 services and hundreds or even thousands of configuration settings. Even a very experienced engineer can't control all these settings manually. I don't trust all the manual tracking of who deployed which configuration on what environment, and I assume that auditors and regulatory agencies are either.
If complexity is too high, an automated solution comes to the rescue.
Managing and controlling test environments is much more than maintaining a tracking list of software artifacts alone. It involves handling the entire stack from client, application, network down to infrastructure. All these elements play a crucial role when you prepare and execute your load and performance tests. You won't end up blaming yourself for running your test on the wrong application or its underlying configuration.
In one of our recent projects, we've used the automated change detection solution Versio.io to track and report the configuration of our customer's environment. Luckily it saved us several times from executing a load and performance test on the wrong application configuration.
Its automated detection and historicization of all application, network, and infrastructure changes saved us a lot of time and improved the quality of our test documentation. We've linked the environment report from Versio to every executed test and created a perfect summary report. Our client appreciated the professional work and planned to use Versio to track and detect all changes in his production environment.
The screenshot below demonstrates how version detects and highlights deviation from the target configuration.
Your Takeaway?
Automated change detection is crucial for your load and performance testing activities because it saves time, provides transparency, and ensures that you run the tests on the correct application configuration.
We are a Versio implementation partner and are happy to support your automated change detection projects.
Keep up the great work! Happy Performance Engineering!