Companies are often not happy with their performance testing suite because it is too expensive, usability is pure, and maintenance efforts are high, and re-use of automated testing scripts is extremely limited. In this post, I will shine a light on essential characteristics of a performance testing platform and give you some guidance how to make your selection.
Over the recent decade, a long list of new open source or cloud based load testing solution provider appeared on this growing market. The need for reliable application is on an all-time-high and organizations spend more on performance engineering than ever before. In some cases, the reason for high investments is simply using the wrong tools. Based on my experience, an outstanding performance-testing platform fulfills those essential requirements:
Ease of use
Deployment, script creation, test execution, and result analysis should be lightweight and intuitive. As a rule of thumb, you should be able to start on a green field, create a script and run a load test within several hours.
Technology support
Capture and replay of the top 3 Browser, Mobile devices, Web 2.0 techniques, Java, Citrix, SAP, and .Net based applications. The list of technologies depends on your technology stack.
Realistic user simulation
Limited user simulation techniques can lead to wrong test results. With the rise of browser based applications, the focus is more on real browser based user simulation. In rare cases, protocol based is also required.
Reusability
Performance testing goes hand in hand with application monitoring on production. It’s a waste of money if you implement those testing and monitoring scripts twice. Your performance testing scripts must be reusable for uptime and health monitoring on production.
Support community
Automating user interactions can be tricky and often requires the help of experienced developers or specialists. It’s essential to get access to a knowledge base and professional support specialists which guide you through automation or tooling issues.
Maintainability
Updating the performance testing platform is often seen as an overhead and brings additional risks such as changes negatively impact further tests. Therefore, close to zero tool maintenance effort is the target.
Costs
Initial license costs are a pain for companies and therefore they often decide to use open source tools. Preferably, those initial spending on performance testing tools must be low. Pay by use is the preferred charging model.
Self-monitoring
Distribute load across injection machines and built-in features to avoid that overloaded machines falsify load test results. It’s highly recommended using a load testing solution which monitors its load agent’s permanent during a test execution, provides health monitoring charts while the test is executed and does never share load generators with other tests.
There are hundreds of load testing solutions on the market and companies have often no idea, which one should be selected, and what decision criteria should be used. Therefore, I’ve created the decision matrix below.
How to use this matrix:
A.) Answer this form for each load testing solution on your short list
B.) Add the appropriate rating for each criterion
4 means that it completely fulfills the criteria
3 means that this is a good fit
2 means that it does not meet the criteria
1 means that it’s far beyond
C.) Calculate the score
Don’t waste time and money with an outdated performance testing platform. The sooner you will start with its replacement, the more value you will gain. Keep doing the good work!