If you want to waste money, use the classic performance testing approach.
What is the Classic Performance Testing approach?
Tools first
Last-minute load tests
Add more hardware
Customer identifies issues
Firefighting
What is Level 5 Load and Performance Testing?
There is a saying, "You can't teach an old dog new skills." Learning can become tricky as we age, but we should never miss out on learning new skills.
# NFRs and Workload modeling
Some load and performance testing practices developed 30 years ago are still relevant, but others have become outdated. For instance, how we specify requirements or perform workload modeling is the same. Unfortunately, there is little tool support for workload modeling and crafting meaningful performance requirements, but these techniques have been established in the last century and are very useful.
# Craft meaningful performance requirements
Specifying response times without expected transaction rates and concurrent user volumes does not help. Instead, a meaningful performance requirement must include the four golden signals: response times, throughput, error rate, and infrastructure boundaries. Ideally, you make them part of your system design documents to allow upfront planning, implementation, and testing for high performance.
# Invite performance engineers to architecture reviews
Performance is a team sport! It takes decades to become an experienced performance architect, and the lessons learned they carry are of tremendous worth to your organization. You can compare their work to chefs in a delicious restaurant. They mentor team members, find problems quickly due to their experience and tell you why systems can't fulfill performance requirements before implementation.
# Conduct workload modeling for as many layers as possible
No need to postpone all performance engineering activities until the system is testable - You can follow the workload modeling approach to calculate concurrency, throughput, and latency limits layer by layer.
# Setup performance monitoring and transaction tracing
If we drive at night, we turn the lights on to see where we are heading. Isn't load testing quite similar to driving a car? Shouldn't we have similar security measures, such as monitoring during a load test execution, in place? I always follow a monitoring-first approach because APM or performance tracing can point me to the root cause of performance issues.
# Specify the load and performance testing approach
Can you build a house without a plan? Without proper planning, it would soon break down, and your investment would be lost. Instead of following a try&error approach, you should design a load test that validates your regular, peak, stress, and durability requirements.
# Developers validate performance requirements
Shift Left is famous for a good reason. Early identification of performance problems saves your teams a tremendous amount of time. Developers quickly identify bottlenecks when they know performance requirements and validate them in the development stages. As a result, you will deploy better software quality to higher stages and reduce the defect resolution effort significantly.
# Establish automated performance gates
Load and performance testing is not a one-time exercise. When your requirements, application, or infrastructure changes, you must re-run the load and performance test. In addition, it's challenging to track all your changes in large environments. This is why we use pipelines to execute a load test after deployment and validate the performance requirements automatically.
# Block release, fix issues, and eliminate technical debt
If your performance gates are validated, you must have the guts to block a release from deployment to production. The same is true for technical debt. Workarounds for performance problems are not acceptable. Your performance gate must turn green, and you must have the backup of your management team no matter what.
# Continuous performance feedback
Load and Performance testing does not end in non-production stages. A successful performance approach also involves performance checks and performance feedback from production. You can only provide your valued customers with excellent user experience and reliable business applications if performance becomes everyone's matter.
Final thoughts
Based on my experience a level 5 load and performance testing approach requires also a culture of discipline, disciplined thought and disciplined action.
Keep up the great work! Happy Performance Engineering!