DevOps is cutting the lines between production and software development. This modern approach to continuous delivery, integration, and deployment is the answer to quick time to market needs. Testing in a DevOps world is often an afterthought. Those who tried to apply traditional testing methods in DevOps gradually failed because of not enough time for test preparation and execution. In this post, I will share some ideas how testing could become a fundamental part of your next DevOps project.
Shift left / testing for the expected
You’ve probably heard from this idea to “shift testing to the left.” It’s about early and continuous testing throughout the application lifecycle. Your testing teams review the requirements, document the test cases along with the expected outputs, execute tests manually or automated and create the test evidence reports.
Things to consider # Time for test execution is limited: Short release sprints reduce the time to execute testing suites massively. We often have just one day or less to execute our regression sets. Risk-based testing and automation help to test the right things even in limited time slots.
# Feedback loops: DevOps testing pays high attention to sense and response. You won’t have time to review all test results in detail, create an excellent test report and after a few days present your test summary. Being able to identify failed test cases and report those failures back to dev teams immediately is indispensable.
# Flexibility: Quick adaption of testing suites is critical. Projects in DevOps mode will deploy new releases frequently on testing stages. In your role as test manager or engineer, you have to find a way how to deal with such permanent changes in your system under test. A flexible automation solution and framework can help you to overcome such challenges.
Shift right / testing for the unexpected, undefined, unknown
In the case of “shift right” testing moves towards production. Load Tests, reliability checks, user experience tests or performance checks are executed on pre-production or production. A characteristic of all of those tests is that expected outcome is often unknown. Real customer requests can drive your test cases, and you investigate how the new application behaves, or you simulate a massive production like the volume on your system under test. Focus your analysis on the undefined and unexpected such as memory leaks, deadlocks, concurrency problems, exceptions and misbalanced communication patterns.
Things to consider # Be creative: I can’t give you a recipe which works for all your applications. Shift right requires an open mindset, passion for data analysis and the vast proportion of creativity. Your success in this stage depends on how long it takes to identify hotspots and find solutions. Experience and the right tools can be a shortcut.
# Monitoring of all user requests: You won’t have enough time for failure reproduction. Integrate APM and UEM solutions which captures all user actions performed on your system under test to avoid time-consuming error reproduction. # Reduce the risks: Don’t release the new solution on day one to your entire user community. A technical go-live followed by a friend and family phase is often the preferred approach to balance the risk of reputational damage due to serious reliability issues.
# Review and improve: You built beautiful apps and ensured high reliability. Once the user and data volumes grow, response times can decline customers will move away due to the wrong user experience. Make regular performance reviews on production a fundamental part of your operational procedures to identify such slowdowns and initiate corrective measures.
New testing techniques
Learn from mistakes and improve your testing procedures day by day. Creative minds and forward-thinking leaders release new testing utilities and practices regularly. Schedule a slot in your weekly agenda for research and continuous optimization. Below are some ideas how testing at the edge of DevOps could be improved. It’s beyond the traditional approach of testing for the expected, but it enriches classic testing.
# Real user requests Time is limited in DevOps projects and therefore teams tend to deploy new versions early on production even if not all functional test cases have been executed. Real users will use the new application in different ways with real data sets. Depending on the design of your new system, you can grant access to a limited user community and watch how those users work with your new application. Failures identified in this stage can give valuable input for your developers and help them to fix such problem spots before a broad user community gets affected. Furthermore, you should check if response times are within expectations and compare them with your non-functional requirements.
# Driving tests with production requests Certain applications such as message queue-based applications often have heavy throughput requirement. Adequate load tests are time consuming and therefore creative minds store messages on production and replay those on their testing stages. The same testing with production requests can be achieved with duplication of user requests by using proxies.
# Static Scans Quality is an important selling factor. You should no longer rely on manual or automated tests alone because both can’t tell you the truth. Certain areas such as dead code, complexity, or vulnerabilities require a static analysis. Extend your testing suite towards static secure code analysis according to security standards such as OWASP top 10.
Keep doing the good things. Happy Testing!