In one of my recent performance engineering projects, I had to deal with a strange monitoring Oxymoron. The issue resulted in a crash of this business application. My lesson learned was: Watch out for monitoring oxymorons.
What is an Oxymoron?
I assume that the term Oxymoron is not very common. However, writers use such terms quite frequently to express something strange with contradictory words.
Often used Oxymorons are:
Virtual reality: Reality or Virtual?
Facebooks privacy: Facebook is not very private?
Organized chaos: Chaos is never organized?
Beautiful Disaster: disaster is not beautiful?
The Problem
My application under test was running without errors when functional testers have executed their manual test cases. Nobody reported a strange behavior or slowdowns. I’ve simulated production volumes on this application several times before, but since the last deployment, something went wrong. Two hours after the load test has started, an out of memory error occurred. The created heap dump has shown that the applications internal monitoring feature holds all the objects from being garbage collected.
Apparently, an application which monitors it’s processing is a oxymoron.
The Solution
The built-in monitoring utility was useful for debugging because it collected call stacks of services and threads. The flip side was that it slowed down processing, caused a higher CPU utilization and resulted in out of memory errors.
We’ve decided to disable the applications built-in monitoring feature and repeat our load test. This test was a success, heap memory utilization was within the expected range and no out of memory error occurred.
Be cautious when a business application provide it’s own monitoring features. Monitoring and handling business services are two totally different things. There are excellent solutions for the former out there.
Comments