Tuning applications for speed, scalability, and stability can be a difficult task. Chances are high that you end in a never-ending trial and error endeavor. In this post, I will share key insights about how T24 TAFJ performance tuning should be conducted.
In one of our recent projects, a Bank approached us to solve some of their most critical application performance problems in T24 TAFJ.
Mastering such tasks requires a holistic approach, which includes
Interview of T24 operational team
Close gaps in the Monitoring stack
Performance analysis
Setup T24 testing stage
Baseline test
Implement tunings
Benchmark test
Deployment of tunings on Production
Create a culture of knowledge sharing and continuous improvement
Based on our experience some key performance problem areas of T24 TAFJ are
Locking
SQL response times
Not adequate Infrastructure
TSA Job configuration
COB
JVM configuration
Separation of concern
Slow BrowserWeb requests
Where to start?
One of the best starting points is to meet the current T24 operational team and talk to them about existing difficulties. These teams are very open and share their experience. Response time problems, system crashes, incident details, and insights about a usual working day help you to get initial ideas.
The out-of-the-box monitoring which is part of T24 TAFJ does not provide insights to all major hotspots. We've experienced situations in which it caused memory leaks and additional overload on T24 itself. Usually, we disable it and introduce a modern full-stack and AI-powered monitoring solution such as Dynatrace. Once the OneAgents are deployed, a detailed user journey is captured, and we can trace every transaction in T24.
As a third step, we start the performance analysis. The needed dashboards are implemented, some service metric configuration and service mappings will be adjusted and after a few hours as the regular users are browsing through T24 we get detailed insights about the key issues in this Core banking system. Ensure that the performance analysis will take place for about one month because some COB jobs might do not be executed every day.
Review the findings with the T24 database, infrastructure, and business teams and share tuning recommendations. Problem analysis in such a complex core banking suite can be overwhelming. To avoid an endless search for the needle in the haystack, it's a good idea to focus your efforts on a standardized approach such as
Horizontal analysis first
Vertical analysis last
But, what does this mean, and how it should help in such a core banking performance analysis task?
Things are easier than we think. Start at the end-user layer and find out long-running requests, failed requests, or throughput issues. Once such problematic items have been identified, you can visualize the service flow and understand at which layer the investigations should be focused. For instance, long-running requests in T24 can have several root causes. If you do not follow the service flow, you can easily end up on the wrong layer without being able to pinpoint the actual problem spot.
Once you have a good idea about hotspots in the T24 core banking solution, you should apply the needed improvements on a testing stage. Ensure that the sizing and the data used for testing are not entirely different from your production.
Run a workload modeling in production and find out the most important and performance-critical transactions. These are good candidates for your baseline and benchmark performance test. You can use one of the major open-source or commercial load testing tools to mimic these key transactions under regular and peak business volume conditions. Before you execute these tests, it's a good idea to double-check if the full-stack monitoring for the testing stage is in place as well.
Baseline - Tuning - Benchmark
Once the load testing scripts has been developed, you can run regular and peak-volume business day performance test. The former stimulates the requests during regular business hours, and the later ensures that the system can handle a 3-5 times higher peak volume as well.
Apply the improvements on your T24 stack and repeat the load tests. These Benchmark results are compared to your earlier performance baseline. Apply additional tunings if the improvements are not within the expected range. If the performance targets have been achieved, it's time to deploy the new code to the T24 core banking production.
Deployment of the improvements
We are often in a hurry and can't wait until tunings are rolled out to all users. This is fully understandable, but there can be situations in which an iterative approach is even better. I've made good experience with more frequent but smaller changes. The benefit is that you can understand the performance impact even better in an iterative tuning deployment approach.
It's a wise idea to create some monitoring cockpits which highlight key performance indicators before and after the tuning. Such dashboards provide evidence that the tunings have improved the performance of the entire system. Don't think too short and consider all the layers from the user to the infrastructure.
In our next blog post, I will share insights on some performance anti-patterns in such a T24 core banking system.
Keep doing the great work. Happy performance tuning!