Performance and Security are tightly interconnected. During one of our regular performance checks, we reviewed the four golden signals, such as
Response times
Request count
Error rate and Exceptions
Utilization
Everything was within the acceptable baseline, except there were some suspicious exceptions. We've zoomed in by following the Trace from the user's click to the app and database layer.
The Trace contained the following error:.
The request was rejected because the URL contained a potentially malicious String "//"
Sample Trace we've investigated:
Below a trace from #Dynatrace
Topology
Service Tomcat/localhost
Request /remote/fgt_lang
Host GBM
Process SpringBoot
Operating system Linux - Ubuntu 22.04.1 LTS (Jammy Jellyfish) (x86)
Bitness 64-bit
Service type Web request service
Service main technology Apache Tomcat
Technology MongoDB Client (driver-core-only, QOS Logback , Java , Apache Tomcat
Metadata
URI ///remote/fgt_lang?lang=/../../../..//////////dev/
HTTP method GET
Response status 500 - Internal Server Error
Context root /
Server name Tomcat/localhost
Client IP xxxx
Thread name https-jsse-nio-8443-exec-28
Thread ID 81
PID 1606829
Request headers
host 54.202.133.52:8443
user-agent python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-1160.el7.x86_64
Trace ID 9f852cdba55482bd4f7232a31d194438
Span ID 4414e26be16058e1
Message:
The request was rejected because the URL contained a potentially malicious String "//"
Show affected entry in 'Code level' tab
Stacktrace:
org.springframework. Security.web.firewall.StrictHttpFirewall.rejectedBlocklistedUrls(StrictHttpFirewall.java:535)
org.springframework. Security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:505)
org.springframework. Security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:196)
org.springframework. Security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1787)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:833)The stack trace explained that this was a cyber attack on our application. Our spring boot http firewall caught this problem.
More Details about this issue CVE-2018-13379
Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal')
https://nvd.nist.gov/vuln/detail/CVE-2018-13379
Lessons learned
Be open-minded and investigate exceptions
Performance tracing helps to find security attacks
Security and Performance teams should work hand-in-hand
I wish you happy Performance & Security Engineering!