Java or Bot?
-
We have a server at work running a big Java application and the Java runtime process is hogging all the CPU time. It doesn't do this on a different server. I've only ever heard bad things about Java, but I wonder if anyone here has seen such behavior from other Java processes? There is also suspicious network activity, with about 20 Mbps constant going in and out of the box. The administrator of the box thinks that's normal server network activity, but I'm worried it might be a bot node. Maybe even part of the network that attacked CP! :sigh:
The difficult we do right away... ...the impossible takes slightly longer.
-
We have a server at work running a big Java application and the Java runtime process is hogging all the CPU time. It doesn't do this on a different server. I've only ever heard bad things about Java, but I wonder if anyone here has seen such behavior from other Java processes? There is also suspicious network activity, with about 20 Mbps constant going in and out of the box. The administrator of the box thinks that's normal server network activity, but I'm worried it might be a bot node. Maybe even part of the network that attacked CP! :sigh:
The difficult we do right away... ...the impossible takes slightly longer.
Time to dig out the network analysis tools. Where's the connections going to/coming from? Is the IP port one that would be expected (e.g. HTTPS), or is it something unexpected (e.g random port number at both ends). If you can capture the packets and they're not encrypted, does the data look like what you'd expect?
"A little song, a little dance, a little seltzer down your pants" Chuckles the clown
-
We have a server at work running a big Java application and the Java runtime process is hogging all the CPU time. It doesn't do this on a different server. I've only ever heard bad things about Java, but I wonder if anyone here has seen such behavior from other Java processes? There is also suspicious network activity, with about 20 Mbps constant going in and out of the box. The administrator of the box thinks that's normal server network activity, but I'm worried it might be a bot node. Maybe even part of the network that attacked CP! :sigh:
The difficult we do right away... ...the impossible takes slightly longer.
Richard Andrew x64 wrote:
The administrator of the box thinks that's normal server network activity
Were they break out in a sweat when they said that? :laugh: A network analyzer would be the thing to use here, the capture need not be very long at 20Mbps.
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment "Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst "I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
-
We have a server at work running a big Java application and the Java runtime process is hogging all the CPU time. It doesn't do this on a different server. I've only ever heard bad things about Java, but I wonder if anyone here has seen such behavior from other Java processes? There is also suspicious network activity, with about 20 Mbps constant going in and out of the box. The administrator of the box thinks that's normal server network activity, but I'm worried it might be a bot node. Maybe even part of the network that attacked CP! :sigh:
The difficult we do right away... ...the impossible takes slightly longer.
Assuming it's not a rogue bot/application on the server...
Richard Andrew x64 wrote:
It doesn't do this on a different server.
I'm not a fan of Java, but if this is the case then my first line of attack would be to assume it's not the JRE or application then (necessarily). It may be a bug that only surfaces with a certain configuration / environment, but it's safe to assume the application is "working" (ish). So, I would start looking at environmental and configuration differences to see what the deal is. Are these machines in different subnets? Are you certain they are configured exactly the same? Was this a manual server provision (and open to mistakes) or automated? Is it the same exact OS with the same exact kernel version? Is the room's ambient temp the same for both servers? Do you know for certain it's the same exact application version and runtimes on both machines? Is one server in a faraday cage (kidding, well maybe not)? And so on...
Jeremy Falcon
-
Assuming it's not a rogue bot/application on the server...
Richard Andrew x64 wrote:
It doesn't do this on a different server.
I'm not a fan of Java, but if this is the case then my first line of attack would be to assume it's not the JRE or application then (necessarily). It may be a bug that only surfaces with a certain configuration / environment, but it's safe to assume the application is "working" (ish). So, I would start looking at environmental and configuration differences to see what the deal is. Are these machines in different subnets? Are you certain they are configured exactly the same? Was this a manual server provision (and open to mistakes) or automated? Is it the same exact OS with the same exact kernel version? Is the room's ambient temp the same for both servers? Do you know for certain it's the same exact application version and runtimes on both machines? Is one server in a faraday cage (kidding, well maybe not)? And so on...
Jeremy Falcon
Jeremy Falcon wrote:
I'm not a fan of Java, but if this is the case then my first line of attack would be to assume it's not the JRE or application then (necessarily).
I agree here. It uses Java 8, which I have been reading is very well tested and patched. But also, I'm sure there are still vulnerabilities.
The difficult we do right away... ...the impossible takes slightly longer.
-
We have a server at work running a big Java application and the Java runtime process is hogging all the CPU time. It doesn't do this on a different server. I've only ever heard bad things about Java, but I wonder if anyone here has seen such behavior from other Java processes? There is also suspicious network activity, with about 20 Mbps constant going in and out of the box. The administrator of the box thinks that's normal server network activity, but I'm worried it might be a bot node. Maybe even part of the network that attacked CP! :sigh:
The difficult we do right away... ...the impossible takes slightly longer.
The Windows Event Logs might show you something. It shows (bot) login attempts (Security log) and errors (Application log). In my experience, "all" public servers get "probed" 24/7.
"Before entering on an understanding, I have meditated for a long time, and have foreseen what might happen. It is not genius which reveals to me suddenly, secretly, what I have to say or to do in a circumstance unexpected by other people; it is reflection, it is meditation." - Napoleon I
-
We have a server at work running a big Java application and the Java runtime process is hogging all the CPU time. It doesn't do this on a different server. I've only ever heard bad things about Java, but I wonder if anyone here has seen such behavior from other Java processes? There is also suspicious network activity, with about 20 Mbps constant going in and out of the box. The administrator of the box thinks that's normal server network activity, but I'm worried it might be a bot node. Maybe even part of the network that attacked CP! :sigh:
The difficult we do right away... ...the impossible takes slightly longer.
I have 10 years at least each in Java, C# and C++. There is nothing wrong with Java. Nor with the other two. What you are describing is what I would generally diagnose as an environment problem. But could be a data caused problem in that a message(s)/request(s) that should have run to completion did not and now it is just spinning. Restarting it would likely demonstrate that. That however would also indicate a likely programming bug. Sufficient execution flow logging, in any type of server application, would allow you to diagnose execution flows running out of control.
