This was amazingly impossible to find the answer to when I was looking for it.

The setup — running JVM, some thread doing something stupid like taking 100% CPU usage. For the purposes of my example, I will explain that I was using Jetty to serve a homegrown web application. With an Apache front-end, requests were being passed to Jetty over the AJP13 protocol using mod_jk.

Basically, the result of months of research turned out to be that a misconfigured client browser was passing a “Content-Length” value in the header of the request that was greater than the amount of data that was actually being supplied by the request. That’s stupid, right? What’s stupider is that in this situation, mod_jk winds up passing this crap along to Jetty, and with AJP13, Jetty sits there waiting for more data to come from the connector. But, there is no more data. So, away it would spin… until the service could be completely restarted, causing a service interruption for end-users. This was not acceptable, so before I was able to find what was actually causing the problem, I spent a long time finding a suitable work-around to the bug. The logical answer seemed to be “killing a java thread” — and I honestly thought that somebody else out there in the world had somehow found a way to kill a java thread. Apparently nobody in the mass of all google search results has ever found a way to kill a java thread that is attached to a running JVM, without destroying the entire JVM… at least not that I found.

So, for all of you guys out there that want/need to do the same thing — here’s your solution:

1) Ensure that your java program is started with the following parameters:

-Dcom.sun.management.jmxremote.port=50199 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Xrunjdwp:transport=dt_socket,address=50100,server=y,suspend=n

This will allow us to attach the java debugger to the running process, after we identify which Thread is causing the problem. Also, make sure that you have your iptables setup appropriately so as to only allow connections on 50100 and 50199 from the hosts/workstations that you manage.

2) Identify the offending thread:
In order to identify the offending thread, I used the Java VisualVM utility that is shipped with the JDK distribution. You can also use JConsole, but VisualVM allows you to add in JConsole plugins, and the look/feel is so much better than JConsole, so I use VisualVM. We’ll need the JTop JConsole plugin, which you can find in /demo/management/JTop.jar. I trust you’ll be able to figure out how to get the plugin working…
The thread that is eating your CPU will be at the top of the list. For the purposes of this example, the image below is an infant thread, so the CPU(sec) column reflects only a few seconds worth of actual usage — yours may/probably will be much higher.

3) Kill the thread.
In this example, the ThreadName is “btpool0-0″. Fire up the java debugger (also shipped with the JDK distribution), and attach to the running JVM…

[root@host ~]# jdb -attach 50100

Get a list of the running threads — this will also give us the thread id as the JVM sees it:

Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
Initializing jdb ...
< threads
Group system:
  (java.lang.ref.Reference$ReferenceHandler)0x25c1             Reference Handler                                         cond. waiting
  (java.lang.ref.Finalizer$FinalizerThread)0x25c2              Finalizer                                                 cond. waiting
  (java.lang.Thread)0x25c3                                     Signal Dispatcher                                         running
  (java.lang.Thread)0x25c4                                     RMI TCP Accept-0                                          running
  (java.lang.Thread)0x25c5                                     RMI TCP Accept-50199                                      running
  (java.lang.Thread)0x25c6                                     RMI TCP Accept-0                                          running
  (java.lang.Thread)0x25c7                                     RMI Scheduler(0)                                          cond. waiting
Group main:
  (java.util.TimerThread)0x25ca                                Timer-0                                                   cond. waiting
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25cb      btpool0-0                                                 running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25cc      btpool0-1 - Acceptor0 SelectChannelConnector@0.0.0.0:8080 running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25cd      btpool0-2 - Acceptor1 SelectChannelConnector@0.0.0.0:8080 running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25ce      btpool0-3 - Acceptor0 Ajp13SocketConnector@0.0.0.0:8009   running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25cf      btpool0-4                                                 running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25d0      btpool0-5                                                 running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25d1      btpool0-6                                                 running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25d2      btpool0-7                                                 running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25d3      btpool0-8                                                 running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25d4      btpool0-9                                                 running
  (java.util.TimerThread)0x25d5                                Timer-1                                                   cond. waiting
  (java.lang.Thread)0x25d6                                     PoolScavenger0                                            cond. waiting
  (oracle.jdbc.pool.OracleImplicitConnectionCacheThread)0x25d8 Thread-15                                                 sleeping
  (java.util.TimerThread)0x25d9                                Timer-2                                                   cond. waiting
  (java.lang.Thread)0x25db                                     Thread-18                                                 cond. waiting
  (java.util.TimerThread)0x25dc                                Timer-3                                                   cond. waiting
  (java.util.TimerThread)0x25dd                                Timer-4                                                   cond. waiting
  (java.lang.Thread)0x25de                                     Northgate Server                                          running
  (java.util.TimerThread)0x25df                                Timer-5                                                   cond. waiting
  (java.util.TimerThread)0x25e0                                Timer-6                                                   cond. waiting
  (java.util.TimerThread)0x25e1                                Timer-7                                                   cond. waiting
  (java.util.TimerThread)0x25e2                                Timer-8                                                   cond. waiting
  (java.util.TimerThread)0x25e3                                Timer-9                                                   cond. waiting
  (java.lang.Thread)0x25e4                                     pool-1-thread-1                                           cond. waiting
  (java.util.TimerThread)0x25e5                                Timer-10                                                  cond. waiting
  (java.lang.Thread)0x25e6                                     DestroyJavaVM                                             running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25e7      btpool0-10                                                running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25e8      btpool0-11                                                running
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25e9      btpool0-12                                                running
  (oracle.jdbc.pool.OracleImplicitConnectionCacheThread)0x25ea Thread-756                                                sleeping
  (org.mortbay.thread.BoundedThreadPool$PoolThread)0x25eb      btpool0-13                                                cond. waiting
Group RMI Runtime:
  (java.lang.Thread)0x25ed                                     JMX server connection timeout 51                          cond. waiting
  (java.lang.Thread)0x25ee                                     RMI TCP Connection(2687)-127.0.0.1                        running
>

The thread id that we’re going to kill is “0x25cb”. The first step of killing the thread is to jump into it, and suspend it…

> thread 0x25cb
btpool0-0[1] suspend 0x25cb

Now, here’s the real trick, and the primary component of killing the thread that I was unable to find in all of the search that I had done… Step to the next frame.

btpool0-0[1] step
>
Step completed: <... snip ...>

This puts you in a position to inject a general exception into the running code (should probably be something that is uncaught), thereby exiting the running thread, and cleaning up objects gracefully within the JVM.

btpool0-0[1] kill 0x25cb new java.lang.Exception()
killing thread: btpool0-0
btpool0-0[1] instance of com.site.package.name(name='btpool0-0', id=9675) killed
btpool0-0[1]

Exit the java debugger, and you’re done!

-dan

3 Responses to “Killing a Java Thread”

  1. Wow, this is really, really awesome! Been struggling with CPU hogging threads that never seem to resolve themselves, and the only way to get past them was to do tomcat restarts. Thanks so much for this man, you rock!

  2. Hi Dan, thanks so much for this information: we needed to kill a thread which was waiting indefinitely on a horrid Sybase Anywhere DB and, voilĂ , it worked !

  3. Hello Dan. Thanks for your method. It clarified me a lot of things, but some threads are killed, and then, when exiting and re-entering in the jdb , the thread is here alive again , and all threads have different addresses.
    Anyone to shed some light on that ?

Leave a Reply

(required)

(required)

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

© 2013 Dan's Blog Suffusion theme by Sayontan Sinha