JProfiler Get rid of your performance problems and memory leaks! Modern Garbage Collection Tuning Java Performance Tuning, 2nd ed The classic and most comprehensive book on tuning Java Java Performance Tuning Newsletter Your source of Java performance news. Subscribe now! Enter email: JProfiler Get rid of your performance problems and memory leaks!

Hotpatching a Java application allows you to fix arbitrary code-level problems in a running application without terminating that application. Here Jack Shirazi covers the basics of how to hotpatch a Java 6 application

Page 1 of 2
next page: Finishing the hotpatching process

Last month the Java Performance Tuning newsletter special on Java 6 lead to an interesting discussion between myself, Kirk Pepperdine the JavaPerformanceTuning.com and TheServerSide columnist, and Heinz Kabutz, the author of the excellent Java Specialists Newsletter, about how to monitor things using the new dynamic attach facility available in Java 6 we announced in last month's newsletter.

During the discussion, we realized some interesting things become possible with the combination of features in Java 6, including the rather amazing capability that you can patch a running Java process without stopping it. In this article, I'm going to walk through the basic procedure to show you one way how this could be achieved. I'll use a simple program which adds and deletes objects to a stack to illustrate the technique. My program adds and deletes objects at the same average rate, and queuing theory indicates that such a queue is potentially unbounded. The example is specific, but the technique I'll use to monitor and fix my app is generic. To keep the code as readable as possible, I've defined everything in the default package and used the current working directory as my classpath in all cases. All code, with unix & windows batch files, can be downloaded here

Initial Diagnosis

My process is up and running, and there is something wrong. I'm not getting the output I was expecting. My app is supposed to be reporting the first ten elements in the stack every once in a while, and I was expecting to see a variable number of elements listed, but I only see one element in the stack every time I get the report.

Figure 1: Only one element is being output

For an abitrary application I would follow standard practice and turn up the debug level in the area of functionality that I think should tell me what is happening, and I would also generate stack dumps (send kill -3 to the process in linux/unix or type cntrl-\ in the Java command console in unix/linux or cntrl-break in the Java command console in Windows or generate from jconsole). That is the normal technique to narrow down the problem, but in this case even if I do apply those techniques it doesn't help me to narrow down the problem - and that is not untypical for many problems I've seen in production.

public class Main {
    static Stack Pile = new Stack();
...

In this case, the Stack that I'm reporting is actually a static object in my Main class. If only I could look at the elements in that stack. Maybe my understanding of queuing theory is all wrong and there should only be one element in the stack. Of course I can kill the process, recompile it, and dump the whole object next time round, but that is not the aim of this article. We want to see how to examine the object without having to stop the process.

Attaching To The Process

So how do I dump and examine that stack? First I need to attach to my process. The basic technique for attaching to a Java 6 process and executing code is detailed in Using Mustang's Attach API (A. Sundararajan). I'll detail it again here though. First I need an "attacher" process - that's straightforward:

public class Attach {
    public static void main(String[] args) throws Exception {
        // This program accepts two parameters:
        //    1. The pid of the JVM on which we want to load an agent
        //    2. The full path of the agent jar file to be loaded

        //Attach to the target Java process
        VirtualMachine vm = VirtualMachine.attach(args[0]);

        // load a specified agent onto the JVM
        vm.loadAgent(args[1], null);
    }
}

Given the process ID of the target process to attach to, a single call attaches us to the process - VirtualMachine.attach(PID). Note that VirtualMachine is Sun specific class in the com.sun.tools.attach package, so this code only works with the Sun JVM, other JVMs should have equivalent attach methods.

The only tricky bit to remember about the attacher is that to compile and run it you need to include the tools.jar from the java distribution in the classpath (e.g. from %JAVAHOME%\lib\tools.jar or $JAVAHOME/lib/tools.jar).

Once attached, a second VirtualMachine method call loads our agent, and we are into our own code from there. That's all there is to attaching and starting our agent!

Getting State

What I need next is the actual agent that will execute inside my process. This can be any class, I just need to keep in mind the entry point - when the class is loaded into my application it will get its agentmain() method called. The basic technique to access the object is some simple reflection. The static I'm after is Main.Pile. To access that using reflection is easy enough: Class.forName("Main").getDeclaredField("Pile").get(null).

So my agent class is actually quite simple (I just make it defensive with a generic try catch just in case I've made a mistake somewhere - I don't want my investigation to unintentionally crash the app):

public class DumpPile {
    public static void agentmain(String agentArgs, Instrumentation inst)
    {
        try
        {
            System.out.println("Dumping Main.Pile ...");
            Object o = Class.forName("Main").getDeclaredField("Pile").get(null);
            System.out.println(o);
            System.out.println("Pile.size() == "+ ((java.util.Stack)o).size());
        }
        catch(Exception e)
        {
            System.out.println(e);
            e.printStackTrace();
        }
    }
}

Running this against my server results in the stack getting dumped to stdout, and I'm not really surprised to find that my stack is much bigger than one element! The subsequent output line from my agent of "Pile.size() == 124" confirms that the list is significantly more than one element.

Figure 2: We get the dump of the Stack

Page 1 of 2
next page: Finishing the hotpatching process