|
|
|
|
Hotpatching a Java application allows you to fix arbitrary code-level problems in a running application without terminating that application. Here Jack Shirazi covers the basics of how to hotpatch a Java 6 applicationPublished February 2007, Author Jack Shirazi
Page 2 of 2
previous page: Starting the hotpatching process
What is next? Well a cursory examination of my reporting object shows that it has a settable "maxCount" variable which determines how many elements to report each time. One obvious explanation of the output I'm seeing is that somehow that has been set to 1, so I'm only reporting one element of the stack each time. However, the technique I used to dump an object in the previous section wouldn't work this time. My reporting object was created in a method and is held by a local variable in that method, and it is not reachable from any class static. [The previous sections showed the generic technique for examining an object that can be reached from static roots of the application - the technique applies to arbitrary objects reachable from the statics, you just need to use knowledge of your application structure to navigate through instance variables and methods. Even private and protected variables are navigable by setting them accessible through the reflection classes (assuming your application's security manager doesn't prevent this)].
So how can I see an object which is only being kept alive from the running thread? One way would
be to get the thread roots and navigate from them in a similar way as from a static, but that sounds difficult.
A simpler way is to insert a logging statement into the reporting class, so that when I get a
report from Reporter, I also see the maxCount value. In this case it would be a simple one line addition,
System.out.println("The maxCount is " + maxCount); at an appropriate place in the
Reporter class.
Well, that would sound simple, but how do I get that one line into the running JVM? Once again we will attach and run an agent, but this time the agent needs to use one of the capabilities of the Instrumentation class. Remember that agentmain() method we needed to define as the entry point to our agent? It is passed an Instrumentation object as the second argument, and that Instrumentation object can let you redefine code on the fly.
To keep it easy, I'll just edit my reporter class to include that logging statement, and recompile it. So I have a new version of the class - I'm very careful about this stage of course, checking a full diff between the old and new versions to make sure that I haven't inadvertently done something stupid - this is going to affect a running app so I really can't afford to make a mistake.
***** Old Reporter.java
{
return topN;
***** New Reporter.java
{
System.out.println("The maxCount is " + maxCount);
return topN;
*****
Now for my agent which is going to load this new version of the class over the old one. Its not particularly difficult (once you know how - isn't that always the way). First off I'll need to get hold of the actual .class file and load that in as a byte array. That is standard java.io stuff. Next I use those bytes to create a ClassDefinition object, the byte array is the second parameter to the constructor and it only needs the existing class object as the first parameter for the constructor and it's done. Finally, we call Instrumentation.redefineClasses using our just constructed ClassDefinition object, and presto, we're done. Here's that agent in full:
public class DumpMaxCount {
public static void agentmain(String agentArgs, Instrumentation inst)
{
try
{
System.out.println("Redefining Reporter class ...");
File f = new File("Reporter.class");
byte[] reporterClassFile = new byte[(int) f.length()];
DataInputStream in = new DataInputStream(new FileInputStream(f));
in.readFully(reporterClassFile);
in.close();
ClassDefinition reporterDef =
new ClassDefinition(Class.forName("Reporter"), reporterClassFile);
inst.redefineClasses(reporterDef);
System.out.println("Redefined Reporter class");
}
catch(Exception e)
{
System.out.println(e);
e.printStackTrace();
}
}
}
One little gotcha, the jar holding our agent needs to specify that it allows redefinitions with
a Can-Redefine-Classes: true line in the manifest file.
Figure 3: Reporter is dynamically redefined, and we now get maxCount printed with each logged line
Attaching and loading this agent results in maxCount now being printed each time the app prints the topN (in this case supposedly the top 10) queue elements. I'm expecting maxCount to be 1 since I only see one element printed and I know from the earlier dump of the actual queue that there is way more than one element in the queue. But maxCount prints out as 10 - what it should be, but not a value that explains the bug I'm seeing.
So I know the queue has more than one element, I know the reporter should be printing the first ten in the queue but is only printing one, time to go look at that printing code - its in a method:
public void report(Stack s)
{
topN = "";
if (s.size() == 0)
return;
int i = 0;
Iterator itr = s.iterator();
for(; itr.hasNext() && i < maxCount; i++);
{
topN += itr.next() + " / ";
}
}
Eagle eyed viewers will see the bug immediately, its a fairly common one and any code checking tool would find it too. Its the semi-colon at the end of the for loop line - Java sees that as the statement for the for loop, and it means that the subsequent block is executed just once after the for loop is terminated (I know, I know, if I had scoped the iterator variables correctly this wouldn't have even compiled. Really I wouldn't write it like that normally, but I needed a nice simple bug to demonstrate the hotpatch technique).
So I know the bug, I can fix the code, and we have already done all we need to hotpatch the app. All I need to do is recompile my Reporter class without the semi-colon, then re-run exactly the same DumpMaxCount agent I previously defined, as that is already setup to reload a new Reporter class.
Figure 4: Reporter is hotpatched, we see 10 elements logged each time now
And there we are. The app is working correctly now. Without stopping it, we were able to inspect arbitrary objects and fix arbitrary code. To me, at least, that's awesomely impressive.
There are limitations of course. Security restrictions may prevent you being able to do this on some apps. Some structures could be too difficult to reach to examine them. Redefined classes are restricted in how they can be redefined: basically you can change the code but not structure, hierarchy or method interfaces. And any loops that are already running do not get redefined, which in some apps could severely restrict patching options.
It's not something I would do on a regular basis, but for downtime-sensitive apps, this hotpatching capability could make a huge difference. Fantastic, and well done Sun.
Page 2 of 2
previous page: Starting the hotpatching process