posts - 156,  comments - 601,  trackbacks - 0


Finding Memory Leaks Using the NetBeans Profiler



 

The NetBeans profiler has powerful capabilities to help you track down memory leaks in Java applications.  Using instrumentation, you see allocations on the heap happen as your program runs and the profiler provides statistical values to highlight patterns in your application's memory allocations.  This "behavioral" approach can help you quickly identify the most likely memory leak candidates, even in situations where each individual leak is very small.  The profiler also has a HeapWalker that can be used to examine the relationships between objects on the heap. This "after the fact" approach can help you quickly identify why a particular object is not being garbage collected by the Java virtual machine. And with its tight integration into the developer work flow, the profiler makes it easy to start/stop profiling sessions and more importantly, to go from profiler results directly into the source code that has the problem.

In this hands-on lab, you are going to use the NetBeans profiler to find out where exactly a memory leaking code is located, first in a web application (Exercise 1) and then in a stand-alone application (Exercise 2).  The latter is based on a real-life use case where the NetBeans profiler was used to find a memory leak.   In Exercise 3, you are going to use the HeapWalker feature of the profiler to analyze a heap.  In Exercise 4, you will learn how to attach the NetBeans profiler to an already-running application.


Expected duration: 100 minutes (excluding homework)




Software Needed

Before you begin, you need to install the following software on your computer. 

  • Java Standard Development Kit (JDK™) version 6.0 (download)
    • Select JDK 6 Update x from the download page
    • The Dynamic attachment (Exercise 4) require JDK 6.0.
    • If you are using Mac OS X, you can use JDK 5.0, with which you can do Exercise 1, 2, and 3.
  • NetBeans IDE 6.0 (download) or later version
    • Download either "Web & Java EE" or "All" bundles.
    • When you install NetBeans IDE, it will ask you which JDK you want to use.  Select JDK 6.0.
  • 5116_nbprofilermemory.zip (download)
    • The zip file contains this document and the lab contents
    • Download it and unzip in a directory of your choice.

Operating Systems/platforms you can use

  • Windows
  • Solaris x86, Solaris Sparc
  • Linux
  • Mac OS

Web/App servers you can use NetBeans profiler


The NetBeans profiling tools can be used with *any* web/application server that is running on JDK 5 update 4 or higher. Please note that the ease-of-use varies quite a bit depending upon which web/application server you are using.  This is because the NetBeans plugins that provide support for web/application servers in the IDE vary in their support of the profiling tools.  The web/appplication servers that work the best with the NetBeans profiler include (1) GlassFish (2) JBostt (3) Tomcat.  On all three of these, you can literally start a profiling session with a single mouse click, assuming that you have registered the server with the IDE.

For other web/application servers (and really, for any other Java application that your run outside the IDE), it gets a bit more complicated.  NetBeans has an "attach" wizard that will step you through what you need to do to modify the startup script for your web/application server so that it can be profiled.  Specific instructions are even included for popular servers such as Weblogic, etc.

Change Log

  • Sep. 16th, 2007: Created
  • Dec. 5th, 2007: Updated to reflect NetBeans IDE 6.0 final release
  • March 14th, 2008: Added information on Web/Application servers that can be used by the NetBeans profiler
  • March 16th, 2008: Direct attachment exercise is added as Exercise 4.3

Things to do (by Authors)

  • HeapWalker exercise need to describe the reference better
  • HeapWalker exercise to provide a step for fixing the problem
  • Need to add profiling point exercise on memory based on the contents here

Lab Exercises


Exercise 0: Perform JDK calibration


In this exercise, you will perform calibration against the JDK running on your system.  The calibration is required in order to perform profiling. It does not affect in any way the behavior of your Java applications.  This needs to be done only once.

Note: Instrumenting the bytecode of the profiled application imposes some overhead. To guarantee the high accuracy of profiling results, NetBeans Profiler needs to collect calibration data in order to "factor out" the time spent in code instrumentation. You need to run the calibration process for each JDK you will use for profiling. The calibration data for each JDK is saved in the .nbprofile directory in your home directory. You are prompted to run the calibration the first time you invoke NetBeans Profiler. You are also prompted if the calibration data for the local machine and JVM is unavailable.

(0.1) Run profiler calibration


1. Select Profile from top-level menu and select Advanced Commands->Run Profiler Calibration.  (Figure-0.20 below)


Figure-0.20: Run Profiler Calibration
  • Select JDK 1.6 (Default) (Figure-0.21 below)
  • Click OK.

Figure-0.21: Select Java Platform to calibrate
  • Observe that the Information dialog box indicating that the Calibration was successful.
  • Click OK to close the dialog box.

Figure-0.22: Information dialog box


Exercise 1: Build and run "memoryleak" Web sample application in Memory Profiling mode

In this exercise, you are going to profile the memoryleak sample application that contains a badly designed code fragment, in which double and float primitives are allocated in a loop and then saved as an entry into a HashMap, which would result in OutOfMemoryError eventually. The goal of using the profiler is to find out where this code fragment is located.

Learning points:

  • How to start a Java application (Web application in this example) in memory profiling mode
  • How to use live profiling data (especially Generation number) to detect abnormal memory allocation patterns
  1. Open "memoryleak" sample project
  2. Run the project in profiling mode
  3. Modify "memoryleak" sample project with a custom class

(1.1) Open "memoryleak" sample project

0. Start NetBeans IDE (if you have not done so yet). 
1. Open memoryleak NetBeans project. 

  • Select File->Open Project (Ctrl+Shift+O). The Open Project dialog box appears.
  • Browse down to <LAB_UNZIPPED_DIRECTORY>/nbprofilermemory/samples directory. 
    • Windows: If you unzipped the 5116_nbprofilermemory.zip file under C:\handsonlabs directory, the directory to which you want to browse down should be C:\handsonlabs\nbprofilermemory\samples.
    • Solaris/Linux: If you unzipped the 5116_nbprofilermemory.zip file under $HOME/handsonlabs directory, the directory to which you want to browse down should be $HOME/handsonlabs/nbprofilermemory/samples.
  • Select memoryleak
  • Click Open Project Folder.
  • Observe that the memoryleak project node appears under Projects tab window.

2. Observe the memory leaking code fragment.  The goal of this exercise is to find this memory leaking code using the NetBeans profiler.

  • Expand memoryleak project node.
  • Expand Source Packages->demo.memoryleak.
  • Double-click LeakThread.java to open it in the source editor window.
  • Observe that two arrays, one with a single float and another with a single double, are being allocated in a loop and saved in a HashMap. (Figure-1.11 below)

Figure-1.11: Code fragment that contains the memory leaking code

                                                                                                                       return to top of the exercise



(1.2) Run the project in "profiling" mode



1. Right click memoryleak project node and select Profile. (Figure-1.20 below)


Figure-1.20: Profile Main Project


If you see the following dialog box, choose JDK 1.6 (Default) as shown below and click OK button.




Trouble-shooting:  If you see the following dialog box, it is because the JDK that is being used by the NetBeans profiler has not been calibrated.
Solution: Perform the calibration as described above.





2. If the project is being profiled for the first time, you may encounter this dialog box. Click OK. (Figure-1.21 below)  If you have profiled the project previously, you will not see this dialog box.


Figure-1.21: Permission to Profile

3. Configure Analyze Memory options.
  • Select Memory on the left.
  • Select Record both object creation and garbage collection.
  • Select Record stack trace for allocations.
  • Make sure the Use defined Profiling Points option is not selected.
  • Click the Run button. (Figure-1.22 below)

Figure-1.22: Analyze Memory options

4. Trigger the memory-leaking.
  • Observe that the browser gets displayed.
  • Click the Start Leaking button.  This will trigger the memory leaking code to get executed.  (Figure-1.23 below )

Figure-1.23: Run the browser

Trouble-shooting: If you are experiencing a problem running this application over Tomcat, it is highly likely due to the fact that you have not installed Tomcat.  Change the deployment platform to GlassFish V2 by (1) Right click memoryleak project node and select Properties (2) In the Project Properties dialog box, select Run on the left and and select GlassFish V2 from the Server drop-down menu.

5. Display the live profiling result.
  • Click Live Results button on the left.
  • Observe that the Live Profiling Results window gets displayed on the right side of the IDE.
  • Click the Generations column in the Live Results window to display the results in sorted order.  (Figure-1.24 below) 

Figure-1.24: Profiling result

<Learning point: Columns of Live Results> The columns of the live results provide object allocation and memory usage information.
  • The Allocated Objects (third column from the right) is the number of objects that the profiler is monitoring. In the example shown in Figure-1.24 above, there are 11,727 instances of char[] that are being monitored. By default this number will be approximately ten percent of the objects actually allocated by your application. By monitoring only a subset of the created objects the profiler is able to dramatically reduce the overhead it places on the JVM, which then allows your application to run at close to full speed.
  • The Live Objects is the number of the Allocated Objects that are still on the JVM's heap and are therefore taking up memory.
  • The two Live Bytes columns show the amount of heap memory being used by the Live Objects. One column displays a graph, the other displays text.
  • The Avg. Age value is calculated using the Live Objects. The age of each object is the number of garbage collections that it has survived. The sum of the ages divided by the number of Live Objects is the Avg. Age.
  • The Generations value is calculated using the Live Objects. As with Avg. Age, the age of an object is the number of garbage collections it has survived. The Generations value is the number of different ages for the Live Objects.
<Learning point: Surviving Generations> Surviving Generations metrics cannot be easily defined with just one line, so let's make a three-line definition: (This is quoted from Uncovering Memory leaks Using NetBeans Profiler article by Jin Sedlacek.)
  • a Generation is a set of instances created within the same GC interval (between two garbage collections)
  • a Surviving Generation is a Generation that survives at least one garbage collection. The number of survived garbage collections - the generation's age - is its unique identifier
  • Surviving Generations (metrics) value is the number of differerent Surviving Generations that are currently alive on the heap (number of Generations with different generation ages)

Typically there are several long-lived objects (like an application's main JFrame etc.) in an application. The generation's age increases during the application's run time but still represents one or a few Surviving Generations.

Most applications also have short-lived objects which are created very frequently (such as Dimension etc.) but these objects are released very soon, typically within only a few garbage collections. This means that they represent only a few Surviving Generations (with generation's age of 1, 2, 3 etc.).

If we merge the two above cases, the total number of Surviving Generations in typical applications is quite stable. You may have long-lived objects representing, for example, 3 Surviving Generations, and short-lived objects representing, for example, 5 Surviving Generations, which means there will be about 8 Surviving Generations during the application's runtime.

Of course, in some application, for example a Swing application, dialogs and other components are created during run time and as a result the number of Surviving Generations grows a bit. But after some period of time the number of Surviving Generations should become stable because all long-lived objects have already been created and newly-created short-lived objects are periodically being released from the heap.

However, if there is a memory leak in an application which prevents newly-created objects from being released from the heap (for example objects stored in a Collection and never removed), the number of Surviving Generations grows. Why? Because between every two garbage collections a new generation (of leaking objects) is created and it survives each successive garbage collection. During run time the number of Surviving Generations whose generation's age differ by one unit increases, and that's exactly what the Surviving Generations metrics is able to detect regardless of how much memory is wasted by such a leak. That's why using the Surviving Generations metrics can help you discover a memory leak much sooner, before it consumes the entire heap and causes an OutOfMemoryError.



6. Detect which object types are not being garbarge-collected.
  • As the program continues to run, the profiler will update the display. Keep your eye on the entries for float[] and double[]. Notice how their Generation values continue to increase. As a result, float[] and double[] continue to move higher in the list. Eventually they will be displayed at the top of the list, right under java.util.HashMap$Entry, which also has an increasing value for generations. As the application continues to run the generations value continues to increase for java.util.HashMap$Entry, float[], and double[], but not for any of the other classes
  • Right click double[] and select Take Snapshot and Show Allocation Stack Traces. (Figure-1.25 below)

Figure-1.25: Take the Snapshot

7. Find the location of the code fragment that leaks memory.
  • Observe that methods in which the double primitive is allocated is displayed.  In this example, there is only one.
  • Double click demo.memoryleak.LeakThread.run().  This will display the code.

  • Observe that the run() method of the LeakThread.java, which contains the code fragment that causes the memory leak is now displayed in the source editor.   (Figure-1.27 below)  So, congratulations!  you found the memory leaking code.

Figure-1.27: Code is found

8. Stop profiling.
  • End the profiling session by choosing Profile > Stop or click the  button.
Note: The memoryleak project is configured to run over Tomcat as a deployment platform.  You can also perform the profiling over GlassFish v2 as well.
  • Right click memoryleak project and select Properties.
  • Observe that the Project Properties dialog box appears.
  • Select Run under Categories and then select GlassFish V2 from the drop-down menu of the Server field.  (If you don't see the Apache Tomcat 6.0.14 in the drop-down menu, it means you have not installed Tomcat during the installation process of NetBeans.)
                                                                                                                       return to top of the exercise


(1.3) Modify "memoryleak" sample application with a custom class


In this step, you are going to create a Person class whose array will be used as an object that will be saved into a Hashmap.

1. Create a Person class.
  • Right click demo.memory.leak and select New->Java Class.
  • Observe that the New Java Class dialog appears.
  • For the Class Name field, enter Person.
  • Click Finish. (Figure-1.31 below)

Figure-1.31: Create a new class

2. Modify the IDE generated Person.java with the code in Code-1.32 and in Figure-1.33 below.

package demo.memoryleak;

public class Person {
    private String name;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}
Code-1.32: Person.java


Figure-1.33: Person.java

3. Modify the LeakThread.java as shown in Code-1.34 and Figure-1.35 below. The code fragments that need to be modified are highlighted in bold font.

package demo.memoryleak;

public class LeakThread extends Thread {
    private java.util.HashMap<float[], Person[]> map =
                new java.util.HashMap<float[], Person[]> ();
    public boolean stop = false;
    public void run () {
        stop = false;
       
        //////////////////////////////////////////////
        while (!stop) {
            try {
                map.put (new float[1], new Person[1]);
                Thread.sleep (100);
                System.gc ();
            } catch (InterruptedException e) {
                return;
            }
        }
        //////////////////////////////////////////////
    }
}
Code-1.34: Modified LeakThread.java


Figure-1.35: Modified LeakThread.java

4. Run the application in profiling mode.
  • Right click the project and select Profile.
  • Observe that the Profile memoryleak dialog box appears.
  • Click Run. (Figure-1.36 below)

Figure-1.36: Run the application in profiling mode.

5. Start the leaking.
  • Browser gets displayed.
  • Click Start Leaking button.
6. Display the profiling result.
  • Click Live Results.
  • Observe that the Live Profiling Results tab window gets displayed.
  • Click Generation column to sort it.
  • Wait a minute or so.
  • Observe that the Generation count of the demo.memoryleark.Person[] keeps increasing.
  • Right click demo.memoryleark.Person[] and select Take Snapshot and Show Allocation Stack Traces. (Figure-1.37 below)

Figure-1.37: Generation count of demo.memoryleark.Person[] keeps increasing
  • Right click and select Go to Source.  (Figure-1.38 below)

Figure-1.38: Go to Source
  • Observe that the source code of the run() method of the LeakThread class gets displayed. (Figure-1.39 below)

Figure-1.39: Source code

7. Stop profiling.
  • End the profiling session by choosing Profile > Stop or click the  button.
                                                                                                                        return to top of the exercise

Summary

In this exercise, you learned how to use the profiler to monitor the memory usage of your application. You learned how to interpret the data collected, especially Generation, which can be used to find out which objects are not being garbage-collected.

                                                                                                                        return to the top




Exercise 2: Build and run "HttpUnit" Java SE sample application in Memory profiling mode

This exercise is based on an actual usage of the NetBeans profiler as part of the development and testing of a production application.  Andrés González of Spain used the NetBeans profiler to track down a memory leak in HttpUnit.  HttpUnit is an open source project that provides a framework for unit testing of web pages. It essentially acts as a web browser so that you can write unit tests to verify that the correct pages are being sent back from your web application.

Andrés was using HttpUnit to run multi-day tests of his application. He discovered something sort of odd - the JVM that was running the tests he created with HttpUnit would eventually report an OutOfMemoryError. The tests had to run for long period of time before the OutOfMemoryError would occur though, so apparently each memory leak was relatively small.

Andrés used the NetBeans profiler to track down the problem and he wrote a blog entry about it (the entry is in Spanish, but Google can translate). There is also this thread out on the HttpUnit mailing lists where he reported what he found.

The HttpUnit sample application included here is not the program that Andrés wrote - it does, however, encounter the same problem in HttpUnit that Andrés has used. The underlying issue has to do with the way that HttpUnit processes web pages that include JavaScript. The framework does have support for a subset of JavaScript, but not the entire language. If it encounters JavaScript that it does not understand it will throw an exception. If the web page you are attempting to test includes JavaScript that HttpUnit does not support and you do not want to wade through all those exceptions in the output then the HttpUnit documentation recommends that your test program call HttpUnitOptions.setExceptionsThrownOnScriptError( false );

The side effect, however, is that HttpUnit will store every exception thrown during its JavaScript processing in an ArrayList and it never removes them. So if your tests access enough web pages that either have JavaScript errors or that include JavaScript that HttpUnit does not support, you can eventually get an OutOfMemoryError.

One additional note on the sample application - it does not require a web server in order to run. HttpUnit has a nice feature where if what you want to test is the response from a servlet then it can host the servlet for you, in the same JVM as your test application. So the sample application consists of:

  • a servlet that returns JavaScript that has an error
  • a main() method that repeatedly requests a page from that servlet.
The process of finding out the offending code fragment is similar to what you've done in Exercise 1.
  1. Open "HttpUnit" sample application
  2. Run the application in "profiling" mode
  3. Display and analyze the collected data
  4. Correct the problem

(2.1) Open "HttpUnit" sample application

Before you start this exercise, it is highly recommended for you to read the above and have a good understanding of the context.

1. Open HttpUnit NetBeans project. 

  • Select File->Open Project (Ctrl+Shift+O). The Open Project dialog box appears.
  • Browse down to <LAB_UNZIPPED_DIRECTORY>/nbprofilermemory/samples directory. 
  • Select HttpUnit
  • Click Open Project Folder.
  • Observe that the HttpUnit project node appears under Projects tab window.
                                                                                                                       return to top of the exercise


(2.2) Run the application in "profiling" mode


1. Make sure the HttpUnit project is the Main project.  (The project node should be in bold-font.)  If it is not the Main project, right click the project and select Set as Main Project.

2. Select Profile from the top-level menu and select Profile Main Project.
  • If the project is being profiled for the first time, you may encounter this dialog box. Click OK. (Figure-2.19 below)  If you have profiled the project previously, you will not see this dialog box.


    Figure-2.19: Permission to Profile
3. Configure Analyze Memory options and run profiling.
  • Select Memory on the left.
  • Select Record both object creation and garbage collection.
  • Select Record stack trace for allocations.
  • Make sure the Use defined Profiling Points is not selected.
  • Click Run button.  (Figure-2.20 below)

Figure-2.20: Configure Analyze Memory options
  • Observe that the application begins running in the Output window of the IDE.  (Figure-2.21 below)  This is a simple test application that emulates the behavior that Andrés saw. It uses HttpUnit to process the HTML that is returned by a servlet. Requests are being repeatedly sent to that servlet by the test.
                                                                                                                       return to top of the exercise


(2.3) Display and analyze the collected data


1. Display Telemetry Overview.
  • Click Telemetry Overview icon.  (Figure-2.21 below)

Figure-2.21: Application is running
  • Observe that the Telemetry Overview window gets displayed. (Figure-2.22 below)

Figure-2.22: VM Telemetry Overview

In the graph on the left, the red shading indicates the allocated size of the JVM heap. The purple overlay indicates the amount of heap space actually in use. In the example above, the allocated heap size at the last update was about 5 megabytes. Of that, a little over 2 megabytes is actually being used to hold Java objects.

The graph on the right shows the count of active threads in the JVM.

The graph in the center shows two important heap statistics.

  • The blue line is the percentage of execution time spent by the JVM doing garbage collection and is graphed against the y-axis on the right edge of the graph. Time spent by the JVM doing garbage collection is time that is not available for it to run your application. So if the blue line indicates a large percentage you may want to consider tuning the JVM by configuring a larger heap size (refer to the -Xmx parameter documentation) or perhaps switching to a different garbage collection algorithm.
  • The red line is surviving generations and is graphed against the y-axis scale on the left edge of the graph. The count of surviving generations is the number of different ages of all the Java objects on the JVM's heap, where "age" is defined as the number of garbage collections that an object has survived. When the value for surviving generations is low, it indicates that most of the objects on the heap have been around about the same amount of time. If, however, the value for surviving generations is increasing at a high rate over time, then it indicates your application is allocating new objects while maintaining references to many of the older objects it already allocated. If those older objects are in fact no longer needed then your application is wasting (or "leaking") memory.
2. Display live profiling result.
  • Click Live Results icon in the profile window.
  • Observe that the Live Profiling Results window shows activity on the heap. The column on the left contains class names. For each class you can see information about the number of objects created, the number that are still in use (live), and a particularly interesting statistic called Generations.
  • Click the Generations column to sort the display by Generations. This will sort it in descending order.  (Figure-2.23 below)

Figure-2.23: Live Profiling Results

3. Detect which object types are not being garbarge-collected.
  • Observe that the two of the classes have huge values for Generation, in comparison to all the other classes: String and char[] array. (Figure-2.25 below)  More importantly, the Generations value for both of them continues to increase as the application runs. Note that, for the rest of the classes, this is not the case - they have stabilized.

  • Note: The generation count for a class is easy to understand. All you have to understand is that each object has an age. The age of an object is simply the number of the Java virtual machine's garbage collections it has survived. So if, for example, an object is created at the beginning of an application and the garbage collector has run 466 times then the age of that object at that point in time is 466. To calculate the Generation value for a class, just count up the number of different ages across all of its objects that are currently on the heap. That count of different ages is the number of generations.

  • Right-click java.lang.String and select Take Snapshot and Show Allocation Stack Traces.  This will take the snapshot of the java.lang.String objects and display more detailed information.

Figure-2.25: Take Snapshot and Show Allocation Stack Traces

4. Stop the profiling
  • Click Stop button to stop the profiling (since we got the profiling data). (Figure-2.26 below)

Figure-2.26: Stop the profiling

5. Analyze the profiled data.

Note: The displayed view shows all the places in the application where Strings were allocated. In a typical Java application, there can be dozens or even hundreds or thousands of places where Strings are allocated. What you want to know is: which of those allocations are resulting in memory leaks? You can use the generation count as a key indicator. Notice that only one of the allocation locations in this group has created Strings that have a large value for Generation count: java.lang.StringBuffer.toString() - it has the value of 20 in the Figure-2.28 below.  If we were to continue running the application and take more snapshots we would see larger values each time.

Now that you know that StringBuffer's toString() method is allocating Strings that appear to be candidate memory leaks, you need to determine how that relates to our application's usage of HttpUnit.
  • Double-click Memory: <time> tab window to display it in maximized window.  (You can click it again to minimize it.)
  • Expand java.lang.StringBuffer.toString() (Figure-2.28 below)  This will display all the places where StringBuffer.toString() method is invoked.
  • Observe that the only strings allocated in StringBuffer.toString() with a large value for generation count are the Strings that resulted from calls to StringBuffer.toString() by a call from the HttpUnit code: com.meterware.httpunit.javascript.JavaScript$JavaScriptEngine.handleScriptException().  You can tell this is the problem code.

Figure-2.28: Expand the suspicious code
  • Expand com.meterware.httpunit.javascript.JavaScript$JavaScriptEngine.handleScriptException().
  • Observe that the profiler goes ahead and expands the entire stack trace. You will see a straight-line back to the main() method, since there is only one way it is being called (in this example).
  • Observe that the main() method then calls com.metaware.httpunit.WebClient.getResponse() method.
  • Right click Main.main(String[]) and select Go To Source. (Figure-2.29 below)

Figure-2.29: Display the source of the main() method
  • Observe the code fragment that calls HttpUnit's getResponse() method on line 46 in Figure-2.30 below, which ends up making the call that results in memory leak.

Figure-2.30: Sample application's call to the code that has a memory leak

6. Take a look at the memory-leaking code.
  • Right click com.meterware.httpunit.javascript.JavaScript$JavaScriptEngine.handleScriptException(Exception,String) and select Go To Source.

Figure-2.31: See the source code that
  • Observe the code that actually has the memory leak: it adds Strings to an ArrayList and they never get removed. (Figure-2.32 below) 

Figure-2.32: Memory leaking code

                                                                                                                       return to top of the exercise


(2.4) Correct the problem


In this step, you are going to correct the problem - actually removing the offending code for the sake of the simplicity of the exercise.

1. Comment out the offending code in the com.meterware.httpunit.javascript.JavaScript$JavaScriptEngine.handleScriptException() method.

        private void handleScriptException( Exception e, String badScript ) {
            final String errorMessage = badScript + " failed: " + e;
            if (!(e instanceof EcmaError) && !(e instanceof EvaluatorException)) {
                e.printStackTrace();
                throw new RuntimeException( errorMessage );
            } else if (isThrowExceptionsOnError()) {
                e.printStackTrace();
                throw new ScriptException( errorMessage );
            } else {
                // _errorMessages.add( errorMessage ); // Commented out
            }
        }
Code-2.33: Remove the offending code

2. Rerun the application in the profiling mode.
3. Observe that the Generation number of the String class remains in the range of 10 to 12.  (Figure-2.34 below)  This indicates that the memory leaking problem has been resolved.


Figure-2.34: String objects are garbage collected.

                                                                                                                        return to top of the exercise

Summary

In this exercise, you have used NetBeans profiler to find the memory leaking code in the production application - HttpUnit code.  By the way, the HttpUnit provides a method that will clear the ArrayList in which these Strings are being collected.

                                                                                                                        return to the top


Exercise 3:  Use "HeapWalker" to analyze memory problem


The HeapWalker provides a complete picture of the objects on the heap and the references between the objects.  The HeapWalker is especially useful for analyzing binary heap dump files produced when an OutOfMemoryError occurs.  The Find Nearest GC Root feature can help you track down memory leaks by showing the owner of the reference that prevents an object from being garbage collected.  For more detailed information on NetBeans HeapWalker, please see "Using HeapWalker" online document.
  1. Open "PrimeNumber" project
  2. Run "PrimeNumber" project in "profiling" mode
  3. Use HeapWalker to analyze the problem
  4. Take and load Heap Dump

(3.1) Open "PrimeNumber" project

1. Open PrimeNumber NetBeans project. 

  • Select File->Open Project (Ctrl+Shift+O). The Open Project dialog box appears.
  • Browse down to <LAB_UNZIPPED_DIRECTORY>/nbprofilermemory/samples directory. 
  • Select PrimeNumber
  • Click Open Project Folder.  The PrimeNumber project node appears under Projects tab window.
2. Observe that the maximum heap size is set to 6M bytes.  This is to use smaller heap space than the default (in order to generate the problem fast.)
  • Right click the PrimeNumber project and select Properties.
  • Select Run.
  • Observe that the VM Options field is set to -Xmx6m.   (Figure-3.10 below)
  • Click OK to close it.

Figure-3.10: Maximum heap size

                                                                                                                       return to top of the exercise


(3.2) Run "PrimeNumber" project in profiling mode


1. Make sure the PrimeNumber project is the Main project.  (The project node should be in bold-font.)  If it is not the Main project, right click the project and select Set as Main Project.
2. Select Profile from the top-level menu and select Profile Main Project.
3. Configure profiling mode.
  • Select Monitor on the left.
  • Click Run button.  (Figure-3.20 below)

Figure-3.20: Monitor application

4. Run the application.
  • For the Enter a Number field, type 1000000.
  • Click Calculate Prime Numbers button.  (Figure-3.21 below)  It will calculate the largest prime number less than or equal to the number that was entered, 1000000 in this example.

Figure-3.21: Calculate Prime Number under 1000000
  • Observe that the result displays 999983.  (Figure-3.22 below)

Figure-3.22: Result

5. Generate OutOfMemoryError condition.
  • Click the Calculate Prime Numbers button again. 
  • Wait until you see a dialog box of Figure-3.23 below. (It might take a couple of minutes)  The dialog box indicates that an OutOfMemoryError was thrown so the profiler requested a standard binary heap dump snapshot from the JVM.
  • Click Yes.

Figure-3.23: Question
  • Observe also that, in the Output window of the IDE, the application displays an java.lang.OutOfMemoryError:  Java heap space.   (Figure-3.24 below).  We are now ready to use the HeapWalker to analyze the memory usage when this error occurred.

Figure-3.24: java.lang.OutOfMemoryError:  Java heap space

                                                                                                                       return to top of the exercise


(3.3) Use HeapWalker to analyze the problem


1. See the Summary view.
  • Observe that the  heapwalker opens up with a Summary view.  You can see the summary view of the heap usage: size, operating system, and the JVM system properties.  (Figure-3.30 below)

Figure-3.30: Summary view of the HeapWalker

2. View the Classes  view.
  • Click the Classes view button and then click the Size column to sort by size.  (Figure-3.31 below)  This is a list of all the classes that are on the heap, along with the number of object instances of each class and the total size occupied by those object instances.

Figure-3.31: Classes view

3. Display the instances.
  • Observe that the most of the heap is taken up by int[] arrays.  You are going to take a look at the object instances for int[] array.
  • Observe that there are 1896 int[] arrays on the heap (in the example picture below), whose total size amounts to 4414336 bytes. (The numbers of your system will be somewhat different from these numbers.)
  • Right click the int[] array and choose Show in Instances View.  (Figure-3.32 below)

Figure-3.32: Show in Instances View of int[] array
  • Observe that the int[] array instances are displayed on the left.
  • Click on the first int[] array instance that has the size of 4000012 bytes.
  • Now that you have selected an int[] array instance, you can see two things over on the right: its fields and a list of any objects that reference this particular instance.  (Figure-3.33 below)

Figure-3.33: Display information on int[] instance that holds large memory

4. Display the contents of the array instance.
  • Expand the <items 0-499> entry.  (Figure-3.35 below) Since this is an array, the list of Field column is actually a list of array indexes, in groups of 500.  The Type and Value columns display the type and the value of entries in the array.

Figure-3.35: View the values of the <items 0-499>
  • Scroll down the items.
  • Observe that this array holds all the prime numbers less than the requested value, with place holder entries, each of which has value of -1, for integers that are not prime.  (Figure-3.36 below)

Figure-3.36: Observe the int[] array contains the prime numbers

We have found something on the heap that should not be there - in this case, this appears to be an array that was used during the calculation that should have been garbage collected after the calculation. So why didn't the JVM's garbage collector remove it? There must be some accidental reference to it that is being made (and that should be cleared). This is where the References information comes in handy.

5. Display reference information.
  • Right click this in the References panel and click Show nearest GC Root. (Figure-3.37 below)  [Note: Garbage Collection (GC) roots are the objects that never get removed from the heap - they are the starting point for the JVM's garbage collector. Any object that is reachable from a GC root cannot be removed from the heap.]

Figure-3.37: Show Nearest GC Root
  • Expand it unitl the entry for primeNumbers is shown.  [Note: There are actually several GC roots in this case, but what is often more interesting is what we can learn by looking at the object references along the way, since if they were to let go of their reference this array would be eligible for removal by the garbage collector. Notice this variable called completeResults_.]
  • Right click completeResults_ and choose Go To Source. (Figure-3.39 below)  [Note: An advantage of an integrated tool - easy access to the source code.]

Figure-3.39: Go To Source
  • Observe that PrimeNumbers.java file opens up in the Source editor window.
  • Observe that the completeResults_ variable is on line 31.   It is a Map object.
  • Right-click it and select Find Usages.  (Figure-3.40 below)

Figure-3.40: Find Usage of the completeResults_
  • Observe that the Find Usage dialog box appears.
  • Set the Scope to PrimeNumbers.
  • Click Find.  (Figure-3.41 below)

Figure-3.41: Find Usage
  • Observe that the Usages window opens with the results
  • Double-click the only result.  (Figure-3.42 below)

Figure-3.42: Result of the Find
  • Observe that the complete list of candidate prime numbers is being put into this Map. (Figure-3.43 below)  But as we can see from the Usages results it is never removed. So we don't need the array anymore, but it cannot be garbage collected - that is a memory leak. And one easily found with the profiler's heapwalker.

Figure-3.43: Code fragment

                                                                                                                       return to top of the exercise


(3.4) Take and load Heap Dump


One last thing to note - the profiler's heapwalker can also be used on any binary heap dump file produced by one of Sun's JVMs. There are a variety of ways to get a JVM to produce a heap dump; as an example, there is a command line flag (-XX:-HeapDumpOnOutOfMemoryError ) that you can use to have the JVM create the file whenever an OutOfMemoryError is thrown. You can then open that file with this feature.

You can also create a Heap Dump from the IDE.

1. Create a Heap Dump.
  • Select Profile from top-level menu and select Take Heap Dump. (Figure-3.46 below)  This will save the heap dump as a file.

Figure-3.46: Take Heap Dump
  • Select Custom Directory.
  • Create a new directory, for example, c:\mydump, in this example.
  • Specify the directory into which you are going to create Heap Dump file.
  • Click OK.  (Figure-3.47 below)

Figure-3.47: Specify the directory
  • Click No for now.  (This is to simulate a situation in which you are reading a Heap Dump file created previously.)

Figure-3.48: Heap Dump Saved.

2.  Load the Heap Dump file.
  • Select Profile from the top-level menu and select Load Heap Dump.  (Figure-3.49 below)

Figure-3.49: Load Heap Dump
  • Observe that the loaded heap dump is displayed. (Figure-3.50 below)

Figure-3.50: Loaded heap dump

                                                                                                                        return to top of the exercise

Summary

In this exercise,  you have learned how to use the HeapWalker of the NetBeans profiler to analyze a memory usage problem, specifically to find a reference to an object that prevents it from being garbage collected.

                                                                                                                        return to the top


Exercise 4: Perform memory usage analysis on locally running applications


With JDK 6 you can attach the profiler to an application that is already running. No special JVM command line flags are necessary when you start that application.  Dynamic attach works even if you do not have a NetBeans project defined for the application.

Please note that Dynamic attach only works if the application being profiled is running on JDK6 and the NetBeans IDE itself is running on JDK6.  JDK5 does support only direct attach but not support dynamic attach.

In other words, "Dynamic" profiling does not require any command line flags when you start the JVM for the application.  Note this only works on JDK 6.  The "Direct" profiling is just the regular old way of starting a profiling session - which works with JDK 5 and higher.  In other words, the JVM for the application is started with the command line flags that tell it to wait for the profiling tool to attach, etc.

  1. Create and run AnagramGame project
  2. Dynamicaly attach profiler to locally running (but external to NetBeans IDE) JDK6-based Java application
  3. Locally attach profiler to locally running (but external to NetBeans IDE) JDK5-based Java application

(4.1) Create and run AnagramGame project


1. Create a new NetBeans project.
  • Select File->New Project. (Figure-4.10 below)


Figure-4.10: Create a new project

  • Expand Samples under Categories and select Java.
  • Select Anagram Game under Projects.  (Figure-4.11 below)


Figure-4.11: Select Anagram Game sample application

  • Select Finish (Figure-4.12 below)


Figure-4.12: Finish creating the project

2. Make sure the project is using JDK 6.
  • Right click the AnagramGame project node and select Properties.
  • Select Sources and make sure the Source Level is set to 1.6.  (Figure-4.13 below)
  • Click OK.


Figure-4.13: Make sure you are using JDK 6.

3. Build the project.

  • Right click the AnagramGame project and select Clean and Build Project.
  • Note that you can run the program at the command line by typing  java -jar "C:\myprojects\AnagramGame\dist\anagrams.jar".  (You will run the application this way later to simulate a locally running Java application.)


Figure-4.14: Take the command line

  • Right click project node and select Close.


Figure-4.15: Command line

4. Run the program external to the NetBeans IDE.

  • Open a terminal window and type java -jar "C:\myprojects\AnagramGame\dist\anagrams.jar"
Note: If you are running on FAT32 system on Windows, open a terminal window, and type java -XX:+PerfBypassFileSystemCheck -jar "C:\myprojects\AnagramGame\dist\anagrams.jar".

Note: If you are running the application over JDK 5, type java -Dcom.sun.management.jmxremote -jar "C:\myprojects\AnagramGame\dist\anagrams.jar"


Figure-4.16: Run the programming outside of the IDE

  • Observe that the Anagrams dialog box appears.
  • Type in abstraction and click Guess.


Figure-4.17: Anagram program

                                                                                                                       return to top of the exercise


(4.2) Attach NetBeans profiler dynamically to the locally running (but externally from NetBeans IDE) application


1. Select Profile->Attach Profiler. (Figure-4.20 below)


Figure-4.20:  Attach profiler

2. Configure Memory Analyzer options.
  • Make sure <External Application> is chosen for the Attach to: field.
  • Select Memory on the left.
  • Select Record both object creation and garbage collection.
  • Select Record stack trace for allocations.
  • Select Attach.

Figure-4.21: Attach Profiler

3. Configure Attach options.
  • Select Local under Attach method.
  • Select Dynamic (JDK 1.6) under Attach invocation.

Figure-4.23: Select Target Type
  • Observe that the Review Attach Settings dialog box appears.
  • Click Next.

Figure-4.24: Review Attach Settings
  • Click Finish.

Figure-4.25: Manual Integration

4. Select the target process to monitor.
  • In the Select Process dialog box, select AnagramGames process.
  • Click OK.  (Figure-4.26 below)

Figure-4.26: Select the target process


Trouble-shooting: If you experience <Error Getting Running Processes> as shown below, it is highly likely because you are running the AnagramGame application on a Windows system that has FAT32 file system.  This symptom is explained in the "hsperfdata is not being created on non-NTFS partitions".
Workaround:  When run the AnagramGame application in a terminal window, type java -XX:+PerfBypassFileSystemCheck -jar "C:\myprojects\AnagramGame\dist\anagrams.jar" not java -jar "C:\myprojects\AnagramGame\dist\anagrams.jar".


Figure-4.27: Process is not being shown


5. Analyze the process.
  • Click Live Results icon.
  • Observe that the live result is being displayed.

Figure-4.26: Display live results


                                                                                                                        return to top of the exercise


(4.3) Attach NetBeans profile directly r to the locally running (but externally from NetBeans IDE) application


1. Stop anagram application.
2. Start the Anagram (or any other application you want to profile) with the flag.
  • -agentpath:"C:\Program Files\NetBeans 6.0.1\profiler2\lib\deployed\jdk15\windows\profilerinterface.dll=\"C:\Program Files\NetBeans 6.0.1\profiler2\lib\"",5140
Example (for starting Anagram application):  java  -agentpath:"C:\Program Files\NetBeans 6.0.1\profiler2\lib\deployed\jdk15\windows\profilerinterface.dll=\"C:\Program Files\NetBeans 6.0.1\profiler2\lib\"",5140 -jar "C:\myprojects\AnagramGame\dist\anagrams.jar" (Figure-4.31 below)
  • Observe that the messages are displayed indicating profiler agent of the application is waiting for the profiler to call at the specified port, port 5140 in this example.

Profiler Agent: Initializing...
Profiler Agent: Options: >"C:\Program Files\NetBeans 6.0.1\profiler2\lib",5140<
Profiler Agent: Initialized succesfully
Profiler Agent: Waiting for connection on port 5140 (Protocol version: 8)


Figure-4.31: Start the java application with a flag

3. Attach the profiler.
  • Select Profile from top-levl menu and select Attach Profiler. (Figure-4.32 below)

Figure-4.32: Attach Profiler
  • Select Application in the Target Type drop down menu.
  • Select Local under Attach Method section.
  • Select Direct under Attach invocation section. (Figure-4.33 below)

Figure-4.33: Select Local and Direct attachment options
  • Observe that the Review Attach Settings  pane is displayed.
  • Click Next. (Figure-4.34 below)

Figure-4.34: Review Attach Settings
  • Read the Manula integration information. (Optional)
  • Click Finish.

Figure-4.35: Read information on Manual integration
  • Click Attach button. (Figure-4.36 below)

Figure-4.36: Perform attachment
  • Observe that the profiling agent is now attached with the profiler. (Figure-4.37 below)

Figure-4.37: Profiler agent

4. Observe the profiling result.
  • Click Live Results button and observe that the Live Profiling Results is being displayed. (Figure-4.38 below)

Figure-4.38: Profiling result

5. Close profiling session.
  • Click Finish button. (Figure-4.39 below)

Figure-4.39: Finish the profiling
  • Observe that the "Connection with the agent closed" message.

Figure-4.40: Connection is closed

Summary

In this exercise,  you have learned how to dynamically and locally attach the NetBeans profiler to a locally running application.

                                                                                                                        return to the top


Exercise 5: Perform dynamic attachment to JDK6 running App server


<tbd>
  1. Create and run AnagramGame project
  2. Dynamicaly attach profiler to locally running (but external to NetBeans IDE) JDK6-based Java application
  3. Locally attach profiler to locally running (but external to NetBeans IDE) JDK5-based Java application

(5.1) Create and run AnagramGame project


1.


Figure


Figure


Figure


Figure


Figure


Figure


Figure


Figure

Step 1: Original files C:\Program Files\glassfish-v2ur1\config\asenv.bat and C:\Program Files\glassfish-v2ur1\domains\domain1\config\domain.xml will be backed up to asenv.bat.backup and domain.xml.backup

Step 2: The following line will be modified in asenv.bat:
set AS_JAVA="C:\Program Files\Java\jdk1.6.0_04"

Step 3: If the domain.xml config file contains <profiler> section, this section will be removed.



Figure






Homework Exercise (for people who are taking Sang Shin's "Advanced Java Programming online course")


<to be provided>





转自: http://www.javapassion.com/handsonlabs/nbprofilermemory/



GoodLuck!
Yours Matthew!


posted on 2009-10-07 11:53 x.matthew 阅读(2821) 评论(1)  编辑  收藏 所属分类: Eclipse Plugins

只有注册用户登录后才能发表评论。


网站导航: