Resolving Java Performance Problems in Production

Posted: February 20th, 2016 | Author: | Filed under: General

Performance problems in production can be tricky. This post describes a proven sequence for resolving urgent production performance problems for Java.

Sources of Performance Problems

The main sources of the performance problems in production are:

  • coding errors
  • internal and external bottlenecks
  • garbage collection (GC)

Coding errors are places in the code where the application acts in a way that leads to significant reduction of speed such as heavy repeated calculations inside a loop. Internal bottlenecks are improper, excessive synchronization, exhausted thread pools and connection pools. External bottlenecks are serially accessed data sources such as databases, file and socket I/O.

Manifestation of Performance Problems

Performance problems in production usually manifest themselves as users complaining about the application being ‘slow’. While ‘slow’ may be a matter of perception, generally it means that the application is abnormally slow. Performance problems may seriously damage your company business and must be resolved as soon as possible. The problem resolution sequence below worked well for me for many years.

Sequence for Resolving Performance Problems

1. Make sure JVM has correct heap settings. The maximum and minimum head should be set to the same value. Example: -Xmx1000k -Xms1000k.

2. Check memory consumption on the server. Make sure that the system is not swapping. Run top or ps to check if the server as a whole and the application itself is not swapping. Increase memory on the server if it doesn’t have enough memory or move some applications out.

3. Check server CPU usage. If the server is underloaded, this is usually an indication of an internal or external bottleneck. Thread dumps, heap dumps and profiling must help to identify the bottlenecks.

4. Take and analyze thread dumps. This method is great because it doesn’t require any instrumentation, takes a minute, and often is enough to isolate the problem. During the periods of slowness, take 5-6 thread dumps. Just send kill -3 if your application is running under *nix. Separate each thread dump by a 10 second interval. Check the thread dumps for places where threads remain within the same execution stack for extended periods of time. Most of the times the problem stack traces will contain application classes. Watch for thread pools all stuck in file or network I/O. Watch for exhausted thread pools. Watch for execution threads blocked on the same objects. Fix found problems.

5. Take and analyze heap dumps. Take 2-3 heap dumps during normal operation using JMap and during the system slowing down or JVM being completely unresponsive. Analyze the heap dumps using JProfiler for objects having large number of instances. Also, compare a normal heap dump with the one taken during the slowdown. Find objects counts that grew significantly since the period of normal operation. Such objects strongly indicate a memory leak. Find and fix the problem in the code.

6. Profile while running a load test. In the QA environment, run a load test the simulates the same sequence of user actions that production users are having a problem with. Use a profiler such as JProfiler to identify CPU and allocation hot spots. Fix found problems.

7. Profile in production. Profiling adds some overhead, so the application will run slower than normally. Use a single server to profile if running a cluster. To minimize overhead, configure JProfiler properly:

  • Use sampling instead of instrumentation
  • Add a narrow list of packages that are interesting from a user-perspective
  • Do not record allocations
  • Only record CPU when there are reports from users about performance issues

8. Tune GC. Enable GC in production by using command line option –XX:+PrintGCDetails. Watch for major GCs occurring abnormally often. The main causes of frequent major GCs are the application running out of heap due to memory leaks and excessive object creation and incorrect GC parameters. Profile the application for memory leaks and excessive object creation. Fix found problems. Adjust GC parameters for better performance. Tuning GC should be the last thing to do. Check our collection of useful resources on tuning Java GC.

9. Cache hard to get data. If the performance bottleneck is found and it is a database or other serially accessed data source or a hard-to-compute data source, cache that data. Here are a few articles on how to approach caching. Or drop a message in Cacheonix support forum; we’ll try to help.

Hope this helps

Slava Imeshev

(No) Comments: Post a response

Presenting on Scaling Java Applications at Code Camp 2012

Posted: September 21st, 2012 | Author: | Filed under: General

This is a great chance to meet developers of open source distributed cache Cacheonix in person. I will be presenting on scaling Java applications horizontally at two separate times on Sunday, October 7 at Silicon Valley Code Camp. Click on the links below to register:

  1. Architecture for scaling Java applications to multiple servers
  2. Developing scalable Java applications with open source clustered cache Cacheonix

Silicon Valley Code Camp is a new type of community event where developers learn from fellow developers. It is totally free and will occur on October 6 and 7.

See you at Code Camp 2012!


Regards,

Slava Imeshev
Main committer, Cacheonix

(No) Comments: Post a response

How to Store a List in a Cache

Posted: September 3rd, 2011 | Author: | Filed under: General

Storing a list in a cache and getting it from the cache is easy. Here is how.

The simplest way is just to put the list to the cache. Note that if you want to use a strongly typed cache, you need to provide a concrete list type:


import java.util.ArrayList;
import org.cacheonix.Cacheonix;
import org.cacheonix.cache.Cache;

...

   /**
    * Puts a list to the cache.
    *
    * @param key  a key identifying the list.
    * @param list the list to cache.
    */
   public void putListToCache(String key, ArrayList<String> list) {

      final Cache<String, ArrayList<String>> listCache = Cacheonix.getInstance().getCache("listCache");
      listCache.put(key, list);
   }


   /**
    * Gets a list from the cache.
    *
    * @param key a key identifying the list.
    * @return the cached list or null if the list wasn't found in the cache or if the list is null.
    */
   public ArrayList<String> getListFromCache(String key) {

      final Cache<String, ArrayList<String>> listCache = Cacheonix.getInstance().getCache("listCache");
      final ArrayList<String> list = listCache.get(key);
      return list;
   }

While the above works, there is a better way, which is creating a first-class object for the cached list.

Usually lists don’t come alone and are accompanied by some metadata. For example, if you are caching a database result set, in future you might want to cache time it took to retrieve the result set in addition to the result set itself. Wrapping the list into a class provides an ability to maintain a stable signature of the cached object. If you need to add more fields in the future, you don’t have to change all places where a cached list is put to and retrieved from the cache.

A cached result set object that contains a list would then look like the following. Note that it declares a signature interface Immutable that signals the cache that this object can be passed by reference:


import java.io.Externalizable
import org.cacheonix.cache.Immutable;

...

  public class CachedResultSet implements Externalizable, Immutable {

    private List<String> list = null;


    public CachedResultSet(List<String> list) {
      this.list = new ArrayList<String>(list);
    }


    /**
     * Required by Externalizable.
     */
    public CachedResultSet() {
    }

    public List getList() {
      return new ArrayList(list);
    }

    public void writeExternal(ObjectOutput objectOutput) throws IOException {

      if (list == null) {

        objectOutput.writeBoolean(true);
      } else {

        objectOutput.writeInt(list.size());
        for (int i = 0; i < list.size(); i++) {

          objectOutput.writeUTF(list.get(i));
        }
      }
    }


    public void readExternal(ObjectInput objectInput) throws IOException, ClassNotFoundException {

      if (!objectInput.readBoolean()) {

        int size = objectInput.readInt();
        list = new ArrayList<String>(size);
        for (int i = 0; i < size; i++) {

          list.add(objectInput.readUTF());
        }
      }
    }
  }

Getting the cached result set would look like this:

   public CachedResultSet getListFromCache(String key) {

      final Cache<String, CachedResultSet> listCache = Cacheonix.getInstance().getCache("listCache");
      final CachedResultSet list = listCache.get(key);
      return list;
   }

Regards,

Slava Imeshev

(No) Comments: Post a response

Testing Private Methods in Java

Posted: August 19th, 2011 | Author: | Filed under: General

Java 5 provides vargargs API that allows to test private methods in a very concise way. Here is how.

Yesterday, during a meetup at Loopt, a question of an API for testing private methods was brought up. I used to use JUnit add-on PrivateAccessor [1] when I was testing private methods. The problem was that applying PrivateAccessor produced prohibitively verbose code, so you had to think really hard to understand what the test was doing.

invoke() with varargs

Luckily, Java 5 provides vargargs API that allows to write really concise code for invoking private methods and return results. Here is the invoke() method that makes use of vargargs. Feel free to use it in your code:

   /**
    * Invokes a private method on an object.
    *
    * @param obj        the object on what to invoke the method.
    * @param methodName the name of the method to invoke.
    * @param parameters a list of parameter
    * @return the result of dispatching the method represented by the object.
    */
   public static <T> T invoke(Object obj, String methodName, Object... parameters) {

      try {

         // Create a list of parameter classes
         final Class[] parameterTypes = new Class[parameters.length];
         for (int i = 0; i < parameters.length; i++) {
            parameterTypes[i] = parameters[i].getClass();
         }

         // Get method
         Class clazz = obj.getClass();
         Method method = clazz.getDeclaredMethod(methodName, parameterTypes);
         method.setAccessible(true);

         // Call method
         return (T)method.invoke(obj, parameters);
      } catch (RuntimeException e) {

         throw e;
      } catch (Exception e) {

         throw new IllegalArgumentException(e);
      }
   }

Complete Example

Below is a complete JUnit tests that tests StringBuilder’s private method append(). Notice how neatly the invocation of append() fits into a short single line:

import java.lang.reflect.Method;
import junit.framework.TestCase;

public final class StringBuilderTest extends TestCase {


   /**
    * Object under test
    */
   private StringBuilder object;


   /**
    * Tests String's private method 'append(StringBuilder sb)'.
    */
   public void testAppend() {
      
      StringBuilder toAppend = new StringBuilder("Test");
      StringBuilder result = invoke(object, "append", toAppend);
      assertEquals("Test", result.toString());
   }


   /**
    * Invokes a private method on an object.
    *
    * @param obj        the object on what to invoke the method.
    * @param methodName the name of the method to invoke.
    * @param parameters a list of parameter
    * @return the result of dispatching the method represented by the object.
    */
   public static <T> T invoke(Object obj, String methodName, Object... parameters) {

      try {

         // Create a list of parameter classes
         final Class[] parameterTypes = new Class[parameters.length];
         for (int i = 0; i < parameters.length; i++) {
            parameterTypes[i] = parameters[i].getClass();
         }

         // Get method
         Class clazz = obj.getClass();
         Method method = clazz.getDeclaredMethod(methodName, parameterTypes);
         method.setAccessible(true);

         // Call method
         return (T)method.invoke(obj, parameters);
      } catch (RuntimeException e) {

         throw e;
      } catch (Exception e) {

         throw new IllegalArgumentException(e);
      }
   }


   public void setUp() throws Exception {

      super.setUp();

      object = new StringBuilder(10);
   }
}

Making invoke() reusable

To make invoke() reusable, I suggest to move it to a separate utility class or at least to create a super class for all your tests that extends junit.TestCase and move invoke() there.

Regards,

Slava Imeshev

References

1. JUnit Add-On PrivateAccessor

(7) Comments: Post a response

Freeing Desk Space with Docking Stand

Posted: June 13th, 2011 | Author: | Filed under: General

My laptop takes quite some space on my  desk. The good news, there is a neat solution that allows to reduce its footprint almost 10 times.

I am using Mac Pro and MacBook Pro as development machines while working on Cacheonix. The Mac Pro is tucked under my desk and is practically invisible. Yet, the MacBook has been occupying a noticeable chunk of the desk surface, which is not nice:

Macbook Taking Space on the Table

The obvious solution was to set it upright, but how? My requirements to the solution were: simple; stable; good-looking.  An extensive research fetched a winner – Power Support’s Docking Stand for MacBook. This neat device satisfies all my requirements. The assembly includes pads to prevent scratching. The aluminum color that matches my MacBook’s body is a nice bonus. $40 price tag is a steal.

Assembly has taken 2 minutes and after getting it all together the MacBook has practically disappeared from the surface of the desk freeing about 50 sq. in:

Macbook with Docking Stand

Too bad PowerSupport stopped making those.  Henge Docks now makes a more advanced and a bit more expensive model that allows to really dock your Macbook.

Regards,

Slava Imeshev
Cacheonix: Reliable Distributed Java Cache

(No) Comments: Post a response