Friday, February 14, 2014

java.lang.ClassCastException: [I cannot be cast to java.util.List.

Caused by: java.lang.ClassCastException: [I cannot be cast to java.util.List.
Initially, I was confused by seeing the sentence "I cannot be cast to..."  However, it turns out that it should be read as:
"[I" cannot be cast to "java.util.List"
In this article, we will look at the class name (i.e., "[I") used in the class file.

What Is "[I"?


"[I" means array of "ints."  If you use jrcmd to generate a heap histogram,[1] you will see the following contents:

--------- Detailed Heap Statistics: ---------
30.0% 65027k   672176     -1k [C
10.2% 22119k   943754     -2k java/lang/String
10.0% 21592k   183036     -3k [Ljava/lang/Object;
 4.7% 10185k   434587     +0k java/util/HashMap$Entry
 4.7% 10114k    27029  -1254k [B
 4.5% 9783k   111539     +0k [Ljava/util/HashMap$Entry;
 1.9% 4075k    34777     +0k java/lang/Class
 1.9% 4058k    86590     +0k java/util/HashMap
 1.6% 3448k   147156     +0k javax/management/ObjectName$Property
 1.2% 2593k    82994     +0k java/util/LinkedHashMap$Entry
 1.1% 2398k    76765     +0k java/util/concurrent/ConcurrentHashMap$Segment
 1.0% 2215k     9311     +0k [I
 0.9% 1975k    18469     +0k [J

Besides "[I", we also find other array's class names:
  • [C
  • [B
  • [J

Array Representation[2]



In Java, arrays are full-fledged objects. Like objects, arrays are always stored on the heap. Also, multi-dimensional arrays are represented as arrays of arrays.

Arrays have a Class instance associated with their class, just like any other object. All arrays of the same dimension and type have the same class. The length of an array (or the lengths of each dimension of a multidimensional array) does not play any role in establishing the array's class. For example, an array of three ints has the same class as an array of three hundred ints. The length of an array is considered part of its instance data.

The name of an array's class has one open square bracket for each dimension plus a letter or string representing the array's type. For example, the class name for an array of ints is "[I". The class name for a three-dimensional array of bytes is "[[[B". The class name for a two-dimensional array of Objects is "[[Ljava.lang.Object".  Read [3] for more details of this naming convention for array classes.

Conclusions


This exception
java.lang.ClassCastException: [I cannot be cast to java.util.List
turned out to be a bug in the application code, which applies a type cast as shown below:

while (iter.hasNext()) 
{ 
  key = iter.next(); 
  val = (List) mEDefToVOAttrsMap.get(key);  <=== this line threw the Exception

The object returned from get() method was created in this way:

iArr = new int[val.size()]; 

This bug was exposed when we switched from JDK 6 to 7 in our tests.  Maybe due to security enforcements or other reasons, JDK 7 provides more runtime checking than JDK 6 does.  Therefore, this hidden bug has been revealed.

To conclude:
The compiler is not backwards compatible[4] because bytecode generated with JDK 7 won't run in Java 1.6 VM (unless compiled with the -target 1.6 flag[5]). But the JVM is backwards compatible, as it can run older bytecodes.

References

  1. Diagnosing OutOfMemoryError or Memory Leaks in JRockit (Xml and More)
  2. Array Representation (The Java Virtual Machine)
  3. The class File Format
  4. Is JDK “upward” or “backward” compatible?
  5. Cross-Compilation Options
  6. Java Products: All About Versions (Xml and More)







Thursday, February 6, 2014

Hotspot: Creating Flight Recording and Viewing Object Allocation

In [1], Marcus has written an excellent article on "Creating Flight Recording." Based on his advice, we have created a JFR recording with object allocations inside/outside TLAB (Thread Local Allocation Buffer) enabled. Then, we installed Java Mission Control (an Eclipse plug-in) on our 64-bit Windows platform and used it to investigate object allocations in the recording.

In this article, we will visit the following topics:
  • Generating JFR recording
  • Downloading and installing Java Mission Control
  • Viewing object allocations using Java Mission Control

Generating JFR Recording


Following Marcus' Time Fixed Recording example, we have used the following command line options:
  • -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:StartFlightRecording=delay=7800s,duration=300s,name=MyRecording,filename=/tmp/myrecording.jfr,settings=default

In our example, we started the recording for 5 minutes after Hotspot has run for 7800 seconds. To configure the recording, you can use either template provided in jre/lib/jfr folder:
  • default.jfc
  • profile.jfc

For our example, we have chosen the default template (note that the settings parameter can also take a path to a template) in which we have enabled the following settings:
<flag name="allocation-profiling-enabled" label="Allocation Profiling">true</flag>

<event path="java/object_alloc_in_new_TLAB">
  <setting name="enabled" control="allocation-profiling-enabled">true</setting>
  <setting name="stackTrace">true</setting>
</event>

<event path="java/object_alloc_outside_TLAB">
  <setting name="enabled" control="allocation-profiling-enabled">true</setting>
  <setting name="stackTrace">true</setting>
</event>

Note that we have saved an original copy of default.jfc before making the changes. After running our benchmark, it has generated a recording file named:
  • myrecording.jfr

Downloading and Installing Java Mission Control



Capability
Oracle JRockit
JDK6 (R28+)
Oracle JDK 7 GA
Oracle JDK
7u40+
Host JRMC/JMC GUI
Yes (JRMC) Yes (JMC) Yes (JMC)
WLDF JFR Events and Analysis
Yes Yes Yes
JFR, JMC Convergence
(JVM Events)
Yes No Yes
                 Table 1  Oracle JDK 7 Java Mission Control Support

You can follow the instructions from [2] to download Java Mission Control—an Eclipse plug-in. However, instead of using https, you should use http as shown above. Also, be noted that
Java Mission Control 5.2.0 does not support Kepler, only Juno.[3,4]
So, you need to download Eclipse Juno from here. If you installed JMC in Eclipse Kepler, you would see the problem as reported in [5].




Viewing object allocations using Java Mission Control


After opening myrecording.jfr, select Memory tab group on the left as shown above.  The Memory tab group shows Information on memory management and garbage collections. It is comprised of these tabs:
  • Overview Tab
  • Garbage Collections Tab
  • GC Times Tab
  • GC Configuration Tab
  • Allocations Tab
  • Object Statistics Tab
The Allocation tab is a good starting point if you suspect that you have problems with object allocation. It contains data about allocated objects and Thread Local Area Buffers (TLABs). This tab has information about how much memory each thread has allocated.

To recap: when we did the recording, we have enabled the following events:
  • java/object_alloc_in_new_TLAB
  • java/object_alloc_outside_TLAB
which are displayed in the above figure:
  • Allocation in new TLAB
  • Allocation outside TLAB


References

  1. Creating Flight Recordings
  2. Oracle Java Mission Control Downloads
  3. Java Mission Control for Eclipse
  4. Oracle® Java Mission Control
    • Oracle® Java Mission Control is a set of plug-ins for Eclipse 3.8 or 4.2.
  5. Mission Control and Flight Recorder on HotSpot JVM
  6. Which JVM?
    • If opening JFR recordings takes forever,  you may need to increase heap size.
  7. JDBC in Java Mission Control 
  8. JDK Mission Control (JMC) 8.1.0 Downloads 

Wednesday, February 5, 2014

Linux: Understanding Processor Queue in Vmstat Output

Linux tools such as mpstat, iostat, vmstat are useful for evaluating overall system performance at the OS level.  In this article, we will look at what statistics vmstat provides, especially focusing on CPU Run Queue (or Processor Queue).


vmstat Sample Ouput


vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity.  It takes an optional delay argument and other options.  For example, the following command asks vmstat make updates every 60 seconds.

$vmstat 60

The first report produced gives averages since the last reboot. Additional reports give information on a sampling period of length delay. The process and memory reports are instantaneous in either case. If no delay is specified, only one report is printed with the average values since boot.

procs -----------memory---------- -swap- ----io--- --system-- -----cpu------
 r  b  swpd   free   buff  cache   si so bi    bo   in   cs   us sy id wa st

 3  0   0 34138256 2654680 30542460 0  0 112 7052 1220 26560   11  9 79  0  0
134 0   0 34070712 2654812 30589228 0  0 19 15130 1174 2061612 79 10  9  1  0
13  0   0 34077516 2654868 30581300 0  0 32 10688 1123 1218261 68 17 15  0  0
 7  0   0 33928980 2654924 30538396 0  0 33  9762 1402 338231  41 12 46  1  0
 7  0   0 33904644 2654932 30556204 0  0 0   7094 1260 22498   12  8 80  0  0
 5  0   0 33869408 2654956 30564832 0  0 77  7298 1203 107386  15  9 76  0  0
 6  0   0 33907632 2654976 30532548 0  0 78  7220 1215 106477  14  9 76  0  0
 5  1   0 33898596 2655004 30539928 0  0 32  7172 1202 21369   10  8 81  0  0
 2  0   0 33883432 2655008 30547144 0  0 19  7158 1208 30007   10  8 81  0  0

As highlighted above, we have a high value (i.e., 134) of Run Queue, which will be discussed further later.

vmstat (Virtual Memory Statistics)


The summary information output by vmstat includes:
  • r
    • CPU Run Queue which is the actual number of lightweight processes in the run queue (including processes waiting to run but that are held up by the CPU)
  • b
    • Number of processes sleeping (usually waiting for IO)
  • swpd
    • Total swap space used (default: KB)
  • free
    • Total free memory (default: KB)
  • buff
    • Total buffer memory usage (default: KB)—represents how much portion of RAM is dedicated to cache disk block
  • cache
    • Total disk cache memory usage  (default: KB)—similar to buff, only this time it caches pages from file reading.
  • si
    • Memory swapped in from disk (in KB/sec)—the amount of memory paged-in 
    • If you see high si/so values, it indicates the system is swapping 
  • so
    • Memory swapped out to disk (in KB/sec)—the amount of memory paged-out
    • If you see high si/so values, it indicates the system is swapping 
  • bi
    • Blocks read in from IO devices (blocks per sec)
  • bo
    • Blocks written to IO devices (blocks per sec)
  • in
    • Interrupts per second
  • cs
    • Context switch per second
  • us
    • Percentage of user CPU utilization
  • sy
    • Percentage of kernel or system CPU utilization
    • High levels of system time mean something is wrong, or the application is making many system calls. Investigating the cause of high system time is  always worthwhile.
  • id
    • Percentage of idle or available CPU
    • The sum of the “us” column and “sy” column should be equal to 100 minus the value in the “id” column, that is, 100 – (“id” column value)
  • wa
    • Time spent waiting for IO—Prior to Linux 2.5.41, included in idle
  • st
    • Time stolen from a virtual machine—Prior to Linux 2.6.11, unknown


CPU Run Queue (Processor Queue)


As we have seen that there was 134 threads (note that we have a 24-processor Linux system) in the Processor Queue at one time, this may be something to be alerted.  It means that there are 134 threads that were running or that could run if there were available CPU.  Keep in mind that the run queue length represents everything on the machine, so sometimes there are other threads from completely separate processes that want to run.

If there are more threads to run than available CPUs, performance will begin to degrade. In general, you want the processor queue length to be 0 on Windows and equal to (or less than) the number of CPUs on Unix systems.

If the run queue length is too long for any significant period of time, it is an indication that your machine is overloaded and you should look into reducing the amount of work the machine is doing (either by moving jobs to another machine, or optimizing your code).[1]

Context Switches


When the machine is overloaded, you will also witness high number of context switches occurring at the same time.  From the line showing high run-queue length (i.e., 134), we also see high number of context switch per second (i.e., 2061612).

There are two types of context switches:
  • Voluntary thread context switches 
    • An executing thread voluntarily takes itself off the CPU
  • Involuntary thread context switches 
    • A thread is taken off the CPU as a result of an expiring time quantum or has been preempted by a higher priority thread
High involuntary context switches are an indication there are more threads ready to run than there are virtual processors available to run them, which results in:
  • A high run queue depth in vmstat
  • A high CPU utilization
  • A high number of migrations
For our case, the high run queue length occurred only for a short duration.  And, it may be due to the fact that our application and database were run on the same system.  At the time of high run queue lengths, database may be performing lots of I/O's.

References

  1. How to Troubleshoot High CPU Usage of Java Applications? (Xml and More)