Hadoop JobTracker crash log,version is 6u23

[Todd Lipcon]

I've seen a similar issue before when there aren't enough file handles
available via ulimit for the Java process. Oracle folks please refer
to "Internal Review ID" 1985110. Basically some of the classloader
code doesn't handle ENFILE well.

See if bumping ulimit -n fixes things.

-Todd

On Wed, Mar 9, 2011 at 10:21 PM, Y. Srinivas Ramakrishna
<y.s.ramakrishna at oracle.com> wrote:
> It looks like you have a sigbus accessing:-
>
> siginfo:si_signo=SIGBUS: si_errno=0, si_code=2 (BUS_ADRERR),
> si_addr=0x0000002ab978f000
>
> which falls in this pmap section:-
>
> 2ab9773000-2ab9794000 r--s 0023b000 08:02 8437945
> /home/hadoop/hadoop-0.19.1-dc/hadoop-0.19.1-dc-core.jar
>
> while your thread is attempting to load a class from that jar file:-
>
> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> J  java.util.zip.ZipFile.getEntry(JLjava/lang/String;Z)J
> J
>  sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
> J  sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
> j  java.net.URLClassLoader$1.run()Ljava/lang/Object;+26
> v  ~StubRoutines::call_stub
> j
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+0
> j  java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+13
> j  java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+47
> j
>  sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+41
> j  java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
> v  ~StubRoutines::call_stub
> j
> org.apache.hadoop.mapred.loadhistory_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+96
> j
> org.apache.jasper.runtime.HttpJspBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3
> j
> javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30
> j
> org.mortbay.jetty.servlet.ServletHolder.handle(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+161
> j
> org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/mortbay/jetty/servlet/ServletHolder;I)V+249
> j
> org.mortbay.jetty.servlet.Dispatcher.dispatch(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;I)V+447
> j
> org.mortbay.jetty.servlet.Dispatcher.include(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+4
> j
> org.apache.jasper.runtime.JspRuntimeLibrary.include(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljava/lang/String;Ljavax/servlet/jsp/JspWriter;Z)V+45
> j
> org.apache.hadoop.mapred.jobdetailshistory_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+225
> j
> org.apache.jasper.runtime.HttpJspBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3
> j
> javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30
> j
> org.mortbay.jetty.servlet.ServletHolder.handle(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+161
> j
> org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/mortbay/jetty/servlet/ServletHolder;I)V+249
> j
> org.mortbay.jetty.servlet.ServletHandler.handle(Ljava/lang/String;Ljava/lang/String;Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+290
> j
> org.mortbay.http.HttpContext.handle(Ljava/lang/String;Ljava/lang/String;Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+157
> j
> org.mortbay.jetty.servlet.WebApplicationContext.handle(Ljava/lang/String;Ljava/lang/String;Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+14
> j
> org.mortbay.http.HttpContext.handle(Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+455
> j
> org.mortbay.http.HttpServer.service(Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)Lorg/mortbay/http/HttpContext;+246
> j
> org.mortbay.http.HttpConnection.service(Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)Lorg/mortbay/http/HttpContext;+24
> j  org.mortbay.http.HttpConnection.handleNext()Z+545
> j  org.mortbay.http.HttpConnection.handle()V+17
> j  org.mortbay.http.SocketListener.handleConnection(Ljava/net/Socket;)V+65
> j  org.mortbay.util.ThreadedServer.handle(Ljava/lang/Object;)V+19
> j  org.mortbay.util.ThreadPool$PoolThread.run()V+124
> v  ~StubRoutines::call_stub
>
>
> Perhaps that mmapped jar file has sudddenly been yanked out from under your
> process? (Did the superuser decide to either delete it or update it while
> it was being used by your process?) Anyhow, this is not GC related so
> doesn't belong.
> i don't think, on this alias.
>
> best.
> -- ramki
>
> On 3/9/2011 9:47 PM, BlueDavy Lin wrote:
>>
>> hi!
>>
>>        The crash log attached,can someone give some advices?
>>
>> #
>> # A fatal error has been detected by the Java Runtime Environment:
>> #
>> #  SIGBUS (0x7) at pc=0x0000003f41b72587, pid=18379, tid=1112861024
>> #
>> # JRE version: 6.0_23-b05
>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode
>> linux-amd64 compressed oops)
>> # Problematic frame:
>> # C  [libc.so.6+0x72587]  memcpy+0x37
>> #
>> # If you would like to submit a bug report, please visit:
>> #   http://java.sun.com/webapps/bugreport/crash.jsp
>> # The crash happened outside the Java Virtual Machine in native code.
>> # See problematic frame for where to report the bug.
>> #
>>
>
>



-- 
Todd Lipcon
Software Engineer, Cloudera