Hadoop JobTracker crash log,version is 6u23

[Y. Srinivas Ramakrishna]

ulimit won't help, i don't think.
I don't know how to look at "internal review id", but
this one is not file descriptor exhaustion, but a sigbus
acessing an mmapped address, because as i said the mmapped
jar was either deleted or updated concurrent with this
class loading going on. I'd check with the sys admin.

-- ramki

On 3/9/2011 10:33 PM, Todd Lipcon wrote:
> I've seen a similar issue before when there aren't enough file handles
> available via ulimit for the Java process. Oracle folks please refer
> to "Internal Review ID" 1985110. Basically some of the classloader
> code doesn't handle ENFILE well.
>
> See if bumping ulimit -n fixes things.
>
> -Todd
>
> On Wed, Mar 9, 2011 at 10:21 PM, Y. Srinivas Ramakrishna
> <y.s.ramakrishna at oracle.com>  wrote:
>> It looks like you have a sigbus accessing:-
>>
>> siginfo:si_signo=SIGBUS: si_errno=0, si_code=2 (BUS_ADRERR),
>> si_addr=0x0000002ab978f000
>>
>> which falls in this pmap section:-
>>
>> 2ab9773000-2ab9794000 r--s 0023b000 08:02 8437945
>> /home/hadoop/hadoop-0.19.1-dc/hadoop-0.19.1-dc-core.jar
>>
>> while your thread is attempting to load a class from that jar file:-
>>
>> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>> J  java.util.zip.ZipFile.getEntry(JLjava/lang/String;Z)J
>> J
>>   sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
>> J  sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
>> j  java.net.URLClassLoader$1.run()Ljava/lang/Object;+26
>> v  ~StubRoutines::call_stub
>> j
>> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+0
>> j  java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+13
>> j  java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+47
>> j
>>   sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+41
>> j  java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
>> v  ~StubRoutines::call_stub
>> j
>> org.apache.hadoop.mapred.loadhistory_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+96
>> j
>> org.apache.jasper.runtime.HttpJspBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3
>> j
>> javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30
>> j
>> org.mortbay.jetty.servlet.ServletHolder.handle(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+161
>> j
>> org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/mortbay/jetty/servlet/ServletHolder;I)V+249
>> j
>> org.mortbay.jetty.servlet.Dispatcher.dispatch(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;I)V+447
>> j
>> org.mortbay.jetty.servlet.Dispatcher.include(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+4
>> j
>> org.apache.jasper.runtime.JspRuntimeLibrary.include(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljava/lang/String;Ljavax/servlet/jsp/JspWriter;Z)V+45
>> j
>> org.apache.hadoop.mapred.jobdetailshistory_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+225
>> j
>> org.apache.jasper.runtime.HttpJspBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3
>> j
>> javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30
>> j
>> org.mortbay.jetty.servlet.ServletHolder.handle(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+161
>> j
>> org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/mortbay/jetty/servlet/ServletHolder;I)V+249
>> j
>> org.mortbay.jetty.servlet.ServletHandler.handle(Ljava/lang/String;Ljava/lang/String;Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+290
>> j
>> org.mortbay.http.HttpContext.handle(Ljava/lang/String;Ljava/lang/String;Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+157
>> j
>> org.mortbay.jetty.servlet.WebApplicationContext.handle(Ljava/lang/String;Ljava/lang/String;Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+14
>> j
>> org.mortbay.http.HttpContext.handle(Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+455
>> j
>> org.mortbay.http.HttpServer.service(Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)Lorg/mortbay/http/HttpContext;+246
>> j
>> org.mortbay.http.HttpConnection.service(Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)Lorg/mortbay/http/HttpContext;+24
>> j  org.mortbay.http.HttpConnection.handleNext()Z+545
>> j  org.mortbay.http.HttpConnection.handle()V+17
>> j  org.mortbay.http.SocketListener.handleConnection(Ljava/net/Socket;)V+65
>> j  org.mortbay.util.ThreadedServer.handle(Ljava/lang/Object;)V+19
>> j  org.mortbay.util.ThreadPool$PoolThread.run()V+124
>> v  ~StubRoutines::call_stub
>>
>>
>> Perhaps that mmapped jar file has sudddenly been yanked out from under your
>> process? (Did the superuser decide to either delete it or update it while
>> it was being used by your process?) Anyhow, this is not GC related so
>> doesn't belong.
>> i don't think, on this alias.
>>
>> best.
>> -- ramki
>>
>> On 3/9/2011 9:47 PM, BlueDavy Lin wrote:
>>>
>>> hi!
>>>
>>>         The crash log attached,can someone give some advices?
>>>
>>> #
>>> # A fatal error has been detected by the Java Runtime Environment:
>>> #
>>> #  SIGBUS (0x7) at pc=0x0000003f41b72587, pid=18379, tid=1112861024
>>> #
>>> # JRE version: 6.0_23-b05
>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode
>>> linux-amd64 compressed oops)
>>> # Problematic frame:
>>> # C  [libc.so.6+0x72587]  memcpy+0x37
>>> #
>>> # If you would like to submit a bug report, please visit:
>>> #   http://java.sun.com/webapps/bugreport/crash.jsp
>>> # The crash happened outside the Java Virtual Machine in native code.
>>> # See problematic frame for where to report the bug.
>>> #
>>>
>>
>>
>
>
>