Hadoop JobTracker crash log,version is 6u23

[Y. Srinivas Ramakrishna]

Just to close the loop, BlueDavy confirmed that the jar file
had been updated at that time, just as we'd suspected.

-- ramki

On 3/9/2011 10:38 PM, Y. Srinivas Ramakrishna wrote:
> ulimit won't help, i don't think.
> I don't know how to look at "internal review id", but
> this one is not file descriptor exhaustion, but a sigbus
> acessing an mmapped address, because as i said the mmapped
> jar was either deleted or updated concurrent with this
> class loading going on. I'd check with the sys admin.
>
> -- ramki
>
> On 3/9/2011 10:33 PM, Todd Lipcon wrote:
>> I've seen a similar issue before when there aren't enough file handles
>> available via ulimit for the Java process. Oracle folks please refer
>> to "Internal Review ID" 1985110. Basically some of the classloader
>> code doesn't handle ENFILE well.
>>
>> See if bumping ulimit -n fixes things.
>>
>> -Todd
>>
>> On Wed, Mar 9, 2011 at 10:21 PM, Y. Srinivas Ramakrishna
>> <y.s.ramakrishna at oracle.com> wrote:
>>> It looks like you have a sigbus accessing:-
>>>
>>> siginfo:si_signo=SIGBUS: si_errno=0, si_code=2 (BUS_ADRERR),
>>> si_addr=0x0000002ab978f000
>>>
>>> which falls in this pmap section:-
>>>
>>> 2ab9773000-2ab9794000 r--s 0023b000 08:02 8437945
>>> /home/hadoop/hadoop-0.19.1-dc/hadoop-0.19.1-dc-core.jar
>>>
>>> while your thread is attempting to load a class from that jar file:-
>>>
>>> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>>> J java.util.zip.ZipFile.getEntry(JLjava/lang/String;Z)J
>>> J
>>> sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
>>> J sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
>>> j java.net.URLClassLoader$1.run()Ljava/lang/Object;+26
>>> v ~StubRoutines::call_stub
>>> j
>>> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+0
>>>
>>> j java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+13
>>> j java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+47
>>> j
>>> sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+41
>>> j java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
>>> v ~StubRoutines::call_stub
>>> j
>>> org.apache.hadoop.mapred.loadhistory_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+96
>>>
>>> j
>>> org.apache.jasper.runtime.HttpJspBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3
>>>
>>> j
>>> javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30
>>>
>>> j
>>> org.mortbay.jetty.servlet.ServletHolder.handle(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+161
>>>
>>> j
>>> org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/mortbay/jetty/servlet/ServletHolder;I)V+249
>>>
>>> j
>>> org.mortbay.jetty.servlet.Dispatcher.dispatch(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;I)V+447
>>>
>>> j
>>> org.mortbay.jetty.servlet.Dispatcher.include(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+4
>>>
>>> j
>>> org.apache.jasper.runtime.JspRuntimeLibrary.include(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljava/lang/String;Ljavax/servlet/jsp/JspWriter;Z)V+45
>>>
>>> j
>>> org.apache.hadoop.mapred.jobdetailshistory_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+225
>>>
>>> j
>>> org.apache.jasper.runtime.HttpJspBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3
>>>
>>> j
>>> javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30
>>>
>>> j
>>> org.mortbay.jetty.servlet.ServletHolder.handle(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+161
>>>
>>> j
>>> org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/mortbay/jetty/servlet/ServletHolder;I)V+249
>>>
>>> j
>>> org.mortbay.jetty.servlet.ServletHandler.handle(Ljava/lang/String;Ljava/lang/String;Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+290
>>>
>>> j
>>> org.mortbay.http.HttpContext.handle(Ljava/lang/String;Ljava/lang/String;Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+157
>>>
>>> j
>>> org.mortbay.jetty.servlet.WebApplicationContext.handle(Ljava/lang/String;Ljava/lang/String;Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+14
>>>
>>> j
>>> org.mortbay.http.HttpContext.handle(Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)V+455
>>>
>>> j
>>> org.mortbay.http.HttpServer.service(Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)Lorg/mortbay/http/HttpContext;+246
>>>
>>> j
>>> org.mortbay.http.HttpConnection.service(Lorg/mortbay/http/HttpRequest;Lorg/mortbay/http/HttpResponse;)Lorg/mortbay/http/HttpContext;+24
>>>
>>> j org.mortbay.http.HttpConnection.handleNext()Z+545
>>> j org.mortbay.http.HttpConnection.handle()V+17
>>> j org.mortbay.http.SocketListener.handleConnection(Ljava/net/Socket;)V+65
>>> j org.mortbay.util.ThreadedServer.handle(Ljava/lang/Object;)V+19
>>> j org.mortbay.util.ThreadPool$PoolThread.run()V+124
>>> v ~StubRoutines::call_stub
>>>
>>>
>>> Perhaps that mmapped jar file has sudddenly been yanked out from under your
>>> process? (Did the superuser decide to either delete it or update it while
>>> it was being used by your process?) Anyhow, this is not GC related so
>>> doesn't belong.
>>> i don't think, on this alias.
>>>
>>> best.
>>> -- ramki
>>>
>>> On 3/9/2011 9:47 PM, BlueDavy Lin wrote:
>>>>
>>>> hi!
>>>>
>>>> The crash log attached,can someone give some advices?
>>>>
>>>> #
>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>> #
>>>> # SIGBUS (0x7) at pc=0x0000003f41b72587, pid=18379, tid=1112861024
>>>> #
>>>> # JRE version: 6.0_23-b05
>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode
>>>> linux-amd64 compressed oops)
>>>> # Problematic frame:
>>>> # C [libc.so.6+0x72587] memcpy+0x37
>>>> #
>>>> # If you would like to submit a bug report, please visit:
>>>> # http://java.sun.com/webapps/bugreport/crash.jsp
>>>> # The crash happened outside the Java Virtual Machine in native code.
>>>> # See problematic frame for where to report the bug.
>>>> #
>>>>
>>>
>>>
>>
>>
>>
>