From peter.helfer.java at gmail.com Tue Nov 6 09:04:28 2007 From: peter.helfer.java at gmail.com (Peter Helfer) Date: Tue, 6 Nov 2007 18:04:28 +0100 Subject: Creating a JavaThread and its java.lang.Thread oop Message-ID: Hi all I am still trying to get my thread pool, so far without real luck... ok, so here is how I'm trying to achieve it: - I'm preparing my own subsystem from runtime/init.cpp, which creates some runtime-code etc.. - When starting up (runtime/thread.cpp: Threads::create_vm(..) ), I'm starting the thread pool after everything is done, shortly before vm_init_completed is set, so I can use Universe::... - I've been seeing a lot of different assertion errors, so I'm now completely exhausted :-( [create_vm(..)]: initialize_class(vmSymbolHandles::java_lang_Compiler(), CHECK_0); reset_vm_info_property(CHECK_0); threadpool_init2(); // HERE COMES THE THREAD POOL quicken_jni_functions(); set_init_completed(); HS_DTRACE_PROBE(hotspot, vm__init__end); Management::record_vm_init_completed(); Now in this function I'm calling the constructor to it which calls the initialize() function: void Threadpool::initialize(){ // is called indirectly by threadpool_init2(); assert(Threadpool::get_SpinWaitEntry() != 0, "the spin wait entry is missing!"); Threadpool::might_be_available = 0; // static peek variable Threadpool::threadID = 1; //static var // MutexLocker ml(ThreadPool); - /* THIS CAUSED SOME LOCK INVERSION.. * it is skipped, but still safe, as all clients to this pool first ask might_be_available > 0, which * isnt the case. But it would be nicer.. */ if(TracePoolGen) FlagSetting fs(TraceThreadEvents, true); for(int i = 0; i < _size; i++){ _pool[i] = createThread((ThreadFunction)Speculation::get_SpinWaitEntry()); Thread::start(_pool[i]); // and start it immediately } Threadpool::might_be_available = _size; } This now calls the real creator: JavaThread* Threadpool::createThread(ThreadFunction entrypoint){ EXCEPTION_MARK; // creates a ExceptionMark and THREAD JavaThread* result_thread = NULL; { result_thread = new JavaThread(entrypoint); //size = 0 if (result_thread->osthread() != NULL) { oop group = Universe::main_thread_group(); bool attach_failed = false; { HandleMark hm(THREAD); Handle thread_group(THREAD, group); result_thread->allocate_threadObj(thread_group, "fromThreadPool", false, THREAD); if (HAS_PENDING_EXCEPTION) { CLEAR_PENDING_EXCEPTION; // cleanup outside the handle mark. attach_failed = true; } } if(attach_failed){ delete result_thread; result_thread = NULL; } } else { delete result_thread; result_thread = NULL; } } return result_thread; } Now I've seen different errors with different configurations... - inverted locks - SIGSEV (that was before splitting the init into 2 parts...) - looping forever with TracePoolGen/TraceThreadEvents set... creating thread 0x8084800 creating thread 0x8086400 creating thread 0x8088000 creating thread 0x8089c00 creating thread 0x808b800 creating thread 0x808d400 creating thread 0x808f000 creating thread 0x8090c00 creating thread 0x8092800 creating thread 0x8094800 Thread::set priority 0x0806d400 [4d6c] main (prio: 5) creating thread 0x80d4800 Thread::set priority 0x080d4800 [4d83] Reference Handler (prio: 10) Thread::start 0x080d4800 [4d83] Reference Handler (prio: 10) creating thread 0x80d6000 .. and here nothing happens anymore - and the current one: Internal Error (/home/peterh/workspace/openjdk/hotspot/src/share/vm/runtime/handles.cpp:40), pid=18280, tid=3084282768 # Error: assert(thread == Thread::current(),"sanity check") Could somebody maybe point me in the right direction, or even hand out a correct piece of code ? It seems that I am missing important parts, but I have no clue what to look for.. and before I smash my monitor, I rather give in and try to continue at a different end of my project.. Thanks & Regards, Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20071106/55782f74/attachment.html From Thomas.Rodriguez at Sun.COM Tue Nov 6 09:56:17 2007 From: Thomas.Rodriguez at Sun.COM (Tom Rodriguez) Date: Tue, 06 Nov 2007 09:56:17 -0800 Subject: Creating a JavaThread and its java.lang.Thread oop In-Reply-To: References: Message-ID: <4730AAC1.5030109@sun.com> I'd recommend creating your thread pool at the absolute end of create_vm. Otherwise you are do your work during bootstrap which is a very tricky period in the VM and you're just asking for trouble. One big clue is way down at the bottom. > (/home/peterh/workspace/openjdk/hotspot/src/share/vm/runtime/handles.cpp:40), > pid=18280, tid=3084282768 > # Error: assert(thread == Thread::current(),"sanity check") > This means that the code appears to be confused about which thread it's running in. If you look at allocate_threadObj it expects to be called from within the newly created thread, not from within the creating thread, which means that this == THREAD, but the way you are calling it this != THREAD. Handles are always specific to the currently running thread so if you pass in an explicit argument it must be equal to the current thread. If you replace the use of this in the Handle constructors with THREAD then it will work properly when called from a different thread or the same thread. By the way, this: if(TracePoolGen) FlagSetting fs(TraceThreadEvents, true); doesn't do what you think. FlagSetting is temporary so it's reset when the destructor runs. If you want to permanently set it, FLAG_SET_DEFAULT(TraceThreadEvents, true) is the recommended idiom. tom Peter Helfer wrote: > Hi all > > I am still trying to get my thread pool, so far without real luck... ok, > so here is how I'm trying to achieve it: > > - I'm preparing my own subsystem from runtime/init.cpp, which creates > some runtime-code etc.. > - When starting up (runtime/thread.cpp: Threads::create_vm(..) ), I'm > starting the thread pool > after everything is done, shortly before vm_init_completed is set, so > I can use Universe::... > - I've been seeing a lot of different assertion errors, so I'm now > completely exhausted :-( > > [create_vm(..)]: > initialize_class(vmSymbolHandles::java_lang_Compiler(), CHECK_0); > reset_vm_info_property(CHECK_0); > threadpool_init2(); // HERE COMES THE THREAD POOL > quicken_jni_functions(); > set_init_completed(); > HS_DTRACE_PROBE(hotspot, vm__init__end); > Management::record_vm_init_completed(); > > Now in this function I'm calling the constructor to it which calls the > initialize() function: > > void Threadpool::initialize(){ // is called indirectly by > threadpool_init2(); > assert(Threadpool::get_SpinWaitEntry() != 0, "the spin wait > entry is missing!"); > Threadpool::might_be_available = 0; // static peek variable > Threadpool::threadID = 1; //static var > > // MutexLocker ml(ThreadPool); - > /* THIS CAUSED SOME LOCK INVERSION.. > * it is skipped, but still safe, as all clients to this pool > first ask might_be_available > 0, which > * isnt the case. But it would be nicer.. > */ > > if(TracePoolGen) FlagSetting fs(TraceThreadEvents, true); > for(int i = 0; i < _size; i++){ > _pool[i] = > createThread((ThreadFunction)Speculation::get_SpinWaitEntry()); > Thread::start(_pool[i]); // and start it immediately > } > Threadpool::might_be_available = _size; > } > > This now calls the real creator: > > JavaThread* Threadpool::createThread(ThreadFunction entrypoint){ > EXCEPTION_MARK; // creates a ExceptionMark and THREAD > JavaThread* result_thread = NULL; > { > result_thread = new JavaThread(entrypoint); //size = 0 > if (result_thread->osthread() != NULL) { > oop group = Universe::main_thread_group(); > bool attach_failed = false; > { > HandleMark hm(THREAD); > Handle thread_group(THREAD, group); > > result_thread->allocate_threadObj(thread_group, "fromThreadPool", false, > THREAD); > if (HAS_PENDING_EXCEPTION) { > CLEAR_PENDING_EXCEPTION; > // cleanup outside the handle mark. > attach_failed = true; > } > } > if(attach_failed){ > delete result_thread; > result_thread = NULL; > } > } else { > delete result_thread; > result_thread = NULL; > } > } > return result_thread; > } > > > Now I've seen different errors with different configurations... > - inverted locks > - SIGSEV (that was before splitting the init into 2 parts...) > - looping forever with TracePoolGen/TraceThreadEvents set... > creating thread 0x8084800 > creating thread 0x8086400 > creating thread 0x8088000 > creating thread 0x8089c00 > creating thread 0x808b800 > creating thread 0x808d400 > creating thread 0x808f000 > creating thread 0x8090c00 > creating thread 0x8092800 > creating thread 0x8094800 > Thread::set priority 0x0806d400 [4d6c] main (prio: 5) > creating thread 0x80d4800 > Thread::set priority 0x080d4800 [4d83] Reference Handler (prio: 10) > Thread::start 0x080d4800 [4d83] Reference Handler (prio: 10) > creating thread 0x80d6000 > .. and here nothing happens anymore > - and the current one: > Internal Error > > Could somebody maybe point me in the right direction, or even hand out a > correct piece of code ? It seems that I am missing important parts, but > I have no clue what to look for.. and before I smash my monitor, I > rather give in and try to continue at a different end of my project.. > > Thanks & Regards, > > Peter > > > > From David.Holmes at Sun.COM Tue Nov 6 14:31:05 2007 From: David.Holmes at Sun.COM (David Holmes - Sun Microsystems) Date: Wed, 07 Nov 2007 08:31:05 +1000 Subject: Creating a JavaThread and its java.lang.Thread oop In-Reply-To: References: Message-ID: <4730EB29.9010202@sun.com> Peter, I agree with Tom. Initializing the pool should happen right at the very end of create_vm - as that should be the safest place. If you need to move it earlier then you can try that one step at a time later. Your thread creation needs to be a combination, but variation, of attach_current_thread and JVM_StartThread. With attach_current_thread you have an existing native thread and need to create the JavaThread and java.lang.Thread. With JVM_StartThread you have a java.lang.Thread and need to create the JavaThread and native thread. With your pool you have nothing to start with so you need to: - create the java.lang.Thread object - invoke JVM_StartThread on it (logically not by an actual call) which will create the JavaThread and native thread At present (and I haven't looked too deeply) it seems to me that you might be trying to create the new threads all associated with the current native thread. Create a raw java.lang.Thread first and then "start" it. BTW if you initialize the pool during VM initialization then you shouldn't need to worry about any locking - the main thread is the only one that will be interacting with the pool. Hope this helps. David Holmes Peter Helfer said the following on 7/11/07 03:04 AM: > Hi all > > I am still trying to get my thread pool, so far without real luck... ok, > so here is how I'm trying to achieve it: > > - I'm preparing my own subsystem from runtime/init.cpp, which creates > some runtime-code etc.. > - When starting up (runtime/thread.cpp: Threads::create_vm(..) ), I'm > starting the thread pool > after everything is done, shortly before vm_init_completed is set, so > I can use Universe::... > - I've been seeing a lot of different assertion errors, so I'm now > completely exhausted :-( > > [create_vm(..)]: > initialize_class(vmSymbolHandles::java_lang_Compiler(), CHECK_0); > reset_vm_info_property(CHECK_0); > threadpool_init2(); // HERE COMES THE THREAD POOL > quicken_jni_functions(); > set_init_completed(); > HS_DTRACE_PROBE(hotspot, vm__init__end); > Management::record_vm_init_completed(); > > Now in this function I'm calling the constructor to it which calls the > initialize() function: > > void Threadpool::initialize(){ // is called indirectly by > threadpool_init2(); > assert(Threadpool::get_SpinWaitEntry() != 0, "the spin wait > entry is missing!"); > Threadpool::might_be_available = 0; // static peek variable > Threadpool::threadID = 1; //static var > > // MutexLocker ml(ThreadPool); - > /* THIS CAUSED SOME LOCK INVERSION.. > * it is skipped, but still safe, as all clients to this pool > first ask might_be_available > 0, which > * isnt the case. But it would be nicer.. > */ > > if(TracePoolGen) FlagSetting fs(TraceThreadEvents, true); > for(int i = 0; i < _size; i++){ > _pool[i] = > createThread((ThreadFunction)Speculation::get_SpinWaitEntry()); > Thread::start(_pool[i]); // and start it immediately > } > Threadpool::might_be_available = _size; > } > > This now calls the real creator: > > JavaThread* Threadpool::createThread(ThreadFunction entrypoint){ > EXCEPTION_MARK; // creates a ExceptionMark and THREAD > JavaThread* result_thread = NULL; > { > result_thread = new JavaThread(entrypoint); //size = 0 > if (result_thread->osthread() != NULL) { > oop group = Universe::main_thread_group(); > bool attach_failed = false; > { > HandleMark hm(THREAD); > Handle thread_group(THREAD, group); > > result_thread->allocate_threadObj(thread_group, "fromThreadPool", false, > THREAD); > if (HAS_PENDING_EXCEPTION) { > CLEAR_PENDING_EXCEPTION; > // cleanup outside the handle mark. > attach_failed = true; > } > } > if(attach_failed){ > delete result_thread; > result_thread = NULL; > } > } else { > delete result_thread; > result_thread = NULL; > } > } > return result_thread; > } > > > Now I've seen different errors with different configurations... > - inverted locks > - SIGSEV (that was before splitting the init into 2 parts...) > - looping forever with TracePoolGen/TraceThreadEvents set... > creating thread 0x8084800 > creating thread 0x8086400 > creating thread 0x8088000 > creating thread 0x8089c00 > creating thread 0x808b800 > creating thread 0x808d400 > creating thread 0x808f000 > creating thread 0x8090c00 > creating thread 0x8092800 > creating thread 0x8094800 > Thread::set priority 0x0806d400 [4d6c] main (prio: 5) > creating thread 0x80d4800 > Thread::set priority 0x080d4800 [4d83] Reference Handler (prio: 10) > Thread::start 0x080d4800 [4d83] Reference Handler (prio: 10) > creating thread 0x80d6000 > .. and here nothing happens anymore > - and the current one: > Internal Error > (/home/peterh/workspace/openjdk/hotspot/src/share/vm/runtime/handles.cpp:40), > pid=18280, tid=3084282768 > # Error: assert(thread == Thread::current(),"sanity check") > > > Could somebody maybe point me in the right direction, or even hand out a > correct piece of code ? It seems that I am missing important parts, but > I have no clue what to look for.. and before I smash my monitor, I > rather give in and try to continue at a different end of my project.. > > Thanks & Regards, > > Peter > > > > From peter.helfer.java at gmail.com Wed Nov 7 15:03:43 2007 From: peter.helfer.java at gmail.com (Peter Helfer) Date: Thu, 8 Nov 2007 00:03:43 +0100 Subject: Code generation Message-ID: Hi all! After solving the thread-thing finally, i'm off to testing of my generated code.. after I figured out that I need to commit a CodeBuffer into a CodeBlob, I thought a RuntimeStub would match my needs; however to generate the RuntimeStub, I need to know the frame_complete as well as the frame size, and the OopMapSet for this 'static constructor': static RuntimeStub* new_runtime_stub( const char* stub_name, CodeBuffer* cb, int frame_complete, int frame_size, OopMapSet* oop_maps, bool caller_must_gc_arguments ); - when using it with a CodeBuffer generated before; and return stub->entry_point(), is that thing completely working, i.e. I dont have to care about, where it is allocated (does this go into a CodeCache ?), and about any relocations ? I just had some assertion problems with the relocs -- is there anything special to look for when generating code ? - the frame_complete is the offset of the code to the point, where this frame is at its maximum size, without arguments for the next invocation ? - the OopMapSet (my usual friend) - I am still missing that kind of good explanation for the OopMaps, and how to use them properly... especially that part here: OopMap* map = new OopMap(framesize, 0); oop_maps->add_gc_map(__ pc() - start, map); // this means that 0 oops have to be handled specially in this part, right ? Now lets assume I really wanted to put some Oops in my frame, I would want to use map->set_oop( VMReg local); - this VMReg can be used to depict either a real register or a location on stack, right ? enum { my_second_oop my_first_oop rbp return_pc framesize } could be a fictious frame layout; now how do I tell OopMap to include my_first_oop and my_second_oop ? thanks, Peter From Steve.Goldman at Sun.COM Thu Nov 8 07:13:08 2007 From: Steve.Goldman at Sun.COM (steve goldman) Date: Thu, 08 Nov 2007 10:13:08 -0500 Subject: Code generation In-Reply-To: References: Message-ID: <47332784.4060004@sun.com> Peter Helfer wrote: > Hi all! > > After solving the thread-thing finally, i'm off to testing of my > generated code.. after I figured out that I need to commit a > CodeBuffer into a CodeBlob, I thought a RuntimeStub would match my > needs; however to generate the RuntimeStub, I need to know the > frame_complete as well as the frame size, and the OopMapSet for this > 'static constructor': > > static RuntimeStub* new_runtime_stub( > const char* stub_name, > CodeBuffer* cb, > int frame_complete, > int frame_size, > OopMapSet* oop_maps, > bool caller_must_gc_arguments > ); > > - when using it with a CodeBuffer generated before; and return > stub->entry_point(), is that thing completely working, i.e. I dont > have to care about, where it is allocated (does this go into a > CodeCache ?), yes. and about any relocations ? I just had some assertion > problems with the relocs -- is there anything special to look for when > generating code ? Don't really understand what was the assertion? > > - the frame_complete is the offset of the code to the point, where > this frame is at its maximum size, without arguments for the next > invocation ? This is the offset where sp is stable (and fp if any is also stable). It is only used for trying to make the async call trace more stable. > > - the OopMapSet (my usual friend) - I am still missing that kind of > good explanation for the OopMaps, and how to use them properly... > especially that part here: > > OopMap* map = new OopMap(framesize, 0); > oop_maps->add_gc_map(__ pc() - start, map); > // this means that 0 oops have to be handled specially in this part, right ? > > Now lets assume I really wanted to put some Oops in my frame, I would > want to use > map->set_oop( VMReg local); - this VMReg can be used to depict either > a real register or a location on stack, right ? yes. > > enum { > my_second_oop > my_first_oop > rbp > return_pc > framesize > } > > could be a fictious frame layout; now how do I tell OopMap to include > my_first_oop and my_second_oop ? The slots for the oops are in 32bit quantities regardless of whether oops are 32/64bits. Unlike register descriptions oops only need a single oopmap to describe them (I'm going to fix that register thing one day...). So you'd do map->set_oop(VMRegImpl::stack2reg(my_first_oop)); ... Once you've generated one of these in the debugger you can do something like "call map->print()" and see that the stack offset is what you think. not that in 64bit world your enum needs 2 enum for each item, e.g. my_second_oop, my_second_oopX If you think understanding the differences between Register/VMReg/OptoReg is ugly and complicated you should have seen it a couple of years ago. :-) Then there's framesize. The framesize in that constructor is measured in words (32bits in 32bit world, 64bits in 64bit world). Oopmaps always deal in frame units of 32bits (the size of a "VMReg") so this can be confusing too. Good places to look at code examples is in SharedRuntime_.cpp where the save/restore registers code is. That code doesn't deal with oops directly only callee save registers but it should help. -- Steve From peter.helfer.java at gmail.com Mon Nov 12 09:01:31 2007 From: peter.helfer.java at gmail.com (Peter Helfer) Date: Mon, 12 Nov 2007 18:01:31 +0100 Subject: Creating a JavaThread and its java.lang.Thread oop In-Reply-To: <4730EB29.9010202@sun.com> References: <4730EB29.9010202@sun.com> Message-ID: Ok.. at first it seemed to work... now I get some NullPointerException from the Thread.(ThreadGroup, String)-Constructor, as it seems - do you see any missing parts ? The pool datastructure is being created somewhere in the way of create_vm(..), as well as some PoolManagerThread to create some new threads. This manager is being notfied totally at the end of create_vm() to start creating threads; so I assume everything is ready to be used... JavaThread* ThreadPool::createThread(ThreadFunction entrypoint, TRAPS){ // 0) arrange the nullResult JavaThread* nullResult = NULL; // 1) create java.lang.Thread klassOop k = SystemDictionary::resolve_or_fail(vmSymbolHandles::java_lang_Thread(), true, CHECK_(nullResult)); instanceKlassHandle klass (THREAD, k); instanceHandle thread_oop = klass->allocate_instance_handle(CHECK_(nullResult)); // 2) use the main thread_group oop thread_group = Universe::main_thread_group(); tty->print("java.lang.Thread object: "); thread_oop()->print(); tty->print("mainThreadGroup: "); thread_group->print(); Handle name = java_lang_String::create_from_str("ThreadPool", CHECK_(nullResult)); tty->print("java.lang.String: "); name()->print(); // 3) call (ThreadGroup g, String name) on the java.lang.Threadobject JavaValue result(T_VOID); // Thread gets assigned specified name and null target JavaCalls::call_special(&result, thread_oop, klass, vmSymbolHandles::object_initializer_name(), vmSymbolHandles::threadgroup_string_void_signature(), thread_group, // Argument 1 name, // Argument 2 THREAD); if (HAS_PENDING_EXCEPTION) { oop ex = PENDING_EXCEPTION; tty->print("EARLIER------------------"); ex->print(); return nullResult; } // 4) now create the JavaThread (this part comes from prims/jvm.cpp::JVM_StartThread JavaThread *native_thread = NULL; //bool throw_illegal_thread_state = false; { // Ensure that the C++ Thread and OSThread structures aren't freed before we operate. MutexLocker mu(Threads_lock); native_thread = new JavaThread(entrypoint, 0); } //if (throw_illegal_thread_state) { // THROW(vmSymbols::java_lang_IllegalThreadStateException()); //} assert(native_thread != NULL, "Starting null thread?"); if (native_thread->osthread() == NULL) { // No one should hold a reference to the 'native_thread'. delete native_thread; THROW_MSG_(vmSymbols::java_lang_OutOfMemoryError(), "unable to create new native thread", nullResult); } // 5) now save the JavaThread into java.lang.Thread, set the priority, and save the // java.lang.Thread object into JavaThread as well. java_lang_Thread::set_thread(thread_oop(), native_thread); java_lang_Thread::set_priority(thread_oop(), NormPriority); native_thread->set_threadObj(thread_oop()); // 6) put the new native thread into Threads-list -> is done one dequeuing from Pool! //{ // MutexLocker ml(Threads_lock); // Threads::add(native_thread); //} // 7) add this thread to the threadGroup as well KlassHandle group(THREAD, SystemDictionary::threadGroup_klass()); Handle threadObj(THREAD, native_thread->threadObj()); JavaCalls::call_special(&result, thread_group, group, vmSymbolHandles::add_method_name(), vmSymbolHandles::thread_void_signature(), threadObj, // Arg 1 THREAD); if (HAS_PENDING_EXCEPTION) { oop ex = PENDING_EXCEPTION; tty->print("LATER ON------------------"); ex->print(); return nullResult; } // 7) finally return the JavaThread created freshly.. the start happens elsewhere.. return native_thread; } from hotspot.log Notify SpecPoolManager to create threads Start creating the Threads SpecMgr: ready/init/max [0, 0, 10]: Adding 10 threads java.lang.Thread object: java.lang.Thread - klass: 'java/lang/Thread' fields: - private 'name' '[C' @56 - private 'priority' 'I' @40 - private 'threadQ' 'Ljava/lang/Thread;' @60 - private 'eetop' 'J' @8 - private 'single_step' 'Z' @48 - private 'daemon' 'Z' @49 - private 'speculative' 'Z' @50 - private 'stillborn' 'Z' @51 - private 'target' 'Ljava/lang/Runnable;' @64 - private 'group' 'Ljava/lang/ThreadGroup;' @68 - private 'contextClassLoader' 'Ljava/lang/ClassLoader;' @72 - private 'inheritedAccessControlContext' 'Ljava/security/AccessControlContext;' @76 - 'threadLocals' 'Ljava/lang/ThreadLocal$ThreadLocalMap;' @80 - 'inheritableThreadLocals' 'Ljava/lang/ThreadLocal$ThreadLocalMap;' @84 - private 'stackSize' 'J' @16 - private 'nativeParkEventPointer' 'J' @24 - private 'tid' 'J' @32 - private 'threadStatus' 'I' @44 - volatile 'parkBlocker' 'Ljava/lang/Object;' @88 - private volatile 'blocker' 'Lsun/nio/ch/Interruptible;' @92 - private 'blockerLock' 'Ljava/lang/Object;' @96 - private 'stopBeforeStart' 'Z' @52 - private 'throwableFromStop' 'Ljava/lang/Throwable;' @100 - private volatile 'uncaughtExceptionHandler' 'Ljava/lang/Thread$UncaughtExceptionHandler;' @104 mainThreadGroup: java.lang.ThreadGroup - klass: 'java/lang/ThreadGroup' fields: - 'parent' 'Ljava/lang/ThreadGroup;' @28 - 'name' 'Ljava/lang/String;' @32 - 'maxPriority' 'I' @8 - 'destroyed' 'Z' @24 - 'daemon' 'Z' @25 - 'vmAllowSuspension' 'Z' @26 - 'nUnstartedThreads' 'I' @12 - 'nthreads' 'I' @16 - 'threads' '[Ljava/lang/Thread;' @36 - 'ngroups' 'I' @20 - 'groups' '[Ljava/lang/ThreadGroup;' @40 java.lang.String: java.lang.String - klass: 'java/lang/String' string: "Speculative" EARLIER------------------java.lang.NullPointerException - klass: 'java/lang/NullPointerException' fields: - private transient 'backtrace' 'Ljava/lang/Object;' @8 - private 'detailMessage' 'Ljava/lang/String;' @12 - private 'cause' 'Ljava/lang/Throwable;' @16 - private 'stackTrace' '[Ljava/lang/StackTraceElement;' @20 2007/11/6, David Holmes - Sun Microsystems : > > Peter, > > I agree with Tom. Initializing the pool should happen right at the very > end of create_vm - as that should be the safest place. If you need to > move it earlier then you can try that one step at a time later. > > Your thread creation needs to be a combination, but variation, of > attach_current_thread and JVM_StartThread. With attach_current_thread > you have an existing native thread and need to create the JavaThread and > java.lang.Thread. With JVM_StartThread you have a java.lang.Thread and > need to create the JavaThread and native thread. With your pool you have > nothing to start with so you need to: > - create the java.lang.Thread object > - invoke JVM_StartThread on it (logically not by an actual call) which > will create the JavaThread and native thread > > At present (and I haven't looked too deeply) it seems to me that you > might be trying to create the new threads all associated with the > current native thread. Create a raw java.lang.Thread first and then > "start" it. > > BTW if you initialize the pool during VM initialization then you > shouldn't need to worry about any locking - the main thread is the only > one that will be interacting with the pool. > > Hope this helps. > > David Holmes > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20071112/3f950665/attachment.html From Thomas.Rodriguez at Sun.COM Mon Nov 12 15:34:49 2007 From: Thomas.Rodriguez at Sun.COM (Tom Rodriguez) Date: Mon, 12 Nov 2007 15:34:49 -0800 Subject: Creating a JavaThread and its java.lang.Thread oop In-Reply-To: References: <4730EB29.9010202@sun.com> Message-ID: <4738E319.5050602@sun.com> Use java_lang_Throwable::print_stack_trace(ex, tty) to get a stack trace for the exception. I suspect that will answer your question. By the way, you shouldn't have to declare nullResult to return NULL. CHECK_NULL is the normal way to do that. tom Peter Helfer wrote: > Ok.. at first it seemed to work... now I get some NullPointerException > from the Thread.(ThreadGroup, String)-Constructor, as it seems - > do you see any missing parts ? > > The pool datastructure is being created somewhere in the way of > create_vm(..), as well as some PoolManagerThread to create some new > threads. This manager is being notfied totally at the end of create_vm() > to start creating threads; so I assume everything is ready to be used... > > > > JavaThread* ThreadPool::createThread(ThreadFunction entrypoint, TRAPS){ > // 0) arrange the nullResult > JavaThread* nullResult = NULL; > > // 1) create java.lang.Thread > klassOop k = > SystemDictionary::resolve_or_fail(vmSymbolHandles::java_lang_Thread(), > true, CHECK_(nullResult)); > instanceKlassHandle klass (THREAD, k); > instanceHandle thread_oop = > klass->allocate_instance_handle(CHECK_(nullResult)); > > // 2) use the main thread_group > oop thread_group = Universe::main_thread_group(); > > tty->print("java.lang.Thread object: "); > thread_oop()->print(); > tty->print("mainThreadGroup: "); > thread_group->print(); > Handle name = java_lang_String::create_from_str("ThreadPool", > CHECK_(nullResult)); > tty->print("java.lang.String: "); > name()->print(); > > // 3) call (ThreadGroup g, String name) on the java.lang.Thread > object > JavaValue result(T_VOID); > // Thread gets assigned specified name and null target > JavaCalls::call_special(&result, > thread_oop, > klass, > vmSymbolHandles::object_initializer_name(), > > vmSymbolHandles::threadgroup_string_void_signature(), > thread_group, // Argument 1 > name, // Argument 2 > THREAD); > > if (HAS_PENDING_EXCEPTION) { > oop ex = PENDING_EXCEPTION; > tty->print("EARLIER------------------"); > ex->print(); > return nullResult; > } > > // 4) now create the JavaThread (this part comes from > prims/jvm.cpp::JVM_StartThread > JavaThread *native_thread = NULL; > //bool throw_illegal_thread_state = false; > { > // Ensure that the C++ Thread and OSThread structures aren't > freed before we operate. > MutexLocker mu(Threads_lock); > native_thread = new JavaThread(entrypoint, 0); > } > //if (throw_illegal_thread_state) { > // THROW(vmSymbols::java_lang_IllegalThreadStateException()); > //} > assert(native_thread != NULL, "Starting null thread?"); > if (native_thread->osthread() == NULL) { > // No one should hold a reference to the 'native_thread'. > delete native_thread; > THROW_MSG_(vmSymbols::java_lang_OutOfMemoryError(), "unable to > create new native thread", nullResult); > } > > // 5) now save the JavaThread into java.lang.Thread, set the priority, > and save the > // java.lang.Thread object into JavaThread as well. > java_lang_Thread::set_thread(thread_oop(), native_thread); > java_lang_Thread::set_priority(thread_oop(), NormPriority); > native_thread->set_threadObj(thread_oop()); > > // 6) put the new native thread into Threads-list -> is done one > dequeuing from Pool! > //{ > // MutexLocker ml(Threads_lock); > // Threads::add(native_thread); > //} > > // 7) add this thread to the threadGroup as well > KlassHandle group(THREAD, SystemDictionary::threadGroup_klass()); > Handle threadObj(THREAD, native_thread->threadObj()); > > JavaCalls::call_special(&result, > thread_group, > group, > vmSymbolHandles::add_method_name(), > vmSymbolHandles::thread_void_signature(), > threadObj, // Arg 1 > THREAD); > > if (HAS_PENDING_EXCEPTION) { > oop ex = PENDING_EXCEPTION; > tty->print("LATER ON------------------"); > ex->print(); > return nullResult; > } > > // 7) finally return the JavaThread created freshly.. the start > happens elsewhere.. > return native_thread; > > } > > > from hotspot.log > > Notify SpecPoolManager to create threads > > Start creating the Threads > SpecMgr: ready/init/max [0, 0, 10]: Adding 10 threads > java.lang.Thread object: java.lang.Thread > - klass: 'java/lang/Thread' > fields: > - private 'name' '[C' @56 > - private 'priority' 'I' @40 > - private 'threadQ' 'Ljava/lang/Thread;' @60 > - private 'eetop' 'J' @8 > - private 'single_step' 'Z' @48 > - private 'daemon' 'Z' @49 > - private 'speculative' 'Z' @50 > - private 'stillborn' 'Z' @51 > - private 'target' 'Ljava/lang/Runnable;' @64 > - private 'group' 'Ljava/lang/ThreadGroup;' @68 > - private 'contextClassLoader' > 'Ljava/lang/ClassLoader;' @72 > - private 'inheritedAccessControlContext' > 'Ljava/security/AccessControlContext;' @76 > - 'threadLocals' > 'Ljava/lang/ThreadLocal$ThreadLocalMap;' @80 > - 'inheritableThreadLocals' > 'Ljava/lang/ThreadLocal$ThreadLocalMap;' @84 > - private 'stackSize' 'J' @16 > - private 'nativeParkEventPointer' 'J' @24 > - private 'tid' 'J' @32 > - private 'threadStatus' 'I' @44 > - volatile 'parkBlocker' 'Ljava/lang/Object;' @88 > - private volatile 'blocker' > 'Lsun/nio/ch/Interruptible;' @92 > - private 'blockerLock' 'Ljava/lang/Object;' @96 > - private 'stopBeforeStart' 'Z' @52 > - private 'throwableFromStop' > 'Ljava/lang/Throwable;' @100 > - private volatile 'uncaughtExceptionHandler' > 'Ljava/lang/Thread$UncaughtExceptionHandler;' @104 > mainThreadGroup: java.lang.ThreadGroup > - klass: 'java/lang/ThreadGroup' > fields: > - 'parent' 'Ljava/lang/ThreadGroup;' @28 > - 'name' 'Ljava/lang/String;' @32 > - 'maxPriority' 'I' @8 > - 'destroyed' 'Z' @24 > - 'daemon' 'Z' @25 > - 'vmAllowSuspension' 'Z' @26 > - 'nUnstartedThreads' 'I' @12 > - 'nthreads' 'I' @16 > - 'threads' '[Ljava/lang/Thread;' @36 > - 'ngroups' 'I' @20 > - 'groups' '[Ljava/lang/ThreadGroup;' @40 > java.lang.String: java.lang.String > - klass: 'java/lang/String' > string: "Speculative" > EARLIER------------------java.lang.NullPointerException > - klass: 'java/lang/NullPointerException' > fields: > - private transient 'backtrace' > 'Ljava/lang/Object;' @8 > - private 'detailMessage' 'Ljava/lang/String;' @12 > - private 'cause' 'Ljava/lang/Throwable;' @16 > - private 'stackTrace' > '[Ljava/lang/StackTraceElement;' @20 > > > > > > > > > 2007/11/6, David Holmes - Sun Microsystems >: > > Peter, > > I agree with Tom. Initializing the pool should happen right at the very > end of create_vm - as that should be the safest place. If you need to > move it earlier then you can try that one step at a time later. > > Your thread creation needs to be a combination, but variation, of > attach_current_thread and JVM_StartThread. With attach_current_thread > you have an existing native thread and need to create the JavaThread > and > java.lang.Thread. With JVM_StartThread you have a java.lang.Thread and > need to create the JavaThread and native thread. With your pool you have > nothing to start with so you need to: > - create the java.lang.Thread object > - invoke JVM_StartThread on it (logically not by an actual call) which > will create the JavaThread and native thread > > At present (and I haven't looked too deeply) it seems to me that you > might be trying to create the new threads all associated with the > current native thread. Create a raw java.lang.Thread first and then > "start" it. > > BTW if you initialize the pool during VM initialization then you > shouldn't need to worry about any locking - the main thread is the only > one that will be interacting with the pool. > > Hope this helps. > > David Holmes > > > From David.Holmes at Sun.COM Mon Nov 12 16:20:48 2007 From: David.Holmes at Sun.COM (David Holmes - Sun Microsystems) Date: Tue, 13 Nov 2007 10:20:48 +1000 Subject: Creating a JavaThread and its java.lang.Thread oop In-Reply-To: <4738E319.5050602@sun.com> References: <4730EB29.9010202@sun.com> <4738E319.5050602@sun.com> Message-ID: <4738EDE0.8070707@sun.com> And note that the thread doing this construction must itself already be an attached Java thread - just checking :) David Holmes Tom Rodriguez said the following on 13/11/07 09:34 AM: > Use java_lang_Throwable::print_stack_trace(ex, tty) to get a stack trace > for the exception. I suspect that will answer your question. > > By the way, you shouldn't have to declare nullResult to return NULL. > CHECK_NULL is the normal way to do that. > > tom > > Peter Helfer wrote: >> Ok.. at first it seemed to work... now I get some NullPointerException >> from the Thread.(ThreadGroup, String)-Constructor, as it seems - >> do you see any missing parts ? >> >> The pool datastructure is being created somewhere in the way of >> create_vm(..), as well as some PoolManagerThread to create some new >> threads. This manager is being notfied totally at the end of >> create_vm() to start creating threads; so I assume everything is ready >> to be used... >> >> >> >> JavaThread* ThreadPool::createThread(ThreadFunction entrypoint, TRAPS){ >> // 0) arrange the nullResult >> JavaThread* nullResult = NULL; >> >> // 1) create java.lang.Thread >> klassOop k = >> SystemDictionary::resolve_or_fail(vmSymbolHandles::java_lang_Thread(), >> true, CHECK_(nullResult)); >> instanceKlassHandle klass (THREAD, k); >> instanceHandle thread_oop = >> klass->allocate_instance_handle(CHECK_(nullResult)); >> >> // 2) use the main thread_group >> oop thread_group = Universe::main_thread_group(); >> >> tty->print("java.lang.Thread object: "); >> thread_oop()->print(); >> tty->print("mainThreadGroup: "); >> thread_group->print(); >> Handle name = java_lang_String::create_from_str("ThreadPool", >> CHECK_(nullResult)); >> tty->print("java.lang.String: "); >> name()->print(); >> >> // 3) call (ThreadGroup g, String name) on the >> java.lang.Thread object >> JavaValue result(T_VOID); >> // Thread gets assigned specified name and null target >> JavaCalls::call_special(&result, >> thread_oop, >> klass, >> vmSymbolHandles::object_initializer_name(), >> >> vmSymbolHandles::threadgroup_string_void_signature(), >> thread_group, // Argument 1 >> name, // Argument 2 >> THREAD); >> >> if (HAS_PENDING_EXCEPTION) { >> oop ex = PENDING_EXCEPTION; >> tty->print("EARLIER------------------"); >> ex->print(); >> return nullResult; >> } >> >> // 4) now create the JavaThread (this part comes from >> prims/jvm.cpp::JVM_StartThread >> JavaThread *native_thread = NULL; >> //bool throw_illegal_thread_state = false; >> { >> // Ensure that the C++ Thread and OSThread structures aren't >> freed before we operate. >> MutexLocker mu(Threads_lock); >> native_thread = new JavaThread(entrypoint, 0); >> } >> //if (throw_illegal_thread_state) { >> // THROW(vmSymbols::java_lang_IllegalThreadStateException()); >> //} >> assert(native_thread != NULL, "Starting null thread?"); >> if (native_thread->osthread() == NULL) { >> // No one should hold a reference to the 'native_thread'. >> delete native_thread; >> THROW_MSG_(vmSymbols::java_lang_OutOfMemoryError(), "unable >> to create new native thread", nullResult); >> } >> >> // 5) now save the JavaThread into java.lang.Thread, set the >> priority, and save the >> // java.lang.Thread object into JavaThread as well. >> java_lang_Thread::set_thread(thread_oop(), native_thread); >> java_lang_Thread::set_priority(thread_oop(), NormPriority); >> native_thread->set_threadObj(thread_oop()); >> >> // 6) put the new native thread into Threads-list -> is done one >> dequeuing from Pool! >> //{ >> // MutexLocker ml(Threads_lock); >> // Threads::add(native_thread); >> //} >> >> // 7) add this thread to the threadGroup as well >> KlassHandle group(THREAD, SystemDictionary::threadGroup_klass()); >> Handle threadObj(THREAD, native_thread->threadObj()); >> >> JavaCalls::call_special(&result, >> thread_group, >> group, >> vmSymbolHandles::add_method_name(), >> vmSymbolHandles::thread_void_signature(), >> threadObj, // Arg 1 >> THREAD); >> >> if (HAS_PENDING_EXCEPTION) { >> oop ex = PENDING_EXCEPTION; >> tty->print("LATER ON------------------"); >> ex->print(); >> return nullResult; >> } >> >> // 7) finally return the JavaThread created freshly.. the start >> happens elsewhere.. >> return native_thread; >> >> } >> >> >> from hotspot.log >> >> Notify SpecPoolManager to create threads >> >> Start creating the Threads >> SpecMgr: ready/init/max [0, 0, 10]: Adding 10 threads >> java.lang.Thread object: java.lang.Thread >> - klass: 'java/lang/Thread' >> fields: >> - private 'name' '[C' @56 >> - private 'priority' 'I' @40 >> - private 'threadQ' 'Ljava/lang/Thread;' @60 >> - private 'eetop' 'J' @8 >> - private 'single_step' 'Z' @48 >> - private 'daemon' 'Z' @49 >> - private 'speculative' 'Z' @50 >> - private 'stillborn' 'Z' @51 >> - private 'target' 'Ljava/lang/Runnable;' @64 >> - private 'group' 'Ljava/lang/ThreadGroup;' @68 >> - private 'contextClassLoader' >> 'Ljava/lang/ClassLoader;' @72 >> - private 'inheritedAccessControlContext' >> 'Ljava/security/AccessControlContext;' @76 >> - 'threadLocals' >> 'Ljava/lang/ThreadLocal$ThreadLocalMap;' @80 >> - 'inheritableThreadLocals' >> 'Ljava/lang/ThreadLocal$ThreadLocalMap;' @84 >> - private 'stackSize' 'J' @16 >> - private 'nativeParkEventPointer' 'J' @24 >> - private 'tid' 'J' @32 >> - private 'threadStatus' 'I' @44 >> - volatile 'parkBlocker' 'Ljava/lang/Object;' @88 >> - private volatile 'blocker' >> 'Lsun/nio/ch/Interruptible;' @92 >> - private 'blockerLock' 'Ljava/lang/Object;' @96 >> - private 'stopBeforeStart' 'Z' @52 >> - private 'throwableFromStop' >> 'Ljava/lang/Throwable;' @100 >> - private volatile 'uncaughtExceptionHandler' >> 'Ljava/lang/Thread$UncaughtExceptionHandler;' @104 >> mainThreadGroup: java.lang.ThreadGroup >> - klass: 'java/lang/ThreadGroup' >> fields: >> - 'parent' 'Ljava/lang/ThreadGroup;' @28 >> - 'name' 'Ljava/lang/String;' @32 >> - 'maxPriority' 'I' @8 >> - 'destroyed' 'Z' @24 >> - 'daemon' 'Z' @25 >> - 'vmAllowSuspension' 'Z' @26 >> - 'nUnstartedThreads' 'I' @12 >> - 'nthreads' 'I' @16 >> - 'threads' '[Ljava/lang/Thread;' @36 >> - 'ngroups' 'I' @20 >> - 'groups' '[Ljava/lang/ThreadGroup;' @40 >> java.lang.String: java.lang.String >> - klass: 'java/lang/String' >> string: "Speculative" >> EARLIER------------------java.lang.NullPointerException >> - klass: 'java/lang/NullPointerException' >> fields: >> - private transient 'backtrace' >> 'Ljava/lang/Object;' @8 >> - private 'detailMessage' 'Ljava/lang/String;' @12 >> - private 'cause' 'Ljava/lang/Throwable;' @16 >> - private 'stackTrace' >> '[Ljava/lang/StackTraceElement;' @20 >> >> >> >> >> >> >> >> >> 2007/11/6, David Holmes - Sun Microsystems > >: >> >> Peter, >> >> I agree with Tom. Initializing the pool should happen right at the >> very >> end of create_vm - as that should be the safest place. If you need to >> move it earlier then you can try that one step at a time later. >> >> Your thread creation needs to be a combination, but variation, of >> attach_current_thread and JVM_StartThread. With attach_current_thread >> you have an existing native thread and need to create the JavaThread >> and >> java.lang.Thread. With JVM_StartThread you have a java.lang.Thread >> and >> need to create the JavaThread and native thread. With your pool >> you have >> nothing to start with so you need to: >> - create the java.lang.Thread object >> - invoke JVM_StartThread on it (logically not by an actual call) >> which >> will create the JavaThread and native thread >> >> At present (and I haven't looked too deeply) it seems to me that you >> might be trying to create the new threads all associated with the >> current native thread. Create a raw java.lang.Thread first and then >> "start" it. >> >> BTW if you initialize the pool during VM initialization then you >> shouldn't need to worry about any locking - the main thread is the >> only >> one that will be interacting with the pool. >> >> Hope this helps. >> >> David Holmes >> >> >> From peter.helfer.java at gmail.com Wed Nov 14 05:19:25 2007 From: peter.helfer.java at gmail.com (Peter Helfer) Date: Wed, 14 Nov 2007 14:19:25 +0100 Subject: Monitor destructor... Message-ID: Hi all I've got some funny error I'm running into: In order to see what deadlocks I'm creating, I extended the Monitor (Mutex) constructor to copy the "name" parameter, instead of keeping it at "UNKNOWN" (as of ClearMonitor). class Monitor{ ... char * _name; // Name of mutex REMOVED const! .. }; static const char* Monitor::_default = "UNKNOWN"; void Monitor::ClearMonitor (Monitor * m) { m->_owner = NULL ; m->_snuck = false ; m->_name = (char*) Monitor::_default; m->_LockWord.FullWord = 0 ; m->_EntryList = NULL ; m->_OnDeck = NULL ; m->_WaitSet = NULL ; m->_WaitLock[0] = 0 ; } Monitor::Monitor() { ClearMonitor(this); } Monitor::Monitor (int Rank, const char * name, bool allow_vm_block) { ClearMonitor (this) ; #ifdef ASSERT if(name){ int len = strnlen(name,63); _name = NEW_C_HEAP_ARRAY(char,len+1); assert(_name, "Not enough mem"); strncpy(_name, name, len+1); } _allow_vm_block = allow_vm_block; _rank = Rank ; #endif } Monitor::~Monitor() { assert ((UNS(_owner)|UNS(_LockWord.FullWord)|UNS(_EntryList)|UNS(_WaitSet)|UNS(_OnDeck)) == 0, "") ; #ifdef ASSERT if(_name != _default){ FreeHeap(_name); } #endif } This is what I am seeing: it only doesnt work in this situation.. The easy work around is to remove the code again, but thats actually not the idea.. :-) /home/peterh/workspace/openjdk/control/build/linux-i586-debug/bin/java -client -Xmx881m -Xms128m -XX:PermSize=32m -XX:MaxPermSize=160m -classpath /home/peterh/workspace/openjdk/control/build/linux-i586-debug/tmp/meta-index BuildMetaIndex -o meta-index *.jar VM option 'PermSize=32m' VM option 'MaxPermSize=160m' ## nof_mallocs = 19744, nof_frees = 12802 ## memory stomp: byte at 0x8189648 in front of object 0x8189660 ### previous object (not sure if correct): 0x8189648 (73 bytes) ### next object (not sure if correct): 0x8189694 (1700358998 bytes) # To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/os.cpp:544 # # An unexpected error has been detected by Java Runtime Environment: # # Internal Error (/home/peterh/workspace/openjdk/hotspot/src/share/vm/runtime/os.cpp:544), pid=7146, tid=2209852304 # Error: memory stomping error # -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20071114/19f17fab/attachment.html From Keith.McGuigan at Sun.COM Wed Nov 14 05:41:55 2007 From: Keith.McGuigan at Sun.COM (Keith McGuigan) Date: Wed, 14 Nov 2007 08:41:55 -0500 Subject: Monitor destructor... In-Reply-To: References: Message-ID: <473AFB23.5080505@sun.com> Peter - Just a guess here, but I see that there's two constructors for Mutex(), one which sets _name to what you want, but the other one (the no-arg one) doesn't appear to initialize _name at all. If you have a Mutex that was created via that constructor, _name will be junk and your deallocation in the desstructor will fail. (By the way, what you have works but it would probably be better to deallocate the memory with FREE_C_HEAP_ARRAY(char,_name), to keep the name pairing). -- - Keith Peter Helfer wrote: > Hi all > > I've got some funny error I'm running into: In order to see what > deadlocks I'm creating, I extended the Monitor (Mutex) constructor to > copy the "name" parameter, instead of keeping it at "UNKNOWN" (as of > ClearMonitor). > > class Monitor{ > ... > char * _name; // Name of mutex REMOVED const! > .. > }; > static const char* Monitor::_default = "UNKNOWN"; > > > void Monitor::ClearMonitor (Monitor * m) { > m->_owner = NULL ; > m->_snuck = false ; > m->_name = (char*) Monitor::_default; > m->_LockWord.FullWord = 0 ; > m->_EntryList = NULL ; > m->_OnDeck = NULL ; > m->_WaitSet = NULL ; > m->_WaitLock[0] = 0 ; > } > > Monitor::Monitor() { ClearMonitor(this); } > > Monitor::Monitor (int Rank, const char * name, bool allow_vm_block) { > ClearMonitor (this) ; > #ifdef ASSERT > if(name){ > int len = strnlen(name,63); > _name = NEW_C_HEAP_ARRAY(char,len+1); > assert(_name, "Not enough mem"); > strncpy(_name, name, len+1); > } > _allow_vm_block = allow_vm_block; > _rank = Rank ; > #endif > } > > Monitor::~Monitor() { > assert > ((UNS(_owner)|UNS(_LockWord.FullWord)|UNS(_EntryList)|UNS(_WaitSet)|UNS(_OnDeck)) > == 0, "") ; > #ifdef ASSERT > if(_name != _default){ > FreeHeap(_name); > } > #endif > } > > This is what I am seeing: it only doesnt work in this situation.. The > easy work around is to remove the code again, but thats actually not the > idea.. :-) > > /home/peterh/workspace/openjdk/control/build/linux-i586-debug/bin/java > -client -Xmx881m -Xms128m -XX:PermSize=32m -XX:MaxPermSize=160m > -classpath > /home/peterh/workspace/openjdk/control/build/linux-i586-debug/tmp/meta-index > BuildMetaIndex -o meta-index *.jar > VM option 'PermSize=32m' > VM option 'MaxPermSize=160m' > ## nof_mallocs = 19744, nof_frees = 12802 > ## memory stomp: byte at 0x8189648 in front of object 0x8189660 > ### previous object (not sure if correct): 0x8189648 (73 bytes) > ### next object (not sure if correct): 0x8189694 (1700358998 bytes) > # To suppress the following error report, specify this argument > # after -XX: or in .hotspotrc: SuppressErrorAt=/os.cpp:544 > # > # An unexpected error has been detected by Java Runtime Environment: > # > # Internal Error > (/home/peterh/workspace/openjdk/hotspot/src/share/vm/runtime/os.cpp:544), > pid=7146, tid=2209852304 > # Error: memory stomping error > # > From Keith.McGuigan at Sun.COM Wed Nov 14 05:43:28 2007 From: Keith.McGuigan at Sun.COM (Keith McGuigan) Date: Wed, 14 Nov 2007 08:43:28 -0500 Subject: Monitor destructor... In-Reply-To: <473AFB23.5080505@sun.com> References: <473AFB23.5080505@sun.com> Message-ID: <473AFB80.7010405@sun.com> Whoops, I take it back - I didn't notice that ClearMonitor() sets this->_name. Ignore my last message... -- - Keith Keith McGuigan wrote: > Peter - > > Just a guess here, but I see that there's two constructors for Mutex(), > one which sets _name to what you want, but the other one (the no-arg > one) doesn't appear to initialize _name at all. If you have a Mutex > that was created via that constructor, _name will be junk and your > deallocation in the desstructor will fail. (By the way, what you have > works but it would probably be better to deallocate the memory with > FREE_C_HEAP_ARRAY(char,_name), to keep the name pairing). > > -- > - Keith > > Peter Helfer wrote: >> Hi all >> >> I've got some funny error I'm running into: In order to see what >> deadlocks I'm creating, I extended the Monitor (Mutex) constructor to >> copy the "name" parameter, instead of keeping it at "UNKNOWN" (as of >> ClearMonitor). >> >> class Monitor{ >> ... >> char * _name; // Name of mutex REMOVED const! >> .. >> }; >> static const char* Monitor::_default = "UNKNOWN"; >> >> >> void Monitor::ClearMonitor (Monitor * m) { >> m->_owner = NULL ; >> m->_snuck = false ; >> m->_name = (char*) Monitor::_default; >> m->_LockWord.FullWord = 0 ; >> m->_EntryList = NULL ; >> m->_OnDeck = NULL ; >> m->_WaitSet = NULL ; >> m->_WaitLock[0] = 0 ; >> } >> >> Monitor::Monitor() { ClearMonitor(this); } >> >> Monitor::Monitor (int Rank, const char * name, bool allow_vm_block) { >> ClearMonitor (this) ; >> #ifdef ASSERT >> if(name){ >> int len = strnlen(name,63); >> _name = NEW_C_HEAP_ARRAY(char,len+1); >> assert(_name, "Not enough mem"); >> strncpy(_name, name, len+1); >> } >> _allow_vm_block = allow_vm_block; >> _rank = Rank ; >> #endif >> } >> >> Monitor::~Monitor() { >> assert >> ((UNS(_owner)|UNS(_LockWord.FullWord)|UNS(_EntryList)|UNS(_WaitSet)|UNS(_OnDeck)) >> == 0, "") ; >> #ifdef ASSERT >> if(_name != _default){ >> FreeHeap(_name); >> } >> #endif >> } >> >> This is what I am seeing: it only doesnt work in this situation.. The >> easy work around is to remove the code again, but thats actually not >> the idea.. :-) >> >> /home/peterh/workspace/openjdk/control/build/linux-i586-debug/bin/java >> -client -Xmx881m -Xms128m -XX:PermSize=32m -XX:MaxPermSize=160m >> -classpath >> /home/peterh/workspace/openjdk/control/build/linux-i586-debug/tmp/meta-index >> BuildMetaIndex -o meta-index *.jar >> VM option 'PermSize=32m' >> VM option 'MaxPermSize=160m' >> ## nof_mallocs = 19744, nof_frees = 12802 >> ## memory stomp: byte at 0x8189648 in front of object 0x8189660 >> ### previous object (not sure if correct): 0x8189648 (73 bytes) >> ### next object (not sure if correct): 0x8189694 (1700358998 bytes) >> # To suppress the following error report, specify this argument >> # after -XX: or in .hotspotrc: SuppressErrorAt=/os.cpp:544 >> # >> # An unexpected error has been detected by Java Runtime Environment: >> # >> # Internal Error >> (/home/peterh/workspace/openjdk/hotspot/src/share/vm/runtime/os.cpp:544), >> pid=7146, tid=2209852304 >> # Error: memory stomping error >> # >> From Thomas.Rodriguez at Sun.COM Wed Nov 14 12:12:12 2007 From: Thomas.Rodriguez at Sun.COM (Tom Rodriguez) Date: Wed, 14 Nov 2007 12:12:12 -0800 Subject: Monitor destructor... In-Reply-To: References: Message-ID: <473B569C.1040406@sun.com> The fact that the names were being lost was recently noticed and a fix is coming for it along with some other locking related changes. I believe the fix declares a 64 byte char array in the monitor class and copies the name into that. It's being done that way since the monitors want padding around them anyway to reduce false sharing. I don't know what's wrong with your changes but fixing it this way would side step your malloc woes. tom Peter Helfer wrote: > Hi all > > I've got some funny error I'm running into: In order to see what > deadlocks I'm creating, I extended the Monitor (Mutex) constructor to > copy the "name" parameter, instead of keeping it at "UNKNOWN" (as of > ClearMonitor). > > class Monitor{ > ... > char * _name; // Name of mutex REMOVED const! > .. > }; > static const char* Monitor::_default = "UNKNOWN"; > > > void Monitor::ClearMonitor (Monitor * m) { > m->_owner = NULL ; > m->_snuck = false ; > m->_name = (char*) Monitor::_default; > m->_LockWord.FullWord = 0 ; > m->_EntryList = NULL ; > m->_OnDeck = NULL ; > m->_WaitSet = NULL ; > m->_WaitLock[0] = 0 ; > } > > Monitor::Monitor() { ClearMonitor(this); } > > Monitor::Monitor (int Rank, const char * name, bool allow_vm_block) { > ClearMonitor (this) ; > #ifdef ASSERT > if(name){ > int len = strnlen(name,63); > _name = NEW_C_HEAP_ARRAY(char,len+1); > assert(_name, "Not enough mem"); > strncpy(_name, name, len+1); > } > _allow_vm_block = allow_vm_block; > _rank = Rank ; > #endif > } > > Monitor::~Monitor() { > assert > ((UNS(_owner)|UNS(_LockWord.FullWord)|UNS(_EntryList)|UNS(_WaitSet)|UNS(_OnDeck)) > == 0, "") ; > #ifdef ASSERT > if(_name != _default){ > FreeHeap(_name); > } > #endif > } > > This is what I am seeing: it only doesnt work in this situation.. The > easy work around is to remove the code again, but thats actually not the > idea.. :-) > > /home/peterh/workspace/openjdk/control/build/linux-i586-debug/bin/java > -client -Xmx881m -Xms128m -XX:PermSize=32m -XX:MaxPermSize=160m > -classpath > /home/peterh/workspace/openjdk/control/build/linux-i586-debug/tmp/meta-index > BuildMetaIndex -o meta-index *.jar > VM option 'PermSize=32m' > VM option 'MaxPermSize=160m' > ## nof_mallocs = 19744, nof_frees = 12802 > ## memory stomp: byte at 0x8189648 in front of object 0x8189660 > ### previous object (not sure if correct): 0x8189648 (73 bytes) > ### next object (not sure if correct): 0x8189694 (1700358998 bytes) > # To suppress the following error report, specify this argument > # after -XX: or in .hotspotrc: SuppressErrorAt=/os.cpp:544 > # > # An unexpected error has been detected by Java Runtime Environment: > # > # Internal Error > (/home/peterh/workspace/openjdk/hotspot/src/share/vm/runtime/os.cpp:544), > pid=7146, tid=2209852304 > # Error: memory stomping error > # > From peter.helfer.java at gmail.com Fri Nov 23 07:43:59 2007 From: peter.helfer.java at gmail.com (Peter Helfer) Date: Fri, 23 Nov 2007 16:43:59 +0100 Subject: figuring out what happenend within the java.lang.Thread. Message-ID: Ok, I'm running again into troubles.. This is what I'm seeing: VM option '+UnlockDiagnosticVMOptions' VM option '-VerifyBeforeExit' PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80DA800 SpecPool::createThread: EXCEPTION OCCURED IN PART1 at java.lang.Thread.(Thread.java:463) PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80DC400 SpecPool::createThread: EXCEPTION OCCURED IN PART3 at java.lang.ThreadGroup.add(ThreadGroup.java:868) PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80DFC00 SpecPool::createThread: EXCEPTION OCCURED IN PART1 at java.lang.Thread.(Thread.java:463) PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80E1C00 SpecPool::createThread: EXCEPTION OCCURED IN PART1 at java.lang.Thread.(Thread.java:463) PARENT[806DC00]: getThread: Linked threads 806DC00 <--> 80E5400 PARENT[806DC00]: getThread: Linked threads 806DC00 <--> 80E3800 Error occurred during initialization of VM java.lang.ExceptionInInitializerError at sun.misc.Launcher.(Launcher.java:71) at sun.misc.Launcher.(Launcher.java:59) at java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java :1322) at java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1304) Caused by: java.lang.NullPointerException at java.net.URLClassLoader.(URLClassLoader.java:589) at sun.misc.Launcher.(Launcher.java:71) at sun.misc.Launcher.(Launcher.java:59) at java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java :1322) at java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1304) Now how can I track or figure out, what has happend within java.lang.Thread. ? - this is just the normal stack trace output, are there any other means ? When using GDB (thx Tom for the script!), I can only see backtraces, which end up in os::raise(), but I'm having issues to set breakpoints, even when using "sharedlibrary libjvm"... Regards, Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20071123/471d5a7a/attachment.html From David.Holmes at Sun.COM Fri Nov 23 13:10:14 2007 From: David.Holmes at Sun.COM (David Holmes - Sun Microsystems) Date: Sat, 24 Nov 2007 07:10:14 +1000 Subject: figuring out what happenend within the java.lang.Thread. In-Reply-To: References: Message-ID: <474741B6.5010305@sun.com> Peter, Try -XX:+TraceExceptions on a non-product build. David Holmes Peter Helfer said the following on 24/11/07 01:43 AM: > Ok, I'm running again into troubles.. > > This is what I'm seeing: > > VM option '+UnlockDiagnosticVMOptions' > VM option '-VerifyBeforeExit' > PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80DA800 > SpecPool::createThread: EXCEPTION OCCURED IN PART1 > at java.lang.Thread.(Thread.java:463) > PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80DC400 > SpecPool::createThread: EXCEPTION OCCURED IN PART3 at > java.lang.ThreadGroup.add(ThreadGroup.java:868) > PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80DFC00 > SpecPool::createThread: EXCEPTION OCCURED IN PART1 > at java.lang.Thread.(Thread.java :463) > PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80E1C00 > SpecPool::createThread: EXCEPTION OCCURED IN PART1 > at java.lang.Thread.(Thread.java:463) > PARENT[806DC00]: getThread: Linked threads 806DC00 <--> 80E5400 > PARENT[806DC00]: getThread: Linked threads 806DC00 <--> 80E3800 > Error occurred during initialization of VM > java.lang.ExceptionInInitializerError > at sun.misc.Launcher.(Launcher.java:71) > at sun.misc.Launcher.(Launcher.java:59) > at > java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1322) > at > java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1304) > Caused by: java.lang.NullPointerException > at java.net.URLClassLoader.(URLClassLoader.java:589) > at sun.misc.Launcher.(Launcher.java:71) > at sun.misc.Launcher.( Launcher.java:59) > at > java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1322) > at java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1304) > > Now how can I track or figure out, what has happend within > java.lang.Thread. ? - this is just the normal stack trace output, > are there any other means ? When using GDB (thx Tom for the script!), I > can only see backtraces, which end up in os::raise(), but I'm having > issues to set breakpoints, even when using "sharedlibrary libjvm"... > > Regards, Peter From Thomas.Rodriguez at Sun.COM Mon Nov 26 15:54:55 2007 From: Thomas.Rodriguez at Sun.COM (Tom Rodriguez) Date: Mon, 26 Nov 2007 15:54:55 -0800 Subject: figuring out what happenend within the java.lang.Thread. In-Reply-To: References: Message-ID: <474B5CCF.5080505@sun.com> It seems like you are initializing your pool too early in the bootstrap. The call to getSystemClassLoader occurs somewhere in the middle of the bootstrap and if you create your threads before that runs then java_system_loader() returns NULL. There should probably be an assert there that it never returns NULL since that would indicate some sort of bootstrap error. Anyway, that might explain the NPEs in your other threads. tom Peter Helfer wrote: > Ok, I'm running again into troubles.. > > This is what I'm seeing: > > VM option '+UnlockDiagnosticVMOptions' > VM option '-VerifyBeforeExit' > PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80DA800 > SpecPool::createThread: EXCEPTION OCCURED IN PART1 > at java.lang.Thread.(Thread.java:463) > PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80DC400 > SpecPool::createThread: EXCEPTION OCCURED IN PART3 at > java.lang.ThreadGroup.add(ThreadGroup.java:868) > PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80DFC00 > SpecPool::createThread: EXCEPTION OCCURED IN PART1 > at java.lang.Thread.(Thread.java :463) > PARENT[80D8800]: getThread: Linked threads 80D8800 <--> 80E1C00 > SpecPool::createThread: EXCEPTION OCCURED IN PART1 > at java.lang.Thread.(Thread.java:463) > PARENT[806DC00]: getThread: Linked threads 806DC00 <--> 80E5400 > PARENT[806DC00]: getThread: Linked threads 806DC00 <--> 80E3800 > Error occurred during initialization of VM > java.lang.ExceptionInInitializerError > at sun.misc.Launcher.(Launcher.java:71) > at sun.misc.Launcher.(Launcher.java:59) > at > java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1322) > at > java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1304) > Caused by: java.lang.NullPointerException > at java.net.URLClassLoader.(URLClassLoader.java:589) > at sun.misc.Launcher.(Launcher.java:71) > at sun.misc.Launcher.( Launcher.java:59) > at > java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1322) > at java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1304) > > Now how can I track or figure out, what has happend within > java.lang.Thread. ? - this is just the normal stack trace output, > are there any other means ? When using GDB (thx Tom for the script!), I > can only see backtraces, which end up in os::raise(), but I'm having > issues to set breakpoints, even when using "sharedlibrary libjvm"... > > Regards, Peter From Y.S.Ramakrishna at Sun.COM Tue Nov 27 23:40:13 2007 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Tue, 27 Nov 2007 23:40:13 -0800 Subject: OOM error In-Reply-To: <35005D6C4D5C8D408B5E62D06458503C0266B075@sfmbx01.ad.hotwirebiz.com> References: <35005D6C4D5C8D408B5E62D06458503C0266B030@sfmbx01.ad.hotwirebiz.com> <5d649bdb0711271446p749bb7cah73451784eeaabd5d@mail.gmail.com> <35005D6C4D5C8D408B5E62D06458503C0266B03D@sfmbx01.ad.hotwirebiz.com> <35005D6C4D5C8D408B5E62D06458503C0266B075@sfmbx01.ad.hotwirebiz.com> Message-ID: In the case of the enclosed pmap (which you indicate is from a JVM that has not encountered the C-heap exhaustion), as you might expect there is space available in the C-heap (shown by the "[heap]" annotated segment in yr pmap dump below). So we'll need to look at the pmap from a failing process. You could one of two things (or perhaps both):- -XX:OnError="pmap %p" will pmap the process just before exit with the error; otherwise you can do -XX:OnError="gcore %p" and then pmap the resulting core file. That should show the C-heap exhausted and you'd be able to see what went wrong. For additional information, since it appears that the process runs for a while before running into this C-heap exhaustion, you might want to run pmap on the process periodically and collect this pmap time-trace. You'd then likely see what it was that was over time consuming address space and running you out of C-heap. Additionally (also in -XX:OnError) pfiles may provide additional information. Hope that helps. PS: Since this is technically not a GC issue, per se, I'd probably take this off the GC list. I am accordingly cross-posting to the hotspot-runtime-dev list where it might be a better fit and where there might be further suggestions. -- ramki ----- Original Message ----- From: Anuj Lal Date: Tuesday, November 27, 2007 4:30 pm Subject: RE: RE: OOM error To: Y Srinivas Ramakrishna Cc: Neo Jia , hotspot-gc-dev at openjdk.java.net > Ramki > > This is pmap info when it is working correctly > We had same issue on other server also. Its becoming p1 issue now. I"ll > opem official sun case for this > > Anuj > > > > 20139: /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/bin/java > -Xms2560m > 00010000 64K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/bin/java > 0002E000 16K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/bin/java > 00032000 135328K rwx-- [ heap ] > 24C00000 5904K r--s- dev:85,5 ino:169421 > 25380000 408K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libfontmana > ger.so > 253F4000 96K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libfontmana > ger.so > 25480000 24K rwx-R [ anon ] > 25486000 488K rw--R [ anon ] > 25580000 24K rwx-R [ anon ] > 25586000 488K rw--R [ anon ] > 25680000 24K ----R [ anon ] > 25686000 488K rw--R [ anon ] > 25780000 24K rwx-R [ anon ] > 25786000 488K rw--R [ anon ] > 25880000 24K rwx-R [ anon ] > 25886000 488K rw--R [ anon ] > 25980000 24K rwx-R [ anon ] > 25986000 488K rw--R [ anon ] > 25A80000 24K rwx-R [ anon ] > 25A86000 488K rw--R [ anon ] > 25B80000 24K rwx-R [ anon ] > 25B86000 488K rw--R [ anon ] > 25C80000 24K rwx-R [ anon ] > 25C86000 488K rw--R [ anon ] > 25D80000 24K ----R [ anon ] > 25D86000 488K rw--R [ anon ] > 25E80000 24K ----R [ anon ] > 25E86000 488K rw--R [ anon ] > 25F80000 24K ----R [ anon ] > 25F86000 488K rw--R [ anon ] > 26080000 24K ----R [ anon ] > 26086000 488K rw--R [ anon ] > 26180000 24K ----R [ anon ] > 26186000 488K rw--R [ anon ] > 26280000 24K ----R [ anon ] > 26286000 488K rw--R [ anon ] > 26380000 24K ----R [ anon ] > 26386000 488K rw--R [ anon ] > 26480000 24K ----R [ anon ] > 26486000 488K rw--R [ anon ] > 26580000 24K ----R [ anon ] > 26586000 488K rw--R [ anon ] > 26680000 24K ----R [ anon ] > 26686000 488K rw--R [ anon ] > 26780000 24K ----R [ anon ] > 26786000 488K rw--R [ anon ] > 26880000 24K ----R [ anon ] > 26886000 488K rw--R [ anon ] > 26980000 504K r--s- dev:85,5 ino:169364 > 26A80000 1424K r--s- dev:85,5 ino:169360 > 26C00000 24K ----R [ anon ] > 26C06000 488K rw--R [ anon ] > 26D00000 24K ----R [ anon ] > 26D06000 488K rw--R [ anon ] > 26E00000 24K ----R [ anon ] > 26E06000 488K rw--R [ anon ] > 26F00000 24K ----R [ anon ] > 26F06000 488K rw--R [ anon ] > 27000000 24K ----R [ anon ] > 27006000 488K rw--R [ anon ] > 27100000 1056K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libmlib_ima > ge.so > 27216000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libmlib_ima > ge.so > 27280000 928K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libawt.so > 27376000 40K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libawt.so > 27380000 152K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libawt.so > 27400000 4432K r--s- dev:85,5 ino:89133 > 27900000 24K ----R [ anon ] > 27906000 488K rw--R [ anon ] > 27A00000 24K ----R [ anon ] > 27A06000 488K rw--R [ anon ] > 27B00000 648K r--s- dev:85,5 ino:89132 > 27C00000 24K ----R [ anon ] > 27C06000 488K rw--R [ anon ] > 27D00000 24K ----R [ anon ] > 27D06000 488K rw--R [ anon ] > 27E00000 24K ----R [ anon ] > 27E06000 488K rw--R [ anon ] > 27F00000 24K ----R [ anon ] > 27F06000 488K rw--R [ anon ] > 28000000 24K ----R [ anon ] > 28006000 488K rw--R [ anon ] > 28100000 24K ----R [ anon ] > 28106000 488K rw--R [ anon ] > 28200000 24K ----R [ anon ] > 28206000 488K rw--R [ anon ] > 28300000 24K ----R [ anon ] > 28306000 488K rw--R [ anon ] > 28400000 24K ----R [ anon ] > 28406000 488K rw--R [ anon ] > 28500000 24K rwx-R [ anon ] > 28506000 488K rw--R [ anon ] > 28600000 24K ----R [ anon ] > 28606000 488K rw--R [ anon ] > 28700000 24K ----R [ anon ] > 28706000 488K rw--R [ anon ] > 28800000 24K ----R [ anon ] > 28806000 488K rw--R [ anon ] > 28900000 24K ----R [ anon ] > 28906000 488K rw--R [ anon ] > 28A00000 24K ----R [ anon ] > 28A06000 488K rw--R [ anon ] > 28B00000 24K ----R [ anon ] > 28B06000 488K rw--R [ anon ] > 28C00000 24K ----R [ anon ] > 28C06000 488K rw--R [ anon ] > 28D00000 24K ----R [ anon ] > 28D06000 488K rw--R [ anon ] > 28E00000 24K ----R [ anon ] > 28E06000 488K rw--R [ anon ] > 28F00000 24K ----R [ anon ] > 28F06000 488K rw--R [ anon ] > 29000000 24K ----R [ anon ] > 29006000 488K rw--R [ anon ] > 29100000 24K ----R [ anon ] > 29106000 488K rw--R [ anon ] > 29200000 24K ----R [ anon ] > 29206000 488K rw--R [ anon ] > 29300000 24K ----R [ anon ] > 29306000 488K rw--R [ anon ] > 29400000 24K ----R [ anon ] > 29406000 488K rw--R [ anon ] > 29500000 24K ----R [ anon ] > 29506000 488K rw--R [ anon ] > 29600000 24K ----R [ anon ] > 29606000 488K rw--R [ anon ] > 29700000 24K ----R [ anon ] > 29706000 488K rw--R [ anon ] > 29800000 24K ----R [ anon ] > 29806000 488K rw--R [ anon ] > 29900000 24K ----R [ anon ] > 29906000 488K rw--R [ anon ] > 29A00000 24K ----R [ anon ] > 29A06000 488K rw--R [ anon ] > 29B00000 24K ----R [ anon ] > 29B06000 488K rw--R [ anon ] > 29C00000 24K ----R [ anon ] > 29C06000 488K rw--R [ anon ] > 29D00000 24K ----R [ anon ] > 29D06000 488K rw--R [ anon ] > 29E00000 24K ----R [ anon ] > 29E06000 488K rw--R [ anon ] > 29F00000 24K ----R [ anon ] > 29F06000 488K rw--R [ anon ] > 2A000000 24K ----R [ anon ] > 2A006000 488K rw--R [ anon ] > 2A100000 24K ----R [ anon ] > 2A106000 488K rw--R [ anon ] > 2A200000 24K ----R [ anon ] > 2A206000 488K rw--R [ anon ] > 2A300000 24K ----R [ anon ] > 2A306000 488K rw--R [ anon ] > 2A400000 24K ----R [ anon ] > 2A406000 488K rw--R [ anon ] > 2A500000 24K ----R [ anon ] > 2A506000 488K rw--R [ anon ] > 2A600000 24K ----R [ anon ] > 2A606000 488K rw--R [ anon ] > 2A700000 24K ----R [ anon ] > 2A706000 488K rw--R [ anon ] > 2A800000 24K ----R [ anon ] > 2A806000 488K rw--R [ anon ] > 2A900000 24K ----R [ anon ] > 2A906000 488K rw--R [ anon ] > 2AA00000 24K ----R [ anon ] > 2AA06000 488K rw--R [ anon ] > 2AB00000 24K ----R [ anon ] > 2AB06000 488K rw--R [ anon ] > 2AC00000 24K ----R [ anon ] > 2AC06000 488K rw--R [ anon ] > 2AD00000 24K ----R [ anon ] > 2AD06000 488K rw--R [ anon ] > 2AE00000 24K ----R [ anon ] > 2AE06000 488K rw--R [ anon ] > 2AF00000 24K ----R [ anon ] > 2AF06000 488K rw--R [ anon ] > 2B000000 24K ----R [ anon ] > 2B006000 488K rw--R [ anon ] > 2B100000 24K ----R [ anon ] > 2B106000 488K rw--R [ anon ] > 2B200000 24K ----R [ anon ] > 2B206000 488K rw--R [ anon ] > 2B300000 24K ----R [ anon ] > 2B306000 488K rw--R [ anon ] > 2B400000 24K ----R [ anon ] > 2B406000 488K rw--R [ anon ] > 2B500000 24K ----R [ anon ] > 2B506000 488K rw--R [ anon ] > 2B600000 24K ----R [ anon ] > 2B606000 488K rw--R [ anon ] > 2B700000 24K ----R [ anon ] > 2B706000 488K rw--R [ anon ] > 2B800000 24K ----R [ anon ] > 2B806000 488K rw--R [ anon ] > 2B900000 24K ----R [ anon ] > 2B906000 488K rw--R [ anon ] > 2BA00000 24K ----R [ anon ] > 2BA06000 488K rw--R [ anon ] > 2BB00000 24K ----R [ anon ] > 2BB06000 488K rw--R [ anon ] > 2BC00000 24K ----R [ anon ] > 2BC06000 488K rw--R [ anon ] > 2BD00000 24K ----R [ anon ] > 2BD06000 488K rw--R [ anon ] > 2BE00000 24K ----R [ anon ] > 2BE06000 488K rw--R [ anon ] > 2BF00000 24K ----R [ anon ] > 2BF06000 488K rw--R [ anon ] > 2C000000 24K ----R [ anon ] > 2C006000 488K rw--R [ anon ] > 2C100000 24K ----R [ anon ] > 2C106000 488K rw--R [ anon ] > 2C200000 24K ----R [ anon ] > 2C206000 488K rw--R [ anon ] > 2C300000 24K ----R [ anon ] > 2C306000 488K rw--R [ anon ] > 2C400000 24K ----R [ anon ] > 2C406000 488K rw--R [ anon ] > 2C500000 24K ----R [ anon ] > 2C506000 488K rw--R [ anon ] > 2C600000 24K ----R [ anon ] > 2C606000 488K rw--R [ anon ] > 2C700000 24K ----R [ anon ] > 2C706000 488K rw--R [ anon ] > 2C800000 24K ----R [ anon ] > 2C806000 488K rw--R [ anon ] > 2C900000 24K ----R [ anon ] > 2C906000 488K rw--R [ anon ] > 2CA00000 24K ----R [ anon ] > 2CA06000 488K rw--R [ anon ] > 2CB00000 24K ----R [ anon ] > 2CB06000 488K rw--R [ anon ] > 2CC00000 24K ----R [ anon ] > 2CC06000 488K rw--R [ anon ] > 2CD00000 24K ----R [ anon ] > 2CD06000 488K rw--R [ anon ] > 2CE00000 24K ----R [ anon ] > 2CE06000 488K rw--R [ anon ] > 2CF00000 24K ----R [ anon ] > 2CF06000 488K rw--R [ anon ] > 2D000000 24K ----R [ anon ] > 2D006000 488K rw--R [ anon ] > 2D100000 24K ----R [ anon ] > 2D106000 488K rw--R [ anon ] > 2D200000 24K ----R [ anon ] > 2D206000 488K rw--R [ anon ] > 2D300000 24K ----R [ anon ] > 2D306000 488K rw--R [ anon ] > 2D400000 24K ----R [ anon ] > 2D406000 488K rw--R [ anon ] > 2D500000 24K ----R [ anon ] > 2D506000 488K rw--R [ anon ] > 2D600000 24K ----R [ anon ] > 2D606000 488K rw--R [ anon ] > 2D700000 24K ----R [ anon ] > 2D706000 488K rw--R [ anon ] > 2D800000 24K ----R [ anon ] > 2D806000 488K rw--R [ anon ] > 2D900000 24K ----R [ anon ] > 2D906000 488K rw--R [ anon ] > 2DA00000 24K ----R [ anon ] > 2DA06000 488K rw--R [ anon ] > 2DB00000 24K ----R [ anon ] > 2DB06000 488K rw--R [ anon ] > 2DC00000 24K ----R [ anon ] > 2DC06000 488K rw--R [ anon ] > 2DD00000 24K ----R [ anon ] > 2DD06000 488K rw--R [ anon ] > 2DE00000 24K ----R [ anon ] > 2DE06000 488K rw--R [ anon ] > 2DEB0000 176K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libjpeg.so > 2DEEC000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libjpeg.so > 2DF00000 24K ----R [ anon ] > 2DF06000 488K rw--R [ anon ] > 2DFA0000 8K rwx-- [ anon ] > 2DFB0000 96K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libdcpr.so > 2DFD6000 80K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libdcpr.so > 2E000000 24K ----R [ anon ] > 2E006000 488K rw--R [ anon ] > 2E090000 312K r--s- dev:85,5 ino:12797 > 2E0F0000 24K r--s- dev:85,5 ino:169374 > 2E100000 24K ----R [ anon ] > 2E106000 488K rw--R [ anon ] > 2E190000 32K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/headless/li > bmawt.so > 2E1A6000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/headless/li > bmawt.so > 2E1B0000 8K rwx-- [ anon ] > 2E1C0000 112K r--s- dev:85,5 ino:104786 > 2E1E0000 112K r--s- dev:85,5 ino:148909 > 2E200000 24K ----R [ anon ] > 2E206000 488K rw--R [ anon ] > 2E290000 24K r--s- dev:85,5 ino:104507 > 2E2A0000 32K r--s- dev:85,5 ino:90208 > 2E2B0000 64K r--s- dev:85,5 ino:103379 > 2E2D0000 32K r--s- dev:85,5 ino:90200 > 2E2E0000 56K r--s- dev:85,5 ino:90170 > 2E300000 24K ----R [ anon ] > 2E306000 488K rw--R [ anon ] > 2E390000 32K r--s- dev:85,5 ino:90131 > 2E3A0000 16K r--s- dev:85,5 ino:90152 > 2E3B0000 40K r--s- dev:85,5 ino:90192 > 2E3C0000 136K r-x-- /usr/lib/libelf.so.1 > 2E3F2000 8K rwx-- /usr/lib/libelf.so.1 > 2E400000 24K ----R [ anon ] > 2E406000 488K rw--R [ anon ] > 2E490000 48K r--s- dev:85,5 ino:90137 > 2E4A0000 248K r-x-- /usr/lib/libresolv.so.2 > 2E4EE000 16K rwx-- /usr/lib/libresolv.so.2 > 2E500000 1480K r--s- dev:85,5 ino:178161 > 2E680000 24K ----R [ anon ] > 2E686000 488K rw--R [ anon ] > 2E710000 56K r--s- dev:85,5 ino:90159 > 2E730000 16K rw--- [ anon ] > 2E740000 40K r-x-- /usr/ucblib/libucb.so.1 > 2E75A000 8K rwx-- /usr/ucblib/libucb.so.1 > 2E760000 24K r-x-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libmuxe > r.so > 2E774000 8K rwx-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libmuxe > r.so > 2E780000 512K rw--R [ anon ] > 2E810000 40K r--s- dev:85,5 ino:90183 > 2E820000 8K r-x-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libwlfi > leio2.so > 2E830000 16K rwx-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libwlfi > leio2.so > 2E840000 64K rw--- [ anon ] > 2E860000 8K r-x-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libstac > kdump.so > 2E870000 8K rwx-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libstac > kdump.so > 2E880000 24K ----R [ anon ] > 2E886000 488K rw--R [ anon ] > 2E910000 48K r--s- dev:85,5 ino:90122 > 2E920000 8K r-x-- /usr/lib/libsendfile.so.1 > 2E932000 8K rwx-- /usr/lib/libsendfile.so.1 > 2E940000 8K r-x-- /usr/lib/libmd5.so.1 > 2E952000 8K rwx-- /usr/lib/libmd5.so.1 > 2E960000 40K r-x-- /usr/lib/libaio.so.1 > 2E97A000 8K rwx-- /usr/lib/libaio.so.1 > 2E980000 24K ----R [ anon ] > 2E986000 488K rw--R [ anon ] > 2EA10000 16K r--s- dev:85,5 ino:91540 > 2EA20000 24K r-x-- /usr/lib/librt.so.1 > 2EA36000 8K rwx-- /usr/lib/librt.so.1 > 2EA40000 200K r--s- dev:85,5 ino:178116 > 2EA80000 24K ----R [ anon ] > 2EA86000 488K rw--R [ anon ] > 2EB10000 32K rw--- [ anon ] > 2EB20000 32K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libnio.so > 2EB36000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libnio.so > 2EB40000 184K r--s- dev:85,5 ino:178351 > 2EB80000 2496K r--s- dev:85,5 ino:43427 > 2EE00000 784K r--s- dev:85,5 ino:8126 > 2EED0000 88K r--s- dev:85,5 ino:178142 > 2EEF0000 32K r--s- dev:85,5 ino:178349 > 2EF00000 64K r--s- dev:85,5 ino:19946 > 2EF20000 328K r--s- dev:85,5 ino:27067 > 2EF80000 776K r--s- dev:85,5 ino:39082 > 2F050000 64K r--s- dev:85,5 ino:16731 > 2F070000 376K r--s- dev:85,5 ino:7814 > 2F0E0000 288K r--s- dev:85,5 ino:7812 > 2F130000 248K r--s- dev:85,5 ino:43449 > 2F180000 1336K r--s- dev:85,5 ino:11522 > 2F2D0000 120K r--s- dev:85,5 ino:7816 > 2F300000 1496K r--s- dev:85,5 ino:7171 > 2F480000 880K r--s- dev:85,5 ino:16719 > 2F560000 56K r--s- dev:85,5 ino:8160 > 2F580000 64K r--s- dev:85,5 ino:7815 > 2F5A0000 336K r--s- dev:85,5 ino:16717 > 2F600000 1656K r--s- dev:85,5 ino:11690 > 2F7B0000 128K r--s- dev:85,5 ino:16720 > 2F7E0000 88K r--s- dev:85,5 ino:23479 > 2F800000 5344K r--s- dev:85,5 ino:25387 > 2FD40000 16K rw--- [ anon ] > 2FD50000 120K r--s- dev:85,5 ino:19782 > 2FD80000 1152K r--s- dev:85,5 ino:7946 > 2FEB0000 24K r--s- dev:85,5 ino:43425 > 2FEC0000 208K r--s- dev:85,5 ino:7944 > 2FF00000 648K r--s- dev:85,5 ino:21820 > 2FFB0000 32K r--s- dev:85,5 ino:7813 > 2FFC0000 240K r--s- dev:85,5 ino:7945 > 30000000 4904K r--s- dev:85,5 ino:157426 > 304D0000 8K rwx-- [ anon ] > 304E0000 24K r--s- dev:85,5 ino:43447 > 304F0000 120K r--s- dev:85,5 ino:16575 > 30520000 352K r--s- dev:85,5 ino:23507 > 30580000 1512K r--s- dev:85,5 ino:156707 > 30700000 616K r--s- dev:85,5 ino:8161 > 307A0000 312K r--s- dev:85,5 ino:12797 > 30800000 3968K r--s- dev:85,5 ino:9009 > 30BF0000 32K r--s- dev:85,5 ino:16718 > 30C00000 5040K r--s- dev:85,5 ino:9011 > 31100000 488K r--s- dev:85,5 ino:21821 > 31180000 480K r--s- dev:85,5 ino:44388 > 31200000 552K r--s- dev:85,5 ino:7744 > 31290000 8K r--s- dev:85,5 ino:169334 > 312A0000 352K r--s- dev:85,5 ino:26985 > 31300000 640K r--s- dev:85,5 ino:176761 > 313B0000 24K r--s- dev:85,5 ino:15965 > 313C0000 24K r--s- dev:85,5 ino:11502 > 313D0000 168K r--s- dev:85,5 ino:21694 > 31400000 6880K r--s- dev:85,5 ino:169093 > 31AC0000 16K r--s- dev:85,5 ino:15966 > 31AD0000 208K r--s- dev:85,5 ino:17946 > 31B10000 432K r--s- dev:85,5 ino:15781 > 31B80000 680K r--s- dev:85,5 ino:170149 > 31C30000 8K r--s- dev:85,5 ino:23497 > 31C40000 224K r--s- dev:85,5 ino:20121 > 31C80000 1000K r--s- dev:85,5 ino:169091 > 31D80000 976K r--s- dev:85,5 ino:169088 > 31E80000 2544K r--s- dev:85,5 ino:188396 > 32100000 512K r--s- dev:85,5 ino:170763 > 32190000 80K r--s- dev:85,5 ino:23502 > 321B0000 40K r--s- dev:85,5 ino:44387 > 321C0000 80K r--s- dev:85,5 ino:17995 > 321E0000 72K r--s- dev:85,5 ino:39073 > 32200000 1496K r--s- dev:85,5 ino:170766 > 32380000 776K r--s- dev:85,5 ino:188388 > 32450000 16K r--s- dev:85,5 ino:15778 > 32460000 112K r--s- dev:85,5 ino:7139 > 32480000 968K r--s- dev:85,5 ino:171312 > 32580000 752K r--s- dev:85,5 ino:176705 > 32650000 32K r--s- dev:85,5 ino:23508 > 32660000 72K r--s- dev:85,5 ino:13123 > 32680000 912K r--s- dev:85,5 ino:176805 > 32770000 32K r--s- dev:85,5 ino:20119 > 32780000 1512K r--s- dev:85,5 ino:178382 > 32900000 2616K r--s- dev:85,5 ino:169089 > 32BA0000 48K r--s- dev:85,5 ino:20243 > 32BB0000 256K r--s- dev:85,5 ino:170462 > 32C00000 52536K r--s- dev:85,5 ino:178597 > 35F60000 152K r--s- dev:85,5 ino:7752 > 35F90000 392K r--s- dev:85,5 ino:170432 > 36000000 1864K r--s- dev:85,5 ino:187108 > 361E0000 24K r--s- dev:85,5 ino:21822 > 361F0000 32K r--s- dev:85,5 ino:7745 > 36200000 1784K r--s- dev:85,5 ino:157391 > 363D0000 120K r--s- dev:85,5 ino:7190 > 36400000 6880K r--s- dev:85,5 ino:94624 > 36AC0000 200K r--s- dev:85,5 ino:188391 > 36B00000 648K r--s- dev:85,5 ino:178589 > 36BB0000 160K r--s- dev:85,5 ino:170190 > 36BE0000 112K r--s- dev:85,5 ino:170154 > 36C00000 6936K r--s- dev:85,5 ino:24574 > 372D0000 8K r--s- dev:85,5 ino:31008 > 372E0000 104K r--s- dev:85,5 ino:170159 > 37300000 2128K r--s- dev:85,5 ino:157398 > 37520000 96K r--s- dev:85,5 ino:170157 > 37540000 216K r--s- dev:85,5 ino:170483 > 37580000 2752K r--s- dev:85,5 ino:24575 > 37840000 32K r--s- dev:85,5 ino:7748 > 37850000 120K r--s- dev:85,5 ino:187721 > 37880000 3432K r--s- dev:85,5 ino:23916 > 37BE0000 8K r--s- dev:85,5 ino:7141 > 37BF0000 48K r--s- dev:85,5 ino:7800 > 37C00000 4096K rwx-- [ anon ] > 38010000 8K r--s- dev:85,5 ino:9010 > 38020000 192K r--s- dev:85,5 ino:187853 > 38060000 88K r--s- dev:85,5 ino:188385 > 38080000 1768K r--s- dev:85,5 ino:23915 > 38240000 192K r--s- dev:85,5 ino:187665 > 38280000 1016K r--s- dev:85,5 ino:7170 > 38390000 40K r--s- dev:85,5 ino:176570 > 383A0000 312K r--s- dev:85,5 ino:188386 > 38400000 131072K rwx-- [ anon ] > 40410000 424K r--s- dev:85,5 ino:187496 > 40480000 864K r--s- dev:85,5 ino:178588 > 40560000 72K r--s- dev:85,5 ino:178111 > 40580000 696K r--s- dev:85,5 ino:20117 > 40640000 8K r--s- dev:85,5 ino:170155 > 40650000 144K r--s- dev:85,5 ino:171315 > 40680000 1072K r--s- dev:85,5 ino:16891 > 407A0000 112K r--s- dev:85,5 ino:188378 > 407C0000 64K r--s- dev:85,5 ino:171316 > 407E0000 72K r--s- dev:85,5 ino:171306 > 40800000 73728K rwx-- [ anon ] > 45000000 8K rwx-- [ anon ] > 45010000 24K r--s- dev:85,5 ino:171297 > 45020000 24K r--s- dev:85,5 ino:171289 > 45030000 8K r--s- dev:85,5 ino:171305 > 45040000 8K r--s- dev:85,5 ino:171296 > 45050000 16K r--s- dev:85,5 ino:171288 > 45060000 8K r--s- dev:85,5 ino:171304 > 45070000 8K r--s- dev:85,5 ino:171295 > 45080000 168K r--s- dev:85,5 ino:171287 > 450B0000 8K r--s- dev:85,5 ino:171311 > 450C0000 40K r--s- dev:85,5 ino:171303 > 450D0000 8K r--s- dev:85,5 ino:171294 > 450E0000 40K r--s- dev:85,5 ino:171286 > 450F0000 8K r--s- dev:85,5 ino:171310 > 45100000 792K r--s- dev:85,5 ino:25468 > 451D0000 8K r--s- dev:85,5 ino:171293 > 451E0000 8K r--s- dev:85,5 ino:171285 > 451F0000 8K r--s- dev:85,5 ino:171309 > 45200000 24K ----R [ anon ] > 45206000 488K rw--R [ anon ] > 45290000 16K r--s- dev:85,5 ino:171301 > 452A0000 48K r--s- dev:85,5 ino:171292 > 452B0000 8K r--s- dev:85,5 ino:171284 > 452C0000 16K r--s- dev:85,5 ino:171308 > 452D0000 16K r--s- dev:85,5 ino:171300 > 452E0000 8K r--s- dev:85,5 ino:171291 > 452F0000 48K r--s- dev:85,5 ino:171307 > 45300000 24K ----R [ anon ] > 45306000 488K rw--R [ anon ] > 45390000 400K r--s- dev:85,5 ino:171302 > 45400000 4096K rwx-- [ anon ] > 45800000 8K rwx-- [ anon ] > 45810000 16K r--s- dev:85,5 ino:171283 > 45820000 72K r--s- dev:85,5 ino:171298 > 45840000 16K r--s- dev:85,5 ino:171290 > 45850000 8K r--s- dev:85,5 ino:171282 > 45860000 112K r--s- dev:85,5 ino:171271 > 45880000 472K r--s- dev:85,5 ino:24753 > 45900000 24K ----R [ anon ] > 45906000 488K rw--R [ anon ] > 45990000 136K r--s- dev:85,5 ino:171270 > 459C0000 64K r--s- dev:85,5 ino:171269 > 459E0000 24K r--s- dev:85,5 ino:176581 > 459F0000 16K r--s- dev:85,5 ino:176577 > 45A00000 24K ----R [ anon ] > 45A06000 488K rw--R [ anon ] > 45A90000 392K r--s- dev:85,5 ino:176527 > 45B00000 40K rwx-- [ anon ] > 45B0A000 8K rwx-- [ anon ] > 45B0C000 8K rwx-- [ anon ] > 45B0E000 8K rwx-- [ anon ] > 45B10000 8K rwx-- [ anon ] > 45B12000 8K rwx-- [ anon ] > 45B14000 8K rwx-- [ anon ] > 45B16000 8K rwx-- [ anon ] > 45B18000 8K rwx-- [ anon ] > 45B1A000 8K rwx-- [ anon ] > 45B1C000 8K rwx-- [ anon ] > 45B1E000 8K rwx-- [ anon ] > 45B20000 8K rwx-- [ anon ] > 45B22000 8K rwx-- [ anon ] > 45B24000 8K rwx-- [ anon ] > 45B26000 8K rwx-- [ anon ] > 45B28000 8K rwx-- [ anon ] > 45B2A000 8K rwx-- [ anon ] > 45B2C000 8K rwx-- [ anon ] > 45B2E000 8K rwx-- [ anon ] > 45B30000 8K rwx-- [ anon ] > 45B32000 8K rwx-- [ anon ] > 45B34000 8K rwx-- [ anon ] > 45B36000 8K rwx-- [ anon ] > 45B38000 8K rwx-- [ anon ] > 45B3A000 152K rwx-- [ anon ] > 45B60000 136K ----R [ anon ] > 45B90000 256K r--s- dev:85,5 ino:176512 > 45BE0000 80K r--s- dev:85,5 ino:188399 > 45C00000 1024K rwx-- [ anon ] > 45D00000 4096K rwx-- [ anon ] > 46100000 32K rwx-- [ anon ] > 46108000 8K rwx-- [ anon ] > 4610A000 8K rwx-- [ anon ] > 4610C000 8K rwx-- [ anon ] > 4610E000 8K rwx-- [ anon ] > 46110000 8K rwx-- [ anon ] > 46112000 8K rwx-- [ anon ] > 46114000 8K rwx-- [ anon ] > 46116000 8K rwx-- [ anon ] > 46118000 8K rwx-- [ anon ] > 4611A000 8K rwx-- [ anon ] > 4611C000 8K rwx-- [ anon ] > 4611E000 8K rwx-- [ anon ] > 46120000 8K rwx-- [ anon ] > 46122000 8K rwx-- [ anon ] > 46124000 8K rwx-- [ anon ] > 46126000 8K rwx-- [ anon ] > 46128000 8K rwx-- [ anon ] > 4612A000 8K rwx-- [ anon ] > 4612C000 8K rwx-- [ anon ] > 4612E000 8K rwx-- [ anon ] > 46130000 8K rwx-- [ anon ] > 46132000 8K rwx-- [ anon ] > 46134000 8K rwx-- [ anon ] > 46136000 8K rwx-- [ anon ] > 46138000 8K rwx-- [ anon ] > 4613A000 152K rwx-- [ anon ] > 46160000 128K ----R [ anon ] > 46180000 8K rwx-- [ anon ] > 46190000 392K r--s- dev:85,5 ino:177742 > 46200000 24K ----R [ anon ] > 46206000 488K rw--R [ anon ] > 46290000 8K r--s- dev:85,5 ino:176568 > 462A0000 8K r--s- dev:85,5 ino:187556 > 462B0000 40K r--s- dev:85,5 ino:178583 > 462C0000 40K r--s- dev:85,5 ino:178576 > 462D0000 16K r--s- dev:85,5 ino:169518 > 462E0000 8K r--s- dev:85,5 ino:168455 > 462F0000 40K r--s- dev:85,5 ino:168458 > 46300000 24K ----R [ anon ] > 46306000 488K rw--R [ anon ] > 46390000 16K r--s- dev:85,5 ino:168457 > 463A0000 24K r--s- dev:85,5 ino:168454 > 463B0000 8K r--s- dev:85,5 ino:168477 > 463C0000 64K r--s- dev:85,5 ino:24571 > 463E0000 96K r--s- dev:85,5 ino:24573 > 46400000 524288K rwx-- [ anon ] > 66400000 2097152K rwx-- [ anon ] > E6400000 16384K rwx-- [ anon ] > E7400000 4096K rwx-- [ anon ] > E7800000 4096K rwx-- [ anon ] > E7C00000 4096K rwx-- [ anon ] > E8000000 4096K rwx-- [ anon ] > E8400000 4096K rwx-- [ anon ] > E8800000 4096K rwx-- [ anon ] > E8C00000 4096K rwx-- [ anon ] > E9000000 4096K rwx-- [ anon ] > E9400000 4096K rwx-- [ anon ] > E9800000 4096K rwx-- [ anon ] > E9C00000 4096K rwx-- [ anon ] > EA000000 4096K rwx-- [ anon ] > EA400000 4096K rwx-- [ anon ] > EA800000 4096K rwx-- [ anon ] > EAC00000 4096K rwx-- [ anon ] > EB000000 4096K rwx-- [ anon ] > EB400000 4096K rwx-- [ anon ] > EB800000 4096K rwx-- [ anon ] > EBC00000 4096K rwx-- [ anon ] > EC000000 4096K rwx-- [ anon ] > EC400000 4096K rwx-- [ anon ] > EC800000 4096K rwx-- [ anon ] > ECC00000 4096K rwx-- [ anon ] > ED000000 4096K rwx-- [ anon ] > ED400000 4096K rwx-- [ anon ] > ED800000 77824K rwx-- [ anon ] > F2400000 65536K ----R [ anon ] > F6410000 8K r--s- dev:85,5 ino:168461 > F6420000 216K r--s- dev:85,5 ino:24576 > F6460000 8K r--s- dev:85,5 ino:24750 > F6470000 8K r--s- dev:85,5 ino:24747 > F6480000 336K r--s- dev:85,5 ino:23914 > F64E0000 88K r--s- dev:85,5 ino:23913 > F6500000 24K ----R [ anon ] > F6506000 488K rw--R [ anon ] > F6590000 32K r--s- dev:85,5 ino:24751 > F65A0000 88K r--s- dev:85,5 ino:9002 > F65C0000 200K r--s- dev:85,5 ino:7151 > F6600000 24K ----R [ anon ] > F6606000 488K rw--R [ anon ] > F6690000 16K r--s- dev:85,5 ino:7143 > F66A0000 72K r--s- dev:85,5 ino:7165 > F66C0000 16K r--s- dev:85,5 ino:7144 > F66D0000 80K r--s- dev:85,5 ino:7158 > F66F0000 16K r--s- dev:85,5 ino:7167 > F6700000 512K rw--R [ anon ] > F6790000 416K r--s- dev:85,5 ino:7161 > F6800000 4096K rwx-- [ anon ] > F6C00000 4096K rwx-- [ anon ] > F7000000 4096K rwx-- [ anon ] > F7400000 4096K rwx-- [ anon ] > F7800000 4096K rwx-- [ anon ] > F7C00000 4096K rwx-- [ anon ] > F8000000 4096K rwx-- [ anon ] > F8400000 4096K rwx-- [ anon ] > F8800000 4096K rwx-- [ anon ] > F8C00000 4096K rwx-- [ anon ] > F9000000 4096K rwx-- [ anon ] > F9400000 4096K rwx-- [ anon ] > F9800000 16384K ----R [ anon ] > FA810000 16K r--s- dev:85,5 ino:7159 > FA820000 368K r--s- dev:85,5 ino:211317 > FA880000 488K r--s- dev:85,5 ino:8285 > FA900000 512K rw--R [ anon ] > FA990000 392K r--s- dev:85,5 ino:102341 > FAA00000 584K rwx-- [ anon ] > FAAA0000 328K r--s- dev:85,5 ino:211325 > FAB00000 512K rw--R [ anon ] > FAB90000 112K r--s- dev:85,5 ino:7179 > FABB0000 16K r--s- dev:85,5 ino:211316 > FABC0000 224K r--s- dev:85,5 ino:102332 > FAC00000 8640K r--s- dev:85,5 ino:25457 > FB480000 512K rw--R [ anon ] > FB510000 64K r--s- dev:85,5 ino:8273 > FB530000 64K r--s- dev:85,5 ino:8276 > FB550000 168K r--s- dev:85,5 ino:8270 > FB580000 512K rw--R [ anon ] > FB610000 48K r--s- dev:85,5 ino:7498 > FB620000 352K r--s- dev:85,5 ino:26726 > FB680000 24K rwx-- [ anon ] > FB686000 8K rwx-- [ anon ] > FB688000 8K rwx-- [ anon ] > FB68A000 8K rwx-- [ anon ] > FB68C000 8K rwx-- [ anon ] > FB68E000 8K rwx-- [ anon ] > FB690000 8K rwx-- [ anon ] > FB692000 8K rwx-- [ anon ] > FB694000 8K rwx-- [ anon ] > FB696000 8K rwx-- [ anon ] > FB698000 8K rwx-- [ anon ] > FB69A000 8K rwx-- [ anon ] > FB69C000 8K rwx-- [ anon ] > FB69E000 8K rwx-- [ anon ] > FB6A0000 8K rwx-- [ anon ] > FB6A2000 8K rwx-- [ anon ] > FB6A4000 8K rwx-- [ anon ] > FB6A6000 8K rwx-- [ anon ] > FB6A8000 8K rwx-- [ anon ] > FB6AA000 8K rwx-- [ anon ] > FB6AC000 8K rwx-- [ anon ] > FB6AE000 8K rwx-- [ anon ] > FB6B0000 8K rwx-- [ anon ] > FB6B2000 8K rwx-- [ anon ] > FB6B4000 8K rwx-- [ anon ] > FB6B6000 8K rwx-- [ anon ] > FB6B8000 8K rwx-- [ anon ] > FB6BA000 8K rwx-- [ anon ] > FB6BC000 8K rwx-- [ anon ] > FB6BE000 8K rwx-- [ anon ] > FB6C0000 8K rwx-- [ anon ] > FB6C2000 8K rwx-- [ anon ] > FB6C4000 8K rwx-- [ anon ] > FB6C6000 8K rwx-- [ anon ] > FB6C8000 8K rwx-- [ anon ] > FB6CA000 8K rwx-- [ anon ] > FB6CC000 8K rwx-- [ anon ] > FB6CE000 8K rwx-- [ anon ] > FB6D0000 8K rwx-- [ anon ] > FB6D2000 8K rwx-- [ anon ] > FB6D4000 8K rwx-- [ anon ] > FB6D6000 8K rwx-- [ anon ] > FB6D8000 8K rwx-- [ anon ] > FB6DA000 8K rwx-- [ anon ] > FB6DC000 8K rwx-- [ anon ] > FB6DE000 8K rwx-- [ anon ] > FB6E0000 8K rwx-- [ anon ] > FB6E2000 8K rwx-- [ anon ] > FB6E4000 8K rwx-- [ anon ] > FB6E6000 8K rwx-- [ anon ] > FB6E8000 8K rwx-- [ anon ] > FB6EA000 8K rwx-- [ anon ] > FB6EC000 8K rwx-- [ anon ] > FB6EE000 8K rwx-- [ anon ] > FB6F0000 8K rwx-- [ anon ] > FB6F2000 8K rwx-- [ anon ] > FB6F4000 8K rwx-- [ anon ] > FB6F6000 8K rwx-- [ anon ] > FB6F8000 8K rwx-- [ anon ] > FB6FA000 8K rwx-- [ anon ] > FB6FC000 8K rwx-- [ anon ] > FB6FE000 8K rwx-- [ anon ] > FB700000 8K rwx-- [ anon ] > FB702000 8K rwx-- [ anon ] > FB704000 8K rwx-- [ anon ] > FB706000 8K rwx-- [ anon ] > FB708000 8K rwx-- [ anon ] > FB70A000 8K rwx-- [ anon ] > FB70C000 8K rwx-- [ anon ] > FB70E000 8K rwx-- [ anon ] > FB710000 8K rwx-- [ anon ] > FB712000 8K rwx-- [ anon ] > FB714000 8K rwx-- [ anon ] > FB716000 8K rwx-- [ anon ] > FB718000 8K rwx-- [ anon ] > FB71A000 8K rwx-- [ anon ] > FB71C000 8K rwx-- [ anon ] > FB71E000 8K rwx-- [ anon ] > FB720000 8K rwx-- [ anon ] > FB722000 8K rwx-- [ anon ] > FB724000 8K rwx-- [ anon ] > FB726000 8K rwx-- [ anon ] > FB728000 8K rwx-- [ anon ] > FB72A000 8K rwx-- [ anon ] > FB72C000 8K rwx-- [ anon ] > FB72E000 8K rwx-- [ anon ] > FB730000 8K rwx-- [ anon ] > FB732000 8K rwx-- [ anon ] > FB734000 304K ----R [ anon ] > FB790000 24K r--s- dev:85,5 ino:102340 > FB7A0000 48K r--s- dev:85,5 ino:8275 > FB7B0000 304K r--s- dev:85,5 ino:16889 > FB800000 39080K r--s- dev:85,5 ino:25497 > FDE30000 8K r--s- dev:85,5 ino:7155 > FDE40000 32K r--s- dev:85,5 ino:8274 > FDE50000 120K r--s- dev:85,5 ino:8269 > FDE80000 512K rw--R [ anon ] > FDF10000 112K r--s- dev:85,5 ino:8271 > FDF30000 96K r--s- dev:85,5 ino:26727 > FDF50000 176K r--s- dev:85,5 ino:25470 > FDF80000 536K r--s- dev:85,5 ino:25490 > FE010000 88K r--s- dev:85,5 ino:26444 > FE030000 160K r--s- dev:85,5 ino:25469 > FE060000 24K r-x-- /usr/lib/nss_files.so.1 > FE076000 8K rwx-- /usr/lib/nss_files.so.1 > FE080000 504K r--s- dev:85,5 ino:24234 > FE110000 72K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libnet.so > FE130000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libnet.so > FE140000 24K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libmanageme > nt.so > FE154000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libmanageme > nt.so > FE160000 112K rw--- [ anon ] > FE180000 1312K r--s- dev:85,5 ino:26635 > FE2D0000 16K r--s- dev:85,5 ino:20250 > FE2E0000 88K r--s- dev:85,5 ino:25489 > FE300000 1176K r--s- dev:85,5 ino:26633 > FE430000 8K rwx-- [ anon ] > FE440000 192K r--s- dev:85,5 ino:26634 > FE480000 2528K r--s- dev:85,5 ino:26520 > FE700000 16K r--s- dev:85,5 ino:8280 > FE710000 192K r--s- dev:85,5 ino:26499 > FE750000 64K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libzip.so > FE760000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libzip.so > FE770000 8K rwx-- [ anon ] > FE780000 144K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libjava.so > FE7B4000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libjava.so > FE7C0000 56K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libverify.s > o > FE7DE000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libverify.s > o > FE7F0000 8K r--s- dev:85,5 ino:25467 > FE800000 7984K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/server/libj > vm.so > FEFDC000 408K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/server/libj > vm.so > FF042000 56K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/server/libj > vm.so > FF060000 32K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/native_thre > ads/libhpi.so > FF078000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/native_thre > ads/libhpi.so > FF07A000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/native_thre > ads/libhpi.so > FF080000 568K r-x-- /usr/lib/libnsl.so.1 > FF11E000 40K rwx-- /usr/lib/libnsl.so.1 > FF128000 24K rwx-- /usr/lib/libnsl.so.1 > FF140000 32K rw-s- dev:0,2 ino:253268838 > FF150000 8K rwx-- [ anon ] > FF160000 8K r---- [ anon ] > FF170000 16K rw--- [ anon ] > FF180000 16K r-x-- /usr/lib/libmp.so.2 > FF194000 8K rwx-- /usr/lib/libmp.so.2 > FF1A0000 8K rwx-- [ anon ] > FF1B0000 224K r-x-- /usr/lib/libm.so.1 > FF1F6000 8K rwx-- /usr/lib/libm.so.1 > FF200000 48K r-x-- /usr/lib/libCrun.so.1 > FF21A000 8K rwx-- /usr/lib/libCrun.so.1 > FF21C000 16K rwx-- /usr/lib/libCrun.so.1 > FF230000 8K r-x-- /usr/lib/libsched.so.1 > FF242000 8K rwx-- /usr/lib/libsched.so.1 > FF250000 40K r-x-- /usr/lib/libsocket.so.1 > FF26A000 8K rwx-- /usr/lib/libsocket.so.1 > FF280000 688K r-x-- /usr/lib/libc.so.1 > FF33C000 24K rwx-- /usr/lib/libc.so.1 > FF342000 8K rwx-- /usr/lib/libc.so.1 > FF350000 8K rwxs- [ anon ] > FF360000 16K rw--- [ anon ] > FF370000 8K rwx-- [ anon ] > FF380000 96K r-x-- /usr/lib/libthread.so.1 > FF3A8000 8K rwx-- /usr/lib/libthread.so.1 > FF3AA000 8K rwx-- /usr/lib/libthread.so.1 > FF3B0000 8K r-x-- /usr/platform/sun4u-us3/lib/libc_psr.so.1 > FF3C0000 192K r-x-- /usr/lib/ld.so.1 > FF3F0000 8K rwx-- /usr/lib/ld.so.1 > FF3F2000 8K rwx-- /usr/lib/ld.so.1 > FF3FA000 8K rwx-- /usr/lib/libdl.so.1 > FFB7C000 24K ----- [ anon ] > FF400000 8192K rw--- [ stack ] > total 3636576K > > > -----Original Message----- > From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] > Sent: Tuesday, November 27, 2007 3:26 PM > To: Anuj Lal > Cc: Neo Jia; hotspot-gc-dev at openjdk.java.net > Subject: Re: RE: OOM error > > > pmap -rs will tell you what's taking up address space. It could be, for > example, > that you have lots of threads (so the space is gone to thread stacks, > so > you might > try -Xss<...> or it could be that you have loaded lots of native > libraries that have > taken up address space, or it could be that you have lots of file > descriptors or > sockets or mmapped files or other native resources that are tying up > your > address space). > > (the reference to "swap space" in the message is usuallysomewhat > misleading; > i think there are plans to fix that message to be smarter / less > misleading). > > -- ramki > > > We are on solaris 9.0 32 bit > > > > We aleady have "low" jvm heap. We don't want to go below this. > > > > I am pretty sure we are not running out of swap space > > > > > > > > This is what we have currently > > > > 8 processes: 46 sleeping, 2 on cpu > > > > CPU states: 95.2% idle, 3.9% user, 0.8% kernel, 0.0% iowait, 0.0% > > swap > > > > Memory: 16G real, 12G free, 3510M swap in use, 18G swap free > > > > > > > > PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND > > > > 20139 weblogic 144 19 0 3389M 2362M cpu/2 87:19 5.16% java > > > > > > > > Anuj > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] > > Sent: Tuesday, November 27, 2007 3:10 PM > > To: Neo Jia > > Cc: Anuj Lal; hotspot-gc-dev at openjdk.java.net > > Subject: Re: OOM error > > > > > > > > > > > > > > > > > > > > Hi Anuj -- > > > > > > > > > > Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: > > requested > > > > > > 32756 bytes for ChunkPool::allocate. Out of swap space? > > > > > > > > This is a request for 32 KB (not 32 MB as in the bug you quoted > below). > > > > > > > > You should probably check pmap -rs on the process. You are almost > > certainly out of > > > > C heap space (so, as you suggested below, reducing the Java heap might > > help by > > > > freeing up your apparently scarce 32-bit virtual address space to the > > c-heap). > > > > I note, though, that you have a rather modest (~2.6 GB?) heap, so on > > Solaris > > > > for example, i'd expect plenty of free address space in the sbrk > > segment. What > > > > platform are you on? > > > > > > > > -- ramki. > > > > > > > > > > We are already setting > > > > > > > > > > > > -XX:ReservedCodeCacheSize=64M > > > > > > > > > > > > > > > > > > > > > > > > As per discussion in thread > > > > > > > > > > > > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5075468 > > > > > > > > > > > > > > > > > > > > > > > > I am sure we have enough heap as well as swap space available. > > > > > > > > > > > > > > > > > > > > > > > > I am wondering above bug is fixed for 1.5.13? > > > > > > > > > > > > What is better option increasing -XX:ReservedCodeCacheSize or > > decreasing > > > > > > heap size so that we have more non heap memory available? > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > ANuj > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jvm parameter > > > > > > > > > > > > with -Xms2560m -Xmx2560m -XX:MaxPermSize=256M > > > > > > > > > > > > -XX:+UseConcMarkSweepGC > > > > > > > > > > > > -XX:+UseParNewGC > > > > > > > > > > > > -XX:+CMSParallelRemarkEnabled > > > > > > > > > > > > -XX:+UseCMSCompactAtFullCollection > > > > > > > > > > > > -XX:CMSFullGCsBeforeCompac > > > > > > > > > > > > tion=0 -XX:+HandlePromotionFailure > > > > > > > > > > > > -XX:CMSInitiatingOccupancyFraction=68 > > > > > > > > > > > > -XX:+UseCMSInitiatingOccupancyOnly - > > > > > > > > > > > > -XX:CMSMarkStackSize=32M > > > > > > > > > > > > XX:CodeCacheMinimumFreeSpace=2M > > > > > > > > > > > > -XX:ReservedCodeCacheSize=64M > > > > > > > > > > > > -XX:NewSize=512M -XX:MaxNewSize=51 > > > > > > > > > > > > 2M -XX:MaxTenuringThreshold=6 -XX:SurvivorRatio=5 > > -XX:TargetSurvivorRatio= > > > > > > > > > > > > -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled > > > > > > -XX:CMSMaxAbortablePrecleanTime=1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wlstart: > > > > > > > > > > > > par new generation total 449408K, used 50959K [0x46400000, > > 0x66400000, > > > > > > 0x66400000) > > > > > > > > > > > > eden space 374528K, 0% used [0x46400000, 0x46400000, > 0x5d1c0000) > > > > > > > > > > > > from space 74880K, 68% used [0x61ae0000, 0x64ca3ed0, > 0x66400000) > > > > > > > > > > > > to space 74880K, 0% used [0x5d1c0000, 0x5d1c0000, > 0x61ae0000) > > > > > > > > > > > > concurrent mark-sweep generation total 2097152K, used 1071568K > > [0x66400000, > > > > > > 0xe6400000, 0xe6400000) > > > > > > > > > > > > concurrent-mark-sweep perm gen total 211688K, used 127055K > > [0xe6400000, > > > > > > 0xf32ba000, 0xf6400000) > > > > > > > > > > > > } > > > > > > > > > > > > , 0.5049826 secs] > > > > > > > > > > > > > > > > > > > > > > > > Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: > > requested > > > > > > 32756 bytes for ChunkPool::allocate. Out of swap space? > > > > > > > > > > > > Java Result: 1 > > > > > > > > > > > > > > > > > > > > -- > > > > > I would remember that if researchers were not ambitious > > > > > probably today we haven't the technology we are using! > > > From alal at hotwire.com Wed Nov 28 13:08:13 2007 From: alal at hotwire.com (Anuj Lal) Date: Wed, 28 Nov 2007 13:08:13 -0800 Subject: OOM error References: <35005D6C4D5C8D408B5E62D06458503C0266B030@sfmbx01.ad.hotwirebiz.com> <5d649bdb0711271446p749bb7cah73451784eeaabd5d@mail.gmail.com> <35005D6C4D5C8D408B5E62D06458503C0266B03D@sfmbx01.ad.hotwirebiz.com> <35005D6C4D5C8D408B5E62D06458503C0266B075@sfmbx01.ad.hotwirebiz.com> Message-ID: <35005D6C4D5C8D408B5E62D06458503C0266B11D@sfmbx01.ad.hotwirebiz.com> Is this XX:OnError= option valid for jdk 1.5 ? According to this thread its only for 1.6 jdk http://forum.java.sun.com/thread.jspa?threadID=5145549 anuj -----Original Message----- From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] Sent: Tuesday, November 27, 2007 11:40 PM To: Anuj Lal Cc: Neo Jia; hotspot-runtime-dev at openjdk.java.net Subject: Re: RE: RE: OOM error In the case of the enclosed pmap (which you indicate is from a JVM that has not encountered the C-heap exhaustion), as you might expect there is space available in the C-heap (shown by the "[heap]" annotated segment in yr pmap dump below). So we'll need to look at the pmap from a failing process. You could one of two things (or perhaps both):- -XX:OnError="pmap %p" will pmap the process just before exit with the error; otherwise you can do -XX:OnError="gcore %p" and then pmap the resulting core file. That should show the C-heap exhausted and you'd be able to see what went wrong. For additional information, since it appears that the process runs for a while before running into this C-heap exhaustion, you might want to run pmap on the process periodically and collect this pmap time-trace. You'd then likely see what it was that was over time consuming address space and running you out of C-heap. Additionally (also in -XX:OnError) pfiles may provide additional information. Hope that helps. PS: Since this is technically not a GC issue, per se, I'd probably take this off the GC list. I am accordingly cross-posting to the hotspot-runtime-dev list where it might be a better fit and where there might be further suggestions. -- ramki ----- Original Message ----- From: Anuj Lal Date: Tuesday, November 27, 2007 4:30 pm Subject: RE: RE: OOM error To: Y Srinivas Ramakrishna Cc: Neo Jia , hotspot-gc-dev at openjdk.java.net > Ramki > > This is pmap info when it is working correctly > We had same issue on other server also. Its becoming p1 issue now. I"ll > opem official sun case for this > > Anuj > > > > 20139: /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/bin/java > -Xms2560m > 00010000 64K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/bin/java > 0002E000 16K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/bin/java > 00032000 135328K rwx-- [ heap ] > 24C00000 5904K r--s- dev:85,5 ino:169421 > 25380000 408K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libfontmana > ger.so > 253F4000 96K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libfontmana > ger.so > 25480000 24K rwx-R [ anon ] > 25486000 488K rw--R [ anon ] > 25580000 24K rwx-R [ anon ] > 25586000 488K rw--R [ anon ] > 25680000 24K ----R [ anon ] > 25686000 488K rw--R [ anon ] > 25780000 24K rwx-R [ anon ] > 25786000 488K rw--R [ anon ] > 25880000 24K rwx-R [ anon ] > 25886000 488K rw--R [ anon ] > 25980000 24K rwx-R [ anon ] > 25986000 488K rw--R [ anon ] > 25A80000 24K rwx-R [ anon ] > 25A86000 488K rw--R [ anon ] > 25B80000 24K rwx-R [ anon ] > 25B86000 488K rw--R [ anon ] > 25C80000 24K rwx-R [ anon ] > 25C86000 488K rw--R [ anon ] > 25D80000 24K ----R [ anon ] > 25D86000 488K rw--R [ anon ] > 25E80000 24K ----R [ anon ] > 25E86000 488K rw--R [ anon ] > 25F80000 24K ----R [ anon ] > 25F86000 488K rw--R [ anon ] > 26080000 24K ----R [ anon ] > 26086000 488K rw--R [ anon ] > 26180000 24K ----R [ anon ] > 26186000 488K rw--R [ anon ] > 26280000 24K ----R [ anon ] > 26286000 488K rw--R [ anon ] > 26380000 24K ----R [ anon ] > 26386000 488K rw--R [ anon ] > 26480000 24K ----R [ anon ] > 26486000 488K rw--R [ anon ] > 26580000 24K ----R [ anon ] > 26586000 488K rw--R [ anon ] > 26680000 24K ----R [ anon ] > 26686000 488K rw--R [ anon ] > 26780000 24K ----R [ anon ] > 26786000 488K rw--R [ anon ] > 26880000 24K ----R [ anon ] > 26886000 488K rw--R [ anon ] > 26980000 504K r--s- dev:85,5 ino:169364 > 26A80000 1424K r--s- dev:85,5 ino:169360 > 26C00000 24K ----R [ anon ] > 26C06000 488K rw--R [ anon ] > 26D00000 24K ----R [ anon ] > 26D06000 488K rw--R [ anon ] > 26E00000 24K ----R [ anon ] > 26E06000 488K rw--R [ anon ] > 26F00000 24K ----R [ anon ] > 26F06000 488K rw--R [ anon ] > 27000000 24K ----R [ anon ] > 27006000 488K rw--R [ anon ] > 27100000 1056K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libmlib_ima > ge.so > 27216000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libmlib_ima > ge.so > 27280000 928K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libawt.so > 27376000 40K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libawt.so > 27380000 152K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libawt.so > 27400000 4432K r--s- dev:85,5 ino:89133 > 27900000 24K ----R [ anon ] > 27906000 488K rw--R [ anon ] > 27A00000 24K ----R [ anon ] > 27A06000 488K rw--R [ anon ] > 27B00000 648K r--s- dev:85,5 ino:89132 > 27C00000 24K ----R [ anon ] > 27C06000 488K rw--R [ anon ] > 27D00000 24K ----R [ anon ] > 27D06000 488K rw--R [ anon ] > 27E00000 24K ----R [ anon ] > 27E06000 488K rw--R [ anon ] > 27F00000 24K ----R [ anon ] > 27F06000 488K rw--R [ anon ] > 28000000 24K ----R [ anon ] > 28006000 488K rw--R [ anon ] > 28100000 24K ----R [ anon ] > 28106000 488K rw--R [ anon ] > 28200000 24K ----R [ anon ] > 28206000 488K rw--R [ anon ] > 28300000 24K ----R [ anon ] > 28306000 488K rw--R [ anon ] > 28400000 24K ----R [ anon ] > 28406000 488K rw--R [ anon ] > 28500000 24K rwx-R [ anon ] > 28506000 488K rw--R [ anon ] > 28600000 24K ----R [ anon ] > 28606000 488K rw--R [ anon ] > 28700000 24K ----R [ anon ] > 28706000 488K rw--R [ anon ] > 28800000 24K ----R [ anon ] > 28806000 488K rw--R [ anon ] > 28900000 24K ----R [ anon ] > 28906000 488K rw--R [ anon ] > 28A00000 24K ----R [ anon ] > 28A06000 488K rw--R [ anon ] > 28B00000 24K ----R [ anon ] > 28B06000 488K rw--R [ anon ] > 28C00000 24K ----R [ anon ] > 28C06000 488K rw--R [ anon ] > 28D00000 24K ----R [ anon ] > 28D06000 488K rw--R [ anon ] > 28E00000 24K ----R [ anon ] > 28E06000 488K rw--R [ anon ] > 28F00000 24K ----R [ anon ] > 28F06000 488K rw--R [ anon ] > 29000000 24K ----R [ anon ] > 29006000 488K rw--R [ anon ] > 29100000 24K ----R [ anon ] > 29106000 488K rw--R [ anon ] > 29200000 24K ----R [ anon ] > 29206000 488K rw--R [ anon ] > 29300000 24K ----R [ anon ] > 29306000 488K rw--R [ anon ] > 29400000 24K ----R [ anon ] > 29406000 488K rw--R [ anon ] > 29500000 24K ----R [ anon ] > 29506000 488K rw--R [ anon ] > 29600000 24K ----R [ anon ] > 29606000 488K rw--R [ anon ] > 29700000 24K ----R [ anon ] > 29706000 488K rw--R [ anon ] > 29800000 24K ----R [ anon ] > 29806000 488K rw--R [ anon ] > 29900000 24K ----R [ anon ] > 29906000 488K rw--R [ anon ] > 29A00000 24K ----R [ anon ] > 29A06000 488K rw--R [ anon ] > 29B00000 24K ----R [ anon ] > 29B06000 488K rw--R [ anon ] > 29C00000 24K ----R [ anon ] > 29C06000 488K rw--R [ anon ] > 29D00000 24K ----R [ anon ] > 29D06000 488K rw--R [ anon ] > 29E00000 24K ----R [ anon ] > 29E06000 488K rw--R [ anon ] > 29F00000 24K ----R [ anon ] > 29F06000 488K rw--R [ anon ] > 2A000000 24K ----R [ anon ] > 2A006000 488K rw--R [ anon ] > 2A100000 24K ----R [ anon ] > 2A106000 488K rw--R [ anon ] > 2A200000 24K ----R [ anon ] > 2A206000 488K rw--R [ anon ] > 2A300000 24K ----R [ anon ] > 2A306000 488K rw--R [ anon ] > 2A400000 24K ----R [ anon ] > 2A406000 488K rw--R [ anon ] > 2A500000 24K ----R [ anon ] > 2A506000 488K rw--R [ anon ] > 2A600000 24K ----R [ anon ] > 2A606000 488K rw--R [ anon ] > 2A700000 24K ----R [ anon ] > 2A706000 488K rw--R [ anon ] > 2A800000 24K ----R [ anon ] > 2A806000 488K rw--R [ anon ] > 2A900000 24K ----R [ anon ] > 2A906000 488K rw--R [ anon ] > 2AA00000 24K ----R [ anon ] > 2AA06000 488K rw--R [ anon ] > 2AB00000 24K ----R [ anon ] > 2AB06000 488K rw--R [ anon ] > 2AC00000 24K ----R [ anon ] > 2AC06000 488K rw--R [ anon ] > 2AD00000 24K ----R [ anon ] > 2AD06000 488K rw--R [ anon ] > 2AE00000 24K ----R [ anon ] > 2AE06000 488K rw--R [ anon ] > 2AF00000 24K ----R [ anon ] > 2AF06000 488K rw--R [ anon ] > 2B000000 24K ----R [ anon ] > 2B006000 488K rw--R [ anon ] > 2B100000 24K ----R [ anon ] > 2B106000 488K rw--R [ anon ] > 2B200000 24K ----R [ anon ] > 2B206000 488K rw--R [ anon ] > 2B300000 24K ----R [ anon ] > 2B306000 488K rw--R [ anon ] > 2B400000 24K ----R [ anon ] > 2B406000 488K rw--R [ anon ] > 2B500000 24K ----R [ anon ] > 2B506000 488K rw--R [ anon ] > 2B600000 24K ----R [ anon ] > 2B606000 488K rw--R [ anon ] > 2B700000 24K ----R [ anon ] > 2B706000 488K rw--R [ anon ] > 2B800000 24K ----R [ anon ] > 2B806000 488K rw--R [ anon ] > 2B900000 24K ----R [ anon ] > 2B906000 488K rw--R [ anon ] > 2BA00000 24K ----R [ anon ] > 2BA06000 488K rw--R [ anon ] > 2BB00000 24K ----R [ anon ] > 2BB06000 488K rw--R [ anon ] > 2BC00000 24K ----R [ anon ] > 2BC06000 488K rw--R [ anon ] > 2BD00000 24K ----R [ anon ] > 2BD06000 488K rw--R [ anon ] > 2BE00000 24K ----R [ anon ] > 2BE06000 488K rw--R [ anon ] > 2BF00000 24K ----R [ anon ] > 2BF06000 488K rw--R [ anon ] > 2C000000 24K ----R [ anon ] > 2C006000 488K rw--R [ anon ] > 2C100000 24K ----R [ anon ] > 2C106000 488K rw--R [ anon ] > 2C200000 24K ----R [ anon ] > 2C206000 488K rw--R [ anon ] > 2C300000 24K ----R [ anon ] > 2C306000 488K rw--R [ anon ] > 2C400000 24K ----R [ anon ] > 2C406000 488K rw--R [ anon ] > 2C500000 24K ----R [ anon ] > 2C506000 488K rw--R [ anon ] > 2C600000 24K ----R [ anon ] > 2C606000 488K rw--R [ anon ] > 2C700000 24K ----R [ anon ] > 2C706000 488K rw--R [ anon ] > 2C800000 24K ----R [ anon ] > 2C806000 488K rw--R [ anon ] > 2C900000 24K ----R [ anon ] > 2C906000 488K rw--R [ anon ] > 2CA00000 24K ----R [ anon ] > 2CA06000 488K rw--R [ anon ] > 2CB00000 24K ----R [ anon ] > 2CB06000 488K rw--R [ anon ] > 2CC00000 24K ----R [ anon ] > 2CC06000 488K rw--R [ anon ] > 2CD00000 24K ----R [ anon ] > 2CD06000 488K rw--R [ anon ] > 2CE00000 24K ----R [ anon ] > 2CE06000 488K rw--R [ anon ] > 2CF00000 24K ----R [ anon ] > 2CF06000 488K rw--R [ anon ] > 2D000000 24K ----R [ anon ] > 2D006000 488K rw--R [ anon ] > 2D100000 24K ----R [ anon ] > 2D106000 488K rw--R [ anon ] > 2D200000 24K ----R [ anon ] > 2D206000 488K rw--R [ anon ] > 2D300000 24K ----R [ anon ] > 2D306000 488K rw--R [ anon ] > 2D400000 24K ----R [ anon ] > 2D406000 488K rw--R [ anon ] > 2D500000 24K ----R [ anon ] > 2D506000 488K rw--R [ anon ] > 2D600000 24K ----R [ anon ] > 2D606000 488K rw--R [ anon ] > 2D700000 24K ----R [ anon ] > 2D706000 488K rw--R [ anon ] > 2D800000 24K ----R [ anon ] > 2D806000 488K rw--R [ anon ] > 2D900000 24K ----R [ anon ] > 2D906000 488K rw--R [ anon ] > 2DA00000 24K ----R [ anon ] > 2DA06000 488K rw--R [ anon ] > 2DB00000 24K ----R [ anon ] > 2DB06000 488K rw--R [ anon ] > 2DC00000 24K ----R [ anon ] > 2DC06000 488K rw--R [ anon ] > 2DD00000 24K ----R [ anon ] > 2DD06000 488K rw--R [ anon ] > 2DE00000 24K ----R [ anon ] > 2DE06000 488K rw--R [ anon ] > 2DEB0000 176K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libjpeg.so > 2DEEC000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libjpeg.so > 2DF00000 24K ----R [ anon ] > 2DF06000 488K rw--R [ anon ] > 2DFA0000 8K rwx-- [ anon ] > 2DFB0000 96K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libdcpr.so > 2DFD6000 80K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libdcpr.so > 2E000000 24K ----R [ anon ] > 2E006000 488K rw--R [ anon ] > 2E090000 312K r--s- dev:85,5 ino:12797 > 2E0F0000 24K r--s- dev:85,5 ino:169374 > 2E100000 24K ----R [ anon ] > 2E106000 488K rw--R [ anon ] > 2E190000 32K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/headless/li > bmawt.so > 2E1A6000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/headless/li > bmawt.so > 2E1B0000 8K rwx-- [ anon ] > 2E1C0000 112K r--s- dev:85,5 ino:104786 > 2E1E0000 112K r--s- dev:85,5 ino:148909 > 2E200000 24K ----R [ anon ] > 2E206000 488K rw--R [ anon ] > 2E290000 24K r--s- dev:85,5 ino:104507 > 2E2A0000 32K r--s- dev:85,5 ino:90208 > 2E2B0000 64K r--s- dev:85,5 ino:103379 > 2E2D0000 32K r--s- dev:85,5 ino:90200 > 2E2E0000 56K r--s- dev:85,5 ino:90170 > 2E300000 24K ----R [ anon ] > 2E306000 488K rw--R [ anon ] > 2E390000 32K r--s- dev:85,5 ino:90131 > 2E3A0000 16K r--s- dev:85,5 ino:90152 > 2E3B0000 40K r--s- dev:85,5 ino:90192 > 2E3C0000 136K r-x-- /usr/lib/libelf.so.1 > 2E3F2000 8K rwx-- /usr/lib/libelf.so.1 > 2E400000 24K ----R [ anon ] > 2E406000 488K rw--R [ anon ] > 2E490000 48K r--s- dev:85,5 ino:90137 > 2E4A0000 248K r-x-- /usr/lib/libresolv.so.2 > 2E4EE000 16K rwx-- /usr/lib/libresolv.so.2 > 2E500000 1480K r--s- dev:85,5 ino:178161 > 2E680000 24K ----R [ anon ] > 2E686000 488K rw--R [ anon ] > 2E710000 56K r--s- dev:85,5 ino:90159 > 2E730000 16K rw--- [ anon ] > 2E740000 40K r-x-- /usr/ucblib/libucb.so.1 > 2E75A000 8K rwx-- /usr/ucblib/libucb.so.1 > 2E760000 24K r-x-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libmuxe > r.so > 2E774000 8K rwx-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libmuxe > r.so > 2E780000 512K rw--R [ anon ] > 2E810000 40K r--s- dev:85,5 ino:90183 > 2E820000 8K r-x-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libwlfi > leio2.so > 2E830000 16K rwx-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libwlfi > leio2.so > 2E840000 64K rw--- [ anon ] > 2E860000 8K r-x-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libstac > kdump.so > 2E870000 8K rwx-- > /opt/p4/3rdparty/bea/weblogic/9.2mp1/server/native/solaris/sparc/libstac > kdump.so > 2E880000 24K ----R [ anon ] > 2E886000 488K rw--R [ anon ] > 2E910000 48K r--s- dev:85,5 ino:90122 > 2E920000 8K r-x-- /usr/lib/libsendfile.so.1 > 2E932000 8K rwx-- /usr/lib/libsendfile.so.1 > 2E940000 8K r-x-- /usr/lib/libmd5.so.1 > 2E952000 8K rwx-- /usr/lib/libmd5.so.1 > 2E960000 40K r-x-- /usr/lib/libaio.so.1 > 2E97A000 8K rwx-- /usr/lib/libaio.so.1 > 2E980000 24K ----R [ anon ] > 2E986000 488K rw--R [ anon ] > 2EA10000 16K r--s- dev:85,5 ino:91540 > 2EA20000 24K r-x-- /usr/lib/librt.so.1 > 2EA36000 8K rwx-- /usr/lib/librt.so.1 > 2EA40000 200K r--s- dev:85,5 ino:178116 > 2EA80000 24K ----R [ anon ] > 2EA86000 488K rw--R [ anon ] > 2EB10000 32K rw--- [ anon ] > 2EB20000 32K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libnio.so > 2EB36000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libnio.so > 2EB40000 184K r--s- dev:85,5 ino:178351 > 2EB80000 2496K r--s- dev:85,5 ino:43427 > 2EE00000 784K r--s- dev:85,5 ino:8126 > 2EED0000 88K r--s- dev:85,5 ino:178142 > 2EEF0000 32K r--s- dev:85,5 ino:178349 > 2EF00000 64K r--s- dev:85,5 ino:19946 > 2EF20000 328K r--s- dev:85,5 ino:27067 > 2EF80000 776K r--s- dev:85,5 ino:39082 > 2F050000 64K r--s- dev:85,5 ino:16731 > 2F070000 376K r--s- dev:85,5 ino:7814 > 2F0E0000 288K r--s- dev:85,5 ino:7812 > 2F130000 248K r--s- dev:85,5 ino:43449 > 2F180000 1336K r--s- dev:85,5 ino:11522 > 2F2D0000 120K r--s- dev:85,5 ino:7816 > 2F300000 1496K r--s- dev:85,5 ino:7171 > 2F480000 880K r--s- dev:85,5 ino:16719 > 2F560000 56K r--s- dev:85,5 ino:8160 > 2F580000 64K r--s- dev:85,5 ino:7815 > 2F5A0000 336K r--s- dev:85,5 ino:16717 > 2F600000 1656K r--s- dev:85,5 ino:11690 > 2F7B0000 128K r--s- dev:85,5 ino:16720 > 2F7E0000 88K r--s- dev:85,5 ino:23479 > 2F800000 5344K r--s- dev:85,5 ino:25387 > 2FD40000 16K rw--- [ anon ] > 2FD50000 120K r--s- dev:85,5 ino:19782 > 2FD80000 1152K r--s- dev:85,5 ino:7946 > 2FEB0000 24K r--s- dev:85,5 ino:43425 > 2FEC0000 208K r--s- dev:85,5 ino:7944 > 2FF00000 648K r--s- dev:85,5 ino:21820 > 2FFB0000 32K r--s- dev:85,5 ino:7813 > 2FFC0000 240K r--s- dev:85,5 ino:7945 > 30000000 4904K r--s- dev:85,5 ino:157426 > 304D0000 8K rwx-- [ anon ] > 304E0000 24K r--s- dev:85,5 ino:43447 > 304F0000 120K r--s- dev:85,5 ino:16575 > 30520000 352K r--s- dev:85,5 ino:23507 > 30580000 1512K r--s- dev:85,5 ino:156707 > 30700000 616K r--s- dev:85,5 ino:8161 > 307A0000 312K r--s- dev:85,5 ino:12797 > 30800000 3968K r--s- dev:85,5 ino:9009 > 30BF0000 32K r--s- dev:85,5 ino:16718 > 30C00000 5040K r--s- dev:85,5 ino:9011 > 31100000 488K r--s- dev:85,5 ino:21821 > 31180000 480K r--s- dev:85,5 ino:44388 > 31200000 552K r--s- dev:85,5 ino:7744 > 31290000 8K r--s- dev:85,5 ino:169334 > 312A0000 352K r--s- dev:85,5 ino:26985 > 31300000 640K r--s- dev:85,5 ino:176761 > 313B0000 24K r--s- dev:85,5 ino:15965 > 313C0000 24K r--s- dev:85,5 ino:11502 > 313D0000 168K r--s- dev:85,5 ino:21694 > 31400000 6880K r--s- dev:85,5 ino:169093 > 31AC0000 16K r--s- dev:85,5 ino:15966 > 31AD0000 208K r--s- dev:85,5 ino:17946 > 31B10000 432K r--s- dev:85,5 ino:15781 > 31B80000 680K r--s- dev:85,5 ino:170149 > 31C30000 8K r--s- dev:85,5 ino:23497 > 31C40000 224K r--s- dev:85,5 ino:20121 > 31C80000 1000K r--s- dev:85,5 ino:169091 > 31D80000 976K r--s- dev:85,5 ino:169088 > 31E80000 2544K r--s- dev:85,5 ino:188396 > 32100000 512K r--s- dev:85,5 ino:170763 > 32190000 80K r--s- dev:85,5 ino:23502 > 321B0000 40K r--s- dev:85,5 ino:44387 > 321C0000 80K r--s- dev:85,5 ino:17995 > 321E0000 72K r--s- dev:85,5 ino:39073 > 32200000 1496K r--s- dev:85,5 ino:170766 > 32380000 776K r--s- dev:85,5 ino:188388 > 32450000 16K r--s- dev:85,5 ino:15778 > 32460000 112K r--s- dev:85,5 ino:7139 > 32480000 968K r--s- dev:85,5 ino:171312 > 32580000 752K r--s- dev:85,5 ino:176705 > 32650000 32K r--s- dev:85,5 ino:23508 > 32660000 72K r--s- dev:85,5 ino:13123 > 32680000 912K r--s- dev:85,5 ino:176805 > 32770000 32K r--s- dev:85,5 ino:20119 > 32780000 1512K r--s- dev:85,5 ino:178382 > 32900000 2616K r--s- dev:85,5 ino:169089 > 32BA0000 48K r--s- dev:85,5 ino:20243 > 32BB0000 256K r--s- dev:85,5 ino:170462 > 32C00000 52536K r--s- dev:85,5 ino:178597 > 35F60000 152K r--s- dev:85,5 ino:7752 > 35F90000 392K r--s- dev:85,5 ino:170432 > 36000000 1864K r--s- dev:85,5 ino:187108 > 361E0000 24K r--s- dev:85,5 ino:21822 > 361F0000 32K r--s- dev:85,5 ino:7745 > 36200000 1784K r--s- dev:85,5 ino:157391 > 363D0000 120K r--s- dev:85,5 ino:7190 > 36400000 6880K r--s- dev:85,5 ino:94624 > 36AC0000 200K r--s- dev:85,5 ino:188391 > 36B00000 648K r--s- dev:85,5 ino:178589 > 36BB0000 160K r--s- dev:85,5 ino:170190 > 36BE0000 112K r--s- dev:85,5 ino:170154 > 36C00000 6936K r--s- dev:85,5 ino:24574 > 372D0000 8K r--s- dev:85,5 ino:31008 > 372E0000 104K r--s- dev:85,5 ino:170159 > 37300000 2128K r--s- dev:85,5 ino:157398 > 37520000 96K r--s- dev:85,5 ino:170157 > 37540000 216K r--s- dev:85,5 ino:170483 > 37580000 2752K r--s- dev:85,5 ino:24575 > 37840000 32K r--s- dev:85,5 ino:7748 > 37850000 120K r--s- dev:85,5 ino:187721 > 37880000 3432K r--s- dev:85,5 ino:23916 > 37BE0000 8K r--s- dev:85,5 ino:7141 > 37BF0000 48K r--s- dev:85,5 ino:7800 > 37C00000 4096K rwx-- [ anon ] > 38010000 8K r--s- dev:85,5 ino:9010 > 38020000 192K r--s- dev:85,5 ino:187853 > 38060000 88K r--s- dev:85,5 ino:188385 > 38080000 1768K r--s- dev:85,5 ino:23915 > 38240000 192K r--s- dev:85,5 ino:187665 > 38280000 1016K r--s- dev:85,5 ino:7170 > 38390000 40K r--s- dev:85,5 ino:176570 > 383A0000 312K r--s- dev:85,5 ino:188386 > 38400000 131072K rwx-- [ anon ] > 40410000 424K r--s- dev:85,5 ino:187496 > 40480000 864K r--s- dev:85,5 ino:178588 > 40560000 72K r--s- dev:85,5 ino:178111 > 40580000 696K r--s- dev:85,5 ino:20117 > 40640000 8K r--s- dev:85,5 ino:170155 > 40650000 144K r--s- dev:85,5 ino:171315 > 40680000 1072K r--s- dev:85,5 ino:16891 > 407A0000 112K r--s- dev:85,5 ino:188378 > 407C0000 64K r--s- dev:85,5 ino:171316 > 407E0000 72K r--s- dev:85,5 ino:171306 > 40800000 73728K rwx-- [ anon ] > 45000000 8K rwx-- [ anon ] > 45010000 24K r--s- dev:85,5 ino:171297 > 45020000 24K r--s- dev:85,5 ino:171289 > 45030000 8K r--s- dev:85,5 ino:171305 > 45040000 8K r--s- dev:85,5 ino:171296 > 45050000 16K r--s- dev:85,5 ino:171288 > 45060000 8K r--s- dev:85,5 ino:171304 > 45070000 8K r--s- dev:85,5 ino:171295 > 45080000 168K r--s- dev:85,5 ino:171287 > 450B0000 8K r--s- dev:85,5 ino:171311 > 450C0000 40K r--s- dev:85,5 ino:171303 > 450D0000 8K r--s- dev:85,5 ino:171294 > 450E0000 40K r--s- dev:85,5 ino:171286 > 450F0000 8K r--s- dev:85,5 ino:171310 > 45100000 792K r--s- dev:85,5 ino:25468 > 451D0000 8K r--s- dev:85,5 ino:171293 > 451E0000 8K r--s- dev:85,5 ino:171285 > 451F0000 8K r--s- dev:85,5 ino:171309 > 45200000 24K ----R [ anon ] > 45206000 488K rw--R [ anon ] > 45290000 16K r--s- dev:85,5 ino:171301 > 452A0000 48K r--s- dev:85,5 ino:171292 > 452B0000 8K r--s- dev:85,5 ino:171284 > 452C0000 16K r--s- dev:85,5 ino:171308 > 452D0000 16K r--s- dev:85,5 ino:171300 > 452E0000 8K r--s- dev:85,5 ino:171291 > 452F0000 48K r--s- dev:85,5 ino:171307 > 45300000 24K ----R [ anon ] > 45306000 488K rw--R [ anon ] > 45390000 400K r--s- dev:85,5 ino:171302 > 45400000 4096K rwx-- [ anon ] > 45800000 8K rwx-- [ anon ] > 45810000 16K r--s- dev:85,5 ino:171283 > 45820000 72K r--s- dev:85,5 ino:171298 > 45840000 16K r--s- dev:85,5 ino:171290 > 45850000 8K r--s- dev:85,5 ino:171282 > 45860000 112K r--s- dev:85,5 ino:171271 > 45880000 472K r--s- dev:85,5 ino:24753 > 45900000 24K ----R [ anon ] > 45906000 488K rw--R [ anon ] > 45990000 136K r--s- dev:85,5 ino:171270 > 459C0000 64K r--s- dev:85,5 ino:171269 > 459E0000 24K r--s- dev:85,5 ino:176581 > 459F0000 16K r--s- dev:85,5 ino:176577 > 45A00000 24K ----R [ anon ] > 45A06000 488K rw--R [ anon ] > 45A90000 392K r--s- dev:85,5 ino:176527 > 45B00000 40K rwx-- [ anon ] > 45B0A000 8K rwx-- [ anon ] > 45B0C000 8K rwx-- [ anon ] > 45B0E000 8K rwx-- [ anon ] > 45B10000 8K rwx-- [ anon ] > 45B12000 8K rwx-- [ anon ] > 45B14000 8K rwx-- [ anon ] > 45B16000 8K rwx-- [ anon ] > 45B18000 8K rwx-- [ anon ] > 45B1A000 8K rwx-- [ anon ] > 45B1C000 8K rwx-- [ anon ] > 45B1E000 8K rwx-- [ anon ] > 45B20000 8K rwx-- [ anon ] > 45B22000 8K rwx-- [ anon ] > 45B24000 8K rwx-- [ anon ] > 45B26000 8K rwx-- [ anon ] > 45B28000 8K rwx-- [ anon ] > 45B2A000 8K rwx-- [ anon ] > 45B2C000 8K rwx-- [ anon ] > 45B2E000 8K rwx-- [ anon ] > 45B30000 8K rwx-- [ anon ] > 45B32000 8K rwx-- [ anon ] > 45B34000 8K rwx-- [ anon ] > 45B36000 8K rwx-- [ anon ] > 45B38000 8K rwx-- [ anon ] > 45B3A000 152K rwx-- [ anon ] > 45B60000 136K ----R [ anon ] > 45B90000 256K r--s- dev:85,5 ino:176512 > 45BE0000 80K r--s- dev:85,5 ino:188399 > 45C00000 1024K rwx-- [ anon ] > 45D00000 4096K rwx-- [ anon ] > 46100000 32K rwx-- [ anon ] > 46108000 8K rwx-- [ anon ] > 4610A000 8K rwx-- [ anon ] > 4610C000 8K rwx-- [ anon ] > 4610E000 8K rwx-- [ anon ] > 46110000 8K rwx-- [ anon ] > 46112000 8K rwx-- [ anon ] > 46114000 8K rwx-- [ anon ] > 46116000 8K rwx-- [ anon ] > 46118000 8K rwx-- [ anon ] > 4611A000 8K rwx-- [ anon ] > 4611C000 8K rwx-- [ anon ] > 4611E000 8K rwx-- [ anon ] > 46120000 8K rwx-- [ anon ] > 46122000 8K rwx-- [ anon ] > 46124000 8K rwx-- [ anon ] > 46126000 8K rwx-- [ anon ] > 46128000 8K rwx-- [ anon ] > 4612A000 8K rwx-- [ anon ] > 4612C000 8K rwx-- [ anon ] > 4612E000 8K rwx-- [ anon ] > 46130000 8K rwx-- [ anon ] > 46132000 8K rwx-- [ anon ] > 46134000 8K rwx-- [ anon ] > 46136000 8K rwx-- [ anon ] > 46138000 8K rwx-- [ anon ] > 4613A000 152K rwx-- [ anon ] > 46160000 128K ----R [ anon ] > 46180000 8K rwx-- [ anon ] > 46190000 392K r--s- dev:85,5 ino:177742 > 46200000 24K ----R [ anon ] > 46206000 488K rw--R [ anon ] > 46290000 8K r--s- dev:85,5 ino:176568 > 462A0000 8K r--s- dev:85,5 ino:187556 > 462B0000 40K r--s- dev:85,5 ino:178583 > 462C0000 40K r--s- dev:85,5 ino:178576 > 462D0000 16K r--s- dev:85,5 ino:169518 > 462E0000 8K r--s- dev:85,5 ino:168455 > 462F0000 40K r--s- dev:85,5 ino:168458 > 46300000 24K ----R [ anon ] > 46306000 488K rw--R [ anon ] > 46390000 16K r--s- dev:85,5 ino:168457 > 463A0000 24K r--s- dev:85,5 ino:168454 > 463B0000 8K r--s- dev:85,5 ino:168477 > 463C0000 64K r--s- dev:85,5 ino:24571 > 463E0000 96K r--s- dev:85,5 ino:24573 > 46400000 524288K rwx-- [ anon ] > 66400000 2097152K rwx-- [ anon ] > E6400000 16384K rwx-- [ anon ] > E7400000 4096K rwx-- [ anon ] > E7800000 4096K rwx-- [ anon ] > E7C00000 4096K rwx-- [ anon ] > E8000000 4096K rwx-- [ anon ] > E8400000 4096K rwx-- [ anon ] > E8800000 4096K rwx-- [ anon ] > E8C00000 4096K rwx-- [ anon ] > E9000000 4096K rwx-- [ anon ] > E9400000 4096K rwx-- [ anon ] > E9800000 4096K rwx-- [ anon ] > E9C00000 4096K rwx-- [ anon ] > EA000000 4096K rwx-- [ anon ] > EA400000 4096K rwx-- [ anon ] > EA800000 4096K rwx-- [ anon ] > EAC00000 4096K rwx-- [ anon ] > EB000000 4096K rwx-- [ anon ] > EB400000 4096K rwx-- [ anon ] > EB800000 4096K rwx-- [ anon ] > EBC00000 4096K rwx-- [ anon ] > EC000000 4096K rwx-- [ anon ] > EC400000 4096K rwx-- [ anon ] > EC800000 4096K rwx-- [ anon ] > ECC00000 4096K rwx-- [ anon ] > ED000000 4096K rwx-- [ anon ] > ED400000 4096K rwx-- [ anon ] > ED800000 77824K rwx-- [ anon ] > F2400000 65536K ----R [ anon ] > F6410000 8K r--s- dev:85,5 ino:168461 > F6420000 216K r--s- dev:85,5 ino:24576 > F6460000 8K r--s- dev:85,5 ino:24750 > F6470000 8K r--s- dev:85,5 ino:24747 > F6480000 336K r--s- dev:85,5 ino:23914 > F64E0000 88K r--s- dev:85,5 ino:23913 > F6500000 24K ----R [ anon ] > F6506000 488K rw--R [ anon ] > F6590000 32K r--s- dev:85,5 ino:24751 > F65A0000 88K r--s- dev:85,5 ino:9002 > F65C0000 200K r--s- dev:85,5 ino:7151 > F6600000 24K ----R [ anon ] > F6606000 488K rw--R [ anon ] > F6690000 16K r--s- dev:85,5 ino:7143 > F66A0000 72K r--s- dev:85,5 ino:7165 > F66C0000 16K r--s- dev:85,5 ino:7144 > F66D0000 80K r--s- dev:85,5 ino:7158 > F66F0000 16K r--s- dev:85,5 ino:7167 > F6700000 512K rw--R [ anon ] > F6790000 416K r--s- dev:85,5 ino:7161 > F6800000 4096K rwx-- [ anon ] > F6C00000 4096K rwx-- [ anon ] > F7000000 4096K rwx-- [ anon ] > F7400000 4096K rwx-- [ anon ] > F7800000 4096K rwx-- [ anon ] > F7C00000 4096K rwx-- [ anon ] > F8000000 4096K rwx-- [ anon ] > F8400000 4096K rwx-- [ anon ] > F8800000 4096K rwx-- [ anon ] > F8C00000 4096K rwx-- [ anon ] > F9000000 4096K rwx-- [ anon ] > F9400000 4096K rwx-- [ anon ] > F9800000 16384K ----R [ anon ] > FA810000 16K r--s- dev:85,5 ino:7159 > FA820000 368K r--s- dev:85,5 ino:211317 > FA880000 488K r--s- dev:85,5 ino:8285 > FA900000 512K rw--R [ anon ] > FA990000 392K r--s- dev:85,5 ino:102341 > FAA00000 584K rwx-- [ anon ] > FAAA0000 328K r--s- dev:85,5 ino:211325 > FAB00000 512K rw--R [ anon ] > FAB90000 112K r--s- dev:85,5 ino:7179 > FABB0000 16K r--s- dev:85,5 ino:211316 > FABC0000 224K r--s- dev:85,5 ino:102332 > FAC00000 8640K r--s- dev:85,5 ino:25457 > FB480000 512K rw--R [ anon ] > FB510000 64K r--s- dev:85,5 ino:8273 > FB530000 64K r--s- dev:85,5 ino:8276 > FB550000 168K r--s- dev:85,5 ino:8270 > FB580000 512K rw--R [ anon ] > FB610000 48K r--s- dev:85,5 ino:7498 > FB620000 352K r--s- dev:85,5 ino:26726 > FB680000 24K rwx-- [ anon ] > FB686000 8K rwx-- [ anon ] > FB688000 8K rwx-- [ anon ] > FB68A000 8K rwx-- [ anon ] > FB68C000 8K rwx-- [ anon ] > FB68E000 8K rwx-- [ anon ] > FB690000 8K rwx-- [ anon ] > FB692000 8K rwx-- [ anon ] > FB694000 8K rwx-- [ anon ] > FB696000 8K rwx-- [ anon ] > FB698000 8K rwx-- [ anon ] > FB69A000 8K rwx-- [ anon ] > FB69C000 8K rwx-- [ anon ] > FB69E000 8K rwx-- [ anon ] > FB6A0000 8K rwx-- [ anon ] > FB6A2000 8K rwx-- [ anon ] > FB6A4000 8K rwx-- [ anon ] > FB6A6000 8K rwx-- [ anon ] > FB6A8000 8K rwx-- [ anon ] > FB6AA000 8K rwx-- [ anon ] > FB6AC000 8K rwx-- [ anon ] > FB6AE000 8K rwx-- [ anon ] > FB6B0000 8K rwx-- [ anon ] > FB6B2000 8K rwx-- [ anon ] > FB6B4000 8K rwx-- [ anon ] > FB6B6000 8K rwx-- [ anon ] > FB6B8000 8K rwx-- [ anon ] > FB6BA000 8K rwx-- [ anon ] > FB6BC000 8K rwx-- [ anon ] > FB6BE000 8K rwx-- [ anon ] > FB6C0000 8K rwx-- [ anon ] > FB6C2000 8K rwx-- [ anon ] > FB6C4000 8K rwx-- [ anon ] > FB6C6000 8K rwx-- [ anon ] > FB6C8000 8K rwx-- [ anon ] > FB6CA000 8K rwx-- [ anon ] > FB6CC000 8K rwx-- [ anon ] > FB6CE000 8K rwx-- [ anon ] > FB6D0000 8K rwx-- [ anon ] > FB6D2000 8K rwx-- [ anon ] > FB6D4000 8K rwx-- [ anon ] > FB6D6000 8K rwx-- [ anon ] > FB6D8000 8K rwx-- [ anon ] > FB6DA000 8K rwx-- [ anon ] > FB6DC000 8K rwx-- [ anon ] > FB6DE000 8K rwx-- [ anon ] > FB6E0000 8K rwx-- [ anon ] > FB6E2000 8K rwx-- [ anon ] > FB6E4000 8K rwx-- [ anon ] > FB6E6000 8K rwx-- [ anon ] > FB6E8000 8K rwx-- [ anon ] > FB6EA000 8K rwx-- [ anon ] > FB6EC000 8K rwx-- [ anon ] > FB6EE000 8K rwx-- [ anon ] > FB6F0000 8K rwx-- [ anon ] > FB6F2000 8K rwx-- [ anon ] > FB6F4000 8K rwx-- [ anon ] > FB6F6000 8K rwx-- [ anon ] > FB6F8000 8K rwx-- [ anon ] > FB6FA000 8K rwx-- [ anon ] > FB6FC000 8K rwx-- [ anon ] > FB6FE000 8K rwx-- [ anon ] > FB700000 8K rwx-- [ anon ] > FB702000 8K rwx-- [ anon ] > FB704000 8K rwx-- [ anon ] > FB706000 8K rwx-- [ anon ] > FB708000 8K rwx-- [ anon ] > FB70A000 8K rwx-- [ anon ] > FB70C000 8K rwx-- [ anon ] > FB70E000 8K rwx-- [ anon ] > FB710000 8K rwx-- [ anon ] > FB712000 8K rwx-- [ anon ] > FB714000 8K rwx-- [ anon ] > FB716000 8K rwx-- [ anon ] > FB718000 8K rwx-- [ anon ] > FB71A000 8K rwx-- [ anon ] > FB71C000 8K rwx-- [ anon ] > FB71E000 8K rwx-- [ anon ] > FB720000 8K rwx-- [ anon ] > FB722000 8K rwx-- [ anon ] > FB724000 8K rwx-- [ anon ] > FB726000 8K rwx-- [ anon ] > FB728000 8K rwx-- [ anon ] > FB72A000 8K rwx-- [ anon ] > FB72C000 8K rwx-- [ anon ] > FB72E000 8K rwx-- [ anon ] > FB730000 8K rwx-- [ anon ] > FB732000 8K rwx-- [ anon ] > FB734000 304K ----R [ anon ] > FB790000 24K r--s- dev:85,5 ino:102340 > FB7A0000 48K r--s- dev:85,5 ino:8275 > FB7B0000 304K r--s- dev:85,5 ino:16889 > FB800000 39080K r--s- dev:85,5 ino:25497 > FDE30000 8K r--s- dev:85,5 ino:7155 > FDE40000 32K r--s- dev:85,5 ino:8274 > FDE50000 120K r--s- dev:85,5 ino:8269 > FDE80000 512K rw--R [ anon ] > FDF10000 112K r--s- dev:85,5 ino:8271 > FDF30000 96K r--s- dev:85,5 ino:26727 > FDF50000 176K r--s- dev:85,5 ino:25470 > FDF80000 536K r--s- dev:85,5 ino:25490 > FE010000 88K r--s- dev:85,5 ino:26444 > FE030000 160K r--s- dev:85,5 ino:25469 > FE060000 24K r-x-- /usr/lib/nss_files.so.1 > FE076000 8K rwx-- /usr/lib/nss_files.so.1 > FE080000 504K r--s- dev:85,5 ino:24234 > FE110000 72K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libnet.so > FE130000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libnet.so > FE140000 24K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libmanageme > nt.so > FE154000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libmanageme > nt.so > FE160000 112K rw--- [ anon ] > FE180000 1312K r--s- dev:85,5 ino:26635 > FE2D0000 16K r--s- dev:85,5 ino:20250 > FE2E0000 88K r--s- dev:85,5 ino:25489 > FE300000 1176K r--s- dev:85,5 ino:26633 > FE430000 8K rwx-- [ anon ] > FE440000 192K r--s- dev:85,5 ino:26634 > FE480000 2528K r--s- dev:85,5 ino:26520 > FE700000 16K r--s- dev:85,5 ino:8280 > FE710000 192K r--s- dev:85,5 ino:26499 > FE750000 64K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libzip.so > FE760000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libzip.so > FE770000 8K rwx-- [ anon ] > FE780000 144K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libjava.so > FE7B4000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libjava.so > FE7C0000 56K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libverify.s > o > FE7DE000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/libverify.s > o > FE7F0000 8K r--s- dev:85,5 ino:25467 > FE800000 7984K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/server/libj > vm.so > FEFDC000 408K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/server/libj > vm.so > FF042000 56K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/server/libj > vm.so > FF060000 32K r-x-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/native_thre > ads/libhpi.so > FF078000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/native_thre > ads/libhpi.so > FF07A000 8K rwx-- > /opt/p4/3rdparty/sun/java/sunos-sparc/1.5.0_13/jre/lib/sparc/native_thre > ads/libhpi.so > FF080000 568K r-x-- /usr/lib/libnsl.so.1 > FF11E000 40K rwx-- /usr/lib/libnsl.so.1 > FF128000 24K rwx-- /usr/lib/libnsl.so.1 > FF140000 32K rw-s- dev:0,2 ino:253268838 > FF150000 8K rwx-- [ anon ] > FF160000 8K r---- [ anon ] > FF170000 16K rw--- [ anon ] > FF180000 16K r-x-- /usr/lib/libmp.so.2 > FF194000 8K rwx-- /usr/lib/libmp.so.2 > FF1A0000 8K rwx-- [ anon ] > FF1B0000 224K r-x-- /usr/lib/libm.so.1 > FF1F6000 8K rwx-- /usr/lib/libm.so.1 > FF200000 48K r-x-- /usr/lib/libCrun.so.1 > FF21A000 8K rwx-- /usr/lib/libCrun.so.1 > FF21C000 16K rwx-- /usr/lib/libCrun.so.1 > FF230000 8K r-x-- /usr/lib/libsched.so.1 > FF242000 8K rwx-- /usr/lib/libsched.so.1 > FF250000 40K r-x-- /usr/lib/libsocket.so.1 > FF26A000 8K rwx-- /usr/lib/libsocket.so.1 > FF280000 688K r-x-- /usr/lib/libc.so.1 > FF33C000 24K rwx-- /usr/lib/libc.so.1 > FF342000 8K rwx-- /usr/lib/libc.so.1 > FF350000 8K rwxs- [ anon ] > FF360000 16K rw--- [ anon ] > FF370000 8K rwx-- [ anon ] > FF380000 96K r-x-- /usr/lib/libthread.so.1 > FF3A8000 8K rwx-- /usr/lib/libthread.so.1 > FF3AA000 8K rwx-- /usr/lib/libthread.so.1 > FF3B0000 8K r-x-- /usr/platform/sun4u-us3/lib/libc_psr.so.1 > FF3C0000 192K r-x-- /usr/lib/ld.so.1 > FF3F0000 8K rwx-- /usr/lib/ld.so.1 > FF3F2000 8K rwx-- /usr/lib/ld.so.1 > FF3FA000 8K rwx-- /usr/lib/libdl.so.1 > FFB7C000 24K ----- [ anon ] > FF400000 8192K rw--- [ stack ] > total 3636576K > > > -----Original Message----- > From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] > Sent: Tuesday, November 27, 2007 3:26 PM > To: Anuj Lal > Cc: Neo Jia; hotspot-gc-dev at openjdk.java.net > Subject: Re: RE: OOM error > > > pmap -rs will tell you what's taking up address space. It could be, for > example, > that you have lots of threads (so the space is gone to thread stacks, > so > you might > try -Xss<...> or it could be that you have loaded lots of native > libraries that have > taken up address space, or it could be that you have lots of file > descriptors or > sockets or mmapped files or other native resources that are tying up > your > address space). > > (the reference to "swap space" in the message is usuallysomewhat > misleading; > i think there are plans to fix that message to be smarter / less > misleading). > > -- ramki > > > We are on solaris 9.0 32 bit > > > > We aleady have "low" jvm heap. We don't want to go below this. > > > > I am pretty sure we are not running out of swap space > > > > > > > > This is what we have currently > > > > 8 processes: 46 sleeping, 2 on cpu > > > > CPU states: 95.2% idle, 3.9% user, 0.8% kernel, 0.0% iowait, 0.0% > > swap > > > > Memory: 16G real, 12G free, 3510M swap in use, 18G swap free > > > > > > > > PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND > > > > 20139 weblogic 144 19 0 3389M 2362M cpu/2 87:19 5.16% java > > > > > > > > Anuj > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] > > Sent: Tuesday, November 27, 2007 3:10 PM > > To: Neo Jia > > Cc: Anuj Lal; hotspot-gc-dev at openjdk.java.net > > Subject: Re: OOM error > > > > > > > > > > > > > > > > > > > > Hi Anuj -- > > > > > > > > > > Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: > > requested > > > > > > 32756 bytes for ChunkPool::allocate. Out of swap space? > > > > > > > > This is a request for 32 KB (not 32 MB as in the bug you quoted > below). > > > > > > > > You should probably check pmap -rs on the process. You are almost > > certainly out of > > > > C heap space (so, as you suggested below, reducing the Java heap might > > help by > > > > freeing up your apparently scarce 32-bit virtual address space to the > > c-heap). > > > > I note, though, that you have a rather modest (~2.6 GB?) heap, so on > > Solaris > > > > for example, i'd expect plenty of free address space in the sbrk > > segment. What > > > > platform are you on? > > > > > > > > -- ramki. > > > > > > > > > > We are already setting > > > > > > > > > > > > -XX:ReservedCodeCacheSize=64M > > > > > > > > > > > > > > > > > > > > > > > > As per discussion in thread > > > > > > > > > > > > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5075468 > > > > > > > > > > > > > > > > > > > > > > > > I am sure we have enough heap as well as swap space available. > > > > > > > > > > > > > > > > > > > > > > > > I am wondering above bug is fixed for 1.5.13? > > > > > > > > > > > > What is better option increasing -XX:ReservedCodeCacheSize or > > decreasing > > > > > > heap size so that we have more non heap memory available? > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > ANuj > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jvm parameter > > > > > > > > > > > > with -Xms2560m -Xmx2560m -XX:MaxPermSize=256M > > > > > > > > > > > > -XX:+UseConcMarkSweepGC > > > > > > > > > > > > -XX:+UseParNewGC > > > > > > > > > > > > -XX:+CMSParallelRemarkEnabled > > > > > > > > > > > > -XX:+UseCMSCompactAtFullCollection > > > > > > > > > > > > -XX:CMSFullGCsBeforeCompac > > > > > > > > > > > > tion=0 -XX:+HandlePromotionFailure > > > > > > > > > > > > -XX:CMSInitiatingOccupancyFraction=68 > > > > > > > > > > > > -XX:+UseCMSInitiatingOccupancyOnly - > > > > > > > > > > > > -XX:CMSMarkStackSize=32M > > > > > > > > > > > > XX:CodeCacheMinimumFreeSpace=2M > > > > > > > > > > > > -XX:ReservedCodeCacheSize=64M > > > > > > > > > > > > -XX:NewSize=512M -XX:MaxNewSize=51 > > > > > > > > > > > > 2M -XX:MaxTenuringThreshold=6 -XX:SurvivorRatio=5 > > -XX:TargetSurvivorRatio= > > > > > > > > > > > > -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled > > > > > > -XX:CMSMaxAbortablePrecleanTime=1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wlstart: > > > > > > > > > > > > par new generation total 449408K, used 50959K [0x46400000, > > 0x66400000, > > > > > > 0x66400000) > > > > > > > > > > > > eden space 374528K, 0% used [0x46400000, 0x46400000, > 0x5d1c0000) > > > > > > > > > > > > from space 74880K, 68% used [0x61ae0000, 0x64ca3ed0, > 0x66400000) > > > > > > > > > > > > to space 74880K, 0% used [0x5d1c0000, 0x5d1c0000, > 0x61ae0000) > > > > > > > > > > > > concurrent mark-sweep generation total 2097152K, used 1071568K > > [0x66400000, > > > > > > 0xe6400000, 0xe6400000) > > > > > > > > > > > > concurrent-mark-sweep perm gen total 211688K, used 127055K > > [0xe6400000, > > > > > > 0xf32ba000, 0xf6400000) > > > > > > > > > > > > } > > > > > > > > > > > > , 0.5049826 secs] > > > > > > > > > > > > > > > > > > > > > > > > Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: > > requested > > > > > > 32756 bytes for ChunkPool::allocate. Out of swap space? > > > > > > > > > > > > Java Result: 1 > > > > > > > > > > > > > > > > > > > > -- > > > > > I would remember that if researchers were not ambitious > > > > > probably today we haven't the technology we are using! > > >