From bradford.wetmore at oracle.com Thu Jan 1 00:11:26 2015 From: bradford.wetmore at oracle.com (Bradford Wetmore) Date: Wed, 31 Dec 2014 16:11:26 -0800 Subject: RFR: JDK-8047769 SecureRandom should be more frugal with file descriptors In-Reply-To: <54A20475.3040206@oracle.com> References: <5492D557.2000807@gmail.com> <5498C5A0.50104@oracle.com> <549AA573.2060801@gmail.com> <54A20475.3040206@oracle.com> Message-ID: <54A490AE.5090103@oracle.com> Just to followup, I've analyzed the whole PIT run. The second one's call stack is identical to: JDK-8067344: Adjust java/lang/invoke/LFCaching/LFGarbageCollectedTest.java for recent changes in java.lang.invoke So, really the only problem is the use of Asserts in your test case. Brad >>> Looks like you have a committer status, will you be pushing this? >> >> I can, yes. As soon as we clear-out the remaining questions, right? > > Yes. The comments below are minor and shouldn't need another review > cycle. I have started a JPRT job for you, I'll run it through "core" > target which will give us: > > jdk_lang, jdk_math, jdk_util, jdk_io, jdk_net, jdk_nio, jdk_security*, > jdk_rmi, jdk_text, jdk_time, jdk_other, core_tools. > > Anything else? I'm off tomorrow, I should have full results Wed. > > Here are the preliminary results for the jobs that have finished. > jdk.testlibrary.Asserts is causing compilation errors, additional > comments below: > > /opt/jprt/T/P1/003505.brwetmor/s/jdk/test/sun/security/provider/FileInputStreamPool/FileInputStreamPoolTest.java:33: > error: package jdk.testlibrary does not exist > import static jdk.testlibrary.Asserts.*; > ^ > /opt/jprt/T/P1/003505.brwetmor/s/jdk/test/sun/security/provider/FileInputStreamPool/FileInputStreamPoolTest.java:52: > error: cannot find symbol > assertEquals(bytes.length, nread, "short read"); > ^ > symbol: method assertEquals(int,int,String) > location: class FileInputStreamPoolTest > /opt/jprt/T/P1/003505.brwetmor/s/jdk/test/sun/security/provider/FileInputStreamPool/FileInputStreamPoolTest.java:53: > error: cannot find symbol > assertTrue(Arrays.equals(readBytes, bytes), > ^ > symbol: method assertTrue(boolean,String) > location: class FileInputStreamPoolTest > 3 errors > > TEST RESULT: Failed. Compilation failed: Compilation failed > > I'm also getting failures in the following test. I unfortunately have > to leave now, so don't have time to look at this. But it did mention > "seed" so I'm mentioning it here. > > TEST: java/lang/invoke/LFCaching/LFGarbageCollectedTest.java > > ACTION: main -- Failed. Execution failed: `main' threw exception: > java.lang.Error: 36 of 39 test cases FAILED! Rerun the test with the > same "-Dseed=" option as in the log file! > REASON: User specified action: run main/othervm LFGarbageCollectedTest > TIME: 213.406 seconds > messages: > command: main LFGarbageCollectedTest > reason: User specified action: run main/othervm LFGarbageCollectedTest > elapsed time (seconds): 213.406 > STDOUT: > -Dseed=6102311124531075592 > -DtestLimit=2000 > Number of iterations according to -DtestLimit is 153 (1989 cases) > Code cache size is 251658240 bytes > Non-profiled code cache size is 123058253 bytes > Number of iterations limited by code cache size is 84 (1092 cases) > Number of iterations limited by non-profiled code cache size is 41 (533 > cases) > Number of iterations is set to 41 (533 cases) > Not enough time to continue execution. Interrupted. > STDERR: > Iteration 0: > Tested LF caching feature with MethodHandles.foldArguments method. > java.lang.AssertionError: Error: Lambda form is not garbage collected > at LFGarbageCollectedTest.doTest(LFGarbageCollectedTest.java:81) > at > LambdaFormTestCase$TestRun.doIteration(LambdaFormTestCase.java:139) > at LambdaFormTestCase$$Lambda$2/5042013.call(Unknown Source) > at > jdk.testlibrary.TimeLimitedRunner.call(TimeLimitedRunner.java:71) > at LambdaFormTestCase.runTests(LambdaFormTestCase.java:201) > at LFGarbageCollectedTest.main(LFGarbageCollectedTest.java:105) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) From bradford.wetmore at oracle.com Thu Jan 1 01:46:42 2015 From: bradford.wetmore at oracle.com (Bradford Wetmore) Date: Wed, 31 Dec 2014 17:46:42 -0800 Subject: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom In-Reply-To: <5491BF67.6070000@gmail.com> References: <5490DE08.708@oracle.com> <5491BF67.6070000@gmail.com> Message-ID: <54A4A702.4010200@oracle.com> >> To the actual proposal: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.03/ >> >> Overall, I'm ok with what's proposed. This is more straightforward to >> parse/understand than trying to adjust NativeSeedGenerator to >> create/call directly (e.g. UNIX: new >> NativeSeedGenerator("/dev/urandom") or Windows: new >> NativeSeedGenerator()). But I'd still like to understand why you >> moved away from this. >> >> One concern is that you're duplicating native libraries in java.base, >> and it would be the third JDK library overall with this type of call. >> There's one in libjava (for java.base/WinCAPISeedGenerator for >> sun.security.provider.NativeSeedGenerator) and sunmscapi (for >> jdk.crypto.mscapi/SunMSCAPI/sun.security.mscapi). Would it work to >> tweak the WinCAPISeedGenerator so you don't have to create a new dll >> for java.base? > > The SystemRandom JNI bindings for Windows are located in: > > java.base/windows/native/libjava/SystemRandomImpl_md.c > > ...so as I understand they are also part of libjava. No new DLL here. True. My thought should have been about having very similar code duplicated in libjava. I'm ok with this, though it's not really clear if/when MS will drop support for ADVAPI32!RtlGenRandom. This always makes me nervous because whatever is put in will likely never change until some MS change breaks it. I'm not familiar with what Alan/Mandy/company have in mind down the road, but I haven't heard of libjava splitting. >> What are the fallbacks for SystemRandomImpl if /dev/urandom or the >> rtlGenRandomFN/CryptGenRandom aren't available? Is that something >> you'll bake into TLR or will you do it here? > > I think it's better to leave it to consumers (TLR/SplittableRandom) as > they know what's good-enough for them. The API allows for arbitrary > number of bytes to be generated and I don't have an easy means of > generating more than 8 "random" bytes just from System.nanoTime() and > System.currentTimeMillis() short of using SecureRandom as a fall-back. webrev.03 only has new code, no changes yet to TLR/SplittableRandom, so I'm not yet quite following where TLR/SR will be changing. Also, what is proposed for platforms that aren't Unix/Windows? Should there be a generic fallback mechanism like ThreadedSeedGenerator? (Note, I'm not suggesting using it, it's rather...SSLLLOOOWWWW...) > The problem is also how to make access to this functionality for > different consumers that are located in different packages (java.util, > java.util.concurrent) and make it somehow usable also for external > access. There is a desire to use this also from stand-alone builds of > java.util.concurrent utilities. That's why my initial approach for > SystemRandom used a public API in java.util. > > The approach used with sun.misc.Unsafe is probably not going to work for > user code in JDK9 with modules, as sun.misc will not be globally > exported. Are any non J2SE packages going to be globally exported? I see > jdk and jdk.net are already mentioned as such globally exported packages > in modules.xml... That's a good question for Alan/Mandy/company. >> Martin wrote: >> >>> https://bugs.openjdk.java.net/browse/JDK-8047769 >> >> If you've been following this bug, I've figured why the NativePRNG$* >> classes are initing and thus opening the /dev/random,urandom. This >> definitely needs some adjustment. > > Something like the following could be used in NativePRNG and > URLSeedGenerator: > > http://cr.openjdk.java.net/~plevart/misc/FileInputStreamPool/FileInputStreamPool.java (See the other active thread in core-libs-dev.) Brad From peter.firmstone at zeus.net.au Thu Jan 1 02:26:17 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Thu, 01 Jan 2015 12:26:17 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54A1A6BC.6070109@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> Message-ID: <1420079177.1431.5.camel@Nokia-N900> Not quite, the constructor signature is the same for super and child classes. ReadSerial is a container for each serialized Object in an ObjectInputStream that provides and controlls access to serial fields, identified by name, type and calling class, such that each class has it's own parameters contained within ReadSerial. Serial parameters, might be better terminology than serial fields. Serial parameters, retrieved from ReadSerial are compatible with serial fields in the serialization specification, providing a compatible upgrade path, and enabling all classes belonging to an Object to obtain parameters during construction, whilst allowing implementations to be independant of an object's serial form. When ReadSerial doesn't contain a parameter, it will return null. // pseudocode public abstract class ReadSerial { protected ReadSerial(){ // check permission } // perform caller class dependant permission and type check public T getParameter( Class type, String name ); } Peter. Sent from my Nokia N900. ----- Original message ----- > So, if I understand this correctly, the way this would get used is: > > class BaseFoo implements Serializable { >? ? ? ? ? private final int x; > >? ? ? ? ? public BaseFoo(ReadSerial rs) { >? ? ? ? ? ? ? ? ? this(rs.getInt("x")); >? ? ? ? ? } > >? ? ? ? ? public BaseFoo(int x) { >? ? ? ? ? ? ? ? ? this.x = x; >? ? ? ? ? } > } > > Right? > > What happens with subclasses?? I think then I need an extra RS arg in my > constructors: > > class SubFoo extends BaseFoo { >? ? ? ? ? private final int y; > >? ? ? ? ? public SubFoo(ReadSerial rs) { >? ? ? ? ? ? ? ? ? this(rs.getInt("y")); >? ? ? ? ? } > >? ? ? ? ? public BaseFoo(ReadSerial rs, int y) { >? ? ? ? ? ? ? ? ? super(rs); >? ? ? ? ? ? ? ? ? this.y = y; >? ? ? ? ? } > } > > Is this what you envision? > > > > > > On 12/27/2014 8:03 PM, Peter Firmstone wrote: > > Is there any interest in developing an explicit API for Serialization?: > > > > 1. Use a public constructor signature with a single argument, > > ReadSerialParameters (read only, writable only by the > > serialization framework) to recreate objects, subclasses (when > > permitted) call this first from their own constructor, they have > > an identical constructor signature.? ReadSerialParameters that are > > null may contain a circular reference and will be available after > > construction, see #3 below. > > 2. Use a factory method (defined by an interface) with one parameter, > > WriteSerialParameters (write only, readable only by the > > serialization framework), this method can be overridden by > > subclasses (when permitted) > > 3. For circular links, a public method (defined by an interface) that > > accepts one argument, ReadSerialParameters, this method is called > > after the constructor completes, subclasses overriding this should > > call the superclass method.? If this method is not called, an > > implementation, if known to possibly contain circular links, > > should check it has been fully initialized in each object method > > called. > > 4. Retains compatibility with current serialization stream format. > > 5. Each serial field has a name, calling class and object reference, > > similar to explicitly declaring "private static final > > ObjectStreamField[] serialPersistentFields ". > > > > Benefits: > > > > 1. An object's internal form is not publicised. > > 2. Each class in an object's heirarchy can use a static method to > > check invarients and throw an exception, prior to > > java.lang.Object's constructor being called, preventing > > construction and avoiding finalizer attacks. > > 3. Final field friendly. > > 4. Compatible with existing serial form. > > 5. Flexible serial form evolution. > > 6. All methods are public and explicitly defined. > > 7. All class ProtectionDomain's exist in the current execution > > context, allowing an object to throw a SecurityException before > > construction. > > 8. Less susceptible to deserialization attacks. > > > > Problems: > > > > 1. Implementations cannot be package private or private.? Implicit > > serialization publicises internal form, any thoughts? > > > > Recommendations: > > > > 1. Create a security check in the serialization framework for > > implicit serialization, allowing administrators to reduce their > > deserialization attack surface. > > 2. For improved security, disallow classes implementing explicit > > serialization from having static state and static initializer > > blocks, only allow static methods, this would require complier and > > verifier changes. > > 3. Alternative to #2, allow final static fields, but don't allow > > static initializer blocks or mutable static fields, similar to > > interfaces. > > > > Penny for your thoughts? > > > > Regards, > > > > Peter Firmstone. From boehm at acm.org Thu Jan 1 07:38:50 2015 From: boehm at acm.org (Hans Boehm) Date: Wed, 31 Dec 2014 23:38:50 -0800 Subject: [concurrency-interest] RFR: 8065804: JEP 171: Clarifications/corrections for fence intrinsics In-Reply-To: References: <5481108A.3030605@oracle.com> <5488B4EF.6060103@redhat.com> <5488EB09.9000501@oracle.com> <54895EFB.3060204@redhat.com> <5490EA49.2050406@oracle.com> <54914CBE.2000201@gmail.com> Message-ID: If we look at using purely store fences and purely load fences in the "initialized flag" example as in this discussion, I think it's worth distinguishing too possible scenarios: 1) We guarantee some form of dependency-based ordering, as most real computer architectures do. This probably invalidates the example from my committee paper that's under discussion here. The problem is, as always, that we don't know how to make this precise at the programming language level. It's the compiler's job to break certain dependencies, like the dependency of the store to x on the load of y in x = 0 * y. Many people are thinking about this problem, both to deal with "out-of-thin-air" issues correctly in various memory models, and to design a version of C++'s memory_order_consume that's more usable. If we had a way to guarantee some well-defined notion of dependency-based ordering, then at least some of the examples here would need to be revisited. 2) We don't guarantee that dependencies imply any sort of ordering. Then I think the weird example under discussion here stands. There is officially nothing to prevent the load of x.a in thread 1 from being reordered with the store to x_init. But there may actually be better examples as to why the store-store ordering in the initializing thread is not always enough. Consider: Thread 1: x.a = 1; if (x.a != 1) world_is_broken = true; StoreStore fence; x_init = true; ... if (world_is_broken) die(); Thread 2: if (x_init) { full fence; x.a++; } I think there is nothing to prevent the read of x.a in Thread 1 from seeing the incremented value, at least if (1) the compiler promotes world_is_broken to a register, and (2) at the assembly level the store to x_init is not dependent on the load of x.a. (1) seems quite plausible, and (2) seems very reasonable if the architecture has a conditional move instruction or the like. (For Itanium, (2) holds even for the naive compilation.) This is not a particularly likely scenario, but I have no idea how would concoct programming rules that would guarantee to prevent this kind of weirdness. The first two statements of Thread 1 might appear inside an "initialize a" library routine that knows nothing about concurrency. Hans On Wed, Dec 17, 2014 at 10:54 AM, Martin Buchholz wrote: > On Wed, Dec 17, 2014 at 1:28 AM, Peter Levart > wrote: > > On 12/17/2014 03:28 AM, David Holmes wrote: > >> > >> On 17/12/2014 10:06 AM, Martin Buchholz wrote: > >> Hans allows for the nonsensical, in my view, possibility that the load > of > >> x.a can happen after the x_init=true store and yet somehow be subject > to the > >> ++ and the ensuing store that has to come before the x_init = true. > > > > Perhaps, he is speaking about why it is dangerous to replace BOTH release > > with just store-store AND acquire with just load-load? > > I'm pretty sure he's talking about weakening EITHER. > > """Clearly, and unsurprisingly, it is unsafe to replace the > load_acquire with a version that restricts only load ordering in this > case. That would allow the store to x in thread 2 to become visible > before the initialization of x by thread 1 is complete, possibly > losing the update, or corrupting the state of x during initialization. > > More interestingly, it is also generally unsafe to restrict the > release ordering constraint in thread 1 to only stores.""" > > (What's "clear and unsurprising" to Hans may not be to the rest of us) > From peter.firmstone at zeus.net.au Thu Jan 1 12:43:47 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Thu, 01 Jan 2015 22:43:47 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54A1A6BC.6070109@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> Message-ID: <1420116227.2489.5.camel@Nokia-N900> Subclass example: class SubFoo extends BaseFoo { public static ReadSerial check(ReadSerial rs){ if (rs.getInt("y") > 128) throw Exception("too big"); return rs; } private final int y; public SubFoo( int x , int y) { super(x); this.y = y; } public SubFoo( ReadSerial rs ){ super(BaseFoo.check(check(rs))); // SubFoo can't get at BaseFoo's rs.getInt("x"), // it's visible only to BaseFoo. Instead SubFoo would get // the default int value of 0. Just in case both classes have // private fields named "x". // ReadSerial is caller sensitive. this.y = rs.getInt("y"); } } Classes in the heirarchy can provide a static method that throws an exception to check invarients while preventing a finaliser attack. We'd want to check invarients for other constructors also, but for berevity... Eg: class BaseFoo implements Serializable{ public static ReadSerial check(ReadSerial rs) throws Exception { if (rs.getInt("x") < 1) throw IllegalArgumentException("message"); return rs; } .... Sent from my Nokia N900. ----- Original message ----- > So, if I understand this correctly, the way this would get used is: > > class BaseFoo implements Serializable { >? ? ? ? ? private final int x; > >? ? ? ? ? public BaseFoo(ReadSerial rs) { >? ? ? ? ? ? ? ? ? this(rs.getInt("x")); >? ? ? ? ? } > >? ? ? ? ? public BaseFoo(int x) { >? ? ? ? ? ? ? ? ? this.x = x; >? ? ? ? ? } > } > > Right? > > What happens with subclasses?? I think then I need an extra RS arg in my > constructors: > > class SubFoo extends BaseFoo { >? ? ? ? ? private final int y; > >? ? ? ? ? public SubFoo(ReadSerial rs) { >? ? ? ? ? ? ? ? ? this(rs.getInt("y")); >? ? ? ? ? } > >? ? ? ? ? public BaseFoo(ReadSerial rs, int y) { >? ? ? ? ? ? ? ? ? super(rs); >? ? ? ? ? ? ? ? ? this.y = y; >? ? ? ? ? } > } > > Is this what you envision? > > > > > > On 12/27/2014 8:03 PM, Peter Firmstone wrote: > > Is there any interest in developing an explicit API for Serialization?: > > > > 1. Use a public constructor signature with a single argument, > > ReadSerialParameters (read only, writable only by the > > serialization framework) to recreate objects, subclasses (when > > permitted) call this first from their own constructor, they have > > an identical constructor signature.? ReadSerialParameters that are > > null may contain a circular reference and will be available after > > construction, see #3 below. > > 2. Use a factory method (defined by an interface) with one parameter, > > WriteSerialParameters (write only, readable only by the > > serialization framework), this method can be overridden by > > subclasses (when permitted) > > 3. For circular links, a public method (defined by an interface) that > > accepts one argument, ReadSerialParameters, this method is called > > after the constructor completes, subclasses overriding this should > > call the superclass method.? If this method is not called, an > > implementation, if known to possibly contain circular links, > > should check it has been fully initialized in each object method > > called. > > 4. Retains compatibility with current serialization stream format. > > 5. Each serial field has a name, calling class and object reference, > > similar to explicitly declaring "private static final > > ObjectStreamField[] serialPersistentFields ". > > > > Benefits: > > > > 1. An object's internal form is not publicised. > > 2. Each class in an object's heirarchy can use a static method to > > check invarients and throw an exception, prior to > > java.lang.Object's constructor being called, preventing > > construction and avoiding finalizer attacks. > > 3. Final field friendly. > > 4. Compatible with existing serial form. > > 5. Flexible serial form evolution. > > 6. All methods are public and explicitly defined. > > 7. All class ProtectionDomain's exist in the current execution > > context, allowing an object to throw a SecurityException before > > construction. > > 8. Less susceptible to deserialization attacks. > > > > Problems: > > > > 1. Implementations cannot be package private or private.? Implicit > > serialization publicises internal form, any thoughts? > > > > Recommendations: > > > > 1. Create a security check in the serialization framework for > > implicit serialization, allowing administrators to reduce their > > deserialization attack surface. > > 2. For improved security, disallow classes implementing explicit > > serialization from having static state and static initializer > > blocks, only allow static methods, this would require complier and > > verifier changes. > > 3. Alternative to #2, allow final static fields, but don't allow > > static initializer blocks or mutable static fields, similar to > > interfaces. > > > > Penny for your thoughts? > > > > Regards, > > > > Peter Firmstone. From peter.levart at gmail.com Thu Jan 1 19:21:48 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 01 Jan 2015 20:21:48 +0100 Subject: RFR: JDK-8047769 SecureRandom should be more frugal with file descriptors In-Reply-To: <54A17899.4060304@oracle.com> References: <5492D557.2000807@gmail.com> <5498C5A0.50104@oracle.com> <549AA573.2060801@gmail.com> <54A119FF.4060103@oracle.com> <54A122AA.3010004@gmail.com> <54A17899.4060304@oracle.com> Message-ID: <54A59E4C.1050603@gmail.com> On 12/29/2014 04:51 PM, Alan Bateman wrote: > On 29/12/2014 09:45, Peter Levart wrote: >> >> Thanks for looking at this, Alan. >> >> You're right about File.getCanonicalFile(). It already checks read >> permission for a file. The additional explicit check is superfluous. >> I have removed it. >> >> With explicit check I wanted the API to behave uniformly regardless >> of whether the returned stream is opened by getInputStream() call or >> an already opened stream is just returned. getCannonicalFile() >> already takes care of it. Here's the updated webrev: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.03/ >> > Updated patch looks good to me. > > -Alan. Thanks, Alan. Peter From chris.hegarty at oracle.com Thu Jan 1 19:56:08 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 1 Jan 2015 19:56:08 +0000 Subject: RFR: JDK-8047769 SecureRandom should be more frugal with file descriptors In-Reply-To: <54A59E4C.1050603@gmail.com> References: <5492D557.2000807@gmail.com> <5498C5A0.50104@oracle.com> <549AA573.2060801@gmail.com> <54A119FF.4060103@oracle.com> <54A122AA.3010004@gmail.com> <54A17899.4060304@oracle.com> <54A59E4C.1050603@gmail.com> Message-ID: <83431B89-27FD-4087-B643-5FDF4052D8BF@oracle.com> This looks very nice Peter. Just a small comment on the test; it may avoid future problems if the test use deleteFileWithRetry, from the test library [1], rather than file.delete(). -Chris. [1] http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/tip/test/lib/testlibrary/jdk/testlibrary/FileUtils.java On 1 Jan 2015, at 19:21, Peter Levart wrote: > > On 12/29/2014 04:51 PM, Alan Bateman wrote: >> On 29/12/2014 09:45, Peter Levart wrote: >>> >>> Thanks for looking at this, Alan. >>> >>> You're right about File.getCanonicalFile(). It already checks read permission for a file. The additional explicit check is superfluous. I have removed it. >>> >>> With explicit check I wanted the API to behave uniformly regardless of whether the returned stream is opened by getInputStream() call or an already opened stream is just returned. getCannonicalFile() already takes care of it. Here's the updated webrev: >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.03/ >> Updated patch looks good to me. >> >> -Alan. > > Thanks, Alan. > > Peter From peter.levart at gmail.com Thu Jan 1 20:22:20 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 01 Jan 2015 21:22:20 +0100 Subject: RFR: JDK-8047769 SecureRandom should be more frugal with file descriptors In-Reply-To: <54A20475.3040206@oracle.com> References: <5492D557.2000807@gmail.com> <5498C5A0.50104@oracle.com> <549AA573.2060801@gmail.com> <54A20475.3040206@oracle.com> Message-ID: <54A5AC7C.5010806@gmail.com> Hi Brad, Here's next webrev which tries to cover all your comments: http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.04/ Answers inline... On 12/30/2014 02:48 AM, Bradford Wetmore wrote: > I'm looking at this version of the webrev. > > http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.03/ > > > I just assigned 8047769 to you. My username is wetmore, Alan is alanb. I'll note you both as reviewers in the changeset. > > On 12/24/2014 3:37 AM, Peter Levart wrote: > >>> Looks like you have a committer status, will you be pushing this? >> >> I can, yes. As soon as we clear-out the remaining questions, right? > > Yes. The comments below are minor and shouldn't need another review > cycle. I'm worried about the failure of the test you observed while running from NetBeans. Perhaps a 0.5s wait is sometimes not enough for ReferenceHandler thread to process (enqueue) a WeakReference. Since there is already a facility in place to help ReferenceHandler thread instead of wait for it, I used it in new version of the test. > I have started a JPRT job for you, I'll run it through "core" target > which will give us: > > jdk_lang, jdk_math, jdk_util, jdk_io, jdk_net, jdk_nio, jdk_security*, > jdk_rmi, jdk_text, jdk_time, jdk_other, core_tools. > > Anything else? I'm off tomorrow, I should have full results Wed. > > Here are the preliminary results for the jobs that have finished. > jdk.testlibrary.Asserts is causing compilation errors, additional > comments below: > > /opt/jprt/T/P1/003505.brwetmor/s/jdk/test/sun/security/provider/FileInputStreamPool/FileInputStreamPoolTest.java:33: > error: package jdk.testlibrary does not exist > import static jdk.testlibrary.Asserts.*; > ^ > /opt/jprt/T/P1/003505.brwetmor/s/jdk/test/sun/security/provider/FileInputStreamPool/FileInputStreamPoolTest.java:52: > error: cannot find symbol > assertEquals(bytes.length, nread, "short read"); > ^ > symbol: method assertEquals(int,int,String) > location: class FileInputStreamPoolTest > /opt/jprt/T/P1/003505.brwetmor/s/jdk/test/sun/security/provider/FileInputStreamPool/FileInputStreamPoolTest.java:53: > error: cannot find symbol > assertTrue(Arrays.equals(readBytes, bytes), > ^ > symbol: method assertTrue(boolean,String) > location: class FileInputStreamPoolTest > 3 errors > > TEST RESULT: Failed. Compilation failed: Compilation failed I changed the test to be self-contained now so one can run it without testlib in classpath. > > I'm also getting failures in the following test. I unfortunately have > to leave now, so don't have time to look at this. But it did mention > "seed" so I'm mentioning it here. > > TEST: java/lang/invoke/LFCaching/LFGarbageCollectedTest.java > > ACTION: main -- Failed. Execution failed: `main' threw exception: > java.lang.Error: 36 of 39 test cases FAILED! Rerun the test with the > same "-Dseed=" option as in the log file! > REASON: User specified action: run main/othervm LFGarbageCollectedTest > TIME: 213.406 seconds > messages: > command: main LFGarbageCollectedTest > reason: User specified action: run main/othervm LFGarbageCollectedTest > elapsed time (seconds): 213.406 > STDOUT: > -Dseed=6102311124531075592 > -DtestLimit=2000 > Number of iterations according to -DtestLimit is 153 (1989 cases) > Code cache size is 251658240 bytes > Non-profiled code cache size is 123058253 bytes > Number of iterations limited by code cache size is 84 (1092 cases) > Number of iterations limited by non-profiled code cache size is 41 > (533 cases) > Number of iterations is set to 41 (533 cases) > Not enough time to continue execution. Interrupted. > STDERR: > Iteration 0: > Tested LF caching feature with MethodHandles.foldArguments method. > java.lang.AssertionError: Error: Lambda form is not garbage collected > at LFGarbageCollectedTest.doTest(LFGarbageCollectedTest.java:81) > at > LambdaFormTestCase$TestRun.doIteration(LambdaFormTestCase.java:139) > at LambdaFormTestCase$$Lambda$2/5042013.call(Unknown Source) > at > jdk.testlibrary.TimeLimitedRunner.call(TimeLimitedRunner.java:71) > at LambdaFormTestCase.runTests(LambdaFormTestCase.java:201) > at LFGarbageCollectedTest.main(LFGarbageCollectedTest.java:105) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> In a couple places, there are a few lines > 80 chars. (It's a pet >>> peeve of mine, this makes side-by-side reviews difficult without a >>> wide monitor. I realize not everyone shares this view, but they're >>> probably not working on a laptop screen regularly.) >> >> I have wrapped the lines to contain them inside the 80 column margin. > > I and my scrubber/slider thank you. :) > > Two minor nits? SeedGenerator.java: Lines 507/518 Done that too. > >>> Do you need to close the InputStream when last holder is GC'd, or do >>> we just let the FileInputStream's finalizer take care of that? >> >> WeakReference is enqueued when it is cleared, so >> at that time we have no access to the referent (UncloseableInputStream) >> any more. We could, in addition, "strongly" reference the underlying >> FileInputStream in the WeakReference subclass itself, but that would >> just prolong the life of FileInputStream (possibly forever if no further >> calls to FileInputStreamPool are made that expunge the references from >> the queue). So yes, we rely on FileInputStream's finalizer, but I don't >> see any other way that would be better than that. > > Makes sense, thanks. > >> NativePRNG and >> URLSeedGenerator don't have a means of closing underlying resources >> either, so this is not making things any worse. > > Yes. > >>> Both of these current calls are contained within a >>> AccessContrller.doPriviledged, the checkRead() seems redundant. >> >> That's right, but from encapsulation, uniformity, security and future >> maintainability standpoint, I would keep this logic in. It doesn't hurt. >> Another possibility is to move doPriviliged call to FileInputStreamPool >> itself and declare it's API exposing security capability (of reading any >> file). > > I see this was addressed later via Alan's review. > >>> In your test case, if assertions are not enabled, the tests at >>> 46/50/51 are noops, e.g. when run by hand. Generally should directly >>> throw Exceptions. >> >> I modified the test to use jdk.testlibrary.Asserts class. Is this ok >> from "run by hand" perspective or should the test be self-contained? > > I've not used this Asserts library yet. Is this part of TestNG, or > something new in jtreg or jprt? If Core-libs is ok with this style of > doing it, I'm ok. I'm kind of old-school and tests should be mostly > self-contained and can be tested via: > > % javac Clazz.java > % java Clazz This should work now. > > without extra classpaths needed. I understand this model doesn't work > with @library and such, so I'm not strongly tied to it. I can be > taught new tricks. > > >>> Core-libs folks? >> >> The documentation doesn't mention threads anywhere in InputStream or >> FileInputStream except in this piece of javadoc for available() method: > ...snip... > > Ok. > > A few minor nits below: > > FileInputStreamPool.java > ======================== > * This method opens an underlying {@link java.io.FileInputStream} for > -> > * This method opens an underlying {@link java.io.FileInputStream} for a > > * among multiple readers of same {@code file} and ignores > -> > * among multiple readers of the same {@code file} and ignores Done. > > FileInputStreamPoolTest.java > ============================ > Generally JTREG labels are immediately following the copyright and > before the imports. > > While what you have is allowed by the JTREG syntax, @test is usually > by itself, or followed by old SCCS or filename info. > > @summary is usually the bug description. Suggest: > > @summary SecureRandom should be more frugal with file descriptors > > 48: This is still using assert. Fixed. > > Maybe issue multiple reads to exercise in1 and in2? e.g. 2 bytes of > in1, 4 bytes of in2, then 2 bytes of in1? The 1st assert makes sure in1 == in2, so there's no point in invoking the same instance via two aliases. > > IIRC, when I ran this under NetBeans last week, the second testCaching > didn't clear the WeakReference. This should not happen any more now that the test is helping to enqueue the WeakReferences instead of waiting for ReferenceHandler to enqueue them. The test can now fail only if System.gc() does not trigger WeakReference processing in the VM. Can you give it a try on your NetBeans environment? > > Thanks, > > Brad Regards, Peter From peter.levart at gmail.com Thu Jan 1 20:57:18 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 01 Jan 2015 21:57:18 +0100 Subject: RFR: JDK-8047769 SecureRandom should be more frugal with file descriptors In-Reply-To: <83431B89-27FD-4087-B643-5FDF4052D8BF@oracle.com> References: <5492D557.2000807@gmail.com> <5498C5A0.50104@oracle.com> <549AA573.2060801@gmail.com> <54A119FF.4060103@oracle.com> <54A122AA.3010004@gmail.com> <54A17899.4060304@oracle.com> <54A59E4C.1050603@gmail.com> <83431B89-27FD-4087-B643-5FDF4052D8BF@oracle.com> Message-ID: <54A5B4AE.6030807@gmail.com> On 01/01/2015 08:56 PM, Chris Hegarty wrote: > This looks very nice Peter. > > Just a small comment on the test; it may avoid future problems if the > test use deleteFileWithRetry, from the test library [1], rather than > file.delete(). > > -Chris. > > [1] > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/tip/test/lib/testlibrary/jdk/testlibrary/FileUtils.java > Hi Chris, Thanks for pointing me to the FileUtils. It got me thinking that the test might not be able to delete the file on Windows, since after return from 2nd call to testCaching(), it might still be open (and it very probably will be in the webrev.04 test). So I "fixed" this. I now treat the unsuccessful deletion as a test failure as it should only occur if the file is not closed at the end which it now should be. Here's the latest webrev: http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.05/ Regards, Peter > On 1 Jan 2015, at 19:21, Peter Levart > wrote: > >> >> On 12/29/2014 04:51 PM, Alan Bateman wrote: >>> On 29/12/2014 09:45, Peter Levart wrote: >>>> >>>> Thanks for looking at this, Alan. >>>> >>>> You're right about File.getCanonicalFile(). It already checks read >>>> permission for a file. The additional explicit check is >>>> superfluous. I have removed it. >>>> >>>> With explicit check I wanted the API to behave uniformly regardless >>>> of whether the returned stream is opened by getInputStream() call >>>> or an already opened stream is just returned. getCannonicalFile() >>>> already takes care of it. Here's the updated webrev: >>>> >>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.03/ >>>> >>> Updated patch looks good to me. >>> >>> -Alan. >> >> Thanks, Alan. >> >> Peter > From peter.levart at gmail.com Thu Jan 1 21:25:53 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 01 Jan 2015 22:25:53 +0100 Subject: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom In-Reply-To: <54A4A702.4010200@oracle.com> References: <5490DE08.708@oracle.com> <5491BF67.6070000@gmail.com> <54A4A702.4010200@oracle.com> Message-ID: <54A5BB61.2010605@gmail.com> Hi Brad, On 01/01/2015 02:46 AM, Bradford Wetmore wrote: > >>> To the actual proposal: >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.03/ >>> >>> Overall, I'm ok with what's proposed. This is more straightforward to >>> parse/understand than trying to adjust NativeSeedGenerator to >>> create/call directly (e.g. UNIX: new >>> NativeSeedGenerator("/dev/urandom") or Windows: new >>> NativeSeedGenerator()). But I'd still like to understand why you >>> moved away from this. >>> >>> One concern is that you're duplicating native libraries in java.base, >>> and it would be the third JDK library overall with this type of call. >>> There's one in libjava (for java.base/WinCAPISeedGenerator for >>> sun.security.provider.NativeSeedGenerator) and sunmscapi (for >>> jdk.crypto.mscapi/SunMSCAPI/sun.security.mscapi). Would it work to >>> tweak the WinCAPISeedGenerator so you don't have to create a new dll >>> for java.base? >> >> The SystemRandom JNI bindings for Windows are located in: >> >> java.base/windows/native/libjava/SystemRandomImpl_md.c >> >> ...so as I understand they are also part of libjava. No new DLL here. > > True. My thought should have been about having very similar code > duplicated in libjava. > > I'm ok with this, though it's not really clear if/when MS will drop > support for ADVAPI32!RtlGenRandom. This always makes me nervous > because whatever is put in will likely never change until some MS > change breaks it. the Microsoft C Runtime Library makes use of this function in its implementation of "rand_s", so it's removal would break a lot of programs. I think this is a relative guarantee that the function is here to stay. > > >>> What are the fallbacks for SystemRandomImpl if /dev/urandom or the >>> rtlGenRandomFN/CryptGenRandom aren't available? Is that something >>> you'll bake into TLR or will you do it here? > > >> I think it's better to leave it to consumers (TLR/SplittableRandom) as >> they know what's good-enough for them. The API allows for arbitrary >> number of bytes to be generated and I don't have an easy means of >> generating more than 8 "random" bytes just from System.nanoTime() and >> System.currentTimeMillis() short of using SecureRandom as a fall-back. > > webrev.03 only has new code, no changes yet to TLR/SplittableRandom, > so I'm not yet quite following where TLR/SR will be changing. Also, > what is proposed for platforms that aren't Unix/Windows? Should there > be a generic fallback mechanism like ThreadedSeedGenerator? (Note, > I'm not suggesting using it, it's rather...SSLLLOOOWWWW...) Are there other non-Unix and non-Windows platforms? I think the planned fall-back for TLR/SplittableRandom is to just use System.currentTimeMillis() & System.nanoTime() - these are the defaults now unless SecureRandom is requested. Regards, Peter From Alan.Bateman at oracle.com Fri Jan 2 09:57:33 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 02 Jan 2015 09:57:33 +0000 Subject: JDK 9 RFR of 4026465: Provide more byte array constructors for BigInteger In-Reply-To: <54A35C56.3040908@oracle.com> References: <48844110-12EF-48E2-B536-FF690C319CDE@oracle.com> <851F1C62-D645-4ACA-833B-D890E26FBEA0@oracle.com> <54A35C56.3040908@oracle.com> Message-ID: <54A66B8D.8070708@oracle.com> On 31/12/2014 02:15, joe darcy wrote: > Hi Brian, > > The new changes generally look good. A few comments, for the new code > like > > 291 } else if ((off < 0) || (off > val.length) || (len < 0) || > 292 ((off + len) > val.length) || ((off + len) < > 0)) { > 293 throw new IndexOutOfBoundsException(); > > it is not immediately clear that the arithmetic on line 292 won't have > some inappropriate overflow behavior. A comment stating why "off + > len" will behave appropriately (assuming it does behave appropriately > ;-) would help. (By line 292, both off and len are non-negative so > that should limit the case analysis.) > I assume this can be reduced down to: if (off < 0 || len < 0 || (off > val.length - len)) { ... } -Alan From claes.redestad at oracle.com Fri Jan 2 15:38:44 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Fri, 02 Jan 2015 16:38:44 +0100 Subject: pre-RFR (s): 8049847: Enhance PrintWriter line.separator handling Message-ID: <54A6BB84.6060508@oracle.com> Hi, this is a proposal to resolve concerns that PrintWriter/BufferedWriter behave inconsistently with j.u.Formatter when setting the line.separator property to override system-appropriate line breaks. Instead of providing a set of new constructors, I propose to simply add a new default method j.l.Appendable#lineSeparator, which by default delegates to System.lineSeparator(). This allows classes such as PrintWriter/BufferedWriter to provide overrides which communicate to j.u.Formatter their intended behavior. This indirectly provides a way to control the lineSeparator by allowing users to override PrintWriter or similar in custom classes which override lineSeparator, e.g.: PrintWriter unixPrintWriter = new PrintWriter(out) { @Override public String lineSeparator() { return "\n"; } }; PrintWriter windowsPrintWriter = new PrintWriter(out) { @Override public String lineSeparator() { return "\r\n"; } }; Bonus: This approach can be used to optimize j.l.Throwable to get rid of all static inner classes to wrap PrintWriter/PrintStream and instead use j.l.Appendable, which more than enough will mitigate the startup hit adding a method to Appendable will incur. bug: https://bugs.openjdk.java.net/browse/JDK-8049847 webrev: http://cr.openjdk.java.net/~redestad/8049847/webrev.00/ I've not filed requests to change the public API just yet, rather wanted to throw this out for preview to see if there's some concern and feedback first. /Claes From rob.mckenna at oracle.com Fri Jan 2 16:39:25 2015 From: rob.mckenna at oracle.com (Rob McKenna) Date: Fri, 02 Jan 2015 16:39:25 +0000 Subject: 5049299 - (process) Use,posix_spawn, not fork, on S10 to avoid swap,exhaustion (jdk7u-dev) In-Reply-To: <54A16A8B.3090407@agnity.com> References: <54A16A8B.3090407@agnity.com> Message-ID: <54A6C9BD.4060705@oracle.com> Hi Amit, As per https://bugs.openjdk.java.net/browse/JDK-5049299, this is fixed in 7u55+ (including 8 fcs). There are some peculiarities however: On JDK8 posix_spawn is the default on Solaris & Mac. vfork is the default on Linux. On JDK7u55+ posix_spawn is the default on Mac only. fork is the default on Solaris and vfork is the default on Linux. On 7u55+ you can change the default launch mechanism using the jdk.lang.Process.launchMechanism startup flag. E.g.: java -Djdk.lang.Process.launchMechanism=posix_spawn ... -Rob On 29/12/14 14:51, Amit Baxi wrote: > Hi Folks, > > We are facing issue with process builder on JRockit JVM with solaris > 5.10 machine. That is realted to following bug: > > http://bugs.java.com/view_bug.do?bug_id=5049299 > > > > Currently we are using JRockit JDK 1.6.0_37-R28.2.5-4.1.0 version. > Please let me know following : > > * Which version of JRockit this bug is fixed ? (as it is fixed in JAVA > 8u 15) > * If yes then is it also fixed for solaris 5.10 or In solaris process > builder still uses fork() call ? > > > As per this page link below Mr. Rob wrote that for solaris it still > using fork which is confusing weather fixed for solaris or not: > > http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-November/022909.html > > > Please confirm for this fix and thanks in advance. > > Waiting for your response, > > Thanks! > *Amit Baxi > > > > * > > > > > From volker.simonis at gmail.com Fri Jan 2 16:54:43 2015 From: volker.simonis at gmail.com (Volker Simonis) Date: Fri, 2 Jan 2015 17:54:43 +0100 Subject: RFR: 5049299 - (process) Use,posix_spawn, not fork, on S10 to avoid swap,exhaustion (jdk7u-dev) In-Reply-To: <54A16A8B.3090407@agnity.com> References: <54A16A8B.3090407@agnity.com> Message-ID: Hi Amit, I think this is the wrong place to ask. This list is about OpenJDK/OracleJDK. As you can read from the bug report (better look at: https://bugs.openjdk.java.net/browse/JDK-5049299) there's a list in which OpenJDK/OracleJDK versions this bug has been fixed. It describes that this bug hasn't been fixed in JDK 6 (only 7 and 8). No idea where you can get this information for JRockit - you'd probably have to ask the one from which you bought it. By the way, in Oracle/OpenJDK 7 and 8 you can control the launch behaviour with: -Djdk.lang.Process.launchMechanism={fork, vfork, posix_spawn} Notice that not every platform supports every setting and different platforms have different default launching mechanisms. You'll have to look at the sources to see what's the default (i.e. the files jdk/src/solaris/classes/java/lang/UNIXProcess.java.{bsd, linux, solaris}'). Regards, Volker On Mon, Dec 29, 2014 at 3:51 PM, Amit Baxi wrote: > Hi Folks, > > We are facing issue with process builder on JRockit JVM with solaris 5.10 > machine. That is realted to following bug: > > http://bugs.java.com/view_bug.do?bug_id=5049299 > > > > Currently we are using JRockit JDK 1.6.0_37-R28.2.5-4.1.0 version. Please > let me know following : > > * Which version of JRockit this bug is fixed ? (as it is fixed in JAVA > 8u 15) > * If yes then is it also fixed for solaris 5.10 or In solaris process > builder still uses fork() call ? > > > As per this page link below Mr. Rob wrote that for solaris it still using > fork which is confusing weather fixed for solaris or not: > > http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-November/022909.html > > Please confirm for this fix and thanks in advance. > > Waiting for your response, > > Thanks! > *Amit Baxi > > > > * > > > > > From huizhe.wang at oracle.com Fri Jan 2 18:39:47 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Fri, 02 Jan 2015 10:39:47 -0800 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> Message-ID: <54A6E5F3.7070900@oracle.com> Lance, Tristan is looking into adding an extension base class for about 60 tests that require file permission, then the current base class would indeed set "minimal" permission. So please wait for his update :-) Best, Joe On 12/30/2014 3:07 PM, Lance @ Oracle wrote: > Hi Tristan, > > I will look at this but doubt I will get to this tomorrow > > > Best, > Lance > > > Lance > Andersen| Principal Member of Technical Staff | +1.781.442.2037 > > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > Sent from my iPad > > On Dec 30, 2014, at 5:28 PM, Tristan Yan > wrote: > >> Hi All >> >> Can I get your review on this change. >> >> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >> >> > > >> >> These fixes originally come from bug >> https://bugs.openjdk.java.net/browse/JDK-8051563 >> as part of our XML >> test colocation work. ThIS change-set mainly covers tests for package >> org.apache.qetest.dtm and org.apache.qetest.trax. >> >> In the meantime I took steps at fixing some of our existed test code >> on below issues: >> >> 1. Add a base test class for all functional tests that enable >> security manager running. A limited minimal permissions set have been >> set for every test. >> 2. Remove all unnecessary exception capture for functional tests that >> we?re using testng to handle all the exceptions. >> 3. Use try-resource block to solve all possible resource leaks >> (including InputStream, OutputStream, Writer, Reader). >> >> Thanks a lot. >> Tristan From tristan.yan at oracle.com Fri Jan 2 18:49:40 2015 From: tristan.yan at oracle.com (Tristan Yan) Date: Fri, 2 Jan 2015 10:49:40 -0800 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: <54A6E5F3.7070900@oracle.com> References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> Message-ID: <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> Hi Joe and Lance Sorry for my late reply. I just uploaded the new webrev which is trying to limit minimal permissions for most of tests. The new changeset has two base classes; class JAXPBaseTest has only minimal set permissions; class JAXPFileBaseTest adds two permissions that need access local files in the directory directory test.src and test.classes. Most of tests only inherit JAXPBaseTest that provides only minimal set permissions. Some tests will inherit JAXPFileBaseTest because tests need access local files. Please help to review the code. http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ Thank you Tristan > On Jan 2, 2015, at 10:39 AM, huizhe wang wrote: > > Lance, > > Tristan is looking into adding an extension base class for about 60 tests that require file permission, then the current base class would indeed set "minimal" permission. So please wait for his update :-) > > Best, > Joe > > On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >> Hi Tristan, >> >> I will look at this but doubt I will get to this tomorrow >> >> >> Best, >> Lance >> >> >> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> Sent from my iPad >> >> On Dec 30, 2014, at 5:28 PM, Tristan Yan > wrote: >> >>> Hi All >>> >>> Can I get your review on this change. >>> >>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ > >>> >>> These fixes originally come from bug https://bugs.openjdk.java.net/browse/JDK-8051563 > as part of our XML test colocation work. ThIS change-set mainly covers tests for package org.apache.qetest.dtm and org.apache.qetest.trax. >>> >>> In the meantime I took steps at fixing some of our existed test code on below issues: >>> >>> 1. Add a base test class for all functional tests that enable security manager running. A limited minimal permissions set have been set for every test. >>> 2. Remove all unnecessary exception capture for functional tests that we?re using testng to handle all the exceptions. >>> 3. Use try-resource block to solve all possible resource leaks (including InputStream, OutputStream, Writer, Reader). >>> >>> Thanks a lot. >>> Tristan > From bradford.wetmore at oracle.com Fri Jan 2 21:29:49 2015 From: bradford.wetmore at oracle.com (Bradford Wetmore) Date: Fri, 02 Jan 2015 13:29:49 -0800 Subject: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom In-Reply-To: <54A5BB61.2010605@gmail.com> References: <5490DE08.708@oracle.com> <5491BF67.6070000@gmail.com> <54A4A702.4010200@oracle.com> <54A5BB61.2010605@gmail.com> Message-ID: <54A70DCD.8080005@oracle.com> > the Microsoft C Runtime Library makes use of this function in its > implementation of "rand_s", so it's removal would break a lot of > programs. I think this is a relative guarantee that the function is here > to stay. Ok. >>>> What are the fallbacks for SystemRandomImpl if /dev/urandom or the >>>> rtlGenRandomFN/CryptGenRandom aren't available? Is that something >>>> you'll bake into TLR or will you do it here? >> > >>> I think it's better to leave it to consumers (TLR/SplittableRandom) as >>> they know what's good-enough for them. The API allows for arbitrary >>> number of bytes to be generated and I don't have an easy means of >>> generating more than 8 "random" bytes just from System.nanoTime() and >>> System.currentTimeMillis() short of using SecureRandom as a fall-back. >> >> webrev.03 only has new code, no changes yet to TLR/SplittableRandom, >> so I'm not yet quite following where TLR/SR will be changing. Also, >> what is proposed for platforms that aren't Unix/Windows? Should there >> be a generic fallback mechanism like ThreadedSeedGenerator? (Note, >> I'm not suggesting using it, it's rather...SSLLLOOOWWWW...) > > Are there other non-Unix and non-Windows platforms? I'm not an expert in non-Oracle Java offerings, but what about the various Java ME devices? Some of the embedded devices are Linux/Windows, but are there others? JavaCard? I know IBM has a large number of OS's they support, for example IBM i. http://en.wikipedia.org/wiki/IBM_i https://www.ibm.com/developerworks/community/wikis/home?lang=en#/wiki/IBM%20i%20Technology%20Updates/page/Java%20on%20IBM%20i I know IBM supports Java on z/OS, but apparently that uses a Linux-style filesystem via UNIX_System_Services. http://en.wikipedia.org/?title=Z/OS Not sure about others. I saw a couple of other non-unix/windows OSs in a wikipedia comparison of JVM's, but most of them seem to be discontinued. > I think the planned > fall-back for TLR/SplittableRandom is to just use > System.currentTimeMillis() & System.nanoTime() - these are the defaults > now unless SecureRandom is requested. Ok. Since this isn't security-critical, IMHO there just needs to be a "reasonable" fall-back option in case /dev/urandom or rtlGenRandomFN can't be used. I don't always closely monitor core-libs-dev, so please cc me if further discussion occurs in a different thread. Cheers, and HNY to you and everyone else here, Brad From brian.goetz at oracle.com Fri Jan 2 22:53:01 2015 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 02 Jan 2015 17:53:01 -0500 Subject: Explicit Serialization API and Security In-Reply-To: <1420116227.2489.5.camel@Nokia-N900> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> Message-ID: <54A7214D.80608@oracle.com> Overall the direction seems promising. Poking at it a bit... - ReadSerial methods are caller-sensitive and only show a class a view of its own fields. - Invariant checking is separate from deserialization, and does not seem entirely built-in -- subclass constructors seem responsible for asking parents to do validity-checking? - I don't see how this invariant-checking mechanism can enforce invariants between superclass fields and subclass fields. For example: class A { int lower, upper; // invariant: lower <= upper } class B extends A { int cur; // invariant: lower <= cur <= upper } To check such an invariant, the serialization library would have to construct the object (in a potentially bad state), invoke the checker at each layer, and then fail deserialization if any checker said no. But, an evil checker could still squirrel away a reference under the carpet. Another challenge in invariant checking is circular data structures. If you have two objects: class Brother { final Brother brother; } that refer to each other, an invariant you might want to check after deserialization is that this.brother.brother == this Obviously you have to patch one or the other instance after construction to retain the circular references; at what point do you do invariant checking? On 1/1/2015 7:43 AM, Peter Firmstone wrote: > Subclass example: > > class SubFoo extends BaseFoo { > > public static ReadSerial check(ReadSerial rs){ > if (rs.getInt("y") > 128) throw Exception("too big"); > return rs; > } > > private final int y; > > public SubFoo( int x , int y) { > super(x); > this.y = y; > } > > public SubFoo( ReadSerial rs ){ > super(BaseFoo.check(check(rs))); > // SubFoo can't get at BaseFoo's rs.getInt("x"), > // it's visible only to BaseFoo. Instead SubFoo would get > // the default int value of 0. Just in case both classes have > // private fields named "x". > // ReadSerial is caller sensitive. > this.y = rs.getInt("y"); > } > } > > Classes in the heirarchy can provide a static method that throws an > exception to check invarients while preventing a finaliser attack. We'd > want to check invarients for other constructors also, but for berevity... > > Eg: > > class BaseFoo implements Serializable{ > > public static ReadSerial check(ReadSerial rs) throws Exception > { > if (rs.getInt("x") < 1) > throw IllegalArgumentException("message"); > return rs; > } > .... > > > Sent from my Nokia N900. > > ----- Original message ----- > > So, if I understand this correctly, the way this would get used is: > > > > class BaseFoo implements Serializable { > > private final int x; > > > > public BaseFoo(ReadSerial rs) { > > this(rs.getInt("x")); > > } > > > > public BaseFoo(int x) { > > this.x = x; > > } > > } > > > > Right? > > > > What happens with subclasses? I think then I need an extra RS arg in my > > constructors: > > > > class SubFoo extends BaseFoo { > > private final int y; > > > > public SubFoo(ReadSerial rs) { > > this(rs.getInt("y")); > > } > > > > public BaseFoo(ReadSerial rs, int y) { > > super(rs); > > this.y = y; > > } > > } > > > > Is this what you envision? > > > > > > > > > > > > On 12/27/2014 8:03 PM, Peter Firmstone wrote: > > > Is there any interest in developing an explicit API for > Serialization?: > > > > > > 1. Use a public constructor signature with a single argument, > > > ReadSerialParameters (read only, writable only by the > > > serialization framework) to recreate objects, subclasses (when > > > permitted) call this first from their own constructor, they have > > > an identical constructor signature. ReadSerialParameters that are > > > null may contain a circular reference and will be available after > > > construction, see #3 below. > > > 2. Use a factory method (defined by an interface) with one parameter, > > > WriteSerialParameters (write only, readable only by the > > > serialization framework), this method can be overridden by > > > subclasses (when permitted) > > > 3. For circular links, a public method (defined by an interface) that > > > accepts one argument, ReadSerialParameters, this method is called > > > after the constructor completes, subclasses overriding this should > > > call the superclass method. If this method is not called, an > > > implementation, if known to possibly contain circular links, > > > should check it has been fully initialized in each object method > > > called. > > > 4. Retains compatibility with current serialization stream format. > > > 5. Each serial field has a name, calling class and object reference, > > > similar to explicitly declaring "private static final > > > ObjectStreamField[] serialPersistentFields ". > > > > > > Benefits: > > > > > > 1. An object's internal form is not publicised. > > > 2. Each class in an object's heirarchy can use a static method to > > > check invarients and throw an exception, prior to > > > java.lang.Object's constructor being called, preventing > > > construction and avoiding finalizer attacks. > > > 3. Final field friendly. > > > 4. Compatible with existing serial form. > > > 5. Flexible serial form evolution. > > > 6. All methods are public and explicitly defined. > > > 7. All class ProtectionDomain's exist in the current execution > > > context, allowing an object to throw a SecurityException before > > > construction. > > > 8. Less susceptible to deserialization attacks. > > > > > > Problems: > > > > > > 1. Implementations cannot be package private or private. Implicit > > > serialization publicises internal form, any thoughts? > > > > > > Recommendations: > > > > > > 1. Create a security check in the serialization framework for > > > implicit serialization, allowing administrators to reduce their > > > deserialization attack surface. > > > 2. For improved security, disallow classes implementing explicit > > > serialization from having static state and static initializer > > > blocks, only allow static methods, this would require complier and > > > verifier changes. > > > 3. Alternative to #2, allow final static fields, but don't allow > > > static initializer blocks or mutable static fields, similar to > > > interfaces. > > > > > > Penny for your thoughts? > > > > > > Regards, > > > > > > Peter Firmstone. > From bradford.wetmore at oracle.com Fri Jan 2 22:58:21 2015 From: bradford.wetmore at oracle.com (Bradford Wetmore) Date: Fri, 02 Jan 2015 14:58:21 -0800 Subject: RFR: JDK-8047769 SecureRandom should be more frugal with file descriptors In-Reply-To: <54A5AC7C.5010806@gmail.com> References: <5492D557.2000807@gmail.com> <5498C5A0.50104@oracle.com> <549AA573.2060801@gmail.com> <54A20475.3040206@oracle.com> <54A5AC7C.5010806@gmail.com> Message-ID: <54A7228D.4040801@oracle.com> On 1/1/2015 12:22 PM, Peter Levart wrote: > Hi Brad, > > Here's next webrev which tries to cover all your comments: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.04/ I downloaded the webrev.05 (Chris' later followup email) and ran it through JPRT. The only error was the previously seen -Dseed which is not your problem. Again, I only ran: jdk_lang, jdk_math, jdk_util, jdk_io, jdk_net, jdk_nio, jdk_security*, jdk_rmi, jdk_text, jdk_time, jdk_other, core_tools. If you need anything else run, let me know. >>>> Looks like you have a committer status, will you be pushing this? >>> >>> I can, yes. As soon as we clear-out the remaining questions, right? >> >> Yes. The comments below are minor and shouldn't need another review >> cycle. > > I'm worried about the failure of the test you observed while running > from NetBeans. Perhaps a 0.5s wait is sometimes not enough for > ReferenceHandler thread to process (enqueue) a WeakReference. Since > there is already a facility in place to help ReferenceHandler thread > instead of wait for it, I used it in new version of the test. BTW, there's now an unnecessary import from java.lang.AssertionError added in webrev.04. >> TEST RESULT: Failed. Compilation failed: Compilation failed > > I changed the test to be self-contained now so one can run it without > testlib in classpath. Thanks. It's compiling fine now. >> Two minor nits? SeedGenerator.java: Lines 507/518 > > Done that too. Thanks. >> Maybe issue multiple reads to exercise in1 and in2? e.g. 2 bytes of >> in1, 4 bytes of in2, then 2 bytes of in1? > > The 1st assert makes sure in1 == in2, so there's no point in invoking > the same instance via two aliases. True. >> IIRC, when I ran this under NetBeans last week, the second testCaching >> didn't clear the WeakReference. > > This should not happen any more now that the test is helping to enqueue > the WeakReferences instead of waiting for ReferenceHandler to enqueue > them. Yes, that hit the refQueue.poll(). It's always interesting to work with core-libs folks, learn something new everyday. Mixing Lambdas/try-with. I need a time-machine for your CFV/jdk8 Committer: http://mail.openjdk.java.net/pipermail/jdk8-dev/2013-August/002896.html I vote yes. > The test can now fail only if System.gc() does not trigger > WeakReference processing in the VM. Can you give it a try on your > NetBeans environment? One last comment. It's now 2015. ;) Brad From brian.burkhalter at oracle.com Sat Jan 3 00:09:39 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 2 Jan 2015 16:09:39 -0800 Subject: JDK 9 RFR of 4026465: Provide more byte array constructors for BigInteger In-Reply-To: <54A66B8D.8070708@oracle.com> References: <48844110-12EF-48E2-B536-FF690C319CDE@oracle.com> <851F1C62-D645-4ACA-833B-D890E26FBEA0@oracle.com> <54A35C56.3040908@oracle.com> <54A66B8D.8070708@oracle.com> Message-ID: <021DF7FC-C49F-4A66-B534-6F38499E64C1@oracle.com> Hello all, Thanks for the comments. A new patch is here: http://cr.openjdk.java.net/~bpb/4026465/webrev.02/ On Dec 30, 2014, at 6:15 PM, joe darcy wrote: > The new changes generally look good. A few comments, for the new code like > > 291 } else if ((off < 0) || (off > val.length) || (len < 0) || > 292 ((off + len) > val.length) || ((off + len) < 0)) { > 293 throw new IndexOutOfBoundsException(); > > it is not immediately clear that the arithmetic on line 292 won't have some inappropriate overflow behavior. A comment stating why "off + len" will behave appropriately (assuming it does behave appropriately ;-) would help. (By line 292, both off and len are non-negative so that should limit the case analysis.) Logic updated. > It might be worthwhile for all the BigInteger constructors which take array arguments to state something about the thread-safely behavior ("arrays are assumed to be unchanged for the duration of the constructor call...?). Verbiage added. > Do have have any code coverage information for the new code by the regression test? It would be good to know whether or not all the guard conditions are properly being executed. No coverage information, but I added some tests for the guard conditions and slightly changed the correct-value part of the test. On Dec 30, 2014, at 6:42 PM, Paul Benedict wrote: > Please add @since 1.9 to the new constructors. Done. On Jan 2, 2015, at 1:57 AM, Alan Bateman wrote: > I assume this can be reduced down to: > if (off < 0 || len < 0 || (off > val.length - len)) { ... } But then if len > val.length, the third inequality tests ?off? for being greater than a negative value. Please see the updated patch. Thanks, Brian From Alan.Bateman at oracle.com Sat Jan 3 08:50:08 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 03 Jan 2015 08:50:08 +0000 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <549D3245.8070508@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> Message-ID: <54A7AD40.7080108@oracle.com> On 26/12/2014 10:02, Ivan Gerasimov wrote: > Hi! > > I changed my mind about handling only entirely quoted entries in PATH. > The webrev was updated. > > Now, on Windows paths with quoted parts are supported. > If the quoted fragment of the path contains the delimiter -- ';', it > will be preserved as a part of the path. > > This is consistent with how Windows command prompt resolves PATH. > > The regression test was extended to cover new functionality. > > The webrev was updated in place: > http://cr.openjdk.java.net/~igerasim/8067951/2/webrev/ > I assume it can't be very common to quote individual elements in this manner, otherwise we would have had bug reports on this many years ago. I agree we should probably support this and the approach in your updated webrev looks reasonable to me. One thing to consider is refactoring it so that the Windows specific handling is the Windows implementation of ClassLoaderHelper rather than in initializePaths. -Alan. From peter.levart at gmail.com Sat Jan 3 10:00:46 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sat, 03 Jan 2015 11:00:46 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <54A7214D.80608@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> Message-ID: <54A7BDCE.7020308@gmail.com> On 01/02/2015 11:53 PM, Brian Goetz wrote: > Overall the direction seems promising. Poking at it a bit... > > - ReadSerial methods are caller-sensitive and only show a class a > view of its own fields. > - Invariant checking is separate from deserialization, and does not > seem entirely built-in -- subclass constructors seem responsible for > asking parents to do validity-checking? > - I don't see how this invariant-checking mechanism can enforce > invariants between superclass fields and subclass fields. For example: > > class A { > int lower, upper; // invariant: lower <= upper > } > > class B extends A { > int cur; // invariant: lower <= cur <= upper > } > > To check such an invariant, the serialization library would have to > construct the object (in a potentially bad state), invoke the checker > at each layer, and then fail deserialization if any checker said no. > But, an evil checker could still squirrel away a reference under the > carpet. Hi, I would like to know what are the potential issues with simple constructor chaining where each constructor checks the invariant of its class state (with the already initialized state of superclass(es)). For example, the above would be written as: class A { final int lower, upper; A(ReadSerial rs) { int l = rs.getInt("lower"); int u = rs.getInt("upper"); if (l > u) throw new IllegalArgumentException(); lower = l; upper = u; } } class B extends A { final int cur; B(ReadSerial rs) { super(rs); int c = rs.getInt("cur"); if (c < lower || c > upper) throw new IllegalArgumentException(); cur = c; } } > > Another challenge in invariant checking is circular data structures. > If you have two objects: > > class Brother { > final Brother brother; > } > > that refer to each other, an invariant you might want to check after > deserialization is that > this.brother.brother == this This is tricky, yes. In general, the graph of objects has to be linked somehow gradually where links are established to half-initialized objects. So there would have to be a special post-deserialization call-back over the deserialized objects just for checking the invariants. But what to do with the instances that "escape" during the process? This is why explicit (de)serialization is tricky - it allows arbitrary user code to be executed during the construction of the object graph. Perhaps we just need some kind of declarative prescription of serialized state and mapping to fields of objects in the form of annotations or such (see what JPA is doing in this field). Combined with post-deserialization invariant-checking call-back method perhaps. Regards, Peter > > Obviously you have to patch one or the other instance after > construction to retain the circular references; at what point do you > do invariant checking? > > On 1/1/2015 7:43 AM, Peter Firmstone wrote: >> Subclass example: >> >> class SubFoo extends BaseFoo { >> >> public static ReadSerial check(ReadSerial rs){ >> if (rs.getInt("y") > 128) throw Exception("too big"); >> return rs; >> } >> >> private final int y; >> >> public SubFoo( int x , int y) { >> super(x); >> this.y = y; >> } >> >> public SubFoo( ReadSerial rs ){ >> super(BaseFoo.check(check(rs))); >> // SubFoo can't get at BaseFoo's rs.getInt("x"), >> // it's visible only to BaseFoo. Instead SubFoo would get >> // the default int value of 0. Just in case both classes have >> // private fields named "x". >> // ReadSerial is caller sensitive. >> this.y = rs.getInt("y"); >> } >> } >> >> Classes in the heirarchy can provide a static method that throws an >> exception to check invarients while preventing a finaliser attack. We'd >> want to check invarients for other constructors also, but for >> berevity... >> >> Eg: >> >> class BaseFoo implements Serializable{ >> >> public static ReadSerial check(ReadSerial rs) throws Exception >> { >> if (rs.getInt("x") < 1) >> throw IllegalArgumentException("message"); >> return rs; >> } >> .... >> >> >> Sent from my Nokia N900. >> >> ----- Original message ----- >> > So, if I understand this correctly, the way this would get used is: >> > >> > class BaseFoo implements Serializable { >> > private final int x; >> > >> > public BaseFoo(ReadSerial rs) { >> > this(rs.getInt("x")); >> > } >> > >> > public BaseFoo(int x) { >> > this.x = x; >> > } >> > } >> > >> > Right? >> > >> > What happens with subclasses? I think then I need an extra RS arg >> in my >> > constructors: >> > >> > class SubFoo extends BaseFoo { >> > private final int y; >> > >> > public SubFoo(ReadSerial rs) { >> > this(rs.getInt("y")); >> > } >> > >> > public BaseFoo(ReadSerial rs, int y) { >> > super(rs); >> > this.y = y; >> > } >> > } >> > >> > Is this what you envision? >> > >> > >> > >> > >> > >> > On 12/27/2014 8:03 PM, Peter Firmstone wrote: >> > > Is there any interest in developing an explicit API for >> Serialization?: >> > > >> > > 1. Use a public constructor signature with a single argument, >> > > ReadSerialParameters (read only, writable only by the >> > > serialization framework) to recreate objects, subclasses (when >> > > permitted) call this first from their own constructor, they have >> > > an identical constructor signature. ReadSerialParameters that are >> > > null may contain a circular reference and will be available after >> > > construction, see #3 below. >> > > 2. Use a factory method (defined by an interface) with one >> parameter, >> > > WriteSerialParameters (write only, readable only by the >> > > serialization framework), this method can be overridden by >> > > subclasses (when permitted) >> > > 3. For circular links, a public method (defined by an interface) >> that >> > > accepts one argument, ReadSerialParameters, this method is called >> > > after the constructor completes, subclasses overriding this should >> > > call the superclass method. If this method is not called, an >> > > implementation, if known to possibly contain circular links, >> > > should check it has been fully initialized in each object method >> > > called. >> > > 4. Retains compatibility with current serialization stream format. >> > > 5. Each serial field has a name, calling class and object >> reference, >> > > similar to explicitly declaring "private static final >> > > ObjectStreamField[] serialPersistentFields ". >> > > >> > > Benefits: >> > > >> > > 1. An object's internal form is not publicised. >> > > 2. Each class in an object's heirarchy can use a static method to >> > > check invarients and throw an exception, prior to >> > > java.lang.Object's constructor being called, preventing >> > > construction and avoiding finalizer attacks. >> > > 3. Final field friendly. >> > > 4. Compatible with existing serial form. >> > > 5. Flexible serial form evolution. >> > > 6. All methods are public and explicitly defined. >> > > 7. All class ProtectionDomain's exist in the current execution >> > > context, allowing an object to throw a SecurityException before >> > > construction. >> > > 8. Less susceptible to deserialization attacks. >> > > >> > > Problems: >> > > >> > > 1. Implementations cannot be package private or private. Implicit >> > > serialization publicises internal form, any thoughts? >> > > >> > > Recommendations: >> > > >> > > 1. Create a security check in the serialization framework for >> > > implicit serialization, allowing administrators to reduce their >> > > deserialization attack surface. >> > > 2. For improved security, disallow classes implementing explicit >> > > serialization from having static state and static initializer >> > > blocks, only allow static methods, this would require complier and >> > > verifier changes. >> > > 3. Alternative to #2, allow final static fields, but don't allow >> > > static initializer blocks or mutable static fields, similar to >> > > interfaces. >> > > >> > > Penny for your thoughts? >> > > >> > > Regards, >> > > >> > > Peter Firmstone. >> From peter.firmstone at zeus.net.au Sat Jan 3 11:24:03 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Sat, 03 Jan 2015 21:24:03 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54A7214D.80608@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> Message-ID: <1420284243.3416.10.camel@Nokia-N900> Thanks Brian, Those are good questions, some thoughts and examples inline: ----- Original message ----- > Overall the direction seems promising.? Poking at it a bit... > >? ? - ReadSerial methods are caller-sensitive and only show a class a view > of its own fields. >? ? - Invariant checking is separate from deserialization, and does not > seem entirely built-in -- subclass constructors seem responsible for > asking parents to do validity-checking? My mistake, the last example wasn't quite right, BaseFoo, should call it's own static check method from within it's constructor, rather than from SubFoo. Each class in the heirarchy needs to check it's invarients using a static method, before calling a superclass constructor, this prevents object construction if an invarient isn't satisfied, because the object isn't created an attacker cannot hold a reference to it. By the time Object's default constructor is called, even though no fields have been set, invarients have already been checked for every member class in the Object. An attacker may choose to call another constructor and not pass on ReadSerial, so all constructors need to perform these checks. >? ? - I don't see how this invariant-checking mechanism can enforce > invariants between superclass fields and subclass fields.? For example: > > class A { >? ? ? ? ? int lower, upper;? // invariant: lower <= upper > } > > class B extends A { >? ? ? ? ? int cur;? // invariant: lower <= cur <= upper > } > > To check such an invariant, the serialization library would have to > construct the object (in a potentially bad state), invoke the checker at > each layer, and then fail deserialization if any checker said no.? The serialization framework should only construct an objects fields and make these available from ReadSerial, each class then calls a static method before calling a superclass constructor that's responsible for an objects creation, to prevent construction of the object, if necessary. Eg, some complexity, but bullet proof: public class A ( public final int lower, upper; private static boolean check(ReadSerial rs){ if (rs.getInt("lower") > rs.getInt("upper")) throw new IllegalArgumentException( "lower bound must be less than or equal to upper"); return true; } public A(ReadSerial rs){ this(check(rs), rs); } private A(boolean checked, ReadSerial rs){ super(); lower = rs.getInt("lower"); upper = rs.getInt("upper"); } // other constructors omitted must also check invarients } class B extends A { public final int cur; private static ReadSerial check(ReadSerial rs) { A a = new A(rs); int cur = rs.getInt("cur"); if ( a.lower > cur || cur > a.upper ) throw new IllegalArgumentException( "cur outside lower and upper bounds"); return rs; } public B(ReadSerial rs) { super(check(rs)); cur = rs.getInt("cur"); } } But, > an evil checker could still squirrel away a reference under the carpet. Not with the above example, if an invarient isn't satisfied, an object instance of B cannot be created. But circular links are still a challenge... > > Another challenge in invariant checking is circular data structures.? If > you have two objects: > > class Brother { >? ? ? ? ? final Brother brother; > } > > that refer to each other, an invariant you might want to check after > deserialization is that >? ? ? this.brother.brother == this > > Obviously you have to patch one or the other instance after construction > to retain the circular references; at what point do you do invariant > checking? There are two cases here I think: One where there is a trust relationship between two classes and the circular relationship is known at compilation time. At least one will provide a mutable setter method, or a Constructor that accepts an argument with a matching argument type, and the other invokes it during construction, I say trusted because 'this' is allowed to escape. In Brother's, because the brother field is final, this was constructed with a trust relationship, because 'this' escapes from at least one Brother during construction. I would argue that the object that allows this to escape should also be responsible for rebuilding the circular relation during deserialisation. However the Brother that didn't let this escape could be serialised first, in any case both are instances of Brother, so either instance could be a subclass, each can correctly create the other. public class Brother { private static Constructor getBroConstructor(ReadSerial rs){ // we can check our subclass has permission here if we like, // even though an object hasn't been created, its class is // in the current execution context. Class type = rs.getType("brother"); if ( !Brother.class.isAssignableFrom(type)) throw new Exception("not my bro"); if ( !rs.samePackageAndClassLoader(Brother.class, type){ Permission toBeMyBro = new BroPermission(); // gets type's ProtectionDomain and checks it has // permission, throws SecurityException rs.checkPermission(type, toBeMyBro); } Class [] paramTypes = {Brother.class}; // Check if constructor is accessible. // and throw exception here if not. return rs.getConstructor("brother", paramTypes); } final Brother brother; public Brother(ReadSerial rs){ this(getBroConstructor(rs)); } private Brother(Constructor c){ Object [] params = {this}; brother = c.newInstance(params); } } Another option might be to provide something like this rs.circularFieldNewInstance("brother", this); But that encourages bad practise; 'this' to escape, although it would allow 'this' to be passed as one of the filelds in another ReadSerial instance, but probably not a great idea. In other cases, there is a circular relationship between untrusted classes and the framework must provide a thread of execution to classes to wire up circular references, after construction. Circular field references that can only be written after construction must be mutable, or if final, their referent must be mutable, depending on the depth of the circular relationship. To enable wiring up the circular relationship, a class needs to implement a method for the framework to call after construction, a suggestion: public interface Circular { void setMutableFields(ReadSerial rs); } Circular fields missing from ReadSerial during construction are populated after. An attacking subclass can override setMutableFields, so the implementing class should make it final, provide and call a protected method for subclasses to implement, should they too contain circular links. The alternative is to use a private method, but it's a lot more convenient to use an interface. Unfortunately the only remaining option to protect invarients is to make all methods throw an exception if invarients haven't been satisfied after construction, even if this method throws an exception and halts deserialisation, an attacker can hold a reference to it. Cheers, Peter. > > On 1/1/2015 7:43 AM, Peter Firmstone wrote: > > Subclass example: > > > > class SubFoo extends BaseFoo { > > > > public static ReadSerial check(ReadSerial rs){ > > if (rs.getInt("y") > 128) throw Exception("too big"); > > return rs; > > } > > > > private final int y; > > > > public SubFoo( int x , int y) { > > super(x); > > this.y = y; > > } > > > > public SubFoo( ReadSerial rs ){ > > super(BaseFoo.check(check(rs))); > > // SubFoo can't get at BaseFoo's rs.getInt("x"), > > // it's visible only to BaseFoo. Instead SubFoo would get > > // the default int value of 0. Just in case both classes have > > // private fields named "x". > > // ReadSerial is caller sensitive. > > this.y = rs.getInt("y"); > > } > > } > > > > Classes in the heirarchy can provide a static method that throws an > > exception to check invarients while preventing a finaliser attack. We'd > > want to check invarients for other constructors also, but for > > berevity... > > > > Eg: > > > > class BaseFoo implements Serializable{ > > > > public static ReadSerial check(ReadSerial rs) throws Exception > > { > > if (rs.getInt("x") < 1) > > throw IllegalArgumentException("message"); > > return rs; > > } > > .... > > > > > > Sent from my Nokia N900. > > > > ----- Original message ----- > > So, if I understand this correctly, the way this would get used is: > > > > class BaseFoo implements Serializable { > > private final int x; > > > > public BaseFoo(ReadSerial rs) { > > this(rs.getInt("x")); > > } > > > > public BaseFoo(int x) { > > this.x = x; > > } > > } > > > > Right? > > > > What happens with subclasses?? I think then I need an extra RS arg in > > my constructors: > > > > class SubFoo extends BaseFoo { > > private final int y; > > > > public SubFoo(ReadSerial rs) { > > this(rs.getInt("y")); > > } > > > > public BaseFoo(ReadSerial rs, int y) { > > super(rs); > > this.y = y; > > } > > } > > > > Is this what you envision? > > > > > > > > > > > > On 12/27/2014 8:03 PM, Peter Firmstone wrote: > > Is there any interest in developing an explicit API for > > Serialization?: > > > > 1. Use a public constructor signature with a single argument, > > ReadSerialParameters (read only, writable only by the > > serialization framework) to recreate objects, subclasses (when > > permitted) call this first from their own constructor, they have > > an identical constructor signature.? ReadSerialParameters that are > > null may contain a circular reference and will be available after > > construction, see #3 below. > > 2. Use a factory method (defined by an interface) with one parameter, > > WriteSerialParameters (write only, readable only by the > > serialization framework), this method can be overridden by > > subclasses (when permitted) > > 3. For circular links, a public method (defined by an interface) that > > accepts one argument, ReadSerialParameters, this method is called > > after the constructor completes, subclasses overriding this should > > call the superclass method.? If this method is not called, an > > implementation, if known to possibly contain circular links, > > should check it has been fully initialized in each object method > > called. > > 4. Retains compatibility with current serialization stream format. > > 5. Each serial field has a name, calling class and object reference, > > similar to explicitly declaring "private static final > > ObjectStreamField[] serialPersistentFields ". > > > > Benefits: > > > > 1. An object's internal form is not publicised. > > 2. Each class in an object's heirarchy can use a static method to > > check invarients and throw an exception, prior to > > java.lang.Object's constructor being called, preventing > > construction and avoiding finalizer attacks. > > 3. Final field friendly. > > 4. Compatible with existing serial form. > > 5. Flexible serial form evolution. > > 6. All methods are public and explicitly defined. > > 7. All class ProtectionDomain's exist in the current execution > > context, allowing an object to throw a SecurityException before > > construction. > > 8. Less susceptible to deserialization attacks. > > > > Problems: > > > > 1. Implementations cannot be package private or private.? Implicit > > serialization publicises internal form, any thoughts? > > > > Recommendations: > > > > 1. Create a security check in the serialization framework for > > implicit serialization, allowing administrators to reduce their > > deserialization attack surface. > > 2. For improved security, disallow classes implementing explicit > > serialization from having static state and static initializer > > blocks, only allow static methods, this would require complier and > > verifier changes. > > 3. Alternative to #2, allow final static fields, but don't allow > > static initializer blocks or mutable static fields, similar to > > interfaces. > > > > Penny for your thoughts? > > > > Regards, > > > > Peter Firmstone. > > From peter.firmstone at zeus.net.au Sat Jan 3 12:38:08 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Sat, 03 Jan 2015 22:38:08 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54A7BDCE.7020308@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> Message-ID: <1420288688.3416.16.camel@Nokia-N900> > Hi, > > I would like to know what are the potential issues with simple > constructor chaining where each constructor checks the invariant of its > class state (with the already initialized state of superclass(es)). Finalizer attack; a subclass can override the finalize method and receive a thread of execution, even when it hasn't gotten hold of a reference during construction. It's best to prevent an object's construction, by throwing any exceptions before Object's default constructor is called. A class can override the finalize method and make it final, but than can cause objects to hang around longer than necessary. It's worth noting that all classes that implement Serializable have a default no arg constructor, so presently don't have the option of preventing object construction and typically have to clone mutable referents after deserialization. Final fields only provide protection for immutable referent objects. So this offers an improvement, at least for objects that don't have circular links and it would allow them to retain compatibility with their existing serial form. Cheers, Peter. For > example, the above would be written as: > > class A { >? ? ? ? ? final int lower, upper; >? ? ? ? ? A(ReadSerial rs) { >? ? ? ? ? ? ? ? ? int l = rs.getInt("lower"); >? ? ? ? ? ? ? ? ? int u = rs.getInt("upper"); >? ? ? ? ? ? ? ? ? if (l > u) throw new IllegalArgumentException(); >? ? ? ? ? ? ? ? ? lower = l; >? ? ? ? ? ? ? ? ? upper = u; >? ? ? ? ? } > } > > class B extends A { >? ? ? ? ? final int cur; >? ? ? ? ? B(ReadSerial rs) { >? ? ? ? ? ? ? ? ? super(rs); >? ? ? ? ? ? ? ? ? int c = rs.getInt("cur"); >? ? ? ? ? ? ? ? ? if (c < lower || c > upper) throw new > IllegalArgumentException();? ? ? ? ? ? ? ? cur = c; >? ? ? ? ? } > } > > > > > > Another challenge in invariant checking is circular data structures.? > > If you have two objects: > > > > class Brother { > > final Brother brother; > > } > > > > that refer to each other, an invariant you might want to check after > > deserialization is that > > this.brother.brother == this > > This is tricky, yes. In general, the graph of objects has to be linked > somehow gradually where links are established to half-initialized > objects. So there would have to be a special post-deserialization > call-back over the deserialized objects just for checking the > invariants. But what to do with the instances that "escape" during the > process? This is why explicit (de)serialization is tricky - it allows > arbitrary user code to be executed during the construction of the object > graph. > > Perhaps we just need some kind of declarative prescription of serialized > state and mapping to fields of objects in the form of annotations or > such (see what JPA is doing in this field). Combined with > post-deserialization invariant-checking call-back method perhaps. > > Regards, Peter > > > > > Obviously you have to patch one or the other instance after > > construction to retain the circular references; at what point do you > > do invariant checking? > > > > On 1/1/2015 7:43 AM, Peter Firmstone wrote: > > > Subclass example: > > > > > > class SubFoo extends BaseFoo { > > > > > > public static ReadSerial check(ReadSerial rs){ > > > if (rs.getInt("y") > 128) throw Exception("too big"); > > > return rs; > > > } > > > > > > private final int y; > > > > > > public SubFoo( int x , int y) { > > > super(x); > > > this.y = y; > > > } > > > > > > public SubFoo( ReadSerial rs ){ > > > super(BaseFoo.check(check(rs))); > > > // SubFoo can't get at BaseFoo's rs.getInt("x"), > > > // it's visible only to BaseFoo. Instead SubFoo would get > > > // the default int value of 0. Just in case both classes have > > > // private fields named "x". > > > // ReadSerial is caller sensitive. > > > this.y = rs.getInt("y"); > > > } > > > } > > > > > > Classes in the heirarchy can provide a static method that throws an > > > exception to check invarients while preventing a finaliser attack. > > > We'd want to check invarients for other constructors also, but for > > > berevity... > > > > > > Eg: > > > > > > class BaseFoo implements Serializable{ > > > > > > public static ReadSerial check(ReadSerial rs) throws Exception > > > { > > > if (rs.getInt("x") < 1) > > > throw IllegalArgumentException("message"); > > > return rs; > > > } > > > .... > > > > > > > > > Sent from my Nokia N900. > > > > > > ----- Original message ----- > > > So, if I understand this correctly, the way this would get used is: > > > > > > class BaseFoo implements Serializable { > > > private final int x; > > > > > > public BaseFoo(ReadSerial rs) { > > > this(rs.getInt("x")); > > > } > > > > > > public BaseFoo(int x) { > > > this.x = x; > > > } > > > } > > > > > > Right? > > > > > > What happens with subclasses?? I think then I need an extra RS arg > > > in my > > > constructors: > > > > > > class SubFoo extends BaseFoo { > > > private final int y; > > > > > > public SubFoo(ReadSerial rs) { > > > this(rs.getInt("y")); > > > } > > > > > > public BaseFoo(ReadSerial rs, int y) { > > > super(rs); > > > this.y = y; > > > } > > > } > > > > > > Is this what you envision? > > > > > > > > > > > > > > > > > > On 12/27/2014 8:03 PM, Peter Firmstone wrote: > > > Is there any interest in developing an explicit API for > > > Serialization?: > > > > > > 1. Use a public constructor signature with a single argument, > > > ReadSerialParameters (read only, writable only by the > > > serialization framework) to recreate objects, subclasses (when > > > permitted) call this first from their own constructor, they have > > > an identical constructor signature. ReadSerialParameters that are > > > null may contain a circular reference and will be available after > > > construction, see #3 below. > > > 2. Use a factory method (defined by an interface) with one > > > parameter, > > > WriteSerialParameters (write only, readable only by the > > > serialization framework), this method can be overridden by > > > subclasses (when permitted) > > > 3. For circular links, a public method (defined by an interface) > > > that > > > accepts one argument, ReadSerialParameters, this method is called > > > after the constructor completes, subclasses overriding this should > > > call the superclass method.? If this method is not called, an > > > implementation, if known to possibly contain circular links, > > > should check it has been fully initialized in each object method > > > called. > > > 4. Retains compatibility with current serialization stream format. > > > 5. Each serial field has a name, calling class and object > > > reference, > > > similar to explicitly declaring "private static final > > > ObjectStreamField[] serialPersistentFields ". > > > > > > Benefits: > > > > > > 1. An object's internal form is not publicised. > > > 2. Each class in an object's heirarchy can use a static method to > > > check invarients and throw an exception, prior to > > > java.lang.Object's constructor being called, preventing > > > construction and avoiding finalizer attacks. > > > 3. Final field friendly. > > > 4. Compatible with existing serial form. > > > 5. Flexible serial form evolution. > > > 6. All methods are public and explicitly defined. > > > 7. All class ProtectionDomain's exist in the current execution > > > context, allowing an object to throw a SecurityException before > > > construction. > > > 8. Less susceptible to deserialization attacks. > > > > > > Problems: > > > > > > 1. Implementations cannot be package private or private.? Implicit > > > serialization publicises internal form, any thoughts? > > > > > > Recommendations: > > > > > > 1. Create a security check in the serialization framework for > > > implicit serialization, allowing administrators to reduce their > > > deserialization attack surface. > > > 2. For improved security, disallow classes implementing explicit > > > serialization from having static state and static initializer > > > blocks, only allow static methods, this would require complier and > > > verifier changes. > > > 3. Alternative to #2, allow final static fields, but don't allow > > > static initializer blocks or mutable static fields, similar to > > > interfaces. > > > > > > Penny for your thoughts? > > > > > > Regards, > > > > > > Peter Firmstone. > > > > From peter.levart at gmail.com Sat Jan 3 15:36:18 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sat, 03 Jan 2015 16:36:18 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <1420288688.3416.16.camel@Nokia-N900> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> Message-ID: <54A80C72.3030906@gmail.com> On 01/03/2015 01:38 PM, Peter Firmstone wrote: > > > Hi, > > > > I would like to know what are the potential issues with simple > > constructor chaining where each constructor checks the invariant of its > > class state (with the already initialized state of superclass(es)). > > Finalizer attack; a subclass can override the finalize method and > receive a thread of execution, even when it hasn't gotten hold of a > reference during construction. It's best to prevent an object's > construction, by throwing any exceptions before Object's default > constructor is called. > As Brian points out, this scheme can only validate intra-class invariants. It can't validate class-against-subclass state. And, the finalize() method will be called anyway - although this time with fully uninitialized instance (all fields default values). If this is safer, then alright. Is there anything in JLS that guarantees finalization for instances which fail construction or deserialization? Wouldn't it be better to "register" for finalization just those instances that complete their construction or deserialization normally? I'm just trying to understand why it is the way it is. Would something like this prevent Finalizer attacks? - leave finalization registration the way it is (at object allocation time). - provide internal API with which a previously registered object can be de-registered - deserialization infrastructure de-registers the instances that fail deserialization Regards, Peter > > A class can override the finalize method and make it final, but than > can cause objects to hang around longer than necessary. > > It's worth noting that all classes that implement Serializable have a > default no arg constructor, so presently don't have the option of > preventing object construction and typically have to clone mutable > referents after deserialization. Final fields only provide protection > for immutable referent objects. > > So this offers an improvement, at least for objects that don't have > circular links and it would allow them to retain compatibility with > their existing serial form. > > Cheers, > > Peter. > > For > > example, the above would be written as: > > > > class A { > > final int lower, upper; > > A(ReadSerial rs) { > > int l = rs.getInt("lower"); > > int u = rs.getInt("upper"); > > if (l > u) throw new IllegalArgumentException(); > > lower = l; > > upper = u; > > } > > } > > > > class B extends A { > > final int cur; > > B(ReadSerial rs) { > > super(rs); > > int c = rs.getInt("cur"); > > if (c < lower || c > upper) throw new > > IllegalArgumentException(); cur = c; > > } > > } > > > > > > > > > > Another challenge in invariant checking is circular data structures. > > > If you have two objects: > > > > > > class Brother { > > > final Brother brother; > > > } > > > > > > that refer to each other, an invariant you might want to check after > > > deserialization is that > > > this.brother.brother == this > > > > This is tricky, yes. In general, the graph of objects has to be linked > > somehow gradually where links are established to half-initialized > > objects. So there would have to be a special post-deserialization > > call-back over the deserialized objects just for checking the > > invariants. But what to do with the instances that "escape" during the > > process? This is why explicit (de)serialization is tricky - it allows > > arbitrary user code to be executed during the construction of the > object > > graph. > > > > Perhaps we just need some kind of declarative prescription of > serialized > > state and mapping to fields of objects in the form of annotations or > > such (see what JPA is doing in this field). Combined with > > post-deserialization invariant-checking call-back method perhaps. > > > > Regards, Peter > > > > > > > > Obviously you have to patch one or the other instance after > > > construction to retain the circular references; at what point do you > > > do invariant checking? > > > > > > On 1/1/2015 7:43 AM, Peter Firmstone wrote: > > > > Subclass example: > > > > > > > > class SubFoo extends BaseFoo { > > > > > > > > public static ReadSerial check(ReadSerial rs){ > > > > if (rs.getInt("y") > 128) throw Exception("too big"); > > > > return rs; > > > > } > > > > > > > > private final int y; > > > > > > > > public SubFoo( int x , int y) { > > > > super(x); > > > > this.y = y; > > > > } > > > > > > > > public SubFoo( ReadSerial rs ){ > > > > super(BaseFoo.check(check(rs))); > > > > // SubFoo can't get at BaseFoo's rs.getInt("x"), > > > > // it's visible only to BaseFoo. Instead SubFoo would get > > > > // the default int value of 0. Just in case both classes have > > > > // private fields named "x". > > > > // ReadSerial is caller sensitive. > > > > this.y = rs.getInt("y"); > > > > } > > > > } > > > > > > > > Classes in the heirarchy can provide a static method that throws an > > > > exception to check invarients while preventing a finaliser attack. > > > > We'd want to check invarients for other constructors also, but for > > > > berevity... > > > > > > > > Eg: > > > > > > > > class BaseFoo implements Serializable{ > > > > > > > > public static ReadSerial check(ReadSerial rs) throws Exception > > > > { > > > > if (rs.getInt("x") < 1) > > > > throw IllegalArgumentException("message"); > > > > return rs; > > > > } > > > > .... > > > > > > > > > > > > Sent from my Nokia N900. > > > > > > > > ----- Original message ----- > > > > So, if I understand this correctly, the way this would get used is: > > > > > > > > class BaseFoo implements Serializable { > > > > private final int x; > > > > > > > > public BaseFoo(ReadSerial rs) { > > > > this(rs.getInt("x")); > > > > } > > > > > > > > public BaseFoo(int x) { > > > > this.x = x; > > > > } > > > > } > > > > > > > > Right? > > > > > > > > What happens with subclasses? I think then I need an extra RS arg > > > > in my > > > > constructors: > > > > > > > > class SubFoo extends BaseFoo { > > > > private final int y; > > > > > > > > public SubFoo(ReadSerial rs) { > > > > this(rs.getInt("y")); > > > > } > > > > > > > > public BaseFoo(ReadSerial rs, int y) { > > > > super(rs); > > > > this.y = y; > > > > } > > > > } > > > > > > > > Is this what you envision? > > > > > > > > > > > > > > > > > > > > > > > > On 12/27/2014 8:03 PM, Peter Firmstone wrote: > > > > Is there any interest in developing an explicit API for > > > > Serialization?: > > > > > > > > 1. Use a public constructor signature with a single argument, > > > > ReadSerialParameters (read only, writable only by the > > > > serialization framework) to recreate objects, subclasses (when > > > > permitted) call this first from their own constructor, they have > > > > an identical constructor signature. ReadSerialParameters that are > > > > null may contain a circular reference and will be available after > > > > construction, see #3 below. > > > > 2. Use a factory method (defined by an interface) with one > > > > parameter, > > > > WriteSerialParameters (write only, readable only by the > > > > serialization framework), this method can be overridden by > > > > subclasses (when permitted) > > > > 3. For circular links, a public method (defined by an interface) > > > > that > > > > accepts one argument, ReadSerialParameters, this method is called > > > > after the constructor completes, subclasses overriding this should > > > > call the superclass method. If this method is not called, an > > > > implementation, if known to possibly contain circular links, > > > > should check it has been fully initialized in each object method > > > > called. > > > > 4. Retains compatibility with current serialization stream format. > > > > 5. Each serial field has a name, calling class and object > > > > reference, > > > > similar to explicitly declaring "private static final > > > > ObjectStreamField[] serialPersistentFields ". > > > > > > > > Benefits: > > > > > > > > 1. An object's internal form is not publicised. > > > > 2. Each class in an object's heirarchy can use a static method to > > > > check invarients and throw an exception, prior to > > > > java.lang.Object's constructor being called, preventing > > > > construction and avoiding finalizer attacks. > > > > 3. Final field friendly. > > > > 4. Compatible with existing serial form. > > > > 5. Flexible serial form evolution. > > > > 6. All methods are public and explicitly defined. > > > > 7. All class ProtectionDomain's exist in the current execution > > > > context, allowing an object to throw a SecurityException before > > > > construction. > > > > 8. Less susceptible to deserialization attacks. > > > > > > > > Problems: > > > > > > > > 1. Implementations cannot be package private or private. Implicit > > > > serialization publicises internal form, any thoughts? > > > > > > > > Recommendations: > > > > > > > > 1. Create a security check in the serialization framework for > > > > implicit serialization, allowing administrators to reduce their > > > > deserialization attack surface. > > > > 2. For improved security, disallow classes implementing explicit > > > > serialization from having static state and static initializer > > > > blocks, only allow static methods, this would require complier and > > > > verifier changes. > > > > 3. Alternative to #2, allow final static fields, but don't allow > > > > static initializer blocks or mutable static fields, similar to > > > > interfaces. > > > > > > > > Penny for your thoughts? > > > > > > > > Regards, > > > > > > > > Peter Firmstone. > > > > > > > From ivan.gerasimov at oracle.com Sat Jan 3 17:39:21 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Sat, 03 Jan 2015 20:39:21 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54A7AD40.7080108@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> Message-ID: <54A82949.4010308@oracle.com> On 03.01.2015 11:50, Alan Bateman wrote: > On 26/12/2014 10:02, Ivan Gerasimov wrote: >> Hi! >> >> I changed my mind about handling only entirely quoted entries in PATH. >> The webrev was updated. >> >> Now, on Windows paths with quoted parts are supported. >> If the quoted fragment of the path contains the delimiter -- ';', it >> will be preserved as a part of the path. >> >> This is consistent with how Windows command prompt resolves PATH. >> >> The regression test was extended to cover new functionality. >> >> The webrev was updated in place: >> http://cr.openjdk.java.net/~igerasim/8067951/2/webrev/ >> > I assume it can't be very common to quote individual elements in this > manner, otherwise we would have had bug reports on this many years ago. Some applications add their paths to PATH quoted, but these paths are not very common locations to load dlls from. In the blog referenced earlier I saw this comment: "Turned out the DirectX SDK inserted its Utilities\Bin\x86 folder at the start of my PATH and used quotes." > I agree we should probably support this and the approach in your > updated webrev looks reasonable to me. One thing to consider is > refactoring it so that the Windows specific handling is the Windows > implementation of ClassLoaderHelper rather than in initializePaths. > Currently, there are tree variants of ClassLoaderHelper: for Windows, for Unix and for MacOS. We have to either duplicate code in Unix and MacOS realizations, or introduce another Helper class for initializing paths only, which would have only two realizations: for Windows and all Unixes. I would choose the second approach. What would you suggest? Sincerely yours, Ivan From chris.hegarty at oracle.com Sat Jan 3 19:34:15 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Sat, 3 Jan 2015 19:34:15 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54A80C72.3030906@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> <54A80C72.3030906@gmail.com> Message-ID: Just a quick comment about the finalization aspect ( as I have been thinking about this too ). On 3 Jan 2015, at 15:36, Peter Levart wrote: > On 01/03/2015 01:38 PM, Peter Firmstone wrote: >> >> > Hi, >> > >> > I would like to know what are the potential issues with simple >> > constructor chaining where each constructor checks the invariant of its >> > class state (with the already initialized state of superclass(es)). >> >> Finalizer attack; a subclass can override the finalize method and >> receive a thread of execution, even when it hasn't gotten hold of a reference during construction. It's best to prevent an object's construction, by throwing any exceptions before Object's default constructor is called. >> > > As Brian points out, this scheme can only validate intra-class invariants. It can't validate class-against-subclass state. And, the finalize() method will be called anyway - although this time with fully uninitialized instance (all fields default values). If this is safer, then alright. In many cases default values will be safer, but not always, e.g. port number of ?0? implies ephemeral port. > Is there anything in JLS that guarantees finalization for instances which fail construction or deserialisation? "An object o is not finalizable until its constructor has invoked the constructor for Object on o and that invocation has completed successfully (that is, without throwing an exception).? [1] It is unfortunate that the way default serialization constructs objects, the j.l.Object constructor is invoked before attempting to set any fields. > Wouldn't it be better to "register" for finalization just those instances that complete their construction or deserialization normally? I'm just trying to understand why it is the way it is. > > Would something like this prevent Finalizer attacks? > > - leave finalization registration the way it is (at object allocation time). > - provide internal API with which a previously registered object can be de-registered > - deserialization infrastructure de-registers the instances that fail deserialisation I have given this some thought too, and also think that deserialized objects should not become finalizable until after the fields have been successfully set. Whether this be an internal API, JLS change, or an extension to the ?magic? serialization mechanism to invoke the first non-serializable classes no-args constructor, I?m not sure. -Chris. [1] http://docs.oracle.com/javase/specs/jls/se7/html/jls-12.html#jls-12.6.1 > > Regards, Peter > From peter.firmstone at zeus.net.au Sat Jan 3 20:29:25 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Sun, 04 Jan 2015 06:29:25 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54A80C72.3030906@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> <54A80C72.3030906@gmail.com> Message-ID: <1420316965.3890.10.camel@Nokia-N900> ----- Original message ----- > > As Brian points out, this scheme can only validate intra-class > invariants. It can't validate class-against-subclass state. Did he say that? It's true that a superclass can't validate subclass state, it can't be expected to know much about it, but it can validate its own and prevent construction. It can prevent extension too, ie be final, or require a permission to be extended. And, the > finalize() method will be called anyway The jvm makes no gaurantee a finalizer will run. - although this time with fully > uninitialized instance (all fields default values). If this is safer, > then alright. > > Is there anything in JLS that guarantees finalization for instances > which fail construction or deserialization? Not that I'm aware of. Wouldn't it be better to > "register" for finalization just those instances that complete their > construction or deserialization normally? I'm just trying to understand > why it is the way it is. Perhaps that might be an option, someone who knows more about finalization might be able to help here. In the early days, the sandbox and bytecode verifier were intended to make java secure, additional private methods were created as vulnerabilities were better understood. I think a problem with Serialization is you have to establish trust before you can use it. It would be nice if there was an input validator like html servers use, to validate the stream before instantiating objects. Eg array size check before array creation, type check before object instantiation and restrict creation to permitted classes, to a subset of what's available on the class path. Techniques that prevent invalid object creation, are the most secure. Circular links between untrusted classes still appear to require at least one object to be created before invarients can be checked and at least one mutated after. Another option might be to require a Permission for Circular links, such that a thread may be run with a Subject that allows it for trusted connections and disallows it for untrusted serial data. Regards, Peter. > > Would something like this prevent Finalizer attacks? > > - leave finalization registration the way it is (at object allocation > time). - provide internal API with which a previously registered object > can be? de-registered > - deserialization infrastructure de-registers the instances that fail > deserialization > > An alternative is to make an object unusable if it fails invarient checks, so all methods throw an exception. > Regards, Peter > > > > > A class can override the finalize method and make it final, but than > > can cause objects to hang around longer than necessary. > > > > It's worth noting that all classes that implement Serializable have a > > default no arg constructor, so presently don't have the option of > > preventing object construction and typically have to clone mutable > > referents after deserialization. Final fields only provide protection > > for immutable referent objects. > > > > So this offers an improvement, at least for objects that don't have > > circular links and it would allow them to retain compatibility with > > their existing serial form. > > > > Cheers, > > > > Peter. > > > > For > > > example, the above would be written as: > > > > > > class A { > > > final int lower, upper; > > > A(ReadSerial rs) { > > > int l = rs.getInt("lower"); > > > int u = rs.getInt("upper"); > > > if (l > u) throw new IllegalArgumentException(); > > > lower = l; > > > upper = u; > > > } > > > } > > > > > > class B extends A { > > > final int cur; > > > B(ReadSerial rs) { > > > super(rs); > > > int c = rs.getInt("cur"); > > > if (c < lower || c > upper) throw new > > > IllegalArgumentException();? ? ? ? ? ? ? ? ? ? ? ? ? ? ? cur = c; > > > } > > > } > > > > > > > > > > > > > > Another challenge in invariant checking is circular data > > > > structures. If you have two objects: > > > > > > > > class Brother { > > > > final Brother brother; > > > > } > > > > > > > > that refer to each other, an invariant you might want to check > > > > after deserialization is that > > > > this.brother.brother == this > > > > > > This is tricky, yes. In general, the graph of objects has to be > > > linked somehow gradually where links are established to > > > half-initialized objects. So there would have to be a special > > > post-deserialization call-back over the deserialized objects just > > > for checking the invariants. But what to do with the instances that > > > "escape" during the process? This is why explicit (de)serialization > > > is tricky - it allows arbitrary user code to be executed during the > > > construction of the > > object > > > graph. > > > > > > Perhaps we just need some kind of declarative prescription of > > serialized > > > state and mapping to fields of objects in the form of annotations or > > > such (see what JPA is doing in this field). Combined with > > > post-deserialization invariant-checking call-back method perhaps. > > > > > > Regards, Peter. From peter.firmstone at zeus.net.au Sat Jan 3 21:15:27 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Sun, 04 Jan 2015 07:15:27 +1000 Subject: Explicit Serialization API and Security In-Reply-To: References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> <54A80C72.3030906@gmail.com> Message-ID: <1420319727.3890.16.camel@Nokia-N900> P.S. Thanks for engaging this difficult subject. It's worth remembering the finalizer attack isn't the only issue, a subclass will have a reference after construction completes, it has a thread of execution and if the superclass hasn't checked invarients, because circular links haven't been wired up, it's game over. For explicit serialization, this is only a problem for circular relationships. A permission check for extension can be performed before Object's constructor is called, or the class can be final. That leaves the case of determining which of two objects is trusted to pass a reference before invarients have been checked. One of the objects in the circular relationship could be the attacker and we just gave it a reference before checking invarients. Once created, an object has little control over references to it. So it would seem that circular links would only be allowed on trusted serial data, for network connections, that means an authenticated, secure connection. So a class that implements Circular would require a permission check to be performed after the Circular object is created, but before passing a reference that wires up a circular link. So it becomes a case of: 1. Do I trust my subclass? 2. Do I trust the framework? 3 Am I trustworthy? 4. Can I trust classes loaded after my constructor completes, to receive my reference, even though I haven't checked invarients and to honour my wishes and delete my reference if I find invarients aren't satisfied? Peter. ----- Original message ----- > Just a quick comment about the finalization aspect ( as I have been > thinking about this too ). > > On 3 Jan 2015, at 15:36, Peter Levart wrote: > > On 01/03/2015 01:38 PM, Peter Firmstone wrote: > > > > > > > Hi, > > > > > > > > I would like to know what are the potential issues with simple > > > > constructor chaining where each constructor checks the invariant > > > > of its class state (with the already initialized state of > > > > superclass(es)). > > > > > > Finalizer attack; a subclass can override the finalize method and > > > receive a thread of execution, even when it hasn't gotten hold of a > > > reference during construction. It's best to prevent an object's > > > construction, by throwing any exceptions before Object's default > > > constructor is called. > > > > > > > As Brian points out, this scheme can only validate intra-class > > invariants. It can't validate class-against-subclass state. And, the > > finalize() method will be called anyway - although this time with > > fully uninitialized instance (all fields default values). If this is > > safer, then alright. > > In many cases default values will be safer, but not always, e.g. port > number of ?0? implies ephemeral port. > > > Is there anything in JLS that guarantees finalization for instances > > which fail construction or deserialisation? > >? "An object o is not finalizable until its constructor has invoked the > constructor for Object on o and that invocation has completed > successfully (that is, without throwing an exception).? [1] > > It is unfortunate that the way default serialization constructs objects, > the j.l.Object constructor is invoked before attempting to set any > fields. > > > Wouldn't it be better to "register" for finalization just those > > instances that complete their construction or deserialization > > normally? I'm just trying to understand why it is the way it is. > > > > Would something like this prevent Finalizer attacks? > > > > - leave finalization registration the way it is (at object allocation > > time). - provide internal API with which a previously registered > > object can be de-registered - deserialization infrastructure > > de-registers the instances that fail deserialisation > > I have given this some thought too, and also think that deserialized > objects should not become finalizable until after the fields have been > successfully set. Whether this be an internal API, JLS change, or an > extension to the ?magic? serialization mechanism to invoke the first > non-serializable classes no-args constructor, I?m not sure. > > -Chris. > > [1] > http://docs.oracle.com/javase/specs/jls/se7/html/jls-12.html#jls-12.6.1 > > > > > Regards, Peter > > From peter.levart at gmail.com Sat Jan 3 23:55:17 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sun, 04 Jan 2015 00:55:17 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <1420316965.3890.10.camel@Nokia-N900> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> <54A80C72.3030906@gmail.com> <1420316965.3890.10.camel@Nokia-N900> Message-ID: <54A88165.3010603@gmail.com> On 01/03/2015 09:29 PM, Peter Firmstone wrote: > ----- Original message ----- > > > > As Brian points out, this scheme can only validate intra-class > > invariants. It can't validate class-against-subclass state. Sorry, I meant it can't validate class -against-superclass state. > > Did he say that? Not directly - his example was pointing that out. Regards, Peter From peter.firmstone at zeus.net.au Sun Jan 4 01:48:50 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Sun, 04 Jan 2015 11:48:50 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54A88165.3010603@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> <54A80C72.3030906@gmail.com> <1420316965.3890.10.camel@Nokia-N900> <54A88165.3010603@gmail.com> Message-ID: <54A89C02.2060903@zeus.net.au> On 4/01/2015 9:55 AM, Peter Levart wrote: >> ----- Original message ----- >> > >> > As Brian points out, this scheme can only validate intra-class >> > invariants. It can't validate class-against-subclass state. > > Sorry, I meant it can't validate class -against-superclass state. > >> >> Did he say that? > > Not directly - his example was pointing that out. > Ah, I think you missed my example :), Brian was more or less asking a question, but the answer is definitely yes. In my example below, subclass B's constructor creates an unextended instance of A, from within B's static check method. It then uses this copy of A to check A's field against B's. At this point, note an instance of B hasn't been constructed, it can still throw an exception, without B ever being constructed, also you'll note that when an unextended copy of A was constructed, A checked its invariants too. B having satisfied all invariants can now be safely constructed. The unextended copy of A is discarded without ever being shared. - I don't see how this invariant-checking mechanism can enforce > invariants between superclass fields and subclass fields.? For example: > > class A { > int lower, upper; // invariant: lower<= upper > } > > class B extends A { > int cur; // invariant: lower<= cur<= upper > } > > To check such an invariant, the serialization library would have to > construct the object (in a potentially bad state), invoke the checker at > each layer, and then fail deserialization if any checker said no.? Eg, some complexity, but bullet proof: public class A ( public final int lower, upper; private static boolean check(ReadSerial rs){ if (rs.getInt("lower")> rs.getInt("upper")) throw new IllegalArgumentException( "lower bound must be less than or equal to upper"); return true; } public A(ReadSerial rs){ this(check(rs), rs); } private A(boolean checked, ReadSerial rs){ super(); lower = rs.getInt("lower"); upper = rs.getInt("upper"); } // other constructors omitted must also check invarients } class B extends A { public final int cur; private static ReadSerial check(ReadSerial rs) { A a = new A(rs); int cur = rs.getInt("cur"); if ( a.lower> cur || cur> a.upper ) throw new IllegalArgumentException( "cur outside lower and upper bounds"); return rs; } public B(ReadSerial rs) { super(check(rs)); cur = rs.getInt("cur"); } } Cheers, Peter. On 4/01/2015 9:55 AM, Peter Levart wrote: > > > > > Regards, Peter From Alan.Bateman at oracle.com Sun Jan 4 10:06:02 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 04 Jan 2015 10:06:02 +0000 Subject: pre-RFR (s): 8049847: Enhance PrintWriter line.separator handling In-Reply-To: <54A6BB84.6060508@oracle.com> References: <54A6BB84.6060508@oracle.com> Message-ID: <54A9108A.2090704@oracle.com> On 02/01/2015 15:38, Claes Redestad wrote: > Hi, > > this is a proposal to resolve concerns that > PrintWriter/BufferedWriter behave > inconsistently with j.u.Formatter when setting the line.separator > property to override > system-appropriate line breaks. > > Instead of providing a set of new constructors, I propose to simply > add a new default > method j.l.Appendable#lineSeparator, which by default delegates to > System.lineSeparator(). > This allows classes such as PrintWriter/BufferedWriter to provide > overrides which > communicate to j.u.Formatter their intended behavior. In the bug report then I assume the inconsistency only arises because the test case attempts to change the line separator in the main method, which is too late. It should work if set on the command-line but this of course sets it system-wide. Your proposal to add a lineSeperator() method to Appendable would allow for customization but I'm not sure that it's the right place (as Appendable is a very general interface and would require every Appendable implementation that writes lines to be updated to make use of the new method). An alternative to consider is adding the notion of lines to java.io.Writer and update it to define newLine() and lineSeperator() methods (i.e. move the BufferWriter::newLine method up to Writer). It's a bit of extra work to update BufferedWriter's spec, adjust a few things in PrintWriter/PrintStream, and special-case Writers in Formatter but something to consider. -Alan From peter.levart at gmail.com Sun Jan 4 10:19:39 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sun, 04 Jan 2015 11:19:39 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <54A89C02.2060903@zeus.net.au> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> <54A80C72.3030906@gmail.com> <1420316965.3890.10.camel@Nokia-N900> <54A88165.3010603@gmail.com> <54A89C02.2060903@zeus.net.au> Message-ID: <54A913BB.30600@gmail.com> On 01/04/2015 02:48 AM, Peter Firmstone wrote: > class B extends A { > > public final int cur; > > private static ReadSerial check(ReadSerial rs) { > A a = new A(rs); What to do if A is abstract? Regards, Peter > int cur = rs.getInt("cur"); > if ( a.lower> cur || cur> a.upper ) > throw new IllegalArgumentException( > "cur outside lower and upper bounds"); > return rs; > } > > public B(ReadSerial rs) { > super(check(rs)); > cur = rs.getInt("cur"); > } > } From peter.levart at gmail.com Sun Jan 4 11:03:35 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sun, 04 Jan 2015 12:03:35 +0100 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54A82949.4010308@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> Message-ID: <54A91E07.1080202@gmail.com> On 01/03/2015 06:39 PM, Ivan Gerasimov wrote: > > On 03.01.2015 11:50, Alan Bateman wrote: >> On 26/12/2014 10:02, Ivan Gerasimov wrote: >>> Hi! >>> >>> I changed my mind about handling only entirely quoted entries in PATH. >>> The webrev was updated. >>> >>> Now, on Windows paths with quoted parts are supported. >>> If the quoted fragment of the path contains the delimiter -- ';', it >>> will be preserved as a part of the path. >>> >>> This is consistent with how Windows command prompt resolves PATH. >>> >>> The regression test was extended to cover new functionality. >>> >>> The webrev was updated in place: >>> http://cr.openjdk.java.net/~igerasim/8067951/2/webrev/ >>> >> I assume it can't be very common to quote individual elements in this >> manner, otherwise we would have had bug reports on this many years ago. > > Some applications add their paths to PATH quoted, but these paths are > not very common locations to load dlls from. > In the blog referenced earlier I saw this comment: "Turned out the > DirectX SDK inserted its Utilities\Bin\x86 folder at the start of my > PATH and used quotes." > >> I agree we should probably support this and the approach in your >> updated webrev looks reasonable to me. One thing to consider is >> refactoring it so that the Windows specific handling is the Windows >> implementation of ClassLoaderHelper rather than in initializePaths. >> > Currently, there are tree variants of ClassLoaderHelper: for Windows, > for Unix and for MacOS. > We have to either duplicate code in Unix and MacOS realizations, or > introduce another Helper class for initializing paths only, which > would have only two realizations: for Windows and all Unixes. > > I would choose the second approach. > What would you suggest? Hi Ivan, There were other cases where one wanted to consolidate duplicated code, but couldn't easily because of subtle differences among OS platforms (UNIXProcess is one of such cases). Perhaps we need something like this: http://cr.openjdk.java.net/~plevart/misc/OS/webrev/ With this compile-time mechanism, one can use switch or if statements in common class to cover subtle differences and only divide code into OS platform specific classes where the bulk of it differs. What do you think? Regards, Peter > > Sincerely yours, > Ivan > From peter.firmstone at zeus.net.au Sun Jan 4 11:07:10 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Sun, 04 Jan 2015 21:07:10 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54A913BB.30600@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> <54A80C72.3030906@gmail.com> <1420316965.3890.10.camel@Nokia-N900> <54A88165.3010603@gmail.com> <54A89C02.2060903@zeus.net.au> <54A913BB.30600@gmail.com> Message-ID: <1420369630.2250.5.camel@Nokia-N900> ----- Original message ----- > > On 01/04/2015 02:48 AM, Peter Firmstone wrote: > > class B extends A { > > > > public final int cur; > > > > private static ReadSerial check(ReadSerial rs) { > > A a = new A(rs); > > What to do if A is abstract? :) Create an anonymous instance, or create a static private unshared class that overrides it, for the purpose of validation and discard it. The important thing is A must retain control of access to its internal state. Cheers, Peter. > > Regards, Peter > > > int cur = rs.getInt("cur"); > > if ( a.lower>? cur || cur>? a.upper ) > > throw new IllegalArgumentException( > > "cur outside lower and upper bounds"); > > return rs; > > } > > > > public B(ReadSerial rs) { > > super(check(rs)); > > cur = rs.getInt("cur"); > > } > > } > From claes.redestad at oracle.com Sun Jan 4 14:10:32 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Sun, 04 Jan 2015 15:10:32 +0100 Subject: pre-RFR (s): 8049847: Enhance PrintWriter line.separator handling In-Reply-To: <54A9108A.2090704@oracle.com> References: <54A6BB84.6060508@oracle.com> <54A9108A.2090704@oracle.com> Message-ID: <54A949D8.2030201@oracle.com> On 2015-01-04 11:06, Alan Bateman wrote: > On 02/01/2015 15:38, Claes Redestad wrote: >> Hi, >> >> this is a proposal to resolve concerns that >> PrintWriter/BufferedWriter behave >> inconsistently with j.u.Formatter when setting the line.separator >> property to override >> system-appropriate line breaks. >> >> Instead of providing a set of new constructors, I propose to simply >> add a new default >> method j.l.Appendable#lineSeparator, which by default delegates to >> System.lineSeparator(). >> This allows classes such as PrintWriter/BufferedWriter to provide >> overrides which >> communicate to j.u.Formatter their intended behavior. > In the bug report then I assume the inconsistency only arises because > the test case attempts to change the line separator in the main > method, which is too late. It should work if set on the command-line > but this of course sets it system-wide. Right, the PrintWriter inconsistency detailed: - methods format/printf uses j.u.Formatter, which uses System.lineSeparator() internally if it encounters an "%n" in the format string. Only responds to when changing line.separator on command-line or similar. - methods println and newLine use the PrintWriter internal lineSeparator value evaluated at object creation time: public PrintWriter(Writer out, boolean autoFlush) { super(out); this.out = out; this.autoFlush = autoFlush; lineSeparator = java.security.AccessController.doPrivileged( new sun.security.action.GetPropertyAction("line.separator")); } This leads to hacks existing in the wild which sets line.separator before creating the PrintWriter, as exemplified here: http://stackoverflow.com/questions/1014287/is-there-a-way-to-make-printwriter-output-to-unix-format Removing this functionality could break compatibility with some applications where developers are intentionally changing the line.separator, while it'd be nice if there was a better way to facilitate this, which I think my proposal would allow with minimal > > Your proposal to add a lineSeperator() method to Appendable would > allow for customization but I'm not sure that it's the right place (as > Appendable is a very general interface and would require every > Appendable implementation that writes lines to be updated to make use > of the new method). Yes, this can seem an awkward place to add this method, but the only place where we'd be able to as easily make java.util.Formatter use the custom lineSeparator, without which we wouldn't be able to resolve the inconsistency above nor provide a more convenient way for developers to avoid the line.separator hack. However, since Appendable is defined rather strictly-coupled with java.util.Formatter ("The Appendable interface must be implemented by any class whose instances are intended to receive formatted output from a java.util.Formatter.") and Formatter provides a higher-order mechanism to print new-lines ("%n") to the Appendable, it doesn't feel entirely arbitrary that the lineSeparator behavior should be accessible via Appendable. > > An alternative to consider is adding the notion of lines to > java.io.Writer and update it to define newLine() and lineSeperator() > methods (i.e. move the BufferWriter::newLine method up to Writer). > It's a bit of extra work to update BufferedWriter's spec, adjust a few > things in PrintWriter/PrintStream, and special-case Writers in > Formatter but something to consider. I need to think about what that would mean, and especially how to solve it on the Formatter side. I worry shoehorning Writer or upcasts into Formatter might be more complex that it's worth. Any ideas? /Claes > > -Alan From peter.levart at gmail.com Sun Jan 4 16:25:14 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sun, 04 Jan 2015 17:25:14 +0100 Subject: RFR: JDK-8047769 SecureRandom should be more frugal with file descriptors In-Reply-To: <54A7228D.4040801@oracle.com> References: <5492D557.2000807@gmail.com> <5498C5A0.50104@oracle.com> <549AA573.2060801@gmail.com> <54A20475.3040206@oracle.com> <54A5AC7C.5010806@gmail.com> <54A7228D.4040801@oracle.com> Message-ID: <54A9696A.3070406@gmail.com> Hi Brad, So I created another webrev (just removed the unneeded import and left-over System.out.println from test): http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.06/ I'll leave it here to rest for a couple of days and if no one objects, I'll push it to jdk9-dev. Thanks everybody for reviews and happy new year! Regards, Peter On 01/02/2015 11:58 PM, Bradford Wetmore wrote: > > On 1/1/2015 12:22 PM, Peter Levart wrote: >> Hi Brad, >> >> Here's next webrev which tries to cover all your comments: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.04/ >> > > I downloaded the webrev.05 (Chris' later followup email) and ran it > through JPRT. The only error was the previously seen -Dseed which is > not your problem. > > Again, I only ran: > > jdk_lang, jdk_math, jdk_util, jdk_io, jdk_net, jdk_nio, > jdk_security*, jdk_rmi, jdk_text, jdk_time, jdk_other, core_tools. > > If you need anything else run, let me know. > >>>>> Looks like you have a committer status, will you be pushing this? >>>> >>>> I can, yes. As soon as we clear-out the remaining questions, right? >>> >>> Yes. The comments below are minor and shouldn't need another review >>> cycle. >> >> I'm worried about the failure of the test you observed while running >> from NetBeans. Perhaps a 0.5s wait is sometimes not enough for >> ReferenceHandler thread to process (enqueue) a WeakReference. Since >> there is already a facility in place to help ReferenceHandler thread >> instead of wait for it, I used it in new version of the test. > > BTW, there's now an unnecessary import from java.lang.AssertionError > added in webrev.04. > >>> TEST RESULT: Failed. Compilation failed: Compilation failed >> >> I changed the test to be self-contained now so one can run it without >> testlib in classpath. > > Thanks. It's compiling fine now. > >>> Two minor nits? SeedGenerator.java: Lines 507/518 >> >> Done that too. > > Thanks. > >>> Maybe issue multiple reads to exercise in1 and in2? e.g. 2 bytes of >>> in1, 4 bytes of in2, then 2 bytes of in1? >> >> The 1st assert makes sure in1 == in2, so there's no point in invoking >> the same instance via two aliases. > > True. > >>> IIRC, when I ran this under NetBeans last week, the second testCaching >>> didn't clear the WeakReference. >> >> This should not happen any more now that the test is helping to enqueue >> the WeakReferences instead of waiting for ReferenceHandler to enqueue >> them. > > Yes, that hit the refQueue.poll(). > > It's always interesting to work with core-libs folks, learn something > new everyday. Mixing Lambdas/try-with. > > I need a time-machine for your CFV/jdk8 Committer: > > http://mail.openjdk.java.net/pipermail/jdk8-dev/2013-August/002896.html > > I vote yes. > >> The test can now fail only if System.gc() does not trigger >> WeakReference processing in the VM. Can you give it a try on your >> NetBeans environment? > > One last comment. It's now 2015. ;) > > Brad > From peter.levart at gmail.com Sun Jan 4 17:58:42 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sun, 04 Jan 2015 18:58:42 +0100 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table with wrong capacity Message-ID: <54A97F52.2010900@gmail.com> Hi, While investigating the following issue: https://bugs.openjdk.java.net/browse/JDK-8029891 I noticed there's a bug in deserialization code of java.util.Hashtable (from day one probably): https://bugs.openjdk.java.net/browse/JDK-8068427 The fix is a trivial one-character replacement: '*' -> '/', but I also corrected some untruthful comments in the neighbourhood (which might have been true from day one, but are not any more): http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ Regards, Peter From xueming.shen at oracle.com Sun Jan 4 19:18:44 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Sun, 04 Jan 2015 11:18:44 -0800 Subject: pre-RFR (s): 8049847: Enhance PrintWriter line.separator handling In-Reply-To: <54A949D8.2030201@oracle.com> References: <54A6BB84.6060508@oracle.com> <54A9108A.2090704@oracle.com> <54A949D8.2030201@oracle.com> Message-ID: <54A99214.7060806@oracle.com> I'm very concerned every time when I see someone tries to achieve something by manipulating the system property, given the lesson we have learned from the past, (property file.encoding, for example). In that particular stackoverflow sample, I would assume the better suggestion is to override the "println", instead of trying to overwrite the system property. I might go further to update the implementation to use the new System.lineSeparator() to disencourage this kind of workaround. I agree with Alan that the general interface "Appendable" probably should not get into the specific "formatting" functionality here. And I also think the "line.separator" should not be in the "Writer" as well, writer/reader does not deal with "formatting", only couple "special" writers need to work with "line" Personally I would not be too concerned about the PrintWriter's format("%n") does not work the same way as PrintWriter.println(), when the "println()" is overrode to write a line separator that is different to the system default (again, it should not be achieved by modifying the system property), given the purpose of defining the "%n" is to provide the functionality of formatting a line separator the same as the underlying platform (otherwise a \n or \r\n probably should be used explicitly ?). If you really want to make such a customized version of PrintWriter to "format" a different line separator for "%n", the workaround is to override the format() to replace the %n. Btw, Formatter is not "tightly" coupled with Appendable, even though we are creating Appendable underneath, a kind of implementation detail. So from specification point of view, it might not be appropriate to specifically specify "%n" to a particular "Appendable". My 2 cents. On 1/4/15 6:10 AM, Claes Redestad wrote: > > On 2015-01-04 11:06, Alan Bateman wrote: >> On 02/01/2015 15:38, Claes Redestad wrote: >>> Hi, >>> >>> this is a proposal to resolve concerns that >>> PrintWriter/BufferedWriter behave >>> inconsistently with j.u.Formatter when setting the line.separator >>> property to override >>> system-appropriate line breaks. >>> >>> Instead of providing a set of new constructors, I propose to simply >>> add a new default >>> method j.l.Appendable#lineSeparator, which by default delegates to >>> System.lineSeparator(). >>> This allows classes such as PrintWriter/BufferedWriter to provide >>> overrides which >>> communicate to j.u.Formatter their intended behavior. >> In the bug report then I assume the inconsistency only arises because >> the test case attempts to change the line separator in the main >> method, which is too late. It should work if set on the command-line >> but this of course sets it system-wide. > > Right, the PrintWriter inconsistency detailed: > - methods format/printf uses j.u.Formatter, which uses > System.lineSeparator() internally if it > encounters an "%n" in the format string. Only responds to when > changing line.separator on > command-line or similar. > - methods println and newLine use the PrintWriter internal > lineSeparator value evaluated at > object creation time: > > public PrintWriter(Writer out, > boolean autoFlush) { > super(out); > this.out = out; > this.autoFlush = autoFlush; > lineSeparator = java.security.AccessController.doPrivileged( > new sun.security.action.GetPropertyAction("line.separator")); > } > > This leads to hacks existing in the wild which sets line.separator > before creating the PrintWriter, > as exemplified here: > http://stackoverflow.com/questions/1014287/is-there-a-way-to-make-printwriter-output-to-unix-format > > Removing this functionality could break compatibility with some > applications where developers > are intentionally changing the line.separator, while it'd be nice if > there was a better way to facilitate > this, which I think my proposal would allow with minimal > >> >> Your proposal to add a lineSeperator() method to Appendable would >> allow for customization but I'm not sure that it's the right place >> (as Appendable is a very general interface and would require every >> Appendable implementation that writes lines to be updated to make use >> of the new method). > > Yes, this can seem an awkward place to add this method, but the only > place where we'd be able > to as easily make java.util.Formatter use the custom lineSeparator, > without which we wouldn't > be able to resolve the inconsistency above nor provide a more > convenient way for developers > to avoid the line.separator hack. > > However, since Appendable is defined rather strictly-coupled with > java.util.Formatter ("The Appendable > interface must be implemented by any class whose instances are > intended to receive formatted > output from a java.util.Formatter.") and Formatter provides a > higher-order mechanism to print new-lines > ("%n") to the Appendable, it doesn't feel entirely arbitrary that the > lineSeparator behavior should be > accessible via Appendable. > >> >> An alternative to consider is adding the notion of lines to >> java.io.Writer and update it to define newLine() and lineSeperator() >> methods (i.e. move the BufferWriter::newLine method up to Writer). >> It's a bit of extra work to update BufferedWriter's spec, adjust a >> few things in PrintWriter/PrintStream, and special-case Writers in >> Formatter but something to consider. > > I need to think about what that would mean, and especially how to > solve it on the Formatter side. > I worry shoehorning Writer or upcasts into Formatter might be more > complex that it's worth. > Any ideas? > > /Claes > >> >> -Alan > From Alan.Bateman at oracle.com Sun Jan 4 19:50:07 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 04 Jan 2015 19:50:07 +0000 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54A82949.4010308@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> Message-ID: <54A9996F.1010109@oracle.com> On 03/01/2015 17:39, Ivan Gerasimov wrote: > > Currently, there are tree variants of ClassLoaderHelper: for Windows, > for Unix and for MacOS. > We have to either duplicate code in Unix and MacOS realizations, or > introduce another Helper class for initializing paths only, which > would have only two realizations: for Windows and all Unixes. When I made the comment then I was thinking of a method such as allowsQuotedPathElements (or a better name) that returns a boolean to indicate if quoting of path elements is allowed or not. That would abstract the capability a bit without needing to do isWindows checks. -Alan. From ivan.gerasimov at oracle.com Sun Jan 4 20:10:47 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Sun, 04 Jan 2015 23:10:47 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54A91E07.1080202@gmail.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A91E07.1080202@gmail.com> Message-ID: <54A99E47.7070507@oracle.com> Hi Peter! > There were other cases where one wanted to consolidate duplicated > code, but couldn't easily because of subtle differences among OS > platforms (UNIXProcess is one of such cases). Perhaps we need > something like this: > > http://cr.openjdk.java.net/~plevart/misc/OS/webrev/ > > With this compile-time mechanism, one can use switch or if statements > in common class to cover subtle differences and only divide code into > OS platform specific classes where the bulk of it differs. I think this mechanism might be useful as a slightly more efficient way of determining the OS, comparing to reading "os.name" property. However, grep found only 8 occurrences of "os.name" usage in src/java.base, so I guess it wouldn't make much difference. In this particular case, it seems to be easier to just introduce another helper class, which has only two realizations: for all Unixes, including MacOS and for Windows. I named it ClassLoaderHelper2, and I'm open for a better name for it. Would you please take a look at the updated webrev? http://cr.openjdk.java.net/~igerasim/8067951/3/webrev/ Sincerely yours, Ivan From ivan.gerasimov at oracle.com Sun Jan 4 20:23:59 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Sun, 04 Jan 2015 23:23:59 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54A9996F.1010109@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> Message-ID: <54A9A15F.9000105@oracle.com> On 04.01.2015 22:50, Alan Bateman wrote: > On 03/01/2015 17:39, Ivan Gerasimov wrote: >> >> Currently, there are tree variants of ClassLoaderHelper: for Windows, >> for Unix and for MacOS. >> We have to either duplicate code in Unix and MacOS realizations, or >> introduce another Helper class for initializing paths only, which >> would have only two realizations: for Windows and all Unixes. > When I made the comment then I was thinking of a method such as > allowsQuotedPathElements (or a better name) that returns a boolean to > indicate if quoting of path elements is allowed or not. That would > abstract the capability a bit without needing to do isWindows checks. > Ah, I see. Though, not needing to check for quotes allows a bit more efficient implementation, so splitting the code for different platforms may also make sense. I did it with another helper class in this webrev: http://cr.openjdk.java.net/~igerasim/8067951/3/webrev/ Sincerely yours, Ivan > -Alan. > > From Alan.Bateman at oracle.com Sun Jan 4 20:25:57 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 04 Jan 2015 20:25:57 +0000 Subject: pre-RFR (s): 8049847: Enhance PrintWriter line.separator handling In-Reply-To: <54A99214.7060806@oracle.com> References: <54A6BB84.6060508@oracle.com> <54A9108A.2090704@oracle.com> <54A949D8.2030201@oracle.com> <54A99214.7060806@oracle.com> Message-ID: <54A9A1D5.7090102@oracle.com> On 04/01/2015 19:18, Xueming Shen wrote: > : > > I agree with Alan that the general interface "Appendable" probably > should not get > into the specific "formatting" functionality here. And I also think > the "line.separator" > should not be in the "Writer" as well, writer/reader does not deal > with "formatting", > only couple "special" writers need to work with "line" I've often wondered about BufferedReader/BufferedWriter supporting lines, it often feels like the readLine and newLine methods should have been further up in the hierarchy in Reader and Writer so the BufferedXXX types just adds buffering. I realize of course that newLine might be very inefficient without buffering. Do have all the history? I'm wondering if there is merit looking at this again as it could be useful beyond the case that Claes wants to address. If Writer did define newLine then then the proposed lineSeperator wouldn't be needed. -Alan From claes.redestad at oracle.com Sun Jan 4 20:29:45 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Sun, 04 Jan 2015 21:29:45 +0100 Subject: pre-RFR (s): 8049847: Enhance PrintWriter line.separator handling In-Reply-To: <54A99214.7060806@oracle.com> References: <54A6BB84.6060508@oracle.com> <54A9108A.2090704@oracle.com> <54A949D8.2030201@oracle.com> <54A99214.7060806@oracle.com> Message-ID: <54A9A2B9.8070902@oracle.com> For the record I would be very happy to update the implementation to use System.lineSeparator() in PrintWriter/BufferedWriter rather than reading line.separator, thus removing the unspecified behavior to be able to hack the current implementation via manipulating system properties. This might break some applications, but, as you imply, any such use case might already be better accommodated by overriding the PrintWriter#println/printf/format methods. How about closing this bug as WNF and filing a new RFE to remove the PrintWriter/BufferedWriter constructor dependencies on line.separator? /Claes On 2015-01-04 20:18, Xueming Shen wrote: > > I'm very concerned every time when I see someone tries to achieve > something by > manipulating the system property, given the lesson we have learned > from the past, > (property file.encoding, for example). In that particular > stackoverflow sample, I > would assume the better suggestion is to override the "println", > instead of trying > to overwrite the system property. I might go further to update the > implementation > to use the new System.lineSeparator() to disencourage this kind of > workaround. > > I agree with Alan that the general interface "Appendable" probably > should not get > into the specific "formatting" functionality here. And I also think > the "line.separator" > should not be in the "Writer" as well, writer/reader does not deal > with "formatting", > only couple "special" writers need to work with "line" > > Personally I would not be too concerned about the PrintWriter's > format("%n") does > not work the same way as PrintWriter.println(), when the "println()" > is overrode to write > a line separator that is different to the system default (again, it > should not be achieved > by modifying the system property), given the purpose of defining the > "%n" is to > provide the functionality of formatting a line separator the same as > the underlying > platform (otherwise a \n or \r\n probably should be used explicitly > ?). If you really > want to make such a customized version of PrintWriter to "format" a > different line > separator for "%n", the workaround is to override the format() to > replace the %n. > > Btw, Formatter is not "tightly" coupled with Appendable, even though > we are creating > Appendable underneath, a kind of implementation detail. So from > specification point > of view, it might not be appropriate to specifically specify "%n" to a > particular "Appendable". > > My 2 cents. > > On 1/4/15 6:10 AM, Claes Redestad wrote: >> >> On 2015-01-04 11:06, Alan Bateman wrote: >>> On 02/01/2015 15:38, Claes Redestad wrote: >>>> Hi, >>>> >>>> this is a proposal to resolve concerns that >>>> PrintWriter/BufferedWriter behave >>>> inconsistently with j.u.Formatter when setting the line.separator >>>> property to override >>>> system-appropriate line breaks. >>>> >>>> Instead of providing a set of new constructors, I propose to >>>> simply add a new default >>>> method j.l.Appendable#lineSeparator, which by default delegates to >>>> System.lineSeparator(). >>>> This allows classes such as PrintWriter/BufferedWriter to provide >>>> overrides which >>>> communicate to j.u.Formatter their intended behavior. >>> In the bug report then I assume the inconsistency only arises >>> because the test case attempts to change the line separator in the >>> main method, which is too late. It should work if set on the >>> command-line but this of course sets it system-wide. >> >> Right, the PrintWriter inconsistency detailed: >> - methods format/printf uses j.u.Formatter, which uses >> System.lineSeparator() internally if it >> encounters an "%n" in the format string. Only responds to when >> changing line.separator on >> command-line or similar. >> - methods println and newLine use the PrintWriter internal >> lineSeparator value evaluated at >> object creation time: >> >> public PrintWriter(Writer out, >> boolean autoFlush) { >> super(out); >> this.out = out; >> this.autoFlush = autoFlush; >> lineSeparator = java.security.AccessController.doPrivileged( >> new >> sun.security.action.GetPropertyAction("line.separator")); >> } >> >> This leads to hacks existing in the wild which sets line.separator >> before creating the PrintWriter, >> as exemplified here: >> http://stackoverflow.com/questions/1014287/is-there-a-way-to-make-printwriter-output-to-unix-format >> >> Removing this functionality could break compatibility with some >> applications where developers >> are intentionally changing the line.separator, while it'd be nice if >> there was a better way to facilitate >> this, which I think my proposal would allow with minimal >> >>> >>> Your proposal to add a lineSeperator() method to Appendable would >>> allow for customization but I'm not sure that it's the right place >>> (as Appendable is a very general interface and would require every >>> Appendable implementation that writes lines to be updated to make >>> use of the new method). >> >> Yes, this can seem an awkward place to add this method, but the only >> place where we'd be able >> to as easily make java.util.Formatter use the custom lineSeparator, >> without which we wouldn't >> be able to resolve the inconsistency above nor provide a more >> convenient way for developers >> to avoid the line.separator hack. >> >> However, since Appendable is defined rather strictly-coupled with >> java.util.Formatter ("The Appendable >> interface must be implemented by any class whose instances are >> intended to receive formatted >> output from a java.util.Formatter.") and Formatter provides a >> higher-order mechanism to print new-lines >> ("%n") to the Appendable, it doesn't feel entirely arbitrary that the >> lineSeparator behavior should be >> accessible via Appendable. >> >>> >>> An alternative to consider is adding the notion of lines to >>> java.io.Writer and update it to define newLine() and lineSeperator() >>> methods (i.e. move the BufferWriter::newLine method up to Writer). >>> It's a bit of extra work to update BufferedWriter's spec, adjust a >>> few things in PrintWriter/PrintStream, and special-case Writers in >>> Formatter but something to consider. >> >> I need to think about what that would mean, and especially how to >> solve it on the Formatter side. >> I worry shoehorning Writer or upcasts into Formatter might be more >> complex that it's worth. >> Any ideas? >> >> /Claes >> >>> >>> -Alan >> > From xueming.shen at oracle.com Sun Jan 4 21:37:22 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Sun, 04 Jan 2015 13:37:22 -0800 Subject: pre-RFR (s): 8049847: Enhance PrintWriter line.separator handling In-Reply-To: <54A9A1D5.7090102@oracle.com> References: <54A6BB84.6060508@oracle.com> <54A9108A.2090704@oracle.com> <54A949D8.2030201@oracle.com> <54A99214.7060806@oracle.com> <54A9A1D5.7090102@oracle.com> Message-ID: <54A9B292.9020504@oracle.com> On 1/4/15 12:25 PM, Alan Bateman wrote: > On 04/01/2015 19:18, Xueming Shen wrote: >> : >> >> I agree with Alan that the general interface "Appendable" probably >> should not get >> into the specific "formatting" functionality here. And I also think >> the "line.separator" >> should not be in the "Writer" as well, writer/reader does not deal >> with "formatting", >> only couple "special" writers need to work with "line" > I've often wondered about BufferedReader/BufferedWriter supporting > lines, it often feels like the readLine and newLine methods should > have been further up in the hierarchy in Reader and Writer so the > BufferedXXX types just adds buffering. I realize of course that > newLine might be very inefficient without buffering. Do have all the > history? I'm wondering if there is merit looking at this again as it > could be useful beyond the case that Claes wants to address. If Writer > did define newLine then then the proposed lineSeperator wouldn't be > needed. > > -Alan I don't have the history. My take on this is that the read/newLine() methods in BufferedReader/Writer are more a pair of utility/convenient/bonus method. They serve parsing and formatting functionality, which probably should belong to the Scanner and Formatter, rather than the io reader and writer. Of course we didn't have scanner and formatter in earlier days:-) And it's convenient to have them in the buffered reader/writer, if the line is only thing you care about on top of "character". When I care about a little more than a "line", I started to use Scanner more often these days. -Sherman From david.holmes at oracle.com Mon Jan 5 00:38:56 2015 From: david.holmes at oracle.com (David Holmes) Date: Mon, 05 Jan 2015 10:38:56 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <1420284243.3416.10.camel@Nokia-N900> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> Message-ID: <54A9DD20.5050802@oracle.com> On 3/01/2015 9:24 PM, Peter Firmstone wrote: > Thanks Brian, > > Those are good questions, some thoughts and examples inline: > > ----- Original message ----- >> Overall the direction seems promising. Poking at it a bit... >> >> - ReadSerial methods are caller-sensitive and only show a class a view >> of its own fields. >> - Invariant checking is separate from deserialization, and does not >> seem entirely built-in -- subclass constructors seem responsible for >> asking parents to do validity-checking? > > My mistake, the last example wasn't quite right, BaseFoo, should call it's own static check method from within it's constructor, rather than from SubFoo. > > Each class in the heirarchy needs to check it's invarients using a static method, before calling a superclass constructor, this prevents object construction if an invarient isn't satisfied, because the object isn't created an attacker cannot hold a reference to it. > > By the time Object's default constructor is called, even though no fields have been set, invarients have already been checked for every member class in the Object. > > An attacker may choose to call another constructor and not pass on ReadSerial, so all constructors need to perform these checks. > > >> - I don't see how this invariant-checking mechanism can enforce >> invariants between superclass fields and subclass fields. For example: >> >> class A { >> int lower, upper; // invariant: lower <= upper >> } >> >> class B extends A { >> int cur; // invariant: lower <= cur <= upper >> } >> >> To check such an invariant, the serialization library would have to >> construct the object (in a potentially bad state), invoke the checker at >> each layer, and then fail deserialization if any checker said no. > > The serialization framework should only construct an objects fields and make these available from ReadSerial, each class then calls a static method before calling a superclass constructor that's responsible for an objects creation, to prevent construction of the object, if necessary. > > Eg, some complexity, but bullet proof: > > public class A ( > > public final int lower, upper; > > private static boolean check(ReadSerial rs){ > if (rs.getInt("lower") > rs.getInt("upper")) > throw new IllegalArgumentException( > "lower bound must be less than or equal to upper"); > return true; > } > > public A(ReadSerial rs){ > this(check(rs), rs); > } > > private A(boolean checked, ReadSerial rs){ > super(); > lower = rs.getInt("lower"); > upper = rs.getInt("upper"); > } > > // other constructors omitted must also check invarients > } > > class B extends A { > > public final int cur; > > private static ReadSerial check(ReadSerial rs) { > A a = new A(rs); It doesn't seem practical in general to have to create an instance of your supertype to validate the passed in serialized form. Why not expose the check method? David ----- > int cur = rs.getInt("cur"); > if ( a.lower > cur || cur > a.upper ) > throw new IllegalArgumentException( > "cur outside lower and upper bounds"); > return rs; > } > > public B(ReadSerial rs) { > super(check(rs)); > cur = rs.getInt("cur"); > } > } > > > > But, >> an evil checker could still squirrel away a reference under the carpet. > > Not with the above example, if an invarient isn't satisfied, an object instance of B cannot be created. > > But circular links are still a challenge... > >> >> Another challenge in invariant checking is circular data structures. If >> you have two objects: >> >> class Brother { >> final Brother brother; >> } >> >> that refer to each other, an invariant you might want to check after >> deserialization is that >> this.brother.brother == this >> >> Obviously you have to patch one or the other instance after construction >> to retain the circular references; at what point do you do invariant >> checking? > > There are two cases here I think: > > One where there is a trust relationship between two classes and the circular relationship is known at compilation time. At least one will provide a mutable setter method, or a Constructor that accepts an argument with a matching argument type, and the other invokes it during construction, I say trusted because 'this' is allowed to escape. > > In Brother's, because the brother field is final, this was constructed with a trust relationship, because 'this' escapes from at least one Brother during construction. I would argue that the object that allows this to escape should also be responsible for rebuilding the circular relation during deserialisation. > > However the Brother that didn't let this escape could be serialised first, in any case both are instances of Brother, so either instance could be a subclass, each can correctly create the other. > > public class Brother { > > private static Constructor getBroConstructor(ReadSerial rs){ > // we can check our subclass has permission here if we like, > // even though an object hasn't been created, its class is > // in the current execution context. > Class type = rs.getType("brother"); > if ( !Brother.class.isAssignableFrom(type)) > throw new Exception("not my bro"); > if ( !rs.samePackageAndClassLoader(Brother.class, type){ > Permission toBeMyBro = new BroPermission(); > // gets type's ProtectionDomain and checks it has > // permission, throws SecurityException > rs.checkPermission(type, toBeMyBro); > } > Class [] paramTypes = {Brother.class}; > // Check if constructor is accessible. > // and throw exception here if not. > return rs.getConstructor("brother", paramTypes); > } > > final Brother brother; > > public Brother(ReadSerial rs){ > this(getBroConstructor(rs)); > } > > private Brother(Constructor c){ > Object [] params = {this}; > brother = c.newInstance(params); > } > } > > Another option might be to provide something like this > > rs.circularFieldNewInstance("brother", this); > > But that encourages bad practise; 'this' to escape, although it would allow 'this' to be passed as one of the filelds in another ReadSerial instance, but probably not a great idea. > > In other cases, there is a circular relationship between untrusted classes and the framework must provide a thread of execution to classes to wire up circular references, after construction. > > Circular field references that can only be written after construction must be mutable, or if final, their referent must be mutable, depending on the depth of the circular relationship. > > To enable wiring up the circular relationship, a class needs to implement a method for the framework to call after construction, a suggestion: > > public interface Circular { > void setMutableFields(ReadSerial rs); > } > > Circular fields missing from ReadSerial during construction are populated after. > > An attacking subclass can override setMutableFields, so the implementing class should make it final, provide and call a protected method for subclasses to implement, should they too contain circular links. > > The alternative is to use a private method, but it's a lot more convenient to use an interface. > > Unfortunately the only remaining option to protect invarients is to make all methods throw an exception if invarients haven't been satisfied after construction, even if this method throws an exception and halts deserialisation, an attacker can hold a reference to it. > > Cheers, > > Peter. > >> >> On 1/1/2015 7:43 AM, Peter Firmstone wrote: >>> Subclass example: >>> >>> class SubFoo extends BaseFoo { >>> >>> public static ReadSerial check(ReadSerial rs){ >>> if (rs.getInt("y") > 128) throw Exception("too big"); >>> return rs; >>> } >>> >>> private final int y; >>> >>> public SubFoo( int x , int y) { >>> super(x); >>> this.y = y; >>> } >>> >>> public SubFoo( ReadSerial rs ){ >>> super(BaseFoo.check(check(rs))); >>> // SubFoo can't get at BaseFoo's rs.getInt("x"), >>> // it's visible only to BaseFoo. Instead SubFoo would get >>> // the default int value of 0. Just in case both classes have >>> // private fields named "x". >>> // ReadSerial is caller sensitive. >>> this.y = rs.getInt("y"); >>> } >>> } >>> >>> Classes in the heirarchy can provide a static method that throws an >>> exception to check invarients while preventing a finaliser attack. We'd >>> want to check invarients for other constructors also, but for >>> berevity... >>> >>> Eg: >>> >>> class BaseFoo implements Serializable{ >>> >>> public static ReadSerial check(ReadSerial rs) throws Exception >>> { >>> if (rs.getInt("x") < 1) >>> throw IllegalArgumentException("message"); >>> return rs; >>> } >>> .... >>> >>> >>> Sent from my Nokia N900. >>> >>> ----- Original message ----- >>> So, if I understand this correctly, the way this would get used is: >>> >>> class BaseFoo implements Serializable { >>> private final int x; >>> >>> public BaseFoo(ReadSerial rs) { >>> this(rs.getInt("x")); >>> } >>> >>> public BaseFoo(int x) { >>> this.x = x; >>> } >>> } >>> >>> Right? >>> >>> What happens with subclasses? I think then I need an extra RS arg in >>> my constructors: >>> >>> class SubFoo extends BaseFoo { >>> private final int y; >>> >>> public SubFoo(ReadSerial rs) { >>> this(rs.getInt("y")); >>> } >>> >>> public BaseFoo(ReadSerial rs, int y) { >>> super(rs); >>> this.y = y; >>> } >>> } >>> >>> Is this what you envision? >>> >>> >>> >>> >>> >>> On 12/27/2014 8:03 PM, Peter Firmstone wrote: >>> Is there any interest in developing an explicit API for >>> Serialization?: >>> >>> 1. Use a public constructor signature with a single argument, >>> ReadSerialParameters (read only, writable only by the >>> serialization framework) to recreate objects, subclasses (when >>> permitted) call this first from their own constructor, they have >>> an identical constructor signature. ReadSerialParameters that are >>> null may contain a circular reference and will be available after >>> construction, see #3 below. >>> 2. Use a factory method (defined by an interface) with one parameter, >>> WriteSerialParameters (write only, readable only by the >>> serialization framework), this method can be overridden by >>> subclasses (when permitted) >>> 3. For circular links, a public method (defined by an interface) that >>> accepts one argument, ReadSerialParameters, this method is called >>> after the constructor completes, subclasses overriding this should >>> call the superclass method. If this method is not called, an >>> implementation, if known to possibly contain circular links, >>> should check it has been fully initialized in each object method >>> called. >>> 4. Retains compatibility with current serialization stream format. >>> 5. Each serial field has a name, calling class and object reference, >>> similar to explicitly declaring "private static final >>> ObjectStreamField[] serialPersistentFields ". >>> >>> Benefits: >>> >>> 1. An object's internal form is not publicised. >>> 2. Each class in an object's heirarchy can use a static method to >>> check invarients and throw an exception, prior to >>> java.lang.Object's constructor being called, preventing >>> construction and avoiding finalizer attacks. >>> 3. Final field friendly. >>> 4. Compatible with existing serial form. >>> 5. Flexible serial form evolution. >>> 6. All methods are public and explicitly defined. >>> 7. All class ProtectionDomain's exist in the current execution >>> context, allowing an object to throw a SecurityException before >>> construction. >>> 8. Less susceptible to deserialization attacks. >>> >>> Problems: >>> >>> 1. Implementations cannot be package private or private. Implicit >>> serialization publicises internal form, any thoughts? >>> >>> Recommendations: >>> >>> 1. Create a security check in the serialization framework for >>> implicit serialization, allowing administrators to reduce their >>> deserialization attack surface. >>> 2. For improved security, disallow classes implementing explicit >>> serialization from having static state and static initializer >>> blocks, only allow static methods, this would require complier and >>> verifier changes. >>> 3. Alternative to #2, allow final static fields, but don't allow >>> static initializer blocks or mutable static fields, similar to >>> interfaces. >>> >>> Penny for your thoughts? >>> >>> Regards, >>> >>> Peter Firmstone. >>> > From martinrb at google.com Mon Jan 5 01:43:26 2015 From: martinrb at google.com (Martin Buchholz) Date: Sun, 4 Jan 2015 17:43:26 -0800 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table with wrong capacity In-Reply-To: <54A97F52.2010900@gmail.com> References: <54A97F52.2010900@gmail.com> Message-ID: Looks good. Not surprising no one ever noticed, since the code is intentionally trying to reduce the number of buckets. I would encapsulate serialization mechanics in a separate serialClone method, as done elsewhere e.g. @SuppressWarnings("unchecked") private static T serialClone(T o) throws Exception { ByteArrayOutputStream bos = new ByteArrayOutputStream(); ObjectOutputStream oos = new ObjectOutputStream(bos); oos.writeObject(o); oos.flush(); oos.close(); ByteArrayInputStream bin = new ByteArrayInputStream(bos.toByteArray()); ObjectInputStream ois = new ObjectInputStream(bin); return (T) ois.readObject(); } On Sun, Jan 4, 2015 at 9:58 AM, Peter Levart wrote: > Hi, > > While investigating the following issue: > > https://bugs.openjdk.java.net/browse/JDK-8029891 > > I noticed there's a bug in deserialization code of java.util.Hashtable > (from day one probably): > > https://bugs.openjdk.java.net/browse/JDK-8068427 > > The fix is a trivial one-character replacement: '*' -> '/', but I also > corrected some untruthful comments in the neighbourhood (which might have > been true from day one, but are not any more): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ > > > Regards, Peter > > From Alan.Bateman at oracle.com Mon Jan 5 09:51:15 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 05 Jan 2015 09:51:15 +0000 Subject: pre-RFR (s): 8049847: Enhance PrintWriter line.separator handling In-Reply-To: <54A9A2B9.8070902@oracle.com> References: <54A6BB84.6060508@oracle.com> <54A9108A.2090704@oracle.com> <54A949D8.2030201@oracle.com> <54A99214.7060806@oracle.com> <54A9A2B9.8070902@oracle.com> Message-ID: <54AA5E93.7020409@oracle.com> On 04/01/2015 20:29, Claes Redestad wrote: > For the record I would be very happy to update the implementation to use > System.lineSeparator() in PrintWriter/BufferedWriter rather than reading > line.separator, thus removing the unspecified behavior to be able to > hack the > current implementation via manipulating system properties. This might > break > some applications, but, as you imply, any such use case might already be > better accommodated by overriding the PrintWriter#println/printf/format > methods. > > How about closing this bug as WNF and filing a new RFE to remove the > PrintWriter/BufferedWriter constructor dependencies on line.separator? > I don't think it make sense to change the line.seperator property in the main method or on the fly. The right thing it is to override the newLine method as Sherman suggested. So if these constructors are changed to use System.lineSeperator() and we document this behavior change in release notes then it seems reasonable for a major release (not for 8uX of course). For printf/format/Formatter then having %n result in a call to newLine() has some appeal. It's doable in Formatter but would require a spec change. -Alan From chris.hegarty at oracle.com Mon Jan 5 10:37:48 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 05 Jan 2015 10:37:48 +0000 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table with wrong capacity In-Reply-To: <54A97F52.2010900@gmail.com> References: <54A97F52.2010900@gmail.com> Message-ID: <54AA697C.3080006@oracle.com> This looks fine to me Peter. Thanks. -Chris. On 04/01/15 17:58, Peter Levart wrote: > Hi, > > While investigating the following issue: > > https://bugs.openjdk.java.net/browse/JDK-8029891 > > I noticed there's a bug in deserialization code of java.util.Hashtable > (from day one probably): > > https://bugs.openjdk.java.net/browse/JDK-8068427 > > The fix is a trivial one-character replacement: '*' -> '/', but I also > corrected some untruthful comments in the neighbourhood (which might > have been true from day one, but are not any more): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ > > > Regards, Peter > From peter.firmstone at zeus.net.au Mon Jan 5 12:01:51 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Mon, 05 Jan 2015 22:01:51 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54AA7C3E.6010703@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> Message-ID: <54AA7D2F.1000703@zeus.net.au> My mistake, thank you. Peter. On 5/01/2015 9:57 PM, David Holmes wrote: > Hi Peter, > > Did you send this to the list? I haven't seen it turn up yet. > > David > > On 5/01/2015 6:51 PM, Peter Firmstone wrote: >> Thanks Dave, >> >> That's another way of checking invariants, you could expose A's check >> method but you'd also need a couple of additional static methods to >> obtain integers upper and lower from ReadSerial, so that B can ensure >> its invariant. >> >> ReadSerial is caller sensitive so B can't obtain A's stream values via >> ReadSerial and must do so via A's API. This prevents the publication of >> A's implementation, you don't wan't B depending on A's serial form. >> Instead A can reorder and rename it's fields and change their values, >> have multiple serial forms and be able to remain compatible with all >> forms relatively easily. >> >> There are some benefits to using a temporary object instance of A; it >> reduces the amount of code required, eg it would be a beneficial for RMI >> to minimise code downloads, the JVM is free to inline access to these >> fields and the temporary instance of A never leaves the cpu cache, so >> it's not likely to become a performance issue. Well it might on real >> time java :) >> >> If circular relationships are mutable, or effectively mutable after >> publication, then we could eventually deprecate the requirement to >> support special treatment of final fields for Serializable. >> >> Cheers, >> >> Peter. >> >> On 5/01/2015 10:38 AM, David Holmes wrote: >>> >>>>> - I don't see how this invariant-checking mechanism can enforce >>>>> invariants between superclass fields and subclass fields. For >>>>> example: >>>>> >>>>> class A { >>>>> int lower, upper; // invariant: lower <= upper >>>>> } >>>>> >>>>> class B extends A { >>>>> int cur; // invariant: lower <= cur <= upper >>>>> } >>>>> >>>> >>>> The serialization framework should only construct an objects fields >>>> and make these available from ReadSerial, each class then calls a >>>> static method before calling a superclass constructor that's >>>> responsible for an objects creation, to prevent construction of the >>>> object, if necessary. >>>> >>>> Eg, some complexity, but bullet proof: >>>> >>>> public class A ( >>>> >>>> public final int lower, upper; >>>> >>>> private static boolean check(ReadSerial rs){ >>>> if (rs.getInt("lower") > rs.getInt("upper")) >>>> throw new IllegalArgumentException( >>>> "lower bound must be less than or equal to upper"); >>>> return true; >>>> } >>>> >>>> public A(ReadSerial rs){ >>>> this(check(rs), rs); >>>> } >>>> >>>> private A(boolean checked, ReadSerial rs){ >>>> super(); >>>> lower = rs.getInt("lower"); >>>> upper = rs.getInt("upper"); >>>> } >>>> >>>> // other constructors omitted must also check invarients >>>> } >>>> >>>> class B extends A { >>>> >>>> public final int cur; >>>> >>>> private static ReadSerial check(ReadSerial rs) { >>>> A a = new A(rs); >>> >>> >>> It doesn't seem practical in general to have to create an instance of >>> your supertype to validate the passed in serialized form. Why not >>> expose the check method? >>> >>> David >>> ----- >>> >>>> int cur = rs.getInt("cur"); >>>> if ( a.lower > cur || cur > a.upper ) >>>> throw new IllegalArgumentException( >>>> "cur outside lower and upper bounds"); >>>> return rs; >>>> } >>>> >>>> public B(ReadSerial rs) { >>>> super(check(rs)); >>>> cur = rs.getInt("cur"); >>>> } >>>> } >>>> >>>> >> From peter.levart at gmail.com Mon Jan 5 14:11:30 2015 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 05 Jan 2015 15:11:30 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <54AA7D2F.1000703@zeus.net.au> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> Message-ID: <54AA9B92.3050705@gmail.com> Hi Peter and others, A assume your approach described here (to use constructors for deserialization) is motivated mainly by the fact that only constructors are allowed to set the final fields. Otherwise the explicit features of your ReadSerial API are more or less accessible even now by using "serialPersistentFields " static field for declaration of "serialized" fields, combined with ObjectOutputStream.PutField and ObjectInputStream.GetField API, accessible from writeObject/readObject methods. readObject() method already acts like a constructor, whith the following notable differences: 1 - it is not chained explicitly to superclass (either in source or automatically by javac) like constructor, but is invoked by (de)serialization infrastructure in correct order for each class in the hierarchy that declares it. 2 - it is a normal method, so it can be called at any time by code in the declaring class or inner classes. 3 - it can't set final fields (without using clumsy reflection) The 1st point is actually in favor of readObject() method, since the (de)serialization API need not be caller-sensitive. The correct per-class context is established before each readObject call-back. 2nd and 3rd points are readObject drawbacks and are both interconnected. If readObject was not universally callable, then the restriction that it can't set final fields could be lifted. Perhaps it is too late from compatibility perspective to change the readObject's 2nd and 3rd point now, but we could compatibly change the 1st point of a special kind of constructor. Say that a constructor declared with exactly one parameter of type ObjectInputStream is treated specially by javac and reflection: - constructor code can't chain to superclass constructor - javac does not automatically generate default constructor chaining code - javac does not allow calling such constructor from any code - this is optional: javac does not allow calling instance methods and explicit use of 'this' keyword in deserialization constructor code (preventing escape of such object before de-serialization ends) - to be really safe, this would probably have to be accompanied with verifier checks too. - public reflection API does not allow invoking such constructor (only deserialization infrastructure can - like it can invoke the default constructor of a non-Serializable superclass on an already allocated instance of a sublclass) No additional changes to VM are apparently necessary (except verifier). Declaring such constructor as an alternative to readObject() method is then possible to make deserialization more safe. Regarding finalizability (and finalizer attacks): The serialization specifies that for 1st (most specific) non-Serializable superclass of a Serializable subclass, default constructor is invoked to initialize the non-Serializable superclass state before deserializing subclass state. As each Serializable class has such a non-Serializable superclass (at least in the form of Object class) and normal constructors are chained, it is apparent that Object's default constructor is called before any deserialization of state begins. As soon as Object's constructor completes, the instance becomes eligible for finalization: "An object o is not finalizable until its constructor has invoked the constructor for Object on o and that invocation has completed successfully (that is, without throwing an exception).? The rule of calling non-Serializable superclass default constructor could be weakened. If 1st (most specific) non-Serializable superclass of a Serializable subclass is Object, then it's constructor is not called. Obviously it is not needed since Object has no state to initialize. What we achieve by this is that such object does not become eligible for finalization just yet. Serialization infrastructure must then explicitly register such objects for finalization, but only after their de-serialization completes normally. This would prevent finalization attacks. Regards, Peter On 01/05/2015 01:01 PM, Peter Firmstone wrote: > My mistake, thank you. > > Peter. > > On 5/01/2015 9:57 PM, David Holmes wrote: >> Hi Peter, >> >> Did you send this to the list? I haven't seen it turn up yet. >> >> David >> >> On 5/01/2015 6:51 PM, Peter Firmstone wrote: >>> Thanks Dave, >>> >>> That's another way of checking invariants, you could expose A's check >>> method but you'd also need a couple of additional static methods to >>> obtain integers upper and lower from ReadSerial, so that B can ensure >>> its invariant. >>> >>> ReadSerial is caller sensitive so B can't obtain A's stream values via >>> ReadSerial and must do so via A's API. This prevents the >>> publication of >>> A's implementation, you don't wan't B depending on A's serial form. >>> Instead A can reorder and rename it's fields and change their values, >>> have multiple serial forms and be able to remain compatible with all >>> forms relatively easily. >>> >>> There are some benefits to using a temporary object instance of A; it >>> reduces the amount of code required, eg it would be a beneficial for >>> RMI >>> to minimise code downloads, the JVM is free to inline access to these >>> fields and the temporary instance of A never leaves the cpu cache, so >>> it's not likely to become a performance issue. Well it might on real >>> time java :) >>> >>> If circular relationships are mutable, or effectively mutable after >>> publication, then we could eventually deprecate the requirement to >>> support special treatment of final fields for Serializable. >>> >>> Cheers, >>> >>> Peter. >>> >>> On 5/01/2015 10:38 AM, David Holmes wrote: >>>> >>>>>> - I don't see how this invariant-checking mechanism can >>>>>> enforce >>>>>> invariants between superclass fields and subclass fields. For >>>>>> example: >>>>>> >>>>>> class A { >>>>>> int lower, upper; // invariant: lower <= upper >>>>>> } >>>>>> >>>>>> class B extends A { >>>>>> int cur; // invariant: lower <= cur <= upper >>>>>> } >>>>>> >>>>> >>>>> The serialization framework should only construct an objects fields >>>>> and make these available from ReadSerial, each class then calls a >>>>> static method before calling a superclass constructor that's >>>>> responsible for an objects creation, to prevent construction of the >>>>> object, if necessary. >>>>> >>>>> Eg, some complexity, but bullet proof: >>>>> >>>>> public class A ( >>>>> >>>>> public final int lower, upper; >>>>> >>>>> private static boolean check(ReadSerial rs){ >>>>> if (rs.getInt("lower") > rs.getInt("upper")) >>>>> throw new IllegalArgumentException( >>>>> "lower bound must be less than or equal to upper"); >>>>> return true; >>>>> } >>>>> >>>>> public A(ReadSerial rs){ >>>>> this(check(rs), rs); >>>>> } >>>>> >>>>> private A(boolean checked, ReadSerial rs){ >>>>> super(); >>>>> lower = rs.getInt("lower"); >>>>> upper = rs.getInt("upper"); >>>>> } >>>>> >>>>> // other constructors omitted must also check invarients >>>>> } >>>>> >>>>> class B extends A { >>>>> >>>>> public final int cur; >>>>> >>>>> private static ReadSerial check(ReadSerial rs) { >>>>> A a = new A(rs); >>>> >>>> >>>> It doesn't seem practical in general to have to create an instance of >>>> your supertype to validate the passed in serialized form. Why not >>>> expose the check method? >>>> >>>> David >>>> ----- >>>> >>>>> int cur = rs.getInt("cur"); >>>>> if ( a.lower > cur || cur > a.upper ) >>>>> throw new IllegalArgumentException( >>>>> "cur outside lower and upper bounds"); >>>>> return rs; >>>>> } >>>>> >>>>> public B(ReadSerial rs) { >>>>> super(check(rs)); >>>>> cur = rs.getInt("cur"); >>>>> } >>>>> } >>>>> >>>>> >>> > From david.lloyd at redhat.com Mon Jan 5 14:17:57 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 05 Jan 2015 08:17:57 -0600 Subject: Explicit Serialization API and Security In-Reply-To: <54A80C72.3030906@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> <54A80C72.3030906@gmail.com> Message-ID: <54AA9D15.701@redhat.com> On 01/03/2015 09:36 AM, Peter Levart wrote: > > On 01/03/2015 01:38 PM, Peter Firmstone wrote: >> >> > Hi, >> > >> > I would like to know what are the potential issues with simple >> > constructor chaining where each constructor checks the invariant of its >> > class state (with the already initialized state of superclass(es)). >> >> Finalizer attack; a subclass can override the finalize method and >> receive a thread of execution, even when it hasn't gotten hold of a >> reference during construction. It's best to prevent an object's >> construction, by throwing any exceptions before Object's default >> constructor is called. >> > > As Brian points out, this scheme can only validate intra-class > invariants. It can't validate class-against-subclass state. And, the > finalize() method will be called anyway - although this time with fully > uninitialized instance (all fields default values). If this is safer, > then alright. > > Is there anything in JLS that guarantees finalization for instances > which fail construction or deserialization? Wouldn't it be better to > "register" for finalization just those instances that complete their > construction or deserialization normally? I'm just trying to understand > why it is the way it is. > > Would something like this prevent Finalizer attacks? > > - leave finalization registration the way it is (at object allocation > time). > - provide internal API with which a previously registered object can be > de-registered > - deserialization infrastructure de-registers the instances that fail > deserialization How about simply forbidding classes with finalizers from being serialized or deserialized with this mechanism? Finalizers never really work the way you want anyway. Seems a better option than essentially doubling (or more) the end-user complexity to me. -- - DML From david.lloyd at redhat.com Mon Jan 5 14:23:29 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 05 Jan 2015 08:23:29 -0600 Subject: Explicit Serialization API and Security In-Reply-To: <1420316965.3890.10.camel@Nokia-N900> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> <54A80C72.3030906@gmail.com> <1420316965.3890.10.camel@Nokia-N900> Message-ID: <54AA9E61.8090600@redhat.com> On 01/03/2015 02:29 PM, Peter Firmstone wrote: > ----- Original message ----- >> Wouldn't it be better to "register" for finalization just those instances that complete their >> construction or deserialization normally? I'm just trying to understand >> why it is the way it is. > > Perhaps that might be an option, someone who knows more about finalization might be able to help here. > > In the early days, the sandbox and bytecode verifier were intended to make java secure, additional private methods were created as vulnerabilities were better understood. > > I think a problem with Serialization is you have to establish trust before you can use it. > > It would be nice if there was an input validator like html servers use, to validate the stream before instantiating objects. Eg array size check before array creation, type check before object instantiation and restrict creation to permitted classes, to a subset of what's available on the class path. You can do this already, albeit to a more limited extent, by customizing class resolution in ObjectInputStream. Other approaches include using readResolve/writeReplace to create serialized representation objects, and the existing validation scheme where the validator is deferred until the deserialize operation is complete. -- - DML From daniel.fuchs at oracle.com Mon Jan 5 14:52:55 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 05 Jan 2015 15:52:55 +0100 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table with wrong capacity In-Reply-To: <54A97F52.2010900@gmail.com> References: <54A97F52.2010900@gmail.com> Message-ID: <54AAA547.8070804@oracle.com> On 04/01/15 18:58, Peter Levart wrote: > Hi, > > While investigating the following issue: > > https://bugs.openjdk.java.net/browse/JDK-8029891 > > I noticed there's a bug in deserialization code of java.util.Hashtable > (from day one probably): > > https://bugs.openjdk.java.net/browse/JDK-8068427 > > The fix is a trivial one-character replacement: '*' -> '/', but I also > corrected some untruthful comments in the neighbourhood (which might > have been true from day one, but are not any more): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ Hi Peter, I wonder whether there should be a guard against loadFactor being zero/negative/NaN after line 1173, like in the constructor e.g. as in lines 188 if (loadFactor <= 0 || Float.isNaN(loadFactor)) 189 throw new IllegalArgumentException("Illegal Load: "+loadFactor); if only to avoid division by zero. best regards, -- daniel > > > Regards, Peter > From paul.sandoz at oracle.com Mon Jan 5 14:57:29 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 5 Jan 2015 15:57:29 +0100 Subject: Stream optimization suggestions In-Reply-To: References: Message-ID: <06DF9F40-BE17-4F91-A225-08FD746FA1D7@oracle.com> On Dec 19, 2014, at 10:19 PM, Dan Smith wrote: > I've submitted a couple of RFEs to support some simple optimizations on Streams. I noticed both issues while writing some code that forced me to choose between cluttering up my straightforward, uniform Stream logic and eating the cost of linear operations that should be O(1). > Thanks. > JDK-8067969 Optimize Stream.count for SIZED Streams That is a reasonable one and i left a comment. > JDK-8067971 Support optimizations for always-true and always-false Predicates > Less convinced about this one. For streams this is a micro-optimization dependent on the identity of a Predicate instance or a method on Predicate declaring it's "trueness" for all values passed to the test method. It's rather fragile and implementation dependent. There are other ways to reorganize the stream pipeline to avoid a filter operation if in certain cases it is not necessary. Given that the expression x -> true is rather concise it is arguable that specific static methods do not carry their weight. Two counter arguments do add some weight: - explicit naming e.g. Predicate.alwaysTrue. It may be easier to read and harder to get wrong. - optimization. A static method can return a singleton instance. Currently the lambda meta factory does not optimise for such cases but i hope this is possible in the future. Stil I don't think these really tip the balance in favour of adding them to the JDK. Paul. > Certainly not high-priority, but would be nice for somebody working with java.util.stream to address someday. > > ?Dan From peter.levart at gmail.com Mon Jan 5 15:01:54 2015 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 05 Jan 2015 16:01:54 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <54AA9D15.701@redhat.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <54A7BDCE.7020308@gmail.com> <1420288688.3416.16.camel@Nokia-N900> <54A80C72.3030906@gmail.com> <54AA9D15.701@redhat.com> Message-ID: <54AAA762.7060607@gmail.com> On 01/05/2015 03:17 PM, David M. Lloyd wrote: >> Would something like this prevent Finalizer attacks? >> >> - leave finalization registration the way it is (at object allocation >> time). This was written incorrectly: "after Object default constructor completes" >> - provide internal API with which a previously registered object can be >> de-registered >> - deserialization infrastructure de-registers the instances that fail >> deserialization > > How about simply forbidding classes with finalizers from being > serialized or deserialized with this mechanism? Finalizers never > really work the way you want anyway. > > Seems a better option than essentially doubling (or more) the end-user > complexity to me. This is invisible to end-user. Just internal mechanics. I thought about this for some more, which I explained in a followup post. Regards, peter From Roger.Riggs at Oracle.com Mon Jan 5 15:25:16 2015 From: Roger.Riggs at Oracle.com (Roger) Date: Mon, 05 Jan 2015 10:25:16 -0500 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54A91E07.1080202@gmail.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A91E07.1080202@gmail.com> Message-ID: <54AAACDC.2010201@Oracle.com> Hi Ivan, An enum for the OS is useful but as proposed is spread over too many files and into the build system too. The construct is not really a compile time constant since it is initialized as part of a static initializer. The initialization could just as easily be done in OS.java by checking the system property. The sun.misc.OS class can be self contained. There should probably also be an isWindows() method. As for switch, there are hidden costs associated with using switch on enum, including an extra generated class for every class that uses the enum and static initialization. Roger On 1/4/2015 6:03 AM, Peter Levart wrote: > > On 01/03/2015 06:39 PM, Ivan Gerasimov wrote: >> ... > Hi Ivan, > > There were other cases where one wanted to consolidate duplicated > code, but couldn't easily because of subtle differences among OS > platforms (UNIXProcess is one of such cases). Perhaps we need > something like this: > > http://cr.openjdk.java.net/~plevart/misc/OS/webrev/ > > With this compile-time mechanism, one can use switch or if statements > in common class to cover subtle differences and only divide code into > OS platform specific classes where the bulk of it differs. > > What do you think? > > Regards, Peter > >> >> Sincerely yours, >> Ivan >> > From claes.redestad at oracle.com Mon Jan 5 15:26:12 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Mon, 05 Jan 2015 16:26:12 +0100 Subject: pre-RFR (s): 8049847: Enhance PrintWriter line.separator handling In-Reply-To: <54AA5E93.7020409@oracle.com> References: <54A6BB84.6060508@oracle.com> <54A9108A.2090704@oracle.com> <54A949D8.2030201@oracle.com> <54A99214.7060806@oracle.com> <54A9A2B9.8070902@oracle.com> <54AA5E93.7020409@oracle.com> Message-ID: <54AAAD14.2010203@oracle.com> On 2015-01-05 10:51, Alan Bateman wrote: > On 04/01/2015 20:29, Claes Redestad wrote: >> For the record I would be very happy to update the implementation to use >> System.lineSeparator() in PrintWriter/BufferedWriter rather than reading >> line.separator, thus removing the unspecified behavior to be able to >> hack the >> current implementation via manipulating system properties. This might >> break >> some applications, but, as you imply, any such use case might already be >> better accommodated by overriding the PrintWriter#println/printf/format >> methods. >> >> How about closing this bug as WNF and filing a new RFE to remove the >> PrintWriter/BufferedWriter constructor dependencies on line.separator? >> > I don't think it make sense to change the line.seperator property in > the main method or on the fly. The right thing it is to override the > newLine method as Sherman suggested. So if these constructors are > changed to use System.lineSeperator() and we document this behavior > change in release notes then it seems reasonable for a major release > (not for 8uX of course). Sounds good to me. > > For printf/format/Formatter then having %n result in a call to > newLine() has some appeal. It's doable in Formatter but would require > a spec change. If we do this (do we have to?), where do we expose the newLine? Currently Appendable is the only thing Formatter knows about, but I guess having it there has been declared distasteful. :-) Having it in Writer and doing some checks and casts could work, I guess, just need to be careful not to impact performance much for non-Writer cases. /Claes > > -Alan > From Roger.Riggs at Oracle.com Mon Jan 5 15:40:00 2015 From: Roger.Riggs at Oracle.com (Roger) Date: Mon, 05 Jan 2015 10:40:00 -0500 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54A9A15F.9000105@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> Message-ID: <54AAB050.4010001@Oracle.com> Hi Ivan, For this small difference in the implementations, I'd recommend against having two different source files. The path initialization function is a one time function and the performance improvement is not significant. I'd suggest a few comments on your 2nd version[1]. - The windows check should check the system property or other definitive os check and could be better expressed (as Alan suggested) as quotesAllowed. - in the loop testing for the quote (") can come before the quotesAllowed check to speed things up (no need to check if they are allowed if they do not occur). This code is unlikely to be executed enough times to optimized. Roger [1] http://cr.openjdk.java.net/~igerasim/8067951/2/webrev/ On 1/4/2015 3:23 PM, Ivan Gerasimov wrote: > > On 04.01.2015 22:50, Alan Bateman wrote: >> On 03/01/2015 17:39, Ivan Gerasimov wrote: >>> >>> Currently, there are tree variants of ClassLoaderHelper: for >>> Windows, for Unix and for MacOS. >>> We have to either duplicate code in Unix and MacOS realizations, or >>> introduce another Helper class for initializing paths only, which >>> would have only two realizations: for Windows and all Unixes. >> When I made the comment then I was thinking of a method such as >> allowsQuotedPathElements (or a better name) that returns a boolean to >> indicate if quoting of path elements is allowed or not. That would >> abstract the capability a bit without needing to do isWindows checks. >> > Ah, I see. > Though, not needing to check for quotes allows a bit more efficient > implementation, so splitting the code for different platforms may also > make sense. > > I did it with another helper class in this webrev: > http://cr.openjdk.java.net/~igerasim/8067951/3/webrev/ > > Sincerely yours, > Ivan > > >> -Alan. >> >> > From Alan.Bateman at oracle.com Mon Jan 5 15:48:19 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 05 Jan 2015 15:48:19 +0000 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54A9A15F.9000105@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> Message-ID: <54AAB243.102@oracle.com> On 04/01/2015 20:23, Ivan Gerasimov wrote: > > Ah, I see. > Though, not needing to check for quotes allows a bit more efficient > implementation, so splitting the code for different platforms may also > make sense. > > I did it with another helper class in this webrev: > http://cr.openjdk.java.net/~igerasim/8067951/3/webrev/ I think we should avoid introducing yet-another new class if possible as we already load too many classes loading at startup. I see your point about using the most efficient implementation but I think you will get that with "if (ClassLoaderHelper allowsQuotedPathElements()) { .. } else { .. }". -Alan. From peter.levart at gmail.com Mon Jan 5 15:48:31 2015 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 05 Jan 2015 16:48:31 +0100 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table with wrong capacity In-Reply-To: <54AAA547.8070804@oracle.com> References: <54A97F52.2010900@gmail.com> <54AAA547.8070804@oracle.com> Message-ID: <54AAB24F.6000008@gmail.com> On 01/05/2015 03:52 PM, Daniel Fuchs wrote: > On 04/01/15 18:58, Peter Levart wrote: >> Hi, >> >> While investigating the following issue: >> >> https://bugs.openjdk.java.net/browse/JDK-8029891 >> >> I noticed there's a bug in deserialization code of java.util.Hashtable >> (from day one probably): >> >> https://bugs.openjdk.java.net/browse/JDK-8068427 >> >> The fix is a trivial one-character replacement: '*' -> '/', but I also >> corrected some untruthful comments in the neighbourhood (which might >> have been true from day one, but are not any more): >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ >> > > Hi Peter, > > I wonder whether there should be a guard against loadFactor being > zero/negative/NaN after line 1173, like in the constructor e.g. as > in lines > > 188 if (loadFactor <= 0 || Float.isNaN(loadFactor)) > 189 throw new IllegalArgumentException("Illegal Load: > "+loadFactor); > > if only to avoid division by zero. > > best regards, > > -- daniel Hi Daniel, This is only possible with forged stream, right? We can extend this bug and make Hashtable more resilient to forged streams too. Another check would be to limit the origlength to be no less than (int)(elements / loadFactor). Can you spot any other check for an invariant that would bail out on forged streams? Regards, Peter > > >> >> >> Regards, Peter >> > From Alan.Bateman at oracle.com Mon Jan 5 15:55:00 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 05 Jan 2015 15:55:00 +0000 Subject: pre-RFR (s): 8049847: Enhance PrintWriter line.separator handling In-Reply-To: <54AAAD14.2010203@oracle.com> References: <54A6BB84.6060508@oracle.com> <54A9108A.2090704@oracle.com> <54A949D8.2030201@oracle.com> <54A99214.7060806@oracle.com> <54A9A2B9.8070902@oracle.com> <54AA5E93.7020409@oracle.com> <54AAAD14.2010203@oracle.com> Message-ID: <54AAB3D4.6090606@oracle.com> On 05/01/2015 15:26, Claes Redestad wrote: > > : > >> >> For printf/format/Formatter then having %n result in a call to >> newLine() has some appeal. It's doable in Formatter but would require >> a spec change. > > If we do this (do we have to?), where do we expose the newLine? > Currently Appendable is the only thing Formatter knows about, but I > guess having it there has been declared distasteful. :-) > > Having it in Writer and doing some checks and casts could work, I > guess, just need to be careful not to impact performance much for > non-Writer cases. We don't have to do this, I was just pointing out that it has some appeal. Formatter does have a constructor that takes a PrintStream for example. -Alan From sundararajan.athijegannathan at oracle.com Mon Jan 5 16:10:56 2015 From: sundararajan.athijegannathan at oracle.com (A. Sundararajan) Date: Mon, 05 Jan 2015 21:40:56 +0530 Subject: RFR 8068279: (typo in the spec) javax.script.ScriptEngineFactory.getLanguageName Message-ID: <54AAB790.4080306@oracle.com> Please review a typo in javadoc comment.. http://cr.openjdk.java.net/~sundar/8068279/ for https://bugs.openjdk.java.net/browse/JDK-8068279 Thanks, -Sundar From james.laskey at oracle.com Mon Jan 5 16:20:24 2015 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Mon, 5 Jan 2015 12:20:24 -0400 Subject: RFR 8068279: (typo in the spec) javax.script.ScriptEngineFactory.getLanguageName In-Reply-To: <54AAB993.6080103@oracle.com> References: <54AAB790.4080306@oracle.com> <54AAB993.6080103@oracle.com> Message-ID: +1 On Jan 5, 2015, at 12:19 PM, A. Sundararajan wrote: > > > > -------- Forwarded Message -------- > Subject: RFR 8068279: (typo in the spec) javax.script.ScriptEngineFactory.getLanguageName > Date: Mon, 05 Jan 2015 21:40:56 +0530 > From: A. Sundararajan > To: core-libs-dev at openjdk.java.net > > Please review a typo in javadoc comment.. > > http://cr.openjdk.java.net/~sundar/8068279/ for > https://bugs.openjdk.java.net/browse/JDK-8068279 > > Thanks, > -Sundar > > From Alan.Bateman at oracle.com Mon Jan 5 16:21:38 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 05 Jan 2015 16:21:38 +0000 Subject: RFR 8068279: (typo in the spec) javax.script.ScriptEngineFactory.getLanguageName In-Reply-To: <54AAB790.4080306@oracle.com> References: <54AAB790.4080306@oracle.com> Message-ID: <54AABA12.9000502@oracle.com> On 05/01/2015 16:10, A. Sundararajan wrote: > Please review a typo in javadoc comment.. > > http://cr.openjdk.java.net/~sundar/8068279/ for > https://bugs.openjdk.java.net/browse/JDK-8068279 Looks okay to me. -Alan From openjdk at duigou.org Mon Jan 5 16:52:24 2015 From: openjdk at duigou.org (Mike Duigou) Date: Mon, 05 Jan 2015 08:52:24 -0800 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes =?UTF-8?Q?table=09with=20wrong=20capacity?= In-Reply-To: References: Message-ID: <7b2072fe0674e465f86518a9f25cef6d@sonic.net> Well spotted Peter. The change looks good though I wonder if it should be: int length = (int)((elements + elements / 20) / loadFactor) + 3; FYI, regarding Daniel's suggestion: When similar invariant checks were added to the HashMap deserialization method we found code which relied upon the illegal values. In some cases the serialized HashMaps were created by alternative serialization implementations which included illegal, but until the checks were added, "harmless" values. The invariant checks should still be added though. It might even be worth adding checks that the other deserialized values are in valid ranges. Mike On 2015-01-05 07:48, core-libs-dev-request at openjdk.java.net wrote: > > Today's Topics: > > 2. Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes > table with wrong capacity (Daniel Fuchs) > Message: 2 > Date: Mon, 05 Jan 2015 15:52:55 +0100 > From: Daniel Fuchs > To: Peter Levart , core-libs-dev > > Subject: Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes > table with wrong capacity > Message-ID: <54AAA547.8070804 at oracle.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > On 04/01/15 18:58, Peter Levart wrote: >> Hi, >> >> While investigating the following issue: >> >> https://bugs.openjdk.java.net/browse/JDK-8029891 >> >> I noticed there's a bug in deserialization code of java.util.Hashtable >> (from day one probably): >> >> https://bugs.openjdk.java.net/browse/JDK-8068427 >> >> The fix is a trivial one-character replacement: '*' -> '/', but I also >> corrected some untruthful comments in the neighbourhood (which might >> have been true from day one, but are not any more): >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ > > Hi Peter, > > I wonder whether there should be a guard against loadFactor being > zero/negative/NaN after line 1173, like in the constructor e.g. as > in lines > > 188 if (loadFactor <= 0 || Float.isNaN(loadFactor)) > 189 throw new IllegalArgumentException("Illegal Load: > "+loadFactor); > > if only to avoid division by zero. > > best regards, > > -- daniel > > >> >> >> Regards, Peter >> > From ivan.gerasimov at oracle.com Mon Jan 5 17:41:21 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Mon, 05 Jan 2015 20:41:21 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AAB243.102@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB243.102@oracle.com> Message-ID: <54AACCC1.6030602@oracle.com> On 05.01.2015 18:48, Alan Bateman wrote: > On 04/01/2015 20:23, Ivan Gerasimov wrote: >> >> Ah, I see. >> Though, not needing to check for quotes allows a bit more efficient >> implementation, so splitting the code for different platforms may >> also make sense. >> >> I did it with another helper class in this webrev: >> http://cr.openjdk.java.net/~igerasim/8067951/3/webrev/ > I think we should avoid introducing yet-another new class if possible > as we already load too many classes loading at startup. I see your > point about using the most efficient implementation but I think you > will get that with "if (ClassLoaderHelper allowsQuotedPathElements()) > { .. } else { .. }". > Alright. Here's another update: http://cr.openjdk.java.net/~igerasim/8067951/4/webrev/ The optimized loop is used both on Unix and on Windows, if there were no quotes found in PATH. Not a big deal, of course, but it should save a few nano-seconds during startup. Sincerely yours, Ivan From ivan.gerasimov at oracle.com Mon Jan 5 17:49:08 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Mon, 05 Jan 2015 20:49:08 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AAB050.4010001@Oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB050.4010001@Oracle.com> Message-ID: <54AACE94.2000802@oracle.com> Hi Roger! I've updated the webrev similarly to what you've suggested: 1) Presence of quotes is checked when counting the separators. 2) If quotes weren't found, we'll execute the same optimized loop as for Unix. http://cr.openjdk.java.net/~igerasim/8067951/4/webrev/ Sincerely yours, Ivan On 05.01.2015 18:40, Roger wrote: > Hi Ivan, > > For this small difference in the implementations, I'd recommend > against having > two different source files. The path initialization function is a one > time function and > the performance improvement is not significant. > > I'd suggest a few comments on your 2nd version[1]. > > - The windows check should check the system property or other > definitive os check > and could be better expressed (as Alan suggested) as quotesAllowed. > - in the loop testing for the quote (") can come before the > quotesAllowed check to > speed things up (no need to check if they are allowed if they do not > occur). > This code is unlikely to be executed enough times to optimized. > > Roger > > [1] http://cr.openjdk.java.net/~igerasim/8067951/2/webrev/ > > On 1/4/2015 3:23 PM, Ivan Gerasimov wrote: >> >> On 04.01.2015 22:50, Alan Bateman wrote: >>> On 03/01/2015 17:39, Ivan Gerasimov wrote: >>>> >>>> Currently, there are tree variants of ClassLoaderHelper: for >>>> Windows, for Unix and for MacOS. >>>> We have to either duplicate code in Unix and MacOS realizations, or >>>> introduce another Helper class for initializing paths only, which >>>> would have only two realizations: for Windows and all Unixes. >>> When I made the comment then I was thinking of a method such as >>> allowsQuotedPathElements (or a better name) that returns a boolean >>> to indicate if quoting of path elements is allowed or not. That >>> would abstract the capability a bit without needing to do isWindows >>> checks. >>> >> Ah, I see. >> Though, not needing to check for quotes allows a bit more efficient >> implementation, so splitting the code for different platforms may >> also make sense. >> >> I did it with another helper class in this webrev: >> http://cr.openjdk.java.net/~igerasim/8067951/3/webrev/ >> >> Sincerely yours, >> Ivan >> >> >>> -Alan. >>> >>> >> > > > From Roger.Riggs at Oracle.com Mon Jan 5 18:10:46 2015 From: Roger.Riggs at Oracle.com (Roger) Date: Mon, 05 Jan 2015 13:10:46 -0500 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AACE94.2000802@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB050.4010001@Oracle.com> <54AACE94.2000802@oracle.com> Message-ID: <54AAD3A6.9010200@Oracle.com> Hi Ivan, Looks pretty good. Just a thought on ClassLoader:1754; If you avoid the local for ClassLoaderHelper.allowsQuotedPathElements then the compiler can optimize the code and may do dead code elimination since it is a final static boolean. On Mac and Unix it reduces to just the original code. (ClassLoaderHelper seems like a weak case for a separate class, IMHO) $.02, Roger On 1/5/2015 12:49 PM, Ivan Gerasimov wrote: > Hi Roger! > > I've updated the webrev similarly to what you've suggested: > 1) Presence of quotes is checked when counting the separators. > 2) If quotes weren't found, we'll execute the same optimized loop as > for Unix. > > http://cr.openjdk.java.net/~igerasim/8067951/4/webrev/ > > Sincerely yours, > Ivan > > > On 05.01.2015 18:40, Roger wrote: >> Hi Ivan, >> >> For this small difference in the implementations, I'd recommend >> against having >> two different source files. The path initialization function is a >> one time function and >> the performance improvement is not significant. >> >> I'd suggest a few comments on your 2nd version[1]. >> >> - The windows check should check the system property or other >> definitive os check >> and could be better expressed (as Alan suggested) as quotesAllowed. >> - in the loop testing for the quote (") can come before the >> quotesAllowed check to >> speed things up (no need to check if they are allowed if they do >> not occur). >> This code is unlikely to be executed enough times to optimized. >> >> Roger >> >> [1] http://cr.openjdk.java.net/~igerasim/8067951/2/webrev/ >> >> On 1/4/2015 3:23 PM, Ivan Gerasimov wrote: >>> >>> On 04.01.2015 22:50, Alan Bateman wrote: >>>> On 03/01/2015 17:39, Ivan Gerasimov wrote: >>>>> >>>>> Currently, there are tree variants of ClassLoaderHelper: for >>>>> Windows, for Unix and for MacOS. >>>>> We have to either duplicate code in Unix and MacOS realizations, >>>>> or introduce another Helper class for initializing paths only, >>>>> which would have only two realizations: for Windows and all Unixes. >>>> When I made the comment then I was thinking of a method such as >>>> allowsQuotedPathElements (or a better name) that returns a boolean >>>> to indicate if quoting of path elements is allowed or not. That >>>> would abstract the capability a bit without needing to do isWindows >>>> checks. >>>> >>> Ah, I see. >>> Though, not needing to check for quotes allows a bit more efficient >>> implementation, so splitting the code for different platforms may >>> also make sense. >>> >>> I did it with another helper class in this webrev: >>> http://cr.openjdk.java.net/~igerasim/8067951/3/webrev/ >>> >>> Sincerely yours, >>> Ivan >>> >>> >>>> -Alan. >>>> >>>> >>> >> >> >> > From stuart.marks at oracle.com Mon Jan 5 18:37:42 2015 From: stuart.marks at oracle.com (Stuart Marks) Date: Mon, 05 Jan 2015 10:37:42 -0800 Subject: RFR(m): 8067316: update RMI test library with better test.timeout.factor handling In-Reply-To: <549A15BF.2000704@oracle.com> References: <549A15BF.2000704@oracle.com> Message-ID: <54AAD9F6.10809@oracle.com> Hi all, Could I get a review for this? (Serves me right for posting this just before the holidays.) Turns out this is the root cause of some test failures. Thanks, s'marks P.S. Happy new year, everyone! On 12/23/14 5:24 PM, Stuart Marks wrote: > Hi all, > > Please review this change, which adds handling of the test.timeout.factor > property to the RMI tests. These tests have a fair number of internal timeouts; > this change multiplies the timeout values by test.timeout.factor so that > timeouts will be less likely on slow systems, where a timeout factor has been > applied. > > In order read this property, a bunch of the tests' security policy files needed > to be updated. This mainly impacted the policy files for the rmid activation > group JVMs. I took the opportunity to clean up these policy files as well so > that they're all fairly consistent among themselves. > > The change updates a couple dozen policy files in a fairly stylized fashion. The > actual code changes are concentrated in the files ActivationLibrary.java, > JavaVM.java, RMID.java, and TestLibrary.java, which all appear at the end of the > webrev. > > Webrev: > > http://cr.openjdk.java.net/~smarks/reviews/8067316/webrev.0/ > > Bug: > > https://bugs.openjdk.java.net/browse/JDK-8067316 > > Sandbox changeset: > > http://hg.openjdk.java.net/jdk9/sandbox/jdk/rev/419205f6b2d9 > > Thanks, > > s'marks From Alan.Bateman at oracle.com Mon Jan 5 18:47:16 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 05 Jan 2015 18:47:16 +0000 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AACCC1.6030602@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB243.102@oracle.com> <54AACCC1.6030602@oracle.com> Message-ID: <54AADC34.1000703@oracle.com> On 05/01/2015 17:41, Ivan Gerasimov wrote: > > : > > Here's another update: > http://cr.openjdk.java.net/~igerasim/8067951/4/webrev/ > > The optimized loop is used both on Unix and on Windows, if there were > no quotes found in PATH. > Not a big deal, of course, but it should save a few nano-seconds > during startup. > This approach looks okay to me. A small improvement would be to change it to a method that returns a boolean. -Alan From kissziszi at gmail.com Mon Jan 5 18:59:47 2015 From: kissziszi at gmail.com (Zoltan Sziladi) Date: Mon, 5 Jan 2015 10:59:47 -0800 Subject: String.indexOf optimization In-Reply-To: References: <534D68B8.3040002@oracle.com> <5350E316.2050406@oracle.com> Message-ID: Hi, This discussion was a long time ago, I was just reading through it to check again what was the last state of the discussion about the String.indexOf. There is one part which I still do not understand, hopefully someone could shed some light on it. A few emails ago Martin mentioned "Hotspot seems to have some intrinsification of String.indexOf, which confuses me. Hotspot seems the right place to provide more optimizations for this, since there has been a fair amount of work creating high-performance low-level implementations of this idea in C." Then Ivan asked what that actually meant, whether hotspot actually replaced the jdk implementation with a low level optimized C implementation, but I never saw an answer to that. Can someone please explain this? If we somehow found an algorithm that beat the naive implementation in the average case, would it be possible to just implement it in the JDK? Also, is there a test set which we could consider conclusive enough to actually change the implementation based on results from that? (For example if I create an implementation that beats the naive algorithm in those testcases, then we could consider it faster in average case) Thanks! Zoltan On Fri, Apr 18, 2014 at 9:43 AM, Martin Buchholz wrote: > I remain skeptical that modifying the implementation of static package > private String.indexOf is the right approach. > > If we can produce high-performance implementations of these, perhaps making > them public in Arrays.java is the right way. > > 1766 if (targetCount == 1) {1767 return (i > <= max) ? i - sourceOffset : -1; > > > If you're going to special case targetCount == 1, you shouldn't have a test > for it in the main loop, since you slow down the general case. Instead, > you can sequester the special cases like this: > > if (targetCount <= 1) { > if (targetCount == 0) ... > else ... > } > > // now assume targetCount >= 2 > > > On Fri, Apr 18, 2014 at 1:32 AM, Ivan Gerasimov > wrote: > > > > > On 16.04.2014 2:53, Martin Buchholz wrote: > > > > Hi Ivan, > > > > There's already an indexOf(int, int) that allows a user to explicitly > > search for a char (or character). > > > > Sure. > > What I meant was not to introduce a special function for searching a > > single character, but to restrict the general case loop to search for a > > substring of at least 2 characters. > > Having this condition hold, we can enter the loop only when two starting > > characters match, and this can save us a few nanoseconds in many cases. > > > > > > Hotspot seems to have some intrinsification of String.indexOf, which > > confuses me. > > > > > > Does it mean that that the java implementation of indexOf is never > > compiled? > > When hotspot replaces the jdk implementation with its own one? > > Is it ever worth to try to optimize the java implementation? > > > > > > Hotspot seems the right place to provide more optimizations for this, > > since there has been a fair amount of work creating high-performance > > low-level implementations of this idea in C. > > > > The hotspot's intrinsic is already optimized for searching substrings of > > different length. > > > > Sincerely yours, > > Ivan > > > > > > > > On Tue, Apr 15, 2014 at 10:13 AM, Ivan Gerasimov < > > ivan.gerasimov at oracle.com> wrote: > > > >> Hi everyone! > >> > >> > >> On 04.04.2014 21:13, Martin Buchholz wrote: > >> > >> Summary: > >> > >> Many people (myself included) have looked at this problem. It's > unlikely > >> that String.indexOf will change. It's hard to beat the naive > >> implementation in the typical case. > >> > >> But we can try to speed up this naive implementation a little bit. > >> > >> We can separate the special case: When the substring's length == 1. > >> This special case can be done fast, and in the general case we can now > >> assume substring's length is at least 2. > >> > >> Here's the webrev with the implementation of this idea: > >> http://cr.openjdk.java.net/~igerasim/indexof/0/webrev/ > >> > >> I've done some benchmarking with JMH and found no performance > degradation > >> on my machine. > >> Of course, more testcases should be created and they should be tried on > >> different machines to be treated as reliable. > >> > >> Benchmark Mode Thr Cnt Sec Mean > >> Mean error Units > >> o.b.IndexOfBench.benchIndexOf_1_A avgt 1 20 5 > >> 704.739 9.104 nsec/op > >> o.b.IndexOfBench.benchIndexOf_1_B avgt 1 20 5 * > >> 573.879* 9.820 nsec/op > >> o.b.IndexOfBench.benchIndexOf_2_A avgt 1 20 5 > >> 668.455 9.882 nsec/op > >> o.b.IndexOfBench.benchIndexOf_2_B avgt 1 20 5 > >> *476.062* 6.063 nsec/op > >> o.b.IndexOfBench.benchIndexOf_3_A avgt 1 20 5 > >> 155.227 1.796 nsec/op > >> o.b.IndexOfBench.benchIndexOf_3_B avgt 1 20 5 * > >> 152.850 * 1.512 nsec/op > >> o.b.IndexOfBench.benchIndexOf_4_A avgt 1 20 5 > >> 656.183 5.904 nsec/op > >> o.b.IndexOfBench.benchIndexOf_4_B avgt 1 20 5 > >> *515.178* 9.107 nsec/op > >> o.b.IndexOfBench.benchIndexOf_5_A avgt 1 20 5 > >> 140.609 7.305 nsec/op > >> o.b.IndexOfBench.benchIndexOf_5_B avgt 1 20 5 > >> *129.603* 1.654 nsec/op > >> o.b.IndexOfBench.benchIndexOf_6_A avgt 1 20 5 > >> 127.713 1.497 nsec/op > >> o.b.IndexOfBench.benchIndexOf_6_B avgt 1 20 5 > >> *122.177* 1.186 nsec/op > >> o.b.IndexOfBench.benchIndexOf_7_A avgt 1 20 5 > >> 430.148 4.875 nsec/op > >> o.b.IndexOfBench.benchIndexOf_7_B avgt 1 20 5 * > >> 387.338* 10.904 nsec/op > >> o.b.IndexOfBench.benchIndexOf_8_A avgt 1 20 5 > >> 2064.563 28.885 nsec/op > >> o.b.IndexOfBench.benchIndexOf_8_B avgt 1 20 5 > >> *1858.669* 24.343 nsec/op > >> > >> Benchmarks ending with A use the current indexOf implementation, with B > >> use the modified version. > >> These numbers show from 1.4% up to 28% performance increase. > >> > >> The full listing is below. > >> > >> Suggestions about what else to test are welcome! > >> > >> Sincerely yours, > >> Ivan > >> > >> --------------------- > >> > >> /** > >> * Copyright (c) 2014, Oracle and/or its affiliates. All rights > reserved. > >> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. > >> * > >> * This code is free software; you can redistribute it and/or modify it > >> * under the terms of the GNU General Public License version 2 only, as > >> * published by the Free Software Foundation. Oracle designates this > >> * particular file as subject to the "Classpath" exception as provided > >> * by Oracle in the LICENSE file that accompanied this code. > >> * > >> * This code is distributed in the hope that it will be useful, but > >> WITHOUT > >> * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or > >> * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License > >> * version 2 for more details (a copy is included in the LICENSE file > that > >> * accompanied this code). > >> * > >> * You should have received a copy of the GNU General Public License > >> version > >> * 2 along with this work; if not, write to the Free Software > Foundation, > >> * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. > >> * > >> * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 > USA > >> * or visit www.oracle.com if you need additional information or have > any > >> * questions. > >> */ > >> package org.benches; > >> > >> import org.openjdk.jmh.annotations.*; > >> import org.openjdk.jmh.logic.BlackHole; > >> import java.util.concurrent.TimeUnit; > >> > >> @BenchmarkMode(Mode.AverageTime) > >> @OutputTimeUnit(TimeUnit.NANOSECONDS) > >> @State(Scope.Benchmark) > >> public class IndexOfBench { > >> > >> > >> // > >> ||| > >> final static char[] source1 = > >> > "abababcabcacabcabcabcabcaccbcabcacbcabcacbcbcabcbcbacbcabcbabacbcbacbcabcabcabcabcabcabcabcacbacbacbacabcabcacbacbcabcbcbcaabdacbacabcabacbacabca".toCharArray(); > >> final static char[] source2 = > >> > "acfacafacfacfacfacafcafcacadcacdacaccacacdacacfcafcafcfacdacadcadcadcdacfacfacdacadcacdcfacfacdacdacdcfacdacdacdacgshgshasdabdahghjgwdshacavcavsc".toCharArray(); > >> final static char[] source3 = > >> > "tyrtytfytytuytfytuytggfghgdytyftytfdytdshfgjhdfytsfuythgsfhgjhfghtuysdfthgfsdhgystfjhgsfguysthgfgjhgdfjhgsjdghfythgsdfjhggfabduikjhfjhkjhfkjhgkjh".toCharArray(); > >> final static char[] target1 = "abd".toCharArray(); > >> > >> final static char[] source4 = > >> > "ahhhahahahahhahahahahhahahahhhahahahahahahahahahahhahahahahahahahahallallalalalalalkakakakakakakakakkakmamamamabammamamamamamaakaklalalaoalalalao".toCharArray(); > >> final static char[] source5 = > >> > "hgjkhjhjkdghkjhdfkjhgkjhdkjdhgkjdfhgkjdhfgkjdfhgkjhdfkjghkdjghkdjfhgkjhkdjhgkjdfhjkghkdjfhgkjdfhgkjdfhgkjdfhkgabhfkjghdkfjhgkjdfhgkjdfhgkjdfhgkhh".toCharArray(); > >> final static char[] target2 = "ab".toCharArray(); > >> > >> final static char[] source6 = > >> > "lskgjsklfjgskldfjgklsfjdlgkjsdflkgjskldfgjsdklfgjsl;kdfgjskldfjglksdfjglksfjglksdfjgklsfdjgslkdfjglksjdflkgsjfalksjdflkfgjsdklfjglskdfjglksdfjghh".toCharArray(); > >> final static char[] target3 = "a".toCharArray(); > >> > >> final static char[] source7 = > >> > "lskgajabfagskldfjgklsabclgkjsdflkabsabcdgjsdklfabclbkdfgjskabfjglksdfjabcdfjglabcfjgklsfabgslkdfjglksjdabcdsjfabcdedflabcjsdklfjglskdfabcksdfjghh".toCharArray(); > >> final static char[] target4 = "abcde".toCharArray(); > >> > >> final static char[] source8 = > >> > "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".toCharArray(); > >> final static char[] target5 = "aaaab".toCharArray(); > >> > >> @GenerateMicroBenchmark > >> public void benchIndexOf_1_A(BlackHole bh) { > >> bh.consume(indexOfA(source1, 0, source1.length, target1, 0, > >> target1.length, 0)); > >> } > >> > >> @GenerateMicroBenchmark > >> public void benchIndexOf_1_B(BlackHole bh) { > >> bh.consume(indexOfB(source1, 0, source1.length, target1, 0, > >> target1.length, 0)); > >> } > >> > >> @GenerateMicroBenchmark > >> public void benchIndexOf_2_A(BlackHole bh) { > >> bh.consume(indexOfA(source2, 0, source2.length, target1, 0, > >> target1.length, 0)); > >> } > >> > >> @GenerateMicroBenchmark > >> public void benchIndexOf_2_B(BlackHole bh) { > >> bh.consume(indexOfB(source2, 0, source2.length, target1, 0, > >> target1.length, 0)); > >> } > >> > >> @GenerateMicroBenchm > >> > > > > > From xueming.shen at oracle.com Mon Jan 5 19:10:23 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Mon, 05 Jan 2015 11:10:23 -0800 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AAD3A6.9010200@Oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB050.4010001@Oracle.com> <54AACE94.2000802@oracle.com> <54AAD3A6.9010200@Oracle.com> Message-ID: <54AAE19F.4090102@oracle.com> Just wonder if we really need that "inQuotes" logic here? A straightforward approach might be "every time you have a quote, skip everything until you hit another one, when counting, or copy everything into the buffer until hit another one, when copying" ? Since you are making copy (into the temporary buffer) anyway for "quote" case, another approach might be to pre-scan the input propName first for quote, if there is one, make a clean copy of the propName without those quotes, then go to the original implementation. This logic can be implemented at the very beginning as private static String[] intializePath(String propname) { if (ClassLoaderHelper.allowsQuotedPathElements) { ... } ... // existing implementation } -Sherman On 01/05/2015 10:10 AM, Roger wrote: > Hi Ivan, > > Looks pretty good. > > Just a thought on ClassLoader:1754; If you avoid the local for ClassLoaderHelper.allowsQuotedPathElements > then the compiler can optimize the code and may do dead code elimination since it is a final static boolean. > On Mac and Unix it reduces to just the original code. > > (ClassLoaderHelper seems like a weak case for a separate class, IMHO) > > $.02, Roger > > > On 1/5/2015 12:49 PM, Ivan Gerasimov wrote: >> Hi Roger! >> >> I've updated the webrev similarly to what you've suggested: >> 1) Presence of quotes is checked when counting the separators. >> 2) If quotes weren't found, we'll execute the same optimized loop as for Unix. >> >> http://cr.openjdk.java.net/~igerasim/8067951/4/webrev/ >> >> Sincerely yours, >> Ivan >> >> >> On 05.01.2015 18:40, Roger wrote: >>> Hi Ivan, >>> >>> For this small difference in the implementations, I'd recommend against having >>> two different source files. The path initialization function is a one time function and >>> the performance improvement is not significant. >>> >>> I'd suggest a few comments on your 2nd version[1]. >>> >>> - The windows check should check the system property or other definitive os check >>> and could be better expressed (as Alan suggested) as quotesAllowed. >>> - in the loop testing for the quote (") can come before the quotesAllowed check to >>> speed things up (no need to check if they are allowed if they do not occur). >>> This code is unlikely to be executed enough times to optimized. >>> >>> Roger >>> >>> [1] http://cr.openjdk.java.net/~igerasim/8067951/2/webrev/ >>> >>> On 1/4/2015 3:23 PM, Ivan Gerasimov wrote: >>>> >>>> On 04.01.2015 22:50, Alan Bateman wrote: >>>>> On 03/01/2015 17:39, Ivan Gerasimov wrote: >>>>>> >>>>>> Currently, there are tree variants of ClassLoaderHelper: for Windows, for Unix and for MacOS. >>>>>> We have to either duplicate code in Unix and MacOS realizations, or introduce another Helper class for initializing paths only, which would have only two realizations: for Windows and all Unixes. >>>>> When I made the comment then I was thinking of a method such as allowsQuotedPathElements (or a better name) that returns a boolean to indicate if quoting of path elements is allowed or not. That would abstract the capability a bit without needing to do isWindows checks. >>>>> >>>> Ah, I see. >>>> Though, not needing to check for quotes allows a bit more efficient implementation, so splitting the code for different platforms may also make sense. >>>> >>>> I did it with another helper class in this webrev: >>>> http://cr.openjdk.java.net/~igerasim/8067951/3/webrev/ >>>> >>>> Sincerely yours, >>>> Ivan >>>> >>>> >>>>> -Alan. >>>>> >>>>> >>>> >>> >>> >>> >> > From lance.andersen at oracle.com Mon Jan 5 19:10:35 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Mon, 5 Jan 2015 14:10:35 -0500 Subject: RFR(m): 8067316: update RMI test library with better test.timeout.factor handling In-Reply-To: <54AAD9F6.10809@oracle.com> References: <549A15BF.2000704@oracle.com> <54AAD9F6.10809@oracle.com> Message-ID: <17319E41-5E9F-42C7-83BD-20812EB85DDB@oracle.com> Hi Stuart, Looks OK to me :-) Best Lance On Jan 5, 2015, at 1:37 PM, Stuart Marks wrote: > Hi all, > > Could I get a review for this? (Serves me right for posting this just before the holidays.) > > Turns out this is the root cause of some test failures. > > Thanks, > > s'marks > > P.S. Happy new year, everyone! > > On 12/23/14 5:24 PM, Stuart Marks wrote: >> Hi all, >> >> Please review this change, which adds handling of the test.timeout.factor >> property to the RMI tests. These tests have a fair number of internal timeouts; >> this change multiplies the timeout values by test.timeout.factor so that >> timeouts will be less likely on slow systems, where a timeout factor has been >> applied. >> >> In order read this property, a bunch of the tests' security policy files needed >> to be updated. This mainly impacted the policy files for the rmid activation >> group JVMs. I took the opportunity to clean up these policy files as well so >> that they're all fairly consistent among themselves. >> >> The change updates a couple dozen policy files in a fairly stylized fashion. The >> actual code changes are concentrated in the files ActivationLibrary.java, >> JavaVM.java, RMID.java, and TestLibrary.java, which all appear at the end of the >> webrev. >> >> Webrev: >> >> http://cr.openjdk.java.net/~smarks/reviews/8067316/webrev.0/ >> >> Bug: >> >> https://bugs.openjdk.java.net/browse/JDK-8067316 >> >> Sandbox changeset: >> >> http://hg.openjdk.java.net/jdk9/sandbox/jdk/rev/419205f6b2d9 >> >> Thanks, >> >> s'marks Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From martinrb at google.com Mon Jan 5 19:31:24 2015 From: martinrb at google.com (Martin Buchholz) Date: Mon, 5 Jan 2015 11:31:24 -0800 Subject: String.indexOf optimization In-Reply-To: References: <534D68B8.3040002@oracle.com> <5350E316.2050406@oracle.com> Message-ID: Evidence that hotspot tries to intrinsify String.indexOf:  do_intrinsic(_indexOf, java_lang_String, indexOf_name, string_int_signature, F_R) \ do_name( indexOf_name, "indexOf") \ So work would have to be done at the hotspot intrinsics level (not easy!) Also, the problem is that Boyer-Moore is a fundamental improvement in string searching, but its overhead is high enough that it's unlikely to help with typical input strings found in the Real World. I think we would want to split into two implementations and only do Boyer-Moore if it looks profitable. Similarly for j.u.r.Pattern's regex compiler. I still think it's sufficiently difficult that effort is best applied elsewhere. On Mon, Jan 5, 2015 at 10:59 AM, Zoltan Sziladi wrote: > Hi, > > This discussion was a long time ago, I was just reading through it to > check again what was the last state of the discussion about the > String.indexOf. > There is one part which I still do not understand, hopefully someone could > shed some light on it. A few emails ago Martin mentioned > > "Hotspot seems to have some intrinsification of String.indexOf, which > confuses me. > Hotspot seems the right place to provide more optimizations for this, > since there has been a fair amount of work creating high-performance > low-level implementations of this idea in C." > > Then Ivan asked what that actually meant, whether hotspot actually > replaced the jdk implementation with a low level optimized C > implementation, but I never saw an answer to that. > > Can someone please explain this? If we somehow found an algorithm that > beat the naive implementation in the average case, would it be possible to > just implement it in the JDK? Also, is there a test set which we could > consider conclusive enough to actually change the implementation based on > results from that? (For example if I create an implementation that beats > the naive algorithm in those testcases, then we could consider it faster in > average case) > > Thanks! > > Zoltan > > On Fri, Apr 18, 2014 at 9:43 AM, Martin Buchholz > wrote: > >> I remain skeptical that modifying the implementation of static package >> private String.indexOf is the right approach. >> >> If we can produce high-performance implementations of these, perhaps >> making >> them public in Arrays.java is the right way. >> >> 1766 if (targetCount == 1) {1767 return (i >> <= max) ? i - sourceOffset : -1; >> >> >> If you're going to special case targetCount == 1, you shouldn't have a >> test >> for it in the main loop, since you slow down the general case. Instead, >> you can sequester the special cases like this: >> >> if (targetCount <= 1) { >> if (targetCount == 0) ... >> else ... >> } >> >> // now assume targetCount >= 2 >> >> >> On Fri, Apr 18, 2014 at 1:32 AM, Ivan Gerasimov >> wrote: >> >> > >> > On 16.04.2014 2:53, Martin Buchholz wrote: >> > >> > Hi Ivan, >> > >> > There's already an indexOf(int, int) that allows a user to explicitly >> > search for a char (or character). >> > >> > Sure. >> > What I meant was not to introduce a special function for searching a >> > single character, but to restrict the general case loop to search for a >> > substring of at least 2 characters. >> > Having this condition hold, we can enter the loop only when two starting >> > characters match, and this can save us a few nanoseconds in many cases. >> > >> > >> > Hotspot seems to have some intrinsification of String.indexOf, which >> > confuses me. >> > >> > >> > Does it mean that that the java implementation of indexOf is never >> > compiled? >> > When hotspot replaces the jdk implementation with its own one? >> > Is it ever worth to try to optimize the java implementation? >> > >> > >> > Hotspot seems the right place to provide more optimizations for this, >> > since there has been a fair amount of work creating high-performance >> > low-level implementations of this idea in C. >> > >> > The hotspot's intrinsic is already optimized for searching substrings >> of >> > different length. >> > >> > Sincerely yours, >> > Ivan >> > >> > >> > >> > On Tue, Apr 15, 2014 at 10:13 AM, Ivan Gerasimov < >> > ivan.gerasimov at oracle.com> wrote: >> > >> >> Hi everyone! >> >> >> >> >> >> On 04.04.2014 21:13, Martin Buchholz wrote: >> >> >> >> Summary: >> >> >> >> Many people (myself included) have looked at this problem. It's >> unlikely >> >> that String.indexOf will change. It's hard to beat the naive >> >> implementation in the typical case. >> >> >> >> But we can try to speed up this naive implementation a little bit. >> >> >> >> We can separate the special case: When the substring's length == 1. >> >> This special case can be done fast, and in the general case we can now >> >> assume substring's length is at least 2. >> >> >> >> Here's the webrev with the implementation of this idea: >> >> http://cr.openjdk.java.net/~igerasim/indexof/0/webrev/ >> >> >> >> I've done some benchmarking with JMH and found no performance >> degradation >> >> on my machine. >> >> Of course, more testcases should be created and they should be tried on >> >> different machines to be treated as reliable. >> >> >> >> Benchmark Mode Thr Cnt Sec Mean >> >> Mean error Units >> >> o.b.IndexOfBench.benchIndexOf_1_A avgt 1 20 5 >> >> 704.739 9.104 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_1_B avgt 1 20 5 * >> >> 573.879* 9.820 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_2_A avgt 1 20 5 >> >> 668.455 9.882 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_2_B avgt 1 20 5 >> >> *476.062* 6.063 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_3_A avgt 1 20 5 >> >> 155.227 1.796 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_3_B avgt 1 20 5 * >> >> 152.850 * 1.512 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_4_A avgt 1 20 5 >> >> 656.183 5.904 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_4_B avgt 1 20 5 >> >> *515.178* 9.107 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_5_A avgt 1 20 5 >> >> 140.609 7.305 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_5_B avgt 1 20 5 >> >> *129.603* 1.654 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_6_A avgt 1 20 5 >> >> 127.713 1.497 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_6_B avgt 1 20 5 >> >> *122.177* 1.186 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_7_A avgt 1 20 5 >> >> 430.148 4.875 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_7_B avgt 1 20 5 * >> >> 387.338* 10.904 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_8_A avgt 1 20 5 >> >> 2064.563 28.885 nsec/op >> >> o.b.IndexOfBench.benchIndexOf_8_B avgt 1 20 5 >> >> *1858.669* 24.343 nsec/op >> >> >> >> >> Benchmarks ending with A use the current indexOf implementation, with B >> >> use the modified version. >> >> These numbers show from 1.4% up to 28% performance increase. >> >> >> >> The full listing is below. >> >> >> >> Suggestions about what else to test are welcome! >> >> >> >> Sincerely yours, >> >> Ivan >> >> >> >> --------------------- >> >> >> >> /** >> >> * Copyright (c) 2014, Oracle and/or its affiliates. All rights >> reserved. >> >> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. >> >> * >> >> * This code is free software; you can redistribute it and/or modify it >> >> * under the terms of the GNU General Public License version 2 only, as >> >> * published by the Free Software Foundation. Oracle designates this >> >> * particular file as subject to the "Classpath" exception as provided >> >> * by Oracle in the LICENSE file that accompanied this code. >> >> * >> >> * This code is distributed in the hope that it will be useful, but >> >> WITHOUT >> >> * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY >> or >> >> * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public >> License >> >> * version 2 for more details (a copy is included in the LICENSE file >> that >> >> * accompanied this code). >> >> * >> >> * You should have received a copy of the GNU General Public License >> >> version >> >> * 2 along with this work; if not, write to the Free Software >> Foundation, >> >> * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. >> >> * >> >> * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 >> USA >> >> * or visit www.oracle.com if you need additional information or have >> any >> >> * questions. >> >> */ >> >> package org.benches; >> >> >> >> import org.openjdk.jmh.annotations.*; >> >> import org.openjdk.jmh.logic.BlackHole; >> >> import java.util.concurrent.TimeUnit; >> >> >> >> @BenchmarkMode(Mode.AverageTime) >> >> @OutputTimeUnit(TimeUnit.NANOSECONDS) >> >> @State(Scope.Benchmark) >> >> public class IndexOfBench { >> >> >> >> >> >> // >> >> ||| >> >> final static char[] source1 = >> >> >> "abababcabcacabcabcabcabcaccbcabcacbcabcacbcbcabcbcbacbcabcbabacbcbacbcabcabcabcabcabcabcabcacbacbacbacabcabcacbacbcabcbcbcaabdacbacabcabacbacabca".toCharArray(); >> >> final static char[] source2 = >> >> >> "acfacafacfacfacfacafcafcacadcacdacaccacacdacacfcafcafcfacdacadcadcadcdacfacfacdacadcacdcfacfacdacdacdcfacdacdacdacgshgshasdabdahghjgwdshacavcavsc".toCharArray(); >> >> final static char[] source3 = >> >> >> "tyrtytfytytuytfytuytggfghgdytyftytfdytdshfgjhdfytsfuythgsfhgjhfghtuysdfthgfsdhgystfjhgsfguysthgfgjhgdfjhgsjdghfythgsdfjhggfabduikjhfjhkjhfkjhgkjh".toCharArray(); >> >> final static char[] target1 = "abd".toCharArray(); >> >> >> >> final static char[] source4 = >> >> >> "ahhhahahahahhahahahahhahahahhhahahahahahahahahahahhahahahahahahahahallallalalalalalkakakakakakakakakkakmamamamabammamamamamamaakaklalalaoalalalao".toCharArray(); >> >> final static char[] source5 = >> >> >> "hgjkhjhjkdghkjhdfkjhgkjhdkjdhgkjdfhgkjdhfgkjdfhgkjhdfkjghkdjghkdjfhgkjhkdjhgkjdfhjkghkdjfhgkjdfhgkjdfhgkjdfhkgabhfkjghdkfjhgkjdfhgkjdfhgkjdfhgkhh".toCharArray(); >> >> final static char[] target2 = "ab".toCharArray(); >> >> >> >> final static char[] source6 = >> >> >> "lskgjsklfjgskldfjgklsfjdlgkjsdflkgjskldfgjsdklfgjsl;kdfgjskldfjglksdfjglksfjglksdfjgklsfdjgslkdfjglksjdflkgsjfalksjdflkfgjsdklfjglskdfjglksdfjghh".toCharArray(); >> >> final static char[] target3 = "a".toCharArray(); >> >> >> >> final static char[] source7 = >> >> >> "lskgajabfagskldfjgklsabclgkjsdflkabsabcdgjsdklfabclbkdfgjskabfjglksdfjabcdfjglabcfjgklsfabgslkdfjglksjdabcdsjfabcdedflabcjsdklfjglskdfabcksdfjghh".toCharArray(); >> >> final static char[] target4 = "abcde".toCharArray(); >> >> >> >> final static char[] source8 = >> >> >> "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".toCharArray(); >> >> final static char[] target5 = "aaaab".toCharArray(); >> >> >> >> @GenerateMicroBenchmark >> >> public void benchIndexOf_1_A(BlackHole bh) { >> >> bh.consume(indexOfA(source1, 0, source1.length, target1, 0, >> >> target1.length, 0)); >> >> } >> >> >> >> @GenerateMicroBenchmark >> >> public void benchIndexOf_1_B(BlackHole bh) { >> >> bh.consume(indexOfB(source1, 0, source1.length, target1, 0, >> >> target1.length, 0)); >> >> } >> >> >> >> @GenerateMicroBenchmark >> >> public void benchIndexOf_2_A(BlackHole bh) { >> >> bh.consume(indexOfA(source2, 0, source2.length, target1, 0, >> >> target1.length, 0)); >> >> } >> >> >> >> @GenerateMicroBenchmark >> >> public void benchIndexOf_2_B(BlackHole bh) { >> >> bh.consume(indexOfB(source2, 0, source2.length, target1, 0, >> >> target1.length, 0)); >> >> } >> >> >> >> @GenerateMicroBenchm >> >> >> > >> > >> > > From ivan.gerasimov at oracle.com Mon Jan 5 20:41:59 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Mon, 05 Jan 2015 23:41:59 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AAE19F.4090102@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB050.4010001@Oracle.com> <54AACE94.2000802@oracle.com> <54AAD3A6.9010200@Oracle.com> <54AAE19F.4090102@oracle.com> Message-ID: <54AAF717.3070602@oracle.com> Thanks Sherman! On 05.01.2015 22:10, Xueming Shen wrote: > > Just wonder if we really need that "inQuotes" logic here? A > straightforward approach might > be "every time you have a quote, skip everything until you hit another > one, when counting, > or copy everything into the buffer until hit another one, when copying" ? > I agree it would work, but, in my opinion, it would be a bit more complicated. The counting loop would look something like this: ------------------------------------ outerLoop: for (int i = 0; i < ldLen; ++i) { char ch = ldPath.charAt(i); if (mayBeQuoted && ch == '\"') { thereAreQuotes = true; for (++i; i < ldLen; ++i) { if (ldPath.charAt(i) == '\"') { continue outerLoop; } } break; // unpaired quote } else if (ch == ps) { psCount++; } } ------------------------------------ which is 3 lines longer, comparing to the loop with inQuotes flag. > Since you are making copy (into the temporary buffer) anyway for > "quote" case, another > approach might be to pre-scan the input propName first for quote, if > there is one, make > a clean copy of the propName without those quotes, then go to the > original implementation. Unfortunately, it wouldn't work in those rare cases, where the PATH contains a quoted path with semicolon in it. Windows allows semicolon in the file names, and it seems to be the only reason to allow quoting the PATH entries. Sincerely yours, Ivan > This logic can be implemented at the very beginning as > > private static String[] intializePath(String propname) { > if (ClassLoaderHelper.allowsQuotedPathElements) { > ... > } > ... // existing implementation > } > > -Sherman > > On 01/05/2015 10:10 AM, Roger wrote: >> Hi Ivan, >> >> Looks pretty good. >> >> Just a thought on ClassLoader:1754; If you avoid the local for >> ClassLoaderHelper.allowsQuotedPathElements >> then the compiler can optimize the code and may do dead code >> elimination since it is a final static boolean. >> On Mac and Unix it reduces to just the original code. >> >> (ClassLoaderHelper seems like a weak case for a separate class, IMHO) >> >> $.02, Roger >> >> >> On 1/5/2015 12:49 PM, Ivan Gerasimov wrote: >>> Hi Roger! >>> >>> I've updated the webrev similarly to what you've suggested: >>> 1) Presence of quotes is checked when counting the separators. >>> 2) If quotes weren't found, we'll execute the same optimized loop as >>> for Unix. >>> >>> http://cr.openjdk.java.net/~igerasim/8067951/4/webrev/ >>> >>> Sincerely yours, >>> Ivan >>> >>> >>> On 05.01.2015 18:40, Roger wrote: >>>> Hi Ivan, >>>> >>>> For this small difference in the implementations, I'd recommend >>>> against having >>>> two different source files. The path initialization function is a >>>> one time function and >>>> the performance improvement is not significant. >>>> >>>> I'd suggest a few comments on your 2nd version[1]. >>>> >>>> - The windows check should check the system property or other >>>> definitive os check >>>> and could be better expressed (as Alan suggested) as quotesAllowed. >>>> - in the loop testing for the quote (") can come before the >>>> quotesAllowed check to >>>> speed things up (no need to check if they are allowed if they do >>>> not occur). >>>> This code is unlikely to be executed enough times to optimized. >>>> >>>> Roger >>>> >>>> [1] http://cr.openjdk.java.net/~igerasim/8067951/2/webrev/ >>>> >>>> On 1/4/2015 3:23 PM, Ivan Gerasimov wrote: >>>>> >>>>> On 04.01.2015 22:50, Alan Bateman wrote: >>>>>> On 03/01/2015 17:39, Ivan Gerasimov wrote: >>>>>>> >>>>>>> Currently, there are tree variants of ClassLoaderHelper: for >>>>>>> Windows, for Unix and for MacOS. >>>>>>> We have to either duplicate code in Unix and MacOS realizations, >>>>>>> or introduce another Helper class for initializing paths only, >>>>>>> which would have only two realizations: for Windows and all Unixes. >>>>>> When I made the comment then I was thinking of a method such as >>>>>> allowsQuotedPathElements (or a better name) that returns a >>>>>> boolean to indicate if quoting of path elements is allowed or >>>>>> not. That would abstract the capability a bit without needing to >>>>>> do isWindows checks. >>>>>> >>>>> Ah, I see. >>>>> Though, not needing to check for quotes allows a bit more >>>>> efficient implementation, so splitting the code for different >>>>> platforms may also make sense. >>>>> >>>>> I did it with another helper class in this webrev: >>>>> http://cr.openjdk.java.net/~igerasim/8067951/3/webrev/ >>>>> >>>>> Sincerely yours, >>>>> Ivan >>>>> >>>>> >>>>>> -Alan. >>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>> >> > > > From brian.goetz at oracle.com Mon Jan 5 20:51:47 2015 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 05 Jan 2015 15:51:47 -0500 Subject: Explicit Serialization API and Security In-Reply-To: <54AA9B92.3050705@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> Message-ID: <54AAF963.2020402@oracle.com> Thanks Peter(s): I think it's just about time to re-sync on the goals, since there are an infinitude of issues with serialization, only some of which we can address. To that end, let me toss out some of the results of our recent explorations with improving serialization, where we started out on something not unlike what Peter is suggesting, and dialed back. One thing we can do to positively improve the serialization experience is to reduce the set of situations where users have to write explicit readObject() methods. In the JDK, we end up having to do so for all manner of security-motivated reasons, and such code is hard to write correctly, hard to validate, hard to audit, and hard to evolve. After an audit of our codebase (thanks Chris!), we identified some of the biggest offenders on this score: - Validating invariants - Ensuring confinement The latter comes up in cases where there's a field: class Foo { private final Bar bar = new BarImpl(); } which implicitly has an invariant: the Bar referenced by Foo.bar is a) exactly of type BarImpl and not some other subclass and b) there is exactly one reference to that BarImpl (assuming it doesn't escape Foo.) Note that both of these still use the default deserialization; having to override readObject() just to do some extra checking is error-prone and messy. So I think the goal of a separate validation method which is called automatically is a good one, since it means we can ditch some percentage of otherwise-needless readObject implementations (which then have to jump through hoops to set final fields.) I agree that being able to play nicely with final fields is also an important goal here. On 1/5/2015 9:11 AM, Peter Levart wrote: > Hi Peter and others, > > A assume your approach described here (to use constructors for > deserialization) is motivated mainly by the fact that only constructors > are allowed to set the final fields. Otherwise the explicit features of > your ReadSerial API are more or less accessible even now by using > "serialPersistentFields " static field for declaration of "serialized" > fields, combined with ObjectOutputStream.PutField and > ObjectInputStream.GetField API, accessible from writeObject/readObject > methods. readObject() method already acts like a constructor, whith the > following notable differences: > > 1 - it is not chained explicitly to superclass (either in source or > automatically by javac) like constructor, but is invoked by > (de)serialization infrastructure in correct order for each class in the > hierarchy that declares it. > 2 - it is a normal method, so it can be called at any time by code in > the declaring class or inner classes. > 3 - it can't set final fields (without using clumsy reflection) > > > The 1st point is actually in favor of readObject() method, since the > (de)serialization API need not be caller-sensitive. The correct > per-class context is established before each readObject call-back. > 2nd and 3rd points are readObject drawbacks and are both interconnected. > If readObject was not universally callable, then the restriction that it > can't set final fields could be lifted. > > Perhaps it is too late from compatibility perspective to change the > readObject's 2nd and 3rd point now, but we could compatibly change the > 1st point of a special kind of constructor. > > Say that a constructor declared with exactly one parameter of type > ObjectInputStream is treated specially by javac and reflection: > - constructor code can't chain to superclass constructor > - javac does not automatically generate default constructor chaining code > - javac does not allow calling such constructor from any code > - this is optional: javac does not allow calling instance methods and > explicit use of 'this' keyword in deserialization constructor code > (preventing escape of such object before de-serialization ends) - to be > really safe, this would probably have to be accompanied with verifier > checks too. > - public reflection API does not allow invoking such constructor (only > deserialization infrastructure can - like it can invoke the default > constructor of a non-Serializable superclass on an already allocated > instance of a sublclass) > > No additional changes to VM are apparently necessary (except verifier). > Declaring such constructor as an alternative to readObject() method is > then possible to make deserialization more safe. > > Regarding finalizability (and finalizer attacks): > > The serialization specifies that for 1st (most specific) > non-Serializable superclass of a Serializable subclass, default > constructor is invoked to initialize the non-Serializable superclass > state before deserializing subclass state. As each Serializable class > has such a non-Serializable superclass (at least in the form of Object > class) and normal constructors are chained, it is apparent that Object's > default constructor is called before any deserialization of state > begins. As soon as Object's constructor completes, the instance becomes > eligible for finalization: > > "An object o is not finalizable until its constructor has invoked the > constructor for Object on o and that invocation has completed > successfully (that is, without throwing an exception).? > > The rule of calling non-Serializable superclass default constructor > could be weakened. If 1st (most specific) non-Serializable superclass of > a Serializable subclass is Object, then it's constructor is not called. > Obviously it is not needed since Object has no state to initialize. What > we achieve by this is that such object does not become eligible for > finalization just yet. Serialization infrastructure must then explicitly > register such objects for finalization, but only after their > de-serialization completes normally. This would prevent finalization > attacks. > > Regards, Peter > > > On 01/05/2015 01:01 PM, Peter Firmstone wrote: >> My mistake, thank you. >> >> Peter. >> >> On 5/01/2015 9:57 PM, David Holmes wrote: >>> Hi Peter, >>> >>> Did you send this to the list? I haven't seen it turn up yet. >>> >>> David >>> >>> On 5/01/2015 6:51 PM, Peter Firmstone wrote: >>>> Thanks Dave, >>>> >>>> That's another way of checking invariants, you could expose A's check >>>> method but you'd also need a couple of additional static methods to >>>> obtain integers upper and lower from ReadSerial, so that B can ensure >>>> its invariant. >>>> >>>> ReadSerial is caller sensitive so B can't obtain A's stream values via >>>> ReadSerial and must do so via A's API. This prevents the >>>> publication of >>>> A's implementation, you don't wan't B depending on A's serial form. >>>> Instead A can reorder and rename it's fields and change their values, >>>> have multiple serial forms and be able to remain compatible with all >>>> forms relatively easily. >>>> >>>> There are some benefits to using a temporary object instance of A; it >>>> reduces the amount of code required, eg it would be a beneficial for >>>> RMI >>>> to minimise code downloads, the JVM is free to inline access to these >>>> fields and the temporary instance of A never leaves the cpu cache, so >>>> it's not likely to become a performance issue. Well it might on real >>>> time java :) >>>> >>>> If circular relationships are mutable, or effectively mutable after >>>> publication, then we could eventually deprecate the requirement to >>>> support special treatment of final fields for Serializable. >>>> >>>> Cheers, >>>> >>>> Peter. >>>> >>>> On 5/01/2015 10:38 AM, David Holmes wrote: >>>>> >>>>>>> - I don't see how this invariant-checking mechanism can >>>>>>> enforce >>>>>>> invariants between superclass fields and subclass fields. For >>>>>>> example: >>>>>>> >>>>>>> class A { >>>>>>> int lower, upper; // invariant: lower <= upper >>>>>>> } >>>>>>> >>>>>>> class B extends A { >>>>>>> int cur; // invariant: lower <= cur <= upper >>>>>>> } >>>>>>> >>>>>> >>>>>> The serialization framework should only construct an objects fields >>>>>> and make these available from ReadSerial, each class then calls a >>>>>> static method before calling a superclass constructor that's >>>>>> responsible for an objects creation, to prevent construction of the >>>>>> object, if necessary. >>>>>> >>>>>> Eg, some complexity, but bullet proof: >>>>>> >>>>>> public class A ( >>>>>> >>>>>> public final int lower, upper; >>>>>> >>>>>> private static boolean check(ReadSerial rs){ >>>>>> if (rs.getInt("lower") > rs.getInt("upper")) >>>>>> throw new IllegalArgumentException( >>>>>> "lower bound must be less than or equal to upper"); >>>>>> return true; >>>>>> } >>>>>> >>>>>> public A(ReadSerial rs){ >>>>>> this(check(rs), rs); >>>>>> } >>>>>> >>>>>> private A(boolean checked, ReadSerial rs){ >>>>>> super(); >>>>>> lower = rs.getInt("lower"); >>>>>> upper = rs.getInt("upper"); >>>>>> } >>>>>> >>>>>> // other constructors omitted must also check invarients >>>>>> } >>>>>> >>>>>> class B extends A { >>>>>> >>>>>> public final int cur; >>>>>> >>>>>> private static ReadSerial check(ReadSerial rs) { >>>>>> A a = new A(rs); >>>>> >>>>> >>>>> It doesn't seem practical in general to have to create an instance of >>>>> your supertype to validate the passed in serialized form. Why not >>>>> expose the check method? >>>>> >>>>> David >>>>> ----- >>>>> >>>>>> int cur = rs.getInt("cur"); >>>>>> if ( a.lower > cur || cur > a.upper ) >>>>>> throw new IllegalArgumentException( >>>>>> "cur outside lower and upper bounds"); >>>>>> return rs; >>>>>> } >>>>>> >>>>>> public B(ReadSerial rs) { >>>>>> super(check(rs)); >>>>>> cur = rs.getInt("cur"); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>> >> > From xueming.shen at oracle.com Mon Jan 5 21:12:58 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Mon, 05 Jan 2015 13:12:58 -0800 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AAF717.3070602@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB050.4010001@Oracle.com> <54AACE94.2000802@oracle.com> <54AAD3A6.9010200@Oracle.com> <54AAE19F.4090102@oracle.com> <54AAF717.3070602@oracle.com> Message-ID: <54AAFE5A.5070406@oracle.com> On 01/05/2015 12:41 PM, Ivan Gerasimov wrote: > Thanks Sherman! > > On 05.01.2015 22:10, Xueming Shen wrote: >> >> Just wonder if we really need that "inQuotes" logic here? A straightforward approach might >> be "every time you have a quote, skip everything until you hit another one, when counting, >> or copy everything into the buffer until hit another one, when copying" ? >> > I agree it would work, but, in my opinion, it would be a bit more complicated. > The counting loop would look something like this: > ------------------------------------ > outerLoop: for (int i = 0; i < ldLen; ++i) { > char ch = ldPath.charAt(i); > if (mayBeQuoted && ch == '\"') { > thereAreQuotes = true; > for (++i; i < ldLen; ++i) { > if (ldPath.charAt(i) == '\"') { > continue outerLoop; > } > } > break; // unpaired quote > } else if (ch == ps) { > psCount++; > } > } > ------------------------------------ > which is 3 lines longer, comparing to the loop with inQuotes flag. > It does not seem like we are doing anything special for "unpaired quote" (just ignore it?), if that is the case, you probably don't need to do anything for it, the code could be something like for (int i = 0; i < ldLen; ++i) { char ch = ldPath.charAt(i); if (mayBeQuoted && ch == '\"') { thereAreQuotes = true; while (++i < ldLen && ldPath.charAt(i) != '\"') {} } else if (ch == ps) { psCount++; } } I have not tried to debug the code though :-) Just an opinion here. -Sherman From ivan.gerasimov at oracle.com Mon Jan 5 21:37:48 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Tue, 06 Jan 2015 00:37:48 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AAFE5A.5070406@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB050.4010001@Oracle.com> <54AACE94.2000802@oracle.com> <54AAD3A6.9010200@Oracle.com> <54AAE19F.4090102@oracle.com> <54AAF717.3070602@oracle.com> <54AAFE5A.5070406@oracle.com> Message-ID: <54AB042C.2010105@oracle.com> Yes, you're right, we can simply ignore the unpaired quotes. I'll incorporate your suggestion, thanks! Sincerely yours, Ivan On 06.01.2015 0:12, Xueming Shen wrote: > On 01/05/2015 12:41 PM, Ivan Gerasimov wrote: >> Thanks Sherman! >> >> On 05.01.2015 22:10, Xueming Shen wrote: >>> >>> Just wonder if we really need that "inQuotes" logic here? A >>> straightforward approach might >>> be "every time you have a quote, skip everything until you hit >>> another one, when counting, >>> or copy everything into the buffer until hit another one, when >>> copying" ? >>> >> I agree it would work, but, in my opinion, it would be a bit more >> complicated. >> The counting loop would look something like this: >> ------------------------------------ >> outerLoop: for (int i = 0; i < ldLen; ++i) { >> char ch = ldPath.charAt(i); >> if (mayBeQuoted && ch == '\"') { >> thereAreQuotes = true; >> for (++i; i < ldLen; ++i) { >> if (ldPath.charAt(i) == '\"') { >> continue outerLoop; >> } >> } >> break; // unpaired quote >> } else if (ch == ps) { >> psCount++; >> } >> } >> ------------------------------------ >> which is 3 lines longer, comparing to the loop with inQuotes flag. >> > > It does not seem like we are doing anything special for "unpaired > quote" (just ignore it?), > if that is the case, you probably don't need to do anything for it, > the code could be > something like > > for (int i = 0; i < ldLen; ++i) { > char ch = ldPath.charAt(i); > if (mayBeQuoted && ch == '\"') { > thereAreQuotes = true; > while (++i < ldLen && ldPath.charAt(i) != '\"') {} > } else if (ch == ps) { > psCount++; > } > } > > I have not tried to debug the code though :-) Just an opinion here. > > -Sherman > > From jawnsy at cpan.org Mon Jan 5 22:46:24 2015 From: jawnsy at cpan.org (Jonathan Yu) Date: Mon, 5 Jan 2015 17:46:24 -0500 Subject: String.indexOf optimization In-Reply-To: References: <534D68B8.3040002@oracle.com> <5350E316.2050406@oracle.com> Message-ID: Hi, Sharing in case you haven't seen it... Aleksey Shipil?v has a talk about String optimizations, which discusses these questions and specifically the Boyer-Moore algorithm. http://shipilev.net/talks/joker-Oct2014-string-catechism.pdf Page 85 talks about the intrinsification of indexOf and compares performance of the builtin vs intrinsified versions by using: -XX:+UnlockDiagnosticVMOptions -XX:DisableIntrinsic=::_indexOf There might also be a video recording somewhere. Jonathan On Mon, Jan 5, 2015 at 2:31 PM, Martin Buchholz wrote: > Evidence that hotspot tries to intrinsify String.indexOf: > > do_intrinsic(_indexOf, > java_lang_String, indexOf_name, string_int_signature, > F_R) \ > do_name( indexOf_name, "indexOf") > \ > > So work would have to be done at the hotspot intrinsics level (not easy!) > > Also, the problem is that Boyer-Moore is a fundamental improvement in > string searching, but its overhead is high enough that it's unlikely to > help with typical input strings found in the Real World. I think we would > want to split into two implementations and only do Boyer-Moore if it looks > profitable. Similarly for j.u.r.Pattern's regex compiler. > > I still think it's sufficiently difficult that effort is best applied > elsewhere. > > > On Mon, Jan 5, 2015 at 10:59 AM, Zoltan Sziladi > wrote: > > > Hi, > > > > This discussion was a long time ago, I was just reading through it to > > check again what was the last state of the discussion about the > > String.indexOf. > > There is one part which I still do not understand, hopefully someone > could > > shed some light on it. A few emails ago Martin mentioned > > > > "Hotspot seems to have some intrinsification of String.indexOf, which > > confuses me. > > Hotspot seems the right place to provide more optimizations for this, > > since there has been a fair amount of work creating high-performance > > low-level implementations of this idea in C." > > > > Then Ivan asked what that actually meant, whether hotspot actually > > replaced the jdk implementation with a low level optimized C > > implementation, but I never saw an answer to that. > > > > Can someone please explain this? If we somehow found an algorithm that > > beat the naive implementation in the average case, would it be possible > to > > just implement it in the JDK? Also, is there a test set which we could > > consider conclusive enough to actually change the implementation based on > > results from that? (For example if I create an implementation that beats > > the naive algorithm in those testcases, then we could consider it faster > in > > average case) > > > > Thanks! > > > > Zoltan > > > > On Fri, Apr 18, 2014 at 9:43 AM, Martin Buchholz > > wrote: > > > >> I remain skeptical that modifying the implementation of static package > >> private String.indexOf is the right approach. > >> > >> If we can produce high-performance implementations of these, perhaps > >> making > >> them public in Arrays.java is the right way. > >> > >> 1766 if (targetCount == 1) {1767 return (i > >> <= max) ? i - sourceOffset : -1; > >> > >> > >> If you're going to special case targetCount == 1, you shouldn't have a > >> test > >> for it in the main loop, since you slow down the general case. Instead, > >> you can sequester the special cases like this: > >> > >> if (targetCount <= 1) { > >> if (targetCount == 0) ... > >> else ... > >> } > >> > >> // now assume targetCount >= 2 > >> > >> > >> On Fri, Apr 18, 2014 at 1:32 AM, Ivan Gerasimov > >> wrote: > >> > >> > > >> > On 16.04.2014 2:53, Martin Buchholz wrote: > >> > > >> > Hi Ivan, > >> > > >> > There's already an indexOf(int, int) that allows a user to explicitly > >> > search for a char (or character). > >> > > >> > Sure. > >> > What I meant was not to introduce a special function for searching a > >> > single character, but to restrict the general case loop to search for > a > >> > substring of at least 2 characters. > >> > Having this condition hold, we can enter the loop only when two > starting > >> > characters match, and this can save us a few nanoseconds in many > cases. > >> > > >> > > >> > Hotspot seems to have some intrinsification of String.indexOf, which > >> > confuses me. > >> > > >> > > >> > Does it mean that that the java implementation of indexOf is never > >> > compiled? > >> > When hotspot replaces the jdk implementation with its own one? > >> > Is it ever worth to try to optimize the java implementation? > >> > > >> > > >> > Hotspot seems the right place to provide more optimizations for this, > >> > since there has been a fair amount of work creating high-performance > >> > low-level implementations of this idea in C. > >> > > >> > The hotspot's intrinsic is already optimized for searching substrings > >> of > >> > different length. > >> > > >> > Sincerely yours, > >> > Ivan > >> > > >> > > >> > > >> > On Tue, Apr 15, 2014 at 10:13 AM, Ivan Gerasimov < > >> > ivan.gerasimov at oracle.com> wrote: > >> > > >> >> Hi everyone! > >> >> > >> >> > >> >> On 04.04.2014 21:13, Martin Buchholz wrote: > >> >> > >> >> Summary: > >> >> > >> >> Many people (myself included) have looked at this problem. It's > >> unlikely > >> >> that String.indexOf will change. It's hard to beat the naive > >> >> implementation in the typical case. > >> >> > >> >> But we can try to speed up this naive implementation a little bit. > >> >> > >> >> We can separate the special case: When the substring's length == 1. > >> >> This special case can be done fast, and in the general case we can > now > >> >> assume substring's length is at least 2. > >> >> > >> >> Here's the webrev with the implementation of this idea: > >> >> http://cr.openjdk.java.net/~igerasim/indexof/0/webrev/ > >> >> > >> >> I've done some benchmarking with JMH and found no performance > >> degradation > >> >> on my machine. > >> >> Of course, more testcases should be created and they should be tried > on > >> >> different machines to be treated as reliable. > >> >> > >> >> Benchmark Mode Thr Cnt Sec > Mean > >> >> Mean error Units > >> >> o.b.IndexOfBench.benchIndexOf_1_A avgt 1 20 5 > >> >> 704.739 9.104 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_1_B avgt 1 20 5 * > >> >> 573.879* 9.820 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_2_A avgt 1 20 5 > >> >> 668.455 9.882 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_2_B avgt 1 20 5 > >> >> *476.062* 6.063 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_3_A avgt 1 20 5 > >> >> 155.227 1.796 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_3_B avgt 1 20 5 * > >> >> 152.850 * 1.512 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_4_A avgt 1 20 5 > >> >> 656.183 5.904 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_4_B avgt 1 20 5 > >> >> *515.178* 9.107 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_5_A avgt 1 20 5 > >> >> 140.609 7.305 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_5_B avgt 1 20 5 > >> >> *129.603* 1.654 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_6_A avgt 1 20 5 > >> >> 127.713 1.497 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_6_B avgt 1 20 5 > >> >> *122.177* 1.186 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_7_A avgt 1 20 5 > >> >> 430.148 4.875 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_7_B avgt 1 20 5 * > >> >> 387.338* 10.904 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_8_A avgt 1 20 5 > >> >> 2064.563 28.885 nsec/op > >> >> o.b.IndexOfBench.benchIndexOf_8_B avgt 1 20 5 > >> >> *1858.669* 24.343 nsec/op > >> > >> >> > >> >> Benchmarks ending with A use the current indexOf implementation, > with B > >> >> use the modified version. > >> >> These numbers show from 1.4% up to 28% performance increase. > >> >> > >> >> The full listing is below. > >> >> > >> >> Suggestions about what else to test are welcome! > >> >> > >> >> Sincerely yours, > >> >> Ivan > >> >> > >> >> --------------------- > >> >> > >> >> /** > >> >> * Copyright (c) 2014, Oracle and/or its affiliates. All rights > >> reserved. > >> >> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. > >> >> * > >> >> * This code is free software; you can redistribute it and/or modify > it > >> >> * under the terms of the GNU General Public License version 2 only, > as > >> >> * published by the Free Software Foundation. Oracle designates this > >> >> * particular file as subject to the "Classpath" exception as > provided > >> >> * by Oracle in the LICENSE file that accompanied this code. > >> >> * > >> >> * This code is distributed in the hope that it will be useful, but > >> >> WITHOUT > >> >> * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY > >> or > >> >> * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public > >> License > >> >> * version 2 for more details (a copy is included in the LICENSE file > >> that > >> >> * accompanied this code). > >> >> * > >> >> * You should have received a copy of the GNU General Public License > >> >> version > >> >> * 2 along with this work; if not, write to the Free Software > >> Foundation, > >> >> * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. > >> >> * > >> >> * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA > 94065 > >> USA > >> >> * or visit www.oracle.com if you need additional information or > have > >> any > >> >> * questions. > >> >> */ > >> >> package org.benches; > >> >> > >> >> import org.openjdk.jmh.annotations.*; > >> >> import org.openjdk.jmh.logic.BlackHole; > >> >> import java.util.concurrent.TimeUnit; > >> >> > >> >> @BenchmarkMode(Mode.AverageTime) > >> >> @OutputTimeUnit(TimeUnit.NANOSECONDS) > >> >> @State(Scope.Benchmark) > >> >> public class IndexOfBench { > >> >> > >> >> > >> >> // > >> >> ||| > >> >> final static char[] source1 = > >> >> > >> > "abababcabcacabcabcabcabcaccbcabcacbcabcacbcbcabcbcbacbcabcbabacbcbacbcabcabcabcabcabcabcabcacbacbacbacabcabcacbacbcabcbcbcaabdacbacabcabacbacabca".toCharArray(); > >> >> final static char[] source2 = > >> >> > >> > "acfacafacfacfacfacafcafcacadcacdacaccacacdacacfcafcafcfacdacadcadcadcdacfacfacdacadcacdcfacfacdacdacdcfacdacdacdacgshgshasdabdahghjgwdshacavcavsc".toCharArray(); > >> >> final static char[] source3 = > >> >> > >> > "tyrtytfytytuytfytuytggfghgdytyftytfdytdshfgjhdfytsfuythgsfhgjhfghtuysdfthgfsdhgystfjhgsfguysthgfgjhgdfjhgsjdghfythgsdfjhggfabduikjhfjhkjhfkjhgkjh".toCharArray(); > >> >> final static char[] target1 = "abd".toCharArray(); > >> >> > >> >> final static char[] source4 = > >> >> > >> > "ahhhahahahahhahahahahhahahahhhahahahahahahahahahahhahahahahahahahahallallalalalalalkakakakakakakakakkakmamamamabammamamamamamaakaklalalaoalalalao".toCharArray(); > >> >> final static char[] source5 = > >> >> > >> > "hgjkhjhjkdghkjhdfkjhgkjhdkjdhgkjdfhgkjdhfgkjdfhgkjhdfkjghkdjghkdjfhgkjhkdjhgkjdfhjkghkdjfhgkjdfhgkjdfhgkjdfhkgabhfkjghdkfjhgkjdfhgkjdfhgkjdfhgkhh".toCharArray(); > >> >> final static char[] target2 = "ab".toCharArray(); > >> >> > >> >> final static char[] source6 = > >> >> > >> > "lskgjsklfjgskldfjgklsfjdlgkjsdflkgjskldfgjsdklfgjsl;kdfgjskldfjglksdfjglksfjglksdfjgklsfdjgslkdfjglksjdflkgsjfalksjdflkfgjsdklfjglskdfjglksdfjghh".toCharArray(); > >> >> final static char[] target3 = "a".toCharArray(); > >> >> > >> >> final static char[] source7 = > >> >> > >> > "lskgajabfagskldfjgklsabclgkjsdflkabsabcdgjsdklfabclbkdfgjskabfjglksdfjabcdfjglabcfjgklsfabgslkdfjglksjdabcdsjfabcdedflabcjsdklfjglskdfabcksdfjghh".toCharArray(); > >> >> final static char[] target4 = "abcde".toCharArray(); > >> >> > >> >> final static char[] source8 = > >> >> > >> > "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".toCharArray(); > >> >> final static char[] target5 = "aaaab".toCharArray(); > >> >> > >> >> @GenerateMicroBenchmark > >> >> public void benchIndexOf_1_A(BlackHole bh) { > >> >> bh.consume(indexOfA(source1, 0, source1.length, target1, 0, > >> >> target1.length, 0)); > >> >> } > >> >> > >> >> @GenerateMicroBenchmark > >> >> public void benchIndexOf_1_B(BlackHole bh) { > >> >> bh.consume(indexOfB(source1, 0, source1.length, target1, 0, > >> >> target1.length, 0)); > >> >> } > >> >> > >> >> @GenerateMicroBenchmark > >> >> public void benchIndexOf_2_A(BlackHole bh) { > >> >> bh.consume(indexOfA(source2, 0, source2.length, target1, 0, > >> >> target1.length, 0)); > >> >> } > >> >> > >> >> @GenerateMicroBenchmark > >> >> public void benchIndexOf_2_B(BlackHole bh) { > >> >> bh.consume(indexOfB(source2, 0, source2.length, target1, 0, > >> >> target1.length, 0)); > >> >> } > >> >> > >> >> @GenerateMicroBenchm > >> >> > >> > > >> > > >> > > > > > From joe.darcy at oracle.com Mon Jan 5 23:58:45 2015 From: joe.darcy at oracle.com (Joseph D. Darcy) Date: Mon, 05 Jan 2015 15:58:45 -0800 Subject: JDK 9 RFR of 4026465: Provide more byte array constructors for BigInteger In-Reply-To: <021DF7FC-C49F-4A66-B534-6F38499E64C1@oracle.com> References: <48844110-12EF-48E2-B536-FF690C319CDE@oracle.com> <851F1C62-D645-4ACA-833B-D890E26FBEA0@oracle.com> <54A35C56.3040908@oracle.com> <54A66B8D.8070708@oracle.com> <021DF7FC-C49F-4A66-B534-6F38499E64C1@oracle.com> Message-ID: <54AB2535.20804@oracle.com> Hi Brian, The new version looks fine. One suggestion to consider: creating a small private helper method to do the len, off, array-length check. (The two copies of the logic are slightly different.) Thanks, -Joe On 1/2/2015 4:09 PM, Brian Burkhalter wrote: > Hello all, > > Thanks for the comments. A new patch is here: > > http://cr.openjdk.java.net/~bpb/4026465/webrev.02/ > > On Dec 30, 2014, at 6:15 PM, joe darcy wrote: > >> The new changes generally look good. A few comments, for the new code like >> >> 291 } else if ((off < 0) || (off > val.length) || (len < 0) || >> 292 ((off + len) > val.length) || ((off + len) < 0)) { >> 293 throw new IndexOutOfBoundsException(); >> >> it is not immediately clear that the arithmetic on line 292 won't have some inappropriate overflow behavior. A comment stating why "off + len" will behave appropriately (assuming it does behave appropriately ;-) would help. (By line 292, both off and len are non-negative so that should limit the case analysis.) > Logic updated. > >> It might be worthwhile for all the BigInteger constructors which take array arguments to state something about the thread-safely behavior ("arrays are assumed to be unchanged for the duration of the constructor call...?). > Verbiage added. > >> Do have have any code coverage information for the new code by the regression test? It would be good to know whether or not all the guard conditions are properly being executed. > No coverage information, but I added some tests for the guard conditions and slightly changed the correct-value part of the test. > > On Dec 30, 2014, at 6:42 PM, Paul Benedict wrote: > >> Please add @since 1.9 to the new constructors. > Done. > > On Jan 2, 2015, at 1:57 AM, Alan Bateman wrote: > >> I assume this can be reduced down to: >> if (off < 0 || len < 0 || (off > val.length - len)) { ... } > > But then if len > val.length, the third inequality tests ?off? for being greater than a negative value. Please see the updated patch. > > Thanks, > > Brian From brian.burkhalter at oracle.com Tue Jan 6 00:41:18 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 5 Jan 2015 16:41:18 -0800 Subject: JDK 9 RFR of 4026465: Provide more byte array constructors for BigInteger In-Reply-To: <54AB2535.20804@oracle.com> References: <48844110-12EF-48E2-B536-FF690C319CDE@oracle.com> <851F1C62-D645-4ACA-833B-D890E26FBEA0@oracle.com> <54A35C56.3040908@oracle.com> <54A66B8D.8070708@oracle.com> <021DF7FC-C49F-4A66-B534-6F38499E64C1@oracle.com> <54AB2535.20804@oracle.com> Message-ID: Hi Joe, The logic differs because one of the cases does not allow for zero length whereas the other does. Thanks, Brian On Jan 5, 2015, at 3:58 PM, Joseph D. Darcy wrote: > The new version looks fine. One suggestion to consider: creating a small private helper method to do the len, off, array-length check. (The two copies of the logic are slightly different.) From daniel.daugherty at oracle.com Tue Jan 6 01:12:36 2015 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 05 Jan 2015 18:12:36 -0700 Subject: RFR(S): 6977426: sun/tools tests can intermittently fail to find app's Java pid In-Reply-To: <5491810F.6030900@oracle.com> References: <5490564F.2010907@oracle.com> <549060CE.5080505@oracle.com> <54914897.2050006@oracle.com> <54915304.4080503@oracle.com> <54916391.9010307@oracle.com> <54916A0A.6080508@oracle.com> <5491810F.6030900@oracle.com> Message-ID: <54AB3684.1000300@oracle.com> > The default JTreg time out is 5 minutes... I believe the default JTREG timeout is 120 seconds or 2 minutes. Dan On 12/17/14 6:11 AM, Yekaterina Kantserova wrote: > > Hopefully the last version :) > > Erik has recommended to skip the whole timeout concept. Instead the > test will loop until a decreasing count is found. Otherwise the test > will timeout. The default JTreg time out is 5 minutes so it would be > enough even for slower configurations. > > The new webrev can be found here: > http://cr.openjdk.java.net/~ykantser/6977426/webrev.02/ > > // Katja > > > > On 12/17/2014 12:33 PM, Daniel Fuchs wrote: >> On 17/12/14 12:05, Yekaterina Kantserova wrote: >>> Daniel, >>> >>> It's really funny to get such a feedback! >>> >>> (1) The output from test right now looks like: >>> >>> ----------System.out:(7/183)---------- >>> call count = 1000 >>> instance count: 1001 >>> call count = 2000 >>> instance count: 1001 >>> Finishing early due to non-increasing instance count >>> increasing count: 1 >>> decreasing or the same count: 1 >>> >>> so printing the value of 'count' is already in there: 'call count ='. >> >> I meant to print the final value for count - but OK - having the >> 'call count' printed every 1000 count should be enough :-) >> >> >>> (2) In the case we want to have an extra condition I think it should be >>> 'count > 2000'. Otherwise if the end time is passed the 'count > 1000' >>> will result in having just one iteration (we will left the loop on >>> 1100) >>> and test will fail on assert. >> >> right. my mistake. you need at least two histograms. >> >>> The test timeout value should be increased as well. Right now it's >>> 120 + >>> 30 sec. >>> >>> * @run main/timeout=150 TestLoggerWeakRefLeak Logger >>> >>> If the test will not be completed in 120 s the extra 30 s will not make >>> the trick and JTreg will interrupt it. What do you think will be >>> enough? >> >> I am not sure - but I guess what you have now should be good enough >> for starter. In the problematic configurations there often is >> an additional timeout factor - so in such case the real jtreg timeout >> would I believe be something like 150*4 anyway - which should hopefully >> leave time enough for the while loop to take two histograms. Simply >> adding the extra condition to keep looping until we have at least two >> should do the trick. >> >> best regards, and thanks again for taking care of 6977426! >> >> -- daniel >> >>> >>> // Katja >>> >>> >>> >>> On 12/17/2014 10:55 AM, Daniel Fuchs wrote: >>>> Hi Katja, >>>> >>>> Sorry for not thinking of that when I replied earlier. >>>> Your new test is so much more readable than the old shell >>>> script :-) >>>> >>>> These are minor suggestions but they might help analysis >>>> if the test fails: >>>> >>>> On 17/12/14 10:10, Yekaterina Kantserova wrote: >>>>> Hi, >>>>> >>>>> Thanks for all reviews! >>>>> >>>>> The new webrev can be found here: >>>>> http://cr.openjdk.java.net/~ykantser/6977426/webrev.01/ >>>> >>>> Not that I believe it's very likely to happen, but I wonder >>>> if the condition to exit the while loop: >>>> >>>> 79 while (now < (startTime + (LOOP_TIME_SEC * 1000))) { >>>> >>>> should also make sure that count > 1000 - and continue if not. >>>> (Thinking of passed timeout related issues when tests where run >>>> with -Xcomp on fastdebug builds during integration nightlies) >>>> >>>> Possibly printing the value of 'count' before the assert >>>> at line 103 could help with debugging too, if the test actually >>>> fails (even though the combination of increasingCount >>>> + decreasingCount would give us a hint) >>>> >>>> best regards, >>>> >>>> -- daniel >>>> >>>>> >>>>> // Katja >>>>> >>>>> >>>>> >>>>> On 12/16/2014 05:41 PM, Jaroslav Bachorik wrote: >>>>>> Hi Katja, >>>>>> >>>>>> This request should probably go to core-libs as well. >>>>>> >>>>>> Other than that I have just a few nits: >>>>>> >>>>>> test/java/util/logging/TestLoggerWeakRefLeak.java >>>>>> >>>>>> L57: if ("Logger".contains(args[0])) -> "Logger".equals(args[0]) ? >>>>>> L127: // in LogManager.loggers that *arebeing* properly managed -> >>>>>> *are being* >>>>>> >>>>>> callLoggerAndCount() and callAnonymousLoggerAndCount() are using the >>>>>> 'count' variable to always return 100. Could they be made to >>>>>> return a >>>>>> constant instead? >>>>>> >>>>>> >>>>>> Cheers, >>>>>> >>>>>> -JB- >>>>>> >>>>>> >>>>>> On 12/16/2014 04:57 PM, Yekaterina Kantserova wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Could I please have a review of this fix. >>>>>>> >>>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-6977426 >>>>>>> webrev: http://cr.openjdk.java.net/~ykantser/6977426/webrev.00/ >>>>>>> >>>>>>> java/util/logging/LoggerWeakRefLeak.sh and >>>>>>> java/util/logging/AnonLoggerWeakRefLeak.sh are merged and >>>>>>> rewritten in >>>>>>> java. sun/tools/common/CommonTests.sh is removed because it doesn't >>>>>>> test >>>>>>> the product but sun/tool/common library functionality. >>>>>>> >>>>>>> Thanks, >>>>>>> Katja >>>>>> >>>>> >>>> >>> >> > > From joe.darcy at oracle.com Tue Jan 6 01:43:26 2015 From: joe.darcy at oracle.com (Joseph D. Darcy) Date: Mon, 05 Jan 2015 17:43:26 -0800 Subject: JDK 9 RFR of 8067669: Documentation for methods in Number incomplete regarding too large values. In-Reply-To: <3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com> References: <20141216125508.GC4911@e6430> <9D55A146-D76D-4E91-B02D-2BC646451CE2@oracle.com> <20141217102612.GE4911@e6430> <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com> <5494B41B.8060003@oracle.com> <0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com> <3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com> Message-ID: <54AB3DBE.5040806@oracle.com> Hello, Getting back to this issue in the new year, taking a closer look at the existing class-level documentation, I don't really see a compelling case for an edit this large. I for one would be content if the bug were closed as "not an issue." HTH, -Joe On 12/22/2014 10:46 AM, Brian Burkhalter wrote: > Please review at your convenience: > > Issue: https://bugs.openjdk.java.net/browse/JDK-8067669 > Patch: http://cr.openjdk.java.net/~bpb/8067669/webrev.00/ > > Summary: > > Add some verbiage. > > I would advocate either changing the verbiage either to the content in the patch or something better, or resolving the issue as ?Not an Issue.? > > Thanks, > > Brian > > On Dec 19, 2014, at 3:34 PM, Brian Burkhalter wrote: > >> On Dec 19, 2014, at 3:26 PM, joe darcy wrote: >> >>> I don't really think the current text is problematic; however, if it is to be changed, I recommending including a citation to the "Narrowing primitive conversion" section of the Java Language Specification. This can be accomplished using the @jls javadoc tag; the syntax is something like >>> >>> * @jls 5.1.3. Narrowing Primitive Conversion >>> >>> Examples of the use of @jls can be found elsewhere in the core libraries. >> This is already in the class level documentation which is why I left it out. No harm adding it to the method docs however. >> >>> For the exact wording how about >>> >>> ... The particular semantics of the conversion operation from the specified number to a {@code long} are defined in subclasses. The operation may involve a narrowing conversion, rounding, or truncation. >> Seems reasonable to me. From brian.burkhalter at oracle.com Tue Jan 6 01:47:02 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 5 Jan 2015 17:47:02 -0800 Subject: JDK 9 RFR of 8067669: Documentation for methods in Number incomplete regarding too large values. In-Reply-To: <54AB3DBE.5040806@oracle.com> References: <20141216125508.GC4911@e6430> <9D55A146-D76D-4E91-B02D-2BC646451CE2@oracle.com> <20141217102612.GE4911@e6430> <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com> <5494B41B.8060003@oracle.com> <0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com> <3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com> <54AB3DBE.5040806@oracle.com> Message-ID: <7D01248E-2405-4F4B-B3A4-F16267A18F81@oracle.com> I personally have no objection but I cannot speak for the others who posted to the discussion thread http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-December/030376.html Thanks, Brian On Jan 5, 2015, at 5:43 PM, Joseph D. Darcy wrote: > Hello, > > Getting back to this issue in the new year, taking a closer look at the existing class-level documentation, I don't really see a compelling case for an edit this large. > > I for one would be content if the bug were closed as "not an issue." > > HTH, > > -Joe > > On 12/22/2014 10:46 AM, Brian Burkhalter wrote: >> Please review at your convenience: >> >> Issue: https://bugs.openjdk.java.net/browse/JDK-8067669 >> Patch: http://cr.openjdk.java.net/~bpb/8067669/webrev.00/ >> >> Summary: >> >> Add some verbiage. >> >> I would advocate either changing the verbiage either to the content in the patch or something better, or resolving the issue as ?Not an Issue.? >> >> Thanks, >> >> Brian From david.holmes at oracle.com Tue Jan 6 02:10:55 2015 From: david.holmes at oracle.com (David Holmes) Date: Tue, 06 Jan 2015 12:10:55 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54AA7D2F.1000703@zeus.net.au> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> Message-ID: <54AB442F.9070800@oracle.com> >> On 5/01/2015 6:51 PM, Peter Firmstone wrote: >>> Thanks Dave, >>> >>> That's another way of checking invariants, you could expose A's check >>> method but you'd also need a couple of additional static methods to >>> obtain integers upper and lower from ReadSerial, so that B can ensure >>> its invariant. Not sure I follow. Currently you do "new A(rs)" which internally calls "A.check(rs)" and I was suggesting you call A.check(rs) directly so avoiding the need to create an A just for validation purposes, and which is subsequently discarded. David >>> >>> ReadSerial is caller sensitive so B can't obtain A's stream values via >>> ReadSerial and must do so via A's API. This prevents the publication of >>> A's implementation, you don't wan't B depending on A's serial form. >>> Instead A can reorder and rename it's fields and change their values, >>> have multiple serial forms and be able to remain compatible with all >>> forms relatively easily. >>> >>> There are some benefits to using a temporary object instance of A; it >>> reduces the amount of code required, eg it would be a beneficial for RMI >>> to minimise code downloads, the JVM is free to inline access to these >>> fields and the temporary instance of A never leaves the cpu cache, so >>> it's not likely to become a performance issue. Well it might on real >>> time java :) >>> >>> If circular relationships are mutable, or effectively mutable after >>> publication, then we could eventually deprecate the requirement to >>> support special treatment of final fields for Serializable. >>> >>> Cheers, >>> >>> Peter. >>> >>> On 5/01/2015 10:38 AM, David Holmes wrote: >>>> >>>>>> - I don't see how this invariant-checking mechanism can enforce >>>>>> invariants between superclass fields and subclass fields. For >>>>>> example: >>>>>> >>>>>> class A { >>>>>> int lower, upper; // invariant: lower <= upper >>>>>> } >>>>>> >>>>>> class B extends A { >>>>>> int cur; // invariant: lower <= cur <= upper >>>>>> } >>>>>> >>>>> >>>>> The serialization framework should only construct an objects fields >>>>> and make these available from ReadSerial, each class then calls a >>>>> static method before calling a superclass constructor that's >>>>> responsible for an objects creation, to prevent construction of the >>>>> object, if necessary. >>>>> >>>>> Eg, some complexity, but bullet proof: >>>>> >>>>> public class A ( >>>>> >>>>> public final int lower, upper; >>>>> >>>>> private static boolean check(ReadSerial rs){ >>>>> if (rs.getInt("lower") > rs.getInt("upper")) >>>>> throw new IllegalArgumentException( >>>>> "lower bound must be less than or equal to upper"); >>>>> return true; >>>>> } >>>>> >>>>> public A(ReadSerial rs){ >>>>> this(check(rs), rs); >>>>> } >>>>> >>>>> private A(boolean checked, ReadSerial rs){ >>>>> super(); >>>>> lower = rs.getInt("lower"); >>>>> upper = rs.getInt("upper"); >>>>> } >>>>> >>>>> // other constructors omitted must also check invarients >>>>> } >>>>> >>>>> class B extends A { >>>>> >>>>> public final int cur; >>>>> >>>>> private static ReadSerial check(ReadSerial rs) { >>>>> A a = new A(rs); >>>> >>>> >>>> It doesn't seem practical in general to have to create an instance of >>>> your supertype to validate the passed in serialized form. Why not >>>> expose the check method? >>>> >>>> David >>>> ----- >>>> >>>>> int cur = rs.getInt("cur"); >>>>> if ( a.lower > cur || cur > a.upper ) >>>>> throw new IllegalArgumentException( >>>>> "cur outside lower and upper bounds"); >>>>> return rs; >>>>> } >>>>> >>>>> public B(ReadSerial rs) { >>>>> super(check(rs)); >>>>> cur = rs.getInt("cur"); >>>>> } >>>>> } >>>>> >>>>> >>> > From peter.levart at gmail.com Tue Jan 6 07:27:00 2015 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 06 Jan 2015 08:27:00 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <54AAF963.2020402@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> Message-ID: <54AB8E44.6010905@gmail.com> Hi Brian, On 01/05/2015 09:51 PM, Brian Goetz wrote: > Thanks Peter(s): I think it's just about time to re-sync on the goals, > since there are an infinitude of issues with serialization, only some > of which we can address. > > To that end, let me toss out some of the results of our recent > explorations with improving serialization, where we started out on > something not unlike what Peter is suggesting, and dialed back. > > One thing we can do to positively improve the serialization experience > is to reduce the set of situations where users have to write explicit > readObject() methods. In the JDK, we end up having to do so for all > manner of security-motivated reasons, and such code is hard to write > correctly, hard to validate, hard to audit, and hard to evolve. > > After an audit of our codebase (thanks Chris!), we identified some of > the biggest offenders on this score: > > - Validating invariants > - Ensuring confinement > > The latter comes up in cases where there's a field: > > class Foo { > private final Bar bar = new BarImpl(); > } > > which implicitly has an invariant: the Bar referenced by Foo.bar is a) > exactly of type BarImpl and not some other subclass and b) there is > exactly one reference to that BarImpl (assuming it doesn't escape Foo.) > > Note that both of these still use the default deserialization; having > to override readObject() just to do some extra checking is error-prone > and messy. So I think the goal of a separate validation method which > is called automatically is a good one, since it means we can ditch > some percentage of otherwise-needless readObject implementations > (which then have to jump through hoops to set final fields.) readObject() can be used on both ends of the spectrum or anywhere in-between. It can be used for explicit deserialization or just for validation of default deserialized state. Would separate validation method give us anything more or simplify things? readObject() can be used just as: private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { in.defaultReadObject(); ... just validation ... } > > I agree that being able to play nicely with final fields is also an > important goal here. Explicit deserialization and final fields don't play nicely together currently, yes. Sometimes data-racey-publication thread-safety is sacrificed just because explicit deserialization would complicate code too much by using reflection. Regards, Peter > > On 1/5/2015 9:11 AM, Peter Levart wrote: >> Hi Peter and others, >> >> A assume your approach described here (to use constructors for >> deserialization) is motivated mainly by the fact that only constructors >> are allowed to set the final fields. Otherwise the explicit features of >> your ReadSerial API are more or less accessible even now by using >> "serialPersistentFields " static field for declaration of "serialized" >> fields, combined with ObjectOutputStream.PutField and >> ObjectInputStream.GetField API, accessible from writeObject/readObject >> methods. readObject() method already acts like a constructor, whith the >> following notable differences: >> >> 1 - it is not chained explicitly to superclass (either in source or >> automatically by javac) like constructor, but is invoked by >> (de)serialization infrastructure in correct order for each class in the >> hierarchy that declares it. >> 2 - it is a normal method, so it can be called at any time by code in >> the declaring class or inner classes. >> 3 - it can't set final fields (without using clumsy reflection) >> >> >> The 1st point is actually in favor of readObject() method, since the >> (de)serialization API need not be caller-sensitive. The correct >> per-class context is established before each readObject call-back. >> 2nd and 3rd points are readObject drawbacks and are both interconnected. >> If readObject was not universally callable, then the restriction that it >> can't set final fields could be lifted. >> >> Perhaps it is too late from compatibility perspective to change the >> readObject's 2nd and 3rd point now, but we could compatibly change the >> 1st point of a special kind of constructor. >> >> Say that a constructor declared with exactly one parameter of type >> ObjectInputStream is treated specially by javac and reflection: >> - constructor code can't chain to superclass constructor >> - javac does not automatically generate default constructor chaining >> code >> - javac does not allow calling such constructor from any code >> - this is optional: javac does not allow calling instance methods and >> explicit use of 'this' keyword in deserialization constructor code >> (preventing escape of such object before de-serialization ends) - to be >> really safe, this would probably have to be accompanied with verifier >> checks too. >> - public reflection API does not allow invoking such constructor (only >> deserialization infrastructure can - like it can invoke the default >> constructor of a non-Serializable superclass on an already allocated >> instance of a sublclass) >> >> No additional changes to VM are apparently necessary (except verifier). >> Declaring such constructor as an alternative to readObject() method is >> then possible to make deserialization more safe. >> >> Regarding finalizability (and finalizer attacks): >> >> The serialization specifies that for 1st (most specific) >> non-Serializable superclass of a Serializable subclass, default >> constructor is invoked to initialize the non-Serializable superclass >> state before deserializing subclass state. As each Serializable class >> has such a non-Serializable superclass (at least in the form of Object >> class) and normal constructors are chained, it is apparent that Object's >> default constructor is called before any deserialization of state >> begins. As soon as Object's constructor completes, the instance becomes >> eligible for finalization: >> >> "An object o is not finalizable until its constructor has invoked the >> constructor for Object on o and that invocation has completed >> successfully (that is, without throwing an exception).? >> >> The rule of calling non-Serializable superclass default constructor >> could be weakened. If 1st (most specific) non-Serializable superclass of >> a Serializable subclass is Object, then it's constructor is not called. >> Obviously it is not needed since Object has no state to initialize. What >> we achieve by this is that such object does not become eligible for >> finalization just yet. Serialization infrastructure must then explicitly >> register such objects for finalization, but only after their >> de-serialization completes normally. This would prevent finalization >> attacks. >> >> Regards, Peter >> >> >> On 01/05/2015 01:01 PM, Peter Firmstone wrote: >>> My mistake, thank you. >>> >>> Peter. >>> >>> On 5/01/2015 9:57 PM, David Holmes wrote: >>>> Hi Peter, >>>> >>>> Did you send this to the list? I haven't seen it turn up yet. >>>> >>>> David >>>> >>>> On 5/01/2015 6:51 PM, Peter Firmstone wrote: >>>>> Thanks Dave, >>>>> >>>>> That's another way of checking invariants, you could expose A's check >>>>> method but you'd also need a couple of additional static methods to >>>>> obtain integers upper and lower from ReadSerial, so that B can ensure >>>>> its invariant. >>>>> >>>>> ReadSerial is caller sensitive so B can't obtain A's stream values >>>>> via >>>>> ReadSerial and must do so via A's API. This prevents the >>>>> publication of >>>>> A's implementation, you don't wan't B depending on A's serial form. >>>>> Instead A can reorder and rename it's fields and change their values, >>>>> have multiple serial forms and be able to remain compatible with all >>>>> forms relatively easily. >>>>> >>>>> There are some benefits to using a temporary object instance of A; it >>>>> reduces the amount of code required, eg it would be a beneficial for >>>>> RMI >>>>> to minimise code downloads, the JVM is free to inline access to these >>>>> fields and the temporary instance of A never leaves the cpu cache, so >>>>> it's not likely to become a performance issue. Well it might on real >>>>> time java :) >>>>> >>>>> If circular relationships are mutable, or effectively mutable after >>>>> publication, then we could eventually deprecate the requirement to >>>>> support special treatment of final fields for Serializable. >>>>> >>>>> Cheers, >>>>> >>>>> Peter. >>>>> >>>>> On 5/01/2015 10:38 AM, David Holmes wrote: >>>>>> >>>>>>>> - I don't see how this invariant-checking mechanism can >>>>>>>> enforce >>>>>>>> invariants between superclass fields and subclass fields. For >>>>>>>> example: >>>>>>>> >>>>>>>> class A { >>>>>>>> int lower, upper; // invariant: lower <= upper >>>>>>>> } >>>>>>>> >>>>>>>> class B extends A { >>>>>>>> int cur; // invariant: lower <= cur <= upper >>>>>>>> } >>>>>>>> >>>>>>> >>>>>>> The serialization framework should only construct an objects fields >>>>>>> and make these available from ReadSerial, each class then calls a >>>>>>> static method before calling a superclass constructor that's >>>>>>> responsible for an objects creation, to prevent construction of the >>>>>>> object, if necessary. >>>>>>> >>>>>>> Eg, some complexity, but bullet proof: >>>>>>> >>>>>>> public class A ( >>>>>>> >>>>>>> public final int lower, upper; >>>>>>> >>>>>>> private static boolean check(ReadSerial rs){ >>>>>>> if (rs.getInt("lower") > rs.getInt("upper")) >>>>>>> throw new IllegalArgumentException( >>>>>>> "lower bound must be less than or equal to >>>>>>> upper"); >>>>>>> return true; >>>>>>> } >>>>>>> >>>>>>> public A(ReadSerial rs){ >>>>>>> this(check(rs), rs); >>>>>>> } >>>>>>> >>>>>>> private A(boolean checked, ReadSerial rs){ >>>>>>> super(); >>>>>>> lower = rs.getInt("lower"); >>>>>>> upper = rs.getInt("upper"); >>>>>>> } >>>>>>> >>>>>>> // other constructors omitted must also check invarients >>>>>>> } >>>>>>> >>>>>>> class B extends A { >>>>>>> >>>>>>> public final int cur; >>>>>>> >>>>>>> private static ReadSerial check(ReadSerial rs) { >>>>>>> A a = new A(rs); >>>>>> >>>>>> >>>>>> It doesn't seem practical in general to have to create an >>>>>> instance of >>>>>> your supertype to validate the passed in serialized form. Why not >>>>>> expose the check method? >>>>>> >>>>>> David >>>>>> ----- >>>>>> >>>>>>> int cur = rs.getInt("cur"); >>>>>>> if ( a.lower > cur || cur > a.upper ) >>>>>>> throw new IllegalArgumentException( >>>>>>> "cur outside lower and upper bounds"); >>>>>>> return rs; >>>>>>> } >>>>>>> >>>>>>> public B(ReadSerial rs) { >>>>>>> super(check(rs)); >>>>>>> cur = rs.getInt("cur"); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> >>>>> >>> >> From sundararajan.athijegannathan at oracle.com Tue Jan 6 07:57:40 2015 From: sundararajan.athijegannathan at oracle.com (A. Sundararajan) Date: Tue, 06 Jan 2015 13:27:40 +0530 Subject: RFR 8068462: javax.script.ScriptEngineFactory.getParameter spec is not completely consistent with the rest of the API Message-ID: <54AB9574.9000701@oracle.com> Please review http://cr.openjdk.java.net/~sundar/8068462/ for https://bugs.openjdk.java.net/browse/JDK-8068462 Thanks, -Sundar From scolebourne at joda.org Tue Jan 6 08:31:27 2015 From: scolebourne at joda.org (Stephen Colebourne) Date: Tue, 6 Jan 2015 08:31:27 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54AB8E44.6010905@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> Message-ID: On 6 January 2015 at 07:27, Peter Levart wrote: > readObject() can be used on both ends of the spectrum or anywhere > in-between. It can be used for explicit deserialization or just for > validation of default deserialized state. Would separate validation method > give us anything more or simplify things? readObject() can be used just as: > > private void readObject(ObjectInputStream in) throws IOException, > ClassNotFoundException { > in.defaultReadObject(); > ... > just validation > ... > } > Explicit deserialization and final fields don't play nicely together > currently, yes. Sometimes data-racey-publication thread-safety is sacrificed > just because explicit deserialization would complicate code too much by > using reflection. I've thought on a number of occasions that what I wanted from serializable was a merger of readObject and readResolve private Object readObjectAndResolve(ObjectInputStream in) throws IOException Using such a method, the input could be read into local variables, and then a normal constructor used to create the object itself. It avoids the final fields problem (as its just reading to local variables) and it handles the validation (by calling a regular constructor/factory). Its also more efficient in many cases, as a common pattern today is to have one object instance created by readObject (or serialization internals) and then another returned by readResolve. If readObjectAndResolve() can't handle cyclic graphs, I can live with that. Stephen From aph at redhat.com Tue Jan 6 09:28:34 2015 From: aph at redhat.com (Andrew Haley) Date: Tue, 06 Jan 2015 09:28:34 +0000 Subject: String.indexOf optimization In-Reply-To: References: <534D68B8.3040002@oracle.com> <5350E316.2050406@oracle.com> Message-ID: <54ABAAC2.2090502@redhat.com> Hi, On 05/01/15 18:59, Zoltan Sziladi wrote: > This discussion was a long time ago, I was just reading through it to check > again what was the last state of the discussion about the String.indexOf. > There is one part which I still do not understand, hopefully someone could > shed some light on it. A few emails ago Martin mentioned > > "Hotspot seems to have some intrinsification of String.indexOf, which > confuses me. > Hotspot seems the right place to provide more optimizations for this, since > there has been a fair amount of work creating high-performance low-level > implementations of this idea in C." > > Then Ivan asked what that actually meant, whether hotspot actually replaced > the jdk implementation with a low level optimized C implementation, but I > never saw an answer to that. You can have a look at an implementation of MacroAssembler::string_indexof in http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b6b89b8f8531/src/cpu/x86/vm/macroAssembler_x86.cpp > Can someone please explain this? If we somehow found an algorithm that beat > the naive implementation in the average case, would it be possible to just > implement it in the JDK? No, because HotSpot would ignore it. Any speed improvements have to be done in the architecture-dependent files. Andrew. From chris.hegarty at oracle.com Tue Jan 6 10:25:49 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 6 Jan 2015 10:25:49 +0000 Subject: Explicit Serialization API and Security In-Reply-To: References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> Message-ID: <4FDA8366-3AA9-4A7E-BD3D-314169C7ABF2@oracle.com> On 6 Jan 2015, at 08:31, Stephen Colebourne wrote: > On 6 January 2015 at 07:27, Peter Levart wrote: >> readObject() can be used on both ends of the spectrum or anywhere >> in-between. It can be used for explicit deserialization or just for >> validation of default deserialized state. Would separate validation method >> give us anything more or simplify things? readObject() can be used just as: >> >> private void readObject(ObjectInputStream in) throws IOException, >> ClassNotFoundException { >> in.defaultReadObject(); >> ... >> just validation >> ... >> } >> Explicit deserialization and final fields don't play nicely together >> currently, yes. Sometimes data-racey-publication thread-safety is sacrificed >> just because explicit deserialization would complicate code too much by >> using reflection. > > I've thought on a number of occasions that what I wanted from > serializable was a merger of readObject and readResolve > > private Object readObjectAndResolve(ObjectInputStream in) throws IOException This is an interesting idea. Just so I understand, readObject is called down the inheritance hierarchy and can read, into locals, its classes serializable fields ( of course if can access its super types fields that are already set ), where as just a single readResolve call is made, if it is defined by or accessible (via inheritance) by the given class. -Chris. > Using such a method, the input could be read into local variables, and > then a normal constructor used to create the object itself. It avoids > the final fields problem (as its just reading to local variables) and > it handles the validation (by calling a regular constructor/factory). > Its also more efficient in many cases, as a common pattern today is to > have one object instance created by readObject (or serialization > internals) and then another returned by readResolve. If > readObjectAndResolve() can't handle cyclic graphs, I can live with > that. > > Stephen From chris.hegarty at oracle.com Tue Jan 6 11:10:09 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 6 Jan 2015 11:10:09 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54AB8E44.6010905@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> Message-ID: On 6 Jan 2015, at 07:27, Peter Levart wrote: Hi Brian, > > On 01/05/2015 09:51 PM, Brian Goetz wrote: >> Thanks Peter(s): I think it's just about time to re-sync on the goals, since there are an infinitude of issues with serialization, only some of which we can address. >> >> To that end, let me toss out some of the results of our recent explorations with improving serialization, where we started out on something not unlike what Peter is suggesting, and dialed back. >> >> One thing we can do to positively improve the serialization experience is to reduce the set of situations where users have to write explicit readObject() methods. In the JDK, we end up having to do so for all manner of security-motivated reasons, and such code is hard to write correctly, hard to validate, hard to audit, and hard to evolve. >> >> After an audit of our codebase (thanks Chris!), we identified some of the biggest offenders on this score: >> >> - Validating invariants >> - Ensuring confinement >> >> The latter comes up in cases where there's a field: >> >> class Foo { >> private final Bar bar = new BarImpl(); >> } >> >> which implicitly has an invariant: the Bar referenced by Foo.bar is a) exactly of type BarImpl and not some other subclass and b) there is exactly one reference to that BarImpl (assuming it doesn't escape Foo.) >> >> Note that both of these still use the default deserialization; having to override readObject() just to do some extra checking is error-prone and messy. So I think the goal of a separate validation method which is called automatically is a good one, since it means we can ditch some percentage of otherwise-needless readObject implementations (which then have to jump through hoops to set final fields.) > > readObject() can be used on both ends of the spectrum or anywhere in-between. It can be used for explicit deserialization or just for validation of default deserialized state. Would separate validation method give us anything more or simplify things? readObject() can be used just as: > > private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { > in.defaultReadObject(); > ... > just validation > ... > } One issue with the above pattern is that defaultReadObject will read and set the fields before the explicit validation code, ?just validate?, is executed. So if validation fails, throws an appropriate exception, the object may be left in an inconsistent state. One approach, with existing serialization, is to use the GetField API rather than defaultReadObject. Read the field values into locals and validate before setting. But this again falls foul for final fields. As an aside: I have been looking at ObjectInputStream, and related classes, recently. The current default implementation of readObject sets non-primitive field values one at a time, without first checking that all their types are assignable. In many cases the setting of field values can be deferred until after they have been validated, at a minimum that the non-primitive types are assignable. Whole hierarchy failure atomicity in many cases, or at a minimum per class descriptor failure atomicity, can be achieved ( either all fields will be set, or contain their default values ). Preliminary webrev [1]. From this exploration, it seems like the natural place for an invariant checker is between the reading and reconstitution of field values, and the assigning of them. If the invariant checker could be called by the serialization mechanism then the users code can use final fields, without needing to muck around with reflection, and also have the added benefit of, in many cases, not potential leaving the object in an inconsistent state. To me, these benefits seems worthwhile, and justify a separate invariant checker mechanism. >> I agree that being able to play nicely with final fields is also an important goal here. > > Explicit deserialization and final fields don't play nicely together currently, yes. Sometimes data-racey-publication thread-safety is sacrificed just because explicit deserialization would complicate code too much by using reflection. Given my above comments, certainly in the case of invariant checking, we should be able to make this experience a little better ( confining the use of reflectively setting final fields and ensuring their safe publication in one place, the Serialization implementation ). -Chris. [1] http://cr.openjdk.java.net/~chegar/failureAtomicity/ > Regards, Peter > >> >> On 1/5/2015 9:11 AM, Peter Levart wrote: >>> Hi Peter and others, >>> >>> A assume your approach described here (to use constructors for >>> deserialization) is motivated mainly by the fact that only constructors >>> are allowed to set the final fields. Otherwise the explicit features of >>> your ReadSerial API are more or less accessible even now by using >>> "serialPersistentFields " static field for declaration of "serialized" >>> fields, combined with ObjectOutputStream.PutField and >>> ObjectInputStream.GetField API, accessible from writeObject/readObject >>> methods. readObject() method already acts like a constructor, whith the >>> following notable differences: >>> >>> 1 - it is not chained explicitly to superclass (either in source or >>> automatically by javac) like constructor, but is invoked by >>> (de)serialization infrastructure in correct order for each class in the >>> hierarchy that declares it. >>> 2 - it is a normal method, so it can be called at any time by code in >>> the declaring class or inner classes. >>> 3 - it can't set final fields (without using clumsy reflection) >>> >>> >>> The 1st point is actually in favor of readObject() method, since the >>> (de)serialization API need not be caller-sensitive. The correct >>> per-class context is established before each readObject call-back. >>> 2nd and 3rd points are readObject drawbacks and are both interconnected. >>> If readObject was not universally callable, then the restriction that it >>> can't set final fields could be lifted. >>> >>> Perhaps it is too late from compatibility perspective to change the >>> readObject's 2nd and 3rd point now, but we could compatibly change the >>> 1st point of a special kind of constructor. >>> >>> Say that a constructor declared with exactly one parameter of type >>> ObjectInputStream is treated specially by javac and reflection: >>> - constructor code can't chain to superclass constructor >>> - javac does not automatically generate default constructor chaining code >>> - javac does not allow calling such constructor from any code >>> - this is optional: javac does not allow calling instance methods and >>> explicit use of 'this' keyword in deserialization constructor code >>> (preventing escape of such object before de-serialization ends) - to be >>> really safe, this would probably have to be accompanied with verifier >>> checks too. >>> - public reflection API does not allow invoking such constructor (only >>> deserialization infrastructure can - like it can invoke the default >>> constructor of a non-Serializable superclass on an already allocated >>> instance of a sublclass) >>> >>> No additional changes to VM are apparently necessary (except verifier). >>> Declaring such constructor as an alternative to readObject() method is >>> then possible to make deserialization more safe. >>> >>> Regarding finalizability (and finalizer attacks): >>> >>> The serialization specifies that for 1st (most specific) >>> non-Serializable superclass of a Serializable subclass, default >>> constructor is invoked to initialize the non-Serializable superclass >>> state before deserializing subclass state. As each Serializable class >>> has such a non-Serializable superclass (at least in the form of Object >>> class) and normal constructors are chained, it is apparent that Object's >>> default constructor is called before any deserialization of state >>> begins. As soon as Object's constructor completes, the instance becomes >>> eligible for finalization: >>> >>> "An object o is not finalizable until its constructor has invoked the >>> constructor for Object on o and that invocation has completed >>> successfully (that is, without throwing an exception).? >>> >>> The rule of calling non-Serializable superclass default constructor >>> could be weakened. If 1st (most specific) non-Serializable superclass of >>> a Serializable subclass is Object, then it's constructor is not called. >>> Obviously it is not needed since Object has no state to initialize. What >>> we achieve by this is that such object does not become eligible for >>> finalization just yet. Serialization infrastructure must then explicitly >>> register such objects for finalization, but only after their >>> de-serialization completes normally. This would prevent finalization >>> attacks. >>> >>> Regards, Peter >>> >>> >>> On 01/05/2015 01:01 PM, Peter Firmstone wrote: >>>> My mistake, thank you. >>>> >>>> Peter. >>>> >>>> On 5/01/2015 9:57 PM, David Holmes wrote: >>>>> Hi Peter, >>>>> >>>>> Did you send this to the list? I haven't seen it turn up yet. >>>>> >>>>> David >>>>> >>>>> On 5/01/2015 6:51 PM, Peter Firmstone wrote: >>>>>> Thanks Dave, >>>>>> >>>>>> That's another way of checking invariants, you could expose A's check >>>>>> method but you'd also need a couple of additional static methods to >>>>>> obtain integers upper and lower from ReadSerial, so that B can ensure >>>>>> its invariant. >>>>>> >>>>>> ReadSerial is caller sensitive so B can't obtain A's stream values via >>>>>> ReadSerial and must do so via A's API. This prevents the >>>>>> publication of >>>>>> A's implementation, you don't wan't B depending on A's serial form. >>>>>> Instead A can reorder and rename it's fields and change their values, >>>>>> have multiple serial forms and be able to remain compatible with all >>>>>> forms relatively easily. >>>>>> >>>>>> There are some benefits to using a temporary object instance of A; it >>>>>> reduces the amount of code required, eg it would be a beneficial for >>>>>> RMI >>>>>> to minimise code downloads, the JVM is free to inline access to these >>>>>> fields and the temporary instance of A never leaves the cpu cache, so >>>>>> it's not likely to become a performance issue. Well it might on real >>>>>> time java :) >>>>>> >>>>>> If circular relationships are mutable, or effectively mutable after >>>>>> publication, then we could eventually deprecate the requirement to >>>>>> support special treatment of final fields for Serializable. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Peter. >>>>>> >>>>>> On 5/01/2015 10:38 AM, David Holmes wrote: >>>>>>> >>>>>>>>> - I don't see how this invariant-checking mechanism can >>>>>>>>> enforce >>>>>>>>> invariants between superclass fields and subclass fields. For >>>>>>>>> example: >>>>>>>>> >>>>>>>>> class A { >>>>>>>>> int lower, upper; // invariant: lower <= upper >>>>>>>>> } >>>>>>>>> >>>>>>>>> class B extends A { >>>>>>>>> int cur; // invariant: lower <= cur <= upper >>>>>>>>> } >>>>>>>>> >>>>>>>> >>>>>>>> The serialization framework should only construct an objects fields >>>>>>>> and make these available from ReadSerial, each class then calls a >>>>>>>> static method before calling a superclass constructor that's >>>>>>>> responsible for an objects creation, to prevent construction of the >>>>>>>> object, if necessary. >>>>>>>> >>>>>>>> Eg, some complexity, but bullet proof: >>>>>>>> >>>>>>>> public class A ( >>>>>>>> >>>>>>>> public final int lower, upper; >>>>>>>> >>>>>>>> private static boolean check(ReadSerial rs){ >>>>>>>> if (rs.getInt("lower") > rs.getInt("upper")) >>>>>>>> throw new IllegalArgumentException( >>>>>>>> "lower bound must be less than or equal to upper"); >>>>>>>> return true; >>>>>>>> } >>>>>>>> >>>>>>>> public A(ReadSerial rs){ >>>>>>>> this(check(rs), rs); >>>>>>>> } >>>>>>>> >>>>>>>> private A(boolean checked, ReadSerial rs){ >>>>>>>> super(); >>>>>>>> lower = rs.getInt("lower"); >>>>>>>> upper = rs.getInt("upper"); >>>>>>>> } >>>>>>>> >>>>>>>> // other constructors omitted must also check invarients >>>>>>>> } >>>>>>>> >>>>>>>> class B extends A { >>>>>>>> >>>>>>>> public final int cur; >>>>>>>> >>>>>>>> private static ReadSerial check(ReadSerial rs) { >>>>>>>> A a = new A(rs); >>>>>>> >>>>>>> >>>>>>> It doesn't seem practical in general to have to create an instance of >>>>>>> your supertype to validate the passed in serialized form. Why not >>>>>>> expose the check method? >>>>>>> >>>>>>> David >>>>>>> ----- >>>>>>> >>>>>>>> int cur = rs.getInt("cur"); >>>>>>>> if ( a.lower > cur || cur > a.upper ) >>>>>>>> throw new IllegalArgumentException( >>>>>>>> "cur outside lower and upper bounds"); >>>>>>>> return rs; >>>>>>>> } >>>>>>>> >>>>>>>> public B(ReadSerial rs) { >>>>>>>> super(check(rs)); >>>>>>>> cur = rs.getInt("cur"); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>> >>>> >>> > From scolebourne at joda.org Tue Jan 6 11:13:32 2015 From: scolebourne at joda.org (Stephen Colebourne) Date: Tue, 6 Jan 2015 11:13:32 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <4FDA8366-3AA9-4A7E-BD3D-314169C7ABF2@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <4FDA8366-3AA9-4A7E-BD3D-314169C7ABF2@oracle.com> Message-ID: On 6 January 2015 at 10:25, Chris Hegarty wrote: > On 6 Jan 2015, at 08:31, Stephen Colebourne wrote: >> I've thought on a number of occasions that what I wanted from >> serializable was a merger of readObject and readResolve >> >> private Object readObjectAndResolve(ObjectInputStream in) throws IOException > > This is an interesting idea. > > Just so I understand, readObject is called down the inheritance hierarchy and can read, into locals, its classes serializable fields ( of course if can access its super types fields that are already set ), where as just a single readResolve call is made, if it is defined by or accessible (via inheritance) by the given class. I tend to work with shallow/no hierarchies so I've not thought too much about the detail. I'd imagine you'd want to have readObjectAndResolve() be a static method called only on the class being deserialized and not superclasses. The method itself would be responsible for any superclass deserialization. (Static because there is no instance to call it on and it should have no access to instance variables). It may be that the input should not be ObjectInputStream, but some simpler but related type. Stephen From Alan.Bateman at oracle.com Tue Jan 6 11:16:09 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 06 Jan 2015 11:16:09 +0000 Subject: RFR 8068462: javax.script.ScriptEngineFactory.getParameter spec is not completely consistent with the rest of the API In-Reply-To: <54AB9574.9000701@oracle.com> References: <54AB9574.9000701@oracle.com> Message-ID: <54ABC3F9.2020201@oracle.com> On 06/01/2015 07:57, A. Sundararajan wrote: > Please review http://cr.openjdk.java.net/~sundar/8068462/ for > https://bugs.openjdk.java.net/browse/JDK-8068462 I think this is okay and doesn't require an update to JSR 223. One suggestion (feel free to ignore) to to split the sentence into two so that the Strings returned by getNames is in a second statement. -Alan. From peter.firmstone at zeus.net.au Tue Jan 6 11:16:29 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Tue, 06 Jan 2015 21:16:29 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54AB442F.9070800@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AB442F.9070800@oracle.com> Message-ID: <1420542989.3295.6.camel@Nokia-N900> ----- Original message ----- > > > On 5/01/2015 6:51 PM, Peter Firmstone wrote: > > > > Thanks Dave, > > > > > > > > That's another way of checking invariants, you could expose A's > > > > check method but you'd also need a couple of additional static > > > > methods to obtain integers upper and lower from ReadSerial, so > > > > that B can ensure its invariant. > > Not sure I follow. Currently you do "new A(rs)" which internally calls > "A.check(rs)" and I was suggesting you call A.check(rs) directly so > avoiding the need to create an A just for validation purposes, and which > is subsequently discarded. > > David > A is also created to allow B to access its upper and lower limit fields, in this case to ensure that the integer cur is between these limits. Intra class invariants. Cheers, Peter. > > > > > > > > ReadSerial is caller sensitive so B can't obtain A's stream values > > > > via ReadSerial and must do so via A's API.? This prevents the > > > > publication of A's implementation, you don't wan't B depending on > > > > A's serial form. Instead A can reorder and rename it's fields and > > > > change their values, have multiple serial forms and be able to > > > > remain compatible with all forms relatively easily. > > > > > > > > There are some benefits to using a temporary object instance of A; > > > > it reduces the amount of code required, eg it would be a > > > > beneficial for RMI to minimise code downloads, the JVM is free to > > > > inline access to these fields and the temporary instance of A > > > > never leaves the cpu cache, so it's not likely to become a > > > > performance issue.? Well it might on real time java :) > > > > > > > > If circular relationships are mutable, or effectively mutable after > > > > publication, then we could eventually deprecate the requirement to > > > > support special treatment of final fields for Serializable. > > > > > > > > Cheers, > > > > > > > > Peter. > > > > > > > > On 5/01/2015 10:38 AM, David Holmes wrote: > > > > > > > > > > > > - I don't see how this invariant-checking mechanism can > > > > > > > enforce invariants between superclass fields and subclass > > > > > > > fields.? ? For example: > > > > > > > > > > > > > > class A { > > > > > > > int lower, upper;? ? // invariant: lower <= upper > > > > > > > } > > > > > > > > > > > > > > class B extends A { > > > > > > > int cur;? ? // invariant: lower <= cur <= upper > > > > > > > } > > > > > > > > > > > > > > > > > > > The serialization framework should only construct an objects > > > > > > fields and make these available from ReadSerial, each class > > > > > > then calls a static method before calling a superclass > > > > > > constructor that's responsible for an objects creation, to > > > > > > prevent construction of the object, if necessary. > > > > > > > > > > > > Eg, some complexity, but bullet proof: > > > > > > > > > > > > public class A ( > > > > > > > > > > > > public final int lower, upper; > > > > > > > > > > > > private static boolean check(ReadSerial rs){ > > > > > > if (rs.getInt("lower") > rs.getInt("upper")) > > > > > > throw new IllegalArgumentException( > > > > > > "lower bound must be less than or equal to upper"); > > > > > > return true; > > > > > > } > > > > > > > > > > > > public A(ReadSerial rs){ > > > > > > this(check(rs), rs); > > > > > > } > > > > > > > > > > > > private A(boolean checked, ReadSerial rs){ > > > > > > super(); > > > > > > lower = rs.getInt("lower"); > > > > > > upper = rs.getInt("upper"); > > > > > > } > > > > > > > > > > > > // other constructors omitted must also check invarients > > > > > > } > > > > > > > > > > > > class B extends A { > > > > > > > > > > > > public final int cur; > > > > > > > > > > > > private static ReadSerial check(ReadSerial rs) { > > > > > > A a = new A(rs); > > > > > > > > > > > > > > > It doesn't seem practical in general to have to create an > > > > > instance of your supertype to validate the passed in serialized > > > > > form. Why not expose the check method? > > > > > > > > > > David > > > > > ----- > > > > > > > > > > > int cur = rs.getInt("cur"); > > > > > > if ( a.lower > cur || cur > a.upper ) > > > > > > throw new IllegalArgumentException( > > > > > > "cur outside lower and upper bounds"); > > > > > > return rs; > > > > > > } > > > > > > > > > > > > public B(ReadSerial rs) { > > > > > > super(check(rs)); > > > > > > cur = rs.getInt("cur"); > > > > > > } > > > > > > } > > > > > > > > > > > > > > > > > > From chris.hegarty at oracle.com Tue Jan 6 11:33:49 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 6 Jan 2015 11:33:49 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54AAF963.2020402@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> Message-ID: <7F6F9513-2613-4253-9574-D92BF163C78E@oracle.com> On 5 Jan 2015, at 20:51, Brian Goetz wrote: > Thanks Peter(s): I think it's just about time to re-sync on the goals, since there are an infinitude of issues with serialization, only some of which we can address. > > To that end, let me toss out some of the results of our recent explorations with improving serialization, where we started out on something not unlike what Peter is suggesting, and dialed back. > > One thing we can do to positively improve the serialization experience is to reduce the set of situations where users have to write explicit readObject() methods. In the JDK, we end up having to do so for all manner of security-motivated reasons, and such code is hard to write correctly, hard to validate, hard to audit, and hard to evolve. > > After an audit of our codebase (thanks Chris!), we identified some of the biggest offenders on this score: > > - Validating invariants > - Ensuring confinement > > The latter comes up in cases where there's a field: > > class Foo { > private final Bar bar = new BarImpl(); > } > > which implicitly has an invariant: the Bar referenced by Foo.bar is a) exactly of type BarImpl and not some other subclass and b) there is exactly one reference to that BarImpl (assuming it doesn't escape Foo.) This type of pattern typically results in a readObject having to be defined to copy/clone the deserialized BarImpl, and then assign Foo.bar to the new copy. Again the issue is with setting the final field. serialPersistentFields can help, in that it can ensure that the deserialized BarImpl is not a back reference, and that no one else can refer to it. But then you have to explicitly define the complete serial form of the class, and evolve it explicitly over time. Cumbersome. Life may be made a little easier, avoid readObject and serialPersistentFields, if we could ensure that BarImpl was not shared, and loaded by the same class loader as Foo ( or a parent of ). Maybe simply popping an annotation, or other such, on the field declaration. class Foo { @Confined private final Bar bar = new BarImpl(); } > Note that both of these still use the default deserialization; having to override readObject() just to do some extra checking is error-prone and messy. So I think the goal of a separate validation method which is called automatically is a good one, since it means we can ditch some percentage of otherwise-needless readObject implementations (which then have to jump through hoops to set final fields.) > > I agree that being able to play nicely with final fields is also an important goal here. Agreed. I don?t have fully formed ideas on how best to deal with the general case of handling final fields, but I think that an invariant checker, and confinement mechanism, may get us a long way. In so much as they allow us to avoid having to explicitly set final fields in user code. -Chris. > On 1/5/2015 9:11 AM, Peter Levart wrote: >> Hi Peter and others, >> >> A assume your approach described here (to use constructors for >> deserialization) is motivated mainly by the fact that only constructors >> are allowed to set the final fields. Otherwise the explicit features of >> your ReadSerial API are more or less accessible even now by using >> "serialPersistentFields " static field for declaration of "serialized" >> fields, combined with ObjectOutputStream.PutField and >> ObjectInputStream.GetField API, accessible from writeObject/readObject >> methods. readObject() method already acts like a constructor, whith the >> following notable differences: >> >> 1 - it is not chained explicitly to superclass (either in source or >> automatically by javac) like constructor, but is invoked by >> (de)serialization infrastructure in correct order for each class in the >> hierarchy that declares it. >> 2 - it is a normal method, so it can be called at any time by code in >> the declaring class or inner classes. >> 3 - it can't set final fields (without using clumsy reflection) >> >> >> The 1st point is actually in favor of readObject() method, since the >> (de)serialization API need not be caller-sensitive. The correct >> per-class context is established before each readObject call-back. >> 2nd and 3rd points are readObject drawbacks and are both interconnected. >> If readObject was not universally callable, then the restriction that it >> can't set final fields could be lifted. >> >> Perhaps it is too late from compatibility perspective to change the >> readObject's 2nd and 3rd point now, but we could compatibly change the >> 1st point of a special kind of constructor. >> >> Say that a constructor declared with exactly one parameter of type >> ObjectInputStream is treated specially by javac and reflection: >> - constructor code can't chain to superclass constructor >> - javac does not automatically generate default constructor chaining code >> - javac does not allow calling such constructor from any code >> - this is optional: javac does not allow calling instance methods and >> explicit use of 'this' keyword in deserialization constructor code >> (preventing escape of such object before de-serialization ends) - to be >> really safe, this would probably have to be accompanied with verifier >> checks too. >> - public reflection API does not allow invoking such constructor (only >> deserialization infrastructure can - like it can invoke the default >> constructor of a non-Serializable superclass on an already allocated >> instance of a sublclass) >> >> No additional changes to VM are apparently necessary (except verifier). >> Declaring such constructor as an alternative to readObject() method is >> then possible to make deserialization more safe. >> >> Regarding finalizability (and finalizer attacks): >> >> The serialization specifies that for 1st (most specific) >> non-Serializable superclass of a Serializable subclass, default >> constructor is invoked to initialize the non-Serializable superclass >> state before deserializing subclass state. As each Serializable class >> has such a non-Serializable superclass (at least in the form of Object >> class) and normal constructors are chained, it is apparent that Object's >> default constructor is called before any deserialization of state >> begins. As soon as Object's constructor completes, the instance becomes >> eligible for finalization: >> >> "An object o is not finalizable until its constructor has invoked the >> constructor for Object on o and that invocation has completed >> successfully (that is, without throwing an exception).? >> >> The rule of calling non-Serializable superclass default constructor >> could be weakened. If 1st (most specific) non-Serializable superclass of >> a Serializable subclass is Object, then it's constructor is not called. >> Obviously it is not needed since Object has no state to initialize. What >> we achieve by this is that such object does not become eligible for >> finalization just yet. Serialization infrastructure must then explicitly >> register such objects for finalization, but only after their >> de-serialization completes normally. This would prevent finalization >> attacks. >> >> Regards, Peter >> >> >> On 01/05/2015 01:01 PM, Peter Firmstone wrote: >>> My mistake, thank you. >>> >>> Peter. >>> >>> On 5/01/2015 9:57 PM, David Holmes wrote: >>>> Hi Peter, >>>> >>>> Did you send this to the list? I haven't seen it turn up yet. >>>> >>>> David >>>> >>>> On 5/01/2015 6:51 PM, Peter Firmstone wrote: >>>>> Thanks Dave, >>>>> >>>>> That's another way of checking invariants, you could expose A's check >>>>> method but you'd also need a couple of additional static methods to >>>>> obtain integers upper and lower from ReadSerial, so that B can ensure >>>>> its invariant. >>>>> >>>>> ReadSerial is caller sensitive so B can't obtain A's stream values via >>>>> ReadSerial and must do so via A's API. This prevents the >>>>> publication of >>>>> A's implementation, you don't wan't B depending on A's serial form. >>>>> Instead A can reorder and rename it's fields and change their values, >>>>> have multiple serial forms and be able to remain compatible with all >>>>> forms relatively easily. >>>>> >>>>> There are some benefits to using a temporary object instance of A; it >>>>> reduces the amount of code required, eg it would be a beneficial for >>>>> RMI >>>>> to minimise code downloads, the JVM is free to inline access to these >>>>> fields and the temporary instance of A never leaves the cpu cache, so >>>>> it's not likely to become a performance issue. Well it might on real >>>>> time java :) >>>>> >>>>> If circular relationships are mutable, or effectively mutable after >>>>> publication, then we could eventually deprecate the requirement to >>>>> support special treatment of final fields for Serializable. >>>>> >>>>> Cheers, >>>>> >>>>> Peter. >>>>> >>>>> On 5/01/2015 10:38 AM, David Holmes wrote: >>>>>> >>>>>>>> - I don't see how this invariant-checking mechanism can >>>>>>>> enforce >>>>>>>> invariants between superclass fields and subclass fields. For >>>>>>>> example: >>>>>>>> >>>>>>>> class A { >>>>>>>> int lower, upper; // invariant: lower <= upper >>>>>>>> } >>>>>>>> >>>>>>>> class B extends A { >>>>>>>> int cur; // invariant: lower <= cur <= upper >>>>>>>> } >>>>>>>> >>>>>>> >>>>>>> The serialization framework should only construct an objects fields >>>>>>> and make these available from ReadSerial, each class then calls a >>>>>>> static method before calling a superclass constructor that's >>>>>>> responsible for an objects creation, to prevent construction of the >>>>>>> object, if necessary. >>>>>>> >>>>>>> Eg, some complexity, but bullet proof: >>>>>>> >>>>>>> public class A ( >>>>>>> >>>>>>> public final int lower, upper; >>>>>>> >>>>>>> private static boolean check(ReadSerial rs){ >>>>>>> if (rs.getInt("lower") > rs.getInt("upper")) >>>>>>> throw new IllegalArgumentException( >>>>>>> "lower bound must be less than or equal to upper"); >>>>>>> return true; >>>>>>> } >>>>>>> >>>>>>> public A(ReadSerial rs){ >>>>>>> this(check(rs), rs); >>>>>>> } >>>>>>> >>>>>>> private A(boolean checked, ReadSerial rs){ >>>>>>> super(); >>>>>>> lower = rs.getInt("lower"); >>>>>>> upper = rs.getInt("upper"); >>>>>>> } >>>>>>> >>>>>>> // other constructors omitted must also check invarients >>>>>>> } >>>>>>> >>>>>>> class B extends A { >>>>>>> >>>>>>> public final int cur; >>>>>>> >>>>>>> private static ReadSerial check(ReadSerial rs) { >>>>>>> A a = new A(rs); >>>>>> >>>>>> >>>>>> It doesn't seem practical in general to have to create an instance of >>>>>> your supertype to validate the passed in serialized form. Why not >>>>>> expose the check method? >>>>>> >>>>>> David >>>>>> ----- >>>>>> >>>>>>> int cur = rs.getInt("cur"); >>>>>>> if ( a.lower > cur || cur > a.upper ) >>>>>>> throw new IllegalArgumentException( >>>>>>> "cur outside lower and upper bounds"); >>>>>>> return rs; >>>>>>> } >>>>>>> >>>>>>> public B(ReadSerial rs) { >>>>>>> super(check(rs)); >>>>>>> cur = rs.getInt("cur"); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> >>>>> >>> >> From chris.hegarty at oracle.com Tue Jan 6 11:46:08 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 6 Jan 2015 11:46:08 +0000 Subject: Explicit Serialization API and Security In-Reply-To: References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <4FDA8366-3AA9-4A7E-BD3D-314169C7ABF2@oracle.com> Message-ID: <50A5EFAE-CD9E-4CE2-9C10-06A9661BFDC5@oracle.com> On 6 Jan 2015, at 11:13, Stephen Colebourne wrote: On 6 January 2015 at 10:25, Chris Hegarty wrote: >> On 6 Jan 2015, at 08:31, Stephen Colebourne wrote: >>> I've thought on a number of occasions that what I wanted from >>> serializable was a merger of readObject and readResolve >>> >>> private Object readObjectAndResolve(ObjectInputStream in) throws IOException >> >> This is an interesting idea. >> >> Just so I understand, readObject is called down the inheritance hierarchy and can read, into locals, its classes serializable fields ( of course if can access its super types fields that are already set ), where as just a single readResolve call is made, if it is defined by or accessible (via inheritance) by the given class. > > I tend to work with shallow/no hierarchies so I've not thought too > much about the detail. I'd imagine you'd want to have > readObjectAndResolve() be a static method called only on the class > being deserialized and not superclasses. The method itself would be > responsible for any superclass deserialization. (Static because there > is no instance to call it on and it should have no access to instance > variables). It may be that the input should not be ObjectInputStream, > but some simpler but related type. With shallow/no hierarchies, and others, the serialization proxy pattern works quite well. The overhead of course being the creation of the proxy itself, but this is typically a minimal container with just the state needed to create the ?real? object, and no behaviour. Whatever the input would have to be to the "static factory? method, readObjectAndWriteReplace, then it would probably have the same overhead, albeit minimal, as the proxy. Though one less serializable type. This could work out nice, but it could not support cyclic graphs ( which I think I could live with ), or private superclass state ( which I think would be limiting ). -Chris. > Stephen From scolebourne at joda.org Tue Jan 6 11:55:30 2015 From: scolebourne at joda.org (Stephen Colebourne) Date: Tue, 6 Jan 2015 11:55:30 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <50A5EFAE-CD9E-4CE2-9C10-06A9661BFDC5@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <4FDA8366-3AA9-4A7E-BD3D-314169C7ABF2@oracle.com> <50A5EFAE-CD9E-4CE2-9C10-06A9661BFDC5@oracle.com> Message-ID: On 6 January 2015 at 11:46, Chris Hegarty wrote: > With shallow/no hierarchies, and others, the serialization proxy pattern works quite well. The overhead of course being the creation of the proxy itself, but this is typically a minimal container with just the state needed to create the ?real? object, and no behaviour. Whatever the input would have to be to the "static factory? method, readObjectAndWriteReplace, then it would probably have the same overhead, albeit minimal, as the proxy. Though one less serializable type. This could work out nice, but it could not support cyclic graphs ( which I think I could live with ), or private superclass state ( which I think would be limiting ). I used a shared proxy on 310. Small serialized form (short class name, shared overhead if more than one type from same package, but no sharing). Its fine, but quite verbose. Oh, and to be clear, with readObjectAndResolve() you'd ban readObject() and readResolve() from being in the same class. On private superclass state, that state must be settable via the constructor or setters of the superclass. Providing you control the superclass and can change your class if the superclass changes, then I don't see private superclass state as a problem. private static Object readObjectAndResolve(in) { String name = in.readStr(""name") int age = in.readInt("age") Address addr = in.readObj("address") return new Person(name, age, address); } works fine even if name is in the superclass AbstractPerson and age/address is in Person. Stephen From sundararajan.athijegannathan at oracle.com Tue Jan 6 12:05:11 2015 From: sundararajan.athijegannathan at oracle.com (A. Sundararajan) Date: Tue, 06 Jan 2015 17:35:11 +0530 Subject: RFR 8068462: javax.script.ScriptEngineFactory.getParameter spec is not completely consistent with the rest of the API In-Reply-To: <54ABC3F9.2020201@oracle.com> References: <54AB9574.9000701@oracle.com> <54ABC3F9.2020201@oracle.com> Message-ID: <54ABCF77.4070601@oracle.com> Thanks. Broke that sentence into two for clarity. -Sundar On Tuesday 06 January 2015 04:46 PM, Alan Bateman wrote: > On 06/01/2015 07:57, A. Sundararajan wrote: >> Please review http://cr.openjdk.java.net/~sundar/8068462/ for >> https://bugs.openjdk.java.net/browse/JDK-8068462 > > I think this is okay and doesn't require an update to JSR 223. > > One suggestion (feel free to ignore) to to split the sentence into two > so that the Strings returned by getNames is in a second statement. > > -Alan. From ivan.gerasimov at oracle.com Tue Jan 6 12:49:30 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Tue, 06 Jan 2015 15:49:30 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AADC34.1000703@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB243.102@oracle.com> <54AACCC1.6030602@oracle.com> <54AADC34.1000703@oracle.com> Message-ID: <54ABD9DA.2040206@oracle.com> Hi Alan! I'm not sure why it would be an improvement to make it a method. As Roger said, with a final variable, the compiler is able to eliminate dead code altogether. I think, if later there's a need to make the decision dynamic/overridable, it can be easily changed, as it's not part of public API. Sincerely yours, Ivan On 05.01.2015 21:47, Alan Bateman wrote: > On 05/01/2015 17:41, Ivan Gerasimov wrote: >> >> : >> >> Here's another update: >> http://cr.openjdk.java.net/~igerasim/8067951/4/webrev/ >> >> The optimized loop is used both on Unix and on Windows, if there were >> no quotes found in PATH. >> Not a big deal, of course, but it should save a few nano-seconds >> during startup. >> > This approach looks okay to me. A small improvement would be to change > it to a method that returns a boolean. > > -Alan > > From james.laskey at oracle.com Tue Jan 6 12:50:21 2015 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Tue, 6 Jan 2015 08:50:21 -0400 Subject: RFR 8068462: javax.script.ScriptEngineFactory.getParameter spec is not completely consistent with the rest of the API In-Reply-To: <54AB959C.7040609@oracle.com> References: <54AB9574.9000701@oracle.com> <54AB959C.7040609@oracle.com> Message-ID: <44F77A10-C426-456D-A2E1-6B380F60C21F@oracle.com> +1 On Jan 6, 2015, at 3:58 AM, A. Sundararajan wrote: > > > > -------- Forwarded Message -------- > Subject: RFR 8068462: javax.script.ScriptEngineFactory.getParameter spec is not completely consistent with the rest of the API > Date: Tue, 06 Jan 2015 13:27:40 +0530 > From: A. Sundararajan > To: core-libs-dev at openjdk.java.net > > Please review http://cr.openjdk.java.net/~sundar/8068462/ for > https://bugs.openjdk.java.net/browse/JDK-8068462 > > Thanks, > -Sundar > > From ivan.gerasimov at oracle.com Tue Jan 6 13:49:08 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Tue, 06 Jan 2015 16:49:08 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AAFE5A.5070406@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB050.4010001@Oracle.com> <54AACE94.2000802@oracle.com> <54AAD3A6.9010200@Oracle.com> <54AAE19F.4090102@oracle.com> <54AAF717.3070602@oracle.com> <54AAFE5A.5070406@oracle.com> Message-ID: <54ABE7D4.5010502@oracle.com> Hi Sherman! I took your suggestion and rewrote the method to moved the logic, which removes the quotes to the top. I think the code became cleaner, so thank you for suggestion! Here's the updated webrev: http://cr.openjdk.java.net/~igerasim/8067951/5/webrev/ Sincerely yours, Ivan On 06.01.2015 0:12, Xueming Shen wrote: > On 01/05/2015 12:41 PM, Ivan Gerasimov wrote: >> Thanks Sherman! >> >> On 05.01.2015 22:10, Xueming Shen wrote: >>> >>> Just wonder if we really need that "inQuotes" logic here? A >>> straightforward approach might >>> be "every time you have a quote, skip everything until you hit >>> another one, when counting, >>> or copy everything into the buffer until hit another one, when >>> copying" ? >>> >> I agree it would work, but, in my opinion, it would be a bit more >> complicated. >> The counting loop would look something like this: >> ------------------------------------ >> outerLoop: for (int i = 0; i < ldLen; ++i) { >> char ch = ldPath.charAt(i); >> if (mayBeQuoted && ch == '\"') { >> thereAreQuotes = true; >> for (++i; i < ldLen; ++i) { >> if (ldPath.charAt(i) == '\"') { >> continue outerLoop; >> } >> } >> break; // unpaired quote >> } else if (ch == ps) { >> psCount++; >> } >> } >> ------------------------------------ >> which is 3 lines longer, comparing to the loop with inQuotes flag. >> > > It does not seem like we are doing anything special for "unpaired > quote" (just ignore it?), > if that is the case, you probably don't need to do anything for it, > the code could be > something like > > for (int i = 0; i < ldLen; ++i) { > char ch = ldPath.charAt(i); > if (mayBeQuoted && ch == '\"') { > thereAreQuotes = true; > while (++i < ldLen && ldPath.charAt(i) != '\"') {} > } else if (ch == ps) { > psCount++; > } > } > > I have not tried to debug the code though :-) Just an opinion here. > > -Sherman > > From peter.levart at gmail.com Tue Jan 6 15:03:04 2015 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 06 Jan 2015 16:03:04 +0100 Subject: Explicit Serialization API and Security In-Reply-To: References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> Message-ID: <54ABF928.3020502@gmail.com> Hi Chris, On 01/06/2015 12:10 PM, Chris Hegarty wrote: > On 6 Jan 2015, at 07:27, Peter Levart wrote: > Hi Brian, >> On 01/05/2015 09:51 PM, Brian Goetz wrote: >>> Thanks Peter(s): I think it's just about time to re-sync on the goals, since there are an infinitude of issues with serialization, only some of which we can address. >>> >>> To that end, let me toss out some of the results of our recent explorations with improving serialization, where we started out on something not unlike what Peter is suggesting, and dialed back. >>> >>> One thing we can do to positively improve the serialization experience is to reduce the set of situations where users have to write explicit readObject() methods. In the JDK, we end up having to do so for all manner of security-motivated reasons, and such code is hard to write correctly, hard to validate, hard to audit, and hard to evolve. >>> >>> After an audit of our codebase (thanks Chris!), we identified some of the biggest offenders on this score: >>> >>> - Validating invariants >>> - Ensuring confinement >>> >>> The latter comes up in cases where there's a field: >>> >>> class Foo { >>> private final Bar bar = new BarImpl(); >>> } >>> >>> which implicitly has an invariant: the Bar referenced by Foo.bar is a) exactly of type BarImpl and not some other subclass and b) there is exactly one reference to that BarImpl (assuming it doesn't escape Foo.) >>> >>> Note that both of these still use the default deserialization; having to override readObject() just to do some extra checking is error-prone and messy. So I think the goal of a separate validation method which is called automatically is a good one, since it means we can ditch some percentage of otherwise-needless readObject implementations (which then have to jump through hoops to set final fields.) >> readObject() can be used on both ends of the spectrum or anywhere in-between. >> It can be used for explicit deserialization or just for validation of default deserialized state. Well, that "in-between" is actually not really available (unless explicit (de)serialization part uses "block data mode", which is not very evolvable). But I think this could be enhanced (see below). >> Would separate validation method give us anything more or simplify things? readObject() can be used just as: >> >> private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { >> in.defaultReadObject(); >> ... >> just validation >> ... >> } > One issue with the above pattern is that defaultReadObject will read and set the fields before the explicit validation code, ?just validate?, is executed. So if validation fails, throws an appropriate exception, the object may be left in an inconsistent state. Right. What about the following addition to ObjectInputStream.GetField/ObjectOutputStream.PutField API. For example: public class Interval implements Serializable { // current fields private final int lo, hi; private static final ObjectStreamField[] serialPersistentFields = { new ObjectStreamField("lo", Integer.TYPE), new ObjectStreamField("hi", Integer.TYPE), // legacy 'len' field new ObjectStreamField("len", Integer.TYPE) }; private static void validate(int lo, int hi) { // invariant if (lo > hi) throw new IllegalArgumentException("lo:" + lo + " > hi:" + hi); } public Interval(int lo, int hi) { validate(lo, hi); this.lo = lo; this.hi = hi; } public int getLo() { return lo; } public int getHi() { return hi; } public int getLen() { return hi - lo; } private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { ObjectInputStream.GetField fields = in.readFields(); // this already validates the types // validate 'lo' and 'hi' fields invariant int lo = fields.get("lo", 0); int hi = fields.get("hi", 0); validate(lo, hi); // validate legacy 'len' field int len = fields.get("len", hi - lo); if (len != hi - lo) throw new IllegalArgumentException("len:" + len + " is wrong"); // set current fields from read data fields.defaultReadFields(); // this is new API! } private void writeObject(ObjectOutputStream out) throws IOException { ObjectOutputStream.PutField fields = out.putFields(); // put current fields fields.defaultPutFields(); // this is new API! // put legacy len field fields.put("len", hi - lo); // write-out buffered puts out.writeFields(); } } I've taken a quick look at what might be needed to do that, and it appears very simple: http://cr.openjdk.java.net/~plevart/misc/ObjectIOStream.GetPutFields/webrev/ > > One approach, with existing serialization, is to use the GetField API rather than defaultReadObject. Read the field values into locals and validate before setting. But this again falls foul for final fields. The above addition to [Get|Put]Fields API is a mix of explicit validation and default setters which seems to cover final fields. > > As an aside: I have been looking at ObjectInputStream, and related classes, recently. The current default implementation of readObject sets non-primitive field values one at a time, without first checking that all their types are assignable. In many cases the setting of field values can be deferred until after they have been validated, at a minimum that the non-primitive types are assignable. Whole hierarchy failure atomicity in many cases, or at a minimum per class descriptor failure atomicity, can be achieved ( either all fields will be set, or contain their default values ). Preliminary webrev [1]. > > From this exploration, it seems like the natural place for an invariant checker is between the reading and reconstitution of field values, and the assigning of them. If the invariant checker could be called by the serialization mechanism then the users code can use final fields, without needing to muck around with reflection, and also have the added benefit of, in many cases, not potential leaving the object in an inconsistent state. > > To me, these benefits seems worthwhile, and justify a separate invariant checker mechanism. The type checking for failure atomicity is a good addition to OIS.defaultReadObject() method behaviour. The separate invariant checker mechanism would have to expose field values before setting them to the fields of object being deserialized. The GetFields API lends itself to that purpose. Whether to reserve another private 'validateReadObject' special method signature for that purpose is a question. The above is an alternative that uses readObject() for both - default field assignment and explicit validation. Regards, Peter > >>> I agree that being able to play nicely with final fields is also an important goal here. >> Explicit deserialization and final fields don't play nicely together currently, yes. Sometimes data-racey-publication thread-safety is sacrificed just because explicit deserialization would complicate code too much by using reflection. > Given my above comments, certainly in the case of invariant checking, we should be able to make this experience a little better ( confining the use of reflectively setting final fields and ensuring their safe publication in one place, the Serialization implementation ). > > -Chris. > > [1] http://cr.openjdk.java.net/~chegar/failureAtomicity/ > From peter.levart at gmail.com Tue Jan 6 15:06:28 2015 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 06 Jan 2015 16:06:28 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <54ABF928.3020502@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> Message-ID: <54ABF9F4.1090705@gmail.com> On 01/06/2015 04:03 PM, Peter Levart wrote: > private void readObject(ObjectInputStream in) throws IOException, > ClassNotFoundException { > ObjectInputStream.GetField fields = in.readFields(); // this > already validates the types Well, not true currently. But type validation could be added at this point. Peter From chris.hegarty at oracle.com Tue Jan 6 17:21:39 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 6 Jan 2015 17:21:39 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54ABF9F4.1090705@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> Message-ID: On 6 Jan 2015, at 15:06, Peter Levart wrote: > On 01/06/2015 04:03 PM, Peter Levart wrote: >> private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { >> ObjectInputStream.GetField fields = in.readFields(); // this already validates the types > > Well, not true currently. But type validation could be added at this point. Right. I think I?ll file a bug to track this as it seems reasonable to add type validation to readFields and defaultReadObject. So we can probably assume/ignore it in this discussion. I like the idea of a callback into the serialization framework to handling the setting of final fields, after validation. I played a little with your patch and added it to a branch in the sandbox** So a simple example, without legacy fields, might looks as below ( without the need for writeObject or serialPersistentFields ). The simple validating readObject is starting to look like boilerplate ? public class SimpleInterval implements Serializable { private final int lo, hi; private static void validate(int lo, int hi) { // invariant if (lo > hi) throw new IllegalArgumentException("lo:" + lo + " > hi:" + hi); } public SimpleInterval(int lo, int hi) { validate(lo, hi); this.lo = lo; this.hi = hi; } public int getLo() { return lo; } public int getHi() { return hi; } private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { ObjectInputStream.GetField fields = in.readFields(); // validate 'lo' and 'hi' fields invariant int lo = fields.get("lo", 0); int hi = fields.get("hi", 0); validate(lo, hi); // set current fields from read data fields.defaultReadFields(); // this is new API! } } -Chris. ** hg clone http://hg.openjdk.java.net/jdk9/sandbox sandbox cd sandbox sh get_source.sh sh common/bin/hgforest.sh update -r serial-exp-branch I also added your example, etc, under: jdk/test/java/io/Serializable/invarientChecker see http://cr.openjdk.java.net/~chegar/docs/sandbox.html From peter.levart at gmail.com Tue Jan 6 17:49:02 2015 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 06 Jan 2015 18:49:02 +0100 Subject: Explicit Serialization API and Security In-Reply-To: References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> Message-ID: <54AC200E.3000602@gmail.com> On 01/06/2015 06:21 PM, Chris Hegarty wrote: > On 6 Jan 2015, at 15:06, Peter Levart wrote: > >> On 01/06/2015 04:03 PM, Peter Levart wrote: >>> private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { >>> ObjectInputStream.GetField fields = in.readFields(); // this already validates the types >> Well, not true currently. But type validation could be added at this point. > Right. I think I?ll file a bug to track this as it seems reasonable to add type validation to readFields and defaultReadObject. So we can probably assume/ignore it in this discussion. > > I like the idea of a callback into the serialization framework to handling the setting of final fields, after validation. I played a little with your patch and added it to a branch in the sandbox** > > So a simple example, without legacy fields, might looks as below ( without the need for writeObject or serialPersistentFields ). The simple validating readObject is starting to look like boilerplate ? Well, 1st and last line are always the same, yes. What's between them is what you would have to write in a special check-only method too. Regards, Peter > > public class SimpleInterval implements Serializable { > > private final int lo, hi; > > private static void validate(int lo, int hi) { > // invariant > if (lo > hi) > throw new IllegalArgumentException("lo:" + lo + " > hi:" + hi); > } > > public SimpleInterval(int lo, int hi) { > validate(lo, hi); > this.lo = lo; > this.hi = hi; > } > > public int getLo() { return lo; } > > public int getHi() { return hi; } > > private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { > ObjectInputStream.GetField fields = in.readFields(); > > // validate 'lo' and 'hi' fields invariant > int lo = fields.get("lo", 0); > int hi = fields.get("hi", 0); > validate(lo, hi); > > // set current fields from read data > fields.defaultReadFields(); // this is new API! > } > } > > -Chris. > > ** hg clone http://hg.openjdk.java.net/jdk9/sandbox sandbox > cd sandbox > sh get_source.sh > sh common/bin/hgforest.sh update -r serial-exp-branch > > I also added your example, etc, under: > jdk/test/java/io/Serializable/invarientChecker > > see http://cr.openjdk.java.net/~chegar/docs/sandbox.html > From xueming.shen at oracle.com Tue Jan 6 18:33:07 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Tue, 06 Jan 2015 10:33:07 -0800 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54ABE7D4.5010502@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB050.4010001@Oracle.com> <54AACE94.2000802@oracle.com> <54AAD3A6.9010200@Oracle.com> <54AAE19F.4090102@oracle.com> <54AAF717.3070602@oracle.com> <54AAFE5A.5070406@oracle.com> <54ABE7D4.5010502@oracle.com> Message-ID: <54AC2A63.1040505@oracle.com> Hi Ivan, Yes, it looks better and should run faster as it doesn't need to worry about quote in second loop. I was a little hesitated when suggested to replace the ps with 0, though I'm pretty much sure a \u0000 should never be a legal character in a windows' path :-) Anyone can think about any possibility of having an embedded \0 character in path? Nitpick: the "posCount" probably can be calculated in the "quote handling" loop ? if (ClassloaderHelper.allowQuotedPathElements ....) { ... if (ch == ps) { ch = '\0'; psCount++; } buf[bufLen++] = ch; ... } else { for (int i = ldPath.indexOf(ps); i >=0 ...) { } } otherwise, it looks fine for me. -Sherman On 01/06/2015 05:49 AM, Ivan Gerasimov wrote: > Hi Sherman! > > I took your suggestion and rewrote the method to moved the logic, which removes the quotes to the top. > I think the code became cleaner, so thank you for suggestion! > > Here's the updated webrev: > http://cr.openjdk.java.net/~igerasim/8067951/5/webrev/ > > Sincerely yours, > Ivan > > > On 06.01.2015 0:12, Xueming Shen wrote: >> On 01/05/2015 12:41 PM, Ivan Gerasimov wrote: >>> Thanks Sherman! >>> >>> On 05.01.2015 22:10, Xueming Shen wrote: >>>> >>>> Just wonder if we really need that "inQuotes" logic here? A straightforward approach might >>>> be "every time you have a quote, skip everything until you hit another one, when counting, >>>> or copy everything into the buffer until hit another one, when copying" ? >>>> >>> I agree it would work, but, in my opinion, it would be a bit more complicated. >>> The counting loop would look something like this: >>> ------------------------------------ >>> outerLoop: for (int i = 0; i < ldLen; ++i) { >>> char ch = ldPath.charAt(i); >>> if (mayBeQuoted && ch == '\"') { >>> thereAreQuotes = true; >>> for (++i; i < ldLen; ++i) { >>> if (ldPath.charAt(i) == '\"') { >>> continue outerLoop; >>> } >>> } >>> break; // unpaired quote >>> } else if (ch == ps) { >>> psCount++; >>> } >>> } >>> ------------------------------------ >>> which is 3 lines longer, comparing to the loop with inQuotes flag. >>> >> >> It does not seem like we are doing anything special for "unpaired quote" (just ignore it?), >> if that is the case, you probably don't need to do anything for it, the code could be >> something like >> >> for (int i = 0; i < ldLen; ++i) { >> char ch = ldPath.charAt(i); >> if (mayBeQuoted && ch == '\"') { >> thereAreQuotes = true; >> while (++i < ldLen && ldPath.charAt(i) != '\"') {} >> } else if (ch == ps) { >> psCount++; >> } >> } >> >> I have not tried to debug the code though :-) Just an opinion here. >> >> -Sherman >> >> > From lance.andersen at oracle.com Tue Jan 6 19:33:18 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Tue, 6 Jan 2015 14:33:18 -0500 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> Message-ID: <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> Hi Tristan, Sorry for the delay but with the holidays and the number of files, it took a while to go through :-) Overall, it looks pretty good. A couple of suggestions, but I would not necessarily hold back from committing: - For tests where you are not caring about the actual exception that is thrown to indicate a failure, such as in DocumentBuilderFactory01.java and testCheckSchemaSupport1, I would just have the method declaration use "throws Exception" vs list all of the possible individual Exceptions, as it keeps it more compact. Glad to see you are not using failUnexcepted() now. - In test classes such as in testCheckSchemaSupport3. java and DocumentBuilderImpl01.java, some tests do not use assertXXX or expect an Exception. Would be good at least to document what could cause a failure or make it clear the expected behavior of the test for passing. -SAXParserTest02.java and other tests where you get a reader/parser such as testXMLReader01, I would at least assert that null is not returned where as it is now, you only validate that an exception is not returned - I know you are porting existing tests, but I would consider consolidating tests as it seems like there is not a real reason to have a testNG class with just 1 test. I would group the like tests (such as SAXTFactoryTest ) in one testNG test class as that allows you to streamline further. - TfClearParamTest.java (as and example) minor nit that you have your @Before/AfterGroups method in between tests. I would suggest grouping all methods such as this DataProviders before or after the actual tests for better organization - TraxSAXWrapper.java, not sure I understand the "Sorry I could not resist comment" - TransformerHandlerAPITest.java has typos in comments: "IllegalArgumentExceptionis thrown?." - Minitest.java, I would add a comment for your Data Provider Best, Lance On Jan 2, 2015, at 1:49 PM, Tristan Yan wrote: > Hi Joe and Lance > Sorry for my late reply. I just uploaded the new webrev which is trying to limit minimal permissions for most of tests. The new changeset has two base classes; class JAXPBaseTest has only minimal set permissions; class JAXPFileBaseTest adds two permissions that need access local files in the directory directory test.src and test.classes. Most of tests only inherit JAXPBaseTest that provides only minimal set permissions. Some tests will inherit JAXPFileBaseTest because tests need access local files. > Please help to review the code. > http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ > > Thank you > Tristan > > >> On Jan 2, 2015, at 10:39 AM, huizhe wang wrote: >> >> Lance, >> >> Tristan is looking into adding an extension base class for about 60 tests that require file permission, then the current base class would indeed set "minimal" permission. So please wait for his update :-) >> >> Best, >> Joe >> >> On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >>> Hi Tristan, >>> >>> I will look at this but doubt I will get to this tomorrow >>> >>> >>> Best, >>> Lance >>> >>> >>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>> Oracle Java Engineering >>> 1 Network Drive >>> Burlington, MA 01803 >>> Lance.Andersen at oracle.com >>> Sent from my iPad >>> >>> On Dec 30, 2014, at 5:28 PM, Tristan Yan wrote: >>> >>>> Hi All >>>> >>>> Can I get your review on this change. >>>> >>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>> >>>> These fixes originally come from bug https://bugs.openjdk.java.net/browse/JDK-8051563 as part of our XML test colocation work. ThIS change-set mainly covers tests for package org.apache.qetest.dtm and org.apache.qetest.trax. >>>> >>>> In the meantime I took steps at fixing some of our existed test code on below issues: >>>> >>>> 1. Add a base test class for all functional tests that enable security manager running. A limited minimal permissions set have been set for every test. >>>> 2. Remove all unnecessary exception capture for functional tests that we?re using testng to handle all the exceptions. >>>> 3. Use try-resource block to solve all possible resource leaks (including InputStream, OutputStream, Writer, Reader). >>>> >>>> Thanks a lot. >>>> Tristan >> > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From bradford.wetmore at oracle.com Tue Jan 6 19:52:10 2015 From: bradford.wetmore at oracle.com (Bradford Wetmore) Date: Tue, 06 Jan 2015 11:52:10 -0800 Subject: RFR: JDK-8047769 SecureRandom should be more frugal with file descriptors In-Reply-To: <54A9696A.3070406@gmail.com> References: <5492D557.2000807@gmail.com> <5498C5A0.50104@oracle.com> <549AA573.2060801@gmail.com> <54A20475.3040206@oracle.com> <54A5AC7C.5010806@gmail.com> <54A7228D.4040801@oracle.com> <54A9696A.3070406@gmail.com> Message-ID: <54AC3CEA.6000702@oracle.com> I only looked at test, looks good to me. I'd rarely ask to remove extra prints in tests. It adds initial debugging data points in case something breaks down the road. Brad On 1/4/2015 8:25 AM, Peter Levart wrote: > Hi Brad, > > So I created another webrev (just removed the unneeded import and > left-over System.out.println from test): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.06/ > > I'll leave it here to rest for a couple of days and if no one objects, > I'll push it to jdk9-dev. > > Thanks everybody for reviews and happy new year! > > Regards, Peter > > On 01/02/2015 11:58 PM, Bradford Wetmore wrote: >> >> On 1/1/2015 12:22 PM, Peter Levart wrote: >>> Hi Brad, >>> >>> Here's next webrev which tries to cover all your comments: >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.04/ >>> >> >> I downloaded the webrev.05 (Chris' later followup email) and ran it >> through JPRT. The only error was the previously seen -Dseed which is >> not your problem. >> >> Again, I only ran: >> >> jdk_lang, jdk_math, jdk_util, jdk_io, jdk_net, jdk_nio, >> jdk_security*, jdk_rmi, jdk_text, jdk_time, jdk_other, core_tools. >> >> If you need anything else run, let me know. >> >>>>>> Looks like you have a committer status, will you be pushing this? >>>>> >>>>> I can, yes. As soon as we clear-out the remaining questions, right? >>>> >>>> Yes. The comments below are minor and shouldn't need another review >>>> cycle. >>> >>> I'm worried about the failure of the test you observed while running >>> from NetBeans. Perhaps a 0.5s wait is sometimes not enough for >>> ReferenceHandler thread to process (enqueue) a WeakReference. Since >>> there is already a facility in place to help ReferenceHandler thread >>> instead of wait for it, I used it in new version of the test. >> >> BTW, there's now an unnecessary import from java.lang.AssertionError >> added in webrev.04. >> >>>> TEST RESULT: Failed. Compilation failed: Compilation failed >>> >>> I changed the test to be self-contained now so one can run it without >>> testlib in classpath. >> >> Thanks. It's compiling fine now. >> >>>> Two minor nits? SeedGenerator.java: Lines 507/518 >>> >>> Done that too. >> >> Thanks. >> >>>> Maybe issue multiple reads to exercise in1 and in2? e.g. 2 bytes of >>>> in1, 4 bytes of in2, then 2 bytes of in1? >>> >>> The 1st assert makes sure in1 == in2, so there's no point in invoking >>> the same instance via two aliases. >> >> True. >> >>>> IIRC, when I ran this under NetBeans last week, the second testCaching >>>> didn't clear the WeakReference. >>> >>> This should not happen any more now that the test is helping to enqueue >>> the WeakReferences instead of waiting for ReferenceHandler to enqueue >>> them. >> >> Yes, that hit the refQueue.poll(). >> >> It's always interesting to work with core-libs folks, learn something >> new everyday. Mixing Lambdas/try-with. >> >> I need a time-machine for your CFV/jdk8 Committer: >> >> http://mail.openjdk.java.net/pipermail/jdk8-dev/2013-August/002896.html >> >> I vote yes. >> >>> The test can now fail only if System.gc() does not trigger >>> WeakReference processing in the VM. Can you give it a try on your >>> NetBeans environment? >> >> One last comment. It's now 2015. ;) >> >> Brad >> > From ivan.gerasimov at oracle.com Tue Jan 6 21:13:27 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Wed, 07 Jan 2015 00:13:27 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AC2A63.1040505@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB050.4010001@Oracle.com> <54AACE94.2000802@oracle.com> <54AAD3A6.9010200@Oracle.com> <54AAE19F.4090102@oracle.com> <54AAF717.3070602@oracle.com> <54AAFE5A.5070406@oracle.com> <54ABE7D4.5010502@oracle.com> <54AC2A63.1040505@oracle.com> Message-ID: <54AC4FF7.6050908@oracle.com> And here's the update: http://cr.openjdk.java.net/~igerasim/8067951/6/webrev/ As you suggested, the psCount is calculated in the first loop along the way. Another change is that a quotation mark is used as the new path separator. I also thought it's better to add a short comment, as now it may look confusing at a first glance. Sincerely yours, Ivan On 06.01.2015 21:33, Xueming Shen wrote: > Hi Ivan, > > Yes, it looks better and should run faster as it doesn't need to worry > about quote > in second loop. I was a little hesitated when suggested to replace the > ps with 0, > though I'm pretty much sure a \u0000 should never be a legal character > in a > windows' path :-) > > Anyone can think about any possibility of having an embedded \0 > character in path? > > Nitpick: > > the "posCount" probably can be calculated in the "quote handling" loop ? > > if (ClassloaderHelper.allowQuotedPathElements ....) { > ... > if (ch == ps) { ch = '\0'; psCount++; } > buf[bufLen++] = ch; > ... > } else { > for (int i = ldPath.indexOf(ps); i >=0 ...) { > } > } > > otherwise, it looks fine for me. > > -Sherman > > On 01/06/2015 05:49 AM, Ivan Gerasimov wrote: >> Hi Sherman! >> >> I took your suggestion and rewrote the method to moved the logic, >> which removes the quotes to the top. >> I think the code became cleaner, so thank you for suggestion! >> >> Here's the updated webrev: >> http://cr.openjdk.java.net/~igerasim/8067951/5/webrev/ >> >> Sincerely yours, >> Ivan >> >> >> On 06.01.2015 0:12, Xueming Shen wrote: >>> On 01/05/2015 12:41 PM, Ivan Gerasimov wrote: >>>> Thanks Sherman! >>>> >>>> On 05.01.2015 22:10, Xueming Shen wrote: >>>>> >>>>> Just wonder if we really need that "inQuotes" logic here? A >>>>> straightforward approach might >>>>> be "every time you have a quote, skip everything until you hit >>>>> another one, when counting, >>>>> or copy everything into the buffer until hit another one, when >>>>> copying" ? >>>>> >>>> I agree it would work, but, in my opinion, it would be a bit more >>>> complicated. >>>> The counting loop would look something like this: >>>> ------------------------------------ >>>> outerLoop: for (int i = 0; i < ldLen; ++i) { >>>> char ch = ldPath.charAt(i); >>>> if (mayBeQuoted && ch == '\"') { >>>> thereAreQuotes = true; >>>> for (++i; i < ldLen; ++i) { >>>> if (ldPath.charAt(i) == '\"') { >>>> continue outerLoop; >>>> } >>>> } >>>> break; // unpaired quote >>>> } else if (ch == ps) { >>>> psCount++; >>>> } >>>> } >>>> ------------------------------------ >>>> which is 3 lines longer, comparing to the loop with inQuotes flag. >>>> >>> >>> It does not seem like we are doing anything special for "unpaired >>> quote" (just ignore it?), >>> if that is the case, you probably don't need to do anything for it, >>> the code could be >>> something like >>> >>> for (int i = 0; i < ldLen; ++i) { >>> char ch = ldPath.charAt(i); >>> if (mayBeQuoted && ch == '\"') { >>> thereAreQuotes = true; >>> while (++i < ldLen && ldPath.charAt(i) != '\"') {} >>> } else if (ch == ps) { >>> psCount++; >>> } >>> } >>> >>> I have not tried to debug the code though :-) Just an opinion here. >>> >>> -Sherman >>> >>> >> > > > From huizhe.wang at oracle.com Tue Jan 6 21:22:05 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Tue, 06 Jan 2015 13:22:05 -0800 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> Message-ID: <54AC51FD.7040808@oracle.com> Thanks for taking the initiative and effort to refactor and clean up all of the Functional tests in previous changeset! We've gone through several iterations for the new ones (xslt tests). I totally agree with Lance, it looks very good overall, a lot better than its original form. It's nice to group all of the tests that required file access, that prepares all other tests to run with minimal permission for a (future) secure-test-target. Unit tests might need a similar treatment, but no pressure to do now :-) A minor comment on utility classes: there's JAXPTestUtilities and TestUtils. The former is an util for all tests while the later contained a couple of SAX handlers, it may make sense to move them into their own classes just as the other Handlers. The constants (XML_DIR and GOLDEN_DIR) then can be declared in a base class for the SAX tests. I understand each group of tests have their own XML source and golden files, thus XML_DIR and GOLDEN_DIR. It may be possible to add a base for each group while put test.src and test.classes into the very base class for all tests. So in general, we would have: TestBase FileTestBase Base for each group (e.g. SAXTestBase, DOMTestBase, XSLTTestBase...) that extends either TestBase or FileTestBase I remember there were a few tests that required a http server. So then we may need a HttpTestBase as well. I know this is not what we've discussed (or planned) previously. But since you've done a great job to incorporate what were previously quite separate test suites into one whole test suite. I can't resist to ask a bit more. Don't get me wrong, what you've done exceeded my expectation in a big way (only a month ago, we were still talking about quick/straight conversion so that you can start to take care of the new features)! SAXParserTest: I noticed Old: testParse11, testParse27 --> New: testParse05. So is testParse03 a new test? I can see SAXException is expected, but not IOE. In fact, this shows that the JAXP spec missed defining how empty string shall be handled (should have been an IAE). Best, Joe On 1/6/2015 11:33 AM, Lance Andersen wrote: > Hi Tristan, > > Sorry for the delay but with the holidays and the number of files, it > took a while to go through :-) > > Overall, it looks pretty good. > > A couple of suggestions, but I would not necessarily hold back from > committing: > > - For tests where you are not caring about the actual exception that > is thrown to indicate a failure, such as > in DocumentBuilderFactory01.java and testCheckSchemaSupport1, I would > just have the method declaration use "throws Exception" vs list all > of the possible individual Exceptions, as it keeps it more compact. > Glad to see you are not using failUnexcepted() now. > > - In test classes such as in testCheckSchemaSupport3. java > and DocumentBuilderImpl01.java, some tests do not use assertXXX or > expect an Exception. Would be good at least to document what could > cause a failure or make it clear the expected behavior of the test for > passing. > > -SAXParserTest02.java and other tests where you get a reader/parser > such as testXMLReader01, I would at least assert that null is not > returned where as it is now, you only validate that an exception is > not returned > > - I know you are porting existing tests, but I would consider > consolidating tests as it seems like there is not a real reason to > have a testNG class with just 1 test. I would group the like tests > (such as SAXTFactoryTest ) in one testNG test class as that allows > you to streamline further. > > - TfClearParamTest.java (as and example) minor nit that you have your > @Before/AfterGroups method in between tests. I would suggest grouping > all methods such as this DataProviders before or after the actual > tests for better organization > > - TraxSAXWrapper.java, not sure I understand the "Sorry I could not > resist comment" > > - TransformerHandlerAPITest.java has typos in comments: > "IllegalArgumentExceptionis thrown?." > > - Minitest.java, I would add a comment for your Data Provider > > Best, > Lance > > On Jan 2, 2015, at 1:49 PM, Tristan Yan > wrote: > >> Hi Joe and Lance >> Sorry for my late reply. I just uploaded the new webrev which is >> trying to limit minimal permissions for most of tests. The new >> changeset has two base classes; class JAXPBaseTest has only minimal >> set permissions; class JAXPFileBaseTest adds two permissions that >> need access local files in the directory directory test.src and >> test.classes. Most of tests only inherit JAXPBaseTest that provides >> only minimal set permissions. Some tests will >> inherit JAXPFileBaseTest because tests need access local files. >> Please help to review the code. >> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >> >> >> Thank you >> Tristan >> >> >>> On Jan 2, 2015, at 10:39 AM, huizhe wang >> > wrote: >>> >>> Lance, >>> >>> Tristan is looking into adding an extension base class for about 60 >>> tests that require file permission, then the current base class >>> would indeed set "minimal" permission. So please wait for his update :-) >>> >>> Best, >>> Joe >>> >>> On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >>>> Hi Tristan, >>>> >>>> I will look at this but doubt I will get to this tomorrow >>>> >>>> >>>> Best, >>>> Lance >>>> >>>> >>>> Lance >>>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>> >>>> Oracle Java Engineering >>>> 1 Network Drive >>>> Burlington, MA 01803 >>>> Lance.Andersen at oracle.com >>>> Sent from my iPad >>>> >>>> On Dec 30, 2014, at 5:28 PM, Tristan Yan >>> > wrote: >>>> >>>>> Hi All >>>>> >>>>> Can I get your review on this change. >>>>> >>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>> >>>>> >>>> > >>>>> >>>>> These fixes originally come from bug >>>>> https://bugs.openjdk.java.net/browse/JDK-8051563 >>>>> as part of our >>>>> XML test colocation work. ThIS change-set mainly covers tests for >>>>> package org.apache.qetest.dtm and org.apache.qetest.trax. >>>>> >>>>> In the meantime I took steps at fixing some of our existed test >>>>> code on below issues: >>>>> >>>>> 1. Add a base test class for all functional tests that enable >>>>> security manager running. A limited minimal permissions set have >>>>> been set for every test. >>>>> 2. Remove all unnecessary exception capture for functional tests >>>>> that we?re using testng to handle all the exceptions. >>>>> 3. Use try-resource block to solve all possible resource leaks >>>>> (including InputStream, OutputStream, Writer, Reader). >>>>> >>>>> Thanks a lot. >>>>> Tristan >>> >> > > > > Lance > Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > From tristan.yan at oracle.com Tue Jan 6 21:31:54 2015 From: tristan.yan at oracle.com (Tristan Yan) Date: Tue, 6 Jan 2015 13:31:54 -0800 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: <54AC51FD.7040808@oracle.com> References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> <54AC51FD.7040808@oracle.com> Message-ID: Thank you Lance and Joe. I am working on the fix. it may take a couple of days that the code needs some refactor. I will send out next review once I finish it. Thanks Tristan > On Jan 6, 2015, at 1:22 PM, huizhe wang wrote: > > Thanks for taking the initiative and effort to refactor and clean up all of the Functional tests in previous changeset! > > We've gone through several iterations for the new ones (xslt tests). I totally agree with Lance, it looks very good overall, a lot better than its original form. > > It's nice to group all of the tests that required file access, that prepares all other tests to run with minimal permission for a (future) secure-test-target. Unit tests might need a similar treatment, but no pressure to do now :-) > > A minor comment on utility classes: there's JAXPTestUtilities and TestUtils. The former is an util for all tests while the later contained a couple of SAX handlers, it may make sense to move them into their own classes just as the other Handlers. The constants (XML_DIR and GOLDEN_DIR) then can be declared in a base class for the SAX tests. > > I understand each group of tests have their own XML source and golden files, thus XML_DIR and GOLDEN_DIR. It may be possible to add a base for each group while put test.src and test.classes into the very base class for all tests. So in general, we would have: > TestBase FileTestBase > Base for each group (e.g. SAXTestBase, DOMTestBase, XSLTTestBase...) that extends either TestBase or FileTestBase > > I remember there were a few tests that required a http server. So then we may need a HttpTestBase as well. > > I know this is not what we've discussed (or planned) previously. But since you've done a great job to incorporate what were previously quite separate test suites into one whole test suite. I can't resist to ask a bit more. Don't get me wrong, what you've done exceeded my expectation in a big way (only a month ago, we were still talking about quick/straight conversion so that you can start to take care of the new features)! > > SAXParserTest: I noticed Old: testParse11, testParse27 --> New: testParse05. So is testParse03 a new test? I can see SAXException is expected, but not IOE. In fact, this shows that the JAXP spec missed defining how empty string shall be handled (should have been an IAE). > > Best, > Joe > > On 1/6/2015 11:33 AM, Lance Andersen wrote: >> Hi Tristan, >> >> Sorry for the delay but with the holidays and the number of files, it took a while to go through :-) >> >> Overall, it looks pretty good. >> >> A couple of suggestions, but I would not necessarily hold back from committing: >> >> - For tests where you are not caring about the actual exception that is thrown to indicate a failure, such as in DocumentBuilderFactory01.java and testCheckSchemaSupport1, I would just have the method declaration use "throws Exception" vs list all of the possible individual Exceptions, as it keeps it more compact. Glad to see you are not using failUnexcepted() now. >> >> - In test classes such as in testCheckSchemaSupport3. java and DocumentBuilderImpl01.java, some tests do not use assertXXX or expect an Exception. Would be good at least to document what could cause a failure or make it clear the expected behavior of the test for passing. >> >> -SAXParserTest02.java and other tests where you get a reader/parser such as testXMLReader01, I would at least assert that null is not returned where as it is now, you only validate that an exception is not returned >> >> - I know you are porting existing tests, but I would consider consolidating tests as it seems like there is not a real reason to have a testNG class with just 1 test. I would group the like tests (such as SAXTFactoryTest ) in one testNG test class as that allows you to streamline further. >> >> - TfClearParamTest.java (as and example) minor nit that you have your @Before/AfterGroups method in between tests. I would suggest grouping all methods such as this DataProviders before or after the actual tests for better organization >> >> - TraxSAXWrapper.java, not sure I understand the "Sorry I could not resist comment" >> >> - TransformerHandlerAPITest.java has typos in comments: "IllegalArgumentExceptionis thrown?." >> >> - Minitest.java, I would add a comment for your Data Provider >> >> Best, >> Lance >> >> On Jan 2, 2015, at 1:49 PM, Tristan Yan > wrote: >> >>> Hi Joe and Lance >>> Sorry for my late reply. I just uploaded the new webrev which is trying to limit minimal permissions for most of tests. The new changeset has two base classes; class JAXPBaseTest has only minimal set permissions; class JAXPFileBaseTest adds two permissions that need access local files in the directory directory test.src and test.classes. Most of tests only inherit JAXPBaseTest that provides only minimal set permissions. Some tests will inherit JAXPFileBaseTest because tests need access local files. >>> Please help to review the code. >>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>> >>> Thank you >>> Tristan >>> >>> >>>> On Jan 2, 2015, at 10:39 AM, huizhe wang > wrote: >>>> >>>> Lance, >>>> >>>> Tristan is looking into adding an extension base class for about 60 tests that require file permission, then the current base class would indeed set "minimal" permission. So please wait for his update :-) >>>> >>>> Best, >>>> Joe >>>> >>>> On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >>>>> Hi Tristan, >>>>> >>>>> I will look at this but doubt I will get to this tomorrow >>>>> >>>>> >>>>> Best, >>>>> Lance >>>>> >>>>> >>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>> Oracle Java Engineering >>>>> 1 Network Drive >>>>> Burlington, MA 01803 >>>>> Lance.Andersen at oracle.com >>>>> Sent from my iPad >>>>> >>>>> On Dec 30, 2014, at 5:28 PM, Tristan Yan > wrote: >>>>> >>>>>> Hi All >>>>>> >>>>>> Can I get your review on this change. >>>>>> >>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ > >>>>>> >>>>>> These fixes originally come from bug https://bugs.openjdk.java.net/browse/JDK-8051563 > as part of our XML test colocation work. ThIS change-set mainly covers tests for package org.apache.qetest.dtm and org.apache.qetest.trax. >>>>>> >>>>>> In the meantime I took steps at fixing some of our existed test code on below issues: >>>>>> >>>>>> 1. Add a base test class for all functional tests that enable security manager running. A limited minimal permissions set have been set for every test. >>>>>> 2. Remove all unnecessary exception capture for functional tests that we?re using testng to handle all the exceptions. >>>>>> 3. Use try-resource block to solve all possible resource leaks (including InputStream, OutputStream, Writer, Reader). >>>>>> >>>>>> Thanks a lot. >>>>>> Tristan >>>> >>> >> >> >> >> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> >> >> > From lance.andersen at oracle.com Tue Jan 6 22:04:10 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Tue, 6 Jan 2015 17:04:10 -0500 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> <54AC51FD.7040808@oracle.com> Message-ID: <078315A4-D463-4891-99CE-6A1785BB5D49@oracle.com> On Jan 6, 2015, at 4:31 PM, Tristan Yan wrote: > Thank you Lance and Joe. You are very welcome. > I am working on the fix. No rush from my perspective, have a lot to keep me busy in the interim before your next webrev . :-) > it may take a couple of days that the code needs some refactor. Understand, as I have been working on tests for RowSets, I have continued to play the refactor dance myself. Best, Lance > I will send out next review once I finish it. > Thanks > Tristan > >> On Jan 6, 2015, at 1:22 PM, huizhe wang wrote: >> >> Thanks for taking the initiative and effort to refactor and clean up all of the Functional tests in previous changeset! >> >> We've gone through several iterations for the new ones (xslt tests). I totally agree with Lance, it looks very good overall, a lot better than its original form. >> >> It's nice to group all of the tests that required file access, that prepares all other tests to run with minimal permission for a (future) secure-test-target. Unit tests might need a similar treatment, but no pressure to do now :-) >> >> A minor comment on utility classes: there's JAXPTestUtilities and TestUtils. The former is an util for all tests while the later contained a couple of SAX handlers, it may make sense to move them into their own classes just as the other Handlers. The constants (XML_DIR and GOLDEN_DIR) then can be declared in a base class for the SAX tests. >> >> I understand each group of tests have their own XML source and golden files, thus XML_DIR and GOLDEN_DIR. It may be possible to add a base for each group while put test.src and test.classes into the very base class for all tests. So in general, we would have: >> TestBase FileTestBase >> Base for each group (e.g. SAXTestBase, DOMTestBase, XSLTTestBase...) that extends either TestBase or FileTestBase >> >> I remember there were a few tests that required a http server. So then we may need a HttpTestBase as well. >> >> I know this is not what we've discussed (or planned) previously. But since you've done a great job to incorporate what were previously quite separate test suites into one whole test suite. I can't resist to ask a bit more. Don't get me wrong, what you've done exceeded my expectation in a big way (only a month ago, we were still talking about quick/straight conversion so that you can start to take care of the new features)! >> >> SAXParserTest: I noticed Old: testParse11, testParse27 --> New: testParse05. So is testParse03 a new test? I can see SAXException is expected, but not IOE. In fact, this shows that the JAXP spec missed defining how empty string shall be handled (should have been an IAE). >> >> Best, >> Joe >> >> On 1/6/2015 11:33 AM, Lance Andersen wrote: >>> Hi Tristan, >>> >>> Sorry for the delay but with the holidays and the number of files, it took a while to go through :-) >>> >>> Overall, it looks pretty good. >>> >>> A couple of suggestions, but I would not necessarily hold back from committing: >>> >>> - For tests where you are not caring about the actual exception that is thrown to indicate a failure, such as in DocumentBuilderFactory01.java and testCheckSchemaSupport1, I would just have the method declaration use "throws Exception" vs list all of the possible individual Exceptions, as it keeps it more compact. Glad to see you are not using failUnexcepted() now. >>> >>> - In test classes such as in testCheckSchemaSupport3. java and DocumentBuilderImpl01.java, some tests do not use assertXXX or expect an Exception. Would be good at least to document what could cause a failure or make it clear the expected behavior of the test for passing. >>> >>> -SAXParserTest02.java and other tests where you get a reader/parser such as testXMLReader01, I would at least assert that null is not returned where as it is now, you only validate that an exception is not returned >>> >>> - I know you are porting existing tests, but I would consider consolidating tests as it seems like there is not a real reason to have a testNG class with just 1 test. I would group the like tests (such as SAXTFactoryTest ) in one testNG test class as that allows you to streamline further. >>> >>> - TfClearParamTest.java (as and example) minor nit that you have your @Before/AfterGroups method in between tests. I would suggest grouping all methods such as this DataProviders before or after the actual tests for better organization >>> >>> - TraxSAXWrapper.java, not sure I understand the "Sorry I could not resist comment" >>> >>> - TransformerHandlerAPITest.java has typos in comments: "IllegalArgumentExceptionis thrown?." >>> >>> - Minitest.java, I would add a comment for your Data Provider >>> >>> Best, >>> Lance >>> >>> On Jan 2, 2015, at 1:49 PM, Tristan Yan wrote: >>> >>>> Hi Joe and Lance >>>> Sorry for my late reply. I just uploaded the new webrev which is trying to limit minimal permissions for most of tests. The new changeset has two base classes; class JAXPBaseTest has only minimal set permissions; class JAXPFileBaseTest adds two permissions that need access local files in the directory directory test.src and test.classes. Most of tests only inherit JAXPBaseTest that provides only minimal set permissions. Some tests will inherit JAXPFileBaseTest because tests need access local files. >>>> Please help to review the code. >>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>> >>>> Thank you >>>> Tristan >>>> >>>> >>>>> On Jan 2, 2015, at 10:39 AM, huizhe wang wrote: >>>>> >>>>> Lance, >>>>> >>>>> Tristan is looking into adding an extension base class for about 60 tests that require file permission, then the current base class would indeed set "minimal" permission. So please wait for his update :-) >>>>> >>>>> Best, >>>>> Joe >>>>> >>>>> On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >>>>>> Hi Tristan, >>>>>> >>>>>> I will look at this but doubt I will get to this tomorrow >>>>>> >>>>>> >>>>>> Best, >>>>>> Lance >>>>>> >>>>>> >>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>> Oracle Java Engineering >>>>>> 1 Network Drive >>>>>> Burlington, MA 01803 >>>>>> Lance.Andersen at oracle.com >>>>>> Sent from my iPad >>>>>> >>>>>> On Dec 30, 2014, at 5:28 PM, Tristan Yan wrote: >>>>>> >>>>>>> Hi All >>>>>>> >>>>>>> Can I get your review on this change. >>>>>>> >>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>>>> >>>>>>> These fixes originally come from bug https://bugs.openjdk.java.net/browse/JDK-8051563 as part of our XML test colocation work. ThIS change-set mainly covers tests for package org.apache.qetest.dtm and org.apache.qetest.trax. >>>>>>> >>>>>>> In the meantime I took steps at fixing some of our existed test code on below issues: >>>>>>> >>>>>>> 1. Add a base test class for all functional tests that enable security manager running. A limited minimal permissions set have been set for every test. >>>>>>> 2. Remove all unnecessary exception capture for functional tests that we?re using testng to handle all the exceptions. >>>>>>> 3. Use try-resource block to solve all possible resource leaks (including InputStream, OutputStream, Writer, Reader). >>>>>>> >>>>>>> Thanks a lot. >>>>>>> Tristan >>>>> >>>> >>> >>> >>> >>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>> Oracle Java Engineering >>> 1 Network Drive >>> Burlington, MA 01803 >>> Lance.Andersen at oracle.com >>> >>> >>> >> > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From lance.andersen at oracle.com Tue Jan 6 22:25:10 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Tue, 6 Jan 2015 17:25:10 -0500 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: <078315A4-D463-4891-99CE-6A1785BB5D49@oracle.com> References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> <54AC51FD.7040808@oracle.com> <078315A4-D463-4891-99CE-6A1785BB5D49@oracle.com> Message-ID: <4B99A07A-A547-4CB5-8E6E-98EA57762906@oracle.com> One more thing :-) Remember to update your copyright year to 2015 also Best Lance On Jan 6, 2015, at 5:04 PM, Lance Andersen wrote: > > On Jan 6, 2015, at 4:31 PM, Tristan Yan wrote: > >> Thank you Lance and Joe. > > You are very welcome. >> I am working on the fix. > > No rush from my perspective, have a lot to keep me busy in the interim before your next webrev . :-) >> it may take a couple of days that the code needs some refactor. > Understand, as I have been working on tests for RowSets, I have continued to play the refactor dance myself. > > Best, > Lance >> I will send out next review once I finish it. >> Thanks >> Tristan >> >>> On Jan 6, 2015, at 1:22 PM, huizhe wang wrote: >>> >>> Thanks for taking the initiative and effort to refactor and clean up all of the Functional tests in previous changeset! >>> >>> We've gone through several iterations for the new ones (xslt tests). I totally agree with Lance, it looks very good overall, a lot better than its original form. >>> >>> It's nice to group all of the tests that required file access, that prepares all other tests to run with minimal permission for a (future) secure-test-target. Unit tests might need a similar treatment, but no pressure to do now :-) >>> >>> A minor comment on utility classes: there's JAXPTestUtilities and TestUtils. The former is an util for all tests while the later contained a couple of SAX handlers, it may make sense to move them into their own classes just as the other Handlers. The constants (XML_DIR and GOLDEN_DIR) then can be declared in a base class for the SAX tests. >>> >>> I understand each group of tests have their own XML source and golden files, thus XML_DIR and GOLDEN_DIR. It may be possible to add a base for each group while put test.src and test.classes into the very base class for all tests. So in general, we would have: >>> TestBase FileTestBase >>> Base for each group (e.g. SAXTestBase, DOMTestBase, XSLTTestBase...) that extends either TestBase or FileTestBase >>> >>> I remember there were a few tests that required a http server. So then we may need a HttpTestBase as well. >>> >>> I know this is not what we've discussed (or planned) previously. But since you've done a great job to incorporate what were previously quite separate test suites into one whole test suite. I can't resist to ask a bit more. Don't get me wrong, what you've done exceeded my expectation in a big way (only a month ago, we were still talking about quick/straight conversion so that you can start to take care of the new features)! >>> >>> SAXParserTest: I noticed Old: testParse11, testParse27 --> New: testParse05. So is testParse03 a new test? I can see SAXException is expected, but not IOE. In fact, this shows that the JAXP spec missed defining how empty string shall be handled (should have been an IAE). >>> >>> Best, >>> Joe >>> >>> On 1/6/2015 11:33 AM, Lance Andersen wrote: >>>> Hi Tristan, >>>> >>>> Sorry for the delay but with the holidays and the number of files, it took a while to go through :-) >>>> >>>> Overall, it looks pretty good. >>>> >>>> A couple of suggestions, but I would not necessarily hold back from committing: >>>> >>>> - For tests where you are not caring about the actual exception that is thrown to indicate a failure, such as in DocumentBuilderFactory01.java and testCheckSchemaSupport1, I would just have the method declaration use "throws Exception" vs list all of the possible individual Exceptions, as it keeps it more compact. Glad to see you are not using failUnexcepted() now. >>>> >>>> - In test classes such as in testCheckSchemaSupport3. java and DocumentBuilderImpl01.java, some tests do not use assertXXX or expect an Exception. Would be good at least to document what could cause a failure or make it clear the expected behavior of the test for passing. >>>> >>>> -SAXParserTest02.java and other tests where you get a reader/parser such as testXMLReader01, I would at least assert that null is not returned where as it is now, you only validate that an exception is not returned >>>> >>>> - I know you are porting existing tests, but I would consider consolidating tests as it seems like there is not a real reason to have a testNG class with just 1 test. I would group the like tests (such as SAXTFactoryTest ) in one testNG test class as that allows you to streamline further. >>>> >>>> - TfClearParamTest.java (as and example) minor nit that you have your @Before/AfterGroups method in between tests. I would suggest grouping all methods such as this DataProviders before or after the actual tests for better organization >>>> >>>> - TraxSAXWrapper.java, not sure I understand the "Sorry I could not resist comment" >>>> >>>> - TransformerHandlerAPITest.java has typos in comments: "IllegalArgumentExceptionis thrown?." >>>> >>>> - Minitest.java, I would add a comment for your Data Provider >>>> >>>> Best, >>>> Lance >>>> >>>> On Jan 2, 2015, at 1:49 PM, Tristan Yan wrote: >>>> >>>>> Hi Joe and Lance >>>>> Sorry for my late reply. I just uploaded the new webrev which is trying to limit minimal permissions for most of tests. The new changeset has two base classes; class JAXPBaseTest has only minimal set permissions; class JAXPFileBaseTest adds two permissions that need access local files in the directory directory test.src and test.classes. Most of tests only inherit JAXPBaseTest that provides only minimal set permissions. Some tests will inherit JAXPFileBaseTest because tests need access local files. >>>>> Please help to review the code. >>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>> >>>>> Thank you >>>>> Tristan >>>>> >>>>> >>>>>> On Jan 2, 2015, at 10:39 AM, huizhe wang wrote: >>>>>> >>>>>> Lance, >>>>>> >>>>>> Tristan is looking into adding an extension base class for about 60 tests that require file permission, then the current base class would indeed set "minimal" permission. So please wait for his update :-) >>>>>> >>>>>> Best, >>>>>> Joe >>>>>> >>>>>> On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >>>>>>> Hi Tristan, >>>>>>> >>>>>>> I will look at this but doubt I will get to this tomorrow >>>>>>> >>>>>>> >>>>>>> Best, >>>>>>> Lance >>>>>>> >>>>>>> >>>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>> Oracle Java Engineering >>>>>>> 1 Network Drive >>>>>>> Burlington, MA 01803 >>>>>>> Lance.Andersen at oracle.com >>>>>>> Sent from my iPad >>>>>>> >>>>>>> On Dec 30, 2014, at 5:28 PM, Tristan Yan wrote: >>>>>>> >>>>>>>> Hi All >>>>>>>> >>>>>>>> Can I get your review on this change. >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>>>>> >>>>>>>> These fixes originally come from bug https://bugs.openjdk.java.net/browse/JDK-8051563 as part of our XML test colocation work. ThIS change-set mainly covers tests for package org.apache.qetest.dtm and org.apache.qetest.trax. >>>>>>>> >>>>>>>> In the meantime I took steps at fixing some of our existed test code on below issues: >>>>>>>> >>>>>>>> 1. Add a base test class for all functional tests that enable security manager running. A limited minimal permissions set have been set for every test. >>>>>>>> 2. Remove all unnecessary exception capture for functional tests that we?re using testng to handle all the exceptions. >>>>>>>> 3. Use try-resource block to solve all possible resource leaks (including InputStream, OutputStream, Writer, Reader). >>>>>>>> >>>>>>>> Thanks a lot. >>>>>>>> Tristan >>>>>> >>>>> >>>> >>>> >>>> >>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>> Oracle Java Engineering >>>> 1 Network Drive >>>> Burlington, MA 01803 >>>> Lance.Andersen at oracle.com >>>> >>>> >>>> >>> >> > > > > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From huizhe.wang at oracle.com Tue Jan 6 22:27:45 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Tue, 06 Jan 2015 14:27:45 -0800 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: <4B99A07A-A547-4CB5-8E6E-98EA57762906@oracle.com> References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> <54AC51FD.7040808@oracle.com> <078315A4-D463-4891-99CE-6A1785BB5D49@oracle.com> <4B99A07A-A547-4CB5-8E6E-98EA57762906@oracle.com> Message-ID: <54AC6161.1000308@oracle.com> On 1/6/2015 2:25 PM, Lance Andersen wrote: > One more thing :-) > > Remember to update your copyright year to 2015 also Indeed, that applies to my webrevs as well :-) Joe > > Best > Lance > On Jan 6, 2015, at 5:04 PM, Lance Andersen > wrote: > >> >> On Jan 6, 2015, at 4:31 PM, Tristan Yan > > wrote: >> >>> Thank you Lance and Joe. >> >> You are very welcome. >>> I am working on the fix. >> >> No rush from my perspective, have a lot to keep me busy in the >> interim before your next webrev . :-) >>> it may take a couple of days that the code needs some refactor. >> Understand, as I have been working on tests for RowSets, I have >> continued to play the refactor dance myself. >> >> Best, >> Lance >>> I will send out next review once I finish it. >>> Thanks >>> Tristan >>> >>>> On Jan 6, 2015, at 1:22 PM, huizhe wang >>> > wrote: >>>> >>>> Thanks for taking the initiative and effort to refactor and clean >>>> up all of the Functional tests in previous changeset! >>>> >>>> We've gone through several iterations for the new ones (xslt >>>> tests). I totally agree with Lance, it looks very good overall, a >>>> lot better than its original form. >>>> >>>> It's nice to group all of the tests that required file access, that >>>> prepares all other tests to run with minimal permission for a >>>> (future) secure-test-target. Unit tests might need a similar >>>> treatment, but no pressure to do now :-) >>>> >>>> A minor comment on utility classes: there's JAXPTestUtilities and >>>> TestUtils. The former is an util for all tests while the later >>>> contained a couple of SAX handlers, it may make sense to move them >>>> into their own classes just as the other Handlers. The constants >>>> (XML_DIR and GOLDEN_DIR) then can be declared in a base class for >>>> the SAX tests. >>>> >>>> I understand each group of tests have their own XML source and >>>> golden files, thus XML_DIR and GOLDEN_DIR. It may be possible to >>>> add a base for each group while put test.src and test.classes into >>>> the very base class for all tests. So in general, we would have: >>>> TestBase FileTestBase >>>> Base for each group (e.g. SAXTestBase, DOMTestBase, >>>> XSLTTestBase...) that extends either TestBase or FileTestBase >>>> >>>> I remember there were a few tests that required a http server. So >>>> then we may need a HttpTestBase as well. >>>> >>>> I know this is not what we've discussed (or planned) previously. >>>> But since you've done a great job to incorporate what were >>>> previously quite separate test suites into one whole test suite. I >>>> can't resist to ask a bit more. Don't get me wrong, what you've >>>> done exceeded my expectation in a big way (only a month ago, we >>>> were still talking about quick/straight conversion so that you can >>>> start to take care of the new features)! >>>> >>>> SAXParserTest: I noticed Old: testParse11, testParse27 --> New: >>>> testParse05. So is testParse03 a new test? I can see SAXException >>>> is expected, but not IOE. In fact, this shows that the JAXP spec >>>> missed defining how empty string shall be handled (should have been >>>> an IAE). >>>> >>>> Best, >>>> Joe >>>> >>>> On 1/6/2015 11:33 AM, Lance Andersen wrote: >>>>> Hi Tristan, >>>>> >>>>> Sorry for the delay but with the holidays and the number of files, >>>>> it took a while to go through :-) >>>>> >>>>> Overall, it looks pretty good. >>>>> >>>>> A couple of suggestions, but I would not necessarily hold back >>>>> from committing: >>>>> >>>>> - For tests where you are not caring about the actual exception >>>>> that is thrown to indicate a failure, such as >>>>> in DocumentBuilderFactory01.java and testCheckSchemaSupport1, I >>>>> would just have the method declaration use "throws Exception" vs >>>>> list all of the possible individual Exceptions, as it keeps it >>>>> more compact. Glad to see you are not using failUnexcepted() now. >>>>> >>>>> - In test classes such as in testCheckSchemaSupport3. java >>>>> and DocumentBuilderImpl01.java, some tests do not use assertXXX or >>>>> expect an Exception. Would be good at least to document what >>>>> could cause a failure or make it clear the expected behavior of >>>>> the test for passing. >>>>> >>>>> -SAXParserTest02.java and other tests where you get a >>>>> reader/parser such as testXMLReader01, I would at least assert >>>>> that null is not returned where as it is now, you only validate >>>>> that an exception is not returned >>>>> >>>>> - I know you are porting existing tests, but I would consider >>>>> consolidating tests as it seems like there is not a real reason to >>>>> have a testNG class with just 1 test. I would group the like >>>>> tests (such as SAXTFactoryTest ) in one testNG test class as that >>>>> allows you to streamline further. >>>>> >>>>> - TfClearParamTest.java (as and example) minor nit that you have >>>>> your @Before/AfterGroups method in between tests. I would suggest >>>>> grouping all methods such as this DataProviders before or after >>>>> the actual tests for better organization >>>>> >>>>> - TraxSAXWrapper.java, not sure I understand the "Sorry I could >>>>> not resist comment" >>>>> >>>>> - TransformerHandlerAPITest.java has typos in comments: >>>>> "IllegalArgumentExceptionis thrown?." >>>>> >>>>> - Minitest.java, I would add a comment for your Data Provider >>>>> >>>>> Best, >>>>> Lance >>>>> >>>>> On Jan 2, 2015, at 1:49 PM, Tristan Yan >>>> > wrote: >>>>> >>>>>> Hi Joe and Lance >>>>>> Sorry for my late reply. I just uploaded the new webrev which is >>>>>> trying to limit minimal permissions for most of tests. The new >>>>>> changeset has two base classes; class JAXPBaseTest has only >>>>>> minimal set permissions; class JAXPFileBaseTest adds two >>>>>> permissions that need access local files in the directory >>>>>> directory test.src and test.classes. Most of tests only >>>>>> inherit JAXPBaseTest that provides only minimal set permissions. >>>>>> Some tests will inherit JAXPFileBaseTest because tests need >>>>>> access local files. >>>>>> Please help to review the code. >>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>>> >>>>>> >>>>>> Thank you >>>>>> Tristan >>>>>> >>>>>> >>>>>>> On Jan 2, 2015, at 10:39 AM, huizhe wang >>>>>> > wrote: >>>>>>> >>>>>>> Lance, >>>>>>> >>>>>>> Tristan is looking into adding an extension base class for about >>>>>>> 60 tests that require file permission, then the current base >>>>>>> class would indeed set "minimal" permission. So please wait for >>>>>>> his update :-) >>>>>>> >>>>>>> Best, >>>>>>> Joe >>>>>>> >>>>>>> On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >>>>>>>> Hi Tristan, >>>>>>>> >>>>>>>> I will look at this but doubt I will get to this tomorrow >>>>>>>> >>>>>>>> >>>>>>>> Best, >>>>>>>> Lance >>>>>>>> >>>>>>>> >>>>>>>> Lance >>>>>>>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>>> >>>>>>>> Oracle Java Engineering >>>>>>>> 1 Network Drive >>>>>>>> Burlington, MA 01803 >>>>>>>> Lance.Andersen at oracle.com >>>>>>>> Sent from my iPad >>>>>>>> >>>>>>>> On Dec 30, 2014, at 5:28 PM, Tristan Yan >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi All >>>>>>>>> >>>>>>>>> Can I get your review on this change. >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>>>>>> >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>>> These fixes originally come from bug >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8051563 >>>>>>>>> as part of >>>>>>>>> our XML test colocation work. ThIS change-set mainly covers >>>>>>>>> tests for package org.apache.qetest.dtm and >>>>>>>>> org.apache.qetest.trax. >>>>>>>>> >>>>>>>>> In the meantime I took steps at fixing some of our existed >>>>>>>>> test code on below issues: >>>>>>>>> >>>>>>>>> 1. Add a base test class for all functional tests that enable >>>>>>>>> security manager running. A limited minimal permissions set >>>>>>>>> have been set for every test. >>>>>>>>> 2. Remove all unnecessary exception capture for functional >>>>>>>>> tests that we?re using testng to handle all the exceptions. >>>>>>>>> 3. Use try-resource block to solve all possible resource leaks >>>>>>>>> (including InputStream, OutputStream, Writer, Reader). >>>>>>>>> >>>>>>>>> Thanks a lot. >>>>>>>>> Tristan >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Lance >>>>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>> Oracle Java Engineering >>>>> 1 Network Drive >>>>> Burlington, MA 01803 >>>>> Lance.Andersen at oracle.com >>>>> >>>>> >>>>> >>>> >>> >> >> >> >> >> Lance >> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> >> >> > > > > Lance > Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > From tristan.yan at oracle.com Tue Jan 6 22:28:45 2015 From: tristan.yan at oracle.com (Tristan Yan) Date: Tue, 6 Jan 2015 14:28:45 -0800 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: <54AC6161.1000308@oracle.com> References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> <54AC51FD.7040808@oracle.com> <078315A4-D463-4891-99CE-6A1785BB5D49@oracle.com> <4B99A07A-A547-4CB5-8E6E-98EA57762906@oracle.com> <54AC6161.1000308@oracle.com> Message-ID: <0A5ABF38-7B96-4E40-B0EF-D089BBE1F01B@oracle.com> Thanks, I will update it. Tristan > On Jan 6, 2015, at 2:27 PM, huizhe wang wrote: > > > On 1/6/2015 2:25 PM, Lance Andersen wrote: >> One more thing :-) >> >> Remember to update your copyright year to 2015 also > > Indeed, that applies to my webrevs as well :-) > > Joe > >> >> Best >> Lance >> On Jan 6, 2015, at 5:04 PM, Lance Andersen > wrote: >> >>> >>> On Jan 6, 2015, at 4:31 PM, Tristan Yan > wrote: >>> >>>> Thank you Lance and Joe. >>> >>> You are very welcome. >>>> I am working on the fix. >>> >>> No rush from my perspective, have a lot to keep me busy in the interim before your next webrev . :-) >>>> it may take a couple of days that the code needs some refactor. >>> Understand, as I have been working on tests for RowSets, I have continued to play the refactor dance myself. >>> >>> Best, >>> Lance >>>> I will send out next review once I finish it. >>>> Thanks >>>> Tristan >>>> >>>>> On Jan 6, 2015, at 1:22 PM, huizhe wang > wrote: >>>>> >>>>> Thanks for taking the initiative and effort to refactor and clean up all of the Functional tests in previous changeset! >>>>> >>>>> We've gone through several iterations for the new ones (xslt tests). I totally agree with Lance, it looks very good overall, a lot better than its original form. >>>>> >>>>> It's nice to group all of the tests that required file access, that prepares all other tests to run with minimal permission for a (future) secure-test-target. Unit tests might need a similar treatment, but no pressure to do now :-) >>>>> >>>>> A minor comment on utility classes: there's JAXPTestUtilities and TestUtils. The former is an util for all tests while the later contained a couple of SAX handlers, it may make sense to move them into their own classes just as the other Handlers. The constants (XML_DIR and GOLDEN_DIR) then can be declared in a base class for the SAX tests. >>>>> >>>>> I understand each group of tests have their own XML source and golden files, thus XML_DIR and GOLDEN_DIR. It may be possible to add a base for each group while put test.src and test.classes into the very base class for all tests. So in general, we would have: >>>>> TestBase FileTestBase >>>>> Base for each group (e.g. SAXTestBase, DOMTestBase, XSLTTestBase...) that extends either TestBase or FileTestBase >>>>> >>>>> I remember there were a few tests that required a http server. So then we may need a HttpTestBase as well. >>>>> >>>>> I know this is not what we've discussed (or planned) previously. But since you've done a great job to incorporate what were previously quite separate test suites into one whole test suite. I can't resist to ask a bit more. Don't get me wrong, what you've done exceeded my expectation in a big way (only a month ago, we were still talking about quick/straight conversion so that you can start to take care of the new features)! >>>>> >>>>> SAXParserTest: I noticed Old: testParse11, testParse27 --> New: testParse05. So is testParse03 a new test? I can see SAXException is expected, but not IOE. In fact, this shows that the JAXP spec missed defining how empty string shall be handled (should have been an IAE). >>>>> >>>>> Best, >>>>> Joe >>>>> >>>>> On 1/6/2015 11:33 AM, Lance Andersen wrote: >>>>>> Hi Tristan, >>>>>> >>>>>> Sorry for the delay but with the holidays and the number of files, it took a while to go through :-) >>>>>> >>>>>> Overall, it looks pretty good. >>>>>> >>>>>> A couple of suggestions, but I would not necessarily hold back from committing: >>>>>> >>>>>> - For tests where you are not caring about the actual exception that is thrown to indicate a failure, such as in DocumentBuilderFactory01.java and testCheckSchemaSupport1, I would just have the method declaration use "throws Exception" vs list all of the possible individual Exceptions, as it keeps it more compact. Glad to see you are not using failUnexcepted() now. >>>>>> >>>>>> - In test classes such as in testCheckSchemaSupport3. java and DocumentBuilderImpl01.java, some tests do not use assertXXX or expect an Exception. Would be good at least to document what could cause a failure or make it clear the expected behavior of the test for passing. >>>>>> >>>>>> -SAXParserTest02.java and other tests where you get a reader/parser such as testXMLReader01, I would at least assert that null is not returned where as it is now, you only validate that an exception is not returned >>>>>> >>>>>> - I know you are porting existing tests, but I would consider consolidating tests as it seems like there is not a real reason to have a testNG class with just 1 test. I would group the like tests (such as SAXTFactoryTest ) in one testNG test class as that allows you to streamline further. >>>>>> >>>>>> - TfClearParamTest.java (as and example) minor nit that you have your @Before/AfterGroups method in between tests. I would suggest grouping all methods such as this DataProviders before or after the actual tests for better organization >>>>>> >>>>>> - TraxSAXWrapper.java, not sure I understand the "Sorry I could not resist comment" >>>>>> >>>>>> - TransformerHandlerAPITest.java has typos in comments: "IllegalArgumentExceptionis thrown?." >>>>>> >>>>>> - Minitest.java, I would add a comment for your Data Provider >>>>>> >>>>>> Best, >>>>>> Lance >>>>>> >>>>>> On Jan 2, 2015, at 1:49 PM, Tristan Yan > wrote: >>>>>> >>>>>>> Hi Joe and Lance >>>>>>> Sorry for my late reply. I just uploaded the new webrev which is trying to limit minimal permissions for most of tests. The new changeset has two base classes; class JAXPBaseTest has only minimal set permissions; class JAXPFileBaseTest adds two permissions that need access local files in the directory directory test.src and test.classes. Most of tests only inherit JAXPBaseTest that provides only minimal set permissions. Some tests will inherit JAXPFileBaseTest because tests need access local files. >>>>>>> Please help to review the code. >>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>>>> >>>>>>> Thank you >>>>>>> Tristan >>>>>>> >>>>>>> >>>>>>>> On Jan 2, 2015, at 10:39 AM, huizhe wang > wrote: >>>>>>>> >>>>>>>> Lance, >>>>>>>> >>>>>>>> Tristan is looking into adding an extension base class for about 60 tests that require file permission, then the current base class would indeed set "minimal" permission. So please wait for his update :-) >>>>>>>> >>>>>>>> Best, >>>>>>>> Joe >>>>>>>> >>>>>>>> On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >>>>>>>>> Hi Tristan, >>>>>>>>> >>>>>>>>> I will look at this but doubt I will get to this tomorrow >>>>>>>>> >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Lance >>>>>>>>> >>>>>>>>> >>>>>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>>>> Oracle Java Engineering >>>>>>>>> 1 Network Drive >>>>>>>>> Burlington, MA 01803 >>>>>>>>> Lance.Andersen at oracle.com >>>>>>>>> Sent from my iPad >>>>>>>>> >>>>>>>>> On Dec 30, 2014, at 5:28 PM, Tristan Yan > wrote: >>>>>>>>> >>>>>>>>>> Hi All >>>>>>>>>> >>>>>>>>>> Can I get your review on this change. >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ > >>>>>>>>>> >>>>>>>>>> These fixes originally come from bug https://bugs.openjdk.java.net/browse/JDK-8051563 > as part of our XML test colocation work. ThIS change-set mainly covers tests for package org.apache.qetest.dtm and org.apache.qetest.trax. >>>>>>>>>> >>>>>>>>>> In the meantime I took steps at fixing some of our existed test code on below issues: >>>>>>>>>> >>>>>>>>>> 1. Add a base test class for all functional tests that enable security manager running. A limited minimal permissions set have been set for every test. >>>>>>>>>> 2. Remove all unnecessary exception capture for functional tests that we?re using testng to handle all the exceptions. >>>>>>>>>> 3. Use try-resource block to solve all possible resource leaks (including InputStream, OutputStream, Writer, Reader). >>>>>>>>>> >>>>>>>>>> Thanks a lot. >>>>>>>>>> Tristan >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>> Oracle Java Engineering >>>>>> 1 Network Drive >>>>>> Burlington, MA 01803 >>>>>> Lance.Andersen at oracle.com >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> >>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>> Oracle Java Engineering >>> 1 Network Drive >>> Burlington, MA 01803 >>> Lance.Andersen at oracle.com >>> >>> >>> >> >> >> >> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> >> >> > From joe.darcy at oracle.com Tue Jan 6 23:12:42 2015 From: joe.darcy at oracle.com (joe darcy) Date: Tue, 06 Jan 2015 15:12:42 -0800 Subject: JDK 9 RFR of 6481080 : (ann) @Deprecated annotation has no effect on packages Message-ID: <54AC6BEA.4080504@oracle.com> Hello, As part of cleaning up the deprecation facility in 9, please review this change to document when a @Deprecated annotation is a no-op: 6481080 : (ann) @Deprecated annotation has no effect on packages http://cr.openjdk.java.net/~darcy/6481080.0/ Since the @Deprecated feature was added in 5, it was a no-op on local variables, parameters, and packages. The JLS documented the no-op situation for local variables and parameters; separately, Alex will handle updating the JLS in 9 to state packages are a no-op too. Patch to java.lang.Deprecated below; the patch also fixes a typo in the JLS section number. Thanks, -Joe --- old/src/java.base/share/classes/java/lang/Deprecated.java 2015-01-06 15:06:40.893641309 -0800 +++ new/src/java.base/share/classes/java/lang/Deprecated.java 2015-01-06 15:06:40.737641309 -0800 @@ -1,5 +1,5 @@ /* - * Copyright (c) 2003, 2013, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2003, 2015, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -34,9 +34,13 @@ * or because a better alternative exists. Compilers warn when a * deprecated program element is used or overridden in non-deprecated code. * + *

Use of the @Deprecated annotation on a local variable + * declaration or on a parameter declaration or a package declaration + * has no effect. + * * @author Neal Gafter * @since 1.5 - * @jls 9.6.3.6 @Deprecated + * @jls 9.6.4.6 @Deprecated */ @Documented @Retention(RetentionPolicy.RUNTIME) From lance.andersen at oracle.com Tue Jan 6 23:47:57 2015 From: lance.andersen at oracle.com (Lance @ Oracle) Date: Tue, 6 Jan 2015 18:47:57 -0500 Subject: JDK 9 RFR of 6481080 : (ann) @Deprecated annotation has no effect on packages In-Reply-To: <54AC6BEA.4080504@oracle.com> References: <54AC6BEA.4080504@oracle.com> Message-ID: Hi joe, The update is clear Best Lance Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com Sent from my iPad > On Jan 6, 2015, at 6:12 PM, joe darcy wrote: > > Hello, > > As part of cleaning up the deprecation facility in 9, please review this change to document when a @Deprecated annotation is a no-op: > > 6481080 : (ann) @Deprecated annotation has no effect on packages > http://cr.openjdk.java.net/~darcy/6481080.0/ > > Since the @Deprecated feature was added in 5, it was a no-op on local variables, parameters, and packages. The JLS documented the no-op situation for local variables and parameters; separately, Alex will handle updating the JLS in 9 to state packages are a no-op too. > > Patch to java.lang.Deprecated below; the patch also fixes a typo in the JLS section number. > > Thanks, > > -Joe > > --- old/src/java.base/share/classes/java/lang/Deprecated.java 2015-01-06 15:06:40.893641309 -0800 > +++ new/src/java.base/share/classes/java/lang/Deprecated.java 2015-01-06 15:06:40.737641309 -0800 > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2003, 2013, Oracle and/or its affiliates. All rights reserved. > + * Copyright (c) 2003, 2015, Oracle and/or its affiliates. All rights reserved. > * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. > * > * This code is free software; you can redistribute it and/or modify it > @@ -34,9 +34,13 @@ > * or because a better alternative exists. Compilers warn when a > * deprecated program element is used or overridden in non-deprecated code. > * > + *

Use of the @Deprecated annotation on a local variable > + * declaration or on a parameter declaration or a package declaration > + * has no effect. > + * > * @author Neal Gafter > * @since 1.5 > - * @jls 9.6.3.6 @Deprecated > + * @jls 9.6.4.6 @Deprecated > */ > @Documented > @Retention(RetentionPolicy.RUNTIME) > From joe.darcy at oracle.com Wed Jan 7 02:44:12 2015 From: joe.darcy at oracle.com (joe darcy) Date: Tue, 06 Jan 2015 18:44:12 -0800 Subject: JDK 9 RFR of 8059175: Zero BigDecimal with negative scale prints leading zeroes in String.format In-Reply-To: <684C24A3-8B54-421F-B38B-41CB2A63FCC5@oracle.com> References: <684C24A3-8B54-421F-B38B-41CB2A63FCC5@oracle.com> Message-ID: <54AC9D7C.9010806@oracle.com> Catching up on reviews, the change looks fine. Cheers, -Joe On 12/23/2014 3:27 PM, Brian Burkhalter wrote: > Please review at your convenience. > > Issue: https://bugs.openjdk.java.net/browse/JDK-8059175 > Patch: http://cr.openjdk.java.net/~bpb/8059175/webrev.00/ > > In java.util.Formatter.BigDecimalLayout.layout() trailing zeroes are erroneously appended when the unscaled value of the BigDecimal is zero and the scale is negative. Some other minor changes are included to make the logic in the method clearer. > > Thanks, > > Brian From andreas.lundblad at oracle.com Wed Jan 7 08:08:38 2015 From: andreas.lundblad at oracle.com (Andreas Lundblad) Date: Wed, 7 Jan 2015 09:08:38 +0100 Subject: JDK 9 RFR of 8067669: Documentation for methods in Number incomplete regarding too large values. In-Reply-To: <54AB3DBE.5040806@oracle.com> References: <20141216125508.GC4911@e6430> <9D55A146-D76D-4E91-B02D-2BC646451CE2@oracle.com> <20141217102612.GE4911@e6430> <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com> <5494B41B.8060003@oracle.com> <0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com> <3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com> <54AB3DBE.5040806@oracle.com> Message-ID: <20150107080837.GB2073@e6430> On Mon, Jan 05, 2015 at 05:43:26PM -0800, Joseph D. Darcy wrote: > Hello, > > Getting back to this issue in the new year, taking a closer look at > the existing class-level documentation, I don't really see a > compelling case for an edit this large. > > I for one would be content if the bug were closed as "not an issue." > > HTH, > > -Joe I originally brought this up when I was trying to create a Comparator. In my use case I was fine with rounding and truncation, so I thougth I could safely use longValue. I was then surprised when I discovered that the method could basically return arbitrary long values. In retrospect I realize that one could interpret "[...] which may involve rounding or truncation" as "may involve rounding or truncation or something else". Nevertheless I find the current formulation a bit confusing. Maybe it's just me. -- Andreas From Alan.Bateman at oracle.com Wed Jan 7 08:31:23 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 07 Jan 2015 08:31:23 +0000 Subject: JDK 9 RFR of 6481080 : (ann) @Deprecated annotation has no effect on packages In-Reply-To: <54AC6BEA.4080504@oracle.com> References: <54AC6BEA.4080504@oracle.com> Message-ID: <54ACEEDB.1040301@oracle.com> On 06/01/2015 23:12, joe darcy wrote: > Hello, > > As part of cleaning up the deprecation facility in 9, please review > this change to document when a @Deprecated annotation is a no-op: > > 6481080 : (ann) @Deprecated annotation has no effect on packages > http://cr.openjdk.java.net/~darcy/6481080.0/ > > Since the @Deprecated feature was added in 5, it was a no-op on local > variables, parameters, and packages. The JLS documented the no-op > situation for local variables and parameters; separately, Alex will > handle updating the JLS in 9 to state packages are a no-op too. > > Patch to java.lang.Deprecated below; the patch also fixes a typo in > the JLS section number. > This looks okay to me too. -Alan From paul.sandoz at oracle.com Wed Jan 7 09:23:10 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 7 Jan 2015 10:23:10 +0100 Subject: JDK 9 RFR of 8059175: Zero BigDecimal with negative scale prints leading zeroes in String.format In-Reply-To: <684C24A3-8B54-421F-B38B-41CB2A63FCC5@oracle.com> References: <684C24A3-8B54-421F-B38B-41CB2A63FCC5@oracle.com> Message-ID: On Dec 24, 2014, at 12:27 AM, Brian Burkhalter wrote: > Please review at your convenience. > > Issue: https://bugs.openjdk.java.net/browse/JDK-8059175 > Patch: http://cr.openjdk.java.net/~bpb/8059175/webrev.00/ > > In java.util.Formatter.BigDecimalLayout.layout() trailing zeroes are erroneously appended when the unscaled value of the BigDecimal is zero and the scale is negative. Some other minor changes are included to make the logic in the method clearer. > Look good. I minor comment, ignore if you wish. If you wanna clean it up further you can merge into one if/else if/else sequence. Paul. From paul.sandoz at oracle.com Wed Jan 7 09:44:26 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 7 Jan 2015 10:44:26 +0100 Subject: JDK 9 RFR of 6481080 : (ann) @Deprecated annotation has no effect on packages In-Reply-To: <54AC6BEA.4080504@oracle.com> References: <54AC6BEA.4080504@oracle.com> Message-ID: <2C52A1B4-65FC-4BD8-95ED-99CEF5045573@oracle.com> On Jan 7, 2015, at 12:12 AM, joe darcy wrote: > Hello, > > As part of cleaning up the deprecation facility in 9, please review this change to document when a @Deprecated annotation is a no-op: > > 6481080 : (ann) @Deprecated annotation has no effect on packages > http://cr.openjdk.java.net/~darcy/6481080.0/ > +1 I am curious to know why @Deprecated was originally allowed to be declared on local variables and packages if it has no effect. I can imagine for the latter there might be some computational issue with javac and I suppose there is still some source code documentation value and other tools can still crunch on it. Paul. From forax at univ-mlv.fr Wed Jan 7 09:45:46 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 07 Jan 2015 10:45:46 +0100 Subject: Bug in ArrayList iterator Message-ID: <54AD004A.4090603@univ-mlv.fr> A simple Java question, what this code does ? ArrayList list = new ArrayList<>(); list.add("foo"); list.add("bar"); for(String s: list) { list.remove(s); } :( R?mi tip: the bug lies in ArrayList.Itr.hasNext() (and AbstractList.Itr.hasNext()). From sbaiduzh at redhat.com Wed Jan 7 09:49:45 2015 From: sbaiduzh at redhat.com (Stanislav Baiduzhyi) Date: Wed, 07 Jan 2015 10:49:45 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <54AD004A.4090603@univ-mlv.fr> References: <54AD004A.4090603@univ-mlv.fr> Message-ID: <2076919.JDFISCuC9J@thinkpad.hell> On Wednesday 07 January 2015 10:45:46 Remi Forax wrote: > A simple Java question, what this code does ? > > ArrayList list = new ArrayList<>(); > list.add("foo"); > list.add("bar"); > for(String s: list) { > list.remove(s); > } > > :( > > R?mi > tip: the bug lies in ArrayList.Itr.hasNext() (and > AbstractList.Itr.hasNext()). You would not invoke list.remove(E) when iterating by index, why do you want to do that iterating through for-each? -- Regards, Stas From daniel.fuchs at oracle.com Wed Jan 7 10:16:15 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Wed, 07 Jan 2015 11:16:15 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <54AD004A.4090603@univ-mlv.fr> References: <54AD004A.4090603@univ-mlv.fr> Message-ID: <54AD076F.8080901@oracle.com> Interesting... I would have expected it to throw java.util.ConcurrentModificationException right away, but it only does so if the list contains exactly 1 or more than 2 elements... best regards, -- daniel On 07/01/15 10:45, Remi Forax wrote: > A simple Java question, what this code does ? > > ArrayList list = new ArrayList<>(); > list.add("foo"); > list.add("bar"); > for(String s: list) { > list.remove(s); > } > > :( > > R?mi > tip: the bug lies in ArrayList.Itr.hasNext() (and > AbstractList.Itr.hasNext()). > From paul.sandoz at oracle.com Wed Jan 7 10:31:23 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 7 Jan 2015 11:31:23 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <54AD004A.4090603@univ-mlv.fr> References: <54AD004A.4090603@univ-mlv.fr> Message-ID: On Jan 7, 2015, at 10:45 AM, Remi Forax wrote: > A simple Java question, what this code does ? > > ArrayList list = new ArrayList<>(); > list.add("foo"); > list.add("bar"); > for(String s: list) { > list.remove(s); > } > > :( > We could improve the best-effort basis by which ConcurrentModificationException is thrown by snapshotting the collection size on construction of the iterator (at the expense of an extra field, but that might pack into a spare 4 bytes due to alignment at least on 64 bits IIUC). Paul. > R?mi > tip: the bug lies in ArrayList.Itr.hasNext() (and AbstractList.Itr.hasNext()). > From daniel.fuchs at oracle.com Wed Jan 7 10:47:56 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Wed, 07 Jan 2015 11:47:56 +0100 Subject: Bug in ArrayList iterator In-Reply-To: References: <54AD004A.4090603@univ-mlv.fr> Message-ID: <54AD0EDC.50203@oracle.com> On 07/01/15 11:31, Paul Sandoz wrote: > > On Jan 7, 2015, at 10:45 AM, Remi Forax wrote: > >> A simple Java question, what this code does ? >> >> ArrayList list = new ArrayList<>(); >> list.add("foo"); >> list.add("bar"); >> for(String s: list) { >> list.remove(s); >> } >> >> :( >> > > We could improve the best-effort basis by which ConcurrentModificationException is thrown by snapshotting the collection size on construction of the iterator (at the expense of an extra field, but that might pack into a spare 4 bytes due to alignment at least on 64 bits IIUC). Or call checkForComodification() when cursor == size in hasNext()? -- daniel > > Paul. > >> R?mi >> tip: the bug lies in ArrayList.Itr.hasNext() (and AbstractList.Itr.hasNext()). >> > From chris.hegarty at oracle.com Wed Jan 7 10:56:01 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 07 Jan 2015 10:56:01 +0000 Subject: Bug in ArrayList iterator In-Reply-To: <54AD0EDC.50203@oracle.com> References: <54AD004A.4090603@univ-mlv.fr> <54AD0EDC.50203@oracle.com> Message-ID: <54AD10C1.8000806@oracle.com> On 07/01/15 10:47, Daniel Fuchs wrote: > On 07/01/15 11:31, Paul Sandoz wrote: >> >> On Jan 7, 2015, at 10:45 AM, Remi Forax wrote: >> >>> A simple Java question, what this code does ? >>> >>> ArrayList list = new ArrayList<>(); >>> list.add("foo"); >>> list.add("bar"); >>> for(String s: list) { >>> list.remove(s); >>> } >>> >>> :( >>> >> >> We could improve the best-effort basis by which >> ConcurrentModificationException is thrown by snapshotting the >> collection size on construction of the iterator This was my thought too. diff --git a/src/java.base/share/classes/java/util/ArrayList.java b/src/java.base/share/classes/java/util/ArrayList.java --- a/src/java.base/share/classes/java/util/ArrayList.java +++ b/src/java.base/share/classes/java/util/ArrayList.java @@ -853,9 +853,10 @@ int cursor; // index of next element to return int lastRet = -1; // index of last element returned; -1 if no such int expectedModCount = modCount; + int itrSize = size; public boolean hasNext() { - return cursor != size; + return cursor != itrSize; } @SuppressWarnings("unchecked") @@ -881,6 +882,7 @@ cursor = lastRet; lastRet = -1; expectedModCount = modCount; + itrSize--; } catch (IndexOutOfBoundsException ex) { throw new ConcurrentModificationException(); } @@ -909,7 +911,7 @@ } final void checkForComodification() { - if (modCount != expectedModCount) + if (modCount != expectedModCount || size != itrSize) throw new ConcurrentModificationException(); } } >> (at the expense of an >> extra field, but that might pack into a spare 4 bytes due to alignment >> at least on 64 bits IIUC). > > Or call checkForComodification() when cursor == size in hasNext()? I'm not sure that we'd want hasNext to possibly throw a CME. I think it best that next do that, if needed. -Chris. > -- daniel > >> >> Paul. >> >>> R?mi >>> tip: the bug lies in ArrayList.Itr.hasNext() (and >>> AbstractList.Itr.hasNext()). >>> >> > From sbaiduzh at redhat.com Wed Jan 7 10:57:13 2015 From: sbaiduzh at redhat.com (Stanislav Baiduzhyi) Date: Wed, 07 Jan 2015 11:57:13 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <54AD10C1.8000806@oracle.com> References: <54AD004A.4090603@univ-mlv.fr> <54AD0EDC.50203@oracle.com> <54AD10C1.8000806@oracle.com> Message-ID: <8718993.ggLpZJQRS0@thinkpad.hell> On Wednesday 07 January 2015 10:56:01 Chris Hegarty wrote: > public boolean hasNext() { > - return cursor != size; > + return cursor != itrSize; > } If the user will invoke list.remove(E) to remove current or previous element then iterator will be skipping some elements. Throwing ConcurrentModification in hasNext() looks better in such case. -- Regards, Stas From paul.sandoz at oracle.com Wed Jan 7 11:16:20 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 7 Jan 2015 12:16:20 +0100 Subject: [9] RFR (M): 8067344: Adjust java/lang/invoke/LFCaching/LFGarbageCollectedTest.java for recent changes in java.lang.invoke In-Reply-To: <549962C4.2040301@oracle.com> References: <549884E7.8040204@oracle.com> <549962C4.2040301@oracle.com> Message-ID: Hi 70 TestMethods testCase = getTestMethod(); 71 if (testCase == TestMethods.EXACT_INVOKER || testCase == TestMethods.INVOKER) { 72 // Invokers aren't collected. 73 return; 74 } Can you just filter those test cases out in the main method within EnumSet.complementOf? On Dec 23, 2014, at 1:40 PM, Vladimir Ivanov wrote: > Spotted some more problems: > - need to skip identity operations (identity_* LambdaForms) in the test, since corresponding LambdaForms reside in a permanent cache; > 82 mtype = adapter.type(); 83 if (mtype.parameterCount() == 0) { 84 // Ignore identity_* LambdaForms. 85 return; 86 } Under what conditions does this arise? i guess it might be non-determinisitic based on the randomly generated arity for the test case, so could filter more tests than absolutely required? > - need to keep original test data for diagnostic purposes, since getTestCaseData() produces new instance. > 78 adapter = getTestMethod().getTestCaseMH(data, TestMethods.Kind.ONE); Could replace "getTestMethod()" with "testCase". Paul. > Updated version: > http://cr.openjdk.java.net/~vlivanov/8067344/webrev.01/ > > Best regards, > Vladimir Ivanov > > On 12/22/14 11:53 PM, Vladimir Ivanov wrote: >> http://cr.openjdk.java.net/~vlivanov/8067344/webrev.00/ >> https://bugs.openjdk.java.net/browse/JDK-8067344 >> >> LFGarbageCollectedTest should be adjusted after JDK-8057020. >> >> There are a couple of problems with the test. >> >> (1) Existing logic to test that LambdaForm instance is collected isn't >> stable enough. Consequent System.GCs can hinder reference enqueueing. >> To speed up the test, I added -XX:SoftRefLRUPolicyMSPerMB=0 and limited >> the heap by -Xmx64m. >> >> (2) MethodType-based invoker caches are deliberately left strongly >> reachable. So, they should be skipped in the test. >> >> (3) Added additional diagnostic output to simplify failure analysis >> (test case details, method handle type and LambdaForm, heap dump >> (optional, -DHEAP_DUMP=true)). >> >> Testing: failing test. >> >> Thanks! >> >> Best regards, >> Vladimir Ivanov > _______________________________________________ > mlvm-dev mailing list > mlvm-dev at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev From chris.hegarty at oracle.com Wed Jan 7 11:20:57 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 07 Jan 2015 11:20:57 +0000 Subject: Bug in ArrayList iterator In-Reply-To: <8718993.ggLpZJQRS0@thinkpad.hell> References: <54AD004A.4090603@univ-mlv.fr> <54AD0EDC.50203@oracle.com> <54AD10C1.8000806@oracle.com> <8718993.ggLpZJQRS0@thinkpad.hell> Message-ID: <54AD1699.8010804@oracle.com> On 07/01/15 10:57, Stanislav Baiduzhyi wrote: > On Wednesday 07 January 2015 10:56:01 Chris Hegarty wrote: >> public boolean hasNext() { >> - return cursor != size; >> + return cursor != itrSize; >> } > > If the user will invoke list.remove(E) to remove current or previous element > then iterator will be skipping some elements. If this happens, then next() and/or remove() will throw CME. Throwing ConcurrentModification > in hasNext() looks better in such case. It is not clear to me that users would be expecting CME from hasNext(), whereas it is more next() and remove() is more obvious. -Chris. From pavel.rappo at oracle.com Wed Jan 7 11:20:51 2015 From: pavel.rappo at oracle.com (Pavel Rappo) Date: Wed, 7 Jan 2015 11:20:51 +0000 Subject: Bug in ArrayList iterator In-Reply-To: <54AD1699.8010804@oracle.com> References: <54AD004A.4090603@univ-mlv.fr> <54AD0EDC.50203@oracle.com> <54AD10C1.8000806@oracle.com> <8718993.ggLpZJQRS0@thinkpad.hell> <54AD1699.8010804@oracle.com> Message-ID: <1E826410-8D9B-46EF-AFBE-EF454DA0B5D0@oracle.com> If I'm not mistaken LinkedList and Vector demonstrate exactly the same buggy behaviour. -Pavel > On 7 Jan 2015, at 11:20, Chris Hegarty wrote: > > On 07/01/15 10:57, Stanislav Baiduzhyi wrote: >> On Wednesday 07 January 2015 10:56:01 Chris Hegarty wrote: >>> public boolean hasNext() { >>> - return cursor != size; >>> + return cursor != itrSize; >>> } >> >> If the user will invoke list.remove(E) to remove current or previous element >> then iterator will be skipping some elements. > > If this happens, then next() and/or remove() will throw CME. > > Throwing ConcurrentModification >> in hasNext() looks better in such case. > > It is not clear to me that users would be expecting CME from hasNext(), whereas it is more next() and remove() is more obvious. > > -Chris. > > From sbaiduzh at redhat.com Wed Jan 7 11:22:06 2015 From: sbaiduzh at redhat.com (Stanislav Baiduzhyi) Date: Wed, 07 Jan 2015 12:22:06 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <54AD1699.8010804@oracle.com> References: <54AD004A.4090603@univ-mlv.fr> <8718993.ggLpZJQRS0@thinkpad.hell> <54AD1699.8010804@oracle.com> Message-ID: <3132623.YULM46OnBs@thinkpad.hell> On Wednesday 07 January 2015 11:20:57 you wrote: > On 07/01/15 10:57, Stanislav Baiduzhyi wrote: > > On Wednesday 07 January 2015 10:56:01 Chris Hegarty wrote: > >> public boolean hasNext() { > >> > >> - return cursor != size; > >> + return cursor != itrSize; > >> > >> } > > > > If the user will invoke list.remove(E) to remove current or previous > > element then iterator will be skipping some elements. > > If this happens, then next() and/or remove() will throw CME. > > Throwing ConcurrentModification > > > in hasNext() looks better in such case. > > It is not clear to me that users would be expecting CME from hasNext(), > whereas it is more next() and remove() is more obvious. Yes, but that exactly how it works now, that's why second invocation of hasNext() returns false and we can see the issue Remi is talking about. If hasNext() will be throwing ConcurrentModification than will work as early warning system that something is wrong with this iterator. -- Regards, Stas From dl at cs.oswego.edu Wed Jan 7 11:25:26 2015 From: dl at cs.oswego.edu (Doug Lea) Date: Wed, 07 Jan 2015 06:25:26 -0500 Subject: Bug in ArrayList iterator In-Reply-To: <54AD004A.4090603@univ-mlv.fr> References: <54AD004A.4090603@univ-mlv.fr> Message-ID: <54AD17A6.7050300@cs.oswego.edu> On 01/07/2015 04:45 AM, Remi Forax wrote: > A simple Java question, what this code does ? > > ArrayList list = new ArrayList<>(); > list.add("foo"); > list.add("bar"); > for(String s: list) { > list.remove(s); > } > I have a vague recollection that this issue has come up before, and that nothing was done about it because it broke compatibility with some existing code. Maybe someone with records of past CCC decisions could check this? -Doug From neugens.limasoftware at gmail.com Wed Jan 7 11:29:44 2015 From: neugens.limasoftware at gmail.com (Mario Torre) Date: Wed, 7 Jan 2015 12:29:44 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <3132623.YULM46OnBs@thinkpad.hell> References: <54AD004A.4090603@univ-mlv.fr> <8718993.ggLpZJQRS0@thinkpad.hell> <54AD1699.8010804@oracle.com> <3132623.YULM46OnBs@thinkpad.hell> Message-ID: 2015-01-07 12:22 GMT+01:00 Stanislav Baiduzhyi : > On Wednesday 07 January 2015 11:20:57 you wrote: >> On 07/01/15 10:57, Stanislav Baiduzhyi wrote: >> > On Wednesday 07 January 2015 10:56:01 Chris Hegarty wrote: >> >> public boolean hasNext() { >> >> >> >> - return cursor != size; >> >> + return cursor != itrSize; >> >> >> >> } >> > >> > If the user will invoke list.remove(E) to remove current or previous >> > element then iterator will be skipping some elements. >> >> If this happens, then next() and/or remove() will throw CME. >> >> Throwing ConcurrentModification >> >> > in hasNext() looks better in such case. >> >> It is not clear to me that users would be expecting CME from hasNext(), >> whereas it is more next() and remove() is more obvious. > > Yes, but that exactly how it works now, that's why second invocation of > hasNext() returns false and we can see the issue Remi is talking about. If > hasNext() will be throwing ConcurrentModification than will work as early > warning system that something is wrong with this iterator. But hasNext isn't modifying things, why should it throw ConcurrentModification? I would find it very confusing, and also likely non backward compatible. Cheers, Mario -- pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF Fingerprint: BA39 9666 94EC 8B73 27FA FC7C 4086 63E3 80F2 40CF Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens Proud GNU Classpath developer: http://www.classpath.org/ OpenJDK: http://openjdk.java.net/projects/caciocavallo/ Please, support open standards: http://endsoftpatents.org/ From paul.sandoz at oracle.com Wed Jan 7 11:33:30 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 7 Jan 2015 12:33:30 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <54AD10C1.8000806@oracle.com> References: <54AD004A.4090603@univ-mlv.fr> <54AD0EDC.50203@oracle.com> <54AD10C1.8000806@oracle.com> Message-ID: <675E4599-5E27-4FE8-A19A-2A5EBEACE7D1@oracle.com> On Jan 7, 2015, at 11:56 AM, Chris Hegarty wrote: > On 07/01/15 10:47, Daniel Fuchs wrote: >> On 07/01/15 11:31, Paul Sandoz wrote: >>> >>> On Jan 7, 2015, at 10:45 AM, Remi Forax wrote: >>> >>>> A simple Java question, what this code does ? >>>> >>>> ArrayList list = new ArrayList<>(); >>>> list.add("foo"); >>>> list.add("bar"); >>>> for(String s: list) { >>>> list.remove(s); >>>> } >>>> >>>> :( >>>> >>> >>> We could improve the best-effort basis by which >>> ConcurrentModificationException is thrown by snapshotting the >>> collection size on construction of the iterator > > This was my thought too. > > diff --git a/src/java.base/share/classes/java/util/ArrayList.java b/src/java.base/share/classes/java/util/ArrayList.java > --- a/src/java.base/share/classes/java/util/ArrayList.java > +++ b/src/java.base/share/classes/java/util/ArrayList.java > @@ -853,9 +853,10 @@ > int cursor; // index of next element to return > int lastRet = -1; // index of last element returned; -1 if no such > int expectedModCount = modCount; > + int itrSize = size; > > public boolean hasNext() { > - return cursor != size; > + return cursor != itrSize; > } > > @SuppressWarnings("unchecked") > @@ -881,6 +882,7 @@ > cursor = lastRet; > lastRet = -1; > expectedModCount = modCount; > + itrSize--; > } catch (IndexOutOfBoundsException ex) { > throw new ConcurrentModificationException(); > } > @@ -909,7 +911,7 @@ > } > > final void checkForComodification() { > - if (modCount != expectedModCount) > + if (modCount != expectedModCount || size != itrSize) > throw new ConcurrentModificationException(); > } > } > Under what conditions can the mod count be the same but the size is different? Would this not indicate there was a bug in some other method that made a structural change but did not update the mod count? Same apples to this: Object[] elementData = ArrayList.this.elementData; if (i >= elementData.length) throw new ConcurrentModificationException(); (Note as per usual in these discussions i am not referring to concurrent operation from multiple threads, under such conditions all bets are off.) Unfortunately the change history in openjdk only goes back so far, and i agree with Doug that there might be some reason why this change was not made earlier on (it seems like an "obvious" thing to do). Paul. From sbaiduzh at redhat.com Wed Jan 7 11:38:03 2015 From: sbaiduzh at redhat.com (Stanislav Baiduzhyi) Date: Wed, 07 Jan 2015 12:38:03 +0100 Subject: Bug in ArrayList iterator In-Reply-To: References: <54AD004A.4090603@univ-mlv.fr> <3132623.YULM46OnBs@thinkpad.hell> Message-ID: <2340046.ecTKAK1KLy@thinkpad.hell> On Wednesday 07 January 2015 12:29:44 Mario Torre wrote: > > Yes, but that exactly how it works now, that's why second invocation of > > hasNext() returns false and we can see the issue Remi is talking about. If > > hasNext() will be throwing ConcurrentModification than will work as early > > warning system that something is wrong with this iterator. > > But hasNext isn't modifying things, why should it throw > ConcurrentModification? I would find it very confusing, and also > likely non backward compatible. Yes. More than that, guide page says directly that you should not use for-each for filtering [1]. Which means that what Remi has reported is not an issue, just incorrectly applied for-each iterator, and probably the least effective way to filter out the list. But as discussion went on with this in search of solution - I still cannot see any other way to fix it than throw ConcurrentModification from hasNext(). [1]: http://docs.oracle.com/javase/8/docs/technotes/guides/language/foreach.html -- Regards, Stas From chris.hegarty at oracle.com Wed Jan 7 12:02:42 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 07 Jan 2015 12:02:42 +0000 Subject: Bug in ArrayList iterator In-Reply-To: <3132623.YULM46OnBs@thinkpad.hell> References: <54AD004A.4090603@univ-mlv.fr> <8718993.ggLpZJQRS0@thinkpad.hell> <54AD1699.8010804@oracle.com> <3132623.YULM46OnBs@thinkpad.hell> Message-ID: <54AD2062.9020507@oracle.com> On 07/01/15 11:22, Stanislav Baiduzhyi wrote: > On Wednesday 07 January 2015 11:20:57 you wrote: >> On 07/01/15 10:57, Stanislav Baiduzhyi wrote: >>> On Wednesday 07 January 2015 10:56:01 Chris Hegarty wrote: >>>> public boolean hasNext() { >>>> >>>> - return cursor != size; >>>> + return cursor != itrSize; >>>> >>>> } >>> >>> If the user will invoke list.remove(E) to remove current or previous >>> element then iterator will be skipping some elements. >> >> If this happens, then next() and/or remove() will throw CME. >> >> Throwing ConcurrentModification >> >>> in hasNext() looks better in such case. >> >> It is not clear to me that users would be expecting CME from hasNext(), >> whereas it is more next() and remove() is more obvious. > > Yes, but that exactly how it works now, that's why second invocation of > hasNext() returns false and we can see the issue Remi is talking about. If > hasNext() will be throwing ConcurrentModification than will work as early > warning system that something is wrong with this iterator. The changes in the patch I suggested will change existing behavior. In Remi's case, hasNext() will now return true, and next() will throw CME. -Chris. From sbaiduzh at redhat.com Wed Jan 7 12:03:15 2015 From: sbaiduzh at redhat.com (Stanislav Baiduzhyi) Date: Wed, 07 Jan 2015 13:03:15 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <54AD2062.9020507@oracle.com> References: <54AD004A.4090603@univ-mlv.fr> <3132623.YULM46OnBs@thinkpad.hell> <54AD2062.9020507@oracle.com> Message-ID: <1993163.p6DuzCE4lR@thinkpad.hell> On Wednesday 07 January 2015 12:02:42 Chris Hegarty wrote: > > Yes, but that exactly how it works now, that's why second invocation of > > hasNext() returns false and we can see the issue Remi is talking about. If > > hasNext() will be throwing ConcurrentModification than will work as early > > warning system that something is wrong with this iterator. > > The changes in the patch I suggested will change existing behavior. In > Remi's case, hasNext() will now return true, and next() will throw CME. That's correct, yes, sorry, missed the idea. That's actually good solution, much better than throwing CME from hasNext(). -- Regards, Stas From Alan.Bateman at oracle.com Wed Jan 7 12:13:27 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 07 Jan 2015 12:13:27 +0000 Subject: Bug in ArrayList iterator In-Reply-To: <54AD17A6.7050300@cs.oswego.edu> References: <54AD004A.4090603@univ-mlv.fr> <54AD17A6.7050300@cs.oswego.edu> Message-ID: <54AD22E7.9090501@oracle.com> On 07/01/2015 11:25, Doug Lea wrote: > On 01/07/2015 04:45 AM, Remi Forax wrote: >> A simple Java question, what this code does ? >> >> ArrayList list = new ArrayList<>(); >> list.add("foo"); >> list.add("bar"); >> for(String s: list) { >> list.remove(s); >> } >> > > I have a vague recollection that this issue has come up before, > and that nothing was done about it because it broke compatibility > with some existing code. Maybe someone with records of past CCC > decisions could check this? One attempt that I recall is JDK-4902078, I think it's come up several others times since then too. -Alan From chris.hegarty at oracle.com Wed Jan 7 12:18:15 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 07 Jan 2015 12:18:15 +0000 Subject: Bug in ArrayList iterator In-Reply-To: <54AD17A6.7050300@cs.oswego.edu> References: <54AD004A.4090603@univ-mlv.fr> <54AD17A6.7050300@cs.oswego.edu> Message-ID: <54AD2407.4010705@oracle.com> On 07/01/15 11:25, Doug Lea wrote: > On 01/07/2015 04:45 AM, Remi Forax wrote: >> A simple Java question, what this code does ? >> >> ArrayList list = new ArrayList<>(); >> list.add("foo"); >> list.add("bar"); >> for(String s: list) { >> list.remove(s); >> } >> > > I have a vague recollection that this issue has come up before, > and that nothing was done about it because it broke compatibility > with some existing code. Maybe someone with records of past CCC > decisions could check this? For the record ( thanks Alan! ) 4902078 [1] proposed to fix this previously, but that was by changing hasNext() to throw CME ( which I don't agree with ). -Chris. [1] https://bugs.openjdk.java.net/browse/JDK-4902078 > > -Doug > From forax at univ-mlv.fr Wed Jan 7 13:07:14 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 07 Jan 2015 14:07:14 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <54AD0EDC.50203@oracle.com> References: <54AD004A.4090603@univ-mlv.fr> <54AD0EDC.50203@oracle.com> Message-ID: <54AD2F82.1030006@univ-mlv.fr> On 01/07/2015 11:47 AM, Daniel Fuchs wrote: > On 07/01/15 11:31, Paul Sandoz wrote: >> >> On Jan 7, 2015, at 10:45 AM, Remi Forax wrote: >> >>> A simple Java question, what this code does ? >>> >>> ArrayList list = new ArrayList<>(); >>> list.add("foo"); >>> list.add("bar"); >>> for(String s: list) { >>> list.remove(s); >>> } >>> >>> :( >>> >> >> We could improve the best-effort basis by which >> ConcurrentModificationException is thrown by snapshotting the >> collection size on construction of the iterator (at the expense of an >> extra field, but that might pack into a spare 4 bytes due to >> alignment at least on 64 bits IIUC). yes, note that AbstractList.Itr exhibit the same issue. I fear that fixing this bug may break some programs. > > Or call checkForComodification() when cursor == size in hasNext()? No, you can't. The contract of AbstractList.modCount is pretty clear about that: https://docs.oracle.com/javase/7/docs/api/java/util/AbstractList.html#modCount hasNext() can not call checkForComodification(). > > -- daniel R?mi > >> >> Paul. >> >>> R?mi >>> tip: the bug lies in ArrayList.Itr.hasNext() (and >>> AbstractList.Itr.hasNext()). >>> >> > From chris.hegarty at oracle.com Wed Jan 7 14:54:43 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 07 Jan 2015 14:54:43 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54AC200E.3000602@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> Message-ID: <54AD48B3.4050607@oracle.com> On 06/01/15 17:49, Peter Levart wrote: > On 01/06/2015 06:21 PM, Chris Hegarty wrote: >> On 6 Jan 2015, at 15:06, Peter Levart wrote: >> >>> On 01/06/2015 04:03 PM, Peter Levart wrote: >>>> private void readObject(ObjectInputStream in) throws IOException, >>>> ClassNotFoundException { >>>> ObjectInputStream.GetField fields = in.readFields(); // this >>>> already validates the types >>> Well, not true currently. But type validation could be added at this >>> point. >> Right. I think I?ll file a bug to track this as it seems reasonable to >> add type validation to readFields and defaultReadObject. So we can >> probably assume/ignore it in this discussion. >> >> I like the idea of a callback into the serialization framework to >> handling the setting of final fields, after validation. I played a >> little with your patch and added it to a branch in the sandbox** >> >> So a simple example, without legacy fields, might looks as below ( >> without the need for writeObject or serialPersistentFields ). The >> simple validating readObject is starting to look like boilerplate ? > > Well, 1st and last line are always the same, yes. What's between them is > what you would have to write in a special check-only method too. I guess what I'm getting at is, if you want just invariant checking, then maybe something like this would be more readable: @SerialInvariantChecker() private static void validate(@SerialParam(name="lo", type=Integer.class)int lo, @SerialParam(name="hi", type=Integer.class)int hi) { if (lo > hi) throw new IllegalArgumentException("lo:" + lo + " > hi:" + hi); } ... and the serialization machinery would call this when appropriate. -Chris > Regards, Peter > >> >> public class SimpleInterval implements Serializable { >> >> private final int lo, hi; >> >> private static void validate(int lo, int hi) { >> // invariant >> if (lo > hi) >> throw new IllegalArgumentException("lo:" + lo + " > hi:" >> + hi); >> } >> >> public SimpleInterval(int lo, int hi) { >> validate(lo, hi); >> this.lo = lo; >> this.hi = hi; >> } >> >> public int getLo() { return lo; } >> >> public int getHi() { return hi; } >> >> private void readObject(ObjectInputStream in) throws IOException, >> ClassNotFoundException { >> ObjectInputStream.GetField fields = in.readFields(); >> >> // validate 'lo' and 'hi' fields invariant >> int lo = fields.get("lo", 0); >> int hi = fields.get("hi", 0); >> validate(lo, hi); >> >> // set current fields from read data >> fields.defaultReadFields(); // this is new API! >> } >> } >> >> -Chris. >> >> ** hg clone http://hg.openjdk.java.net/jdk9/sandbox sandbox >> cd sandbox >> sh get_source.sh >> sh common/bin/hgforest.sh update -r serial-exp-branch >> >> I also added your example, etc, under: >> jdk/test/java/io/Serializable/invarientChecker >> >> see http://cr.openjdk.java.net/~chegar/docs/sandbox.html >> > From peter.levart at gmail.com Wed Jan 7 16:02:11 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 07 Jan 2015 17:02:11 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <54AD48B3.4050607@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> Message-ID: <54AD5883.3070705@gmail.com> On 01/07/2015 03:54 PM, Chris Hegarty wrote: > On 06/01/15 17:49, Peter Levart wrote: >> On 01/06/2015 06:21 PM, Chris Hegarty wrote: >>> On 6 Jan 2015, at 15:06, Peter Levart wrote: >>> >>>> On 01/06/2015 04:03 PM, Peter Levart wrote: >>>>> private void readObject(ObjectInputStream in) throws IOException, >>>>> ClassNotFoundException { >>>>> ObjectInputStream.GetField fields = in.readFields(); // this >>>>> already validates the types >>>> Well, not true currently. But type validation could be added at this >>>> point. >>> Right. I think I?ll file a bug to track this as it seems reasonable to >>> add type validation to readFields and defaultReadObject. So we can >>> probably assume/ignore it in this discussion. >>> >>> I like the idea of a callback into the serialization framework to >>> handling the setting of final fields, after validation. I played a >>> little with your patch and added it to a branch in the sandbox** >>> >>> So a simple example, without legacy fields, might looks as below ( >>> without the need for writeObject or serialPersistentFields ). The >>> simple validating readObject is starting to look like boilerplate ? >> >> Well, 1st and last line are always the same, yes. What's between them is >> what you would have to write in a special check-only method too. > > I guess what I'm getting at is, if you want just invariant checking, > then maybe something like this would be more readable: > > @SerialInvariantChecker() > private static void validate(@SerialParam(name="lo", > type=Integer.class)int lo, > @SerialParam(name="hi", > type=Integer.class)int hi) > { > if (lo > hi) > throw new IllegalArgumentException("lo:" + lo + " > hi:" + > hi); > } > > ... and the serialization machinery would call this when appropriate. Nice. We have method parameter names accessible via reflection since JDK8 and types have always been, so no @SerialParam is necessary. Peter > > -Chris > > >> Regards, Peter >> >>> >>> public class SimpleInterval implements Serializable { >>> >>> private final int lo, hi; >>> >>> private static void validate(int lo, int hi) { >>> // invariant >>> if (lo > hi) >>> throw new IllegalArgumentException("lo:" + lo + " > hi:" >>> + hi); >>> } >>> >>> public SimpleInterval(int lo, int hi) { >>> validate(lo, hi); >>> this.lo = lo; >>> this.hi = hi; >>> } >>> >>> public int getLo() { return lo; } >>> >>> public int getHi() { return hi; } >>> >>> private void readObject(ObjectInputStream in) throws IOException, >>> ClassNotFoundException { >>> ObjectInputStream.GetField fields = in.readFields(); >>> >>> // validate 'lo' and 'hi' fields invariant >>> int lo = fields.get("lo", 0); >>> int hi = fields.get("hi", 0); >>> validate(lo, hi); >>> >>> // set current fields from read data >>> fields.defaultReadFields(); // this is new API! >>> } >>> } >>> >>> -Chris. >>> >>> ** hg clone http://hg.openjdk.java.net/jdk9/sandbox sandbox >>> cd sandbox >>> sh get_source.sh >>> sh common/bin/hgforest.sh update -r serial-exp-branch >>> >>> I also added your example, etc, under: >>> jdk/test/java/io/Serializable/invarientChecker >>> >>> see http://cr.openjdk.java.net/~chegar/docs/sandbox.html >>> >> From david.lloyd at redhat.com Wed Jan 7 16:14:58 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 07 Jan 2015 10:14:58 -0600 Subject: Explicit Serialization API and Security In-Reply-To: <54AD5883.3070705@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> Message-ID: <54AD5B82.20306@redhat.com> On 01/07/2015 10:02 AM, Peter Levart wrote: > On 01/07/2015 03:54 PM, Chris Hegarty wrote: >> On 06/01/15 17:49, Peter Levart wrote: >>> On 01/06/2015 06:21 PM, Chris Hegarty wrote: >>>> On 6 Jan 2015, at 15:06, Peter Levart wrote: >>>> >>>>> On 01/06/2015 04:03 PM, Peter Levart wrote: >>>>>> private void readObject(ObjectInputStream in) throws IOException, >>>>>> ClassNotFoundException { >>>>>> ObjectInputStream.GetField fields = in.readFields(); // this >>>>>> already validates the types >>>>> Well, not true currently. But type validation could be added at this >>>>> point. >>>> Right. I think I?ll file a bug to track this as it seems reasonable to >>>> add type validation to readFields and defaultReadObject. So we can >>>> probably assume/ignore it in this discussion. >>>> >>>> I like the idea of a callback into the serialization framework to >>>> handling the setting of final fields, after validation. I played a >>>> little with your patch and added it to a branch in the sandbox** >>>> >>>> So a simple example, without legacy fields, might looks as below ( >>>> without the need for writeObject or serialPersistentFields ). The >>>> simple validating readObject is starting to look like boilerplate ? >>> >>> Well, 1st and last line are always the same, yes. What's between them is >>> what you would have to write in a special check-only method too. >> >> I guess what I'm getting at is, if you want just invariant checking, >> then maybe something like this would be more readable: >> >> @SerialInvariantChecker() >> private static void validate(@SerialParam(name="lo", >> type=Integer.class)int lo, >> @SerialParam(name="hi", >> type=Integer.class)int hi) >> { >> if (lo > hi) >> throw new IllegalArgumentException("lo:" + lo + " > hi:" + >> hi); >> } >> >> ... and the serialization machinery would call this when appropriate. > > Nice. > > We have method parameter names accessible via reflection since JDK8 and > types have always been, so no @SerialParam is necessary. I tend to disagree, a bit. Most superficially, there's no reason to bind serialization stream parameter names to method parameter names. But more importantly, this implies a pretty complex implementation when you consider we already have an API to represent the fields of a class in a stream (GetFields) as well as a well-established convention of using specifically-named private members for this kind of thing (in the form of readObject/writeObject, objectStreamFields, etc.). You could just as well support this approach, and probably with a substantially simpler implementation: private static void validate(GetField fields) { if (fields.getInt("lo") > fields.getInt("hi")) { ... } } But given these principles, I think this idea proposal was superior and is a sort of obvious "why wasn't it always this way" kind of thing: >>>On 01/06/2015 06:21 PM, Chris Hegarty wrote: >>>>[...] >>>> private void readObject(ObjectInputStream in) throws IOException, >>>> ClassNotFoundException { >>>> ObjectInputStream.GetField fields = in.readFields(); >>>> >>>> // validate 'lo' and 'hi' fields invariant >>>> int lo = fields.get("lo", 0); >>>> int hi = fields.get("hi", 0); >>>> validate(lo, hi); >>>> >>>> // set current fields from read data >>>> fields.defaultReadFields(); // this is new API! >>>> } -- - DML From brian.burkhalter at oracle.com Wed Jan 7 17:39:46 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 7 Jan 2015 09:39:46 -0800 Subject: JDK 9 RFR of 8059175: Zero BigDecimal with negative scale prints leading zeroes in String.format In-Reply-To: References: <684C24A3-8B54-421F-B38B-41CB2A63FCC5@oracle.com> Message-ID: <91872432-14DD-411B-A404-05B9CEFB2E85@oracle.com> On Jan 7, 2015, at 1:23 AM, Paul Sandoz wrote: >> In java.util.Formatter.BigDecimalLayout.layout() trailing zeroes are erroneously appended when the unscaled value of the BigDecimal is zero and the scale is negative. Some other minor changes are included to make the logic in the method clearer. >> > > Look good. > > I minor comment, ignore if you wish. If you wanna clean it up further you can merge into one if/else if/else sequence. Thanks for the suggestion; I like that better also. An updated version is here: http://cr.openjdk.java.net/~bpb/8059175/webrev.01/ For the sake of formality, a +1 on this minor tweak from a Reviewer would be appreciated. Thanks, Brian From joe.darcy at oracle.com Wed Jan 7 21:16:44 2015 From: joe.darcy at oracle.com (joe darcy) Date: Wed, 07 Jan 2015 13:16:44 -0800 Subject: JDK 9 RFR of 6481080 : (ann) @Deprecated annotation has no effect on packages In-Reply-To: <2C52A1B4-65FC-4BD8-95ED-99CEF5045573@oracle.com> References: <54AC6BEA.4080504@oracle.com> <2C52A1B4-65FC-4BD8-95ED-99CEF5045573@oracle.com> Message-ID: <54ADA23C.8080307@oracle.com> On 1/7/2015 1:44 AM, Paul Sandoz wrote: > On Jan 7, 2015, at 12:12 AM, joe darcy wrote: > >> Hello, >> >> As part of cleaning up the deprecation facility in 9, please review this change to document when a @Deprecated annotation is a no-op: >> >> 6481080 : (ann) @Deprecated annotation has no effect on packages >> http://cr.openjdk.java.net/~darcy/6481080.0/ >> > +1 > > I am curious to know why @Deprecated was originally allowed to be declared on local variables and packages if it has no effect. I can imagine for the latter there might be some computational issue with javac and I suppose there is still some source code documentation value and other tools can still crunch on it. Back in JDK 5 when annotations were added, the Deprecated annotation type was initially unconstrained in what it could be applied to (no @Target meta-annotation) [1]. I believe this was just a small oversight in part of the libraries work in 5. A target list was added in 7 as an outgrowth of the JDK 308 work (to make sure Deprecated wasn't a TYPE_USE). [2] For compatibility, the target list included all the JDK 5-era values. [3] HTH, -Joe [1] http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/Deprecated.html [2] https://bugs.openjdk.java.net/browse/JDK-6908218 [3] http://docs.oracle.com/javase/7/docs/api/java/lang/Deprecated.html From brian.burkhalter at oracle.com Wed Jan 7 22:04:42 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 7 Jan 2015 14:04:42 -0800 Subject: JDK 9 RFR of 8067669: Documentation for methods in Number incomplete regarding too large values. In-Reply-To: <20150107080837.GB2073@e6430> References: <20141216125508.GC4911@e6430> <9D55A146-D76D-4E91-B02D-2BC646451CE2@oracle.com> <20141217102612.GE4911@e6430> <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com> <5494B41B.8060003@oracle.com> <0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com> <3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com> <54AB3DBE.5040806@oracle.com> <20150107080837.GB2073@e6430> Message-ID: On Jan 7, 2015, at 12:08 AM, Andreas Lundblad wrote: > On Mon, Jan 05, 2015 at 05:43:26PM -0800, Joseph D. Darcy wrote: >> Hello, >> >> Getting back to this issue in the new year, taking a closer look at >> the existing class-level documentation, I don't really see a >> compelling case for an edit this large. >> >> I for one would be content if the bug were closed as "not an issue." >> >> HTH, >> >> -Joe > > > I originally brought this up when I was trying to create a Comparator. In my use case I was fine with rounding and truncation, so I thougth I could safely use longValue. I was then surprised when I discovered that the method could basically return arbitrary long values. With this in mind, perhaps simply an additional sentence in the class-level documentation would be in order, so to speak (insert as penultimate sentence): ?In particular, conversions may not preserve the mathematical ordering of the numeric values represented by Number instances." Brian From joe.darcy at oracle.com Wed Jan 7 22:10:09 2015 From: joe.darcy at oracle.com (Joseph D. Darcy) Date: Wed, 07 Jan 2015 14:10:09 -0800 Subject: JDK 9 RFR of 8059175: Zero BigDecimal with negative scale prints leading zeroes in String.format In-Reply-To: <91872432-14DD-411B-A404-05B9CEFB2E85@oracle.com> References: <684C24A3-8B54-421F-B38B-41CB2A63FCC5@oracle.com> <91872432-14DD-411B-A404-05B9CEFB2E85@oracle.com> Message-ID: <54ADAEC1.8040109@oracle.com> On 1/7/2015 9:39 AM, Brian Burkhalter wrote: > On Jan 7, 2015, at 1:23 AM, Paul Sandoz wrote: > >>> In java.util.Formatter.BigDecimalLayout.layout() trailing zeroes are erroneously appended when the unscaled value of the BigDecimal is zero and the scale is negative. Some other minor changes are included to make the logic in the method clearer. >>> >> Look good. >> >> I minor comment, ignore if you wish. If you wanna clean it up further you can merge into one if/else if/else sequence. > Thanks for the suggestion; I like that better also. An updated version is here: > > http://cr.openjdk.java.net/~bpb/8059175/webrev.01/ > > For the sake of formality, a +1 on this minor tweak from a Reviewer would be appreciated. > > +1 -Joe From joe.darcy at oracle.com Thu Jan 8 00:03:26 2015 From: joe.darcy at oracle.com (Joseph D. Darcy) Date: Wed, 07 Jan 2015 16:03:26 -0800 Subject: JDK 9 RFR of 8067669: Documentation for methods in Number incomplete regarding too large values. In-Reply-To: References: <20141216125508.GC4911@e6430> <9D55A146-D76D-4E91-B02D-2BC646451CE2@oracle.com> <20141217102612.GE4911@e6430> <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com> <5494B41B.8060003@oracle.com> <0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com> <3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com> <54AB3DBE.5040806@oracle.com> <20150107080837.GB2073@e6430> Message-ID: <54ADC94E.4090607@oracle.com> On 1/7/2015 2:04 PM, Brian Burkhalter wrote: > > On Jan 7, 2015, at 12:08 AM, Andreas Lundblad > > wrote: > >> On Mon, Jan 05, 2015 at 05:43:26PM -0800, Joseph D. Darcy wrote: >>> Hello, >>> >>> Getting back to this issue in the new year, taking a closer look at >>> the existing class-level documentation, I don't really see a >>> compelling case for an edit this large. >>> >>> I for one would be content if the bug were closed as "not an issue." >>> >>> HTH, >>> >>> -Joe >> >> >> I originally brought this up when I was trying to create a >> Comparator. In my use case I was fine with rounding and >> truncation, so I thougth I could safely use longValue. I was then >> surprised when I discovered that the method could basically return >> arbitrary long values. > > With this in mind, perhaps simply an additional sentence in the > class-level documentation would be in order, so to speak (insert as > penultimate sentence): > > ?In particular, conversions may not preserve the mathematical ordering > of the numeric values represented by Number instances." > > Brian On the matter of writing a Comparator, the basic problem is the Number interface is not strong enough to allow you to write such functionality. First, the Number type basically only means "convertible to a primitive." There are no strict requirements on consistency between the different conversion methods and there is not even a requirement that calling the same Number method on the same object returns the same value (since Numbers are not required to be immutable). Even just writing a semantically correct Comparator for Number subclasses in the JDK is nontrivial. Multiple Number subclasses (BigInteger and BigDecimal) have values outside of the set of values that can be represented with a primitive type. So a general Comparator is not really well-defined. Cheers, -Joe From david.holmes at oracle.com Thu Jan 8 02:24:54 2015 From: david.holmes at oracle.com (David Holmes) Date: Thu, 08 Jan 2015 12:24:54 +1000 Subject: Bug in ArrayList iterator In-Reply-To: <54AD004A.4090603@univ-mlv.fr> References: <54AD004A.4090603@univ-mlv.fr> Message-ID: <54ADEA76.3010701@oracle.com> On 7/01/2015 7:45 PM, Remi Forax wrote: > A simple Java question, what this code does ? > > ArrayList list = new ArrayList<>(); > list.add("foo"); > list.add("bar"); > for(String s: list) { > list.remove(s); > } > > :( > > R?mi > tip: the bug lies in ArrayList.Itr.hasNext() (and > AbstractList.Itr.hasNext()). This is not a bug. The only supported way to remove from a collection you are iterating over is to use an Iterator's remove method. http://docs.oracle.com/javase/8/docs/api/java/util/Iterator.html#remove-- "The behavior of an iterator is unspecified if the underlying collection is modified while the iteration is in progress in any way other than by calling this method." and as already pointed out the forEach docs make it clear you can't use forEach in such a context as the Iterator is not exposed to you. So don't do that. David From tristan.yan at oracle.com Thu Jan 8 05:41:11 2015 From: tristan.yan at oracle.com (Tristan Yan) Date: Wed, 7 Jan 2015 21:41:11 -0800 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: <54AC6161.1000308@oracle.com> References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> <54AC51FD.7040808@oracle.com> <078315A4-D463-4891-99CE-6A1785BB5D49@oracle.com> <4B99A07A-A547-4CB5-8E6E-98EA57762906@oracle.com> <54AC6161.1000308@oracle.com> Message-ID: <1716E308-BE26-4EF6-9C48-FF28D5A640B0@oracle.com> Hi Joe/Lance and others. Please review updated version for this one. http://cr.openjdk.java.net/~tyan/8051563/webrev.01/ Thank you Tristan > On Jan 6, 2015, at 2:27 PM, huizhe wang > wrote: > > > On 1/6/2015 2:25 PM, Lance Andersen wrote: >> One more thing :-) >> >> Remember to update your copyright year to 2015 also > > Indeed, that applies to my webrevs as well :-) > > Joe > >> >> Best >> Lance >> On Jan 6, 2015, at 5:04 PM, Lance Andersen > wrote: >> >>> >>> On Jan 6, 2015, at 4:31 PM, Tristan Yan > wrote: >>> >>>> Thank you Lance and Joe. >>> >>> You are very welcome. >>>> I am working on the fix. >>> >>> No rush from my perspective, have a lot to keep me busy in the interim before your next webrev . :-) >>>> it may take a couple of days that the code needs some refactor. >>> Understand, as I have been working on tests for RowSets, I have continued to play the refactor dance myself. >>> >>> Best, >>> Lance >>>> I will send out next review once I finish it. >>>> Thanks >>>> Tristan >>>> >>>>> On Jan 6, 2015, at 1:22 PM, huizhe wang > wrote: >>>>> >>>>> Thanks for taking the initiative and effort to refactor and clean up all of the Functional tests in previous changeset! >>>>> >>>>> We've gone through several iterations for the new ones (xslt tests). I totally agree with Lance, it looks very good overall, a lot better than its original form. >>>>> >>>>> It's nice to group all of the tests that required file access, that prepares all other tests to run with minimal permission for a (future) secure-test-target. Unit tests might need a similar treatment, but no pressure to do now :-) >>>>> >>>>> A minor comment on utility classes: there's JAXPTestUtilities and TestUtils. The former is an util for all tests while the later contained a couple of SAX handlers, it may make sense to move them into their own classes just as the other Handlers. The constants (XML_DIR and GOLDEN_DIR) then can be declared in a base class for the SAX tests. >>>>> >>>>> I understand each group of tests have their own XML source and golden files, thus XML_DIR and GOLDEN_DIR. It may be possible to add a base for each group while put test.src and test.classes into the very base class for all tests. So in general, we would have: >>>>> TestBase FileTestBase >>>>> Base for each group (e.g. SAXTestBase, DOMTestBase, XSLTTestBase...) that extends either TestBase or FileTestBase >>>>> >>>>> I remember there were a few tests that required a http server. So then we may need a HttpTestBase as well. >>>>> >>>>> I know this is not what we've discussed (or planned) previously. But since you've done a great job to incorporate what were previously quite separate test suites into one whole test suite. I can't resist to ask a bit more. Don't get me wrong, what you've done exceeded my expectation in a big way (only a month ago, we were still talking about quick/straight conversion so that you can start to take care of the new features)! >>>>> >>>>> SAXParserTest: I noticed Old: testParse11, testParse27 --> New: testParse05. So is testParse03 a new test? I can see SAXException is expected, but not IOE. In fact, this shows that the JAXP spec missed defining how empty string shall be handled (should have been an IAE). >>>>> >>>>> Best, >>>>> Joe >>>>> >>>>> On 1/6/2015 11:33 AM, Lance Andersen wrote: >>>>>> Hi Tristan, >>>>>> >>>>>> Sorry for the delay but with the holidays and the number of files, it took a while to go through :-) >>>>>> >>>>>> Overall, it looks pretty good. >>>>>> >>>>>> A couple of suggestions, but I would not necessarily hold back from committing: >>>>>> >>>>>> - For tests where you are not caring about the actual exception that is thrown to indicate a failure, such as in DocumentBuilderFactory01.java and testCheckSchemaSupport1, I would just have the method declaration use "throws Exception" vs list all of the possible individual Exceptions, as it keeps it more compact. Glad to see you are not using failUnexcepted() now. >>>>>> >>>>>> - In test classes such as in testCheckSchemaSupport3. java and DocumentBuilderImpl01.java, some tests do not use assertXXX or expect an Exception. Would be good at least to document what could cause a failure or make it clear the expected behavior of the test for passing. >>>>>> >>>>>> -SAXParserTest02.java and other tests where you get a reader/parser such as testXMLReader01, I would at least assert that null is not returned where as it is now, you only validate that an exception is not returned >>>>>> >>>>>> - I know you are porting existing tests, but I would consider consolidating tests as it seems like there is not a real reason to have a testNG class with just 1 test. I would group the like tests (such as SAXTFactoryTest ) in one testNG test class as that allows you to streamline further. >>>>>> >>>>>> - TfClearParamTest.java (as and example) minor nit that you have your @Before/AfterGroups method in between tests. I would suggest grouping all methods such as this DataProviders before or after the actual tests for better organization >>>>>> >>>>>> - TraxSAXWrapper.java, not sure I understand the "Sorry I could not resist comment" >>>>>> >>>>>> - TransformerHandlerAPITest.java has typos in comments: "IllegalArgumentExceptionis thrown?." >>>>>> >>>>>> - Minitest.java, I would add a comment for your Data Provider >>>>>> >>>>>> Best, >>>>>> Lance >>>>>> >>>>>> On Jan 2, 2015, at 1:49 PM, Tristan Yan > wrote: >>>>>> >>>>>>> Hi Joe and Lance >>>>>>> Sorry for my late reply. I just uploaded the new webrev which is trying to limit minimal permissions for most of tests. The new changeset has two base classes; class JAXPBaseTest has only minimal set permissions; class JAXPFileBaseTest adds two permissions that need access local files in the directory directory test.src and test.classes. Most of tests only inherit JAXPBaseTest that provides only minimal set permissions. Some tests will inherit JAXPFileBaseTest because tests need access local files. >>>>>>> Please help to review the code. >>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>>>> >>>>>>> Thank you >>>>>>> Tristan >>>>>>> >>>>>>> >>>>>>>> On Jan 2, 2015, at 10:39 AM, huizhe wang > wrote: >>>>>>>> >>>>>>>> Lance, >>>>>>>> >>>>>>>> Tristan is looking into adding an extension base class for about 60 tests that require file permission, then the current base class would indeed set "minimal" permission. So please wait for his update :-) >>>>>>>> >>>>>>>> Best, >>>>>>>> Joe >>>>>>>> >>>>>>>> On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >>>>>>>>> Hi Tristan, >>>>>>>>> >>>>>>>>> I will look at this but doubt I will get to this tomorrow >>>>>>>>> >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Lance >>>>>>>>> >>>>>>>>> >>>>>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>>>> Oracle Java Engineering >>>>>>>>> 1 Network Drive >>>>>>>>> Burlington, MA 01803 >>>>>>>>> Lance.Andersen at oracle.com >>>>>>>>> Sent from my iPad >>>>>>>>> >>>>>>>>> On Dec 30, 2014, at 5:28 PM, Tristan Yan > wrote: >>>>>>>>> >>>>>>>>>> Hi All >>>>>>>>>> >>>>>>>>>> Can I get your review on this change. >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ > >>>>>>>>>> >>>>>>>>>> These fixes originally come from bug https://bugs.openjdk.java.net/browse/JDK-8051563 > as part of our XML test colocation work. ThIS change-set mainly covers tests for package org.apache.qetest.dtm and org.apache.qetest.trax. >>>>>>>>>> >>>>>>>>>> In the meantime I took steps at fixing some of our existed test code on below issues: >>>>>>>>>> >>>>>>>>>> 1. Add a base test class for all functional tests that enable security manager running. A limited minimal permissions set have been set for every test. >>>>>>>>>> 2. Remove all unnecessary exception capture for functional tests that we?re using testng to handle all the exceptions. >>>>>>>>>> 3. Use try-resource block to solve all possible resource leaks (including InputStream, OutputStream, Writer, Reader). >>>>>>>>>> >>>>>>>>>> Thanks a lot. >>>>>>>>>> Tristan >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>> Oracle Java Engineering >>>>>> 1 Network Drive >>>>>> Burlington, MA 01803 >>>>>> Lance.Andersen at oracle.com >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> >>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>> Oracle Java Engineering >>> 1 Network Drive >>> Burlington, MA 01803 >>> Lance.Andersen at oracle.com >>> >>> >>> >> >> >> >> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> >> >> > From forax at univ-mlv.fr Thu Jan 8 08:11:47 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 08 Jan 2015 09:11:47 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <54ADEA76.3010701@oracle.com> References: <54AD004A.4090603@univ-mlv.fr> <54ADEA76.3010701@oracle.com> Message-ID: <54AE3BC3.3090305@univ-mlv.fr> On 01/08/2015 03:24 AM, David Holmes wrote: > On 7/01/2015 7:45 PM, Remi Forax wrote: >> A simple Java question, what this code does ? >> >> ArrayList list = new ArrayList<>(); >> list.add("foo"); >> list.add("bar"); >> for(String s: list) { >> list.remove(s); >> } >> >> :( >> >> R?mi >> tip: the bug lies in ArrayList.Itr.hasNext() (and >> AbstractList.Itr.hasNext()). > > This is not a bug. The only supported way to remove from a collection > you are iterating over is to use an Iterator's remove method. > > http://docs.oracle.com/javase/8/docs/api/java/util/Iterator.html#remove-- > > "The behavior of an iterator is unspecified if the underlying > collection is modified while the iteration is in progress in any way > other than by calling this method." > > and as already pointed out the forEach docs make it clear you can't > use forEach in such a context as the Iterator is not exposed to you. > > So don't do that. Hi David, while I agree that fail-fast behavior should be done on the best effort basis, i think like Paul said that the size should be stored in a field of the iterator because i don't see the point to try to support the fact that the size of the collection can change during the iteration. so in my opinion, it's not a bug but the current implementation of ArrayList.Itr should be changed or maybe there is a case where supporting a size changed is valid ? > > David > > > R?mi From peter.firmstone at zeus.net.au Thu Jan 8 08:32:01 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Thu, 08 Jan 2015 18:32:01 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54AD5883.3070705@gmail.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> Message-ID: <54AE4081.2090600@zeus.net.au> Thank you all for participating in this discussion. Initially a constructor signature for deserialization was proposed to enforce invariants and encapsulation, however there appears to be interest in using alternative methods, although they appear to be improvements over the status quo, I'm having trouble working out how to: * Enforce intra class invariants with alternate proposals without breaking encapsulation. * How the static method proposal can be used to replace final fields referents without needing to implement readObject(). * And how the alternative GetFields readObject() implementation can enforce invariants and prevent finalizer attack, while also allowing subclassing? What started me down this path, is our project (Apache River) is heavily dependant on Serialization, I wanted to create a secure ObjectInputStream subclass that restricted deserialized classes to those I had audited. My aim was to allow deserialization to enforce security using verification, similar to how an http server verifies its input. Unfortunately before I can achieve the goal of secure deserialization, there's a denial of service issue in ObjectInputStream: I'd also like to propose a system property, to allow limiting the size of arrays, created during deserialization. Presently I can craft an inputstream to take down the JVM, if I can do it, so can an attacker, I'd like to get that fixed if possible. I've attached a text file that contains three classes using the original proposed serial constructor, A, B and C which have intra class invariants that must be satiisfied. C also contains a circular link and implements an interface called Circular, the Serialization framework calls it after construction to provide access to fields with circular links. Can the other proposals provide similar safety? Notes: 1. The class with the circular link is final and every method checks if invariants are satisfied. 2. The method with @SerialConstructor annotation, is the only constructor called by the serialization frame work, other than the Circular interface method, there are no other methods that need to be implemented. 3. Encapsulation is preserved, classes have full control over their invariants, their internal implementation and what's serialized. Regards, Peter. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Explicit serialization.txt URL: From paul.sandoz at oracle.com Thu Jan 8 08:36:19 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 8 Jan 2015 09:36:19 +0100 Subject: Bug in ArrayList iterator In-Reply-To: <54AE3BC3.3090305@univ-mlv.fr> References: <54AD004A.4090603@univ-mlv.fr> <54ADEA76.3010701@oracle.com> <54AE3BC3.3090305@univ-mlv.fr> Message-ID: <99CA4037-B6AC-453A-A7F4-47C677A93D85@oracle.com> > On 01/08/2015 03:24 AM, David Holmes wrote: >> >> So don't do that. > Yes, don't. However, it can happen unintentionally and such an exception helps catch bugs. This can more easily occur when there is some "distance" between iterating and operating on the list. So it's about strengthening the best-effort basis of the implementation (ideally without unduly perturbing existing time and space characteristics). On Jan 8, 2015, at 9:11 AM, Remi Forax wrote: > Hi David, > while I agree that fail-fast behavior should be done on the best effort basis, > i think like Paul said that the size should be stored in a field of the iterator because > i don't see the point to try to support the fact that the size of the collection > can change during the iteration. > > so in my opinion, it's not a bug but the current implementation of ArrayList.Itr should be changed > or maybe there is a case where supporting a size changed is valid ? > Any size change (via addition or removal, bulk or otherwise) is a structural modification in some form or another to the underlying data structure. The only allowed modification i am aware of when iterating is that performed by the iterator itself via Iterator.remove. Paul. From martinrb at google.com Thu Jan 8 08:36:13 2015 From: martinrb at google.com (Martin Buchholz) Date: Thu, 8 Jan 2015 00:36:13 -0800 Subject: Bug in ArrayList iterator In-Reply-To: <54AE3BC3.3090305@univ-mlv.fr> References: <54AD004A.4090603@univ-mlv.fr> <54ADEA76.3010701@oracle.com> <54AE3BC3.3090305@univ-mlv.fr> Message-ID: I would not like to add to the overhead of ArrayList's iterator. Those writing performance-critical code are avoiding the iterator and doing for (int i = 0, limit = list.size(); i < limit; i++) { E elt = list.get(i); ... } On Thu, Jan 8, 2015 at 12:11 AM, Remi Forax wrote: > > On 01/08/2015 03:24 AM, David Holmes wrote: > >> On 7/01/2015 7:45 PM, Remi Forax wrote: >> >>> A simple Java question, what this code does ? >>> >>> ArrayList list = new ArrayList<>(); >>> list.add("foo"); >>> list.add("bar"); >>> for(String s: list) { >>> list.remove(s); >>> } >>> >>> :( >>> >>> R?mi >>> tip: the bug lies in ArrayList.Itr.hasNext() (and >>> AbstractList.Itr.hasNext()). >>> >> >> This is not a bug. The only supported way to remove from a collection you >> are iterating over is to use an Iterator's remove method. >> >> http://docs.oracle.com/javase/8/docs/api/java/util/Iterator.html#remove-- >> >> "The behavior of an iterator is unspecified if the underlying collection >> is modified while the iteration is in progress in any way other than by >> calling this method." >> >> and as already pointed out the forEach docs make it clear you can't use >> forEach in such a context as the Iterator is not exposed to you. >> >> So don't do that. >> > > Hi David, > while I agree that fail-fast behavior should be done on the best effort > basis, > i think like Paul said that the size should be stored in a field of the > iterator because > i don't see the point to try to support the fact that the size of the > collection > can change during the iteration. > > so in my opinion, it's not a bug but the current implementation of > ArrayList.Itr should be changed > or maybe there is a case where supporting a size changed is valid ? > > >> David >> >> >> >> > R?mi > > From chris.hegarty at oracle.com Thu Jan 8 14:24:29 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 8 Jan 2015 14:24:29 +0000 Subject: RFR [9] 8068599: Apply mutability, serializability, and thread-safety, caveat to all Collectors that do not accept a Collection supplier Message-ID: This is a review request for a small spec clarification to java.util.stream.Collectors, that appears to be a minor oversight in the original spec. Many of the Collectors that accumulate, group, etc, elements into a Collection, and do not accept a Collection supplier, come with a caveat on the Collection they create, along the lines of, "There are no guarantees on the type, mutability, serializability, or thread-safety of the ... returned". This caveat should be applied to all Collectors that create such Collections. Specdiff: http://cr.openjdk.java.net/~chegar/8068599/specdiff/java/util/stream/Collectors.html webrev: http://cr.openjdk.java.net/~chegar/8068599/webrev/ Thanks, -Chris. From sean.coffey at oracle.com Thu Jan 8 14:41:17 2015 From: sean.coffey at oracle.com (=?UTF-8?B?U2XDoW4gQ29mZmV5?=) Date: Thu, 08 Jan 2015 14:41:17 +0000 Subject: RFR : 8066612: Add a test that will call getDeclaredFields() on all classes and try to set them accessible. Message-ID: <54AE970D.4020805@oracle.com> I've taken suggested test code from Daniel and am looking to backport 8066612 to jdk7u-dev. The test differs slightly in that it used the non-lambda approach (ClassNameListBuilder) I've reduced the number of run counts also (one ovm run for secure and one for non-secure mode) - As a result and given the large number of classes loaded, I've bumped PermSize up to 128m - no harm to stress that side of things also I guess. Test passes on all JPRT platforms. I'll revert to smaller class load counts if issues are seen with this test approach. webrev : http://cr.openjdk.java.net/~coffeys/webrev.8066612.jdk7u/webrev/ bug ID : https://bugs.openjdk.java.net/browse/JDK-8066612 regards, Sean. From daniel.fuchs at oracle.com Thu Jan 8 14:55:20 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Thu, 08 Jan 2015 15:55:20 +0100 Subject: RFR : 8066612: Add a test that will call getDeclaredFields() on all classes and try to set them accessible. In-Reply-To: <54AE970D.4020805@oracle.com> References: <54AE970D.4020805@oracle.com> Message-ID: <54AE9A58.70607@oracle.com> Hi Se?n, Thanks for taking care of that :-) It looks good to me. You might want to set the copyright year to 2015. Best regards, -- daniel On 08/01/15 15:41, Se?n Coffey wrote: > I've taken suggested test code from Daniel and am looking to backport > 8066612 to jdk7u-dev. The test differs slightly in that it used the > non-lambda approach (ClassNameListBuilder) > > I've reduced the number of run counts also (one ovm run for secure and > one for non-secure mode) - As a result and given the large number of > classes loaded, I've bumped PermSize up to 128m - no harm to stress that > side of things also I guess. > > Test passes on all JPRT platforms. I'll revert to smaller class load > counts if issues are seen with this test approach. > > webrev : http://cr.openjdk.java.net/~coffeys/webrev.8066612.jdk7u/webrev/ > bug ID : https://bugs.openjdk.java.net/browse/JDK-8066612 > > regards, > Sean. > > From erik.joelsson at oracle.com Thu Jan 8 14:55:41 2015 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Thu, 08 Jan 2015 15:55:41 +0100 Subject: RFR: JDK-8042707: Source changes needed to build JDK 9 with Visual Studio 2013 (VS2013) Message-ID: <54AE9A6D.2030101@oracle.com> Hello, Please review this patch, which adds support for building with different versions of Visual Studio and in particular adds support for VS2013. In order to control which version to use, I've introduced a new configure parameter "--with-toolchain-version". On windows, this parameter will have the valid values 2010, 2012 and 2013. The default is still 2010. Note that 2012 was added for convenience, but has not been tested to actually work. The longer term goal is to switch the official compiler used for JDK 9 to VS2013. This is just the first step. The main difference that needed to be addressed was that _STATIC_CPPLIB is no longer supported since VS2012, so we will now have to bundle another runtime dll (MSVCP). Also, since the build needs to be compatible with multiple versions of VS, we can no longer hard code the name msvcr100.dll. I changed the names of the dlls into macros that get added to the preprocessor command line. Note that the implementation for msvcp*.dll in java_md.c could perhaps be more elegant. I'm not familiar enough with the APIs, but if someone would like to improve on it, please do. Bug: https://bugs.openjdk.java.net/browse/JDK-8042707 Webrevs: http://cr.openjdk.java.net/~erikj/8042707/webrev.root.01/ http://cr.openjdk.java.net/~erikj/8042707/webrev.jdk.01/ Here is a patch for make/jprt.properties that adds new build and test platforms using the new compilers for windows and macosx. If you apply this and run JPRT, windows and mac platforms will be built and tested with both the old and the new compilers. I would like to commit this change too, but will do it separately later. diff -r ef5c7075496d make/jprt.properties --- a/make/jprt.properties +++ b/make/jprt.properties @@ -115,6 +115,12 @@ ${my.i586.default.build.configure.args} \ ${jprt.productOpen.build.configure.args} +jprt.windows_i586_6.2.build.configure.args= \ + --with-toolchain-version=2013 \ + ${jprt.i586.build.configure.args} +jprt.windows_x64_6.2.build.configure.args= \ + --with-toolchain-version=2013 + ######## # # Build targets and options (default/jdk) @@ -130,8 +136,11 @@ linux_i586_2.6-{product|fastdebug}, \ linux_x64_2.6-{product|fastdebug}, \ macosx_x64_10.7-{product|fastdebug}, \ - windows_i586_6.1-{product|fastdebug}, \ - windows_x64_6.1-{product|fastdebug} + macosx_x64_10.9-{product|fastdebug}, \ + windows_i586_6.1-{product|fastdebug}, \ + windows_x64_6.1-{product|fastdebug}, \ + windows_i586_6.2-{product|fastdebug}, \ + windows_x64_6.2-{product|fastdebug} # Test target list (no fastdebug & limited c2 testing) my.test.target.set= \ @@ -140,8 +149,11 @@ linux_i586_2.6-product-{c1|c2}-TESTNAME, \ linux_x64_2.6-product-c2-TESTNAME, \ macosx_x64_10.7-product-c2-TESTNAME, \ + macosx_x64_10.9-product-c2-TESTNAME, \ windows_i586_6.1-product-c1-TESTNAME, \ - windows_x64_6.1-product-c2-TESTNAME + windows_x64_6.1-product-c2-TESTNAME, \ + windows_i586_6.2-product-c1-TESTNAME, \ + windows_x64_6.2-product-c2-TESTNAME # Default vm test targets (testset=default) my.test.targets.default= \ /Erik From andreas.lundblad at oracle.com Thu Jan 8 15:13:56 2015 From: andreas.lundblad at oracle.com (Andreas Lundblad) Date: Thu, 8 Jan 2015 16:13:56 +0100 Subject: JDK 9 RFR of 8067669: Documentation for methods in Number incomplete regarding too large values. In-Reply-To: <54ADC94E.4090607@oracle.com> References: <9D55A146-D76D-4E91-B02D-2BC646451CE2@oracle.com> <20141217102612.GE4911@e6430> <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com> <5494B41B.8060003@oracle.com> <0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com> <3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com> <54AB3DBE.5040806@oracle.com> <20150107080837.GB2073@e6430> <54ADC94E.4090607@oracle.com> Message-ID: <20150108151355.GA11334@e6430> > > On the matter of writing a Comparator, the basic problem is > the Number interface is not strong enough to allow you to write such > functionality. > > First, the Number type basically only means "convertible to a > primitive." There are no strict requirements on consistency between > the different conversion methods and there is not even a requirement > that calling the same Number method on the same object returns the > same value (since Numbers are not required to be immutable). > > Even just writing a semantically correct Comparator for > Number subclasses in the JDK is nontrivial. Multiple Number > subclasses (BigInteger and BigDecimal) have values outside of the > set of values that can be represented with a primitive type. > > So a general Comparator is not really well-defined. Right. This is something I discovered in the process. (Feel free to look at how for I got [1].) Although I was the one who brought it up, I think the whole Comparator discussion is slightly off topic. I still think that explicitly mentioning rounding and truncation is a bit confusing, as even the JDK implementations resort to other methods. [1] http://stackoverflow.com/a/4192900/276052 From david.lloyd at redhat.com Thu Jan 8 15:32:26 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 08 Jan 2015 09:32:26 -0600 Subject: Explicit Serialization API and Security In-Reply-To: <54AE4081.2090600@zeus.net.au> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> <54AE4081.2090600@zeus.net.au> Message-ID: <54AEA30A.9090203@redhat.com> On 01/08/2015 02:32 AM, Peter Firmstone wrote: > Thank you all for participating in this discussion. > > Initially a constructor signature for deserialization was proposed to > enforce invariants and encapsulation, however there appears to be > interest in using alternative methods, although they appear to be > improvements over the status quo, I'm having trouble working out how to: > > * Enforce intra class invariants with alternate proposals without > breaking encapsulation. > * How the static method proposal can be used to replace final fields > referents without needing to implement readObject(). > * And how the alternative GetFields readObject() implementation can > enforce invariants and prevent finalizer attack, while also > allowing subclassing? > > What started me down this path, is our project (Apache River) is heavily > dependant on Serialization, I wanted to create a secure > ObjectInputStream subclass that restricted deserialized classes to those > I had audited. My aim was to allow deserialization to enforce security > using verification, similar to how an http server verifies its input. Now we're getting down to brass tacks. :-) Maybe I missed this part of the discussion, but can you elaborate on why ObjectInputValidation is inadequate for validating intra-class invariants? I was thinking this is really the only way to effectively achieve intra-class validation, because (barring complex graph traversal algorithms) there isn't really any better way to guarantee that any given subgraph of the object graph is actually fully initialized. As far as replacing final fields, I think the seed of the best idea came from Chris Hegarty - the problem isn't the readObject() method, the problem is that there's no way to initialize your final fields if you use it. By adding a "defaultReadFields()" (name not important) method like he proposes, you can validate your single class. But I don't see any reason why this couldn't be taken another step farther, and allow the readObject method to actually change, add, and remove fields from its internal map before doing so. This would allow readObject to arbitrarily transform its content, and assign to final fields, while only minimally changing the API and implementation. > Unfortunately before I can achieve the goal of secure deserialization, > there's a denial of service issue in ObjectInputStream: I'd also like to > propose a system property, to allow limiting the size of arrays, created > during deserialization. Presently I can craft an inputstream to take > down the JVM, if I can do it, so can an attacker, I'd like to get that > fixed if possible. I agree this would be a very good (and easy to implement) idea. You can limit the underlying stream but that does you no good if the attacker can cause the construction of any number of very large objects before the end of stream is hit. I wonder if this could be as simple as adding a protected method on ObjectInputStream in the vein of: protected boolean validateArrayCreation(Class elementType, int dimension) { return true; } Or elementType could be arrayType; I don't think it matters too much. Method returns false and InvalidObjectException is thrown. Implementations could do a simple global max array size, different max sizes by primitive vs ref, different by type, or even have a "budget" of heap and use a combination of this method and overridden resovleClass() to create a rough estimate of memory usage, failing at a heuristic threshold. > I've attached a text file that contains three classes using the original > proposed serial constructor, A, B and C which have intra class > invariants that must be satiisfied. C also contains a circular link and > implements an interface called Circular, the Serialization framework > calls it after construction to provide access to fields with circular > links. Can the other proposals provide similar safety? > > Notes: > > 1. The class with the circular link is final and every method checks > if invariants are satisfied. > 2. The method with @SerialConstructor annotation, is the only > constructor called by the serialization frame work, other than the > Circular interface method, there are no other methods that need to > be implemented. > 3. Encapsulation is preserved, classes have full control over their > invariants, their internal implementation and what's serialized. I was previously firmly of the opinion that a serialization constructor is the best way to accomplish this, but this discussion thread has at least given me quite some doubt if not outright changed my mind, though I think there's still an unanswered question regarding validation (above). If nothing else, this discussion has yielded some good ideas for enhancements to JBoss Marshalling. :-) -- - DML From paul.sandoz at oracle.com Thu Jan 8 15:46:34 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 8 Jan 2015 16:46:34 +0100 Subject: RFR [9] 8068599: Apply mutability, serializability, and thread-safety, caveat to all Collectors that do not accept a Collection supplier In-Reply-To: References: Message-ID: On Jan 8, 2015, at 3:24 PM, Chris Hegarty wrote: > This is a review request for a small spec clarification to java.util.stream.Collectors, that appears to be a minor oversight in the original spec. > > Many of the Collectors that accumulate, group, etc, elements into a Collection, and do not accept a Collection supplier, come with a caveat on the Collection they create, along the lines of, "There are no guarantees on the type, mutability, serializability, or thread-safety of the ... returned". > > This caveat should be applied to all Collectors that create such Collections. > > Specdiff: > http://cr.openjdk.java.net/~chegar/8068599/specdiff/java/util/stream/Collectors.html > webrev: > http://cr.openjdk.java.net/~chegar/8068599/webrev/ > Thanks for doing this. The toMap docs are missing 'thread-safety", otherwise looks good. Paul. From chris.hegarty at oracle.com Thu Jan 8 16:06:38 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 8 Jan 2015 16:06:38 +0000 Subject: RFR [9] 8068599: Apply mutability, serializability, and thread-safety, caveat to all Collectors that do not accept a Collection supplier In-Reply-To: References: Message-ID: On 8 Jan 2015, at 15:46, Paul Sandoz wrote: > > On Jan 8, 2015, at 3:24 PM, Chris Hegarty wrote: > >> This is a review request for a small spec clarification to java.util.stream.Collectors, that appears to be a minor oversight in the original spec. >> >> Many of the Collectors that accumulate, group, etc, elements into a Collection, and do not accept a Collection supplier, come with a caveat on the Collection they create, along the lines of, "There are no guarantees on the type, mutability, serializability, or thread-safety of the ... returned". >> >> This caveat should be applied to all Collectors that create such Collections. >> >> Specdiff: >> http://cr.openjdk.java.net/~chegar/8068599/specdiff/java/util/stream/Collectors.html >> webrev: >> http://cr.openjdk.java.net/~chegar/8068599/webrev/ >> > > Thanks for doing this. The toMap docs are missing 'thread-safety", otherwise looks good. Fixed. Webrev and specdiff have been updated in place. Thanks, -Chris. From ecki at zusammenkunft.net Thu Jan 8 16:38:11 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Thu, 8 Jan 2015 17:38:11 +0100 Subject: HashMap collision speed (regression 7->8) Message-ID: <20150108173811.00003a76.ecki@zusammenkunft.net> Hello, I think it was topic before, but I just wanted to point out, that it is still an topic on the internetz. :) Motivated by a StackOverflow question regarding HashMap performance regression in Java 8 http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java-8/27760442 I made a JMH test and compared 7 and 8 speed. (the test is not very scientific as I dont really know how to squeeze the longrunning loop which alters state into the harness, but the results seem to be consitent wth theory and stopwatch testing) https://gist.github.com/ecki/9f69773eb29428a36077 What can be seen is, that with a good distribution of hash keys 8 looks faster than 7, and with a bad distribution of hash keys Java 7 is faster (unless you supply a Comparator for the key). (and a good distributed hashkey with comparable seems to be a bit slower) I think the regression is somewhat expected, but I guess its not widely known. (I do not use a cached hashcode, but it has a nearly trivial implementation just to make it more life like. the tests also compares different initial sizes, but they do not have an measurable effect on the performance, I show only default size below:) java version "1.7.0_72" Benchmark (initialSize) Mode Samp Score Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 10847,318 ? 5596,690 ms/op n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 10761,430 ? 5376,975 ms/op n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 3613,923 ? 254,823 ms/op n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3656,229 ? 274,350 ms/op java version "1.8.0_25" Benchmark (initialSize) Mode Samp Score Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 14309,880 ? 1811,709 ms/op <-slower n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 8232,037 ? 5974,530 ms/op n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 3304,698 ? 116,866 ms/op n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3425,762 ? 107,659 ms/op Greetings Bernd From paul.sandoz at oracle.com Thu Jan 8 16:40:39 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 8 Jan 2015 17:40:39 +0100 Subject: RFR [9] 8068599: Apply mutability, serializability, and thread-safety, caveat to all Collectors that do not accept a Collection supplier In-Reply-To: References: Message-ID: <84036ED0-EF56-48AE-8877-CC9E38592365@oracle.com> On Jan 8, 2015, at 5:06 PM, Chris Hegarty wrote: > On 8 Jan 2015, at 15:46, Paul Sandoz wrote: > >> >> On Jan 8, 2015, at 3:24 PM, Chris Hegarty wrote: >> >>> This is a review request for a small spec clarification to java.util.stream.Collectors, that appears to be a minor oversight in the original spec. >>> >>> Many of the Collectors that accumulate, group, etc, elements into a Collection, and do not accept a Collection supplier, come with a caveat on the Collection they create, along the lines of, "There are no guarantees on the type, mutability, serializability, or thread-safety of the ... returned". >>> >>> This caveat should be applied to all Collectors that create such Collections. >>> >>> Specdiff: >>> http://cr.openjdk.java.net/~chegar/8068599/specdiff/java/util/stream/Collectors.html >>> webrev: >>> http://cr.openjdk.java.net/~chegar/8068599/webrev/ >>> >> >> Thanks for doing this. The toMap docs are missing 'thread-safety", otherwise looks good. > > Fixed. Webrev and specdiff have been updated in place. > +1 Paul. From chris.hegarty at oracle.com Thu Jan 8 17:09:12 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 8 Jan 2015 17:09:12 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54AEA30A.9090203@redhat.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> <54AE4081.2090600@zeus.net.au> <54AEA30A.9090203@redhat.com> Message-ID: <97533E9F-0281-4EDD-9504-C38DE9453BE8@oracle.com> Peter, David I would like to see how far we can push the existing Serialization mechanism, before embarking on a road that may lead us to substantially different one. Whether that be constructor based, to otherwise ( we know we will need a replacement for Unsafe.allocateInstance, for third-party Serialization libraries, in the future ). Just to summarise the potential improvements that have been discussed so far ( I can create JIRA issues for these ): 1) Validate invariants A clear and easy to understand mechanism that can validate the deserialized fields. Does not prevent the use of final fields, as the serialization framework will be responsible for setting them. Something along the lines of what David suggested: private static void validate(GetField fields) { if (fields.getInt("lo") > fields.getInt("hi")) { ... } } This could be a ?special? method, or annotation driven. TBD. Note: the validate method is static, so the object instance is not required to be created before running the validation. 2) Failure atomicity for defaultReadObject and readFields In many cases the setting of field values can be deferred until after they have been validated, at a minimum that the non-primitive types are assignable. Whole hierarchy failure atomicity in many cases ( where there are not ?special readObjectXXX methods ), or at a minimum per class failure atomicity, can be achieved ( either all fields will be set, or contain their default values ). Coupled with 1) above this could work well. 3) Finalization It is clear that the finalization attack is an issue for deserialized objects. I think that a deserialized object should not be ?finalizable? until after a certain point in its construction. I would like to investigate further the possibility of making the VM aware of the first no-args default constructor, or j.l.Object, being called by the serialization mechanism, and possibly treating it differently. Or maybe spin ?field holder? classes for finalizable types being deserialized, and deserialze into them first. This doesn?t need to be ultra-performant. 4) Confinement A new mechanism to ensure that a deserialized field value is not shared, and loaded by the same loader, or parent of, the object being constructed?s loader. This will eliminate the need to define a readObject to essentially copy the deserialized field value, as it cannot always be trusted. And again the issue with final fields. Something along the lines of: class Foo { @Confined private final Bar bar = new BarImpl(); } @Confined could be placed on fields, or possibly classes. There may have been more, sorry if I missed anything. -Chris. On 8 Jan 2015, at 15:32, David M. Lloyd wrote: > On 01/08/2015 02:32 AM, Peter Firmstone wrote: >> Thank you all for participating in this discussion. >> >> Initially a constructor signature for deserialization was proposed to >> enforce invariants and encapsulation, however there appears to be >> interest in using alternative methods, although they appear to be >> improvements over the status quo, I'm having trouble working out how to: >> >> * Enforce intra class invariants with alternate proposals without >> breaking encapsulation. >> * How the static method proposal can be used to replace final fields >> referents without needing to implement readObject(). >> * And how the alternative GetFields readObject() implementation can >> enforce invariants and prevent finalizer attack, while also >> allowing subclassing? >> >> What started me down this path, is our project (Apache River) is heavily >> dependant on Serialization, I wanted to create a secure >> ObjectInputStream subclass that restricted deserialized classes to those >> I had audited. My aim was to allow deserialization to enforce security >> using verification, similar to how an http server verifies its input. > > Now we're getting down to brass tacks. :-) > > Maybe I missed this part of the discussion, but can you elaborate on why ObjectInputValidation is inadequate for validating intra-class invariants? I was thinking this is really the only way to effectively achieve intra-class validation, because (barring complex graph traversal algorithms) there isn't really any better way to guarantee that any given subgraph of the object graph is actually fully initialized. > > As far as replacing final fields, I think the seed of the best idea came from Chris Hegarty - the problem isn't the readObject() method, the problem is that there's no way to initialize your final fields if you use it. By adding a "defaultReadFields()" (name not important) method like he proposes, you can validate your single class. But I don't see any reason why this couldn't be taken another step farther, and allow the readObject method to actually change, add, and remove fields from its internal map before doing so. This would allow readObject to arbitrarily transform its content, and assign to final fields, while only minimally changing the API and implementation. > >> Unfortunately before I can achieve the goal of secure deserialization, >> there's a denial of service issue in ObjectInputStream: I'd also like to >> propose a system property, to allow limiting the size of arrays, created >> during deserialization. Presently I can craft an inputstream to take >> down the JVM, if I can do it, so can an attacker, I'd like to get that >> fixed if possible. > > I agree this would be a very good (and easy to implement) idea. You can limit the underlying stream but that does you no good if the attacker can cause the construction of any number of very large objects before the end of stream is hit. > > I wonder if this could be as simple as adding a protected method on ObjectInputStream in the vein of: > > protected boolean validateArrayCreation(Class elementType, int dimension) { > return true; > } > > Or elementType could be arrayType; I don't think it matters too much. Method returns false and InvalidObjectException is thrown. Implementations could do a simple global max array size, different max sizes by primitive vs ref, different by type, or even have a "budget" of heap and use a combination of this method and overridden resovleClass() to create a rough estimate of memory usage, failing at a heuristic threshold. > >> I've attached a text file that contains three classes using the original >> proposed serial constructor, A, B and C which have intra class >> invariants that must be satiisfied. C also contains a circular link and >> implements an interface called Circular, the Serialization framework >> calls it after construction to provide access to fields with circular >> links. Can the other proposals provide similar safety? >> >> Notes: >> >> 1. The class with the circular link is final and every method checks >> if invariants are satisfied. >> 2. The method with @SerialConstructor annotation, is the only >> constructor called by the serialization frame work, other than the >> Circular interface method, there are no other methods that need to >> be implemented. >> 3. Encapsulation is preserved, classes have full control over their >> invariants, their internal implementation and what's serialized. > > I was previously firmly of the opinion that a serialization constructor is the best way to accomplish this, but this discussion thread has at least given me quite some doubt if not outright changed my mind, though I think there's still an unanswered question regarding validation (above). > > If nothing else, this discussion has yielded some good ideas for enhancements to JBoss Marshalling. :-) > > -- > - DML From chris.hegarty at oracle.com Thu Jan 8 19:17:29 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 8 Jan 2015 19:17:29 +0000 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <20150108173811.00003a76.ecki@zusammenkunft.net> References: <20150108173811.00003a76.ecki@zusammenkunft.net> Message-ID: <770C5B8A-4B80-4F25-B774-5903397B5F76@oracle.com> Interesting observations. The class description of HashMap in Java SE 8 alludes to the behaviour you are seeing with respect to Comparable keys. "Note that using many keys with the same hashCode() is a sure way to slow down performance of any hash table. To ameliorate impact, when keys are Comparable, this class may use comparison order among keys to help break ties.? -Chris. On 8 Jan 2015, at 16:38, Bernd Eckenfels wrote: > Hello, > > I think it was topic before, but I just wanted to point out, that it is > still an topic on the internetz. :) > > Motivated by a StackOverflow question regarding HashMap performance > regression in Java 8 > > http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java-8/27760442 > > I made a JMH test and compared 7 and 8 speed. (the test is not very scientific as I dont really know how to squeeze the longrunning loop which alters state into the harness, but the results seem to be consitent wth theory and stopwatch testing) > > https://gist.github.com/ecki/9f69773eb29428a36077 > > What can be seen is, that with a good distribution of hash keys 8 looks > faster than 7, and with a bad distribution of hash keys Java 7 is > faster (unless you supply a Comparator for the key). (and a good distributed hashkey with comparable seems to be a bit slower) > > I think the regression is somewhat expected, but I guess its not widely > known. > > (I do not use a cached hashcode, but it has a nearly trivial implementation just to make it more life like. the tests also compares different initial sizes, but they do not have > an measurable effect on the performance, I show only default size below:) > > java version "1.7.0_72" > > Benchmark (initialSize) Mode Samp Score Error Units > n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 10847,318 ? 5596,690 ms/op > n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 10761,430 ? 5376,975 ms/op > n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 3613,923 ? 254,823 ms/op > n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3656,229 ? 274,350 ms/op > > > java version "1.8.0_25" > > Benchmark (initialSize) Mode Samp Score Error Units > n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 14309,880 ? 1811,709 ms/op <-slower > n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 8232,037 ? 5974,530 ms/op > n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 3304,698 ? 116,866 ms/op > n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3425,762 ? 107,659 ms/op > > > Greetings > Bernd From peter.levart at gmail.com Thu Jan 8 19:22:13 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 08 Jan 2015 20:22:13 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <20150108173811.00003a76.ecki@zusammenkunft.net> References: <20150108173811.00003a76.ecki@zusammenkunft.net> Message-ID: <54AED8E5.5080109@gmail.com> Hi Bernd, It seems that only bad hash codes (without comparable keys) in JDK8 HM are worse than JDK7 HM. Since you have already taken time to measure JDK7 vs JDK8 HM, could you try to take the JDK8 source and just replace the internal "comparableClassFor" method with the following implementation: static final ClassValue selfComparable = new ClassValue() { @Override protected Boolean computeValue(Class c) { Type[] as; ParameterizedType p; for (Type t : c.getGenericInterfaces()) { if (t instanceof ParameterizedType && (p = (ParameterizedType) t).getRawType() == Comparable.class && (as = p.getActualTypeArguments()).length == 1 && as[0] == c) // type arg is c return true; } return false; } }; static Class comparableClassFor(Object x) { if (x instanceof Comparable) { Class c = x.getClass(); if (c == String.class || selfComparable.get(c)) { return c; } } return null; } ...and retry your measurements. I just want to see if it has any impact. Thanks, Peter On 01/08/2015 05:38 PM, Bernd Eckenfels wrote: > Hello, > > I think it was topic before, but I just wanted to point out, that it is > still an topic on the internetz. :) > > Motivated by a StackOverflow question regarding HashMap performance > regression in Java 8 > > http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java-8/27760442 > > I made a JMH test and compared 7 and 8 speed. (the test is not very scientific as I dont really know how to squeeze the longrunning loop which alters state into the harness, but the results seem to be consitent wth theory and stopwatch testing) > > https://gist.github.com/ecki/9f69773eb29428a36077 > > What can be seen is, that with a good distribution of hash keys 8 looks > faster than 7, and with a bad distribution of hash keys Java 7 is > faster (unless you supply a Comparator for the key). (and a good distributed hashkey with comparable seems to be a bit slower) > > I think the regression is somewhat expected, but I guess its not widely > known. > > (I do not use a cached hashcode, but it has a nearly trivial implementation just to make it more life like. the tests also compares different initial sizes, but they do not have > an measurable effect on the performance, I show only default size below:) > > java version "1.7.0_72" > > Benchmark (initialSize) Mode Samp Score Error Units > n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 10847,318 ? 5596,690 ms/op > n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 10761,430 ? 5376,975 ms/op > n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 3613,923 ? 254,823 ms/op > n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3656,229 ? 274,350 ms/op > > > java version "1.8.0_25" > > Benchmark (initialSize) Mode Samp Score Error Units > n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 14309,880 ? 1811,709 ms/op <-slower > n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 8232,037 ? 5974,530 ms/op > n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 3304,698 ? 116,866 ms/op > n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3425,762 ? 107,659 ms/op > > > Greetings > Bernd From ecki at zusammenkunft.net Thu Jan 8 19:41:20 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Thu, 8 Jan 2015 20:41:20 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54AED8E5.5080109@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> Message-ID: <20150108204120.0000323b.ecki@zusammenkunft.net> Hello Peter, yes it is only keys without an Compareable interface, but they are quite common. I think the main problem with the internal comparator (based on instance identity) is, that it would work for looking up the same instance again, but not for the case where the actual instance is re-created (as in my example code). I would love to test your modified code, but I don't have a OpenJDK build environment handy. Or actually I can try to get one, is there somewhere a Virtulisation or Cloud Image available which is pre-installed? I have (1.6) a compiled benchmark.jar here, in case anyone wants to try it: https://onedrive.live.com/redir?resid=A98B6F4E09966AFD!20440&authkey=!AFFk03-5jq21Xz0&ithint=file%2cjar BTW: I am (as usual) not expecting commoents on that, but just to mention it: the expected good behavior of degenerated hashmaps (due to the tree) was reason for removing the hashcode secret randomization. I wonder if that was such a good idea if colliding lookups (with more than a handfull of entries in a bucket) have this 50% penalty. Greetings Bernd Am Thu, 08 Jan 2015 20:22:13 +0100 schrieb Peter Levart : > Hi Bernd, > > It seems that only bad hash codes (without comparable keys) in JDK8 > HM are worse than JDK7 HM. > > > Since you have already taken time to measure JDK7 vs JDK8 HM, could > you try to take the JDK8 source and just replace the internal > "comparableClassFor" method with the following implementation: > > static final ClassValue selfComparable = new > ClassValue() { > @Override > protected Boolean computeValue(Class c) { > Type[] as; ParameterizedType p; > for (Type t : c.getGenericInterfaces()) { > if (t instanceof ParameterizedType && > (p = (ParameterizedType) t).getRawType() == > Comparable.class && > (as = p.getActualTypeArguments()).length == 1 && > as[0] == c) // type arg is c > return true; > } > return false; > } > }; > > static Class comparableClassFor(Object x) { > if (x instanceof Comparable) { > Class c = x.getClass(); > if (c == String.class || selfComparable.get(c)) { > return c; > } > } > return null; > } > > > ...and retry your measurements. I just want to see if it has any > impact. > > > Thanks, Peter > > > On 01/08/2015 05:38 PM, Bernd Eckenfels wrote: > > Hello, > > > > I think it was topic before, but I just wanted to point out, that > > it is still an topic on the internetz. :) > > > > Motivated by a StackOverflow question regarding HashMap performance > > regression in Java 8 > > > > http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java-8/27760442 > > > > I made a JMH test and compared 7 and 8 speed. (the test is not very > > scientific as I dont really know how to squeeze the longrunning > > loop which alters state into the harness, but the results seem to > > be consitent wth theory and stopwatch testing) > > > > https://gist.github.com/ecki/9f69773eb29428a36077 > > > > What can be seen is, that with a good distribution of hash keys 8 > > looks faster than 7, and with a bad distribution of hash keys Java > > 7 is faster (unless you supply a Comparator for the key). (and a > > good distributed hashkey with comparable seems to be a bit slower) > > > > I think the regression is somewhat expected, but I guess its not > > widely known. > > > > (I do not use a cached hashcode, but it has a nearly trivial > > implementation just to make it more life like. the tests also > > compares different initial sizes, but they do not have an > > measurable effect on the performance, I show only default size > > below:) > > > > java version "1.7.0_72" > > > > Benchmark (initialSize) Mode Samp Score > > Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 > > 10847,318 ? 5596,690 ms/op n.e.j.h.HashMapCollision.badDistWithComp > > 16 avgt 4 10761,430 ? 5376,975 ms/op > > n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 3613,923 ? > > 254,823 ms/op n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt > > 4 3656,229 ? 274,350 ms/op > > java version "1.8.0_25" > > > > Benchmark (initialSize) Mode Samp Score > > Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 > > 14309,880 ? 1811,709 ms/op <-slower > > n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 8232,037 ? > > 5974,530 ms/op n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt > > 4 3304,698 ? 116,866 ms/op > > n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3425,762 ? > > 107,659 ms/op > > > > > > Greetings > > Bernd > > From brian.goetz at oracle.com Thu Jan 8 20:10:40 2015 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 08 Jan 2015 15:10:40 -0500 Subject: Explicit Serialization API and Security In-Reply-To: <97533E9F-0281-4EDD-9504-C38DE9453BE8@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> <54AE4081.2090600@zeus.net.au> <54AEA30A.9090203@redhat.com> <97533E9F-0281-4EDD-9504-C38DE9453BE8@oracle.com> Message-ID: <54AEE440.2060709@oracle.com> > 1) Validate invariants > > A clear and easy to understand mechanism that can validate the deserialized > fields. Does not prevent the use of final fields, as the serialization framework > will be responsible for setting them. Something along the lines of what David > suggested: > > private static void validate(GetField fields) { > if (fields.getInt("lo") > fields.getInt("hi")) { ... } > } > > This could be a ?special? method, or annotation driven. TBD. > > Note: the validate method is static, so the object instance is not required to > be created before running the validation. Sort of... This is true if the fields participating in the invariant are primitives. But if they're refs, what do you do? What if you want to validate something like count == list.size() // fields are int count, List list ? Then wouldn't GetField.getObject have to deserialize the object referred to by that field? From sbaiduzh at redhat.com Thu Jan 8 19:49:22 2015 From: sbaiduzh at redhat.com (Stanislav Baiduzhyi) Date: Thu, 08 Jan 2015 20:49:22 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <20150108204120.0000323b.ecki@zusammenkunft.net> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> Message-ID: <1591732.ZXfyiIqWOs@thinkpad.hell> On Thursday 08 January 2015 20:41:20 Bernd Eckenfels wrote: > Hello Peter, > > yes it is only keys without an Compareable interface, but they are > quite common. I think the main problem with the internal comparator > (based on instance identity) is, that it would work for looking up the > same instance again, but not for the case where the actual instance is > re-created (as in my example code). > > I would love to test your modified code, but I don't have a > OpenJDK build environment handy. Or actually I can try to get one, is > there somewhere a Virtulisation or Cloud Image available which is > pre-installed? It should not be a problem. Create a custom class names HashMap in custom package named java.util, and copy all the source from original HashMap into your implementation. Custom one should be earlier in the classpath than the one from rt.jar, but if not - just repack rt.jar with your implementation. That should do the trick. > I have (1.6) a compiled benchmark.jar here, in case anyone wants to try > it: > > https://onedrive.live.com/redir?resid=A98B6F4E09966AFD!20440&authkey=!AFFk03 > -5jq21Xz0&ithint=file%2cjar > > BTW: I am (as usual) not expecting commoents on that, but just to > mention it: the expected good behavior of degenerated hashmaps (due to > the tree) was reason for removing the hashcode secret randomization. I > wonder if that was such a good idea if colliding lookups (with more > than a handfull of entries in a bucket) have this 50% penalty. > > Greetings > Bernd > > > > Am Thu, 08 Jan 2015 20:22:13 +0100 > > schrieb Peter Levart : > > Hi Bernd, > > > > It seems that only bad hash codes (without comparable keys) in JDK8 > > HM are worse than JDK7 HM. > > > > > > Since you have already taken time to measure JDK7 vs JDK8 HM, could > > you try to take the JDK8 source and just replace the internal > > > > "comparableClassFor" method with the following implementation: > > static final ClassValue selfComparable = new > > > > ClassValue() { > > > > @Override > > protected Boolean computeValue(Class c) { > > > > Type[] as; ParameterizedType p; > > for (Type t : c.getGenericInterfaces()) { > > > > if (t instanceof ParameterizedType && > > > > (p = (ParameterizedType) t).getRawType() == > > > > Comparable.class && > > > > (as = p.getActualTypeArguments()).length == 1 && > > as[0] == c) // type arg is c > > return true; > > > > } > > return false; > > > > } > > > > }; > > > > static Class comparableClassFor(Object x) { > > > > if (x instanceof Comparable) { > > > > Class c = x.getClass(); > > if (c == String.class || selfComparable.get(c)) { > > > > return c; > > > > } > > > > } > > return null; > > > > } > > > > ...and retry your measurements. I just want to see if it has any > > impact. > > > > > > Thanks, Peter > > > > On 01/08/2015 05:38 PM, Bernd Eckenfels wrote: > > > Hello, > > > > > > I think it was topic before, but I just wanted to point out, that > > > it is still an topic on the internetz. :) > > > > > > Motivated by a StackOverflow question regarding HashMap performance > > > regression in Java 8 > > > > > > http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java > > > -8/27760442 > > > > > > I made a JMH test and compared 7 and 8 speed. (the test is not very > > > scientific as I dont really know how to squeeze the longrunning > > > loop which alters state into the harness, but the results seem to > > > be consitent wth theory and stopwatch testing) > > > > > > https://gist.github.com/ecki/9f69773eb29428a36077 > > > > > > What can be seen is, that with a good distribution of hash keys 8 > > > looks faster than 7, and with a bad distribution of hash keys Java > > > 7 is faster (unless you supply a Comparator for the key). (and a > > > good distributed hashkey with comparable seems to be a bit slower) > > > > > > I think the regression is somewhat expected, but I guess its not > > > widely known. > > > > > > (I do not use a cached hashcode, but it has a nearly trivial > > > implementation just to make it more life like. the tests also > > > compares different initial sizes, but they do not have an > > > measurable effect on the performance, I show only default size > > > below:) > > > > > > java version "1.7.0_72" > > > > > > Benchmark (initialSize) Mode Samp Score > > > Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 > > > 10847,318 ? 5596,690 ms/op n.e.j.h.HashMapCollision.badDistWithComp > > > 16 avgt 4 10761,430 ? 5376,975 ms/op > > > n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 3613,923 ? > > > 254,823 ms/op n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt > > > 4 3656,229 ? 274,350 ms/op > > > java version "1.8.0_25" > > > > > > Benchmark (initialSize) Mode Samp Score > > > Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 > > > 14309,880 ? 1811,709 ms/op <-slower > > > n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 8232,037 ? > > > 5974,530 ms/op n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt > > > 4 3304,698 ? 116,866 ms/op > > > n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3425,762 ? > > > 107,659 ms/op > > > > > > > > > Greetings > > > Bernd -- Regards, Stas From kissziszi at gmail.com Thu Jan 8 20:34:03 2015 From: kissziszi at gmail.com (Zoltan Sziladi) Date: Thu, 8 Jan 2015 12:34:03 -0800 Subject: String.indexOf optimization In-Reply-To: <54ABAAC2.2090502@redhat.com> References: <534D68B8.3040002@oracle.com> <5350E316.2050406@oracle.com> <54ABAAC2.2090502@redhat.com> Message-ID: Hi, Thanks everyone for all the info. So, just to go step by step in understanding this. Andrew said HotSpot would ignore my implementation. So why is there an implementation of indexOf at all in the JDK, if that's not the code that's executed? Is it just a default fallback? When is the indexOf function not intrinsified? When do people usually disable intrinsification? Sorry if these are newbie questions, I'm new to this part of Java. Regards, Zoltan On Tue, Jan 6, 2015 at 1:28 AM, Andrew Haley wrote: > Hi, > > On 05/01/15 18:59, Zoltan Sziladi wrote: > > > This discussion was a long time ago, I was just reading through it to > check > > again what was the last state of the discussion about the String.indexOf. > > There is one part which I still do not understand, hopefully someone > could > > shed some light on it. A few emails ago Martin mentioned > > > > "Hotspot seems to have some intrinsification of String.indexOf, which > > confuses me. > > Hotspot seems the right place to provide more optimizations for this, > since > > there has been a fair amount of work creating high-performance low-level > > implementations of this idea in C." > > > > Then Ivan asked what that actually meant, whether hotspot actually > replaced > > the jdk implementation with a low level optimized C implementation, but I > > never saw an answer to that. > > You can have a look at an implementation of MacroAssembler::string_indexof > in > > > http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b6b89b8f8531/src/cpu/x86/vm/macroAssembler_x86.cpp > > > Can someone please explain this? If we somehow found an algorithm that > beat > > the naive implementation in the average case, would it be possible to > just > > implement it in the JDK? > > No, because HotSpot would ignore it. Any speed improvements have to be > done in the architecture-dependent files. > > Andrew. > > From vitalyd at gmail.com Thu Jan 8 20:42:37 2015 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Thu, 8 Jan 2015 15:42:37 -0500 Subject: String.indexOf optimization In-Reply-To: References: <534D68B8.3040002@oracle.com> <5350E316.2050406@oracle.com> <54ABAAC2.2090502@redhat.com> Message-ID: The java impl you saw would be called by (a) interpreter, (b) if you explicitly disable intrinsification of this function, or (c) some other JVM that doesn't intrinsify this method (or any method). People don't usually disable intrinsics; if they do, it's because they hit some JIT bug and may disable it. On Thu, Jan 8, 2015 at 3:34 PM, Zoltan Sziladi wrote: > Hi, > > Thanks everyone for all the info. > So, just to go step by step in understanding this. > Andrew said HotSpot would ignore my implementation. So why is there an > implementation of indexOf at all in the JDK, if that's not the code that's > executed? Is it just a default fallback? When is the indexOf function not > intrinsified? When do people usually disable intrinsification? > Sorry if these are newbie questions, I'm new to this part of Java. > > Regards, > Zoltan > > On Tue, Jan 6, 2015 at 1:28 AM, Andrew Haley wrote: > > > Hi, > > > > On 05/01/15 18:59, Zoltan Sziladi wrote: > > > > > This discussion was a long time ago, I was just reading through it to > > check > > > again what was the last state of the discussion about the > String.indexOf. > > > There is one part which I still do not understand, hopefully someone > > could > > > shed some light on it. A few emails ago Martin mentioned > > > > > > "Hotspot seems to have some intrinsification of String.indexOf, which > > > confuses me. > > > Hotspot seems the right place to provide more optimizations for this, > > since > > > there has been a fair amount of work creating high-performance > low-level > > > implementations of this idea in C." > > > > > > Then Ivan asked what that actually meant, whether hotspot actually > > replaced > > > the jdk implementation with a low level optimized C implementation, > but I > > > never saw an answer to that. > > > > You can have a look at an implementation of > MacroAssembler::string_indexof > > in > > > > > > > http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b6b89b8f8531/src/cpu/x86/vm/macroAssembler_x86.cpp > > > > > Can someone please explain this? If we somehow found an algorithm that > > beat > > > the naive implementation in the average case, would it be possible to > > just > > > implement it in the JDK? > > > > No, because HotSpot would ignore it. Any speed improvements have to be > > done in the architecture-dependent files. > > > > Andrew. > > > > > From david.lloyd at redhat.com Thu Jan 8 22:03:19 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 08 Jan 2015 16:03:19 -0600 Subject: Explicit Serialization API and Security In-Reply-To: <54AEE440.2060709@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> <54AE4081.2090600@zeus.net.au> <54AEA30A.9090203@redhat.com> <97533E9F-0281-4EDD-9504-C38DE9453BE8@oracle.com> <54AEE440.2060709@oracle.com> Message-ID: <54AEFEA7.80503@redhat.com> On 01/08/2015 02:10 PM, Brian Goetz wrote: >> 1) Validate invariants >> >> A clear and easy to understand mechanism that can validate the >> deserialized >> fields. Does not prevent the use of final fields, as the >> serialization framework >> will be responsible for setting them. Something along the lines >> of what David >> suggested: >> >> private static void validate(GetField fields) { >> if (fields.getInt("lo") > fields.getInt("hi")) { ... } >> } >> >> This could be a ?special? method, or annotation driven. TBD. >> >> Note: the validate method is static, so the object instance is >> not required to >> be created before running the validation. > > Sort of... > > This is true if the fields participating in the invariant are > primitives. But if they're refs, what do you do? What if you want to > validate something like > > count == list.size() // fields are int count, List list > > ? Then wouldn't GetField.getObject have to deserialize the object > referred to by that field? In fact you cannot validate invariants across multiple objects at all using this method *or* readObject() (existing or enhanced) unless the object in question is an enum, Class, or String (and a few other special cases) because you can't rely on initialization order during deserialization. That's the very reason why OIS#registerValidation() even exists - inter-object validation is not possible until after the root-most readObject has completed, which is the time when validations are executed. Robust validation of a given object class may require two stages - "near" validation and "spanning" validation - to fully validate. However the readObject() approach and its proposed variations (including the static validate() version) can still be useful for fail-fast and non-complex validations; you just have to understand that (just as today) any Object you examine might not be fully initialized yet. -- - DML From peter.levart at gmail.com Thu Jan 8 22:10:10 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 08 Jan 2015 23:10:10 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <20150108204120.0000323b.ecki@zusammenkunft.net> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> Message-ID: <54AF0042.3080304@gmail.com> Bernd, I tried to change the "comparableClassFor" myself and it didn't work (HashMap is used very early in boot-up sequence and initializing ClassValue at that time triggers a NPE). Anyway, caching of "comparableClassFor" result would only potentially improve the "badDistWithComp" result. But can not improve "badDistNoComp" which is the one with speed regression as you're benchmark suggests. But your feeling that this is caused by "simpler" hashCode bits spreading function is not correct. I tried to replace the hash() method with the one that was in HM before and I get comparable results. This is the JDK8 HashMap.hash() method: static final int hash(Object key) { int h; return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> 16); } Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 avgt 4 3171.264 1152.995 ms/op j.t.HashMapCollision.badDistWithComp 16 avgt 4 2819.342 422.861 ms/op j.t.HashMapCollision.goodDistNoComp 16 avgt 4 1026.064 72.049 ms/op j.t.HashMapCollision.goodDistWithComp 16 avgt 4 1025.312 39.858 ms/op ...and this is my re-interpretation of pre JDK8 HashMap.hash(): static final int randomHash = mix32(System.currentTimeMillis() ^ System.nanoTime()); private static int mix32(long z) { z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL; return (int)(((z ^ (z >>> 33)) * 0xc4ceb9fe1a85ec53L) >>> 32); } static final int hash(Object k) { int h = k == null ? randomHash : randomHash ^ k.hashCode(); // This function ensures that hashCodes that differ only by // constant multiples at each bit position have a bounded // number of collisions (approximately 8 at default load factor). h ^= (h >>> 20) ^ (h >>> 12); return h ^ (h >>> 7) ^ (h >>> 4); } Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 avgt 4 3257.348 1079.088 ms/op j.t.HashMapCollision.badDistWithComp 16 avgt 4 2866.740 414.687 ms/op j.t.HashMapCollision.goodDistNoComp 16 avgt 4 1041.068 99.370 ms/op j.t.HashMapCollision.goodDistWithComp 16 avgt 4 1041.653 53.925 ms/op Your benchmark does not show much difference. Perhaps the regression for "badDistNoComp" case could be caused by the fact that with really bad hashCode and no Comparable interface, the red-black tree becomes less performant to search than a simple linked list of Nodes... Regards, Peter On 01/08/2015 08:41 PM, Bernd Eckenfels wrote: > Hello Peter, > > yes it is only keys without an Compareable interface, but they are > quite common. I think the main problem with the internal comparator > (based on instance identity) is, that it would work for looking up the > same instance again, but not for the case where the actual instance is > re-created (as in my example code). > > I would love to test your modified code, but I don't have a > OpenJDK build environment handy. Or actually I can try to get one, is > there somewhere a Virtulisation or Cloud Image available which is > pre-installed? > > I have (1.6) a compiled benchmark.jar here, in case anyone wants to try > it: > > https://onedrive.live.com/redir?resid=A98B6F4E09966AFD!20440&authkey=!AFFk03-5jq21Xz0&ithint=file%2cjar > > BTW: I am (as usual) not expecting commoents on that, but just to > mention it: the expected good behavior of degenerated hashmaps (due to > the tree) was reason for removing the hashcode secret randomization. I > wonder if that was such a good idea if colliding lookups (with more > than a handfull of entries in a bucket) have this 50% penalty. > > Greetings > Bernd > > > > Am Thu, 08 Jan 2015 20:22:13 +0100 > schrieb Peter Levart : > >> Hi Bernd, >> >> It seems that only bad hash codes (without comparable keys) in JDK8 >> HM are worse than JDK7 HM. >> >> >> Since you have already taken time to measure JDK7 vs JDK8 HM, could >> you try to take the JDK8 source and just replace the internal >> "comparableClassFor" method with the following implementation: >> >> static final ClassValue selfComparable = new >> ClassValue() { >> @Override >> protected Boolean computeValue(Class c) { >> Type[] as; ParameterizedType p; >> for (Type t : c.getGenericInterfaces()) { >> if (t instanceof ParameterizedType && >> (p = (ParameterizedType) t).getRawType() == >> Comparable.class && >> (as = p.getActualTypeArguments()).length == 1 && >> as[0] == c) // type arg is c >> return true; >> } >> return false; >> } >> }; >> >> static Class comparableClassFor(Object x) { >> if (x instanceof Comparable) { >> Class c = x.getClass(); >> if (c == String.class || selfComparable.get(c)) { >> return c; >> } >> } >> return null; >> } >> >> >> ...and retry your measurements. I just want to see if it has any >> impact. >> >> >> Thanks, Peter >> >> >> On 01/08/2015 05:38 PM, Bernd Eckenfels wrote: >>> Hello, >>> >>> I think it was topic before, but I just wanted to point out, that >>> it is still an topic on the internetz. :) >>> >>> Motivated by a StackOverflow question regarding HashMap performance >>> regression in Java 8 >>> >>> http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java-8/27760442 >>> >>> I made a JMH test and compared 7 and 8 speed. (the test is not very >>> scientific as I dont really know how to squeeze the longrunning >>> loop which alters state into the harness, but the results seem to >>> be consitent wth theory and stopwatch testing) >>> >>> https://gist.github.com/ecki/9f69773eb29428a36077 >>> >>> What can be seen is, that with a good distribution of hash keys 8 >>> looks faster than 7, and with a bad distribution of hash keys Java >>> 7 is faster (unless you supply a Comparator for the key). (and a >>> good distributed hashkey with comparable seems to be a bit slower) >>> >>> I think the regression is somewhat expected, but I guess its not >>> widely known. >>> >>> (I do not use a cached hashcode, but it has a nearly trivial >>> implementation just to make it more life like. the tests also >>> compares different initial sizes, but they do not have an >>> measurable effect on the performance, I show only default size >>> below:) >>> >>> java version "1.7.0_72" >>> >>> Benchmark (initialSize) Mode Samp Score >>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 >>> 10847,318 ? 5596,690 ms/op n.e.j.h.HashMapCollision.badDistWithComp >>> 16 avgt 4 10761,430 ? 5376,975 ms/op >>> n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 3613,923 ? >>> 254,823 ms/op n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt >>> 4 3656,229 ? 274,350 ms/op >>> java version "1.8.0_25" >>> >>> Benchmark (initialSize) Mode Samp Score >>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 >>> 14309,880 ? 1811,709 ms/op <-slower >>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 8232,037 ? >>> 5974,530 ms/op n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt >>> 4 3304,698 ? 116,866 ms/op >>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3425,762 ? >>> 107,659 ms/op >>> >>> >>> Greetings >>> Bernd From huizhe.wang at oracle.com Thu Jan 8 22:54:58 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Thu, 08 Jan 2015 14:54:58 -0800 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: <1716E308-BE26-4EF6-9C48-FF28D5A640B0@oracle.com> References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> <54AC51FD.7040808@oracle.com> <078315A4-D463-4891-99CE-6A1785BB5D49@oracle.com> <4B99A07A-A547-4CB5-8E6E-98EA57762906@oracle.com> <54AC6161.1000308@oracle.com> <1716E308-BE26-4EF6-9C48-FF28D5A640B0@oracle.com> Message-ID: <54AF0AC2.10206@oracle.com> Thanks for the update. I think the webrev is ready for putback. Best, Joe On 1/7/2015 9:41 PM, Tristan Yan wrote: > Hi Joe/Lance and others. > > Please review updated version for this one. > > http://cr.openjdk.java.net/~tyan/8051563/webrev.01/ > > > Thank you > Tristan > >> On Jan 6, 2015, at 2:27 PM, huizhe wang > > wrote: >> >> >> On 1/6/2015 2:25 PM, Lance Andersen wrote: >>> One more thing :-) >>> >>> Remember to update your copyright year to 2015 also >> >> Indeed, that applies to my webrevs as well :-) >> >> Joe >> >>> >>> Best >>> Lance >>> On Jan 6, 2015, at 5:04 PM, Lance Andersen >>> > wrote: >>> >>>> >>>> On Jan 6, 2015, at 4:31 PM, Tristan Yan >>> > wrote: >>>> >>>>> Thank you Lance and Joe. >>>> >>>> You are very welcome. >>>>> I am working on the fix. >>>> >>>> No rush from my perspective, have a lot to keep me busy in the >>>> interim before your next webrev . :-) >>>>> it may take a couple of days that the code needs some refactor. >>>> Understand, as I have been working on tests for RowSets, I have >>>> continued to play the refactor dance myself. >>>> >>>> Best, >>>> Lance >>>>> I will send out next review once I finish it. >>>>> Thanks >>>>> Tristan >>>>> >>>>>> On Jan 6, 2015, at 1:22 PM, huizhe wang >>>>> > wrote: >>>>>> >>>>>> Thanks for taking the initiative and effort to refactor and clean >>>>>> up all of the Functional tests in previous changeset! >>>>>> >>>>>> We've gone through several iterations for the new ones (xslt >>>>>> tests). I totally agree with Lance, it looks very good overall, a >>>>>> lot better than its original form. >>>>>> >>>>>> It's nice to group all of the tests that required file access, >>>>>> that prepares all other tests to run with minimal permission for >>>>>> a (future) secure-test-target. Unit tests might need a similar >>>>>> treatment, but no pressure to do now :-) >>>>>> >>>>>> A minor comment on utility classes: there's JAXPTestUtilities and >>>>>> TestUtils. The former is an util for all tests while the later >>>>>> contained a couple of SAX handlers, it may make sense to move >>>>>> them into their own classes just as the other Handlers. The >>>>>> constants (XML_DIR and GOLDEN_DIR) then can be declared in a base >>>>>> class for the SAX tests. >>>>>> >>>>>> I understand each group of tests have their own XML source and >>>>>> golden files, thus XML_DIR and GOLDEN_DIR. It may be possible to >>>>>> add a base for each group while put test.src and test.classes >>>>>> into the very base class for all tests. So in general, we would have: >>>>>> TestBase FileTestBase >>>>>> Base for each group (e.g. SAXTestBase, DOMTestBase, >>>>>> XSLTTestBase...) that extends either TestBase or FileTestBase >>>>>> >>>>>> I remember there were a few tests that required a http server. So >>>>>> then we may need a HttpTestBase as well. >>>>>> >>>>>> I know this is not what we've discussed (or planned) previously. >>>>>> But since you've done a great job to incorporate what were >>>>>> previously quite separate test suites into one whole test suite. >>>>>> I can't resist to ask a bit more. Don't get me wrong, what you've >>>>>> done exceeded my expectation in a big way (only a month ago, we >>>>>> were still talking about quick/straight conversion so that you >>>>>> can start to take care of the new features)! >>>>>> >>>>>> SAXParserTest: I noticed Old: testParse11, testParse27 --> New: >>>>>> testParse05. So is testParse03 a new test? I can see >>>>>> SAXException is expected, but not IOE. In fact, this shows that >>>>>> the JAXP spec missed defining how empty string shall be handled >>>>>> (should have been an IAE). >>>>>> >>>>>> Best, >>>>>> Joe >>>>>> >>>>>> On 1/6/2015 11:33 AM, Lance Andersen wrote: >>>>>>> Hi Tristan, >>>>>>> >>>>>>> Sorry for the delay but with the holidays and the number of >>>>>>> files, it took a while to go through :-) >>>>>>> >>>>>>> Overall, it looks pretty good. >>>>>>> >>>>>>> A couple of suggestions, but I would not necessarily hold back >>>>>>> from committing: >>>>>>> >>>>>>> - For tests where you are not caring about the actual exception >>>>>>> that is thrown to indicate a failure, such as >>>>>>> in DocumentBuilderFactory01.java and testCheckSchemaSupport1, I >>>>>>> would just have the method declaration use "throws Exception" >>>>>>> vs list all of the possible individual Exceptions, as it >>>>>>> keeps it more compact. Glad to see you are not using >>>>>>> failUnexcepted() now. >>>>>>> >>>>>>> - In test classes such as in testCheckSchemaSupport3. java >>>>>>> and DocumentBuilderImpl01.java, some tests do not use assertXXX >>>>>>> or expect an Exception. Would be good at least to document what >>>>>>> could cause a failure or make it clear the expected behavior of >>>>>>> the test for passing. >>>>>>> >>>>>>> -SAXParserTest02.java and other tests where you get a >>>>>>> reader/parser such as testXMLReader01, I would at least assert >>>>>>> that null is not returned where as it is now, you only validate >>>>>>> that an exception is not returned >>>>>>> >>>>>>> - I know you are porting existing tests, but I would consider >>>>>>> consolidating tests as it seems like there is not a real reason >>>>>>> to have a testNG class with just 1 test. I would group the >>>>>>> like tests (such as SAXTFactoryTest ) in one testNG test class >>>>>>> as that allows you to streamline further. >>>>>>> >>>>>>> - TfClearParamTest.java (as and example) minor nit that you have >>>>>>> your @Before/AfterGroups method in between tests. I would >>>>>>> suggest grouping all methods such as this DataProviders before >>>>>>> or after the actual tests for better organization >>>>>>> >>>>>>> - TraxSAXWrapper.java, not sure I understand the "Sorry I could >>>>>>> not resist comment" >>>>>>> >>>>>>> - TransformerHandlerAPITest.java has typos in comments: >>>>>>> "IllegalArgumentExceptionis thrown?." >>>>>>> >>>>>>> - Minitest.java, I would add a comment for your Data Provider >>>>>>> >>>>>>> Best, >>>>>>> Lance >>>>>>> >>>>>>> On Jan 2, 2015, at 1:49 PM, Tristan Yan >>>>>> > wrote: >>>>>>> >>>>>>>> Hi Joe and Lance >>>>>>>> Sorry for my late reply. I just uploaded the new webrev which >>>>>>>> is trying to limit minimal permissions for most of tests. The >>>>>>>> new changeset has two base classes; class JAXPBaseTest has only >>>>>>>> minimal set permissions; class JAXPFileBaseTest adds two >>>>>>>> permissions that need access local files in the directory >>>>>>>> directory test.src and test.classes. Most of tests only >>>>>>>> inherit JAXPBaseTest that provides only minimal set >>>>>>>> permissions. Some tests will inherit JAXPFileBaseTest because >>>>>>>> tests need access local files. >>>>>>>> Please help to review the code. >>>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>>>>> >>>>>>>> >>>>>>>> Thank you >>>>>>>> Tristan >>>>>>>> >>>>>>>> >>>>>>>>> On Jan 2, 2015, at 10:39 AM, huizhe wang >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>> Lance, >>>>>>>>> >>>>>>>>> Tristan is looking into adding an extension base class for >>>>>>>>> about 60 tests that require file permission, then the current >>>>>>>>> base class would indeed set "minimal" permission. So please >>>>>>>>> wait for his update :-) >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Joe >>>>>>>>> >>>>>>>>> On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >>>>>>>>>> Hi Tristan, >>>>>>>>>> >>>>>>>>>> I will look at this but doubt I will get to this tomorrow >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Lance >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Lance Andersen| >>>>>>>>>> Principal Member of Technical Staff | +1.781.442.2037 >>>>>>>>>> >>>>>>>>>> Oracle Java Engineering >>>>>>>>>> 1 Network Drive >>>>>>>>>> Burlington, MA 01803 >>>>>>>>>> Lance.Andersen at oracle.com >>>>>>>>>> Sent from my iPad >>>>>>>>>> >>>>>>>>>> On Dec 30, 2014, at 5:28 PM, Tristan Yan >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Hi All >>>>>>>>>>> >>>>>>>>>>> Can I get your review on this change. >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> These fixes originally come from bug >>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8051563 >>>>>>>>>>> as part >>>>>>>>>>> of our XML test colocation work. ThIS change-set mainly >>>>>>>>>>> covers tests for package org.apache.qetest.dtm and >>>>>>>>>>> org.apache.qetest.trax. >>>>>>>>>>> >>>>>>>>>>> In the meantime I took steps at fixing some of our existed >>>>>>>>>>> test code on below issues: >>>>>>>>>>> >>>>>>>>>>> 1. Add a base test class for all functional tests that >>>>>>>>>>> enable security manager running. A limited minimal >>>>>>>>>>> permissions set have been set for every test. >>>>>>>>>>> 2. Remove all unnecessary exception capture for functional >>>>>>>>>>> tests that we?re using testng to handle all the exceptions. >>>>>>>>>>> 3. Use try-resource block to solve all possible resource >>>>>>>>>>> leaks (including InputStream, OutputStream, Writer, Reader). >>>>>>>>>>> >>>>>>>>>>> Thanks a lot. >>>>>>>>>>> Tristan >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Lance >>>>>>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>> Oracle Java Engineering >>>>>>> 1 Network Drive >>>>>>> Burlington, MA 01803 >>>>>>> Lance.Andersen at oracle.com >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>>> >>>> Lance >>>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>> Oracle Java Engineering >>>> 1 Network Drive >>>> Burlington, MA 01803 >>>> Lance.Andersen at oracle.com >>>> >>>> >>>> >>> >>> >>> >>> >>> Lance >>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>> Oracle Java Engineering >>> 1 Network Drive >>> Burlington, MA 01803 >>> Lance.Andersen at oracle.com >>> >>> >>> >> > From kissziszi at gmail.com Thu Jan 8 23:05:38 2015 From: kissziszi at gmail.com (Zoltan Sziladi) Date: Thu, 8 Jan 2015 15:05:38 -0800 Subject: String.indexOf optimization In-Reply-To: References: <534D68B8.3040002@oracle.com> <5350E316.2050406@oracle.com> <54ABAAC2.2090502@redhat.com> Message-ID: Thanks for the info. So that basically means we have 2 implementations of indexOf currently, one is in HotSpot, the other is in the JDK itself, which rarely gets executed. Aside from this later fact, isn't it still worth improving the JDK implementation if it is possible? I know that the intrinsified method gets executed most of the time, but still, if we can improve the JDK implementation also, why not? I don't know much about other JVMs but maybe a few don't intrinsify it? Is there any existing test suite which is considered conclusive enough that if an implementation beats the naive algorithm in those testcases then it could be considered as a replacement in the JDK? Regards, Zoltan On Thu, Jan 8, 2015 at 12:42 PM, Vitaly Davidovich wrote: > The java impl you saw would be called by (a) interpreter, (b) if you > explicitly disable intrinsification of this function, or (c) some other JVM > that doesn't intrinsify this method (or any method). > > People don't usually disable intrinsics; if they do, it's because they hit > some JIT bug and may disable it. > > On Thu, Jan 8, 2015 at 3:34 PM, Zoltan Sziladi > wrote: > >> Hi, >> >> Thanks everyone for all the info. >> So, just to go step by step in understanding this. >> Andrew said HotSpot would ignore my implementation. So why is there an >> implementation of indexOf at all in the JDK, if that's not the code that's >> executed? Is it just a default fallback? When is the indexOf function not >> intrinsified? When do people usually disable intrinsification? >> Sorry if these are newbie questions, I'm new to this part of Java. >> >> Regards, >> Zoltan >> >> On Tue, Jan 6, 2015 at 1:28 AM, Andrew Haley wrote: >> >> > Hi, >> > >> > On 05/01/15 18:59, Zoltan Sziladi wrote: >> > >> > > This discussion was a long time ago, I was just reading through it to >> > check >> > > again what was the last state of the discussion about the >> String.indexOf. >> > > There is one part which I still do not understand, hopefully someone >> > could >> > > shed some light on it. A few emails ago Martin mentioned >> > > >> > > "Hotspot seems to have some intrinsification of String.indexOf, which >> > > confuses me. >> > > Hotspot seems the right place to provide more optimizations for this, >> > since >> > > there has been a fair amount of work creating high-performance >> low-level >> > > implementations of this idea in C." >> > > >> > > Then Ivan asked what that actually meant, whether hotspot actually >> > replaced >> > > the jdk implementation with a low level optimized C implementation, >> but I >> > > never saw an answer to that. >> > >> > You can have a look at an implementation of >> MacroAssembler::string_indexof >> > in >> > >> > >> > >> http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b6b89b8f8531/src/cpu/x86/vm/macroAssembler_x86.cpp >> > >> > > Can someone please explain this? If we somehow found an algorithm that >> > beat >> > > the naive implementation in the average case, would it be possible to >> > just >> > > implement it in the JDK? >> > >> > No, because HotSpot would ignore it. Any speed improvements have to be >> > done in the architecture-dependent files. >> > >> > Andrew. >> > >> > >> > > From ecki at zusammenkunft.net Thu Jan 8 23:24:13 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 9 Jan 2015 00:24:13 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54AF0042.3080304@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> Message-ID: <20150109002413.00000678.ecki@zusammenkunft.net> Hello Peter, hm not sure what you mean, i was not suggesting the regression is caused by simpler hashCode bits. (do you mean my comment about the removed randomizer? that is not a problem for the benchmark, but it might be an opportunity for DOS (again)). I think the regression itself is caused by the fact that a tree is used instead of the linear search. The benchmark does colide heavily, but only places 15 or so entries in one bucket. This is enough to trigger the migration from linear search to tree, but not enough to hurt for linear search performance (at least I think this is the case). The way the nodes are constructed they actually do sort pretty good if compareable is implemented. It would most likely not help if Compareable cannot distinguish more than hashCode would. Gruss Bernd Am Thu, 08 Jan 2015 23:10:10 +0100 schrieb Peter Levart : > Bernd, > > I tried to change the "comparableClassFor" myself and it didn't work > (HashMap is used very early in boot-up sequence and initializing > ClassValue at that time triggers a NPE). > > Anyway, caching of "comparableClassFor" result would only potentially > improve the "badDistWithComp" result. But can not improve > "badDistNoComp" which is the one with speed regression as you're > benchmark suggests. > > But your feeling that this is caused by "simpler" hashCode bits > spreading function is not correct. I tried to replace the hash() > method with the one that was in HM before and I get comparable > results. This is the JDK8 HashMap.hash() method: > > static final int hash(Object key) { > int h; > return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> 16); > } > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 avgt 4 > 3171.264 1152.995 ms/op > j.t.HashMapCollision.badDistWithComp 16 avgt 4 > 2819.342 422.861 ms/op > j.t.HashMapCollision.goodDistNoComp 16 avgt 4 > 1026.064 72.049 ms/op > j.t.HashMapCollision.goodDistWithComp 16 avgt 4 > 1025.312 39.858 ms/op > > > ...and this is my re-interpretation of pre JDK8 HashMap.hash(): > > > static final int randomHash = mix32(System.currentTimeMillis() ^ > System.nanoTime()); > > private static int mix32(long z) { > z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL; > return (int)(((z ^ (z >>> 33)) * 0xc4ceb9fe1a85ec53L) >>> > 32); } > > static final int hash(Object k) { > int h = k == null ? randomHash : randomHash ^ k.hashCode(); > > // This function ensures that hashCodes that differ only by > // constant multiples at each bit position have a bounded > // number of collisions (approximately 8 at default load > factor). h ^= (h >>> 20) ^ (h >>> 12); > return h ^ (h >>> 7) ^ (h >>> 4); > } > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 avgt 4 > 3257.348 1079.088 ms/op > j.t.HashMapCollision.badDistWithComp 16 avgt 4 > 2866.740 414.687 ms/op > j.t.HashMapCollision.goodDistNoComp 16 avgt 4 > 1041.068 99.370 ms/op > j.t.HashMapCollision.goodDistWithComp 16 avgt 4 > 1041.653 53.925 ms/op > > > Your benchmark does not show much difference. Perhaps the regression > for "badDistNoComp" case could be caused by the fact that with really > bad hashCode and no Comparable interface, the red-black tree becomes > less performant to search than a simple linked list of Nodes... > > > Regards, Peter > > > > > > On 01/08/2015 08:41 PM, Bernd Eckenfels wrote: > > Hello Peter, > > > > yes it is only keys without an Compareable interface, but they are > > quite common. I think the main problem with the internal comparator > > (based on instance identity) is, that it would work for looking up > > the same instance again, but not for the case where the actual > > instance is re-created (as in my example code). > > > > I would love to test your modified code, but I don't have a > > OpenJDK build environment handy. Or actually I can try to get one, > > is there somewhere a Virtulisation or Cloud Image available which is > > pre-installed? > > > > I have (1.6) a compiled benchmark.jar here, in case anyone wants to > > try it: > > > > https://onedrive.live.com/redir?resid=A98B6F4E09966AFD!20440&authkey=!AFFk03-5jq21Xz0&ithint=file%2cjar > > > > BTW: I am (as usual) not expecting commoents on that, but just to > > mention it: the expected good behavior of degenerated hashmaps (due > > to the tree) was reason for removing the hashcode secret > > randomization. I wonder if that was such a good idea if colliding > > lookups (with more than a handfull of entries in a bucket) have > > this 50% penalty. > > > > Greetings > > Bernd > > > > > > > > Am Thu, 08 Jan 2015 20:22:13 +0100 > > schrieb Peter Levart : > > > >> Hi Bernd, > >> > >> It seems that only bad hash codes (without comparable keys) in JDK8 > >> HM are worse than JDK7 HM. > >> > >> > >> Since you have already taken time to measure JDK7 vs JDK8 HM, could > >> you try to take the JDK8 source and just replace the internal > >> "comparableClassFor" method with the following implementation: > >> > >> static final ClassValue selfComparable = new > >> ClassValue() { > >> @Override > >> protected Boolean computeValue(Class c) { > >> Type[] as; ParameterizedType p; > >> for (Type t : c.getGenericInterfaces()) { > >> if (t instanceof ParameterizedType && > >> (p = (ParameterizedType) t).getRawType() == > >> Comparable.class && > >> (as = p.getActualTypeArguments()).length == > >> 1 && as[0] == c) // type arg is c > >> return true; > >> } > >> return false; > >> } > >> }; > >> > >> static Class comparableClassFor(Object x) { > >> if (x instanceof Comparable) { > >> Class c = x.getClass(); > >> if (c == String.class || selfComparable.get(c)) { > >> return c; > >> } > >> } > >> return null; > >> } > >> > >> > >> ...and retry your measurements. I just want to see if it has any > >> impact. > >> > >> > >> Thanks, Peter > >> > >> > >> On 01/08/2015 05:38 PM, Bernd Eckenfels wrote: > >>> Hello, > >>> > >>> I think it was topic before, but I just wanted to point out, that > >>> it is still an topic on the internetz. :) > >>> > >>> Motivated by a StackOverflow question regarding HashMap > >>> performance regression in Java 8 > >>> > >>> http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java-8/27760442 > >>> > >>> I made a JMH test and compared 7 and 8 speed. (the test is not > >>> very scientific as I dont really know how to squeeze the > >>> longrunning loop which alters state into the harness, but the > >>> results seem to be consitent wth theory and stopwatch testing) > >>> > >>> https://gist.github.com/ecki/9f69773eb29428a36077 > >>> > >>> What can be seen is, that with a good distribution of hash keys 8 > >>> looks faster than 7, and with a bad distribution of hash keys Java > >>> 7 is faster (unless you supply a Comparator for the key). (and a > >>> good distributed hashkey with comparable seems to be a bit slower) > >>> > >>> I think the regression is somewhat expected, but I guess its not > >>> widely known. > >>> > >>> (I do not use a cached hashcode, but it has a nearly trivial > >>> implementation just to make it more life like. the tests also > >>> compares different initial sizes, but they do not have an > >>> measurable effect on the performance, I show only default size > >>> below:) > >>> > >>> java version "1.7.0_72" > >>> > >>> Benchmark (initialSize) Mode Samp Score > >>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 > >>> 10847,318 ? 5596,690 ms/op > >>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 10761,430 ? > >>> 5376,975 ms/op n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt > >>> 4 3613,923 ? 254,823 ms/op > >>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3656,229 ? > >>> 274,350 ms/op java version "1.8.0_25" > >>> > >>> Benchmark (initialSize) Mode Samp Score > >>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 > >>> 14309,880 ? 1811,709 ms/op <-slower > >>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 8232,037 ? > >>> 5974,530 ms/op n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt > >>> 4 3304,698 ? 116,866 ms/op > >>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3425,762 ? > >>> 107,659 ms/op > >>> > >>> > >>> Greetings > >>> Bernd > > From vitalyd at gmail.com Thu Jan 8 23:32:15 2015 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Thu, 8 Jan 2015 18:32:15 -0500 Subject: String.indexOf optimization In-Reply-To: References: <534D68B8.3040002@oracle.com> <5350E316.2050406@oracle.com> <54ABAAC2.2090502@redhat.com> Message-ID: I'm fairly certain that nobody would object if you send a patch with reproducible benchmark code demonstrating the gain. I think this mostly comes down to cost/benefit of doing this work, and the benefit is marginalized out of the gate by virtue of the intrinsic. You'd probably also have to demonstrate no loss in performance for the common case of a short string. I don't have an answer to your test suite question, unfortunately. Moreover, any such benchmarks would probably be measuring peak performance, which means the JIT intrinsic code. Sent from my phone On Jan 8, 2015 6:05 PM, "Zoltan Sziladi" wrote: > Thanks for the info. > So that basically means we have 2 implementations of indexOf currently, > one is in HotSpot, the other is in the JDK itself, which rarely gets > executed. Aside from this later fact, isn't it still worth improving the > JDK implementation if it is possible? I know that the intrinsified method > gets executed most of the time, but still, if we can improve the JDK > implementation also, why not? I don't know much about other JVMs but maybe > a few don't intrinsify it? > > Is there any existing test suite which is considered conclusive enough > that if an implementation beats the naive algorithm in those testcases then > it could be considered as a replacement in the JDK? > > Regards, > Zoltan > > > On Thu, Jan 8, 2015 at 12:42 PM, Vitaly Davidovich > wrote: > >> The java impl you saw would be called by (a) interpreter, (b) if you >> explicitly disable intrinsification of this function, or (c) some other JVM >> that doesn't intrinsify this method (or any method). >> >> People don't usually disable intrinsics; if they do, it's because they >> hit some JIT bug and may disable it. >> >> On Thu, Jan 8, 2015 at 3:34 PM, Zoltan Sziladi >> wrote: >> >>> Hi, >>> >>> Thanks everyone for all the info. >>> So, just to go step by step in understanding this. >>> Andrew said HotSpot would ignore my implementation. So why is there an >>> implementation of indexOf at all in the JDK, if that's not the code >>> that's >>> executed? Is it just a default fallback? When is the indexOf function not >>> intrinsified? When do people usually disable intrinsification? >>> Sorry if these are newbie questions, I'm new to this part of Java. >>> >>> Regards, >>> Zoltan >>> >>> On Tue, Jan 6, 2015 at 1:28 AM, Andrew Haley wrote: >>> >>> > Hi, >>> > >>> > On 05/01/15 18:59, Zoltan Sziladi wrote: >>> > >>> > > This discussion was a long time ago, I was just reading through it to >>> > check >>> > > again what was the last state of the discussion about the >>> String.indexOf. >>> > > There is one part which I still do not understand, hopefully someone >>> > could >>> > > shed some light on it. A few emails ago Martin mentioned >>> > > >>> > > "Hotspot seems to have some intrinsification of String.indexOf, which >>> > > confuses me. >>> > > Hotspot seems the right place to provide more optimizations for this, >>> > since >>> > > there has been a fair amount of work creating high-performance >>> low-level >>> > > implementations of this idea in C." >>> > > >>> > > Then Ivan asked what that actually meant, whether hotspot actually >>> > replaced >>> > > the jdk implementation with a low level optimized C implementation, >>> but I >>> > > never saw an answer to that. >>> > >>> > You can have a look at an implementation of >>> MacroAssembler::string_indexof >>> > in >>> > >>> > >>> > >>> http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b6b89b8f8531/src/cpu/x86/vm/macroAssembler_x86.cpp >>> > >>> > > Can someone please explain this? If we somehow found an algorithm >>> that >>> > beat >>> > > the naive implementation in the average case, would it be possible to >>> > just >>> > > implement it in the JDK? >>> > >>> > No, because HotSpot would ignore it. Any speed improvements have to be >>> > done in the architecture-dependent files. >>> > >>> > Andrew. >>> > >>> > >>> >> >> > From vitalyd at gmail.com Thu Jan 8 23:38:53 2015 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Thu, 8 Jan 2015 18:38:53 -0500 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <20150109002413.00000678.ecki@zusammenkunft.net> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> Message-ID: Hmmm, 15 entries doesn't seem like a large enough number to drop linear search. I'd have expected something like 60-80 entries (in some binary vs linear search benchmarks I've come across, that seems to be crossover point). It's hard to beat linear search for small sets when the comparison function is dirt cheap. Sent from my phone On Jan 8, 2015 6:25 PM, "Bernd Eckenfels" wrote: > Hello Peter, > > hm not sure what you mean, i was not suggesting the regression is > caused by simpler hashCode bits. (do you mean my comment about the > removed randomizer? that is not a problem for the benchmark, but it > might be an opportunity for DOS (again)). > > I think the regression itself is caused by the fact that a tree is used > instead of the linear search. The benchmark does colide heavily, but > only places 15 or so entries in one bucket. This is enough to trigger > the migration from linear search to tree, but not enough to hurt for > linear search performance (at least I think this is the case). > > The way the nodes are constructed they actually do sort pretty good if > compareable is implemented. It would most likely not help if > Compareable cannot distinguish more than hashCode would. > > Gruss > Bernd > > > > Am Thu, 08 Jan 2015 23:10:10 +0100 > schrieb Peter Levart : > > > Bernd, > > > > I tried to change the "comparableClassFor" myself and it didn't work > > (HashMap is used very early in boot-up sequence and initializing > > ClassValue at that time triggers a NPE). > > > > Anyway, caching of "comparableClassFor" result would only potentially > > improve the "badDistWithComp" result. But can not improve > > "badDistNoComp" which is the one with speed regression as you're > > benchmark suggests. > > > > But your feeling that this is caused by "simpler" hashCode bits > > spreading function is not correct. I tried to replace the hash() > > method with the one that was in HM before and I get comparable > > results. This is the JDK8 HashMap.hash() method: > > > > static final int hash(Object key) { > > int h; > > return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> 16); > > } > > > > Benchmark (initialSize) Mode > > Samples Score Score error Units > > j.t.HashMapCollision.badDistNoComp 16 avgt 4 > > 3171.264 1152.995 ms/op > > j.t.HashMapCollision.badDistWithComp 16 avgt 4 > > 2819.342 422.861 ms/op > > j.t.HashMapCollision.goodDistNoComp 16 avgt 4 > > 1026.064 72.049 ms/op > > j.t.HashMapCollision.goodDistWithComp 16 avgt 4 > > 1025.312 39.858 ms/op > > > > > > ...and this is my re-interpretation of pre JDK8 HashMap.hash(): > > > > > > static final int randomHash = mix32(System.currentTimeMillis() ^ > > System.nanoTime()); > > > > private static int mix32(long z) { > > z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL; > > return (int)(((z ^ (z >>> 33)) * 0xc4ceb9fe1a85ec53L) >>> > > 32); } > > > > static final int hash(Object k) { > > int h = k == null ? randomHash : randomHash ^ k.hashCode(); > > > > // This function ensures that hashCodes that differ only by > > // constant multiples at each bit position have a bounded > > // number of collisions (approximately 8 at default load > > factor). h ^= (h >>> 20) ^ (h >>> 12); > > return h ^ (h >>> 7) ^ (h >>> 4); > > } > > > > Benchmark (initialSize) Mode > > Samples Score Score error Units > > j.t.HashMapCollision.badDistNoComp 16 avgt 4 > > 3257.348 1079.088 ms/op > > j.t.HashMapCollision.badDistWithComp 16 avgt 4 > > 2866.740 414.687 ms/op > > j.t.HashMapCollision.goodDistNoComp 16 avgt 4 > > 1041.068 99.370 ms/op > > j.t.HashMapCollision.goodDistWithComp 16 avgt 4 > > 1041.653 53.925 ms/op > > > > > > Your benchmark does not show much difference. Perhaps the regression > > for "badDistNoComp" case could be caused by the fact that with really > > bad hashCode and no Comparable interface, the red-black tree becomes > > less performant to search than a simple linked list of Nodes... > > > > > > Regards, Peter > > > > > > > > > > > > On 01/08/2015 08:41 PM, Bernd Eckenfels wrote: > > > Hello Peter, > > > > > > yes it is only keys without an Compareable interface, but they are > > > quite common. I think the main problem with the internal comparator > > > (based on instance identity) is, that it would work for looking up > > > the same instance again, but not for the case where the actual > > > instance is re-created (as in my example code). > > > > > > I would love to test your modified code, but I don't have a > > > OpenJDK build environment handy. Or actually I can try to get one, > > > is there somewhere a Virtulisation or Cloud Image available which is > > > pre-installed? > > > > > > I have (1.6) a compiled benchmark.jar here, in case anyone wants to > > > try it: > > > > > > > https://onedrive.live.com/redir?resid=A98B6F4E09966AFD!20440&authkey=!AFFk03-5jq21Xz0&ithint=file%2cjar > > > > > > BTW: I am (as usual) not expecting commoents on that, but just to > > > mention it: the expected good behavior of degenerated hashmaps (due > > > to the tree) was reason for removing the hashcode secret > > > randomization. I wonder if that was such a good idea if colliding > > > lookups (with more than a handfull of entries in a bucket) have > > > this 50% penalty. > > > > > > Greetings > > > Bernd > > > > > > > > > > > > Am Thu, 08 Jan 2015 20:22:13 +0100 > > > schrieb Peter Levart : > > > > > >> Hi Bernd, > > >> > > >> It seems that only bad hash codes (without comparable keys) in JDK8 > > >> HM are worse than JDK7 HM. > > >> > > >> > > >> Since you have already taken time to measure JDK7 vs JDK8 HM, could > > >> you try to take the JDK8 source and just replace the internal > > >> "comparableClassFor" method with the following implementation: > > >> > > >> static final ClassValue selfComparable = new > > >> ClassValue() { > > >> @Override > > >> protected Boolean computeValue(Class c) { > > >> Type[] as; ParameterizedType p; > > >> for (Type t : c.getGenericInterfaces()) { > > >> if (t instanceof ParameterizedType && > > >> (p = (ParameterizedType) t).getRawType() == > > >> Comparable.class && > > >> (as = p.getActualTypeArguments()).length == > > >> 1 && as[0] == c) // type arg is c > > >> return true; > > >> } > > >> return false; > > >> } > > >> }; > > >> > > >> static Class comparableClassFor(Object x) { > > >> if (x instanceof Comparable) { > > >> Class c = x.getClass(); > > >> if (c == String.class || selfComparable.get(c)) { > > >> return c; > > >> } > > >> } > > >> return null; > > >> } > > >> > > >> > > >> ...and retry your measurements. I just want to see if it has any > > >> impact. > > >> > > >> > > >> Thanks, Peter > > >> > > >> > > >> On 01/08/2015 05:38 PM, Bernd Eckenfels wrote: > > >>> Hello, > > >>> > > >>> I think it was topic before, but I just wanted to point out, that > > >>> it is still an topic on the internetz. :) > > >>> > > >>> Motivated by a StackOverflow question regarding HashMap > > >>> performance regression in Java 8 > > >>> > > >>> > http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java-8/27760442 > > >>> > > >>> I made a JMH test and compared 7 and 8 speed. (the test is not > > >>> very scientific as I dont really know how to squeeze the > > >>> longrunning loop which alters state into the harness, but the > > >>> results seem to be consitent wth theory and stopwatch testing) > > >>> > > >>> https://gist.github.com/ecki/9f69773eb29428a36077 > > >>> > > >>> What can be seen is, that with a good distribution of hash keys 8 > > >>> looks faster than 7, and with a bad distribution of hash keys Java > > >>> 7 is faster (unless you supply a Comparator for the key). (and a > > >>> good distributed hashkey with comparable seems to be a bit slower) > > >>> > > >>> I think the regression is somewhat expected, but I guess its not > > >>> widely known. > > >>> > > >>> (I do not use a cached hashcode, but it has a nearly trivial > > >>> implementation just to make it more life like. the tests also > > >>> compares different initial sizes, but they do not have an > > >>> measurable effect on the performance, I show only default size > > >>> below:) > > >>> > > >>> java version "1.7.0_72" > > >>> > > >>> Benchmark (initialSize) Mode Samp Score > > >>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 > > >>> 10847,318 ? 5596,690 ms/op > > >>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 10761,430 ? > > >>> 5376,975 ms/op n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt > > >>> 4 3613,923 ? 254,823 ms/op > > >>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3656,229 ? > > >>> 274,350 ms/op java version "1.8.0_25" > > >>> > > >>> Benchmark (initialSize) Mode Samp Score > > >>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt 4 > > >>> 14309,880 ? 1811,709 ms/op <-slower > > >>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 8232,037 ? > > >>> 5974,530 ms/op n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt > > >>> 4 3304,698 ? 116,866 ms/op > > >>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 3425,762 ? > > >>> 107,659 ms/op > > >>> > > >>> > > >>> Greetings > > >>> Bernd > > > > > From ecki at zusammenkunft.net Fri Jan 9 00:10:35 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 9 Jan 2015 01:10:35 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> Message-ID: <20150109011035.000000f5.ecki@zusammenkunft.net> Hello Vitaly, the TREEIFY_THRESHOLD in HashMap is 8 (6 for UNTREEIFY). Not sure if there have been benchmarks for this, I dont see an justification in the source comments. There is a comment, that it is expected to waste a factor of two time and space when not compareable. I would also expect that larger thresholds make sense (especially when key is not compareable). The JEP 180 also does not mention results from the benchmarks, maybe somebody knows details? http://openjdk.java.net/jeps/180 Gruss Bernd Am Thu, 8 Jan 2015 18:38:53 -0500 schrieb Vitaly Davidovich : > Hmmm, 15 entries doesn't seem like a large enough number to drop > linear search. I'd have expected something like 60-80 entries (in > some binary vs linear search benchmarks I've come across, that seems > to be crossover point). It's hard to beat linear search for small > sets when the comparison function is dirt cheap. > > Sent from my phone > On Jan 8, 2015 6:25 PM, "Bernd Eckenfels" > wrote: > > > Hello Peter, > > > > hm not sure what you mean, i was not suggesting the regression is > > caused by simpler hashCode bits. (do you mean my comment about the > > removed randomizer? that is not a problem for the benchmark, but it > > might be an opportunity for DOS (again)). > > > > I think the regression itself is caused by the fact that a tree is > > used instead of the linear search. The benchmark does colide > > heavily, but only places 15 or so entries in one bucket. This is > > enough to trigger the migration from linear search to tree, but not > > enough to hurt for linear search performance (at least I think this > > is the case). > > > > The way the nodes are constructed they actually do sort pretty good > > if compareable is implemented. It would most likely not help if > > Compareable cannot distinguish more than hashCode would. > > > > Gruss > > Bernd > > > > > > > > Am Thu, 08 Jan 2015 23:10:10 +0100 > > schrieb Peter Levart : > > > > > Bernd, > > > > > > I tried to change the "comparableClassFor" myself and it didn't > > > work (HashMap is used very early in boot-up sequence and > > > initializing ClassValue at that time triggers a NPE). > > > > > > Anyway, caching of "comparableClassFor" result would only > > > potentially improve the "badDistWithComp" result. But can not > > > improve "badDistNoComp" which is the one with speed regression as > > > you're benchmark suggests. > > > > > > But your feeling that this is caused by "simpler" hashCode bits > > > spreading function is not correct. I tried to replace the hash() > > > method with the one that was in HM before and I get comparable > > > results. This is the JDK8 HashMap.hash() method: > > > > > > static final int hash(Object key) { > > > int h; > > > return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> > > > 16); } > > > > > > Benchmark (initialSize) Mode > > > Samples Score Score error Units > > > j.t.HashMapCollision.badDistNoComp 16 avgt 4 > > > 3171.264 1152.995 ms/op > > > j.t.HashMapCollision.badDistWithComp 16 avgt 4 > > > 2819.342 422.861 ms/op > > > j.t.HashMapCollision.goodDistNoComp 16 avgt 4 > > > 1026.064 72.049 ms/op > > > j.t.HashMapCollision.goodDistWithComp 16 avgt 4 > > > 1025.312 39.858 ms/op > > > > > > > > > ...and this is my re-interpretation of pre JDK8 HashMap.hash(): > > > > > > > > > static final int randomHash = > > > mix32(System.currentTimeMillis() ^ System.nanoTime()); > > > > > > private static int mix32(long z) { > > > z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL; > > > return (int)(((z ^ (z >>> 33)) * 0xc4ceb9fe1a85ec53L) >>> > > > 32); } > > > > > > static final int hash(Object k) { > > > int h = k == null ? randomHash : randomHash ^ > > > k.hashCode(); > > > > > > // This function ensures that hashCodes that differ only > > > by // constant multiples at each bit position have a bounded > > > // number of collisions (approximately 8 at default load > > > factor). h ^= (h >>> 20) ^ (h >>> 12); > > > return h ^ (h >>> 7) ^ (h >>> 4); > > > } > > > > > > Benchmark (initialSize) Mode > > > Samples Score Score error Units > > > j.t.HashMapCollision.badDistNoComp 16 avgt 4 > > > 3257.348 1079.088 ms/op > > > j.t.HashMapCollision.badDistWithComp 16 avgt 4 > > > 2866.740 414.687 ms/op > > > j.t.HashMapCollision.goodDistNoComp 16 avgt 4 > > > 1041.068 99.370 ms/op > > > j.t.HashMapCollision.goodDistWithComp 16 avgt 4 > > > 1041.653 53.925 ms/op > > > > > > > > > Your benchmark does not show much difference. Perhaps the > > > regression for "badDistNoComp" case could be caused by the fact > > > that with really bad hashCode and no Comparable interface, the > > > red-black tree becomes less performant to search than a simple > > > linked list of Nodes... > > > > > > > > > Regards, Peter > > > > > > > > > > > > > > > > > > On 01/08/2015 08:41 PM, Bernd Eckenfels wrote: > > > > Hello Peter, > > > > > > > > yes it is only keys without an Compareable interface, but they > > > > are quite common. I think the main problem with the internal > > > > comparator (based on instance identity) is, that it would work > > > > for looking up the same instance again, but not for the case > > > > where the actual instance is re-created (as in my example code). > > > > > > > > I would love to test your modified code, but I don't have a > > > > OpenJDK build environment handy. Or actually I can try to get > > > > one, is there somewhere a Virtulisation or Cloud Image > > > > available which is pre-installed? > > > > > > > > I have (1.6) a compiled benchmark.jar here, in case anyone > > > > wants to try it: > > > > > > > > > > https://onedrive.live.com/redir?resid=A98B6F4E09966AFD!20440&authkey=!AFFk03-5jq21Xz0&ithint=file%2cjar > > > > > > > > BTW: I am (as usual) not expecting commoents on that, but just > > > > to mention it: the expected good behavior of degenerated > > > > hashmaps (due to the tree) was reason for removing the hashcode > > > > secret randomization. I wonder if that was such a good idea if > > > > colliding lookups (with more than a handfull of entries in a > > > > bucket) have this 50% penalty. > > > > > > > > Greetings > > > > Bernd > > > > > > > > > > > > > > > > Am Thu, 08 Jan 2015 20:22:13 +0100 > > > > schrieb Peter Levart : > > > > > > > >> Hi Bernd, > > > >> > > > >> It seems that only bad hash codes (without comparable keys) in > > > >> JDK8 HM are worse than JDK7 HM. > > > >> > > > >> > > > >> Since you have already taken time to measure JDK7 vs JDK8 HM, > > > >> could you try to take the JDK8 source and just replace the > > > >> internal "comparableClassFor" method with the following > > > >> implementation: > > > >> > > > >> static final ClassValue selfComparable = new > > > >> ClassValue() { > > > >> @Override > > > >> protected Boolean computeValue(Class c) { > > > >> Type[] as; ParameterizedType p; > > > >> for (Type t : c.getGenericInterfaces()) { > > > >> if (t instanceof ParameterizedType && > > > >> (p = (ParameterizedType) t).getRawType() > > > >> == Comparable.class && > > > >> (as = p.getActualTypeArguments()).length > > > >> == 1 && as[0] == c) // type arg is c > > > >> return true; > > > >> } > > > >> return false; > > > >> } > > > >> }; > > > >> > > > >> static Class comparableClassFor(Object x) { > > > >> if (x instanceof Comparable) { > > > >> Class c = x.getClass(); > > > >> if (c == String.class || selfComparable.get(c)) { > > > >> return c; > > > >> } > > > >> } > > > >> return null; > > > >> } > > > >> > > > >> > > > >> ...and retry your measurements. I just want to see if it has > > > >> any impact. > > > >> > > > >> > > > >> Thanks, Peter > > > >> > > > >> > > > >> On 01/08/2015 05:38 PM, Bernd Eckenfels wrote: > > > >>> Hello, > > > >>> > > > >>> I think it was topic before, but I just wanted to point out, > > > >>> that it is still an topic on the internetz. :) > > > >>> > > > >>> Motivated by a StackOverflow question regarding HashMap > > > >>> performance regression in Java 8 > > > >>> > > > >>> > > http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java-8/27760442 > > > >>> > > > >>> I made a JMH test and compared 7 and 8 speed. (the test is not > > > >>> very scientific as I dont really know how to squeeze the > > > >>> longrunning loop which alters state into the harness, but the > > > >>> results seem to be consitent wth theory and stopwatch testing) > > > >>> > > > >>> https://gist.github.com/ecki/9f69773eb29428a36077 > > > >>> > > > >>> What can be seen is, that with a good distribution of hash > > > >>> keys 8 looks faster than 7, and with a bad distribution of > > > >>> hash keys Java 7 is faster (unless you supply a Comparator > > > >>> for the key). (and a good distributed hashkey with comparable > > > >>> seems to be a bit slower) > > > >>> > > > >>> I think the regression is somewhat expected, but I guess its > > > >>> not widely known. > > > >>> > > > >>> (I do not use a cached hashcode, but it has a nearly trivial > > > >>> implementation just to make it more life like. the tests also > > > >>> compares different initial sizes, but they do not have an > > > >>> measurable effect on the performance, I show only default size > > > >>> below:) > > > >>> > > > >>> java version "1.7.0_72" > > > >>> > > > >>> Benchmark (initialSize) Mode Samp Score > > > >>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt > > > >>> 4 10847,318 ? 5596,690 ms/op > > > >>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 > > > >>> 10761,430 ? 5376,975 ms/op > > > >>> n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 > > > >>> 3613,923 ? 254,823 ms/op > > > >>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 > > > >>> 3656,229 ? 274,350 ms/op java version "1.8.0_25" > > > >>> > > > >>> Benchmark (initialSize) Mode Samp Score > > > >>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt > > > >>> 4 14309,880 ? 1811,709 ms/op <-slower > > > >>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 > > > >>> 8232,037 ? 5974,530 ms/op > > > >>> n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 > > > >>> 3304,698 ? 116,866 ms/op > > > >>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 > > > >>> 3425,762 ? 107,659 ms/op > > > >>> > > > >>> > > > >>> Greetings > > > >>> Bernd > > > > > > > > > From joe.darcy at oracle.com Fri Jan 9 02:53:54 2015 From: joe.darcy at oracle.com (Joseph D. Darcy) Date: Thu, 08 Jan 2015 18:53:54 -0800 Subject: [9] RFR (M): 8067344: Adjust java/lang/invoke/LFCaching/LFGarbageCollectedTest.java for recent changes in java.lang.invoke In-Reply-To: References: <549884E7.8040204@oracle.com> <549962C4.2040301@oracle.com> Message-ID: <54AF42C2.9070105@oracle.com> Hello, I don't have a comment on the changes to the test per se, but as someone who keeps an eye on test failures that occur in regression tests in the jdk repo of the JDK 9 dev forest, I'd like to see this test stop failing, either by the test being fixed for, barring that, the testing being @ignore-d in some way until the semantics of the test can be corrected. Thanks, -Joe On 1/7/2015 3:16 AM, Paul Sandoz wrote: > Hi > > 70 TestMethods testCase = getTestMethod(); > 71 if (testCase == TestMethods.EXACT_INVOKER || testCase == TestMethods.INVOKER) { > 72 // Invokers aren't collected. > 73 return; > 74 } > > Can you just filter those test cases out in the main method within EnumSet.complementOf? > > On Dec 23, 2014, at 1:40 PM, Vladimir Ivanov wrote: > >> Spotted some more problems: >> - need to skip identity operations (identity_* LambdaForms) in the test, since corresponding LambdaForms reside in a permanent cache; >> > 82 mtype = adapter.type(); > 83 if (mtype.parameterCount() == 0) { > 84 // Ignore identity_* LambdaForms. > 85 return; > 86 } > > Under what conditions does this arise? i guess it might be non-determinisitic based on the randomly generated arity for the test case, so could filter more tests than absolutely required? > > >> - need to keep original test data for diagnostic purposes, since getTestCaseData() produces new instance. >> > 78 adapter = getTestMethod().getTestCaseMH(data, TestMethods.Kind.ONE); > > > Could replace "getTestMethod()" with "testCase". > > Paul. > >> Updated version: >> http://cr.openjdk.java.net/~vlivanov/8067344/webrev.01/ >> >> Best regards, >> Vladimir Ivanov >> >> On 12/22/14 11:53 PM, Vladimir Ivanov wrote: >>> http://cr.openjdk.java.net/~vlivanov/8067344/webrev.00/ >>> https://bugs.openjdk.java.net/browse/JDK-8067344 >>> >>> LFGarbageCollectedTest should be adjusted after JDK-8057020. >>> >>> There are a couple of problems with the test. >>> >>> (1) Existing logic to test that LambdaForm instance is collected isn't >>> stable enough. Consequent System.GCs can hinder reference enqueueing. >>> To speed up the test, I added -XX:SoftRefLRUPolicyMSPerMB=0 and limited >>> the heap by -Xmx64m. >>> >>> (2) MethodType-based invoker caches are deliberately left strongly >>> reachable. So, they should be skipped in the test. >>> >>> (3) Added additional diagnostic output to simplify failure analysis >>> (test case details, method handle type and LambdaForm, heap dump >>> (optional, -DHEAP_DUMP=true)). >>> >>> Testing: failing test. >>> >>> Thanks! >>> >>> Best regards, >>> Vladimir Ivanov >> _______________________________________________ >> mlvm-dev mailing list >> mlvm-dev at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev From tristan.yan at oracle.com Fri Jan 9 04:50:34 2015 From: tristan.yan at oracle.com (Tristan Yan) Date: Thu, 8 Jan 2015 20:50:34 -0800 Subject: RFR: JDK-8051563: Convert JAXP function tests in xslt components: : org.apache.qetest.dtm package, org.apache.qetest.trax package to openjdk In-Reply-To: <54AF0AC2.10206@oracle.com> References: <0B695EBE-5E9C-4B2A-B166-0BFF8644A8A0@oracle.com> <54A6E5F3.7070900@oracle.com> <55271FB0-8179-4545-BF3D-4098C9808B65@oracle.com> <637AA2A0-8BD2-4B45-8697-2E556F372041@oracle.com> <54AC51FD.7040808@oracle.com> <078315A4-D463-4891-99CE-6A1785BB5D49@oracle.com> <4B99A07A-A547-4CB5-8E6E-98EA57762906@oracle.com> <54AC6161.1000308@oracle.com> <1716E308-BE26-4EF6-9C48-FF28D5A640B0@oracle.com> <54AF0AC2.10206@oracle.com> Message-ID: <899B072B-4AE1-451D-B631-B7A2459A1B24@oracle.com> Thanks Joe and Lance. do you mind sponsor this for me. Tristan > On Jan 8, 2015, at 2:54 PM, huizhe wang wrote: > > Thanks for the update. I think the webrev is ready for putback. > > Best, > Joe > > On 1/7/2015 9:41 PM, Tristan Yan wrote: >> Hi Joe/Lance and others. >> >> Please review updated version for this one. >> >> http://cr.openjdk.java.net/~tyan/8051563/webrev.01/ >> >> Thank you >> Tristan >> >>> On Jan 6, 2015, at 2:27 PM, huizhe wang > wrote: >>> >>> >>> On 1/6/2015 2:25 PM, Lance Andersen wrote: >>>> One more thing :-) >>>> >>>> Remember to update your copyright year to 2015 also >>> >>> Indeed, that applies to my webrevs as well :-) >>> >>> Joe >>> >>>> >>>> Best >>>> Lance >>>> On Jan 6, 2015, at 5:04 PM, Lance Andersen > wrote: >>>> >>>>> >>>>> On Jan 6, 2015, at 4:31 PM, Tristan Yan > wrote: >>>>> >>>>>> Thank you Lance and Joe. >>>>> >>>>> You are very welcome. >>>>>> I am working on the fix. >>>>> >>>>> No rush from my perspective, have a lot to keep me busy in the interim before your next webrev . :-) >>>>>> it may take a couple of days that the code needs some refactor. >>>>> Understand, as I have been working on tests for RowSets, I have continued to play the refactor dance myself. >>>>> >>>>> Best, >>>>> Lance >>>>>> I will send out next review once I finish it. >>>>>> Thanks >>>>>> Tristan >>>>>> >>>>>>> On Jan 6, 2015, at 1:22 PM, huizhe wang > wrote: >>>>>>> >>>>>>> Thanks for taking the initiative and effort to refactor and clean up all of the Functional tests in previous changeset! >>>>>>> >>>>>>> We've gone through several iterations for the new ones (xslt tests). I totally agree with Lance, it looks very good overall, a lot better than its original form. >>>>>>> >>>>>>> It's nice to group all of the tests that required file access, that prepares all other tests to run with minimal permission for a (future) secure-test-target. Unit tests might need a similar treatment, but no pressure to do now :-) >>>>>>> >>>>>>> A minor comment on utility classes: there's JAXPTestUtilities and TestUtils. The former is an util for all tests while the later contained a couple of SAX handlers, it may make sense to move them into their own classes just as the other Handlers. The constants (XML_DIR and GOLDEN_DIR) then can be declared in a base class for the SAX tests. >>>>>>> >>>>>>> I understand each group of tests have their own XML source and golden files, thus XML_DIR and GOLDEN_DIR. It may be possible to add a base for each group while put test.src and test.classes into the very base class for all tests. So in general, we would have: >>>>>>> TestBase FileTestBase >>>>>>> Base for each group (e.g. SAXTestBase, DOMTestBase, XSLTTestBase...) that extends either TestBase or FileTestBase >>>>>>> >>>>>>> I remember there were a few tests that required a http server. So then we may need a HttpTestBase as well. >>>>>>> >>>>>>> I know this is not what we've discussed (or planned) previously. But since you've done a great job to incorporate what were previously quite separate test suites into one whole test suite. I can't resist to ask a bit more. Don't get me wrong, what you've done exceeded my expectation in a big way (only a month ago, we were still talking about quick/straight conversion so that you can start to take care of the new features)! >>>>>>> >>>>>>> SAXParserTest: I noticed Old: testParse11, testParse27 --> New: testParse05. So is testParse03 a new test? I can see SAXException is expected, but not IOE. In fact, this shows that the JAXP spec missed defining how empty string shall be handled (should have been an IAE). >>>>>>> >>>>>>> Best, >>>>>>> Joe >>>>>>> >>>>>>> On 1/6/2015 11:33 AM, Lance Andersen wrote: >>>>>>>> Hi Tristan, >>>>>>>> >>>>>>>> Sorry for the delay but with the holidays and the number of files, it took a while to go through :-) >>>>>>>> >>>>>>>> Overall, it looks pretty good. >>>>>>>> >>>>>>>> A couple of suggestions, but I would not necessarily hold back from committing: >>>>>>>> >>>>>>>> - For tests where you are not caring about the actual exception that is thrown to indicate a failure, such as in DocumentBuilderFactory01.java and testCheckSchemaSupport1, I would just have the method declaration use "throws Exception" vs list all of the possible individual Exceptions, as it keeps it more compact. Glad to see you are not using failUnexcepted() now. >>>>>>>> >>>>>>>> - In test classes such as in testCheckSchemaSupport3. java and DocumentBuilderImpl01.java, some tests do not use assertXXX or expect an Exception. Would be good at least to document what could cause a failure or make it clear the expected behavior of the test for passing. >>>>>>>> >>>>>>>> -SAXParserTest02.java and other tests where you get a reader/parser such as testXMLReader01, I would at least assert that null is not returned where as it is now, you only validate that an exception is not returned >>>>>>>> >>>>>>>> - I know you are porting existing tests, but I would consider consolidating tests as it seems like there is not a real reason to have a testNG class with just 1 test. I would group the like tests (such as SAXTFactoryTest ) in one testNG test class as that allows you to streamline further. >>>>>>>> >>>>>>>> - TfClearParamTest.java (as and example) minor nit that you have your @Before/AfterGroups method in between tests. I would suggest grouping all methods such as this DataProviders before or after the actual tests for better organization >>>>>>>> >>>>>>>> - TraxSAXWrapper.java, not sure I understand the "Sorry I could not resist comment" >>>>>>>> >>>>>>>> - TransformerHandlerAPITest.java has typos in comments: "IllegalArgumentExceptionis thrown?." >>>>>>>> >>>>>>>> - Minitest.java, I would add a comment for your Data Provider >>>>>>>> >>>>>>>> Best, >>>>>>>> Lance >>>>>>>> >>>>>>>> On Jan 2, 2015, at 1:49 PM, Tristan Yan > wrote: >>>>>>>> >>>>>>>>> Hi Joe and Lance >>>>>>>>> Sorry for my late reply. I just uploaded the new webrev which is trying to limit minimal permissions for most of tests. The new changeset has two base classes; class JAXPBaseTest has only minimal set permissions; class JAXPFileBaseTest adds two permissions that need access local files in the directory directory test.src and test.classes. Most of tests only inherit JAXPBaseTest that provides only minimal set permissions. Some tests will inherit JAXPFileBaseTest because tests need access local files. >>>>>>>>> Please help to review the code. >>>>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ >>>>>>>>> >>>>>>>>> Thank you >>>>>>>>> Tristan >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Jan 2, 2015, at 10:39 AM, huizhe wang > wrote: >>>>>>>>>> >>>>>>>>>> Lance, >>>>>>>>>> >>>>>>>>>> Tristan is looking into adding an extension base class for about 60 tests that require file permission, then the current base class would indeed set "minimal" permission. So please wait for his update :-) >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Joe >>>>>>>>>> >>>>>>>>>> On 12/30/2014 3:07 PM, Lance @ Oracle wrote: >>>>>>>>>>> Hi Tristan, >>>>>>>>>>> >>>>>>>>>>> I will look at this but doubt I will get to this tomorrow >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Lance >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>>>>>> Oracle Java Engineering >>>>>>>>>>> 1 Network Drive >>>>>>>>>>> Burlington, MA 01803 >>>>>>>>>>> Lance.Andersen at oracle.com >>>>>>>>>>> Sent from my iPad >>>>>>>>>>> >>>>>>>>>>> On Dec 30, 2014, at 5:28 PM, Tristan Yan > wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi All >>>>>>>>>>>> >>>>>>>>>>>> Can I get your review on this change. >>>>>>>>>>>> >>>>>>>>>>>> http://cr.openjdk.java.net/~tyan/8051563/webrev.00/ > >>>>>>>>>>>> >>>>>>>>>>>> These fixes originally come from bug https://bugs.openjdk.java.net/browse/JDK-8051563 > as part of our XML test colocation work. ThIS change-set mainly covers tests for package org.apache.qetest.dtm and org.apache.qetest.trax. >>>>>>>>>>>> >>>>>>>>>>>> In the meantime I took steps at fixing some of our existed test code on below issues: >>>>>>>>>>>> >>>>>>>>>>>> 1. Add a base test class for all functional tests that enable security manager running. A limited minimal permissions set have been set for every test. >>>>>>>>>>>> 2. Remove all unnecessary exception capture for functional tests that we?re using testng to handle all the exceptions. >>>>>>>>>>>> 3. Use try-resource block to solve all possible resource leaks (including InputStream, OutputStream, Writer, Reader). >>>>>>>>>>>> >>>>>>>>>>>> Thanks a lot. >>>>>>>>>>>> Tristan >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>>> Oracle Java Engineering >>>>>>>> 1 Network Drive >>>>>>>> Burlington, MA 01803 >>>>>>>> Lance.Andersen at oracle.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>> Oracle Java Engineering >>>>> 1 Network Drive >>>>> Burlington, MA 01803 >>>>> Lance.Andersen at oracle.com >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>> Oracle Java Engineering >>>> 1 Network Drive >>>> Burlington, MA 01803 >>>> Lance.Andersen at oracle.com >>>> >>>> >>>> >>> >> > From peter.levart at gmail.com Fri Jan 9 10:16:01 2015 From: peter.levart at gmail.com (Peter Levart) Date: Fri, 09 Jan 2015 11:16:01 +0100 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table with wrong capacity In-Reply-To: <7b2072fe0674e465f86518a9f25cef6d@sonic.net> References: <7b2072fe0674e465f86518a9f25cef6d@sonic.net> Message-ID: <54AFAA61.30008@gmail.com> On 01/05/2015 05:52 PM, Mike Duigou wrote: > Well spotted Peter. The change looks good though I wonder if it should > be: > > int length = (int)((elements + elements / 20) / loadFactor) + 3; > > FYI, regarding Daniel's suggestion: When similar invariant checks were > added to the HashMap deserialization method we found code which relied > upon the illegal values. In some cases the serialized HashMaps were > created by alternative serialization implementations which included > illegal, but until the checks were added, "harmless" values. > > The invariant checks should still be added though. It might even be > worth adding checks that the other deserialized values are in valid > ranges. > > Mike Hi Mike and others, Yes, your suggested length computation is more in "spirit" with the comment above that states: "Compute new length with a bit of room 5% to grow...", since it takes loadFactor into account for that additional 5% too. I changed it to your suggested expression. Here's new webrev: http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.02/ In addition to valid loadFactor, # of elements is checked to be non-negative. The original length is just "repaired" if it appears to be less than the enforced auto-growth invariant of Hashtable. I also expanded the test to exercise more combinations of # of elements and loadFactor. Here's what gets printed with current Hashtable implementation: ser. deser. size load lentgh length valid range ok? ------- ----- ------- ------- ----------------- ------ 10 0.15 127 4 67... 134 NOT-OK 10 0.50 31 8 21... 42 NOT-OK 10 0.75 15 10 14... 28 NOT-OK 10 1.00 15 13 11... 22 OK 10 2.50 7 7 5... 10 OK 50 0.15 511 12 334... 668 NOT-OK 50 0.50 127 30 101... 202 NOT-OK 50 0.75 127 42 67... 134 NOT-OK 50 1.00 63 55 51... 102 OK 50 2.50 31 31 21... 42 OK 500 0.15 4095 103 3334... 6668 NOT-OK 500 0.50 1023 278 1001... 2002 NOT-OK 500 0.75 1023 403 667... 1334 NOT-OK 500 1.00 511 511 501... 1002 OK 500 2.50 255 255 201... 402 OK 5000 0.15 65535 1003 33334... 66668 NOT-OK 5000 0.50 16383 2753 10001... 20002 NOT-OK 5000 0.75 8191 4003 6667... 13334 NOT-OK 5000 1.00 8191 5253 5001... 10002 OK 5000 2.50 2047 2047 2001... 4002 OK With patched Hashtable, the test passes: ser. deser. size load lentgh length valid range ok? ------- ----- ------- ------- ----------------- ------ 10 0.15 127 69 67... 134 OK 10 0.50 31 23 21... 42 OK 10 0.75 15 15 14... 28 OK 10 1.00 15 13 11... 22 OK 10 2.50 7 7 5... 10 OK 50 0.15 511 349 334... 668 OK 50 0.50 127 107 101... 202 OK 50 0.75 127 71 67... 134 OK 50 1.00 63 55 51... 102 OK 50 2.50 31 23 21... 42 OK 500 0.15 4095 3501 3334... 6668 OK 500 0.50 1023 1023 1001... 2002 OK 500 0.75 1023 703 667... 1334 OK 500 1.00 511 511 501... 1002 OK 500 2.50 255 213 201... 402 OK 5000 0.15 65535 35003 33334... 66668 OK 5000 0.50 16383 10503 10001... 20002 OK 5000 0.75 8191 7003 6667... 13334 OK 5000 1.00 8191 5253 5001... 10002 OK 5000 2.50 2047 2047 2001... 4002 OK Regards, Peter > > On 2015-01-05 07:48, core-libs-dev-request at openjdk.java.net wrote: >> >> Today's Topics: >> >> 2. Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes >> table with wrong capacity (Daniel Fuchs) > > >> Message: 2 >> Date: Mon, 05 Jan 2015 15:52:55 +0100 >> From: Daniel Fuchs >> To: Peter Levart , core-libs-dev >> >> Subject: Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes >> table with wrong capacity >> Message-ID: <54AAA547.8070804 at oracle.com> >> Content-Type: text/plain; charset=utf-8; format=flowed >> >> On 04/01/15 18:58, Peter Levart wrote: >>> Hi, >>> >>> While investigating the following issue: >>> >>> https://bugs.openjdk.java.net/browse/JDK-8029891 >>> >>> I noticed there's a bug in deserialization code of java.util.Hashtable >>> (from day one probably): >>> >>> https://bugs.openjdk.java.net/browse/JDK-8068427 >>> >>> The fix is a trivial one-character replacement: '*' -> '/', but I also >>> corrected some untruthful comments in the neighbourhood (which might >>> have been true from day one, but are not any more): >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ >>> >> >> Hi Peter, >> >> I wonder whether there should be a guard against loadFactor being >> zero/negative/NaN after line 1173, like in the constructor e.g. as >> in lines >> >> 188 if (loadFactor <= 0 || Float.isNaN(loadFactor)) >> 189 throw new IllegalArgumentException("Illegal Load: >> "+loadFactor); >> >> if only to avoid division by zero. >> >> best regards, >> >> -- daniel >> >> >>> >>> >>> Regards, Peter >>> >> > From sean.coffey at oracle.com Fri Jan 9 11:07:04 2015 From: sean.coffey at oracle.com (=?UTF-8?B?U2XDoW4gQ29mZmV5?=) Date: Fri, 09 Jan 2015 11:07:04 +0000 Subject: RFR : 8066612: Add a test that will call getDeclaredFields() on all classes and try to set them accessible. In-Reply-To: <54AE9A58.70607@oracle.com> References: <54AE970D.4020805@oracle.com> <54AE9A58.70607@oracle.com> Message-ID: <54AFB658.1030407@oracle.com> Thanks for reviewing Daniel. I'm mark you as author. If I can get a jdk7u Reviewer to review, I'll push this. Will update copyright before pushing also. regards, Sean. On 08/01/15 14:55, Daniel Fuchs wrote: > Hi Se?n, > > Thanks for taking care of that :-) > > It looks good to me. You might want to set the copyright year to 2015. > Best regards, > > -- daniel > > On 08/01/15 15:41, Se?n Coffey wrote: >> I've taken suggested test code from Daniel and am looking to backport >> 8066612 to jdk7u-dev. The test differs slightly in that it used the >> non-lambda approach (ClassNameListBuilder) >> >> I've reduced the number of run counts also (one ovm run for secure and >> one for non-secure mode) - As a result and given the large number of >> classes loaded, I've bumped PermSize up to 128m - no harm to stress that >> side of things also I guess. >> >> Test passes on all JPRT platforms. I'll revert to smaller class load >> counts if issues are seen with this test approach. >> >> webrev : >> http://cr.openjdk.java.net/~coffeys/webrev.8066612.jdk7u/webrev/ >> bug ID : https://bugs.openjdk.java.net/browse/JDK-8066612 >> >> regards, >> Sean. >> >> > From magnus.ihse.bursie at oracle.com Fri Jan 9 14:45:33 2015 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Fri, 09 Jan 2015 15:45:33 +0100 Subject: RFR: JDK-8042707: Source changes needed to build JDK 9 with Visual Studio 2013 (VS2013) In-Reply-To: <54AE9A6D.2030101@oracle.com> References: <54AE9A6D.2030101@oracle.com> Message-ID: <54AFE98D.3070105@oracle.com> On 2015-01-08 15:55, Erik Joelsson wrote: > Hello, > > Please review this patch, which adds support for building with > different versions of Visual Studio and in particular adds support for > VS2013. In order to control which version to use, I've introduced a > new configure parameter "--with-toolchain-version". On windows, this > parameter will have the valid values 2010, 2012 and 2013. The default > is still 2010. Note that 2012 was added for convenience, but has not > been tested to actually work. The longer term goal is to switch the > official compiler used for JDK 9 to VS2013. This is just the first step. > > The main difference that needed to be addressed was that > _STATIC_CPPLIB is no longer supported since VS2012, so we will now > have to bundle another runtime dll (MSVCP). Also, since the build > needs to be compatible with multiple versions of VS, we can no longer > hard code the name msvcr100.dll. I changed the names of the dlls into > macros that get added to the preprocessor command line. Note that the > implementation for msvcp*.dll in java_md.c could perhaps be more > elegant. I'm not familiar enough with the APIs, but if someone would > like to improve on it, please do. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8042707 > Webrevs: > http://cr.openjdk.java.net/~erikj/8042707/webrev.root.01/ > http://cr.openjdk.java.net/~erikj/8042707/webrev.jdk.01/ It looks basically good to me. Some remarks on toolchain_windows.m4, though. In TOOLCHAIN_CHECK_POSSIBLE_VISUAL_STUDIO_ROOT: # TODO: improve detection for other versions of VS This seems to be done now, so perhaps this can be removed :) --- Why is "vs_version" lowercase? It might be better to have local variables in lower case (or not, I'm not sure we are consistent on this?) but this breaks with the rest of the variables in the function and looks strange, like it was intentionally signalling something. (This goes for the vs_version in the other functions as well.) --- # Work around the insanely named ProgramFiles(x86) env variable PROGRAMFILES_X86="`env | $SED -n 's/^ProgramFiles(x86)=//p'`" Yay! :-) I spent some time going crazy about that one before I gave up. Never thought of that solution. --- # FIXME will just assume default Visual Studio version if test "x$with_tools_dir" != x; then TOOLCHAIN_CHECK_POSSIBLE_VISUAL_STUDIO_ROOT([$with_tools_dir/../..], This seems broken. Have you tested it? You have to pass a version as the first argument to TOOLCHAIN_CHECK_POSSIBLE_VISUAL_STUDIO_ROOT, yes? I think we need to examine an explicitely given toolchain to have it's version determined. Otherwise, the msvcr/msvcp handling code will not function correctly. I suggest that we first check if --with-toolchain-version is set and if so, respect it. Otherwise, check the path given for the known names (VS_VS_INSTALLDIR_*), and if none matches, abort and complain that version must be specified. Hm, actually, maybe the entire block of testing with_tools_dir should be lifted up into TOOLCHAIN_FIND_VISUAL_STUDIO and handled separately..? It doesn't really fit into TOOLCHAIN_FIND_VISUAL_STUDIO_BAT_FILE anyhow. /Magnus From claes.redestad at oracle.com Fri Jan 9 15:27:05 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Fri, 09 Jan 2015 16:27:05 +0100 Subject: RFR: 8068498: Remove constructor dependency on line.separator from PrintWriter and BufferedWriter Message-ID: <54AFF349.6060503@oracle.com> Hi, please review this change to remove the dependency on the line.separator property at object construction time by instead using the System.lineSeparator() utility. In itself, this change will break some applications manipulating the line.separator property to change the per-instance behavior, which is an unspecified implementation behavior. This was partially discussed and suggested earlier on this list[1], and the intent is to provide a well-specified method of overriding the line separator for a PrintWriter and possibly BufferedWriter as a follow-on to this. webrev: http://cr.openjdk.java.net/~redestad/8068498/webrev.01/ bug: https://bugs.openjdk.java.net/browse/JDK-8068498 Testing: JTREG java/lang java/io /Claes [1] http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-January/030530.html From lance.andersen at oracle.com Fri Jan 9 15:35:18 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 9 Jan 2015 10:35:18 -0500 Subject: RFR: 8068732, Adding Initial RowSet tests Message-ID: Hi all, Please find the webrev for adding an initial set of tests for RowSets. The webrev is at http://cr.openjdk.java.net/~lancea/8068732/webrev.00/ Best, Lance Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From Alan.Bateman at oracle.com Fri Jan 9 16:12:36 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 09 Jan 2015 16:12:36 +0000 Subject: RFR: 8068498: Remove constructor dependency on line.separator from PrintWriter and BufferedWriter In-Reply-To: <54AFF349.6060503@oracle.com> References: <54AFF349.6060503@oracle.com> Message-ID: <54AFFDF4.8070006@oracle.com> On 09/01/2015 15:27, Claes Redestad wrote: > Hi, > > please review this change to remove the dependency on the line.separator > property at object construction time by instead using the > System.lineSeparator() > utility. > > In itself, this change will break some applications manipulating the > line.separator > property to change the per-instance behavior, which is an unspecified > implementation > behavior. This was partially discussed and suggested earlier on this > list[1], and the > intent is to provide a well-specified method of overriding the line > separator for a > PrintWriter and possibly BufferedWriter as a follow-on to this. > > webrev: http://cr.openjdk.java.net/~redestad/8068498/webrev.01/ > bug: https://bugs.openjdk.java.net/browse/JDK-8068498 This looks good to me and the behavior change looks reasonable for a major release. -Alan From xueming.shen at oracle.com Fri Jan 9 16:16:43 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Fri, 09 Jan 2015 08:16:43 -0800 Subject: RFR: 8068498: Remove constructor dependency on line.separator from PrintWriter and BufferedWriter In-Reply-To: <54AFF349.6060503@oracle.com> References: <54AFF349.6060503@oracle.com> Message-ID: <54AFFEEB.9020505@oracle.com> On 1/9/15 7:27 AM, Claes Redestad wrote: > Hi, > > please review this change to remove the dependency on the line.separator > property at object construction time by instead using the > System.lineSeparator() > utility. > > In itself, this change will break some applications manipulating the > line.separator > property to change the per-instance behavior, which is an unspecified > implementation > behavior. This was partially discussed and suggested earlier on this > list[1], and the > intent is to provide a well-specified method of overriding the line > separator for a > PrintWriter and possibly BufferedWriter as a follow-on to this. > > webrev: http://cr.openjdk.java.net/~redestad/8068498/webrev.01/ > bug: https://bugs.openjdk.java.net/browse/JDK-8068498 > > Testing: JTREG java/lang java/io > > /Claes > > [1] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-January/030530.html > +1 From daniel.fuchs at oracle.com Fri Jan 9 16:56:28 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Fri, 09 Jan 2015 17:56:28 +0100 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() Message-ID: <54B0083C.1090409@oracle.com> Hi, Please find below a webrev for: 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() https://bugs.openjdk.java.net/browse/JDK-8068730 The java.time.Clock.system() method (and variants thereof) are specified to "obtain a clock that returns the current instant using best available system clock". However the current implementation of the clock returned is based on System.currentTimeMillis() whereas the underlying native clock used by System.currentTimeMillis() has often a greater precision than milliseconds (for instance, on Linux, System.currentTimeMillis() is based on gettimeofday, which offers microseconds precision). This patch enhances the implementation of the java.time.Clock.SystemClock, so that it offer at least the same precision than the underlying clock available on the system. There is no change in the public API. http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.00/ Some more details on the patch: native (hotspot) side: - adds a new method to the os class: os::javaTimeSystemUTC(jlong &seconds, jlong &nanos) which allows to get the time in the form of a number of seconds and number of nanoseconds (see os.hpp and various os_.cpp files) - adds a JVM_GetNanoTimeAdjustment method, which takes an offset (in seconds) as parameter and returns a nano time adjustment compared to the offset. Calls os::javaTimeSystemUTC to compute the adjustment (see jvm.cpp) java (jdk) side: - adds a native sun.misc.VM.getNanoTimeAdjustment method (which is bound to JVM_GetNanoTimeAdjustment) (see VM.java and VM.c) - modifies java.time.Clock.SystemClock to call the new sun.misc.VM.getNanoTimeAdjustment instead of System.currentTimeMillis. - fixes java.util.Formatter - which wasn't expecting greater than millisecond precision. testing: - A new test is added to test sun.misc.VM.getNanoTimeAdjustment In particular it checks the edge cases. - New tests are added to TestClock_System.java to check for the increased precision. There is also a test for the edge cases there. - Some java.time tests where tripped by the increased precision, and had to be modified to take that into account Note: comparing the new os::javaTimeSystemUTC with os::javaTimeMillis can help in reviewing the changes. best regards, -- daniel From ecki at zusammenkunft.net Fri Jan 9 17:25:59 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 9 Jan 2015 18:25:59 +0100 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B0083C.1090409@oracle.com> References: <54B0083C.1090409@oracle.com> Message-ID: <20150109182559.00004552.ecki@zusammenkunft.net> Hello Daniel, this is good news. I do wonder: is there a plan to make this an intrinsic(+vsyscall on Linux) just like System.currentTimeMillis()? Because using it for high precision timestamps would only make sense if it is similar lightweight. Greetings Bernd Am Fri, 09 Jan 2015 17:56:28 +0100 schrieb Daniel Fuchs : > Hi, > > Please find below a webrev for: > > 8068730: Increase the precision of the implementation > of java.time.Clock.systemUTC() > https://bugs.openjdk.java.net/browse/JDK-8068730 > > The java.time.Clock.system() method (and variants thereof) are > specified to "obtain a clock that returns the current instant > using best available system clock". However the current > implementation of the clock returned is based on > System.currentTimeMillis() whereas the underlying native clock > used by System.currentTimeMillis() has often a greater precision > than milliseconds (for instance, on Linux, System.currentTimeMillis() > is based on gettimeofday, which offers microseconds precision). > > This patch enhances the implementation of the > java.time.Clock.SystemClock, so that it offer at least the > same precision than the underlying clock available on the system. > > There is no change in the public API. > > http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.00/ > > Some more details on the patch: > > native (hotspot) side: > > - adds a new method to the os class: > os::javaTimeSystemUTC(jlong &seconds, jlong &nanos) > which allows to get the time in the form of a number > of seconds and number of nanoseconds > (see os.hpp and various os_.cpp files) > > - adds a JVM_GetNanoTimeAdjustment method, which takes > an offset (in seconds) as parameter and returns a > nano time adjustment compared to the offset. > Calls os::javaTimeSystemUTC to compute the adjustment > (see jvm.cpp) > > java (jdk) side: > > - adds a native sun.misc.VM.getNanoTimeAdjustment method > (which is bound to JVM_GetNanoTimeAdjustment) > (see VM.java and VM.c) > > - modifies java.time.Clock.SystemClock to call the new > sun.misc.VM.getNanoTimeAdjustment instead of > System.currentTimeMillis. > > - fixes java.util.Formatter - which wasn't expecting > greater than millisecond precision. > > testing: > > - A new test is added to test sun.misc.VM.getNanoTimeAdjustment > In particular it checks the edge cases. > - New tests are added to TestClock_System.java to check for > the increased precision. > There is also a test for the edge cases there. > - Some java.time tests where tripped by the increased precision, > and had to be modified to take that into account > > Note: comparing the new os::javaTimeSystemUTC with os::javaTimeMillis > can help in reviewing the changes. > > best regards, > > -- daniel From daniel.fuchs at oracle.com Fri Jan 9 17:46:45 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Fri, 09 Jan 2015 18:46:45 +0100 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <20150109182559.00004552.ecki@zusammenkunft.net> References: <54B0083C.1090409@oracle.com> <20150109182559.00004552.ecki@zusammenkunft.net> Message-ID: <54B01405.1070001@oracle.com> On 09/01/15 18:25, Bernd Eckenfels wrote: > Hello Daniel, > > this is good news. I do wonder: is there a plan to make this an > intrinsic(+vsyscall on Linux) just like System.currentTimeMillis()? I guess it is a possibility if there is a need, but not in this changeset, and not by me, as I wouldn't know where to start ;-) For the record I have done some microbenchmarking and the performance of the new implementation does not seem to be very far from the old. I have compared Instant.ofEpochMilli(System.currentTimeMillis()) with systemUTC.instant() on my machine: t.j.t.ClockBenchmark.testInstantOfEpochMillis thrpt 40 22610258.156 ? 209829.169 ops/s t.j.t.ClockBenchmark.testSystemUTC thrpt 40 18569691.759 ? 160916.846 ops/s best regards, -- daniel > > Because using it for high precision timestamps would only make sense if > it is similar lightweight. > > Greetings > Bernd > > > Am Fri, 09 Jan 2015 > 17:56:28 +0100 schrieb Daniel Fuchs : > >> Hi, >> >> Please find below a webrev for: >> >> 8068730: Increase the precision of the implementation >> of java.time.Clock.systemUTC() >> https://bugs.openjdk.java.net/browse/JDK-8068730 >> >> The java.time.Clock.system() method (and variants thereof) are >> specified to "obtain a clock that returns the current instant >> using best available system clock". However the current >> implementation of the clock returned is based on >> System.currentTimeMillis() whereas the underlying native clock >> used by System.currentTimeMillis() has often a greater precision >> than milliseconds (for instance, on Linux, System.currentTimeMillis() >> is based on gettimeofday, which offers microseconds precision). >> >> This patch enhances the implementation of the >> java.time.Clock.SystemClock, so that it offer at least the >> same precision than the underlying clock available on the system. >> >> There is no change in the public API. >> >> http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.00/ >> >> Some more details on the patch: >> >> native (hotspot) side: >> >> - adds a new method to the os class: >> os::javaTimeSystemUTC(jlong &seconds, jlong &nanos) >> which allows to get the time in the form of a number >> of seconds and number of nanoseconds >> (see os.hpp and various os_.cpp files) >> >> - adds a JVM_GetNanoTimeAdjustment method, which takes >> an offset (in seconds) as parameter and returns a >> nano time adjustment compared to the offset. >> Calls os::javaTimeSystemUTC to compute the adjustment >> (see jvm.cpp) >> >> java (jdk) side: >> >> - adds a native sun.misc.VM.getNanoTimeAdjustment method >> (which is bound to JVM_GetNanoTimeAdjustment) >> (see VM.java and VM.c) >> >> - modifies java.time.Clock.SystemClock to call the new >> sun.misc.VM.getNanoTimeAdjustment instead of >> System.currentTimeMillis. >> >> - fixes java.util.Formatter - which wasn't expecting >> greater than millisecond precision. >> >> testing: >> >> - A new test is added to test sun.misc.VM.getNanoTimeAdjustment >> In particular it checks the edge cases. >> - New tests are added to TestClock_System.java to check for >> the increased precision. >> There is also a test for the edge cases there. >> - Some java.time tests where tripped by the increased precision, >> and had to be modified to take that into account >> >> Note: comparing the new os::javaTimeSystemUTC with os::javaTimeMillis >> can help in reviewing the changes. >> >> best regards, >> >> -- daniel > From scolebourne at joda.org Fri Jan 9 18:26:51 2015 From: scolebourne at joda.org (Stephen Colebourne) Date: Fri, 9 Jan 2015 18:26:51 +0000 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B0083C.1090409@oracle.com> References: <54B0083C.1090409@oracle.com> Message-ID: Well that is a nice surprise ;-) And I think the implementation via an adjustment is very sensible. In java.time.Instant, a readObject() has been added. However, I don't believe this will ever be called because Instant has a writeReplace() method and so is not deserialized. (There may be some security related "evil serialization stream" reason why readObject() should exist, I can't say). In java.util.Formatter, there should be no reason to retain use of the MILLI_OF_SECOND, as any TemporalAccessor that can return NANO_OF_SECOND should also return MILLI_OF_SECOND, however the spec is not quite tight enough to guarantee that sadly. As such, I think the catch will have to be retained. TestFormatter has some commented out System.out statements that should probably be removed. I suspect that this change will break some user code, but it certainly doesn't break the spec. As such, I think the change should go in. I do believe that this change means that a new method should be added to Clock however: public static Clock tickMillis(ZoneId zone) { return new TickClock(system(zone), NANOS_PER_MILLI); } While this can be achieved without a new method, the API would feel slightly strange without it now better-than-milli clocks exist. I recommend raising a separate RFE to track this. Stephen On 9 January 2015 at 16:56, Daniel Fuchs wrote: > Hi, > > Please find below a webrev for: > > 8068730: Increase the precision of the implementation > of java.time.Clock.systemUTC() > https://bugs.openjdk.java.net/browse/JDK-8068730 > > The java.time.Clock.system() method (and variants thereof) are > specified to "obtain a clock that returns the current instant > using best available system clock". However the current > implementation of the clock returned is based on > System.currentTimeMillis() whereas the underlying native clock > used by System.currentTimeMillis() has often a greater precision > than milliseconds (for instance, on Linux, System.currentTimeMillis() > is based on gettimeofday, which offers microseconds precision). > > This patch enhances the implementation of the > java.time.Clock.SystemClock, so that it offer at least the > same precision than the underlying clock available on the system. > > There is no change in the public API. > > http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.00/ > > Some more details on the patch: > > native (hotspot) side: > > - adds a new method to the os class: > os::javaTimeSystemUTC(jlong &seconds, jlong &nanos) > which allows to get the time in the form of a number > of seconds and number of nanoseconds > (see os.hpp and various os_.cpp files) > > - adds a JVM_GetNanoTimeAdjustment method, which takes > an offset (in seconds) as parameter and returns a > nano time adjustment compared to the offset. > Calls os::javaTimeSystemUTC to compute the adjustment > (see jvm.cpp) > > java (jdk) side: > > - adds a native sun.misc.VM.getNanoTimeAdjustment method > (which is bound to JVM_GetNanoTimeAdjustment) > (see VM.java and VM.c) > > - modifies java.time.Clock.SystemClock to call the new > sun.misc.VM.getNanoTimeAdjustment instead of > System.currentTimeMillis. > > - fixes java.util.Formatter - which wasn't expecting > greater than millisecond precision. > > testing: > > - A new test is added to test sun.misc.VM.getNanoTimeAdjustment > In particular it checks the edge cases. > - New tests are added to TestClock_System.java to check for > the increased precision. > There is also a test for the edge cases there. > - Some java.time tests where tripped by the increased precision, > and had to be modified to take that into account > > Note: comparing the new os::javaTimeSystemUTC with os::javaTimeMillis > can help in reviewing the changes. > > best regards, > > -- daniel From brent.christian at oracle.com Fri Jan 9 18:31:52 2015 From: brent.christian at oracle.com (Brent Christian) Date: Fri, 09 Jan 2015 10:31:52 -0800 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <20150109011035.000000f5.ecki@zusammenkunft.net> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> Message-ID: <54B01E98.308@oracle.com> Hi, Bernd The initial discussion of the change that set TREEIFY_THRESHOLD to 8 (it was initially 16) can be read here: http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-July/018685.html ...continuing here... http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-August/019853.html The code review discussion is archived here: http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-August/020131.html -Brent On 1/8/15 4:10 PM, Bernd Eckenfels wrote: > Hello Vitaly, > > the TREEIFY_THRESHOLD in HashMap is 8 (6 for UNTREEIFY). Not sure if > there have been benchmarks for this, I dont see an justification in the > source comments. There is a comment, that it is expected to waste a > factor of two time and space when not compareable. > > I would also expect that larger thresholds make sense (especially when key is not > compareable). > > The JEP 180 also does not mention results from the benchmarks, maybe > somebody knows details? > > http://openjdk.java.net/jeps/180 > > Gruss > Bernd > > > Am Thu, 8 Jan 2015 18:38:53 -0500 > schrieb Vitaly Davidovich : > >> Hmmm, 15 entries doesn't seem like a large enough number to drop >> linear search. I'd have expected something like 60-80 entries (in >> some binary vs linear search benchmarks I've come across, that seems >> to be crossover point). It's hard to beat linear search for small >> sets when the comparison function is dirt cheap. >> >> Sent from my phone >> On Jan 8, 2015 6:25 PM, "Bernd Eckenfels" >> wrote: >> >>> Hello Peter, >>> >>> hm not sure what you mean, i was not suggesting the regression is >>> caused by simpler hashCode bits. (do you mean my comment about the >>> removed randomizer? that is not a problem for the benchmark, but it >>> might be an opportunity for DOS (again)). >>> >>> I think the regression itself is caused by the fact that a tree is >>> used instead of the linear search. The benchmark does colide >>> heavily, but only places 15 or so entries in one bucket. This is >>> enough to trigger the migration from linear search to tree, but not >>> enough to hurt for linear search performance (at least I think this >>> is the case). >>> >>> The way the nodes are constructed they actually do sort pretty good >>> if compareable is implemented. It would most likely not help if >>> Compareable cannot distinguish more than hashCode would. >>> >>> Gruss >>> Bernd >>> >>> >>> >>> Am Thu, 08 Jan 2015 23:10:10 +0100 >>> schrieb Peter Levart : >>> >>>> Bernd, >>>> >>>> I tried to change the "comparableClassFor" myself and it didn't >>>> work (HashMap is used very early in boot-up sequence and >>>> initializing ClassValue at that time triggers a NPE). >>>> >>>> Anyway, caching of "comparableClassFor" result would only >>>> potentially improve the "badDistWithComp" result. But can not >>>> improve "badDistNoComp" which is the one with speed regression as >>>> you're benchmark suggests. >>>> >>>> But your feeling that this is caused by "simpler" hashCode bits >>>> spreading function is not correct. I tried to replace the hash() >>>> method with the one that was in HM before and I get comparable >>>> results. This is the JDK8 HashMap.hash() method: >>>> >>>> static final int hash(Object key) { >>>> int h; >>>> return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> >>>> 16); } >>>> >>>> Benchmark (initialSize) Mode >>>> Samples Score Score error Units >>>> j.t.HashMapCollision.badDistNoComp 16 avgt 4 >>>> 3171.264 1152.995 ms/op >>>> j.t.HashMapCollision.badDistWithComp 16 avgt 4 >>>> 2819.342 422.861 ms/op >>>> j.t.HashMapCollision.goodDistNoComp 16 avgt 4 >>>> 1026.064 72.049 ms/op >>>> j.t.HashMapCollision.goodDistWithComp 16 avgt 4 >>>> 1025.312 39.858 ms/op >>>> >>>> >>>> ...and this is my re-interpretation of pre JDK8 HashMap.hash(): >>>> >>>> >>>> static final int randomHash = >>>> mix32(System.currentTimeMillis() ^ System.nanoTime()); >>>> >>>> private static int mix32(long z) { >>>> z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL; >>>> return (int)(((z ^ (z >>> 33)) * 0xc4ceb9fe1a85ec53L) >>> >>>> 32); } >>>> >>>> static final int hash(Object k) { >>>> int h = k == null ? randomHash : randomHash ^ >>>> k.hashCode(); >>>> >>>> // This function ensures that hashCodes that differ only >>>> by // constant multiples at each bit position have a bounded >>>> // number of collisions (approximately 8 at default load >>>> factor). h ^= (h >>> 20) ^ (h >>> 12); >>>> return h ^ (h >>> 7) ^ (h >>> 4); >>>> } >>>> >>>> Benchmark (initialSize) Mode >>>> Samples Score Score error Units >>>> j.t.HashMapCollision.badDistNoComp 16 avgt 4 >>>> 3257.348 1079.088 ms/op >>>> j.t.HashMapCollision.badDistWithComp 16 avgt 4 >>>> 2866.740 414.687 ms/op >>>> j.t.HashMapCollision.goodDistNoComp 16 avgt 4 >>>> 1041.068 99.370 ms/op >>>> j.t.HashMapCollision.goodDistWithComp 16 avgt 4 >>>> 1041.653 53.925 ms/op >>>> >>>> >>>> Your benchmark does not show much difference. Perhaps the >>>> regression for "badDistNoComp" case could be caused by the fact >>>> that with really bad hashCode and no Comparable interface, the >>>> red-black tree becomes less performant to search than a simple >>>> linked list of Nodes... >>>> >>>> >>>> Regards, Peter >>>> >>>> >>>> >>>> >>>> >>>> On 01/08/2015 08:41 PM, Bernd Eckenfels wrote: >>>>> Hello Peter, >>>>> >>>>> yes it is only keys without an Compareable interface, but they >>>>> are quite common. I think the main problem with the internal >>>>> comparator (based on instance identity) is, that it would work >>>>> for looking up the same instance again, but not for the case >>>>> where the actual instance is re-created (as in my example code). >>>>> >>>>> I would love to test your modified code, but I don't have a >>>>> OpenJDK build environment handy. Or actually I can try to get >>>>> one, is there somewhere a Virtulisation or Cloud Image >>>>> available which is pre-installed? >>>>> >>>>> I have (1.6) a compiled benchmark.jar here, in case anyone >>>>> wants to try it: >>>>> >>>>> >>> https://onedrive.live.com/redir?resid=A98B6F4E09966AFD!20440&authkey=!AFFk03-5jq21Xz0&ithint=file%2cjar >>>>> >>>>> BTW: I am (as usual) not expecting commoents on that, but just >>>>> to mention it: the expected good behavior of degenerated >>>>> hashmaps (due to the tree) was reason for removing the hashcode >>>>> secret randomization. I wonder if that was such a good idea if >>>>> colliding lookups (with more than a handfull of entries in a >>>>> bucket) have this 50% penalty. >>>>> >>>>> Greetings >>>>> Bernd >>>>> >>>>> >>>>> >>>>> Am Thu, 08 Jan 2015 20:22:13 +0100 >>>>> schrieb Peter Levart : >>>>> >>>>>> Hi Bernd, >>>>>> >>>>>> It seems that only bad hash codes (without comparable keys) in >>>>>> JDK8 HM are worse than JDK7 HM. >>>>>> >>>>>> >>>>>> Since you have already taken time to measure JDK7 vs JDK8 HM, >>>>>> could you try to take the JDK8 source and just replace the >>>>>> internal "comparableClassFor" method with the following >>>>>> implementation: >>>>>> >>>>>> static final ClassValue selfComparable = new >>>>>> ClassValue() { >>>>>> @Override >>>>>> protected Boolean computeValue(Class c) { >>>>>> Type[] as; ParameterizedType p; >>>>>> for (Type t : c.getGenericInterfaces()) { >>>>>> if (t instanceof ParameterizedType && >>>>>> (p = (ParameterizedType) t).getRawType() >>>>>> == Comparable.class && >>>>>> (as = p.getActualTypeArguments()).length >>>>>> == 1 && as[0] == c) // type arg is c >>>>>> return true; >>>>>> } >>>>>> return false; >>>>>> } >>>>>> }; >>>>>> >>>>>> static Class comparableClassFor(Object x) { >>>>>> if (x instanceof Comparable) { >>>>>> Class c = x.getClass(); >>>>>> if (c == String.class || selfComparable.get(c)) { >>>>>> return c; >>>>>> } >>>>>> } >>>>>> return null; >>>>>> } >>>>>> >>>>>> >>>>>> ...and retry your measurements. I just want to see if it has >>>>>> any impact. >>>>>> >>>>>> >>>>>> Thanks, Peter >>>>>> >>>>>> >>>>>> On 01/08/2015 05:38 PM, Bernd Eckenfels wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I think it was topic before, but I just wanted to point out, >>>>>>> that it is still an topic on the internetz. :) >>>>>>> >>>>>>> Motivated by a StackOverflow question regarding HashMap >>>>>>> performance regression in Java 8 >>>>>>> >>>>>>> >>> http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java-8/27760442 >>>>>>> >>>>>>> I made a JMH test and compared 7 and 8 speed. (the test is not >>>>>>> very scientific as I dont really know how to squeeze the >>>>>>> longrunning loop which alters state into the harness, but the >>>>>>> results seem to be consitent wth theory and stopwatch testing) >>>>>>> >>>>>>> https://gist.github.com/ecki/9f69773eb29428a36077 >>>>>>> >>>>>>> What can be seen is, that with a good distribution of hash >>>>>>> keys 8 looks faster than 7, and with a bad distribution of >>>>>>> hash keys Java 7 is faster (unless you supply a Comparator >>>>>>> for the key). (and a good distributed hashkey with comparable >>>>>>> seems to be a bit slower) >>>>>>> >>>>>>> I think the regression is somewhat expected, but I guess its >>>>>>> not widely known. >>>>>>> >>>>>>> (I do not use a cached hashcode, but it has a nearly trivial >>>>>>> implementation just to make it more life like. the tests also >>>>>>> compares different initial sizes, but they do not have an >>>>>>> measurable effect on the performance, I show only default size >>>>>>> below:) >>>>>>> >>>>>>> java version "1.7.0_72" >>>>>>> >>>>>>> Benchmark (initialSize) Mode Samp Score >>>>>>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt >>>>>>> 4 10847,318 ? 5596,690 ms/op >>>>>>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 >>>>>>> 10761,430 ? 5376,975 ms/op >>>>>>> n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 >>>>>>> 3613,923 ? 254,823 ms/op >>>>>>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 >>>>>>> 3656,229 ? 274,350 ms/op java version "1.8.0_25" >>>>>>> >>>>>>> Benchmark (initialSize) Mode Samp Score >>>>>>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt >>>>>>> 4 14309,880 ? 1811,709 ms/op <-slower >>>>>>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 >>>>>>> 8232,037 ? 5974,530 ms/op >>>>>>> n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 >>>>>>> 3304,698 ? 116,866 ms/op >>>>>>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 >>>>>>> 3425,762 ? 107,659 ms/op >>>>>>> >>>>>>> >>>>>>> Greetings >>>>>>> Bernd >>>> >>>> >>> >> > From ecki at zusammenkunft.net Fri Jan 9 19:22:30 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 9 Jan 2015 20:22:30 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B01E98.308@oracle.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> Message-ID: <20150109202230.00000e85.ecki@zusammenkunft.net> Hello Brent, thank you for digging that out! It describes basically what is documented in the javadoc as well. Having 8 colliding keys is regarded as extremly rare when hashCode is poision distributed. From that point of view using this threshold makes sense. However it was not checked (at least I havent seen it in the archive) to what point a linear search is faster implemented as the treeifying (especially if it is known the keys wont sort very nicely anyway). At least I could not see it in the discussions. I would think this aspect needs a bit of thought as well, now that the bit scrambeling is removed. But it is not only about attackers trying to provoke that, it seems to me (from the SO question and other discussions) that there is unfortunatelly some bad hashCode() out there. My naive suggestion would be to delay the treeifying a bit longer if the hash keys are not Compareable and the hash tables are large(er). Gruss Bernd Am Fri, 09 Jan 2015 10:31:52 -0800 schrieb Brent Christian : > Hi, Bernd > > The initial discussion of the change that set TREEIFY_THRESHOLD to 8 > (it was initially 16) can be read here: > > http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-July/018685.html > > ...continuing here... > > http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-August/019853.html > > > The code review discussion is archived here: > > http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-August/020131.html > > -Brent > > On 1/8/15 4:10 PM, Bernd Eckenfels wrote: > > Hello Vitaly, > > > > the TREEIFY_THRESHOLD in HashMap is 8 (6 for UNTREEIFY). Not sure if > > there have been benchmarks for this, I dont see an justification in > > the source comments. There is a comment, that it is expected to > > waste a factor of two time and space when not compareable. > > > > I would also expect that larger thresholds make sense (especially > > when key is not compareable). > > > > The JEP 180 also does not mention results from the benchmarks, maybe > > somebody knows details? > > > > http://openjdk.java.net/jeps/180 > > > > Gruss > > Bernd > > > > > > Am Thu, 8 Jan 2015 18:38:53 -0500 > > schrieb Vitaly Davidovich : > > > >> Hmmm, 15 entries doesn't seem like a large enough number to drop > >> linear search. I'd have expected something like 60-80 entries (in > >> some binary vs linear search benchmarks I've come across, that > >> seems to be crossover point). It's hard to beat linear search for > >> small sets when the comparison function is dirt cheap. > >> > >> Sent from my phone > >> On Jan 8, 2015 6:25 PM, "Bernd Eckenfels" > >> wrote: > >> > >>> Hello Peter, > >>> > >>> hm not sure what you mean, i was not suggesting the regression is > >>> caused by simpler hashCode bits. (do you mean my comment about the > >>> removed randomizer? that is not a problem for the benchmark, but > >>> it might be an opportunity for DOS (again)). > >>> > >>> I think the regression itself is caused by the fact that a tree is > >>> used instead of the linear search. The benchmark does colide > >>> heavily, but only places 15 or so entries in one bucket. This is > >>> enough to trigger the migration from linear search to tree, but > >>> not enough to hurt for linear search performance (at least I > >>> think this is the case). > >>> > >>> The way the nodes are constructed they actually do sort pretty > >>> good if compareable is implemented. It would most likely not help > >>> if Compareable cannot distinguish more than hashCode would. > >>> > >>> Gruss > >>> Bernd > >>> > >>> > >>> > >>> Am Thu, 08 Jan 2015 23:10:10 +0100 > >>> schrieb Peter Levart : > >>> > >>>> Bernd, > >>>> > >>>> I tried to change the "comparableClassFor" myself and it didn't > >>>> work (HashMap is used very early in boot-up sequence and > >>>> initializing ClassValue at that time triggers a NPE). > >>>> > >>>> Anyway, caching of "comparableClassFor" result would only > >>>> potentially improve the "badDistWithComp" result. But can not > >>>> improve "badDistNoComp" which is the one with speed regression as > >>>> you're benchmark suggests. > >>>> > >>>> But your feeling that this is caused by "simpler" hashCode bits > >>>> spreading function is not correct. I tried to replace the hash() > >>>> method with the one that was in HM before and I get comparable > >>>> results. This is the JDK8 HashMap.hash() method: > >>>> > >>>> static final int hash(Object key) { > >>>> int h; > >>>> return (key == null) ? 0 : (h = key.hashCode()) ^ (h > >>>> >>> 16); } > >>>> > >>>> Benchmark (initialSize) Mode > >>>> Samples Score Score error Units > >>>> j.t.HashMapCollision.badDistNoComp 16 avgt 4 > >>>> 3171.264 1152.995 ms/op > >>>> j.t.HashMapCollision.badDistWithComp 16 avgt 4 > >>>> 2819.342 422.861 ms/op > >>>> j.t.HashMapCollision.goodDistNoComp 16 avgt 4 > >>>> 1026.064 72.049 ms/op > >>>> j.t.HashMapCollision.goodDistWithComp 16 avgt 4 > >>>> 1025.312 39.858 ms/op > >>>> > >>>> > >>>> ...and this is my re-interpretation of pre JDK8 HashMap.hash(): > >>>> > >>>> > >>>> static final int randomHash = > >>>> mix32(System.currentTimeMillis() ^ System.nanoTime()); > >>>> > >>>> private static int mix32(long z) { > >>>> z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL; > >>>> return (int)(((z ^ (z >>> 33)) * 0xc4ceb9fe1a85ec53L) > >>>> >>> 32); } > >>>> > >>>> static final int hash(Object k) { > >>>> int h = k == null ? randomHash : randomHash ^ > >>>> k.hashCode(); > >>>> > >>>> // This function ensures that hashCodes that differ > >>>> only by // constant multiples at each bit position have a bounded > >>>> // number of collisions (approximately 8 at default > >>>> load factor). h ^= (h >>> 20) ^ (h >>> 12); > >>>> return h ^ (h >>> 7) ^ (h >>> 4); > >>>> } > >>>> > >>>> Benchmark (initialSize) Mode > >>>> Samples Score Score error Units > >>>> j.t.HashMapCollision.badDistNoComp 16 avgt 4 > >>>> 3257.348 1079.088 ms/op > >>>> j.t.HashMapCollision.badDistWithComp 16 avgt 4 > >>>> 2866.740 414.687 ms/op > >>>> j.t.HashMapCollision.goodDistNoComp 16 avgt 4 > >>>> 1041.068 99.370 ms/op > >>>> j.t.HashMapCollision.goodDistWithComp 16 avgt 4 > >>>> 1041.653 53.925 ms/op > >>>> > >>>> > >>>> Your benchmark does not show much difference. Perhaps the > >>>> regression for "badDistNoComp" case could be caused by the fact > >>>> that with really bad hashCode and no Comparable interface, the > >>>> red-black tree becomes less performant to search than a simple > >>>> linked list of Nodes... > >>>> > >>>> > >>>> Regards, Peter > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 01/08/2015 08:41 PM, Bernd Eckenfels wrote: > >>>>> Hello Peter, > >>>>> > >>>>> yes it is only keys without an Compareable interface, but they > >>>>> are quite common. I think the main problem with the internal > >>>>> comparator (based on instance identity) is, that it would work > >>>>> for looking up the same instance again, but not for the case > >>>>> where the actual instance is re-created (as in my example code). > >>>>> > >>>>> I would love to test your modified code, but I don't have a > >>>>> OpenJDK build environment handy. Or actually I can try to get > >>>>> one, is there somewhere a Virtulisation or Cloud Image > >>>>> available which is pre-installed? > >>>>> > >>>>> I have (1.6) a compiled benchmark.jar here, in case anyone > >>>>> wants to try it: > >>>>> > >>>>> > >>> https://onedrive.live.com/redir?resid=A98B6F4E09966AFD!20440&authkey=!AFFk03-5jq21Xz0&ithint=file%2cjar > >>>>> > >>>>> BTW: I am (as usual) not expecting commoents on that, but just > >>>>> to mention it: the expected good behavior of degenerated > >>>>> hashmaps (due to the tree) was reason for removing the hashcode > >>>>> secret randomization. I wonder if that was such a good idea if > >>>>> colliding lookups (with more than a handfull of entries in a > >>>>> bucket) have this 50% penalty. > >>>>> > >>>>> Greetings > >>>>> Bernd > >>>>> > >>>>> > >>>>> > >>>>> Am Thu, 08 Jan 2015 20:22:13 +0100 > >>>>> schrieb Peter Levart : > >>>>> > >>>>>> Hi Bernd, > >>>>>> > >>>>>> It seems that only bad hash codes (without comparable keys) in > >>>>>> JDK8 HM are worse than JDK7 HM. > >>>>>> > >>>>>> > >>>>>> Since you have already taken time to measure JDK7 vs JDK8 HM, > >>>>>> could you try to take the JDK8 source and just replace the > >>>>>> internal "comparableClassFor" method with the following > >>>>>> implementation: > >>>>>> > >>>>>> static final ClassValue selfComparable = new > >>>>>> ClassValue() { > >>>>>> @Override > >>>>>> protected Boolean computeValue(Class c) { > >>>>>> Type[] as; ParameterizedType p; > >>>>>> for (Type t : c.getGenericInterfaces()) { > >>>>>> if (t instanceof ParameterizedType && > >>>>>> (p = (ParameterizedType) t).getRawType() > >>>>>> == Comparable.class && > >>>>>> (as = p.getActualTypeArguments()).length > >>>>>> == 1 && as[0] == c) // type arg is c > >>>>>> return true; > >>>>>> } > >>>>>> return false; > >>>>>> } > >>>>>> }; > >>>>>> > >>>>>> static Class comparableClassFor(Object x) { > >>>>>> if (x instanceof Comparable) { > >>>>>> Class c = x.getClass(); > >>>>>> if (c == String.class || selfComparable.get(c)) > >>>>>> { return c; > >>>>>> } > >>>>>> } > >>>>>> return null; > >>>>>> } > >>>>>> > >>>>>> > >>>>>> ...and retry your measurements. I just want to see if it has > >>>>>> any impact. > >>>>>> > >>>>>> > >>>>>> Thanks, Peter > >>>>>> > >>>>>> > >>>>>> On 01/08/2015 05:38 PM, Bernd Eckenfels wrote: > >>>>>>> Hello, > >>>>>>> > >>>>>>> I think it was topic before, but I just wanted to point out, > >>>>>>> that it is still an topic on the internetz. :) > >>>>>>> > >>>>>>> Motivated by a StackOverflow question regarding HashMap > >>>>>>> performance regression in Java 8 > >>>>>>> > >>>>>>> > >>> http://stackoverflow.com/questions/27759527/using-java-7-hashmap-in-java-8/27760442 > >>>>>>> > >>>>>>> I made a JMH test and compared 7 and 8 speed. (the test is not > >>>>>>> very scientific as I dont really know how to squeeze the > >>>>>>> longrunning loop which alters state into the harness, but the > >>>>>>> results seem to be consitent wth theory and stopwatch testing) > >>>>>>> > >>>>>>> https://gist.github.com/ecki/9f69773eb29428a36077 > >>>>>>> > >>>>>>> What can be seen is, that with a good distribution of hash > >>>>>>> keys 8 looks faster than 7, and with a bad distribution of > >>>>>>> hash keys Java 7 is faster (unless you supply a Comparator > >>>>>>> for the key). (and a good distributed hashkey with comparable > >>>>>>> seems to be a bit slower) > >>>>>>> > >>>>>>> I think the regression is somewhat expected, but I guess its > >>>>>>> not widely known. > >>>>>>> > >>>>>>> (I do not use a cached hashcode, but it has a nearly trivial > >>>>>>> implementation just to make it more life like. the tests also > >>>>>>> compares different initial sizes, but they do not have an > >>>>>>> measurable effect on the performance, I show only default size > >>>>>>> below:) > >>>>>>> > >>>>>>> java version "1.7.0_72" > >>>>>>> > >>>>>>> Benchmark (initialSize) Mode Samp Score > >>>>>>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt > >>>>>>> 4 10847,318 ? 5596,690 ms/op > >>>>>>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 > >>>>>>> 10761,430 ? 5376,975 ms/op > >>>>>>> n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 > >>>>>>> 3613,923 ? 254,823 ms/op > >>>>>>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 > >>>>>>> 3656,229 ? 274,350 ms/op java version "1.8.0_25" > >>>>>>> > >>>>>>> Benchmark (initialSize) Mode Samp Score > >>>>>>> Error Units n.e.j.h.HashMapCollision.badDistNoComp 16 avgt > >>>>>>> 4 14309,880 ? 1811,709 ms/op <-slower > >>>>>>> n.e.j.h.HashMapCollision.badDistWithComp 16 avgt 4 > >>>>>>> 8232,037 ? 5974,530 ms/op > >>>>>>> n.e.j.h.HashMapCollision.goodDistNoComp 16 avgt 4 > >>>>>>> 3304,698 ? 116,866 ms/op > >>>>>>> n.e.j.h.HashMapCollision.goodDistWithComp 16 avgt 4 > >>>>>>> 3425,762 ? 107,659 ms/op > >>>>>>> > >>>>>>> > >>>>>>> Greetings > >>>>>>> Bernd > >>>> > >>>> > >>> > >> > > From dl at cs.oswego.edu Fri Jan 9 19:51:03 2015 From: dl at cs.oswego.edu (Doug Lea) Date: Fri, 09 Jan 2015 14:51:03 -0500 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <20150109202230.00000e85.ecki@zusammenkunft.net> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> Message-ID: <54B03127.8080706@cs.oswego.edu> On 01/09/2015 02:22 PM, Bernd Eckenfels wrote: > My naive suggestion would be to delay the treeifying a bit longer if > the hash keys are not Compareable and the hash tables are large(er). > Feel free to experiment. Bear in mind that the main rationale for treeifying HashMap and ConcurrentHashMap is that there are bad guys out there who hand-craft millions of keys with identical hashCodes to try to bring down servers. The treeified versions successfully resist this, at the expense of adding a further penalty to classes with already-terrible hashCodes. (One arguable benefit is that people who now notice this will be more motivated to fix their hashCodes.) The main penalty here vs linear lists for cases with only dozens (vs millions) of clashing elements is the call to comparableClassFor for non-Strings, which hits some reflective goop. I posted something a few years ago mentioning that this could be made faster with some other JDK support. But as Peter Levart noted, bootstrapping issues preclude the most straightforward solution. -Doug From martinrb at google.com Fri Jan 9 23:29:13 2015 From: martinrb at google.com (Martin Buchholz) Date: Fri, 9 Jan 2015 15:29:13 -0800 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B03127.8080706@cs.oswego.edu> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> Message-ID: Given the prevalence of sub-optimal hashcodes, my own intuition is also that raising the treeification threshold from 8 will be a win. But it depends on the element type - the more expensive it is to compare elements, the lower the optimal treeification threshold. String (or a wrapper around Strings) is the most common element type, and comparison of unequal Strings is usually fast because ... (wait a minute, I thought String.equals would compare hash codes, but it doesn't! but even if it did, our determined adversary has crafted String keys all of the same length and all with the same hash code and all with a long common prefix ...) Did I just talk myself into supporting Doug's choice of 8? We probably don't want to get into the business of measuring the cost of equals and compareTo... Or maybe we do, given that this overhead only comes into play when we consider treeification?! Is HashMap going to grow its very own JIT?! On Fri, Jan 9, 2015 at 11:51 AM, Doug Lea

wrote: > On 01/09/2015 02:22 PM, Bernd Eckenfels wrote: > > My naive suggestion would be to delay the treeifying a bit longer if >> the hash keys are not Compareable and the hash tables are large(er). >> >> > Feel free to experiment. Bear in mind that the main rationale for > treeifying HashMap and ConcurrentHashMap is that there are bad guys > out there who hand-craft millions of keys with identical hashCodes > to try to bring down servers. The treeified versions successfully > resist this, at the expense of adding a further penalty to classes > with already-terrible hashCodes. (One arguable benefit is that people > who now notice this will be more motivated to fix their hashCodes.) > The main penalty here vs linear lists for cases with only dozens > (vs millions) of clashing elements is the call to comparableClassFor > for non-Strings, which hits some reflective goop. > I posted something a few years ago mentioning that this could be > made faster with some other JDK support. But as Peter Levart noted, > bootstrapping issues preclude the most straightforward solution. > > -Doug > > From dl at cs.oswego.edu Sat Jan 10 00:20:33 2015 From: dl at cs.oswego.edu (Doug Lea) Date: Fri, 09 Jan 2015 19:20:33 -0500 Subject: HashMap collision speed (regression 7->8) In-Reply-To: References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> Message-ID: <54B07051.5050409@cs.oswego.edu> On 01/09/2015 06:29 PM, Martin Buchholz wrote: > Given the prevalence of sub-optimal hashcodes, my own intuition is also that > raising the treeification threshold from 8 will be a win. That's what I thought at first. But 8 is a better choice for String and other Comparable keys, which account for the majority of HashMaps out there. (For non-comparables, infinity is the best threshold.) How much slower should we make the most common cases to make the others faster? The only way to decide empirically is to take a large corpus of programs and vary thresholds. Short of that, speeding up comparableClassFor is still the best bet for reducing impact on non-comparables. -Doug From huizhe.wang at oracle.com Sat Jan 10 01:12:20 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Fri, 09 Jan 2015 17:12:20 -0800 Subject: RFR: 8068732, Adding Initial RowSet tests In-Reply-To: References: Message-ID: <54B07C74.3000404@oracle.com> Hi Lance, Looks good to me. Are classes CachedRowSetTests and WebRowSetTests used? The Common* tests seem to me all extends CommonCachedRowSetTests. A minor point: would it make sense to add a rowSetType data provider that includes listener(s)? Some of the tests in CommonCachedRowSetTests are disabled, did they not work? The unsetMatchColumn - SQLException tests that follow them imply that the setMatchColumn method works. Best, Joe On 1/9/2015 7:35 AM, Lance Andersen wrote: > Hi all, > > Please find the webrev for adding an initial set of tests for RowSets. The webrev is at http://cr.openjdk.java.net/~lancea/8068732/webrev.00/ > > Best, > Lance > > > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > From lance.andersen at oracle.com Sat Jan 10 01:49:51 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 9 Jan 2015 20:49:51 -0500 Subject: RFR: 8068732, Adding Initial RowSet tests In-Reply-To: <54B07C74.3000404@oracle.com> References: <54B07C74.3000404@oracle.com> Message-ID: <3A1EAA62-B632-453D-94A8-D0E18D1B1186@oracle.com> Hi Joe, On Jan 9, 2015, at 8:12 PM, huizhe wang wrote: > Hi Lance, > > Looks good to me. Thank you > > Are classes CachedRowSetTests and WebRowSetTests used? The Common* tests seem to me all extends CommonCachedRowSetTests. Yes, they are, Each type of RowSet (Cached/Web/Join/Filter) have a XXXRowSetTest class which extend some form of CommonXXXRowSet: CommonCachedRowSetTests extends CommonRowSetTests. CommonWebRowSetTests extends CommonCachedRowSetTests BaseRowSetTests extends CommonRowSetTests WebRowSetTests extends CommonWebRowSetTests CachedRowSetTests extends CommonCachedRowSetTests FilteredRowSetTests extends CommonWebRowSetTests JoinRowSetTests extends CommonWebRowSetTests The above is similar to how the XXXRowSet interfaces are desgined Many of the tests are applicable to other RowSets but some are only applicable to a subset and reduces potential duplication of common tests > > A minor point: would it make sense to add a rowSetType data provider that includes listener(s)? I can possibly look at this for some of the tests in CommonCachedRowSet but in some cases like BaseRowSet, it would not be applicable based on how the API was originally designed. > > Some of the tests in CommonCachedRowSetTests are disabled, did they not work? The unsetMatchColumn - SQLException tests that follow them imply that the setMatchColumn method works. Yes there are some tests which I have left in but disabled as I found implementation bugs. setMatchColumn is a good example as you should be able to specify the index or columnLabel interchangeably but the implementation does not account for this You will notice this and some of the other RowSet tests where the tests are overridden because of implementation bugs and are basically a no-op. This allows me to just enable or remove the overridden tests but not lose the coverage where it is actually not buggy. Let me know if the above is clear(as mud) otherwise I will try and clarify further?. :-) There probably some additional refactoring but wanted to get a baseline in and will revisit as I add additional tests as I did for the BaseRowSet tests Have a nice weekend! Best, Lance > > Best, > Joe > > On 1/9/2015 7:35 AM, Lance Andersen wrote: >> Hi all, >> >> Please find the webrev for adding an initial set of tests for RowSets. The webrev is at http://cr.openjdk.java.net/~lancea/8068732/webrev.00/ >> >> Best, >> Lance >> >> >> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> >> >> > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From huizhe.wang at oracle.com Sat Jan 10 02:43:04 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Fri, 09 Jan 2015 18:43:04 -0800 Subject: RFR: 8068732, Adding Initial RowSet tests In-Reply-To: <3A1EAA62-B632-453D-94A8-D0E18D1B1186@oracle.com> References: <54B07C74.3000404@oracle.com> <3A1EAA62-B632-453D-94A8-D0E18D1B1186@oracle.com> Message-ID: <54B091B8.5030506@oracle.com> Hi Lance, Thanks for the explanation. That's an interesting/smart use of the annotation (enabled). I was guessing there might be issues in the implementation :-) It's good to get a baseline in and revisit later, similarly as you've seen in the newly-migrated jaxp tests. I'm just glad I have a good set of working tests that I can use now, and can worry about refining them later. This set of tests of yours is in a much better shape than those, I think you can push them as they are. Best, Joe On 1/9/2015 5:49 PM, Lance Andersen wrote: > Hi Joe, > > On Jan 9, 2015, at 8:12 PM, huizhe wang > wrote: > >> Hi Lance, >> >> Looks good to me. > > Thank you >> >> Are classes CachedRowSetTests and WebRowSetTests used? The Common* >> tests seem to me all extends CommonCachedRowSetTests. > > Yes, they are, Each type of RowSet (Cached/Web/Join/Filter) have a > XXXRowSetTest class which extend some form of CommonXXXRowSet: > > CommonCachedRowSetTests extends CommonRowSetTests. > CommonWebRowSetTests extends CommonCachedRowSetTests > > BaseRowSetTests extends CommonRowSetTests > WebRowSetTests extends CommonWebRowSetTests > CachedRowSetTests extends CommonCachedRowSetTests > FilteredRowSetTests extends CommonWebRowSetTests > JoinRowSetTests extends CommonWebRowSetTests > > The above is similar to how the XXXRowSet interfaces are desgined > > Many of the tests are applicable to other RowSets but some are only > applicable to a subset and reduces potential duplication of common tests > >> >> A minor point: would it make sense to add a rowSetType data provider >> that includes listener(s)? > > I can possibly look at this for some of the tests in > CommonCachedRowSet but in some cases like BaseRowSet, it would not be > applicable based on how the API was originally designed. >> >> Some of the tests in CommonCachedRowSetTests are disabled, did they >> not work? The unsetMatchColumn - SQLException tests that follow them >> imply that the setMatchColumn method works. > Yes there are some tests which I have left in but disabled as I found > implementation bugs. > > setMatchColumn is a good example as you should be able to specify the > index or columnLabel interchangeably but the implementation does not > account for this > > You will notice this and some of the other RowSet tests where the > tests are overridden because of implementation bugs and are basically > a no-op. This allows me to just enable or remove the overridden tests > but not lose the coverage where it is actually not buggy. > > Let me know if the above is clear(as mud) otherwise I will try and > clarify further?. :-) > > There probably some additional refactoring but wanted to get a > baseline in and will revisit as I add additional tests as I did for > the BaseRowSet tests > > Have a nice weekend! > > Best, > Lance >> >> Best, >> Joe >> >> On 1/9/2015 7:35 AM, Lance Andersen wrote: >>> Hi all, >>> >>> Please find the webrev for adding an initial set of tests for >>> RowSets. The webrev is at >>> http://cr.openjdk.java.net/~lancea/8068732/webrev.00/ >>> >>> >>> Best, >>> Lance >>> >>> >>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>> Oracle Java Engineering >>> 1 Network Drive >>> Burlington, MA 01803 >>> Lance.Andersen at oracle.com >>> >>> >>> >> > > > > Lance > Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > From peter.firmstone at zeus.net.au Sat Jan 10 07:00:22 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Sat, 10 Jan 2015 17:00:22 +1000 Subject: Explicit Serialization API and Security Message-ID: <54B0CE06.2020408@zeus.net.au> Again, thank you all for engaging in discussion of this very difficult topic. While we can't presently check intra object dependencies during deserialization with readObject(), the examples I provide can do this. ObjectInputValidation is sufficient for enforcing business rules, however it can't stop an attacker. By the time the validator runs, the attackers object of choice has been instantiated and it's game over. That's right, the attacker may choose any Serializable class from the classpath, and may even extend non serializable classes, with a zero arg constructor, such as ClassLoader. To trust deserialization of an Object, as opposed to a String or primitive, the ObjectInputStream (or overriding subclass) must limit the classes allowed to be deserialized. For a class to be trusted, it must be trusted to check its parameters and enforce its invariants during object deserialization and to not deserialize with priviliged context. Validating an entire object graph's invariants, requires each class in the graph to take responsibility for validating and enforcing its own invariants. As shown in my earlier example, intra class invariants can be enforced using Serialization constructors, from which static methods are called to check invariants prior to super class constructors being called. Presently, I override ObjectInputStream and use a Permission called DeserializationPermission to limit classes that can be deserialized. Untrusted connections are run from unprivileged context to limit classes that can be instantiated, while those with trusted connections are run with a Subject that allows any class to be deserialized. ReadSerial, could do this: Class c = rs.getType("foo"); And Foo f = rs.getObject("foo", Foo.class); Which performs instanceof type check, (prior to Object deserialization, if first time deserialized, otherwise after) and generic complile time type checked method return. Thus we must restrict the classes that can be deserialized with ObjectInputStream. If Foo is mutable and we want a private copy, the caller needs to copy or clone it before checking invariants, as it would any mutable constructor parameter. Each level of validation is the responsibility of the component with the most knowledge. 1. Trusted class lists - administration, it might change. 2. Object invariants and intra object invariants - Classes, not objects. 3. Business rules, but not security - ObjectInputValidation. So summing up, in order to secure deserialization we must validate all data input, preferably before allowing object instantiation. Once an object has been constructed an attacker can gain a reference, whether by a finalizer attack or some other cleverly crafted stream, the attacker cannot obtain a reference to an object that doesn't exist. Circular links can allow an attacker to obtain an object reference ,prior to its invariants being checked, when we rely on readObject() to throw an exception. Delaying finalizer registration won't save you. Classes with circular links should be final and use a transient boolean "initialized" field, that every method checks to prevent an attacker doing anything useful, should they gain a reference to an incorrectly constructed object. The real question is, do we continue to plaster over the issues with the Serialization framework's api, and continue to create special cases such as a second final field freeze after a constructor completes? To be honest, I don't like this second final field freeze, because invariants haven't been checked. Don't get me wrong, Serialization is quite clever, but implementing Serializable properly can consume a lot of time due to api complexity and is presently a security problem. In my examples all constructors share the same invariant checks and because it's public API, the invariants can be tested easily with unit tests. It seems like we're trying to give constructor properties to a private instance method at the expense of increasing complexity, wouldn't it be simpler in the long term to provide a Serialization constructor? P.S. David, I like your suggestion of using a protected method, for limiting array size. Thank you, Peter. > On 01/08/2015 02:10 PM, Brian Goetz wrote: >>> >> 1) Validate invariants >>> >> >>> >> A clear and easy to understand mechanism that can validate the >>> >> deserialized >>> >> fields. Does not prevent the use of final fields, as the >>> >> serialization framework >>> >> will be responsible for setting them. Something along the lines >>> >> of what David >>> >> suggested: >>> >> >>> >> private static void validate(GetField fields) { >>> >> if (fields.getInt("lo")> fields.getInt("hi")) { ... } >>> >> } >>> >> >>> >> This could be a ?special? method, or annotation driven. TBD. >>> >> >>> >> Note: the validate method is static, so the object instance is >>> >> not required to >>> >> be created before running the validation. >> > >> > Sort of... >> > >> > This is true if the fields participating in the invariant are >> > primitives. But if they're refs, what do you do? What if you want to >> > validate something like >> > >> > count == list.size() // fields are int count, List list >> > >> > ? Then wouldn't GetField.getObject have to deserialize the object >> > referred to by that field? > In fact you cannot validate invariants across multiple objects at all > using this method*or* readObject() (existing or enhanced) unless the > object in question is an enum, Class, or String (and a few other special > cases) because you can't rely on initialization order during > deserialization. That's the very reason why OIS#registerValidation() > even exists - inter-object validation is not possible until after the > root-most readObject has completed, which is the time when validations > are executed. Robust validation of a given object class may require two > stages - "near" validation and "spanning" validation - to fully > validate. However the readObject() approach and its proposed variations > (including the static validate() version) can still be useful for > fail-fast and non-complex validations; you just have to understand that > (just as today) any Object you examine might not be fully initialized yet. > > -- - DML From peter.levart at gmail.com Sat Jan 10 13:01:18 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sat, 10 Jan 2015 14:01:18 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B07051.5050409@cs.oswego.edu> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> Message-ID: <54B1229E.5060904@gmail.com> On 01/10/2015 01:20 AM, Doug Lea wrote: > On 01/09/2015 06:29 PM, Martin Buchholz wrote: >> Given the prevalence of sub-optimal hashcodes, my own intuition is >> also that >> raising the treeification threshold from 8 will be a win. > > That's what I thought at first. But 8 is a better choice for String > and other Comparable keys, which account for the majority of HashMaps > out there. (For non-comparables, infinity is the best threshold.) > How much slower should we make the most common cases to make the others > faster? The only way to decide empirically is to take a large > corpus of programs and vary thresholds. Short of that, speeding up > comparableClassFor is still the best bet for reducing impact on > non-comparables. Hi Doug, comparableClassFor() for non-comparables that don't implement Comparable is already as fast as it can be (the 1st check is instanceof Comparable). For other comparables (and non-comparables) that implement Comparable (except for String which is special-cased), we could improve the situation by caching the result. Here's another attempt at that. This time it uses plain old JDK1 stuff, so it actually works even in HashMap (using IdentityHashMap so no danger of circular usage if it is to be applied to CHM also): http://cr.openjdk.java.net/~plevart/jdk9-dev/Class.getGenericDerivative/webrev.01/ With this patch, the results of Bernd's JMH benchmark do give some boost to keys that implement Comparable (badDistWithComp case). These are the results with original JDK9 HashMap: Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 avgt 6 3104.047 278.057 ms/op j.t.HashMapCollision.badDistWithComp 16 avgt 6 2754.499 243.780 ms/op j.t.HashMapCollision.goodDistNoComp 16 avgt 6 1031.992 26.422 ms/op j.t.HashMapCollision.goodDistWithComp 16 avgt 6 1082.347 30.981 ms/op And this is with patch applied: Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 avgt 6 3081.419 386.125 ms/op j.t.HashMapCollision.badDistWithComp 16 avgt 6 2116.030 281.160 ms/op j.t.HashMapCollision.goodDistNoComp 16 avgt 6 1015.224 81.843 ms/op j.t.HashMapCollision.goodDistWithComp 16 avgt 6 1078.719 38.351 ms/op Caching is performed as part of Class generic types information caching (ClassRepository), so there's no overhead for those that don't need generic types information. All logic is kept inside (C)HM. Regards, Peter > > -Doug > From martinrb at google.com Sun Jan 11 01:00:58 2015 From: martinrb at google.com (Martin Buchholz) Date: Sat, 10 Jan 2015 17:00:58 -0800 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B1229E.5060904@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> Message-ID: Why not treeify only when trying to insert an element that is instanceof Comparable? That way, we will not attempt to use less efficient treebins when there will be no benefit. Something like: diff --git a/src/java.base/share/classes/java/util/HashMap.java b/src/java.base/share/classes/java/util/HashMap.java --- a/src/java.base/share/classes/java/util/HashMap.java +++ b/src/java.base/share/classes/java/util/HashMap.java @@ -639,7 +639,8 @@ for (int binCount = 0; ; ++binCount) { if ((e = p.next) == null) { p.next = newNode(hash, key, value, null); - if (binCount >= TREEIFY_THRESHOLD - 1) // -1 for 1st + if (binCount >= TREEIFY_THRESHOLD - 1 // -1 for 1st + && key instanceof Comparable) treeifyBin(tab, hash); break; } @@ -1127,7 +1128,8 @@ t.putTreeVal(this, tab, hash, key, v); else { tab[i] = newNode(hash, key, v, first); - if (binCount >= TREEIFY_THRESHOLD - 1) + if (binCount >= TREEIFY_THRESHOLD - 1 + && key instanceof Comparable) treeifyBin(tab, hash); } ++modCount; @@ -1199,7 +1201,8 @@ t.putTreeVal(this, tab, hash, key, v); else { tab[i] = newNode(hash, key, v, first); - if (binCount >= TREEIFY_THRESHOLD - 1) + if (binCount >= TREEIFY_THRESHOLD - 1 + && key instanceof Comparable) treeifyBin(tab, hash); } ++modCount; @@ -1258,7 +1261,8 @@ t.putTreeVal(this, tab, hash, key, value); else { tab[i] = newNode(hash, key, value, first); - if (binCount >= TREEIFY_THRESHOLD - 1) + if (binCount >= TREEIFY_THRESHOLD - 1 + && key instanceof Comparable) treeifyBin(tab, hash); } ++modCount; From martinrb at google.com Sun Jan 11 01:21:59 2015 From: martinrb at google.com (Martin Buchholz) Date: Sat, 10 Jan 2015 17:21:59 -0800 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B07051.5050409@cs.oswego.edu> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> Message-ID: On Fri, Jan 9, 2015 at 4:20 PM, Doug Lea
wrote: > On 01/09/2015 06:29 PM, Martin Buchholz wrote: > >> Given the prevalence of sub-optimal hashcodes, my own intuition is also >> that >> raising the treeification threshold from 8 will be a win. >> > > That's what I thought at first. But 8 is a better choice for String > and other Comparable keys, which account for the majority of HashMaps > out there. (For non-comparables, infinity is the best threshold.) > Non-malicious Strings are only rarely going to end up in treebins, because their hash codes are not awful (just sub-optimal - hmmm ... is anyone putting all possible 2-char Strings into a HashMap?). I'm not convinced that comparableClassFor should special-case Strings. Linear search through a non-treebin of non-malicious Strings is likely to be fast, because we will short-circuit if hash codes or length don't match, which compareTo cannot. So compareTo is likely to have more overhead per element, although of course in the end log(N) beats N. From ecki at zusammenkunft.net Sun Jan 11 01:22:05 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Sun, 11 Jan 2015 02:22:05 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> Message-ID: <20150111022205.0000007b.ecki@zusammenkunft.net> Am Sat, 10 Jan 2015 17:00:58 -0800 schrieb Martin Buchholz : > Why not treeify only when trying to insert an element that is > instanceof Comparable? That way, we will not attempt to use less > efficient treebins when there will be no benefit. This is my thinking as well, one thing which was unclear to me: when the keys are not Comparable but their instance is reused for put/get, will the system hashcode method allow to do a proper tree lookup anyway (or actually will the code do a tree lookup). In my Bench equal key objects with different instances are used, and this is quite common. But in some scenarios it might be likely that the actual instance of a put key will also be get. Gruss Bernd > > Something like: > > diff --git a/src/java.base/share/classes/java/util/HashMap.java > b/src/java.base/share/classes/java/util/HashMap.java > --- a/src/java.base/share/classes/java/util/HashMap.java > +++ b/src/java.base/share/classes/java/util/HashMap.java > @@ -639,7 +639,8 @@ > for (int binCount = 0; ; ++binCount) { > if ((e = p.next) == null) { > p.next = newNode(hash, key, value, null); > - if (binCount >= TREEIFY_THRESHOLD - 1) // -1 > for 1st > + if (binCount >= TREEIFY_THRESHOLD - 1 // -1 > for 1st > + && key instanceof Comparable) > treeifyBin(tab, hash); > break; > } > @@ -1127,7 +1128,8 @@ > t.putTreeVal(this, tab, hash, key, v); > else { > tab[i] = newNode(hash, key, v, first); > - if (binCount >= TREEIFY_THRESHOLD - 1) > + if (binCount >= TREEIFY_THRESHOLD - 1 > + && key instanceof Comparable) > treeifyBin(tab, hash); > } > ++modCount; > @@ -1199,7 +1201,8 @@ > t.putTreeVal(this, tab, hash, key, v); > else { > tab[i] = newNode(hash, key, v, first); > - if (binCount >= TREEIFY_THRESHOLD - 1) > + if (binCount >= TREEIFY_THRESHOLD - 1 > + && key instanceof Comparable) > treeifyBin(tab, hash); > } > ++modCount; > @@ -1258,7 +1261,8 @@ > t.putTreeVal(this, tab, hash, key, value); > else { > tab[i] = newNode(hash, key, value, first); > - if (binCount >= TREEIFY_THRESHOLD - 1) > + if (binCount >= TREEIFY_THRESHOLD - 1 > + && key instanceof Comparable) > treeifyBin(tab, hash); > } > ++modCount; From martinrb at google.com Sun Jan 11 01:27:13 2015 From: martinrb at google.com (Martin Buchholz) Date: Sat, 10 Jan 2015 17:27:13 -0800 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B1229E.5060904@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> Message-ID: Peter, You are adding the ability to add "app-specific storage" to Class objects ("Class-local variables"?), which is pretty unusual. I was thinking instead of a very dumb 1-element cache, remembering Class and comparableClassFor, which will work for typical homogeneous HashMaps. On Sat, Jan 10, 2015 at 5:01 AM, Peter Levart wrote: > > On 01/10/2015 01:20 AM, Doug Lea wrote: > > On 01/09/2015 06:29 PM, Martin Buchholz wrote: > > Given the prevalence of sub-optimal hashcodes, my own intuition is also > that > raising the treeification threshold from 8 will be a win. > > > That's what I thought at first. But 8 is a better choice for String > and other Comparable keys, which account for the majority of HashMaps > out there. (For non-comparables, infinity is the best threshold.) > How much slower should we make the most common cases to make the others > faster? The only way to decide empirically is to take a large > corpus of programs and vary thresholds. Short of that, speeding up > comparableClassFor is still the best bet for reducing impact on > non-comparables. > > > Hi Doug, > > comparableClassFor() for non-comparables that don't implement Comparable > is already as fast as it can be (the 1st check is instanceof Comparable). > For other comparables (and non-comparables) that implement Comparable > (except for String which is special-cased), we could improve the situation > by caching the result. > > Here's another attempt at that. This time it uses plain old JDK1 stuff, so > it actually works even in HashMap (using IdentityHashMap so no danger of > circular usage if it is to be applied to CHM also): > > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Class.getGenericDerivative/webrev.01/ > > With this patch, the results of Bernd's JMH benchmark do give some boost > to keys that implement Comparable (badDistWithComp case). > > These are the results with original JDK9 HashMap: > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 avgt 6 > 3104.047 278.057 ms/op > j.t.HashMapCollision.badDistWithComp 16 avgt 6 > 2754.499 243.780 ms/op > j.t.HashMapCollision.goodDistNoComp 16 avgt 6 > 1031.992 26.422 ms/op > j.t.HashMapCollision.goodDistWithComp 16 avgt 6 > 1082.347 30.981 ms/op > > And this is with patch applied: > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 avgt 6 > 3081.419 386.125 ms/op > j.t.HashMapCollision.badDistWithComp 16 avgt 6 > 2116.030 281.160 ms/op > j.t.HashMapCollision.goodDistNoComp 16 avgt 6 > 1015.224 81.843 ms/op > j.t.HashMapCollision.goodDistWithComp 16 avgt 6 > 1078.719 38.351 ms/op > > > Caching is performed as part of Class generic types information caching > (ClassRepository), so there's no overhead for those that don't need generic > types information. All logic is kept inside (C)HM. > > Regards, Peter > > > -Doug > > > From martinrb at google.com Sun Jan 11 02:13:52 2015 From: martinrb at google.com (Martin Buchholz) Date: Sat, 10 Jan 2015 18:13:52 -0800 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <20150111022205.0000007b.ecki@zusammenkunft.net> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <20150111022205.0000007b.ecki@zusammenkunft.net> Message-ID: On further reflection, treebins *can* help when elements are not mutually comparable, *if* they have different hashcodes. The treebin is sorted by hashcode first, then by their Comparable relationship. On Sat, Jan 10, 2015 at 5:22 PM, Bernd Eckenfels wrote: > Am Sat, 10 Jan 2015 17:00:58 -0800 > schrieb Martin Buchholz : > > > Why not treeify only when trying to insert an element that is > > instanceof Comparable? That way, we will not attempt to use less > > efficient treebins when there will be no benefit. > > This is my thinking as well, one thing which was unclear to me: when > the keys are not Comparable but their instance is reused for put/get, > will the system hashcode method allow to do a proper tree lookup > anyway (or actually will the code do a tree lookup). > > In my Bench equal key objects with different instances are used, and > this is quite common. But in some scenarios it might be likely that the > actual instance of a put key will also be get. > > Gruss > Bernd > > > > > > > Something like: > > > > diff --git a/src/java.base/share/classes/java/util/HashMap.java > > b/src/java.base/share/classes/java/util/HashMap.java > > --- a/src/java.base/share/classes/java/util/HashMap.java > > +++ b/src/java.base/share/classes/java/util/HashMap.java > > @@ -639,7 +639,8 @@ > > for (int binCount = 0; ; ++binCount) { > > if ((e = p.next) == null) { > > p.next = newNode(hash, key, value, null); > > - if (binCount >= TREEIFY_THRESHOLD - 1) // -1 > > for 1st > > + if (binCount >= TREEIFY_THRESHOLD - 1 // -1 > > for 1st > > + && key instanceof Comparable) > > treeifyBin(tab, hash); > > break; > > } > > @@ -1127,7 +1128,8 @@ > > t.putTreeVal(this, tab, hash, key, v); > > else { > > tab[i] = newNode(hash, key, v, first); > > - if (binCount >= TREEIFY_THRESHOLD - 1) > > + if (binCount >= TREEIFY_THRESHOLD - 1 > > + && key instanceof Comparable) > > treeifyBin(tab, hash); > > } > > ++modCount; > > @@ -1199,7 +1201,8 @@ > > t.putTreeVal(this, tab, hash, key, v); > > else { > > tab[i] = newNode(hash, key, v, first); > > - if (binCount >= TREEIFY_THRESHOLD - 1) > > + if (binCount >= TREEIFY_THRESHOLD - 1 > > + && key instanceof Comparable) > > treeifyBin(tab, hash); > > } > > ++modCount; > > @@ -1258,7 +1261,8 @@ > > t.putTreeVal(this, tab, hash, key, value); > > else { > > tab[i] = newNode(hash, key, value, first); > > - if (binCount >= TREEIFY_THRESHOLD - 1) > > + if (binCount >= TREEIFY_THRESHOLD - 1 > > + && key instanceof Comparable) > > treeifyBin(tab, hash); > > } > > ++modCount; > > From peter.levart at gmail.com Sun Jan 11 11:35:01 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sun, 11 Jan 2015 12:35:01 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> Message-ID: <54B25FE5.9000301@gmail.com> On 01/11/2015 02:21 AM, Martin Buchholz wrote: > On Fri, Jan 9, 2015 at 4:20 PM, Doug Lea
wrote: > >> On 01/09/2015 06:29 PM, Martin Buchholz wrote: >> >>> Given the prevalence of sub-optimal hashcodes, my own intuition is also >>> that >>> raising the treeification threshold from 8 will be a win. >>> >> That's what I thought at first. But 8 is a better choice for String >> and other Comparable keys, which account for the majority of HashMaps >> out there. (For non-comparables, infinity is the best threshold.) >> > Non-malicious Strings are only rarely going to end up in treebins, because > their hash codes are not awful (just sub-optimal - hmmm ... is anyone > putting all possible 2-char Strings into a HashMap?). I'm not convinced > that comparableClassFor should special-case Strings. > Linear search through a non-treebin of non-malicious Strings is likely to > be fast, because we will short-circuit if hash codes or length don't match, > which compareTo cannot. So compareTo is likely to have more overhead per > element, although of course in the end log(N) beats N. As I understand the code, compareTo is only called when hashes are equal to break the tie, otherwise hash is always used 1st in tree bins, but if hashes are the same, then equlas() does check the length 1st but compareTo can't. Peter From peter.levart at gmail.com Sun Jan 11 11:55:48 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sun, 11 Jan 2015 12:55:48 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> Message-ID: <54B264C4.90506@gmail.com> On 01/11/2015 02:27 AM, Martin Buchholz wrote: > Peter, > > You are adding the ability to add "app-specific storage" to Class > objects ("Class-local variables"?), which is pretty unusual. Well, that was my intention, since the logic about what should be cached is very specific to the usecase and might change in the future. Anyway, this is only internal API. Users have a public alternative in ClassValue. That's one reason. The other is space overhead introduced when caching with ClassValue and inability to initialize ClassValue so very early in the boot-up sequence. > > I was thinking instead of a very dumb 1-element cache, remembering > Class and comparableClassFor, which will work for typical homogeneous > HashMaps. This seems like a good idea. We would actually need only one field of type Class and a boolean flag. Unfortunately, comparableClassFor is a static method used also from various other contexts that don't have access to HashMap instance, for example from TreeNode. We would have to extend the internal API with an additional HashMap argument to pass the HM instance around. Not to mention that this would be tricky because retaining the last used comparable Class object in the HM instance could prevent GC from releasing a ClassLoader in an app server environment for example. A WeakReference> would have to be used and new WeakReference object created each time cached value changes. Unless we cache only the 1st comparableClassFor result and never change it, which has the same cache-hit ratio for homogeneous keys. Right, here's what this looks like: http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HomogeneousKeysCache/webrev.01/ I modified Bernd's JMH benchmark a little to use ThreadLocalRandom insted of Random, so results express more what is going on with HashMap and less with Random synchronization: http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HashMapCollision.java Results: Original JDK9 HashMap: Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 avgt 6 3101.247 435.866 ms/op j.t.HashMapCollision.badDistWithComp 16 avgt 6 2410.202 478.247 ms/op j.t.HashMapCollision.goodDistNoComp 16 avgt 6 615.100 7.063 ms/op j.t.HashMapCollision.goodDistWithComp 16 avgt 6 614.229 159.558 ms/op Caching of comparableClassFor (in ClassRepository - good for heterogeneous keys too): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 avgt 6 3305.967 652.791 ms/op j.t.HashMapCollision.badDistWithComp 16 avgt 6 2030.965 241.910 ms/op j.t.HashMapCollision.goodDistNoComp 16 avgt 6 611.202 6.440 ms/op j.t.HashMapCollision.goodDistWithComp 16 avgt 6 582.890 4.896 ms/op Caching of comparableClassFor (internally - good for homogeneous keys only): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 avgt 6 3265.673 660.030 ms/op j.t.HashMapCollision.badDistWithComp 16 avgt 6 1875.204 224.682 ms/op j.t.HashMapCollision.goodDistNoComp 16 avgt 6 598.949 25.484 ms/op j.t.HashMapCollision.goodDistWithComp 16 avgt 6 585.278 8.103 ms/op Regards, Peter > > On Sat, Jan 10, 2015 at 5:01 AM, Peter Levart > wrote: > > > On 01/10/2015 01:20 AM, Doug Lea wrote: >> On 01/09/2015 06:29 PM, Martin Buchholz wrote: >>> Given the prevalence of sub-optimal hashcodes, my own intuition >>> is also that >>> raising the treeification threshold from 8 will be a win. >> >> That's what I thought at first. But 8 is a better choice for String >> and other Comparable keys, which account for the majority of >> HashMaps >> out there. (For non-comparables, infinity is the best threshold.) >> How much slower should we make the most common cases to make the >> others >> faster? The only way to decide empirically is to take a large >> corpus of programs and vary thresholds. Short of that, speeding up >> comparableClassFor is still the best bet for reducing impact on >> non-comparables. > > Hi Doug, > > comparableClassFor() for non-comparables that don't implement > Comparable is already as fast as it can be (the 1st check is > instanceof Comparable). For other comparables (and > non-comparables) that implement Comparable (except for String > which is special-cased), we could improve the situation by caching > the result. > > Here's another attempt at that. This time it uses plain old JDK1 > stuff, so it actually works even in HashMap (using IdentityHashMap > so no danger of circular usage if it is to be applied to CHM also): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Class.getGenericDerivative/webrev.01/ > > > With this patch, the results of Bernd's JMH benchmark do give some > boost to keys that implement Comparable (badDistWithComp case). > > These are the results with original JDK9 HashMap: > > Benchmark (initialSize) Mode Samples Score Score > error Units > j.t.HashMapCollision.badDistNoComp 16 avgt 6 > 3104.047 278.057 ms/op > j.t.HashMapCollision.badDistWithComp 16 avgt 6 > 2754.499 243.780 ms/op > j.t.HashMapCollision.goodDistNoComp 16 avgt 6 > 1031.992 26.422 ms/op > j.t.HashMapCollision.goodDistWithComp 16 avgt 6 > 1082.347 30.981 ms/op > > And this is with patch applied: > > Benchmark (initialSize) Mode Samples Score Score > error Units > j.t.HashMapCollision.badDistNoComp 16 avgt 6 > 3081.419 386.125 ms/op > j.t.HashMapCollision.badDistWithComp 16 avgt 6 > 2116.030 281.160 ms/op > j.t.HashMapCollision.goodDistNoComp 16 avgt 6 > 1015.224 81.843 ms/op > j.t.HashMapCollision.goodDistWithComp 16 avgt 6 > 1078.719 38.351 ms/op > > > Caching is performed as part of Class generic types information > caching (ClassRepository), so there's no overhead for those that > don't need generic types information. All logic is kept inside (C)HM. > > Regards, Peter > >> >> -Doug >> > > From peter.levart at gmail.com Sun Jan 11 15:09:02 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sun, 11 Jan 2015 16:09:02 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B264C4.90506@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> Message-ID: <54B2920E.4000803@gmail.com> Hi, I wasn't comfortable with Bernd's HMH benchmark results jitter, so I changed the mode of operation to be SingleShotTime (since a particular invocation is from 0.6 to 3sec anyway). GC is triggered before each invocation (-gc true). I also added -XX:-TieredCompilation VM option and run 6 forks of 10 iterations of each test. By Doug's suggestion I also added a variant of unchanged HashMap where TREEIFY_THRESHOLD = 1 << 20, UNTREEIFY_THRESHOLD = TREEIFY_THRESHOLD - 2, MIN_TREEIFY_CAPACITY = TREEIFY_THRESHOLD * 4 as a reference to compare with. Here are the results: Original JDK9 HashMap: Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3011.738 78.249 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2984.280 48.315 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 682.060 52.341 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 685.705 55.183 ms Original JDK9 HashMap with TREEIFY_THRESHOLD = 1 << 20: Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 2780.771 236.647 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2541.740 233.429 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 757.364 67.869 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 671.617 54.943 ms Caching of comparableClassFor (in ClassRepository - good for heterogeneous keys too): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3014.888 71.778 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2279.757 54.159 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 760.743 70.674 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 725.188 67.853 ms Caching of comparableClassFor (internally - good for homogeneous keys only): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3026.707 84.571 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2137.296 66.140 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 635.964 8.213 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 685.129 46.783 ms Regards, Peter On 01/11/2015 12:55 PM, Peter Levart wrote: > > On 01/11/2015 02:27 AM, Martin Buchholz wrote: >> Peter, >> >> You are adding the ability to add "app-specific storage" to Class >> objects ("Class-local variables"?), which is pretty unusual. > > Well, that was my intention, since the logic about what should be > cached is very specific to the usecase and might change in the future. > Anyway, this is only internal API. Users have a public alternative in > ClassValue. That's one reason. The other is space overhead introduced > when caching with ClassValue and inability to initialize ClassValue so > very early in the boot-up sequence. > >> >> I was thinking instead of a very dumb 1-element cache, remembering >> Class and comparableClassFor, which will work for typical homogeneous >> HashMaps. > > This seems like a good idea. We would actually need only one field of > type Class and a boolean flag. > > Unfortunately, comparableClassFor is a static method used also from > various other contexts that don't have access to HashMap instance, for > example from TreeNode. We would have to extend the internal API with > an additional HashMap argument to pass the HM instance around. Not to > mention that this would be tricky because retaining the last used > comparable Class object in the HM instance could prevent GC from > releasing a ClassLoader in an app server environment for example. A > WeakReference> would have to be used and new WeakReference > object created each time cached value changes. Unless we cache only > the 1st comparableClassFor result and never change it, which has the > same cache-hit ratio for homogeneous keys. > > Right, here's what this looks like: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HomogeneousKeysCache/webrev.01/ > > I modified Bernd's JMH benchmark a little to use ThreadLocalRandom > insted of Random, so results express more what is going on with > HashMap and less with Random synchronization: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HashMapCollision.java > > Results: > > Original JDK9 HashMap: > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 avgt > 6 3101.247 435.866 ms/op > j.t.HashMapCollision.badDistWithComp 16 avgt > 6 2410.202 478.247 ms/op > j.t.HashMapCollision.goodDistNoComp 16 avgt > 6 615.100 7.063 ms/op > j.t.HashMapCollision.goodDistWithComp 16 avgt > 6 614.229 159.558 ms/op > > Caching of comparableClassFor (in ClassRepository - good for > heterogeneous keys too): > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 avgt > 6 3305.967 652.791 ms/op > j.t.HashMapCollision.badDistWithComp 16 avgt > 6 2030.965 241.910 ms/op > j.t.HashMapCollision.goodDistNoComp 16 avgt > 6 611.202 6.440 ms/op > j.t.HashMapCollision.goodDistWithComp 16 avgt > 6 582.890 4.896 ms/op > > Caching of comparableClassFor (internally - good for homogeneous keys > only): > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 avgt > 6 3265.673 660.030 ms/op > j.t.HashMapCollision.badDistWithComp 16 avgt > 6 1875.204 224.682 ms/op > j.t.HashMapCollision.goodDistNoComp 16 avgt > 6 598.949 25.484 ms/op > j.t.HashMapCollision.goodDistWithComp 16 avgt > 6 585.278 8.103 ms/op > > > Regards, Peter > >> >> On Sat, Jan 10, 2015 at 5:01 AM, Peter Levart > > wrote: >> >> >> On 01/10/2015 01:20 AM, Doug Lea wrote: >>> On 01/09/2015 06:29 PM, Martin Buchholz wrote: >>>> Given the prevalence of sub-optimal hashcodes, my own intuition >>>> is also that >>>> raising the treeification threshold from 8 will be a win. >>> >>> That's what I thought at first. But 8 is a better choice for String >>> and other Comparable keys, which account for the majority of >>> HashMaps >>> out there. (For non-comparables, infinity is the best threshold.) >>> How much slower should we make the most common cases to make the >>> others >>> faster? The only way to decide empirically is to take a large >>> corpus of programs and vary thresholds. Short of that, speeding up >>> comparableClassFor is still the best bet for reducing impact on >>> non-comparables. >> >> Hi Doug, >> >> comparableClassFor() for non-comparables that don't implement >> Comparable is already as fast as it can be (the 1st check is >> instanceof Comparable). For other comparables (and >> non-comparables) that implement Comparable (except for String >> which is special-cased), we could improve the situation by >> caching the result. >> >> Here's another attempt at that. This time it uses plain old JDK1 >> stuff, so it actually works even in HashMap (using >> IdentityHashMap so no danger of circular usage if it is to be >> applied to CHM also): >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/Class.getGenericDerivative/webrev.01/ >> >> >> With this patch, the results of Bernd's JMH benchmark do give >> some boost to keys that implement Comparable (badDistWithComp case). >> >> These are the results with original JDK9 HashMap: >> >> Benchmark (initialSize) Mode Samples Score Score >> error Units >> j.t.HashMapCollision.badDistNoComp 16 avgt 6 >> 3104.047 278.057 ms/op >> j.t.HashMapCollision.badDistWithComp 16 avgt 6 >> 2754.499 243.780 ms/op >> j.t.HashMapCollision.goodDistNoComp 16 avgt 6 >> 1031.992 26.422 ms/op >> j.t.HashMapCollision.goodDistWithComp 16 avgt 6 >> 1082.347 30.981 ms/op >> >> And this is with patch applied: >> >> Benchmark (initialSize) Mode Samples Score Score >> error Units >> j.t.HashMapCollision.badDistNoComp 16 avgt 6 >> 3081.419 386.125 ms/op >> j.t.HashMapCollision.badDistWithComp 16 avgt 6 >> 2116.030 281.160 ms/op >> j.t.HashMapCollision.goodDistNoComp 16 avgt 6 >> 1015.224 81.843 ms/op >> j.t.HashMapCollision.goodDistWithComp 16 avgt 6 >> 1078.719 38.351 ms/op >> >> >> Caching is performed as part of Class generic types information >> caching (ClassRepository), so there's no overhead for those that >> don't need generic types information. All logic is kept inside (C)HM. >> >> Regards, Peter >> >>> >>> -Doug >>> >> >> > From peter.levart at gmail.com Sun Jan 11 19:26:33 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sun, 11 Jan 2015 20:26:33 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B2920E.4000803@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> Message-ID: <54B2CE69.4050300@gmail.com> Hi, Just to illustrate what Bend's benchmark keys are made of so one can interpret the results accordingly. The full set of keys consists of 250,000 distinct keys (500*500). With GOOD_HASH multiplier, each of them has it's own distinct hashCode() and a HashMap filled with all 250k keys looks like: Capacity: 524288 Load factor: 0.75 Size: 250000 Bin sizes: 0*274288 1*250000 total=250000 Empty bins: 52.3 % Unique hash codes per bin: 0*274288 1*250000 total=250000 With BAD_HASH multiplier, there are only 18963 unique hashCodes: Capacity: 524288 Load factor: 0.75 Size: 250000 Bin sizes: 0*505325 1*74 2*74 3*74 4*74 5*74 6*74 7*74 8*74 9*74 10*74 11*74 12*74 13*8822 14*9253 total=250000 Empty bins: 96.4 % Unique hash codes per bin: 0*505325 1*18963 total=18963 Although majority of entries constitute the bins of size 13 or 14, there's only a single hashCode value per bin. So in this benchmark, treeifying with non-comparable keys is a waste of effort. Are there (non-forged) sets of non-comparable keys with hashCodes where treeifying makes sense? Regards, Peter P.S. The below modified benchmark source is update here: http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HashMapCollision.java The HashMap simulator that prints stats about bins and hashCodes is here: http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HashMapSimulator.java On 01/11/2015 04:09 PM, Peter Levart wrote: > Hi, > > I wasn't comfortable with Bernd's HMH benchmark results jitter, so I > changed the mode of operation to be SingleShotTime (since a particular > invocation is from 0.6 to 3sec anyway). GC is triggered before each > invocation (-gc true). I also added -XX:-TieredCompilation VM option > and run 6 forks of 10 iterations of each test. By Doug's suggestion I > also added a variant of unchanged HashMap where TREEIFY_THRESHOLD = 1 > << 20, UNTREEIFY_THRESHOLD = TREEIFY_THRESHOLD - 2, > MIN_TREEIFY_CAPACITY = TREEIFY_THRESHOLD * 4 as a reference to compare > with. Here are the results: > > Original JDK9 HashMap: > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3011.738 78.249 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2984.280 48.315 ms > j.t.HashMapCollision.goodDistNoComp 16 ss > 60 682.060 52.341 ms > j.t.HashMapCollision.goodDistWithComp 16 ss > 60 685.705 55.183 ms > > Original JDK9 HashMap with TREEIFY_THRESHOLD = 1 << 20: > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 2780.771 236.647 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2541.740 233.429 ms > j.t.HashMapCollision.goodDistNoComp 16 ss > 60 757.364 67.869 ms > j.t.HashMapCollision.goodDistWithComp 16 ss > 60 671.617 54.943 ms > > Caching of comparableClassFor (in ClassRepository - good for > heterogeneous keys too): > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3014.888 71.778 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2279.757 54.159 ms > j.t.HashMapCollision.goodDistNoComp 16 ss > 60 760.743 70.674 ms > j.t.HashMapCollision.goodDistWithComp 16 ss > 60 725.188 67.853 ms > > Caching of comparableClassFor (internally - good for homogeneous keys > only): > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3026.707 84.571 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2137.296 66.140 ms > j.t.HashMapCollision.goodDistNoComp 16 ss > 60 635.964 8.213 ms > j.t.HashMapCollision.goodDistWithComp 16 ss > 60 685.129 46.783 ms > > > > Regards, Peter > > On 01/11/2015 12:55 PM, Peter Levart wrote: >> >> On 01/11/2015 02:27 AM, Martin Buchholz wrote: >>> Peter, >>> >>> You are adding the ability to add "app-specific storage" to Class >>> objects ("Class-local variables"?), which is pretty unusual. >> >> Well, that was my intention, since the logic about what should be >> cached is very specific to the usecase and might change in the >> future. Anyway, this is only internal API. Users have a public >> alternative in ClassValue. That's one reason. The other is space >> overhead introduced when caching with ClassValue and inability to >> initialize ClassValue so very early in the boot-up sequence. >> >>> >>> I was thinking instead of a very dumb 1-element cache, remembering >>> Class and comparableClassFor, which will work for typical >>> homogeneous HashMaps. >> >> This seems like a good idea. We would actually need only one field of >> type Class and a boolean flag. >> >> Unfortunately, comparableClassFor is a static method used also from >> various other contexts that don't have access to HashMap instance, >> for example from TreeNode. We would have to extend the internal API >> with an additional HashMap argument to pass the HM instance around. >> Not to mention that this would be tricky because retaining the last >> used comparable Class object in the HM instance could prevent GC from >> releasing a ClassLoader in an app server environment for example. A >> WeakReference> would have to be used and new WeakReference >> object created each time cached value changes. Unless we cache only >> the 1st comparableClassFor result and never change it, which has the >> same cache-hit ratio for homogeneous keys. >> >> Right, here's what this looks like: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HomogeneousKeysCache/webrev.01/ >> >> I modified Bernd's JMH benchmark a little to use ThreadLocalRandom >> insted of Random, so results express more what is going on with >> HashMap and less with Random synchronization: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HashMapCollision.java >> >> Results: >> >> Original JDK9 HashMap: >> >> Benchmark (initialSize) Mode >> Samples Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 avgt >> 6 3101.247 435.866 ms/op >> j.t.HashMapCollision.badDistWithComp 16 avgt >> 6 2410.202 478.247 ms/op >> j.t.HashMapCollision.goodDistNoComp 16 avgt >> 6 615.100 7.063 ms/op >> j.t.HashMapCollision.goodDistWithComp 16 avgt >> 6 614.229 159.558 ms/op >> >> Caching of comparableClassFor (in ClassRepository - good for >> heterogeneous keys too): >> >> Benchmark (initialSize) Mode >> Samples Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 avgt >> 6 3305.967 652.791 ms/op >> j.t.HashMapCollision.badDistWithComp 16 avgt >> 6 2030.965 241.910 ms/op >> j.t.HashMapCollision.goodDistNoComp 16 avgt >> 6 611.202 6.440 ms/op >> j.t.HashMapCollision.goodDistWithComp 16 avgt >> 6 582.890 4.896 ms/op >> >> Caching of comparableClassFor (internally - good for homogeneous keys >> only): >> >> Benchmark (initialSize) Mode >> Samples Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 avgt >> 6 3265.673 660.030 ms/op >> j.t.HashMapCollision.badDistWithComp 16 avgt >> 6 1875.204 224.682 ms/op >> j.t.HashMapCollision.goodDistNoComp 16 avgt >> 6 598.949 25.484 ms/op >> j.t.HashMapCollision.goodDistWithComp 16 avgt >> 6 585.278 8.103 ms/op >> >> >> Regards, Peter >> >>> >>> On Sat, Jan 10, 2015 at 5:01 AM, Peter Levart >>> > wrote: >>> >>> >>> On 01/10/2015 01:20 AM, Doug Lea wrote: >>>> On 01/09/2015 06:29 PM, Martin Buchholz wrote: >>>>> Given the prevalence of sub-optimal hashcodes, my own >>>>> intuition is also that >>>>> raising the treeification threshold from 8 will be a win. >>>> >>>> That's what I thought at first. But 8 is a better choice for >>>> String >>>> and other Comparable keys, which account for the majority of >>>> HashMaps >>>> out there. (For non-comparables, infinity is the best threshold.) >>>> How much slower should we make the most common cases to make >>>> the others >>>> faster? The only way to decide empirically is to take a large >>>> corpus of programs and vary thresholds. Short of that, speeding up >>>> comparableClassFor is still the best bet for reducing impact on >>>> non-comparables. >>> >>> Hi Doug, >>> >>> comparableClassFor() for non-comparables that don't implement >>> Comparable is already as fast as it can be (the 1st check is >>> instanceof Comparable). For other comparables (and >>> non-comparables) that implement Comparable (except for String >>> which is special-cased), we could improve the situation by >>> caching the result. >>> >>> Here's another attempt at that. This time it uses plain old JDK1 >>> stuff, so it actually works even in HashMap (using >>> IdentityHashMap so no danger of circular usage if it is to be >>> applied to CHM also): >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/Class.getGenericDerivative/webrev.01/ >>> >>> >>> With this patch, the results of Bernd's JMH benchmark do give >>> some boost to keys that implement Comparable (badDistWithComp case). >>> >>> These are the results with original JDK9 HashMap: >>> >>> Benchmark (initialSize) Mode Samples Score Score >>> error Units >>> j.t.HashMapCollision.badDistNoComp 16 avgt 6 >>> 3104.047 278.057 ms/op >>> j.t.HashMapCollision.badDistWithComp 16 avgt 6 >>> 2754.499 243.780 ms/op >>> j.t.HashMapCollision.goodDistNoComp 16 avgt 6 >>> 1031.992 26.422 ms/op >>> j.t.HashMapCollision.goodDistWithComp 16 avgt 6 >>> 1082.347 30.981 ms/op >>> >>> And this is with patch applied: >>> >>> Benchmark (initialSize) Mode Samples Score Score >>> error Units >>> j.t.HashMapCollision.badDistNoComp 16 avgt 6 >>> 3081.419 386.125 ms/op >>> j.t.HashMapCollision.badDistWithComp 16 avgt 6 >>> 2116.030 281.160 ms/op >>> j.t.HashMapCollision.goodDistNoComp 16 avgt 6 >>> 1015.224 81.843 ms/op >>> j.t.HashMapCollision.goodDistWithComp 16 avgt 6 >>> 1078.719 38.351 ms/op >>> >>> >>> Caching is performed as part of Class generic types information >>> caching (ClassRepository), so there's no overhead for those that >>> don't need generic types information. All logic is kept inside >>> (C)HM. >>> >>> Regards, Peter >>> >>>> >>>> -Doug >>>> >>> >>> >> > From dl at cs.oswego.edu Sun Jan 11 21:00:33 2015 From: dl at cs.oswego.edu (Doug Lea) Date: Sun, 11 Jan 2015 16:00:33 -0500 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B2CE69.4050300@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> Message-ID: <54B2E471.3090905@cs.oswego.edu> On 01/11/2015 02:26 PM, Peter Levart wrote: > Although majority of entries constitute the bins of size 13 or 14, there's only > a single hashCode value per bin. > > So in this benchmark, treeifying with non-comparable keys is a waste of effort. On the other hand, the waste seems to only cost about 10% in your runs. I wonder why the original report using jdk7 vs jdk8 seemed larger. > > Are there (non-forged) sets of non-comparable keys with hashCodes where > treeifying makes sense? Try using a class like: class FHC { float f; int hashCode() { return Float.floatToIntBits(f); } } and populate with instances with integral values for f. Similarly for doubles. Pre-jdk8, we devised a bit-smearing function that (among other constraints) did OK for float/double keys with integral values, that are not all that rare. With treeification, we don't need to penalize classes with decent hashCodes by bit-smearing to still get OK performance for these kinds of cases where the tree-based hashCode comparison helps more than Comparability per se. Also... It looks like the simplest path to a minor improvement is just to cache internally (your fourth test below). But I now recall not doing this because it adds to footprint and the field could prevent class unloading, for only a small benefit. (Every time HashMap has changed, there have been reports of performance regressions even though typical performance generally improves.) -Doug >> Original JDK9 HashMap: >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3011.738 78.249 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2984.280 48.315 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 682.060 52.341 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 685.705 55.183 ms >> >> Original JDK9 HashMap with TREEIFY_THRESHOLD = 1 << 20: >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 2780.771 236.647 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2541.740 233.429 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 757.364 67.869 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 671.617 54.943 ms >> >> Caching of comparableClassFor (in ClassRepository - good for heterogeneous >> keys too): >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3014.888 71.778 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2279.757 54.159 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 760.743 70.674 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 725.188 67.853 ms >> >> Caching of comparableClassFor (internally - good for homogeneous keys only): >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3026.707 84.571 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2137.296 66.140 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 635.964 8.213 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 685.129 46.783 ms >> >> From peter.levart at gmail.com Sun Jan 11 23:26:35 2015 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 12 Jan 2015 00:26:35 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B2E471.3090905@cs.oswego.edu> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> Message-ID: <54B306AB.8010509@gmail.com> On 01/11/2015 10:00 PM, Doug Lea wrote: > On 01/11/2015 02:26 PM, Peter Levart wrote: > >> Although majority of entries constitute the bins of size 13 or 14, >> there's only >> a single hashCode value per bin. >> >> So in this benchmark, treeifying with non-comparable keys is a waste >> of effort. > > On the other hand, the waste seems to only cost about 10% in your runs. > I wonder why the original report using jdk7 vs jdk8 seemed larger. I don't know. I ran the same benchmark with same VM options on JDK 7 too. Here are all results together: Original JDK 7 HashMap (and JVM): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 2839.458 157.299 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2673.924 187.063 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 686.972 32.928 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 631.001 6.574 ms Original JDK 9 HashMap: Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3011.738 78.249 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2984.280 48.315 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 682.060 52.341 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 685.705 55.183 ms Original JDK 9 HashMap with TREEIFY_THRESHOLD = 1 << 20: Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 2780.771 236.647 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2541.740 233.429 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 757.364 67.869 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 671.617 54.943 ms Caching of comparableClassFor (in ClassRepository - good for heterogeneous keys too): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3014.888 71.778 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2279.757 54.159 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 760.743 70.674 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 725.188 67.853 ms Caching of comparableClassFor (internally - good for homogeneous keys only): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3026.707 84.571 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2137.296 66.140 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 635.964 8.213 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 685.129 46.783 ms > >> >> Are there (non-forged) sets of non-comparable keys with hashCodes where >> treeifying makes sense? > > Try using a class like: > class FHC { float f; int hashCode() { return > Float.floatToIntBits(f); } } > and populate with instances with integral values for f. > > Similarly for doubles. > > Pre-jdk8, we devised a bit-smearing function that (among other > constraints) did OK for float/double keys with integral values, > that are not all that rare. With treeification, we don't need to > penalize classes with decent hashCodes by bit-smearing to still > get OK performance for these kinds of cases where the tree-based > hashCode comparison helps more than Comparability per se. I see. These keys actually have unique or near unique hashCodes but which are not good for power of two length tables without bit-smearing. With tree bins we don't need heavy bit-smearing to get decent performance in speed, but the table gets quite sparse anyway (although this is the smaller of the space overheads - tree nodes are bigger). For example, for 1M integral Floats, we get the following: >>> Float ... Capacity: 2097152 Load factor: 0.75 Size: 1000000 Bin sizes: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 6*0 7*0 8*0 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 Empty bins: 93.8 % Unique hash codes per bin: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 6*0 7*0 8*0 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 > > Also... > > It looks like the simplest path to a minor improvement is > just to cache internally (your fourth test below). But I now > recall not doing this because it adds to footprint and > the field could prevent class unloading, for only a small > benefit. Footprint, yes (one reference field in HM instance), while class unloading is taken care of using WeakReference: http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HomogeneousKeysCache/webrev.01/ > > (Every time HashMap has changed, there have been reports of > performance regressions even though typical performance > generally improves.) > > -Doug Regards, Peter > > >>> Original JDK9 HashMap: >>> >>> Benchmark (initialSize) Mode Samples >>> Score Score error Units >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>> 3011.738 78.249 ms >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>> 2984.280 48.315 ms >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>> 682.060 52.341 ms >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>> 685.705 55.183 ms >>> >>> Original JDK9 HashMap with TREEIFY_THRESHOLD = 1 << 20: >>> >>> Benchmark (initialSize) Mode Samples >>> Score Score error Units >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>> 2780.771 236.647 ms >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>> 2541.740 233.429 ms >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>> 757.364 67.869 ms >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>> 671.617 54.943 ms >>> >>> Caching of comparableClassFor (in ClassRepository - good for >>> heterogeneous >>> keys too): >>> >>> Benchmark (initialSize) Mode Samples >>> Score Score error Units >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>> 3014.888 71.778 ms >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>> 2279.757 54.159 ms >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>> 760.743 70.674 ms >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>> 725.188 67.853 ms >>> >>> Caching of comparableClassFor (internally - good for homogeneous >>> keys only): >>> >>> Benchmark (initialSize) Mode Samples >>> Score Score error Units >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>> 3026.707 84.571 ms >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>> 2137.296 66.140 ms >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>> 635.964 8.213 ms >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>> 685.129 46.783 ms >>> >>> > From david.holmes at oracle.com Mon Jan 12 02:20:58 2015 From: david.holmes at oracle.com (David Holmes) Date: Mon, 12 Jan 2015 12:20:58 +1000 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B0083C.1090409@oracle.com> References: <54B0083C.1090409@oracle.com> Message-ID: <54B32F8A.5020006@oracle.com> Hi Daniel, Looking at the hotspot part ... On 10/01/2015 2:56 AM, Daniel Fuchs wrote: > Hi, > > Please find below a webrev for: > > 8068730: Increase the precision of the implementation > of java.time.Clock.systemUTC() > https://bugs.openjdk.java.net/browse/JDK-8068730 > > The java.time.Clock.system() method (and variants thereof) are > specified to "obtain a clock that returns the current instant > using best available system clock". However the current > implementation of the clock returned is based on > System.currentTimeMillis() whereas the underlying native clock > used by System.currentTimeMillis() has often a greater precision > than milliseconds (for instance, on Linux, System.currentTimeMillis() > is based on gettimeofday, which offers microseconds precision). > > This patch enhances the implementation of the > java.time.Clock.SystemClock, so that it offer at least the > same precision than the underlying clock available on the system. > > There is no change in the public API. > > http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.00/ > > Some more details on the patch: > > native (hotspot) side: > > - adds a new method to the os class: > os::javaTimeSystemUTC(jlong &seconds, jlong &nanos) > which allows to get the time in the form of a number > of seconds and number of nanoseconds > (see os.hpp and various os_.cpp files) Everything seems consistent with javaTimeMillis code. Only nit on Windows is that windows_time_stamp isn't a very descriptive name (no idea what it returns) - perhaps windows_time_as_micros and adjust the calculations accordingly? > - adds a JVM_GetNanoTimeAdjustment method, which takes > an offset (in seconds) as parameter and returns a > nano time adjustment compared to the offset. > Calls os::javaTimeSystemUTC to compute the adjustment > (see jvm.cpp) IIUC JVM_GetNanoTimeAdjustment(jlong offset_secs) returns the difference in nanoseconds between current UTC time (from Epoch) and the UTC time since the Epoch that offset_secs represents - is that right? I don't find the name particularly intuitive and I guess you renamed it somewhere along the way as the mapfiles are no longer in alphabetical order (but would be if this was JVM_GetTimeXXXX). So please fix mapfiles. > java (jdk) side: > > - adds a native sun.misc.VM.getNanoTimeAdjustment method > (which is bound to JVM_GetNanoTimeAdjustment) > (see VM.java and VM.c) Just curious: why the indirection through sun.misc.VM? Cheers, David > - modifies java.time.Clock.SystemClock to call the new > sun.misc.VM.getNanoTimeAdjustment instead of > System.currentTimeMillis. > > - fixes java.util.Formatter - which wasn't expecting > greater than millisecond precision. > > testing: > > - A new test is added to test sun.misc.VM.getNanoTimeAdjustment > In particular it checks the edge cases. > - New tests are added to TestClock_System.java to check for > the increased precision. > There is also a test for the edge cases there. > - Some java.time tests where tripped by the increased precision, > and had to be modified to take that into account > > Note: comparing the new os::javaTimeSystemUTC with os::javaTimeMillis > can help in reviewing the changes. > > best regards, > > -- daniel From peter.levart at gmail.com Mon Jan 12 08:26:36 2015 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 12 Jan 2015 09:26:36 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B306AB.8010509@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> <54B306AB.8010509@gmail.com> Message-ID: <54B3853C.2020507@gmail.com> On 01/12/2015 12:26 AM, Peter Levart wrote: > With tree bins we don't need heavy bit-smearing to get decent > performance in speed, but the table gets quite sparse anyway (although > this is the smaller of the space overheads - tree nodes are bigger). > For example, for 1M integral Floats, we get the following: > > >>> Float ... > Capacity: 2097152 > Load factor: 0.75 > Size: 1000000 > Bin sizes: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 6*0 > 7*0 8*0 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 > Empty bins: 93.8 % > Unique hash codes per bin: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 6*0 > 7*0 8*0 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 Hi Doug, Did bit smearing function used in JDK 7 have negative impact on any hashCodes or was it replaced with simpler one just because it is not needed when tree bins are used? At least on my i7 PC the old bit smearing compared to new one does not show that much difference in relative speed (jdk7's is about 20% slower) while absolute time is in range of a few ns: Benchmark Mode Samples Score Error Units j.t.HashSmearingTest.testHash7 avgt 60 3.472 ? 0.105 ns/op j.t.HashSmearingTest.testHash8 avgt 60 2.881 ? 0.087 ns/op Using the following JMH benchmark: @State(Scope.Thread) @Fork(6) @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.NANOSECONDS) @Warmup(iterations = 7) @Measurement(iterations = 10) public class HashSmearingTest { static int hash7(Object key) { if (key == null) return 0; int h = key.hashCode(); h ^= (h >>> 20) ^ (h >>> 12); return h ^ (h >>> 7) ^ (h >>> 4); } static int hash8(Object key) { int h; return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> 16); } private int cnt = ThreadLocalRandom.current().nextInt(); @Benchmark public int testHash7() { return hash7(cnt++); } @Benchmark public int testHash8() { return hash8(cnt++); } } Regards, Peter From andrey.x.nazarov at oracle.com Mon Jan 12 09:25:21 2015 From: andrey.x.nazarov at oracle.com (Andrey Nazarov) Date: Mon, 12 Jan 2015 12:25:21 +0300 Subject: RFR 8067437: New tests for mJRE feature removal. In-Reply-To: <54998A08.5060303@oracle.com> References: <5499484E.70101@oracle.com> <549989C1.5060506@oracle.com> <54998A08.5060303@oracle.com> Message-ID: <54B39301.9060209@oracle.com> Hi Kumar, I've updated patch with your comments. http://cr.openjdk.java.net/~anazarov/8067437/webrev.02/ Also see comments inline. Thank you for review. On 23.12.2014 18:28, Kumar Srinivasan wrote: > Missed clarifying this, I will be sponsoring this patch. > > Kumar > > On 12/23/2014 7:26 AM, Kumar Srinivasan wrote: >> Neil, >> >> Since you contributed the dev work, appreciate if you can provide some >> feedback on the cleanup and new tests. >> >> Andrey, >> Please see embedded comments below: >> >>> Please review new tests for the java launcher. >>> >>> New tests in MultipleJRE.java includes: >>> 1. java -help must not contain information about obsolete flags >>> >>> "-version:", "-jre-restrict-search", "-jre-no-restrict-search" >>> >>> 2. java should emit error if obsolete flag is specified (combo test) >>> >>> 3. java should ignore manifest attributes: >>> "JRE-Version","JRE-Restrict-Search" >> >> MultipleJRE.java >> Prefer to see braces for "for-while-loops" lines 71, 72, its ok if >> you skip braces for >> one-liner if-then-else clauses. added braces >> >> I would also suggest dumping the TestResult separately upon failure >> to assist in debugging, >> because it will be likely be "yours truly" who will triage future bugs. >> >> 112 if (!tr.testStatus) { >> - 113 throw new RuntimeException("test case: failed\n" + >> tr.toString()); >> + System.out.println(tr); >> + throw new RuntimeException("test case: failed\n" + >> cmd); >> 114 } >> added System.out.println(tr) in several places. >> >> >>> >>> Removed tests: >>> >>> 1. Tests covered by MultipleJRE.java are removed in shell script >>> MultipleJRE.sh. Script can be ported to Java by another changeset. >> >> Amen, I want to see MultipleJRE.sh be whittled away and finally removed. >> >>> >>> 2. Removed tests that check java argument parsing through re-exec >>> when another java version specified through flag -version: >>> >>> Do we have tests for argument parsing in java launcher except >>> Arrrghs.java? >> >> Besides Arrrghs.java, there are tests in langtools >> http://hg.openjdk.java.net/jdk9/dev/langtools/file/20475c78a0a6/test/tools/javac/Paths/wcMineField.sh >> >> http://hg.openjdk.java.net/jdk9/dev/langtools/file/20475c78a0a6/test/tools/javac/Paths/MineField.sh Arrrghs contains tests that check parsing of arguments with quotes and wildcard for Windows only. Do you why Unix systems are skipped? >> >> >> So why is the BugId added to this test ? This test no longer tests >> any mJRE features ? Removed the BugID. I planned to fix both issues by one changeset, but then decide to split changeset due to amount of work. The Arrrghs.java no longer tests any mJRE features. >> >> Thanks for doing this!. >> >> Kumar >> >>> >>> Webrev: http://cr.openjdk.java.net/~anazarov/8067437/webrev.00/ >>> >>> >>> >>> >>> --Thanks, Andrey >> > --Thanks, Andrey. From claes.redestad at oracle.com Mon Jan 12 10:07:39 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Mon, 12 Jan 2015 11:07:39 +0100 Subject: RFR: 8068498: Remove constructor dependency on line.separator from PrintWriter and BufferedWriter In-Reply-To: <54AFFDF4.8070006@oracle.com> References: <54AFF349.6060503@oracle.com> <54AFFDF4.8070006@oracle.com> Message-ID: <54B39CEB.1000107@oracle.com> On 01/09/2015 05:12 PM, Alan Bateman wrote: >> webrev: http://cr.openjdk.java.net/~redestad/8068498/webrev.01/ >> bug: https://bugs.openjdk.java.net/browse/JDK-8068498 > This looks good to me and the behavior change looks reasonable for a > major release. Thanks! I was expecting some opposition, but I guess abusing system properties to affect behavior is not a common practice. I'll update the copyright headers and push this later this week. /Claes > > -Alan From chris.hegarty at oracle.com Mon Jan 12 11:37:06 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 12 Jan 2015 11:37:06 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54AEE440.2060709@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> <54AE4081.2090600@zeus.net.au> <54AEA30A.9090203@redhat.com> <97533E9F-0281-4EDD-9504-C38DE9453BE8@oracle.com> <54AEE440.2060709@oracle.com> Message-ID: <54B3B1E2.9070603@oracle.com> On 08/01/15 20:10, Brian Goetz wrote: >> 1) Validate invariants >> >> A clear and easy to understand mechanism that can validate the >> deserialized >> fields. Does not prevent the use of final fields, as the >> serialization framework >> will be responsible for setting them. Something along the lines >> of what David >> suggested: >> >> private static void validate(GetField fields) { >> if (fields.getInt("lo") > fields.getInt("hi")) { ... } >> } >> >> This could be a ?special? method, or annotation driven. TBD. >> >> Note: the validate method is static, so the object instance is >> not required to >> be created before running the validation. > > Sort of... > > This is true if the fields participating in the invariant are > primitives. But if they're refs, what do you do? What if you want to > validate something like > > count == list.size() // fields are int count, List list > > ? Then wouldn't GetField.getObject have to deserialize the object > referred to by that field? Yes it would. For clarity, I would like to describe how things currently work. 1) Allocate a new instance of the deserialized type. 2) Call the first non-Serializable types no-arg constructor ( may be j.l.Object ). 3) For each type in the deserialized types hierarchy, starting with the top most ( closest to j.l.Object ), 3a) create objects representing all fields values for the type [this step is recursive and will go to 1 until all non-primitive types have been created ] 3b) [ holder for invariant validation ] 3c) assign objects to their respective members of the containing instance [ For simplicity, ignore cyclic references are readObjectXXX for now, I will address them separately. ] Without any user visible side-effects, no readObjectXXX methods, it would appear that there is no reason why 1 & 2 must happen before 3a. Since objects representing field values are created recursively, then all the objects representing the field values are created, per class in the hierarchy, before being assigned. If we have no reaObjectXXX methods, then the objects being created in 3a could be stored locally, repeating 3a as needed, and only assign after all types in the hierarchy have been walked. Essentially the sequence of steps could be, 3, 3a+, [3b], 1, 2, 3c+. Given this, an the invariant could be validated at 3b, without the need for the creation of the containing object. Cyclic references: If we encounter a cyclic reference, then we can "de-optimize"; stop, create the required instances reachable in the graph, fill in whatever fields are currently known, then continue. readObjectXXX: Since these are instance methods then they must have visibility to any deserialized state in the super type. These can be handled similarly to cyclic references, but can be determine up front, at step 3, rather than in the flow. -Chris. From daniel.fuchs at oracle.com Mon Jan 12 11:36:14 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 12 Jan 2015 12:36:14 +0100 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: References: <54B0083C.1090409@oracle.com> Message-ID: <54B3B1AE.5060108@oracle.com> On 09/01/15 19:26, Stephen Colebourne wrote: > Well that is a nice surprise ;-) And I think the implementation via an > adjustment is very sensible. Thanks Stephen :-) > In java.time.Instant, a readObject() has been added. However, I don't > believe this will ever be called because Instant has a writeReplace() > method and so is not deserialized. (There may be some security related > "evil serialization stream" reason why readObject() should exist, I > can't say). Well - I don't know the history of that - so I can't comment. I have not touched java.time.Instant in this patch. I added a readObject to Clock.SystemClock to reinitialize the new transient offset field. It is not strictly required, but for consistency I believe it's better. > In java.util.Formatter, there should be no reason to retain use of the > MILLI_OF_SECOND, as any TemporalAccessor that can return > NANO_OF_SECOND should also return MILLI_OF_SECOND, however the spec is > not quite tight enough to guarantee that sadly. As such, I think the > catch will have to be retained. OK. Thanks for confirming. > TestFormatter has some commented out System.out statements that should > probably be removed. Right. Done. > I suspect that this change will break some user code, but it certainly > doesn't break the spec. As such, I think the change should go in. Thanks :-) > I do believe that this change means that a new method should be added > to Clock however: > > public static Clock tickMillis(ZoneId zone) { > return new TickClock(system(zone), NANOS_PER_MILLI); > } > > While this can be achieved without a new method, the API would feel > slightly strange without it now better-than-milli clocks exist. I > recommend raising a separate RFE to track this. Ah - I see. I didn't think of that. It looks like a sensible RFE. Agreed. best regards, -- daniel > > Stephen > > > > On 9 January 2015 at 16:56, Daniel Fuchs wrote: >> Hi, >> >> Please find below a webrev for: >> >> 8068730: Increase the precision of the implementation >> of java.time.Clock.systemUTC() >> https://bugs.openjdk.java.net/browse/JDK-8068730 >> >> The java.time.Clock.system() method (and variants thereof) are >> specified to "obtain a clock that returns the current instant >> using best available system clock". However the current >> implementation of the clock returned is based on >> System.currentTimeMillis() whereas the underlying native clock >> used by System.currentTimeMillis() has often a greater precision >> than milliseconds (for instance, on Linux, System.currentTimeMillis() >> is based on gettimeofday, which offers microseconds precision). >> >> This patch enhances the implementation of the >> java.time.Clock.SystemClock, so that it offer at least the >> same precision than the underlying clock available on the system. >> >> There is no change in the public API. >> >> http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.00/ >> >> Some more details on the patch: >> >> native (hotspot) side: >> >> - adds a new method to the os class: >> os::javaTimeSystemUTC(jlong &seconds, jlong &nanos) >> which allows to get the time in the form of a number >> of seconds and number of nanoseconds >> (see os.hpp and various os_.cpp files) >> >> - adds a JVM_GetNanoTimeAdjustment method, which takes >> an offset (in seconds) as parameter and returns a >> nano time adjustment compared to the offset. >> Calls os::javaTimeSystemUTC to compute the adjustment >> (see jvm.cpp) >> >> java (jdk) side: >> >> - adds a native sun.misc.VM.getNanoTimeAdjustment method >> (which is bound to JVM_GetNanoTimeAdjustment) >> (see VM.java and VM.c) >> >> - modifies java.time.Clock.SystemClock to call the new >> sun.misc.VM.getNanoTimeAdjustment instead of >> System.currentTimeMillis. >> >> - fixes java.util.Formatter - which wasn't expecting >> greater than millisecond precision. >> >> testing: >> >> - A new test is added to test sun.misc.VM.getNanoTimeAdjustment >> In particular it checks the edge cases. >> - New tests are added to TestClock_System.java to check for >> the increased precision. >> There is also a test for the edge cases there. >> - Some java.time tests where tripped by the increased precision, >> and had to be modified to take that into account >> >> Note: comparing the new os::javaTimeSystemUTC with os::javaTimeMillis >> can help in reviewing the changes. >> >> best regards, >> >> -- daniel From tim.bell at oracle.com Mon Jan 12 11:43:21 2015 From: tim.bell at oracle.com (Tim Bell) Date: Mon, 12 Jan 2015 03:43:21 -0800 Subject: RFR: JDK-8042707: Source changes needed to build JDK 9 with Visual Studio 2013 (VS2013) In-Reply-To: <54AE9A6D.2030101@oracle.com> References: <54AE9A6D.2030101@oracle.com> Message-ID: <54B3B359.208@oracle.com> Erik: Looks good to me. Thanks for picking this up - I struggled with the problem for months. Tim > Please review this patch, which adds support for building with > different versions of Visual Studio and in particular adds support for > VS2013. In order to control which version to use, I've introduced a > new configure parameter "--with-toolchain-version". On windows, this > parameter will have the valid values 2010, 2012 and 2013. The default > is still 2010. Note that 2012 was added for convenience, but has not > been tested to actually work. The longer term goal is to switch the > official compiler used for JDK 9 to VS2013. This is just the first step. > > The main difference that needed to be addressed was that > _STATIC_CPPLIB is no longer supported since VS2012, so we will now > have to bundle another runtime dll (MSVCP). Also, since the build > needs to be compatible with multiple versions of VS, we can no longer > hard code the name msvcr100.dll. I changed the names of the dlls into > macros that get added to the preprocessor command line. Note that the > implementation for msvcp*.dll in java_md.c could perhaps be more > elegant. I'm not familiar enough with the APIs, but if someone would > like to improve on it, please do. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8042707 > Webrevs: > http://cr.openjdk.java.net/~erikj/8042707/webrev.root.01/ > http://cr.openjdk.java.net/~erikj/8042707/webrev.jdk.01/ > > Here is a patch for make/jprt.properties that adds new build and test > platforms using the new compilers for windows and macosx. If you apply > this and run JPRT, windows and mac platforms will be built and tested > with both the old and the new compilers. I would like to commit this > change too, but will do it separately later. > > diff -r ef5c7075496d make/jprt.properties > --- a/make/jprt.properties > +++ b/make/jprt.properties > @@ -115,6 +115,12 @@ > ${my.i586.default.build.configure.args} \ > ${jprt.productOpen.build.configure.args} > > +jprt.windows_i586_6.2.build.configure.args= \ > + --with-toolchain-version=2013 \ > + ${jprt.i586.build.configure.args} > +jprt.windows_x64_6.2.build.configure.args= \ > + --with-toolchain-version=2013 > + > ######## > # > # Build targets and options (default/jdk) > @@ -130,8 +136,11 @@ > linux_i586_2.6-{product|fastdebug}, \ > linux_x64_2.6-{product|fastdebug}, \ > macosx_x64_10.7-{product|fastdebug}, \ > - windows_i586_6.1-{product|fastdebug}, \ > - windows_x64_6.1-{product|fastdebug} > + macosx_x64_10.9-{product|fastdebug}, \ > + windows_i586_6.1-{product|fastdebug}, \ > + windows_x64_6.1-{product|fastdebug}, \ > + windows_i586_6.2-{product|fastdebug}, \ > + windows_x64_6.2-{product|fastdebug} > > # Test target list (no fastdebug & limited c2 testing) > my.test.target.set= \ > @@ -140,8 +149,11 @@ > linux_i586_2.6-product-{c1|c2}-TESTNAME, \ > linux_x64_2.6-product-c2-TESTNAME, \ > macosx_x64_10.7-product-c2-TESTNAME, \ > + macosx_x64_10.9-product-c2-TESTNAME, \ > windows_i586_6.1-product-c1-TESTNAME, \ > - windows_x64_6.1-product-c2-TESTNAME > + windows_x64_6.1-product-c2-TESTNAME, \ > + windows_i586_6.2-product-c1-TESTNAME, \ > + windows_x64_6.2-product-c2-TESTNAME > > # Default vm test targets (testset=default) > my.test.targets.default= \ > > /Erik From scolebourne at joda.org Mon Jan 12 11:40:57 2015 From: scolebourne at joda.org (Stephen Colebourne) Date: Mon, 12 Jan 2015 11:40:57 +0000 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B3B1AE.5060108@oracle.com> References: <54B0083C.1090409@oracle.com> <54B3B1AE.5060108@oracle.com> Message-ID: On 12 January 2015 at 11:36, Daniel Fuchs wrote: >> In java.time.Instant, a readObject() has been added. However, I don't >> believe this will ever be called because Instant has a writeReplace() >> method and so is not deserialized. (There may be some security related >> "evil serialization stream" reason why readObject() should exist, I >> can't say). > > I have not touched java.time.Instant in this patch. Oops, I misread the webrev! >> I do believe that this change means that a new method should be added >> to Clock however: > > Ah - I see. I didn't think of that. It looks like a sensible > RFE. Agreed. I look forward to that webrev! Stephen From chris.hegarty at oracle.com Mon Jan 12 11:51:03 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 12 Jan 2015 11:51:03 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54AEFEA7.80503@redhat.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> <54AE4081.2090600@zeus.net.au> <54AEA30A.9090203@redhat.com> <97533E9F-0281-4EDD-9504-C38DE9453BE8@oracle.com> <54AEE440.2060709@oracle.com> <54AEFEA7.80503@redhat.com> Message-ID: <54B3B527.7000609@oracle.com> On 08/01/15 22:03, David M. Lloyd wrote: >> .... >>> private static void validate(GetField fields) { >>> if (fields.getInt("lo") > fields.getInt("hi")) { ... } >>> } > ... > In fact you cannot validate invariants across multiple objects at all > using this method *or* readObject() (existing or enhanced) unless the > object in question is an enum, Class, or String (and a few other special > cases) because you can't rely on initialization order during > deserialization. That's the very reason why OIS#registerValidation() > even exists - inter-object validation is not possible until after the > root-most readObject has completed, which is the time when validations > are executed. Robust validation of a given object class may require two > stages - "near" validation and "spanning" validation - to fully > validate. However the readObject() approach and its proposed variations > (including the static validate() version) can still be useful for > fail-fast and non-complex validations; you just have to understand that > (just as today) any Object you examine might not be fully initialized yet. If I may, I'd like to build a little on this proposal: 1) Specify that validate is called down the hierarchy, from j.l.Object. 2) Provide access to persistent supertype's fields, so they can participate in the validation. public static abstract class GetField { .... /** * Returns the persistent fields of the supertype, read from * the stream, or null if the the supertype is not Serializable. */ public abstract GetField superTypeFields(); } -Chris. From daniel.fuchs at oracle.com Mon Jan 12 12:00:34 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 12 Jan 2015 13:00:34 +0100 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B32F8A.5020006@oracle.com> References: <54B0083C.1090409@oracle.com> <54B32F8A.5020006@oracle.com> Message-ID: <54B3B762.9060508@oracle.com> On 12/01/15 03:20, David Holmes wrote: > Hi Daniel, > > Looking at the hotspot part ... > > On 10/01/2015 2:56 AM, Daniel Fuchs wrote: Hi David, [...] >> http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.00/ >> >> Some more details on the patch: >> >> native (hotspot) side: >> >> - adds a new method to the os class: >> os::javaTimeSystemUTC(jlong &seconds, jlong &nanos) >> which allows to get the time in the form of a number >> of seconds and number of nanoseconds >> (see os.hpp and various os_.cpp files) > > Everything seems consistent with javaTimeMillis code. Only nit on > Windows is that windows_time_stamp isn't a very descriptive name (no > idea what it returns) - perhaps windows_time_as_micros and adjust the > calculations accordingly? I'm not a big fan of the current name either. I would gladly rename it. I did think of windows_to_java_time_micros, but it actually returns tenth of micros - so it would be lying... Is there a good name for 'tenth of micros'? Otherwise - I assume I could rename it into: windows_to_java_time_max_precision Would that be a better name? Or would windows_time_as_tenth_of_micros be better? >> - adds a JVM_GetNanoTimeAdjustment method, which takes >> an offset (in seconds) as parameter and returns a >> nano time adjustment compared to the offset. >> Calls os::javaTimeSystemUTC to compute the adjustment >> (see jvm.cpp) > > IIUC JVM_GetNanoTimeAdjustment(jlong offset_secs) returns the difference > in nanoseconds between current UTC time (from Epoch) and the UTC time > since the Epoch that offset_secs represents - is that right? Yes. Exactly. > I don't find the name particularly intuitive Well - I borrowed the name from the java time API. Instant has a factory method that takes a number of seconds and a 'nanoAdjustement': http://docs.oracle.com/javase/8/docs/api/java/time/Instant.html#ofEpochSecond-long-long- > and I guess you renamed it > somewhere along the way as the mapfiles are no longer in alphabetical > order (but would be if this was JVM_GetTimeXXXX). So please fix mapfiles. You have an eagle's eye ;-) Yes - in my early prototype it was named JVM_GetTimeStamp - which was not a great name. Mapfiles fixed. Thanks for spotting that! >> java (jdk) side: >> >> - adds a native sun.misc.VM.getNanoTimeAdjustment method >> (which is bound to JVM_GetNanoTimeAdjustment) >> (see VM.java and VM.c) > > Just curious: why the indirection through sun.misc.VM? In my early prototype I didn't want to add new .c files, and so I looked for a home for this new method. When I asked around whether adding it to sun.misc.VM would be appropriate nobody seemed too shocked. If you think it's better I could make it a private static method in java.time.Clock - but then I'd have to change the new unit test for that (GetNanoTimeAdjustment.java) to use reflection (and would probably also need to add an "initialize" native method to java.time.Clock as well). Here is the new webrev with Stephen & your feedback included http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.01/ (windows_time_stamp not renamed yet) Thanks a lot David! -- daniel > > Cheers, > David > >> - modifies java.time.Clock.SystemClock to call the new >> sun.misc.VM.getNanoTimeAdjustment instead of >> System.currentTimeMillis. >> >> - fixes java.util.Formatter - which wasn't expecting >> greater than millisecond precision. >> >> testing: >> >> - A new test is added to test sun.misc.VM.getNanoTimeAdjustment >> In particular it checks the edge cases. >> - New tests are added to TestClock_System.java to check for >> the increased precision. >> There is also a test for the edge cases there. >> - Some java.time tests where tripped by the increased precision, >> and had to be modified to take that into account >> >> Note: comparing the new os::javaTimeSystemUTC with os::javaTimeMillis >> can help in reviewing the changes. >> >> best regards, >> >> -- daniel From ivan.gerasimov at oracle.com Mon Jan 12 12:03:37 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Mon, 12 Jan 2015 15:03:37 +0300 Subject: RFR(S): 8067471: Use private static final char[0] for empty Strings In-Reply-To: <54A3F985.1080405@oracle.com> References: <54915954.6090804@oracle.com> <54A2A6A6.9060303@oracle.com> <54A2CC6B.7080002@oracle.com> <54A2D14C.5020303@oracle.com> <54A3F985.1080405@oracle.com> Message-ID: <54B3B819.8080507@oracle.com> On 31.12.2014 16:26, Lev Priima wrote: > Thanks Ivan! > > I've updated: http://cr.openjdk.java.net/~lpriima/8067471/webrev.05/. > I would say I'm Okay with the latest variant, if the performance team doesn't have any objections. Sincerely yours, Ivan > Best Regards, > Lev > > > From dl at cs.oswego.edu Mon Jan 12 12:15:26 2015 From: dl at cs.oswego.edu (Doug Lea) Date: Mon, 12 Jan 2015 07:15:26 -0500 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B3853C.2020507@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> <54B306AB.8010509@gmail.com> <54B3853C.2020507@gmail.com> Message-ID: <54B3BADE.8070607@cs.oswego.edu> On 01/12/2015 03:26 AM, Peter Levart wrote: > Did bit smearing function used in JDK 7 have negative impact on any hashCodes or > was it replaced with simpler one just because it is not needed when tree bins > are used? I don't have detailed records, but in pre-release jdk8 tests, smearing was detectably but not hugely slower for the cases of String, identity-hash, and Integer keys. It is worth rechecking with jdk9. We believe these cases together account for the majority of HashMap instances -- based on some old instrumentation of a reference load, around 90% of the them. (It would be great if someone re-ran this on a more recent corpus.) The design goal of HashMap is to work best in common cases, and to work well in all others except pathological cases of non-comparable elements with many duplicate hashCodes. One issue that might be causing treeified bins to be worse than they would otherwise be is that the recursion in TreeBin.find makes it harder for compilers (JITs) to perform class analysis to determine concrete types, leading to more virtual dispatching and slower code. Someone from a compiler group once asked whether this code could be changed to help JITs. Allocating emulated stack-frames during lookups seems not to work well, but it is possible that a scheme that temporarily reused internal node pointers might help. I haven't looked into this. -Doug From claes.redestad at oracle.com Mon Jan 12 12:26:31 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Mon, 12 Jan 2015 13:26:31 +0100 Subject: RFR(S): 8067471: Use private static final char[0] for empty Strings In-Reply-To: <54B3B819.8080507@oracle.com> References: <54915954.6090804@oracle.com> <54A2A6A6.9060303@oracle.com> <54A2CC6B.7080002@oracle.com> <54A2D14C.5020303@oracle.com> <54A3F985.1080405@oracle.com> <54B3B819.8080507@oracle.com> Message-ID: <54B3BD77.6020907@oracle.com> On 01/12/2015 01:03 PM, Ivan Gerasimov wrote: > > On 31.12.2014 16:26, Lev Priima wrote: >> Thanks Ivan! >> >> I've updated: http://cr.openjdk.java.net/~lpriima/8067471/webrev.05/. >> > > I would say I'm Okay with the latest variant, if the performance team > doesn't have any objections. No objection from me, at least. Re-ran some of the micros that have been floating around in this thread now and got similar (congruent?) results. /Claes > > Sincerely yours, > Ivan > >> Best Regards, >> Lev >> >> >> > From ivan.gerasimov at oracle.com Mon Jan 12 12:42:03 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Mon, 12 Jan 2015 15:42:03 +0300 Subject: RFR: 8067951: System.loadLibrary cannot find library when path contains quoted entry In-Reply-To: <54AC4FF7.6050908@oracle.com> References: <54996EB6.9080908@oracle.com> <29A1F964-41A3-497D-8FB7-4574EF2CFBF3@oracle.com> <549A992F.7030307@oracle.com> <549B14FE.7070800@oracle.com> <549C2341.6010807@oracle.com> <549D3245.8070508@oracle.com> <54A7AD40.7080108@oracle.com> <54A82949.4010308@oracle.com> <54A9996F.1010109@oracle.com> <54A9A15F.9000105@oracle.com> <54AAB050.4010001@Oracle.com> <54AACE94.2000802@oracle.com> <54AAD3A6.9010200@Oracle.com> <54AAE19F.4090102@oracle.com> <54AAF717.3070602@oracle.com> <54AAFE5A.5070406@oracle.com> <54ABE7D4.5010502@oracle.com> <54AC2A63.1040505@oracle.com> <54AC4FF7.6050908@oracle.com> Message-ID: <54B3C11B.9090005@oracle.com> Hi everyone! May I assume the last webrev is good to go? - It uses static final in the if clause, so that the additional code is eliminated on Unix; - Quotation marks are used as new delimiter, so it's impossible for them to collide with whatever is found in the property value; - The last loop is as small as was the original, though handles empty paths more accurately. Sincerely yours, Ivan On 07.01.2015 0:13, Ivan Gerasimov wrote: > And here's the update: > http://cr.openjdk.java.net/~igerasim/8067951/6/webrev/ > > As you suggested, the psCount is calculated in the first loop along > the way. > Another change is that a quotation mark is used as the new path > separator. > > I also thought it's better to add a short comment, as now it may look > confusing at a first glance. > > Sincerely yours, > Ivan > > On 06.01.2015 21:33, Xueming Shen wrote: >> Hi Ivan, >> >> Yes, it looks better and should run faster as it doesn't need to >> worry about quote >> in second loop. I was a little hesitated when suggested to replace >> the ps with 0, >> though I'm pretty much sure a \u0000 should never be a legal >> character in a >> windows' path :-) >> >> Anyone can think about any possibility of having an embedded \0 >> character in path? >> >> Nitpick: >> >> the "posCount" probably can be calculated in the "quote handling" loop ? >> >> if (ClassloaderHelper.allowQuotedPathElements ....) { >> ... >> if (ch == ps) { ch = '\0'; psCount++; } >> buf[bufLen++] = ch; >> ... >> } else { >> for (int i = ldPath.indexOf(ps); i >=0 ...) { >> } >> } >> >> otherwise, it looks fine for me. >> >> -Sherman >> >> On 01/06/2015 05:49 AM, Ivan Gerasimov wrote: >>> Hi Sherman! >>> >>> I took your suggestion and rewrote the method to moved the logic, >>> which removes the quotes to the top. >>> I think the code became cleaner, so thank you for suggestion! >>> >>> Here's the updated webrev: >>> http://cr.openjdk.java.net/~igerasim/8067951/5/webrev/ >>> >>> Sincerely yours, >>> Ivan >>> >>> >>> On 06.01.2015 0:12, Xueming Shen wrote: >>>> On 01/05/2015 12:41 PM, Ivan Gerasimov wrote: >>>>> Thanks Sherman! >>>>> >>>>> On 05.01.2015 22:10, Xueming Shen wrote: >>>>>> >>>>>> Just wonder if we really need that "inQuotes" logic here? A >>>>>> straightforward approach might >>>>>> be "every time you have a quote, skip everything until you hit >>>>>> another one, when counting, >>>>>> or copy everything into the buffer until hit another one, when >>>>>> copying" ? >>>>>> >>>>> I agree it would work, but, in my opinion, it would be a bit more >>>>> complicated. >>>>> The counting loop would look something like this: >>>>> ------------------------------------ >>>>> outerLoop: for (int i = 0; i < ldLen; ++i) { >>>>> char ch = ldPath.charAt(i); >>>>> if (mayBeQuoted && ch == '\"') { >>>>> thereAreQuotes = true; >>>>> for (++i; i < ldLen; ++i) { >>>>> if (ldPath.charAt(i) == '\"') { >>>>> continue outerLoop; >>>>> } >>>>> } >>>>> break; // unpaired quote >>>>> } else if (ch == ps) { >>>>> psCount++; >>>>> } >>>>> } >>>>> ------------------------------------ >>>>> which is 3 lines longer, comparing to the loop with inQuotes flag. >>>>> >>>> >>>> It does not seem like we are doing anything special for "unpaired >>>> quote" (just ignore it?), >>>> if that is the case, you probably don't need to do anything for it, >>>> the code could be >>>> something like >>>> >>>> for (int i = 0; i < ldLen; ++i) { >>>> char ch = ldPath.charAt(i); >>>> if (mayBeQuoted && ch == '\"') { >>>> thereAreQuotes = true; >>>> while (++i < ldLen && ldPath.charAt(i) != '\"') {} >>>> } else if (ch == ps) { >>>> psCount++; >>>> } >>>> } >>>> >>>> I have not tried to debug the code though :-) Just an opinion here. >>>> >>>> -Sherman >>>> >>>> >>> >> >> >> > > > From scolebourne at joda.org Mon Jan 12 12:49:39 2015 From: scolebourne at joda.org (Stephen Colebourne) Date: Mon, 12 Jan 2015 12:49:39 +0000 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B3B762.9060508@oracle.com> References: <54B0083C.1090409@oracle.com> <54B32F8A.5020006@oracle.com> <54B3B762.9060508@oracle.com> Message-ID: On 12 January 2015 at 12:00, Daniel Fuchs wrote: > I'm not a big fan of the current name either. > I would gladly rename it. I did think of > windows_to_java_time_micros, but it actually returns tenth of > micros - so it would be lying... > Is there a good name for 'tenth of micros'? .NET refers to them as "ticks". Stephen From konstantin.shefov at oracle.com Mon Jan 12 14:37:18 2015 From: konstantin.shefov at oracle.com (Konstantin Shefov) Date: Mon, 12 Jan 2015 17:37:18 +0300 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <19036153-CA73-435C-B1EA-7335D632927E@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <19036153-CA73-435C-B1EA-7335D632927E@oracle.com> Message-ID: <54B3DC1E.1060900@oracle.com> Hello, Chris You have asked "Have you seen NetworkInterface.getName() return names with ?_?, ?.?, or ?:? ,or is this theoretical?" I can answer your question. I have named one of network interfaces on my Linux desktop like "eth0.1_55". ifconfig says: eth0.1_55 Link encap:Ethernet HWaddr 08:00:27:10:c0:3e inet addr:10.162.82.115 Bcast:10.162.83.255 Mask:255.255.254.0 inet6 addr: 2606:b400:81c:1050:6568:8bee:d39b:68aa/64 Scope:Global inet6 addr: 2606:b400:81c:1044:a00:27ff:fe10:c03e/64 Scope:Global inet6 addr: 2606:b400:81c:1050:a00:27ff:fe10:c03e/64 Scope:Global inet6 addr: fe80::a00:27ff:fe10:c03e/64 Scope:Link inet6 addr: 2606:b400:81c:1044:6568:8bee:d39b:68aa/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1198 errors:0 dropped:0 overruns:0 frame:0 TX packets:255 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:144325 (144.3 KB) TX bytes:31843 (31.8 KB) Simple java code Enumeration nis = NetworkInterface.getNetworkInterfaces(); System.out.println(nis.nextElement().getName()); returns exactly "eth0.1_55". This means we can include dot "." and underscore "_" symbols to URI scope id. -Konstantin On 11.12.2014 18:29, Chris Hegarty wrote: > On 11 Dec 2014, at 11:09, Konstantin Shefov wrote: > >> CC'ed core-libs-dev at openjdk.java.net >> >> On 10.12.2014 18:21, Konstantin Shefov wrote: >>> Hello, >>> >>> Please, review the bug fix: https://bugs.openjdk.java.net/browse/JDK-6933879 >>> Webrev: http://cr.openjdk.java.net/~kshefov/6933879/webrev.00 >>> >>> It is suggested to add some more symbols allowed for scope id in IPv6 URI, namely "_", "." and ":", because these symbols may be included in network interface names. Now only alphanumeric characters are allowed to be in a scope id. > From Inet6Address, "Textual representation of IPv6 scoped addresses? [1] : > > ?? > As a string. This must be the exact string that is returned by > NetworkInterface.getName() for the particular interface in > question. When an Inet6Address is created in this way, the > numeric scope-id is determined at the time the object is > created by querying the relevant NetworkInterface." > > Have you seen NetworkInterface.getName() return names with ?_?, ?.?, or ?:? ,or is this theoretical ? > > -Chris. > > [1] http://docs.oracle.com/javase/8/docs/api/java/net/Inet6Address.html#scoped > > >>> Thanks >>> -Konstantin From daniel.fuchs at oracle.com Mon Jan 12 14:41:00 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 12 Jan 2015 15:41:00 +0100 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: References: <54B0083C.1090409@oracle.com> <54B32F8A.5020006@oracle.com> <54B3B762.9060508@oracle.com> Message-ID: <54B3DCFC.1010705@oracle.com> On 12/01/15 13:49, Stephen Colebourne wrote: > On 12 January 2015 at 12:00, Daniel Fuchs wrote: >> I'm not a big fan of the current name either. >> I would gladly rename it. I did think of >> windows_to_java_time_micros, but it actually returns tenth of >> micros - so it would be lying... >> Is there a good name for 'tenth of micros'? > > .NET refers to them as "ticks". Ah! That's a good one. 'windows_to_time_ticks' I like this better :-) Thanks Stephen! -- daniel > > Stephen > From david.lloyd at redhat.com Mon Jan 12 15:26:43 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 12 Jan 2015 09:26:43 -0600 Subject: Explicit Serialization API and Security In-Reply-To: <54B3B527.7000609@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> <54AE4081.2090600@zeus.net.au> <54AEA30A.9090203@redhat.com> <97533E9F-0281-4EDD-9504-C38DE9453BE8@oracle.com> <54AEE440.2060709@oracle.com> <54AEFEA7.80503@redhat.com> <54B3B527.7000609@oracle.com> Message-ID: <54B3E7B3.70203@redhat.com> On 01/12/2015 05:51 AM, Chris Hegarty wrote: > > On 08/01/15 22:03, David M. Lloyd wrote: > >>> .... >>>> private static void validate(GetField fields) { >>>> if (fields.getInt("lo") > fields.getInt("hi")) { ... } >>>> } > >> ... >> In fact you cannot validate invariants across multiple objects at all >> using this method *or* readObject() (existing or enhanced) unless the >> object in question is an enum, Class, or String (and a few other special >> cases) because you can't rely on initialization order during >> deserialization. That's the very reason why OIS#registerValidation() >> even exists - inter-object validation is not possible until after the >> root-most readObject has completed, which is the time when validations >> are executed. Robust validation of a given object class may require two >> stages - "near" validation and "spanning" validation - to fully >> validate. However the readObject() approach and its proposed variations >> (including the static validate() version) can still be useful for >> fail-fast and non-complex validations; you just have to understand that >> (just as today) any Object you examine might not be fully initialized >> yet. > > > If I may, I'd like to build a little on this proposal: > > 1) Specify that validate is called down the hierarchy, from > j.l.Object. > > 2) Provide access to persistent supertype's fields, so they can > participate in the validation. > > public static abstract class GetField { > .... > > /** > * Returns the persistent fields of the supertype, read from > * the stream, or null if the the supertype is not Serializable. > */ > public abstract GetField superTypeFields(); > } Maybe limited to accessible fields? -- - DML From chris.hegarty at oracle.com Mon Jan 12 15:56:53 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 12 Jan 2015 15:56:53 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54B0CE06.2020408@zeus.net.au> References: <54B0CE06.2020408@zeus.net.au> Message-ID: <54B3EEC5.5040805@oracle.com> On 10/01/15 07:00, Peter Firmstone wrote: > Again, thank you all for engaging in discussion of this very difficult > topic. > > While we can't presently check intra object dependencies during > deserialization with readObject(), the examples I provide can do this. I have replied to Davids mail with a small change to GetField ( added superTypeFields() ) to return the deserialized supertypes fields. This gives subtypes the ability to check values of the supertypes persistent state. As with your original proposal, it is limited to the persistent state as read off the stream, and not the transient state, but I think it gets us most of the way there. In your original proposal, it looks quite cumbersome to hook up the static validator method in the constructor hierarchy ( as it is to do this with standard constructors too ). It also relies on the fact that the subtype has to create an instance of the supertype. I just wonder if we can push on the alternative static validator proposal to come up with something a but more attractive ( maybe not! ). > ObjectInputValidation is sufficient for enforcing business rules, > however it can't stop an attacker. By the time the validator runs, the > attackers object of choice has been instantiated and it's game over. > That's right, the attacker may choose any Serializable class from the > classpath, and may even extend non serializable classes, with a zero arg > constructor, such as ClassLoader. > > To trust deserialization of an Object, as opposed to a String or > primitive, the ObjectInputStream (or overriding subclass) must limit the > classes allowed to be deserialized. Agreed. I am working on fleshing out a more concrete proposal on confinement, that I mentioned in previous mails. Hopefully, I will have something in the next few days. > For a class to be trusted, it must be trusted to check its parameters > and enforce its invariants during object deserialization and to not > deserialize with priviliged context. > > Validating an entire object graph's invariants, requires each class in > the graph to take responsibility for validating and enforcing its own > invariants. Agreed, and I think that the revised static validator method gives us this. > As shown in my earlier example, intra class invariants can be enforced > using Serialization constructors, from which static methods are called > to check invariants prior to super class constructors being called. Yes, but it is cumbersome and easy to make mistakes. > Presently, I override ObjectInputStream and use a Permission called > DeserializationPermission to limit classes that can be deserialized. > Untrusted connections are run from unprivileged context to limit classes > that can be instantiated, while those with trusted connections are run > with a Subject that allows any class to be deserialized. Interesting. I think there is overlap here with confinement. > ReadSerial, could do this: > Class c = rs.getType("foo"); > > And > > Foo f = rs.getObject("foo", Foo.class); > > Which performs instanceof type check, (prior to Object deserialization, > if first time deserialized, otherwise after) and generic complile time > type checked method return. Thus we must restrict the classes that can > be deserialized with ObjectInputStream. Have you seen the changes I proposing for failure atomicity, preliminary webrev http://cr.openjdk.java.net/~chegar/failureAtomicity/ .. and I think we can go further than this, creating the containing object lazily, if there are no readObjectXXX methods in the hierarchy. > If Foo is mutable and we want a private copy, the caller needs to copy > or clone it before checking invariants, as it would any mutable > constructor parameter. > > Each level of validation is the responsibility of the component with the > most knowledge. > > 1. Trusted class lists - administration, it might change. > 2. Object invariants and intra object invariants - Classes, not objects. > 3. Business rules, but not security - ObjectInputValidation. > > So summing up, in order to secure deserialization we must validate all > data input, preferably before allowing object instantiation. Yes, where possible. If we can phase out readObjectXXX and replace it with a static validator, then I think this is possible. > Once an object has been constructed an attacker can gain a reference, > whether by a finalizer attack or some other cleverly crafted stream, the > attacker cannot obtain a reference to an object that doesn't exist. > > Circular links can allow an attacker to obtain an object reference > ,prior to its invariants being checked, when we rely on readObject() to > throw an exception. Delaying finalizer registration won't save you. Yes, circular references are nasty. > Classes with circular links should be final and use a transient boolean > "initialized" field, that every method checks to prevent an attacker > doing anything useful, should they gain a reference to an incorrectly > constructed object. > > The real question is, do we continue to plaster over the issues with the > Serialization framework's api, and continue to create special cases such > as a second final field freeze after a constructor completes? To be > honest, I don't like this second final field freeze, because invariants > haven't been checked. I think some of the discussion on this mailing list can provide APIs that avoid this, in the common case. > Don't get me wrong, Serialization is quite clever, but implementing > Serializable properly can consume a lot of time due to api complexity > and is presently a security problem. Yes, we too have spent too much time on this already. :-( I hope this discussion will result in concrete changes to help make Serialization more secure by default, and provide additional tools, confinement, etc, when this is not possible. > In my examples all constructors share the same invariant checks and > because it's public API, the invariants can be tested easily with unit > tests. > > It seems like we're trying to give constructor properties to a private > instance method at the expense of increasing complexity, wouldn't it be > simpler in the long term to provide a Serialization constructor? I think long term we will need a replacement for Unsafe.allocateInstance. There are a number of issues in JIRA about possibly adding support, with suitable permissions, to reflection, to create an instance without running its constructor. I need to track down the bugs and details, not sure if this is even a realistic possibility. > P.S. David, I like your suggestion of using a protected method, for > limiting array size. Agreed. This looks like it may have potential. I thought some work was done in this area in the recent past, restricting the size of the graph or objects. Let me see if I can find out more. -Chris. > Thank you, > > Peter. >> On 01/08/2015 02:10 PM, Brian Goetz wrote: >>>> >> 1) Validate invariants >>>> >> >>>> >> A clear and easy to understand mechanism that can validate >>>> the >>>> >> deserialized >>>> >> fields. Does not prevent the use of final fields, as the >>>> >> serialization framework >>>> >> will be responsible for setting them. Something along the >>>> lines >>>> >> of what David >>>> >> suggested: >>>> >> >>>> >> private static void validate(GetField fields) { >>>> >> if (fields.getInt("lo")> fields.getInt("hi")) { ... } >>>> >> } >>>> >> >>>> >> This could be a ?special? method, or annotation driven. TBD. >>>> >> >>>> >> Note: the validate method is static, so the object >>>> instance is >>>> >> not required to >>>> >> be created before running the validation. >>> > >>> > Sort of... >>> > >>> > This is true if the fields participating in the invariant are >>> > primitives. But if they're refs, what do you do? What if you >>> want to >>> > validate something like >>> > >>> > count == list.size() // fields are int count, List list >>> > >>> > ? Then wouldn't GetField.getObject have to deserialize the object >>> > referred to by that field? >> In fact you cannot validate invariants across multiple objects at all >> using this method*or* readObject() (existing or enhanced) unless the >> object in question is an enum, Class, or String (and a few other special >> cases) because you can't rely on initialization order during >> deserialization. That's the very reason why OIS#registerValidation() >> even exists - inter-object validation is not possible until after the >> root-most readObject has completed, which is the time when validations >> are executed. Robust validation of a given object class may require two >> stages - "near" validation and "spanning" validation - to fully >> validate. However the readObject() approach and its proposed variations >> (including the static validate() version) can still be useful for >> fail-fast and non-complex validations; you just have to understand that >> (just as today) any Object you examine might not be fully initialized >> yet. >> >> -- - DML From scolebourne at joda.org Mon Jan 12 16:15:19 2015 From: scolebourne at joda.org (Stephen Colebourne) Date: Mon, 12 Jan 2015 16:15:19 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54B3B1E2.9070603@oracle.com> References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> <54AE4081.2090600@zeus.net.au> <54AEA30A.9090203@redhat.com> <97533E9F-0281-4EDD-9504-C38DE9453BE8@oracle.com> <54AEE440.2060709@oracle.com> <54B3B1E2.9070603@oracle.com> Message-ID: On 12 January 2015 at 11:37, Chris Hegarty wrote: > For clarity, I would like to describe how things currently work. > > 1) Allocate a new instance of the deserialized type. > 2) Call the first non-Serializable types no-arg constructor > ( may be j.l.Object ). > 3) For each type in the deserialized types hierarchy, starting > with the top most ( closest to j.l.Object ), > 3a) create objects representing all fields values for the type > [this step is recursive and will go to 1 until all > non-primitive types have been created ] > 3b) [ holder for invariant validation ] > 3c) assign objects to their respective members of the > containing instance Just to note that for me, the problem is that (1) happens at all. IMO, serialization should be separated entirely. Objects should only be created via standard constructors or factory methods. That is the direction that the static readObjectAndResolve() was in. Talk of dedicated serialization constructors or more use of unsafe doesn't really strike me as much of a way forward, except perhaps where compatibility is concerned. Stephen From daniel.fuchs at oracle.com Mon Jan 12 17:17:24 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 12 Jan 2015 18:17:24 +0100 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B3DCFC.1010705@oracle.com> References: <54B0083C.1090409@oracle.com> <54B32F8A.5020006@oracle.com> <54B3B762.9060508@oracle.com> <54B3DCFC.1010705@oracle.com> Message-ID: <54B401A4.8040207@oracle.com> On 12/01/15 15:41, Daniel Fuchs wrote: > On 12/01/15 13:49, Stephen Colebourne wrote: >> On 12 January 2015 at 12:00, Daniel Fuchs >> wrote: >>> I'm not a big fan of the current name either. >>> I would gladly rename it. I did think of >>> windows_to_java_time_micros, but it actually returns tenth of >>> micros - so it would be lying... >>> Is there a good name for 'tenth of micros'? >> >> .NET refers to them as "ticks". > > Ah! That's a good one. 'windows_to_time_ticks' I like this better :-) Here is the new webrev. Testing on windows went well :-) http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.02/ best regards, -- daniel > > Thanks Stephen! > > -- daniel > >> >> Stephen >> > From vladimir.x.ivanov at oracle.com Mon Jan 12 18:06:54 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Mon, 12 Jan 2015 21:06:54 +0300 Subject: [9] RFR (M): 8067344: Adjust java/lang/invoke/LFCaching/LFGarbageCollectedTest.java for recent changes in java.lang.invoke In-Reply-To: References: <549884E7.8040204@oracle.com> <549962C4.2040301@oracle.com> Message-ID: <54B40D3E.3080608@oracle.com> Paul, Thanks for the review! Updated webrev: http://cr.openjdk.java.net/~vlivanov/8067344/webrev.02 > 70 TestMethods testCase = getTestMethod(); > 71 if (testCase == TestMethods.EXACT_INVOKER || testCase == TestMethods.INVOKER) { > 72 // Invokers aren't collected. > 73 return; > 74 } > > Can you just filter those test cases out in the main method within EnumSet.complementOf? Good point! Done. > 82 mtype = adapter.type(); > 83 if (mtype.parameterCount() == 0) { > 84 // Ignore identity_* LambdaForms. > 85 return; > 86 } > > Under what conditions does this arise? i guess it might be non-determinisitic based on the randomly generated arity for the test case, so could filter more tests than absolutely required? Some transformations can rarely degenerate into identity. I share your concern, so I decided to check LambdaFor.debugName instead. >> - need to keep original test data for diagnostic purposes, since getTestCaseData() produces new instance. >> > > 78 adapter = getTestMethod().getTestCaseMH(data, TestMethods.Kind.ONE); > > > Could replace "getTestMethod()" with "testCase". Done. Best regards, Vladimir Ivanov From paul.sandoz at oracle.com Mon Jan 12 18:42:11 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 12 Jan 2015 19:42:11 +0100 Subject: [9] RFR (M): 8067344: Adjust java/lang/invoke/LFCaching/LFGarbageCollectedTest.java for recent changes in java.lang.invoke In-Reply-To: <54B40D3E.3080608@oracle.com> References: <549884E7.8040204@oracle.com> <549962C4.2040301@oracle.com> <54B40D3E.3080608@oracle.com> Message-ID: <6C72F39E-3CD3-4005-BD47-0FEEFFCD1F43@oracle.com> On Jan 12, 2015, at 7:06 PM, Vladimir Ivanov wrote: > Paul, > > Thanks for the review! > Look good, +1, Paul. > Updated webrev: > http://cr.openjdk.java.net/~vlivanov/8067344/webrev.02 > >> 70 TestMethods testCase = getTestMethod(); >> 71 if (testCase == TestMethods.EXACT_INVOKER || testCase == TestMethods.INVOKER) { >> 72 // Invokers aren't collected. >> 73 return; >> 74 } >> >> Can you just filter those test cases out in the main method within EnumSet.complementOf? > Good point! Done. > >> 82 mtype = adapter.type(); >> 83 if (mtype.parameterCount() == 0) { >> 84 // Ignore identity_* LambdaForms. >> 85 return; >> 86 } >> >> Under what conditions does this arise? i guess it might be non-determinisitic based on the randomly generated arity for the test case, so could filter more tests than absolutely required? > Some transformations can rarely degenerate into identity. I share your concern, so I decided to check LambdaFor.debugName instead. > >>> - need to keep original test data for diagnostic purposes, since getTestCaseData() produces new instance. >>> >> >> 78 adapter = getTestMethod().getTestCaseMH(data, TestMethods.Kind.ONE); >> >> >> Could replace "getTestMethod()" with "testCase". > Done. > From ecki at zusammenkunft.net Mon Jan 12 19:09:48 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Mon, 12 Jan 2015 20:09:48 +0100 Subject: Charset.lookupViaProviders uses new ServiceLoader instance on each miss. Message-ID: <20150112200948.000048bf.ecki@zusammenkunft.net> Hello, I see some strange scalability problem in an application which uses a few exernal Charsets which are provided by ServiceLoader. When I check the code for Charset.forName()/lookup() I can the 2-entry LRU cache for charsets, and if those caches do not find the charset, it will check standard, extended and loaded charsets. The later function is here: http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/42770c335bf7/src/java.base/share/classes/java/nio/charset/Charset.java#l341 And it uses ServiceLoader.load() on each lookup. This is unfortunate as it creates a new ServideLoader instance which in turn has its own (empty) service provider cache. So every time a extended charset is loaded (or an unknown charset is requested) it has to parse all the manifests. But what it is even worse, it also has to actually instantiate the found provider (again), so in some cases that provider can also not cache. Do I read the code correct? I think a quick fix would be to hold onto the ServiceLoader. Since it is initialized with the System classloader, there is no need to have a new one on each load. It migh also be worth considerung another 2-level LRU cache inside lookupViaProviders. I am not sure if a reload would be needed for that kind of system serviceloader? Here is a Java7 stacktrace, I think it has'nt changed much in Java9: java.lang.Thread.State: RUNNABLE at sun.misc.ASCIICaseInsensitiveComparator.isUpper(ASCIICaseInsensitiveComparator.java:89) at sun.misc.ASCIICaseInsensitiveComparator.toLower(ASCIICaseInsensitiveComparator.java:93) at sun.misc.ASCIICaseInsensitiveComparator.compare(ASCIICaseInsensitiveComparator.java:48) at sun.misc.ASCIICaseInsensitiveComparator.compare(ASCIICaseInsensitiveComparator.java:36) at java.util.TreeMap.put(TreeMap.java:545) at sun.nio.cs.AbstractCharsetProvider.put(AbstractCharsetProvider.java:84) at sun.nio.cs.AbstractCharsetProvider.charset(AbstractCharsetProvider.java:98) - locked <0x00002b0df808c1e0> (a sun.nio.cs.ext.ExtendedCharsets) at sun.nio.cs.ext.ExtendedCharsets.(ExtendedCharsets.java:349) at sun.reflect.GeneratedConstructorAccessor398.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>at java.lang.Class.newInstance(Class.java:374) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373) at java.util.ServiceLoader$1.next(ServiceLoader.java:445) at java.nio.charset.Charset$1.getNext(Charset.java:357) at java.nio.charset.Charset$1.hasNext(Charset.java:370) at java.nio.charset.Charset$2.run(Charset.java:413) at java.nio.charset.Charset$2.run(Charset.java:411) at java.security.AccessController.doPrivileged(Native Method) >>at java.nio.charset.Charset.lookupViaProviders(Charset.java:410) at java.nio.charset.Charset.lookup2(Charset.java:487) at java.nio.charset.Charset.lookup(Charset.java:475) at java.nio.charset.Charset.isSupported(Charset.java:517) at sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:68) at java.io.InputStreamReader.(InputStreamReader.java:100) In the specific case it also looks like no thread local decoder was used, but I havent looked into that, yet. Gruss Bernd From vladimir.x.ivanov at oracle.com Mon Jan 12 19:12:10 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Mon, 12 Jan 2015 22:12:10 +0300 Subject: [9] RFR (M): 8067344: Adjust java/lang/invoke/LFCaching/LFGarbageCollectedTest.java for recent changes in java.lang.invoke In-Reply-To: <6C72F39E-3CD3-4005-BD47-0FEEFFCD1F43@oracle.com> References: <549884E7.8040204@oracle.com> <549962C4.2040301@oracle.com> <54B40D3E.3080608@oracle.com> <6C72F39E-3CD3-4005-BD47-0FEEFFCD1F43@oracle.com> Message-ID: <54B41C8A.4060103@oracle.com> Thanks, Paul! Best regards, Vladimir Ivanov On 1/12/15 9:42 PM, Paul Sandoz wrote: > On Jan 12, 2015, at 7:06 PM, Vladimir Ivanov wrote: >> Paul, >> >> Thanks for the review! >> > > Look good, +1, > Paul. > >> Updated webrev: >> http://cr.openjdk.java.net/~vlivanov/8067344/webrev.02 >> >>> 70 TestMethods testCase = getTestMethod(); >>> 71 if (testCase == TestMethods.EXACT_INVOKER || testCase == TestMethods.INVOKER) { >>> 72 // Invokers aren't collected. >>> 73 return; >>> 74 } >>> >>> Can you just filter those test cases out in the main method within EnumSet.complementOf? >> Good point! Done. >> >>> 82 mtype = adapter.type(); >>> 83 if (mtype.parameterCount() == 0) { >>> 84 // Ignore identity_* LambdaForms. >>> 85 return; >>> 86 } >>> >>> Under what conditions does this arise? i guess it might be non-determinisitic based on the randomly generated arity for the test case, so could filter more tests than absolutely required? >> Some transformations can rarely degenerate into identity. I share your concern, so I decided to check LambdaFor.debugName instead. >> >>>> - need to keep original test data for diagnostic purposes, since getTestCaseData() produces new instance. >>>> >>> >>> 78 adapter = getTestMethod().getTestCaseMH(data, TestMethods.Kind.ONE); >>> >>> >>> Could replace "getTestMethod()" with "testCase". >> Done. >> > > From martinrb at google.com Mon Jan 12 19:42:03 2015 From: martinrb at google.com (Martin Buchholz) Date: Mon, 12 Jan 2015 11:42:03 -0800 Subject: Charset.lookupViaProviders uses new ServiceLoader instance on each miss. In-Reply-To: <20150112200948.000048bf.ecki@zusammenkunft.net> References: <20150112200948.000048bf.ecki@zusammenkunft.net> Message-ID: Historical notes: I added the two-element cache many years ago, assuming that code that repeatedly accessed more than 2 charsets would be rare. I have been hoping for a high-quality cache implementation to be added to the JDK, perhaps via guava, and to use that in places like Charset. I agree that lookup of non-builtin charsets is heavy-weight. I don't recall anyone trying to optimize the lookup itself. Perhaps you've found an important opportunity to optimize! On Mon, Jan 12, 2015 at 11:09 AM, Bernd Eckenfels wrote: > Hello, > > I see some strange scalability problem in an application which uses a > few exernal Charsets which are provided by ServiceLoader. When I check > the code for Charset.forName()/lookup() I can the 2-entry LRU cache for > charsets, and if those caches do not find the charset, it will check > standard, extended and loaded charsets. > > The later function is here: > > > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/42770c335bf7/src/java.base/share/classes/java/nio/charset/Charset.java#l341 > > And it uses ServiceLoader.load() on each lookup. > > This is unfortunate as it creates a new ServideLoader instance which in > turn has its own (empty) service provider cache. So every time a > extended charset is loaded (or an unknown charset is requested) it has > to parse all the manifests. But what it is even worse, it also has to > actually instantiate the found provider (again), so in some cases that > provider can also not cache. > > Do I read the code correct? > > I think a quick fix would be to hold onto the ServiceLoader. Since it > is initialized with the System classloader, there is no need to have a > new one on each load. It migh also be worth considerung another 2-level > LRU cache inside lookupViaProviders. I am not sure if a reload would be > needed for that kind of system serviceloader? > > Here is a Java7 stacktrace, I think it has'nt changed much in Java9: > > > java.lang.Thread.State: RUNNABLE > at > sun.misc.ASCIICaseInsensitiveComparator.isUpper(ASCIICaseInsensitiveComparator.java:89) > at > sun.misc.ASCIICaseInsensitiveComparator.toLower(ASCIICaseInsensitiveComparator.java:93) > at > sun.misc.ASCIICaseInsensitiveComparator.compare(ASCIICaseInsensitiveComparator.java:48) > at > sun.misc.ASCIICaseInsensitiveComparator.compare(ASCIICaseInsensitiveComparator.java:36) > at java.util.TreeMap.put(TreeMap.java:545) > at > sun.nio.cs.AbstractCharsetProvider.put(AbstractCharsetProvider.java:84) > at > sun.nio.cs.AbstractCharsetProvider.charset(AbstractCharsetProvider.java:98) > - locked <0x00002b0df808c1e0> (a sun.nio.cs.ext.ExtendedCharsets) > at sun.nio.cs.ext.ExtendedCharsets.(ExtendedCharsets.java:349) > at sun.reflect.GeneratedConstructorAccessor398.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > >>at java.lang.Class.newInstance(Class.java:374) > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373) > at java.util.ServiceLoader$1.next(ServiceLoader.java:445) > at java.nio.charset.Charset$1.getNext(Charset.java:357) > at java.nio.charset.Charset$1.hasNext(Charset.java:370) > at java.nio.charset.Charset$2.run(Charset.java:413) > at java.nio.charset.Charset$2.run(Charset.java:411) > at java.security.AccessController.doPrivileged(Native Method) > >>at java.nio.charset.Charset.lookupViaProviders(Charset.java:410) > at java.nio.charset.Charset.lookup2(Charset.java:487) > at java.nio.charset.Charset.lookup(Charset.java:475) > at java.nio.charset.Charset.isSupported(Charset.java:517) > at sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:68) > at java.io.InputStreamReader.(InputStreamReader.java:100) > > In the specific case it also looks like no thread local decoder was used, > but I havent looked into that, yet. > > Gruss > Bernd > > From martinrb at google.com Mon Jan 12 20:11:57 2015 From: martinrb at google.com (Martin Buchholz) Date: Mon, 12 Jan 2015 12:11:57 -0800 Subject: String.indexOf optimization In-Reply-To: References: <534D68B8.3040002@oracle.com> <5350E316.2050406@oracle.com> <54ABAAC2.2090502@redhat.com> Message-ID: If there's a clean improvement in the java code, it's worth putting in. You can try benchmarking with -Xint. Are we talking about this method? static int indexOf(char[] source, int sourceOffset, int sourceCount, char[] target, int targetOffset, int targetCount, int fromIndex) { It does look like we can tighten the code up a little... On Thu, Jan 8, 2015 at 3:05 PM, Zoltan Sziladi wrote: > Thanks for the info. > So that basically means we have 2 implementations of indexOf currently, one > is in HotSpot, the other is in the JDK itself, which rarely gets executed. > Aside from this later fact, isn't it still worth improving the JDK > implementation if it is possible? I know that the intrinsified method gets > executed most of the time, but still, if we can improve the JDK > implementation also, why not? I don't know much about other JVMs but maybe > a few don't intrinsify it? > > Is there any existing test suite which is considered conclusive enough that > if an implementation beats the naive algorithm in those testcases then it > could be considered as a replacement in the JDK? > > Regards, > Zoltan > > > On Thu, Jan 8, 2015 at 12:42 PM, Vitaly Davidovich > wrote: > > > The java impl you saw would be called by (a) interpreter, (b) if you > > explicitly disable intrinsification of this function, or (c) some other > JVM > > that doesn't intrinsify this method (or any method). > > > > People don't usually disable intrinsics; if they do, it's because they > hit > > some JIT bug and may disable it. > > > > On Thu, Jan 8, 2015 at 3:34 PM, Zoltan Sziladi > > wrote: > > > >> Hi, > >> > >> Thanks everyone for all the info. > >> So, just to go step by step in understanding this. > >> Andrew said HotSpot would ignore my implementation. So why is there an > >> implementation of indexOf at all in the JDK, if that's not the code > that's > >> executed? Is it just a default fallback? When is the indexOf function > not > >> intrinsified? When do people usually disable intrinsification? > >> Sorry if these are newbie questions, I'm new to this part of Java. > >> > >> Regards, > >> Zoltan > >> > >> On Tue, Jan 6, 2015 at 1:28 AM, Andrew Haley wrote: > >> > >> > Hi, > >> > > >> > On 05/01/15 18:59, Zoltan Sziladi wrote: > >> > > >> > > This discussion was a long time ago, I was just reading through it > to > >> > check > >> > > again what was the last state of the discussion about the > >> String.indexOf. > >> > > There is one part which I still do not understand, hopefully someone > >> > could > >> > > shed some light on it. A few emails ago Martin mentioned > >> > > > >> > > "Hotspot seems to have some intrinsification of String.indexOf, > which > >> > > confuses me. > >> > > Hotspot seems the right place to provide more optimizations for > this, > >> > since > >> > > there has been a fair amount of work creating high-performance > >> low-level > >> > > implementations of this idea in C." > >> > > > >> > > Then Ivan asked what that actually meant, whether hotspot actually > >> > replaced > >> > > the jdk implementation with a low level optimized C implementation, > >> but I > >> > > never saw an answer to that. > >> > > >> > You can have a look at an implementation of > >> MacroAssembler::string_indexof > >> > in > >> > > >> > > >> > > >> > http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b6b89b8f8531/src/cpu/x86/vm/macroAssembler_x86.cpp > >> > > >> > > Can someone please explain this? If we somehow found an algorithm > that > >> > beat > >> > > the naive implementation in the average case, would it be possible > to > >> > just > >> > > implement it in the JDK? > >> > > >> > No, because HotSpot would ignore it. Any speed improvements have to > be > >> > done in the architecture-dependent files. > >> > > >> > Andrew. > >> > > >> > > >> > > > > > From patrick at reini.net Mon Jan 12 20:31:59 2015 From: patrick at reini.net (Patrick Reinhart) Date: Mon, 12 Jan 2015 21:31:59 +0100 Subject: Explicit Serialization API and Security In-Reply-To: References: <549F56DA.7040004@zeus.net.au> <54A1A6BC.6070109@oracle.com> <1420116227.2489.5.camel@Nokia-N900> <54A7214D.80608@oracle.com> <1420284243.3416.10.camel@Nokia-N900> <54A9DD20.5050802@oracle.com> <54AA50A8.7030801@zeus.net.au> <54AA7C3E.6010703@oracle.com> <54AA7D2F.1000703@zeus.net.au> <54AA9B92.3050705@gmail.com> <54AAF963.2020402@oracle.com> <54AB8E44.6010905@gmail.com> <54ABF928.3020502@gmail.com> <54ABF9F4.1090705@gmail.com> <54AC200E.3000602@gmail.com> <54AD48B3.4050607@oracle.com> <54AD5883.3070705@gmail.com> <54AE4081.2090600@zeus.net.au> <54AEA30A.9090203@redhat.com> <97533E9F-0281-4EDD-9504-C38DE9453BE8@oracle.com> <54AEE440.2060709@oracle.com> <54B3B1E2.9070603@oracle.com> Message-ID: <7667243D-ADCF-4C3D-8041-5F3E510B2C0C@reini.net> > Am 12.01.2015 um 17:15 schrieb Stephen Colebourne : > > On 12 January 2015 at 11:37, Chris Hegarty wrote: >> For clarity, I would like to describe how things currently work. >> >> 1) Allocate a new instance of the deserialized type. >> 2) Call the first non-Serializable types no-arg constructor >> ( may be j.l.Object ). >> 3) For each type in the deserialized types hierarchy, starting >> with the top most ( closest to j.l.Object ), >> 3a) create objects representing all fields values for the type >> [this step is recursive and will go to 1 until all >> non-primitive types have been created ] >> 3b) [ holder for invariant validation ] >> 3c) assign objects to their respective members of the >> containing instance > > Just to note that for me, the problem is that (1) happens at all. IMO, > serialization should be separated entirely. Objects should only be > created via standard constructors or factory methods. That is the > direction that the static readObjectAndResolve() was in. Talk of > dedicated serialization constructors or more use of unsafe doesn't > really strike me as much of a way forward, except perhaps where > compatibility is concerned. > > Stephen That?s the way we mostly implemented the serialization in our product in order to be more open for simpler conversion of enhancements being done on existing classes. In our case we serialize using a proxy class with the writeReplace() on the original and readObject()/readResolve() method on the serialize proxy class. One of the most annoying part of the existing Serialization for me that should also be looked into is the way that errors are being reported if something goes wrong while serializing/deserializing objects. In that part I spent the most of my time in the past. - Patrick From patrick at reini.net Mon Jan 12 21:12:04 2015 From: patrick at reini.net (Patrick Reinhart) Date: Mon, 12 Jan 2015 22:12:04 +0100 Subject: Offer for help Message-ID: <2E0D251F-9F07-4D97-BE9E-D5433A5A850C@reini.net> Hi Pavel, I would like to help on some open Issues that needed being solved. Should I apply the transferTo method on the existing codebase, where possible? Or what would be a issue that need some attention? -Patrick From Ulf.Zibis at gmx.de Mon Jan 12 21:26:24 2015 From: Ulf.Zibis at gmx.de (Ulf Zibis) Date: Mon, 12 Jan 2015 22:26:24 +0100 Subject: Charset.lookupViaProviders uses new ServiceLoader instance on each miss. In-Reply-To: References: <20150112200948.000048bf.ecki@zusammenkunft.net> Message-ID: <54B43C00.9080204@gmx.de> Am 12.01.2015 um 20:42 schrieb Martin Buchholz: > Historical notes: > > I added the two-element cache many years ago, assuming that code that > repeatedly accessed more than 2 charsets would be rare. I suspect this opinion, see: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6795535 -Ulf From Roger.Riggs at Oracle.com Mon Jan 12 21:36:36 2015 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Mon, 12 Jan 2015 16:36:36 -0500 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B401A4.8040207@oracle.com> References: <54B0083C.1090409@oracle.com> <54B32F8A.5020006@oracle.com> <54B3B762.9060508@oracle.com> <54B3DCFC.1010705@oracle.com> <54B401A4.8040207@oracle.com> Message-ID: <54B43E64.90802@Oracle.com> Hi Daniel, Looks fine, a great enhancement. Thanks, Roger On 1/12/2015 12:17 PM, Daniel Fuchs wrote: ... >> Ah! That's a good one. 'windows_to_time_ticks' I like this better :-) > > Here is the new webrev. Testing on windows went well :-) > > http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.02/ > > best regards, > > -- daniel From peter.levart at gmail.com Mon Jan 12 22:12:13 2015 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 12 Jan 2015 23:12:13 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B306AB.8010509@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> <54B306AB.8010509@gmail.com> Message-ID: <54B446BD.7050306@gmail.com> Hi, I added results obtained with JDK 8 (FCS and u20) - same machine, same VM options, just different JDKs: Original JDK 7 HashMap (and JVM): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 2839.458 157.299 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2673.924 187.063 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 686.972 32.928 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 631.001 6.574 ms Original JDK 8 HashMap (JDK 8 FCS JVM): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3186.305 74.890 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2479.155 136.924 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 673.819 13.236 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 673.636 8.676 ms Original JDK 8 HashMap (JDK 8u20 JVM): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3107.455 72.524 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2986.006 9.796 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 631.295 7.281 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 641.041 17.139 ms Original JDK 9 HashMap: Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3011.738 78.249 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2984.280 48.315 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 682.060 52.341 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 685.705 55.183 ms Original JDK 9 HashMap with TREEIFY_THRESHOLD = 1 << 20: Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 2780.771 236.647 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2541.740 233.429 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 757.364 67.869 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 671.617 54.943 ms Caching of comparableClassFor (in ClassRepository - good for heterogeneous keys too): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3014.888 71.778 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2279.757 54.159 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 760.743 70.674 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 725.188 67.853 ms Caching of comparableClassFor (internally - good for homogeneous keys only): Benchmark (initialSize) Mode Samples Score Score error Units j.t.HashMapCollision.badDistNoComp 16 ss 60 3026.707 84.571 ms j.t.HashMapCollision.badDistWithComp 16 ss 60 2137.296 66.140 ms j.t.HashMapCollision.goodDistNoComp 16 ss 60 635.964 8.213 ms j.t.HashMapCollision.goodDistWithComp 16 ss 60 685.129 46.783 ms Peter On 01/12/2015 12:26 AM, Peter Levart wrote: > > On 01/11/2015 10:00 PM, Doug Lea wrote: >> On 01/11/2015 02:26 PM, Peter Levart wrote: >> >>> Although majority of entries constitute the bins of size 13 or 14, >>> there's only >>> a single hashCode value per bin. >>> >>> So in this benchmark, treeifying with non-comparable keys is a waste >>> of effort. >> >> On the other hand, the waste seems to only cost about 10% in your runs. >> I wonder why the original report using jdk7 vs jdk8 seemed larger. > > I don't know. I ran the same benchmark with same VM options on JDK 7 > too. Here are all results together: > > Original JDK 7 HashMap (and JVM): > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 2839.458 157.299 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2673.924 187.063 ms > j.t.HashMapCollision.goodDistNoComp 16 ss > 60 686.972 32.928 ms > j.t.HashMapCollision.goodDistWithComp 16 ss > 60 631.001 6.574 ms > > Original JDK 9 HashMap: > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3011.738 78.249 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2984.280 48.315 ms > j.t.HashMapCollision.goodDistNoComp 16 ss > 60 682.060 52.341 ms > j.t.HashMapCollision.goodDistWithComp 16 ss > 60 685.705 55.183 ms > > Original JDK 9 HashMap with TREEIFY_THRESHOLD = 1 << 20: > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 2780.771 236.647 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2541.740 233.429 ms > j.t.HashMapCollision.goodDistNoComp 16 ss > 60 757.364 67.869 ms > j.t.HashMapCollision.goodDistWithComp 16 ss > 60 671.617 54.943 ms > > Caching of comparableClassFor (in ClassRepository - good for > heterogeneous keys too): > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3014.888 71.778 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2279.757 54.159 ms > j.t.HashMapCollision.goodDistNoComp 16 ss > 60 760.743 70.674 ms > j.t.HashMapCollision.goodDistWithComp 16 ss > 60 725.188 67.853 ms > > Caching of comparableClassFor (internally - good for homogeneous keys > only): > > Benchmark (initialSize) Mode > Samples Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3026.707 84.571 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2137.296 66.140 ms > j.t.HashMapCollision.goodDistNoComp 16 ss > 60 635.964 8.213 ms > j.t.HashMapCollision.goodDistWithComp 16 ss > 60 685.129 46.783 ms > >> >>> >>> Are there (non-forged) sets of non-comparable keys with hashCodes where >>> treeifying makes sense? >> >> Try using a class like: >> class FHC { float f; int hashCode() { return >> Float.floatToIntBits(f); } } >> and populate with instances with integral values for f. >> >> Similarly for doubles. >> >> Pre-jdk8, we devised a bit-smearing function that (among other >> constraints) did OK for float/double keys with integral values, >> that are not all that rare. With treeification, we don't need to >> penalize classes with decent hashCodes by bit-smearing to still >> get OK performance for these kinds of cases where the tree-based >> hashCode comparison helps more than Comparability per se. > > I see. These keys actually have unique or near unique hashCodes but > which are not good for power of two length tables without > bit-smearing. With tree bins we don't need heavy bit-smearing to get > decent performance in speed, but the table gets quite sparse anyway > (although this is the smaller of the space overheads - tree nodes are > bigger). For example, for 1M integral Floats, we get the following: > > >>> Float ... > Capacity: 2097152 > Load factor: 0.75 > Size: 1000000 > Bin sizes: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 6*0 > 7*0 8*0 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 > Empty bins: 93.8 % > Unique hash codes per bin: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 6*0 > 7*0 8*0 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 > > >> >> Also... >> >> It looks like the simplest path to a minor improvement is >> just to cache internally (your fourth test below). But I now >> recall not doing this because it adds to footprint and >> the field could prevent class unloading, for only a small >> benefit. > > Footprint, yes (one reference field in HM instance), while class > unloading is taken care of using WeakReference: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HomogeneousKeysCache/webrev.01/ > >> >> (Every time HashMap has changed, there have been reports of >> performance regressions even though typical performance >> generally improves.) >> >> -Doug > > Regards, Peter > >> >> >>>> Original JDK9 HashMap: >>>> >>>> Benchmark (initialSize) Mode Samples >>>> Score Score error Units >>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>> 3011.738 78.249 ms >>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>> 2984.280 48.315 ms >>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>> 682.060 52.341 ms >>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>> 685.705 55.183 ms >>>> >>>> Original JDK9 HashMap with TREEIFY_THRESHOLD = 1 << 20: >>>> >>>> Benchmark (initialSize) Mode Samples >>>> Score Score error Units >>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>> 2780.771 236.647 ms >>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>> 2541.740 233.429 ms >>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>> 757.364 67.869 ms >>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>> 671.617 54.943 ms >>>> >>>> Caching of comparableClassFor (in ClassRepository - good for >>>> heterogeneous >>>> keys too): >>>> >>>> Benchmark (initialSize) Mode Samples >>>> Score Score error Units >>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>> 3014.888 71.778 ms >>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>> 2279.757 54.159 ms >>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>> 760.743 70.674 ms >>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>> 725.188 67.853 ms >>>> >>>> Caching of comparableClassFor (internally - good for homogeneous >>>> keys only): >>>> >>>> Benchmark (initialSize) Mode Samples >>>> Score Score error Units >>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>> 3026.707 84.571 ms >>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>> 2137.296 66.140 ms >>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>> 635.964 8.213 ms >>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>> 685.129 46.783 ms >>>> >>>> >> > From otaviojava at java.net Mon Jan 12 22:23:35 2015 From: otaviojava at java.net (=?UTF-8?Q?Ot=C3=A1vio_Gon=C3=A7alves_de_Santana?=) Date: Mon, 12 Jan 2015 20:23:35 -0200 Subject: IndexOf with Char instead of String Message-ID: These String literals may be replaced by equivalent character literals[1], gaining some performance enhancement. Webrev: https://dl.dropboxusercontent.com/u/16109193/open_jdk/jdk/index_of_character/index.html [1] @State(Scope.Thread) @OutputTimeUnit(TimeUnit.SECONDS) public class IndexOfBenchmark { private static final String WORDS = "I love java Language 8 because has Lambda, Stream and MetaSpace"; @Param("10000") private int param; @GenerateMicroBenchmark public void indexOfString(BlackHole bh) { for (int i = 0; i <= param; i++) { int index = WORDS.indexOf(","); bh.consume(index); } } @GenerateMicroBenchmark public void indexOfChar(BlackHole bh) { for (int i = 0; i <= param; i++) { int index = WORDS.indexOf(','); bh.consume(index); } } } Benchmark (param) Mode Samples Mean Mean error Units m.IndexOfBenchmark.indexOfChar 10000 thrpt 10 5579.120 114.179 ops/s m.IndexOfBenchmark.indexOfString 10000 thrpt 10 4562.178 46.751 ops/s -- Ot?vio Gon?alves de Santana blog: http://otaviosantana.blogspot.com.br/ twitter: http://twitter.com/otaviojava site: *http://about.me/otaviojava * 55 (11) 98255-3513 -------------- next part -------------- A non-text attachment was scrubbed... Name: 10350_updated.patch Type: text/x-patch Size: 54809 bytes Desc: not available URL: From martinrb at google.com Mon Jan 12 22:41:20 2015 From: martinrb at google.com (Martin Buchholz) Date: Mon, 12 Jan 2015 14:41:20 -0800 Subject: Charset.lookupViaProviders uses new ServiceLoader instance on each miss. In-Reply-To: <54B43C00.9080204@gmx.de> References: <20150112200948.000048bf.ecki@zusammenkunft.net> <54B43C00.9080204@gmx.de> Message-ID: I don't disagree about 3 elements in the cache being better than 2. When I implemented this crappy cache, it was all about "2 is better than 1". Yeah, maybe we could get a "real" cache someday... On Mon, Jan 12, 2015 at 1:26 PM, Ulf Zibis wrote: > > Am 12.01.2015 um 20:42 schrieb Martin Buchholz: > >> Historical notes: >> >> I added the two-element cache many years ago, assuming that code that >> repeatedly accessed more than 2 charsets would be rare. >> > > I suspect this opinion, see: http://bugs.java.com/ > bugdatabase/view_bug.do?bug_id=6795535 > > -Ulf > > From xueming.shen at oracle.com Mon Jan 12 23:31:24 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Mon, 12 Jan 2015 15:31:24 -0800 Subject: Charset.lookupViaProviders uses new ServiceLoader instance on each miss. In-Reply-To: References: <20150112200948.000048bf.ecki@zusammenkunft.net> <54B43C00.9080204@gmx.de> Message-ID: <54B4594C.2060208@oracle.com> We do have map based cache in standard and extended charset provider implementation already. The standard version is prehashedmap based, it probably should be fast enough. The extended version is a treemap based, which probably is slow. I'm a little surprised that the 2-elements cache in Charset actually makes big difference for the prehashedmap based cache in standard charsets provider. Yes, we don't cache the non-builtin provider. It might trigger performance hit if certain number of charsets from that provider is frequently looked up. -Sherman On 01/12/2015 02:41 PM, Martin Buchholz wrote: > I don't disagree about 3 elements in the cache being better than 2. > When I implemented this crappy cache, it was all about "2 is better than 1". > Yeah, maybe we could get a "real" cache someday... > > On Mon, Jan 12, 2015 at 1:26 PM, Ulf Zibis wrote: > >> Am 12.01.2015 um 20:42 schrieb Martin Buchholz: >> >>> Historical notes: >>> >>> I added the two-element cache many years ago, assuming that code that >>> repeatedly accessed more than 2 charsets would be rare. >>> >> I suspect this opinion, see: http://bugs.java.com/ >> bugdatabase/view_bug.do?bug_id=6795535 >> >> -Ulf >> >> From ecki at zusammenkunft.net Mon Jan 12 23:57:27 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Tue, 13 Jan 2015 00:57:27 +0100 Subject: Charset.lookupViaProviders uses new ServiceLoader instance on each miss. In-Reply-To: References: <20150112200948.000048bf.ecki@zusammenkunft.net> <54B43C00.9080204@gmx.de> Message-ID: <20150113005727.00003c23.ecki@zusammenkunft.net> Hello, I see typical use of "ASCII", "ISO88591", "UTF8" and then the 1-2 platform/filename encodings in case of Windows (ANSI+OEM). That makes at least 5 commonly used ones (not sure about UTF-16((BL)E) internally?). And this hopes, that all charsets are not used by any of their aliases. But of course, this does not mean they are requested in high frequency and alternating. (Maybe we should instrument that?) The problem is the standard charset provider is rather fast, but it will also evict the hit from the cache. Thats why I was thinking a somewhat larger cache which is only used after first+2nd level and standard/extended charset lookups (ie before ServiceLoader) While digging through the source, I found a place where defaultCharset() -> string -> string verify -> string lookup is done: http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/sun/nio/cs/StreamDecoder.java#59 Can this be changed to: # if (csn == null) # return return new StreamDecoder(in, l, Charset.defaultCharset()); and there are a few places where literal lookup can be replaced with static StandardCharsets.XXX to further reduce number of lookups: http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/net/URI.java#2778 http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/sun/net/www/ParseUtil.java#181 http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/com/sun/crypto/provider/PBKDF2KeyImpl.java#63 http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/sun/print/IPPPrintService.java#1642 http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/com/sun/tools/internal/ws/processor/util/IndentingWriter.java#169 (defaultCharset()) Gruss Bernd Am Mon, 12 Jan 2015 14:41:20 -0800 schrieb Martin Buchholz : > I don't disagree about 3 elements in the cache being better than 2. > When I implemented this crappy cache, it was all about "2 is better > than 1". Yeah, maybe we could get a "real" cache someday... > > On Mon, Jan 12, 2015 at 1:26 PM, Ulf Zibis wrote: > > > > > Am 12.01.2015 um 20:42 schrieb Martin Buchholz: > > > >> Historical notes: > >> > >> I added the two-element cache many years ago, assuming that code > >> that repeatedly accessed more than 2 charsets would be rare. > >> > > > > I suspect this opinion, see: http://bugs.java.com/ > > bugdatabase/view_bug.do?bug_id=6795535 > > > > -Ulf > > > > From ecki at zusammenkunft.net Tue Jan 13 00:37:55 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Tue, 13 Jan 2015 01:37:55 +0100 Subject: Charset.lookupViaProviders uses new ServiceLoader instance on each miss. In-Reply-To: <54B4594C.2060208@oracle.com> References: <20150112200948.000048bf.ecki@zusammenkunft.net> <54B43C00.9080204@gmx.de> <54B4594C.2060208@oracle.com> Message-ID: <20150113013755.00004b36.ecki@zusammenkunft.net> Am Mon, 12 Jan 2015 15:31:24 -0800 schrieb Xueming Shen : > We do have map based cache in standard and extended charset provider > implementation already. The standard version is prehashedmap based, it > probably should be fast enough. The extended version is a treemap > based, which probably is slow. > > I'm a little surprised that the 2-elements cache in Charset actually > makes big difference for the prehashedmap based cache in standard > charsets provider. One option would be to have the cache after the prehased fast provider and/or at least check for well known standard charsets before they pollute the cache: UTF8, ISO1 and ASCII. Both will avoid that the often used but easily obtained charsets drain the 2 slots. > Yes, we don't cache the non-builtin provider. It might trigger > performance hit if certain number of charsets from that provider is > frequently looked up. Especially because not even the ServiceLoader (and therefore the provider) is cached. And since AbstractCharsetProvider is caching in an instance variable, its losing its memory every time. This may affect custom providers. BTW: I just found out that the stacktrace I was showing is from Java 6, where the ExtendedCharsets is found via Service Loader and reinitialized on each lookup (hence the many hasmap.put() operations). I just tested it, on 7 and 8, this is no longer a problem. So the really slow re-initizing of ExtendedCharsets confused the picture. With 6 the service loader returns: provider class=sun.nio.cs.ext.ExtendedCharsets instance=sun.nio.cs.ext.ExtendedCharsets at 5e743399 source=jar:file:/C:/Program%20Files/Java/jdk1.6.0_38/jre/lib/charsets.jar!/sun/nio/cs/ext/ Big5 Big5-HKSCS ... seems to not only affect custom charset providers, but in my stacktraces I often see the constructor of ExtendedCharsets which puts 2 dozen of pre-initialized charstes (again and again). Gruss Bernd Gruss Bernd > > -Sherman > > On 01/12/2015 02:41 PM, Martin Buchholz wrote: > > I don't disagree about 3 elements in the cache being better than 2. > > When I implemented this crappy cache, it was all about "2 is better > > than 1". Yeah, maybe we could get a "real" cache someday... > > > > On Mon, Jan 12, 2015 at 1:26 PM, Ulf Zibis wrote: > > > >> Am 12.01.2015 um 20:42 schrieb Martin Buchholz: > >> > >>> Historical notes: > >>> > >>> I added the two-element cache many years ago, assuming that code > >>> that repeatedly accessed more than 2 charsets would be rare. > >>> > >> I suspect this opinion, see: http://bugs.java.com/ > >> bugdatabase/view_bug.do?bug_id=6795535 > >> > >> -Ulf > >> > >> > From martinrb at google.com Tue Jan 13 00:42:26 2015 From: martinrb at google.com (Martin Buchholz) Date: Mon, 12 Jan 2015 16:42:26 -0800 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B3BADE.8070607@cs.oswego.edu> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> <54B306AB.8010509@gmail.com> <54B3853C.2020507@gmail.com> <54B3BADE.8070607@cs.oswego.edu> Message-ID: Overall, I'm very happy with the way that Doug's heroic treebinization project worked out - we have expected O(1) and worst case O(log N) when trees are possible. I didn't consider the problem of not hurting iterator performance. I might consider having the iterator read-ahead all the elements in the bucket into an array and discard the pointers in the tree nodes that are there only to help the iterator. My intuition is to build a tree ordered only by hashcode, and then within each of those nodes, have a sequence of either homogenous Comparable trees or non-Comparable linked list, and each of those terminal structures, being homogeneous and concrete, will be easy for hotspot to deal with. But all of these ideas are expensive to implement and test, and make this complicated code even more so - probably we should declare victory instead. On Mon, Jan 12, 2015 at 4:15 AM, Doug Lea
wrote: > On 01/12/2015 03:26 AM, Peter Levart wrote: > >> Did bit smearing function used in JDK 7 have negative impact on any >> hashCodes or >> was it replaced with simpler one just because it is not needed when tree >> bins >> are used? >> > > I don't have detailed records, but in pre-release jdk8 tests, smearing was > detectably but not hugely slower for the cases of String, identity-hash, > and Integer keys. It is worth rechecking with jdk9. We believe these > cases together account for the majority of HashMap instances -- based on > some old instrumentation of a reference load, around 90% of the them. > (It would be great if someone re-ran this on a more recent corpus.) > The design goal of HashMap is to work best in common cases, > and to work well in all others except pathological cases of > non-comparable elements with many duplicate hashCodes. > > One issue that might be causing treeified bins to be worse than they > would otherwise be is that the recursion in TreeBin.find makes > it harder for compilers (JITs) to perform class analysis to determine > concrete types, leading to more virtual dispatching and slower code. > Someone from a compiler group once asked whether this code could > be changed to help JITs. Allocating emulated stack-frames > during lookups seems not to work well, but it is possible that > a scheme that temporarily reused internal node pointers might help. > I haven't looked into this. > > -Doug > > From peter.firmstone at zeus.net.au Tue Jan 13 00:51:09 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Tue, 13 Jan 2015 10:51:09 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54B3EEC5.5040805@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> Message-ID: <1421110269.7217.4.camel@Nokia-N900> ----- Original message ----- > On 10/01/15 07:00, Peter Firmstone wrote: > > Again, thank you all for engaging in discussion of this very difficult > > topic. > > > > While we can't presently check intra object dependencies during > > deserialization with readObject(), the examples I provide can do this. > > I have replied to Davids mail with a small change to GetField ( added > superTypeFields() ) to return the deserialized supertypes fields. This > gives subtypes the ability to check values of the supertypes persistent > state. Unfortunately this breaks encapsulation, a class is then locked into using that serial form as public api forever. > > As with your original proposal, it is limited to the persistent state as > read off the stream, and not the transient state, but I think it gets us > most of the way there. > > In your original proposal, it looks quite cumbersome to hook up the > static validator method in the constructor hierarchy ( as it is to do > this with standard constructors too ). It also relies on the fact that > the subtype has to create an instance of the supertype. I just wonder if > we can push on the alternative static validator proposal to come up with > something a but more attractive ( maybe not! ). Before proposing the constructor, I had considered a static validator method, with the following limitations: * A static method invariant check, while simple, cannot accommodate intra object invariants in class inheritance hierarchies without breaking encapsulation and allowing child classes to examine stream content. * When we break encapsulation, complexity of implementing Serializable safely increases; how to evolve serial form without breaking compatibility. * An Object under construction, usually copies mutable parameters before checking invariants, how can we guarantee that these objects are not shared and haven't mutated between the time the invariants are checked and when fields are set by reflection after construction? I'd also considered a static factory method: * It offers maximum flexibility for evolution; it can return objects of any class it likes. * Lacks the ability to recreate inherited private state, without breaking encapsulation. * Issues establishing intra object dependencies in inheritance hierarchies. Advantages that led me to propose the use of a constructor: * If existing constructors throw an exception when invariants haven't been satisfied, best practise dictates they must do so prior to calling Object's super class constructor to avoid finalizer attacks; static validation methods should already exist, if not the class should be refactored anyway, whether it implements Serializable or not. * Improved code reuse; invariant validation code is shared with other constructors. * Ability to satisfy complex intra object invariants in class inheritance heirarchies without breaking encapsulation. * Although it appears complex at first glance, once learnt, it's easily proven, preserves encapsulation and is easy to implement and test. * No serial form lock in, encapsulation has not been broken. Only the implementing class knows and uses the field names and types directly. It just seems like the best compromise, it's not perfect, is has less flaws. > > Have you seen the changes I proposing for failure atomicity, preliminary > webrev > http://cr.openjdk.java.net/~chegar/failureAtomicity/ > > .. and I think we can go further than this, creating the containing > object lazily, if there are no readObjectXXX methods in the hierarchy. > > Yes, but need some more time to absorb it, before commenting. Regards, Peter. From david.holmes at oracle.com Tue Jan 13 03:52:16 2015 From: david.holmes at oracle.com (David Holmes) Date: Tue, 13 Jan 2015 13:52:16 +1000 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B3B762.9060508@oracle.com> References: <54B0083C.1090409@oracle.com> <54B32F8A.5020006@oracle.com> <54B3B762.9060508@oracle.com> Message-ID: <54B49670.9080905@oracle.com> On 12/01/2015 10:00 PM, Daniel Fuchs wrote: > On 12/01/15 03:20, David Holmes wrote: >> Hi Daniel, >> >> Looking at the hotspot part ... >> >> On 10/01/2015 2:56 AM, Daniel Fuchs wrote: > > Hi David, > > [...] > >>> http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.00/ >>> >>> Some more details on the patch: >>> >>> native (hotspot) side: >>> >>> - adds a new method to the os class: >>> os::javaTimeSystemUTC(jlong &seconds, jlong &nanos) >>> which allows to get the time in the form of a number >>> of seconds and number of nanoseconds >>> (see os.hpp and various os_.cpp files) >> >> Everything seems consistent with javaTimeMillis code. Only nit on >> Windows is that windows_time_stamp isn't a very descriptive name (no >> idea what it returns) - perhaps windows_time_as_micros and adjust the >> calculations accordingly? > > I'm not a big fan of the current name either. > I would gladly rename it. I did think of > windows_to_java_time_micros, but it actually returns tenth of > micros - so it would be lying... > Is there a good name for 'tenth of micros'? hundreds of nanos? :) > Otherwise - I assume I could rename it into: > windows_to_java_time_max_precision > Would that be a better name? > Or would windows_time_as_tenth_of_micros be better? The windows_ticks name seems adequate. >>> - adds a JVM_GetNanoTimeAdjustment method, which takes >>> an offset (in seconds) as parameter and returns a >>> nano time adjustment compared to the offset. >>> Calls os::javaTimeSystemUTC to compute the adjustment >>> (see jvm.cpp) >> >> IIUC JVM_GetNanoTimeAdjustment(jlong offset_secs) returns the difference >> in nanoseconds between current UTC time (from Epoch) and the UTC time >> since the Epoch that offset_secs represents - is that right? > > Yes. Exactly. > >> I don't find the name particularly intuitive > > Well - I borrowed the name from the java time API. > Instant has a factory method that takes a number of seconds > and a 'nanoAdjustement': > http://docs.oracle.com/javase/8/docs/api/java/time/Instant.html#ofEpochSecond-long-long- > > >> and I guess you renamed it >> somewhere along the way as the mapfiles are no longer in alphabetical >> order (but would be if this was JVM_GetTimeXXXX). So please fix mapfiles. > > You have an eagle's eye ;-) > Yes - in my early prototype it was named JVM_GetTimeStamp - which was > not a great name. > Mapfiles fixed. Thanks for spotting that! > >>> java (jdk) side: >>> >>> - adds a native sun.misc.VM.getNanoTimeAdjustment method >>> (which is bound to JVM_GetNanoTimeAdjustment) >>> (see VM.java and VM.c) >> >> Just curious: why the indirection through sun.misc.VM? > > In my early prototype I didn't want to add new .c files, > and so I looked for a home for this new method. > When I asked around whether adding it to sun.misc.VM would > be appropriate nobody seemed too shocked. > > If you think it's better I could make it a private static > method in java.time.Clock - but then I'd have to change > the new unit test for that (GetNanoTimeAdjustment.java) to > use reflection (and would probably also need to add an > "initialize" native method to java.time.Clock as well). I was just curious, I'm not aware of any general policy on the core-libs side regarding placement of native methods. > Here is the new webrev with Stephen & your feedback included > http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.01/ > > (windows_time_stamp not renamed yet) I checked the updated webrev.02 and everything looks good on the hotspot side. What are your plans for pushing this? David ----- > Thanks a lot David! > > -- daniel > >> >> Cheers, >> David >> >>> - modifies java.time.Clock.SystemClock to call the new >>> sun.misc.VM.getNanoTimeAdjustment instead of >>> System.currentTimeMillis. >>> >>> - fixes java.util.Formatter - which wasn't expecting >>> greater than millisecond precision. >>> >>> testing: >>> >>> - A new test is added to test sun.misc.VM.getNanoTimeAdjustment >>> In particular it checks the edge cases. >>> - New tests are added to TestClock_System.java to check for >>> the increased precision. >>> There is also a test for the edge cases there. >>> - Some java.time tests where tripped by the increased precision, >>> and had to be modified to take that into account >>> >>> Note: comparing the new os::javaTimeSystemUTC with os::javaTimeMillis >>> can help in reviewing the changes. >>> >>> best regards, >>> >>> -- daniel > From erik.joelsson at oracle.com Tue Jan 13 08:41:33 2015 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Tue, 13 Jan 2015 09:41:33 +0100 Subject: RFR: JDK-8042707: Source changes needed to build JDK 9 with Visual Studio 2013 (VS2013) In-Reply-To: <54AFE98D.3070105@oracle.com> References: <54AE9A6D.2030101@oracle.com> <54AFE98D.3070105@oracle.com> Message-ID: <54B4DA3D.4020702@oracle.com> Hello, New webrev for root repo: http://cr.openjdk.java.net/~erikj/8042707/webrev.root.02/ On 2015-01-09 15:45, Magnus Ihse Bursie wrote: > > It looks basically good to me. Some remarks on toolchain_windows.m4, > though. > > In TOOLCHAIN_CHECK_POSSIBLE_VISUAL_STUDIO_ROOT: > # TODO: improve detection for other versions of VS > > This seems to be done now, so perhaps this can be removed :) Done. > --- > Why is "vs_version" lowercase? It might be better to have local > variables in lower case (or not, I'm not sure we are consistent on > this?) but this breaks with the rest of the variables in the function > and looks strange, like it was intentionally signalling something. > (This goes for the vs_version in the other functions as well.) Changed to upper case. Also introduced a common TOOLCHAIN_VERSION variable that is used in any non windows specific m4 file. > --- > # Work around the insanely named ProgramFiles(x86) env variable > PROGRAMFILES_X86="`env | $SED -n 's/^ProgramFiles(x86)=//p'`" > > Yay! :-) I spent some time going crazy about that one before I gave > up. Never thought of that solution. I think I found that on StackOverflow or similar site so can't claim credit. > --- > # FIXME will just assume default Visual Studio version > if test "x$with_tools_dir" != x; then > TOOLCHAIN_CHECK_POSSIBLE_VISUAL_STUDIO_ROOT([$with_tools_dir/../..], > > This seems broken. Have you tested it? You have to pass a version as > the first argument to TOOLCHAIN_CHECK_POSSIBLE_VISUAL_STUDIO_ROOT, > yes? I think we need to examine an explicitely given toolchain to have > it's version determined. Otherwise, the msvcr/msvcp handling code will > not function correctly. I suggest that we first check if > --with-toolchain-version is set and if so, respect it. Otherwise, > check the path given for the known names (VS_VS_INSTALLDIR_*), and if > none matches, abort and complain that version must be specified. > > Hm, actually, maybe the entire block of testing with_tools_dir should > be lifted up into TOOLCHAIN_FIND_VISUAL_STUDIO and handled > separately..? It doesn't really fit into > TOOLCHAIN_FIND_VISUAL_STUDIO_BAT_FILE anyhow. > I tinkered some with this, but ended up putting it back in TOOLCHAIN_FIND_VISUAL_STUDIO_BAT_FILE because it was simpler. It works now, but setting --with-tools-dir assumes you are pointing to the default version of Visual Studio. If you aren't, you must also set --with-toolchain-version for it to behave correctly. I suppose we could implement detection logic that would figure out which version it was, but I would rather not spend time on it when it's not the preferred way to configure this. /Erik From daniel.fuchs at oracle.com Tue Jan 13 09:53:58 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 13 Jan 2015 10:53:58 +0100 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B49670.9080905@oracle.com> References: <54B0083C.1090409@oracle.com> <54B32F8A.5020006@oracle.com> <54B3B762.9060508@oracle.com> <54B49670.9080905@oracle.com> Message-ID: <54B4EB36.9010909@oracle.com> On 13/01/15 04:52, David Holmes wrote: >> Here is the new webrev with Stephen & your feedback included >> http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.01/ >> >> (windows_time_stamp not renamed yet) > > I checked the updated webrev.02 and everything looks good on the hotspot > side. > > What are your plans for pushing this? Thanks David, My understanding is that I'll need to push that through hs-rt - since it has changes for hotspot. I believe I'll need two reviewers - is that right? Roger Riggs has also reviewed the changes on core-libs-dev, so is this good to go, or do both reviewers need to be members of the hsx project? Also - I am not sure whether I need a sponsor (I have never pushed anything to hs-rt - I usually push in jdk9-dev) or whether I am allowed to push this myself (I'll use jprt in that case). best regards, -- daniel > > David > ----- From peter.levart at gmail.com Tue Jan 13 10:00:06 2015 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 13 Jan 2015 11:00:06 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <54B3EEC5.5040805@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> Message-ID: <54B4ECA6.8050308@gmail.com> Hi Chris, I stepped back and asked myself what problem(s) we are searching solution(s) for and tried to see how existing API solves or doesn't solve them. I think that existing readObject() + GetFields API already solves most of the problems discussed so far as it: - provides encapsulation (using GetFields API we only have access to deserialized values of the class where readObject() method being invoked is declared) - provides a means to check intra-object invariants in an "encapsulated" way - the superclass state is already initialized/deserialized when readObject() is called, so we can check GetFields provided values against superclass state in a way that preserves encapsulation. What readObject()/GetFields combo doesn't provide is a relatively nice solution for final fields (we have to resort to clumsy reflection). So why not adding just that to the API? For example, extend ObjectInputStream with the following method: ObjectInputStream { ... FieldAccess fields(); ... } where FieldAccess is something like: public interface FieldAccess { FieldAccess set(String fieldName, boolean value); FieldAccess set(String fieldName, byte value); FieldAccess set(String fieldName, char value); FieldAccess set(String fieldName, short value); FieldAccess set(String fieldName, int value); FieldAccess set(String fieldName, long value); FieldAccess set(String fieldName, float value); FieldAccess set(String fieldName, double value); FieldAccess set(String fieldName, T value); } ...which would confine access to just those fields declared in the class where readObject() being invoked is declared and just for the instance and time of readObject() call-back. Simple, flexible and stright-forward. It solves the problem with final fields. It doesn't provide any auto-magic, which, although it might seem pretty at first, can sometimes just be in the way because of inflexibility. Sugar can be added though (in the form of GetFields.defaultReadFields() or even special check() methods), but flexibility is a necessary ingredient, I think. What do you think? Regards, Peter On 01/12/2015 04:56 PM, Chris Hegarty wrote: > On 10/01/15 07:00, Peter Firmstone wrote: >> Again, thank you all for engaging in discussion of this very difficult >> topic. >> >> While we can't presently check intra object dependencies during >> deserialization with readObject(), the examples I provide can do this. > > I have replied to Davids mail with a small change to GetField ( added > superTypeFields() ) to return the deserialized supertypes fields. This > gives subtypes the ability to check values of the supertypes > persistent state. > > As with your original proposal, it is limited to the persistent state > as read off the stream, and not the transient state, but I think it > gets us most of the way there. > > In your original proposal, it looks quite cumbersome to hook up the > static validator method in the constructor hierarchy ( as it is to do > this with standard constructors too ). It also relies on the fact that > the subtype has to create an instance of the supertype. I just wonder > if we can push on the alternative static validator proposal to come up > with something a but more attractive ( maybe not! ). > >> ObjectInputValidation is sufficient for enforcing business rules, >> however it can't stop an attacker. By the time the validator runs, the >> attackers object of choice has been instantiated and it's game over. >> That's right, the attacker may choose any Serializable class from the >> classpath, and may even extend non serializable classes, with a zero arg >> constructor, such as ClassLoader. >> >> To trust deserialization of an Object, as opposed to a String or >> primitive, the ObjectInputStream (or overriding subclass) must limit the >> classes allowed to be deserialized. > > Agreed. I am working on fleshing out a more concrete proposal on > confinement, that I mentioned in previous mails. Hopefully, I will > have something in the next few days. > >> For a class to be trusted, it must be trusted to check its parameters >> and enforce its invariants during object deserialization and to not >> deserialize with priviliged context. >> >> Validating an entire object graph's invariants, requires each class in >> the graph to take responsibility for validating and enforcing its own >> invariants. > > Agreed, and I think that the revised static validator method gives us > this. > >> As shown in my earlier example, intra class invariants can be enforced >> using Serialization constructors, from which static methods are called >> to check invariants prior to super class constructors being called. > > Yes, but it is cumbersome and easy to make mistakes. > >> Presently, I override ObjectInputStream and use a Permission called >> DeserializationPermission to limit classes that can be deserialized. >> Untrusted connections are run from unprivileged context to limit classes >> that can be instantiated, while those with trusted connections are run >> with a Subject that allows any class to be deserialized. > > Interesting. I think there is overlap here with confinement. > >> ReadSerial, could do this: >> Class c = rs.getType("foo"); >> >> And >> >> Foo f = rs.getObject("foo", Foo.class); >> >> Which performs instanceof type check, (prior to Object deserialization, >> if first time deserialized, otherwise after) and generic complile time >> type checked method return. Thus we must restrict the classes that can >> be deserialized with ObjectInputStream. > > Have you seen the changes I proposing for failure atomicity, > preliminary webrev > http://cr.openjdk.java.net/~chegar/failureAtomicity/ > > .. and I think we can go further than this, creating the containing > object lazily, if there are no readObjectXXX methods in the hierarchy. > >> If Foo is mutable and we want a private copy, the caller needs to copy >> or clone it before checking invariants, as it would any mutable >> constructor parameter. >> >> Each level of validation is the responsibility of the component with the >> most knowledge. >> >> 1. Trusted class lists - administration, it might change. >> 2. Object invariants and intra object invariants - Classes, not >> objects. >> 3. Business rules, but not security - ObjectInputValidation. >> >> So summing up, in order to secure deserialization we must validate all >> data input, preferably before allowing object instantiation. > > Yes, where possible. If we can phase out readObjectXXX and replace it > with a static validator, then I think this is possible. > >> Once an object has been constructed an attacker can gain a reference, >> whether by a finalizer attack or some other cleverly crafted stream, the >> attacker cannot obtain a reference to an object that doesn't exist. >> >> Circular links can allow an attacker to obtain an object reference >> ,prior to its invariants being checked, when we rely on readObject() to >> throw an exception. Delaying finalizer registration won't save you. > > Yes, circular references are nasty. > >> Classes with circular links should be final and use a transient boolean >> "initialized" field, that every method checks to prevent an attacker >> doing anything useful, should they gain a reference to an incorrectly >> constructed object. >> >> The real question is, do we continue to plaster over the issues with the >> Serialization framework's api, and continue to create special cases such >> as a second final field freeze after a constructor completes? To be >> honest, I don't like this second final field freeze, because invariants >> haven't been checked. > > I think some of the discussion on this mailing list can provide APIs > that avoid this, in the common case. > >> Don't get me wrong, Serialization is quite clever, but implementing >> Serializable properly can consume a lot of time due to api complexity >> and is presently a security problem. > > Yes, we too have spent too much time on this already. :-( I hope this > discussion will result in concrete changes to help make Serialization > more secure by default, and provide additional tools, confinement, > etc, when this is not possible. > >> In my examples all constructors share the same invariant checks and >> because it's public API, the invariants can be tested easily with unit >> tests. >> >> It seems like we're trying to give constructor properties to a private >> instance method at the expense of increasing complexity, wouldn't it be >> simpler in the long term to provide a Serialization constructor? > > I think long term we will need a replacement for > Unsafe.allocateInstance. There are a number of issues in JIRA about > possibly adding support, with suitable permissions, to reflection, to > create an instance without running its constructor. I need to track > down the bugs and details, not sure if this is even a realistic > possibility. > >> P.S. David, I like your suggestion of using a protected method, for >> limiting array size. > > Agreed. This looks like it may have potential. I thought some work > was done in this area in the recent past, restricting the size of the > graph or objects. Let me see if I can find out more. > > -Chris. > >> Thank you, >> >> Peter. >>> On 01/08/2015 02:10 PM, Brian Goetz wrote: >>>>> >> 1) Validate invariants >>>>> >> >>>>> >> A clear and easy to understand mechanism that can validate >>>>> the >>>>> >> deserialized >>>>> >> fields. Does not prevent the use of final fields, as the >>>>> >> serialization framework >>>>> >> will be responsible for setting them. Something along the >>>>> lines >>>>> >> of what David >>>>> >> suggested: >>>>> >> >>>>> >> private static void validate(GetField fields) { >>>>> >> if (fields.getInt("lo")> fields.getInt("hi")) { ... } >>>>> >> } >>>>> >> >>>>> >> This could be a ?special? method, or annotation driven. >>>>> TBD. >>>>> >> >>>>> >> Note: the validate method is static, so the object >>>>> instance is >>>>> >> not required to >>>>> >> be created before running the validation. >>>> > >>>> > Sort of... >>>> > >>>> > This is true if the fields participating in the invariant are >>>> > primitives. But if they're refs, what do you do? What if you >>>> want to >>>> > validate something like >>>> > >>>> > count == list.size() // fields are int count, List list >>>> > >>>> > ? Then wouldn't GetField.getObject have to deserialize the object >>>> > referred to by that field? >>> In fact you cannot validate invariants across multiple objects at all >>> using this method*or* readObject() (existing or enhanced) unless the >>> object in question is an enum, Class, or String (and a few other >>> special >>> cases) because you can't rely on initialization order during >>> deserialization. That's the very reason why OIS#registerValidation() >>> even exists - inter-object validation is not possible until after the >>> root-most readObject has completed, which is the time when validations >>> are executed. Robust validation of a given object class may require >>> two >>> stages - "near" validation and "spanning" validation - to fully >>> validate. However the readObject() approach and its proposed >>> variations >>> (including the static validate() version) can still be useful for >>> fail-fast and non-complex validations; you just have to understand that >>> (just as today) any Object you examine might not be fully initialized >>> yet. >>> >>> -- - DML From peter.firmstone at zeus.net.au Tue Jan 13 10:24:53 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Tue, 13 Jan 2015 20:24:53 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54B3EEC5.5040805@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> Message-ID: <1421144693.7530.20.camel@Nokia-N900> Could we use a static validator method and generate bytecode for constructors dynamically? The developer can optionally implement the constructors. static GetField invariantCheck(GetField f); Create a caller sensitive GetField implementation and add a two new methods to GetField: abstract Object createSuper(); // to access superclass object methods for inavariant checking. abstract Class getType(String name); Set fields from within constructors. The generated constructors are straight forward: 1. Call static method. 2. Call super class constructor with result from static method. 3. Set final fields 4. How to set transient fields, implement a private method called from within the constructor? Require a permission to extend GetField? From dl at cs.oswego.edu Tue Jan 13 11:36:37 2015 From: dl at cs.oswego.edu (Doug Lea) Date: Tue, 13 Jan 2015 06:36:37 -0500 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B446BD.7050306@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> <54B306AB.8010509@gmail.com> <54B446BD.7050306@gmail.com> Message-ID: <54B50345.4020808@cs.oswego.edu> On 01/12/2015 05:12 PM, Peter Levart wrote: > Hi, > > I added results obtained with JDK 8 (FCS and u20) - same machine, same VM > options, just different JDKs: Thanks very much. It remains a mystery why the original report by Bernd showed a 40% difference from jdk7. The approx 10% hit seen in these more careful measurements is about what we expected as the intended tradeoff for avoiding DOS attacks and better performance in more common cases. There are still some tweaky improvements we should keep exploring, but I don't think there's any need for fundamental changes. -Doug > > Original JDK 7 HashMap (and JVM): > > Benchmark (initialSize) Mode Samples > Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 2839.458 157.299 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2673.924 187.063 ms > j.t.HashMapCollision.goodDistNoComp 16 ss 60 > 686.972 32.928 ms > j.t.HashMapCollision.goodDistWithComp 16 ss 60 > 631.001 6.574 ms > > Original JDK 8 HashMap (JDK 8 FCS JVM): > > Benchmark (initialSize) Mode Samples > Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3186.305 74.890 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2479.155 136.924 ms > j.t.HashMapCollision.goodDistNoComp 16 ss 60 > 673.819 13.236 ms > j.t.HashMapCollision.goodDistWithComp 16 ss 60 > 673.636 8.676 ms > > Original JDK 8 HashMap (JDK 8u20 JVM): > > Benchmark (initialSize) Mode Samples > Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3107.455 72.524 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2986.006 9.796 ms > j.t.HashMapCollision.goodDistNoComp 16 ss 60 > 631.295 7.281 ms > j.t.HashMapCollision.goodDistWithComp 16 ss 60 > 641.041 17.139 ms > > Original JDK 9 HashMap: > > Benchmark (initialSize) Mode Samples > Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3011.738 78.249 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2984.280 48.315 ms > j.t.HashMapCollision.goodDistNoComp 16 ss 60 > 682.060 52.341 ms > j.t.HashMapCollision.goodDistWithComp 16 ss 60 > 685.705 55.183 ms > > Original JDK 9 HashMap with TREEIFY_THRESHOLD = 1 << 20: > > Benchmark (initialSize) Mode Samples > Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 2780.771 236.647 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2541.740 233.429 ms > j.t.HashMapCollision.goodDistNoComp 16 ss 60 > 757.364 67.869 ms > j.t.HashMapCollision.goodDistWithComp 16 ss 60 > 671.617 54.943 ms > > Caching of comparableClassFor (in ClassRepository - good for heterogeneous keys > too): > > Benchmark (initialSize) Mode Samples > Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3014.888 71.778 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2279.757 54.159 ms > j.t.HashMapCollision.goodDistNoComp 16 ss 60 > 760.743 70.674 ms > j.t.HashMapCollision.goodDistWithComp 16 ss 60 > 725.188 67.853 ms > > Caching of comparableClassFor (internally - good for homogeneous keys only): > > Benchmark (initialSize) Mode Samples > Score Score error Units > j.t.HashMapCollision.badDistNoComp 16 ss 60 > 3026.707 84.571 ms > j.t.HashMapCollision.badDistWithComp 16 ss 60 > 2137.296 66.140 ms > j.t.HashMapCollision.goodDistNoComp 16 ss 60 > 635.964 8.213 ms > j.t.HashMapCollision.goodDistWithComp 16 ss 60 > 685.129 46.783 ms > > > Peter > > On 01/12/2015 12:26 AM, Peter Levart wrote: >> >> On 01/11/2015 10:00 PM, Doug Lea wrote: >>> On 01/11/2015 02:26 PM, Peter Levart wrote: >>> >>>> Although majority of entries constitute the bins of size 13 or 14, there's only >>>> a single hashCode value per bin. >>>> >>>> So in this benchmark, treeifying with non-comparable keys is a waste of effort. >>> >>> On the other hand, the waste seems to only cost about 10% in your runs. >>> I wonder why the original report using jdk7 vs jdk8 seemed larger. >> >> I don't know. I ran the same benchmark with same VM options on JDK 7 too. Here >> are all results together: >> >> Original JDK 7 HashMap (and JVM): >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 2839.458 157.299 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2673.924 187.063 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 686.972 32.928 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 631.001 6.574 ms >> >> Original JDK 9 HashMap: >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3011.738 78.249 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2984.280 48.315 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 682.060 52.341 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 685.705 55.183 ms >> >> Original JDK 9 HashMap with TREEIFY_THRESHOLD = 1 << 20: >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 2780.771 236.647 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2541.740 233.429 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 757.364 67.869 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 671.617 54.943 ms >> >> Caching of comparableClassFor (in ClassRepository - good for heterogeneous >> keys too): >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3014.888 71.778 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2279.757 54.159 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 760.743 70.674 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 725.188 67.853 ms >> >> Caching of comparableClassFor (internally - good for homogeneous keys only): >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3026.707 84.571 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2137.296 66.140 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 635.964 8.213 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 685.129 46.783 ms >> >>> >>>> >>>> Are there (non-forged) sets of non-comparable keys with hashCodes where >>>> treeifying makes sense? >>> >>> Try using a class like: >>> class FHC { float f; int hashCode() { return Float.floatToIntBits(f); } } >>> and populate with instances with integral values for f. >>> >>> Similarly for doubles. >>> >>> Pre-jdk8, we devised a bit-smearing function that (among other >>> constraints) did OK for float/double keys with integral values, >>> that are not all that rare. With treeification, we don't need to >>> penalize classes with decent hashCodes by bit-smearing to still >>> get OK performance for these kinds of cases where the tree-based >>> hashCode comparison helps more than Comparability per se. >> >> I see. These keys actually have unique or near unique hashCodes but which are >> not good for power of two length tables without bit-smearing. With tree bins >> we don't need heavy bit-smearing to get decent performance in speed, but the >> table gets quite sparse anyway (although this is the smaller of the space >> overheads - tree nodes are bigger). For example, for 1M integral Floats, we >> get the following: >> >> >>> Float ... >> Capacity: 2097152 >> Load factor: 0.75 >> Size: 1000000 >> Bin sizes: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 6*0 7*0 8*0 >> 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 >> Empty bins: 93.8 % >> Unique hash codes per bin: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 6*0 7*0 8*0 >> 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 >> >> >>> >>> Also... >>> >>> It looks like the simplest path to a minor improvement is >>> just to cache internally (your fourth test below). But I now >>> recall not doing this because it adds to footprint and >>> the field could prevent class unloading, for only a small >>> benefit. >> >> Footprint, yes (one reference field in HM instance), while class unloading is >> taken care of using WeakReference: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HomogeneousKeysCache/webrev.01/ >> >>> >>> (Every time HashMap has changed, there have been reports of >>> performance regressions even though typical performance >>> generally improves.) >>> >>> -Doug >> >> Regards, Peter >> >>> >>> >>>>> Original JDK9 HashMap: >>>>> >>>>> Benchmark (initialSize) Mode Samples >>>>> Score Score error Units >>>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>>> 3011.738 78.249 ms >>>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>>> 2984.280 48.315 ms >>>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>>> 682.060 52.341 ms >>>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>>> 685.705 55.183 ms >>>>> >>>>> Original JDK9 HashMap with TREEIFY_THRESHOLD = 1 << 20: >>>>> >>>>> Benchmark (initialSize) Mode Samples >>>>> Score Score error Units >>>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>>> 2780.771 236.647 ms >>>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>>> 2541.740 233.429 ms >>>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>>> 757.364 67.869 ms >>>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>>> 671.617 54.943 ms >>>>> >>>>> Caching of comparableClassFor (in ClassRepository - good for heterogeneous >>>>> keys too): >>>>> >>>>> Benchmark (initialSize) Mode Samples >>>>> Score Score error Units >>>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>>> 3014.888 71.778 ms >>>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>>> 2279.757 54.159 ms >>>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>>> 760.743 70.674 ms >>>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>>> 725.188 67.853 ms >>>>> >>>>> Caching of comparableClassFor (internally - good for homogeneous keys only): >>>>> >>>>> Benchmark (initialSize) Mode Samples >>>>> Score Score error Units >>>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>>> 3026.707 84.571 ms >>>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>>> 2137.296 66.140 ms >>>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>>> 635.964 8.213 ms >>>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>>> 685.129 46.783 ms >>>>> >>>>> >>> >> > From david.holmes at oracle.com Tue Jan 13 12:26:39 2015 From: david.holmes at oracle.com (David Holmes) Date: Tue, 13 Jan 2015 22:26:39 +1000 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B4EB36.9010909@oracle.com> References: <54B0083C.1090409@oracle.com> <54B32F8A.5020006@oracle.com> <54B3B762.9060508@oracle.com> <54B49670.9080905@oracle.com> <54B4EB36.9010909@oracle.com> Message-ID: <54B50EFF.9070308@oracle.com> Hi Daniel, On 13/01/2015 7:53 PM, Daniel Fuchs wrote: > On 13/01/15 04:52, David Holmes wrote: >>> Here is the new webrev with Stephen & your feedback included >>> http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.01/ >>> >>> (windows_time_stamp not renamed yet) >> >> I checked the updated webrev.02 and everything looks good on the hotspot >> side. >> >> What are your plans for pushing this? > > Thanks David, > > My understanding is that I'll need to push that through hs-rt - since > it has changes for hotspot. I believe I'll need two reviewers - is that > right? Roger Riggs has also reviewed the changes on core-libs-dev, so > is this good to go, or do both reviewers need to be members of the hsx > project? hsx project is defunct, so technically we're all just jdk9 reviewers, but we still like two reviews from hotspot team members. > Also - I am not sure whether I need a sponsor (I have never pushed > anything to hs-rt - I usually push in jdk9-dev) or whether I am > allowed to push this myself (I'll use jprt in that case). If you are a jdk9 committer then you can push yourself via jprt - full forest submission from top-level: "jprt submit -forest -testset hotspot" David > best regards, > > -- daniel > >> >> David >> ----- > From peter.levart at gmail.com Tue Jan 13 15:51:01 2015 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 13 Jan 2015 16:51:01 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B50345.4020808@cs.oswego.edu> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> <54B306AB.8010509@gmail.com> <54B446BD.7050306@gmail.com> <54B50345.4020808@cs.oswego.edu> Message-ID: <54B53EE5.1070707@gmail.com> On 01/13/2015 12:36 PM, Doug Lea wrote: > On 01/12/2015 05:12 PM, Peter Levart wrote: >> Hi, >> >> I added results obtained with JDK 8 (FCS and u20) - same machine, >> same VM >> options, just different JDKs: > > Thanks very much. It remains a mystery why the original report > by Bernd showed a 40% difference from jdk7. The approx 10% hit > seen in these more careful measurements is about what we > expected as the intended tradeoff for avoiding DOS attacks > and better performance in more common cases. There are > still some tweaky improvements we should keep exploring, > but I don't think there's any need for fundamental changes. > > -Doug Perhaps it was because of different HW. I ran the tests on Intel? Core? i7-2600K CPU with 8 MB of Cache. I wonder what Bernd used. The number of different keys is 250.000, they are accessed randomly in a loop. Such HashMap consists of majority of TreeNode(s). TreeNode(s) have 9 reference fields + int and boolean. I don't claim that such HashMap fits intirely into Cache, but half of it perhaps does. Maybe it is just that cache-hit ratio was larger on my HW than on Bernd's and that Bernd's cache was thrashing more than mine. With JDK 7 (normal Nodes have just 3 reference fields + int) such HashMap has roughly half the memory footprint. Bernd, what did you run your benchmark on? Peter > >> >> Original JDK 7 HashMap (and JVM): >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 2839.458 157.299 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2673.924 187.063 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 686.972 32.928 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 631.001 6.574 ms >> >> Original JDK 8 HashMap (JDK 8 FCS JVM): >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3186.305 74.890 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2479.155 136.924 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 673.819 13.236 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 673.636 8.676 ms >> >> Original JDK 8 HashMap (JDK 8u20 JVM): >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3107.455 72.524 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2986.006 9.796 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 631.295 7.281 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 641.041 17.139 ms >> >> Original JDK 9 HashMap: >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3011.738 78.249 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2984.280 48.315 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 682.060 52.341 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 685.705 55.183 ms >> >> Original JDK 9 HashMap with TREEIFY_THRESHOLD = 1 << 20: >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 2780.771 236.647 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2541.740 233.429 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 757.364 67.869 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 671.617 54.943 ms >> >> Caching of comparableClassFor (in ClassRepository - good for >> heterogeneous keys >> too): >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3014.888 71.778 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2279.757 54.159 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 760.743 70.674 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 725.188 67.853 ms >> >> Caching of comparableClassFor (internally - good for homogeneous keys >> only): >> >> Benchmark (initialSize) Mode Samples >> Score Score error Units >> j.t.HashMapCollision.badDistNoComp 16 ss 60 >> 3026.707 84.571 ms >> j.t.HashMapCollision.badDistWithComp 16 ss 60 >> 2137.296 66.140 ms >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >> 635.964 8.213 ms >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >> 685.129 46.783 ms >> >> >> Peter >> >> On 01/12/2015 12:26 AM, Peter Levart wrote: >>> >>> On 01/11/2015 10:00 PM, Doug Lea wrote: >>>> On 01/11/2015 02:26 PM, Peter Levart wrote: >>>> >>>>> Although majority of entries constitute the bins of size 13 or 14, >>>>> there's only >>>>> a single hashCode value per bin. >>>>> >>>>> So in this benchmark, treeifying with non-comparable keys is a >>>>> waste of effort. >>>> >>>> On the other hand, the waste seems to only cost about 10% in your >>>> runs. >>>> I wonder why the original report using jdk7 vs jdk8 seemed larger. >>> >>> I don't know. I ran the same benchmark with same VM options on JDK 7 >>> too. Here >>> are all results together: >>> >>> Original JDK 7 HashMap (and JVM): >>> >>> Benchmark (initialSize) Mode Samples >>> Score Score error Units >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>> 2839.458 157.299 ms >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>> 2673.924 187.063 ms >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>> 686.972 32.928 ms >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>> 631.001 6.574 ms >>> >>> Original JDK 9 HashMap: >>> >>> Benchmark (initialSize) Mode Samples >>> Score Score error Units >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>> 3011.738 78.249 ms >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>> 2984.280 48.315 ms >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>> 682.060 52.341 ms >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>> 685.705 55.183 ms >>> >>> Original JDK 9 HashMap with TREEIFY_THRESHOLD = 1 << 20: >>> >>> Benchmark (initialSize) Mode Samples >>> Score Score error Units >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>> 2780.771 236.647 ms >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>> 2541.740 233.429 ms >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>> 757.364 67.869 ms >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>> 671.617 54.943 ms >>> >>> Caching of comparableClassFor (in ClassRepository - good for >>> heterogeneous >>> keys too): >>> >>> Benchmark (initialSize) Mode Samples >>> Score Score error Units >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>> 3014.888 71.778 ms >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>> 2279.757 54.159 ms >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>> 760.743 70.674 ms >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>> 725.188 67.853 ms >>> >>> Caching of comparableClassFor (internally - good for homogeneous >>> keys only): >>> >>> Benchmark (initialSize) Mode Samples >>> Score Score error Units >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>> 3026.707 84.571 ms >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>> 2137.296 66.140 ms >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>> 635.964 8.213 ms >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>> 685.129 46.783 ms >>> >>>> >>>>> >>>>> Are there (non-forged) sets of non-comparable keys with hashCodes >>>>> where >>>>> treeifying makes sense? >>>> >>>> Try using a class like: >>>> class FHC { float f; int hashCode() { return >>>> Float.floatToIntBits(f); } } >>>> and populate with instances with integral values for f. >>>> >>>> Similarly for doubles. >>>> >>>> Pre-jdk8, we devised a bit-smearing function that (among other >>>> constraints) did OK for float/double keys with integral values, >>>> that are not all that rare. With treeification, we don't need to >>>> penalize classes with decent hashCodes by bit-smearing to still >>>> get OK performance for these kinds of cases where the tree-based >>>> hashCode comparison helps more than Comparability per se. >>> >>> I see. These keys actually have unique or near unique hashCodes but >>> which are >>> not good for power of two length tables without bit-smearing. With >>> tree bins >>> we don't need heavy bit-smearing to get decent performance in speed, >>> but the >>> table gets quite sparse anyway (although this is the smaller of the >>> space >>> overheads - tree nodes are bigger). For example, for 1M integral >>> Floats, we >>> get the following: >>> >>> >>> Float ... >>> Capacity: 2097152 >>> Load factor: 0.75 >>> Size: 1000000 >>> Bin sizes: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 6*0 >>> 7*0 8*0 >>> 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 >>> Empty bins: 93.8 % >>> Unique hash codes per bin: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 6*0 >>> 7*0 8*0 >>> 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 >>> >>> >>>> >>>> Also... >>>> >>>> It looks like the simplest path to a minor improvement is >>>> just to cache internally (your fourth test below). But I now >>>> recall not doing this because it adds to footprint and >>>> the field could prevent class unloading, for only a small >>>> benefit. >>> >>> Footprint, yes (one reference field in HM instance), while class >>> unloading is >>> taken care of using WeakReference: >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HomogeneousKeysCache/webrev.01/ >>> >>> >>>> >>>> (Every time HashMap has changed, there have been reports of >>>> performance regressions even though typical performance >>>> generally improves.) >>>> >>>> -Doug >>> >>> Regards, Peter >>> >>>> >>>> >>>>>> Original JDK9 HashMap: >>>>>> >>>>>> Benchmark (initialSize) Mode Samples >>>>>> Score Score error Units >>>>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>>>> 3011.738 78.249 ms >>>>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>>>> 2984.280 48.315 ms >>>>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>>>> 682.060 52.341 ms >>>>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>>>> 685.705 55.183 ms >>>>>> >>>>>> Original JDK9 HashMap with TREEIFY_THRESHOLD = 1 << 20: >>>>>> >>>>>> Benchmark (initialSize) Mode Samples >>>>>> Score Score error Units >>>>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>>>> 2780.771 236.647 ms >>>>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>>>> 2541.740 233.429 ms >>>>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>>>> 757.364 67.869 ms >>>>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>>>> 671.617 54.943 ms >>>>>> >>>>>> Caching of comparableClassFor (in ClassRepository - good for >>>>>> heterogeneous >>>>>> keys too): >>>>>> >>>>>> Benchmark (initialSize) Mode Samples >>>>>> Score Score error Units >>>>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>>>> 3014.888 71.778 ms >>>>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>>>> 2279.757 54.159 ms >>>>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>>>> 760.743 70.674 ms >>>>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>>>> 725.188 67.853 ms >>>>>> >>>>>> Caching of comparableClassFor (internally - good for homogeneous >>>>>> keys only): >>>>>> >>>>>> Benchmark (initialSize) Mode Samples >>>>>> Score Score error Units >>>>>> j.t.HashMapCollision.badDistNoComp 16 ss 60 >>>>>> 3026.707 84.571 ms >>>>>> j.t.HashMapCollision.badDistWithComp 16 ss 60 >>>>>> 2137.296 66.140 ms >>>>>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 >>>>>> 635.964 8.213 ms >>>>>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 >>>>>> 685.129 46.783 ms >>>>>> >>>>>> >>>> >>> >> > From magnus.ihse.bursie at oracle.com Tue Jan 13 16:04:56 2015 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Tue, 13 Jan 2015 17:04:56 +0100 Subject: RFR: JDK-8042707: Source changes needed to build JDK 9 with Visual Studio 2013 (VS2013) In-Reply-To: <54B4DA3D.4020702@oracle.com> References: <54AE9A6D.2030101@oracle.com> <54AFE98D.3070105@oracle.com> <54B4DA3D.4020702@oracle.com> Message-ID: <54B54228.2020005@oracle.com> On 2015-01-13 09:41, Erik Joelsson wrote: > Hello, > > New webrev for root repo: > http://cr.openjdk.java.net/~erikj/8042707/webrev.root.02/ Fixes look good to me. Thanks! /Magnus > > On 2015-01-09 15:45, Magnus Ihse Bursie wrote: >> >> It looks basically good to me. Some remarks on toolchain_windows.m4, >> though. >> >> In TOOLCHAIN_CHECK_POSSIBLE_VISUAL_STUDIO_ROOT: >> # TODO: improve detection for other versions of VS >> >> This seems to be done now, so perhaps this can be removed :) > Done. >> --- >> Why is "vs_version" lowercase? It might be better to have local >> variables in lower case (or not, I'm not sure we are consistent on >> this?) but this breaks with the rest of the variables in the function >> and looks strange, like it was intentionally signalling something. >> (This goes for the vs_version in the other functions as well.) > Changed to upper case. Also introduced a common TOOLCHAIN_VERSION > variable that is used in any non windows specific m4 file. >> --- >> # Work around the insanely named ProgramFiles(x86) env variable >> PROGRAMFILES_X86="`env | $SED -n 's/^ProgramFiles(x86)=//p'`" >> >> Yay! :-) I spent some time going crazy about that one before I gave >> up. Never thought of that solution. > I think I found that on StackOverflow or similar site so can't claim > credit. >> --- >> # FIXME will just assume default Visual Studio version >> if test "x$with_tools_dir" != x; then >> TOOLCHAIN_CHECK_POSSIBLE_VISUAL_STUDIO_ROOT([$with_tools_dir/../..], >> >> This seems broken. Have you tested it? You have to pass a version as >> the first argument to TOOLCHAIN_CHECK_POSSIBLE_VISUAL_STUDIO_ROOT, >> yes? I think we need to examine an explicitely given toolchain to >> have it's version determined. Otherwise, the msvcr/msvcp handling >> code will not function correctly. I suggest that we first check if >> --with-toolchain-version is set and if so, respect it. Otherwise, >> check the path given for the known names (VS_VS_INSTALLDIR_*), and if >> none matches, abort and complain that version must be specified. >> >> Hm, actually, maybe the entire block of testing with_tools_dir should >> be lifted up into TOOLCHAIN_FIND_VISUAL_STUDIO and handled >> separately..? It doesn't really fit into >> TOOLCHAIN_FIND_VISUAL_STUDIO_BAT_FILE anyhow. >> > I tinkered some with this, but ended up putting it back in > TOOLCHAIN_FIND_VISUAL_STUDIO_BAT_FILE because it was simpler. It works > now, but setting --with-tools-dir assumes you are pointing to the > default version of Visual Studio. If you aren't, you must also set > --with-toolchain-version for it to behave correctly. I suppose we > could implement detection logic that would figure out which version it > was, but I would rather not spend time on it when it's not the > preferred way to configure this. > > /Erik From chris.hegarty at oracle.com Tue Jan 13 16:46:06 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 13 Jan 2015 16:46:06 +0000 Subject: RFR : 8066612: Add a test that will call getDeclaredFields() on all classes and try to set them accessible. In-Reply-To: <54AFB658.1030407@oracle.com> References: <54AE970D.4020805@oracle.com> <54AE9A58.70607@oracle.com> <54AFB658.1030407@oracle.com> Message-ID: <5A25478A-6964-462D-81BF-38AC05FD73ED@oracle.com> This looks fine to me too ( removal of streams usage and JRT, for 7uXX ).. -Chris. On 9 Jan 2015, at 11:07, Se?n Coffey wrote: > Thanks for reviewing Daniel. I'm mark you as author. If I can get a jdk7u Reviewer to review, I'll push this. > Will update copyright before pushing also. > > regards, > Sean. > > On 08/01/15 14:55, Daniel Fuchs wrote: >> Hi Se?n, >> >> Thanks for taking care of that :-) >> >> It looks good to me. You might want to set the copyright year to 2015. >> Best regards, >> >> -- daniel >> >> On 08/01/15 15:41, Se?n Coffey wrote: >>> I've taken suggested test code from Daniel and am looking to backport >>> 8066612 to jdk7u-dev. The test differs slightly in that it used the >>> non-lambda approach (ClassNameListBuilder) >>> >>> I've reduced the number of run counts also (one ovm run for secure and >>> one for non-secure mode) - As a result and given the large number of >>> classes loaded, I've bumped PermSize up to 128m - no harm to stress that >>> side of things also I guess. >>> >>> Test passes on all JPRT platforms. I'll revert to smaller class load >>> counts if issues are seen with this test approach. >>> >>> webrev : http://cr.openjdk.java.net/~coffeys/webrev.8066612.jdk7u/webrev/ >>> bug ID : https://bugs.openjdk.java.net/browse/JDK-8066612 >>> >>> regards, >>> Sean. >>> >>> >> > From ecki at zusammenkunft.net Tue Jan 13 17:11:00 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Tue, 13 Jan 2015 18:11:00 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B53EE5.1070707@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> <54B306AB.8010509@gmail.com> <54B446BD.7050306@gmail.com> <54B50345.4020808@cs.oswego.edu> <54B53EE5.1070707@gmail.com> Message-ID: <20150113181100.00002002.ecki@zusammenkunft.net> Hello, Windows 7 x64 on Core i7 Q720 (4C/8T/6MB L3, 1,6GHz) 8GB RAM my numbers in the gist are from a notebook, but I plan to repeat my and your benachmark versions on a different machine tomorow (which has less memory pressure), I will report. I was testing single threaded, will see if this makes a difference. Gruss Bernd Am Tue, 13 Jan 2015 16:51:01 +0100 schrieb Peter Levart : > On 01/13/2015 12:36 PM, Doug Lea wrote: > > On 01/12/2015 05:12 PM, Peter Levart wrote: > >> Hi, > >> > >> I added results obtained with JDK 8 (FCS and u20) - same machine, > >> same VM > >> options, just different JDKs: > > > > Thanks very much. It remains a mystery why the original report > > by Bernd showed a 40% difference from jdk7. The approx 10% hit > > seen in these more careful measurements is about what we > > expected as the intended tradeoff for avoiding DOS attacks > > and better performance in more common cases. There are > > still some tweaky improvements we should keep exploring, > > but I don't think there's any need for fundamental changes. > > > > -Doug > > Perhaps it was because of different HW. I ran the tests on Intel? > Core? i7-2600K CPU with 8 MB of Cache. I wonder what Bernd used. The > number of different keys is 250.000, they are accessed randomly in a > loop. Such HashMap consists of majority of TreeNode(s). TreeNode(s) > have 9 reference fields + int and boolean. I don't claim that such > HashMap fits intirely into Cache, but half of it perhaps does. Maybe > it is just that cache-hit ratio was larger on my HW than on Bernd's > and that Bernd's cache was thrashing more than mine. With JDK 7 > (normal Nodes have just 3 reference fields + int) such HashMap has > roughly half the memory footprint. > > Bernd, what did you run your benchmark on? > > > Peter > > > > >> > >> Original JDK 7 HashMap (and JVM): > >> > >> Benchmark (initialSize) Mode > >> Samples Score Score error Units > >> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >> 2839.458 157.299 ms > >> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >> 2673.924 187.063 ms > >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >> 686.972 32.928 ms > >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >> 631.001 6.574 ms > >> > >> Original JDK 8 HashMap (JDK 8 FCS JVM): > >> > >> Benchmark (initialSize) Mode > >> Samples Score Score error Units > >> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >> 3186.305 74.890 ms > >> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >> 2479.155 136.924 ms > >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >> 673.819 13.236 ms > >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >> 673.636 8.676 ms > >> > >> Original JDK 8 HashMap (JDK 8u20 JVM): > >> > >> Benchmark (initialSize) Mode > >> Samples Score Score error Units > >> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >> 3107.455 72.524 ms > >> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >> 2986.006 9.796 ms > >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >> 631.295 7.281 ms > >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >> 641.041 17.139 ms > >> > >> Original JDK 9 HashMap: > >> > >> Benchmark (initialSize) Mode > >> Samples Score Score error Units > >> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >> 3011.738 78.249 ms > >> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >> 2984.280 48.315 ms > >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >> 682.060 52.341 ms > >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >> 685.705 55.183 ms > >> > >> Original JDK 9 HashMap with TREEIFY_THRESHOLD = 1 << 20: > >> > >> Benchmark (initialSize) Mode > >> Samples Score Score error Units > >> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >> 2780.771 236.647 ms > >> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >> 2541.740 233.429 ms > >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >> 757.364 67.869 ms > >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >> 671.617 54.943 ms > >> > >> Caching of comparableClassFor (in ClassRepository - good for > >> heterogeneous keys > >> too): > >> > >> Benchmark (initialSize) Mode > >> Samples Score Score error Units > >> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >> 3014.888 71.778 ms > >> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >> 2279.757 54.159 ms > >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >> 760.743 70.674 ms > >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >> 725.188 67.853 ms > >> > >> Caching of comparableClassFor (internally - good for homogeneous > >> keys only): > >> > >> Benchmark (initialSize) Mode > >> Samples Score Score error Units > >> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >> 3026.707 84.571 ms > >> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >> 2137.296 66.140 ms > >> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >> 635.964 8.213 ms > >> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >> 685.129 46.783 ms > >> > >> > >> Peter > >> > >> On 01/12/2015 12:26 AM, Peter Levart wrote: > >>> > >>> On 01/11/2015 10:00 PM, Doug Lea wrote: > >>>> On 01/11/2015 02:26 PM, Peter Levart wrote: > >>>> > >>>>> Although majority of entries constitute the bins of size 13 or > >>>>> 14, there's only > >>>>> a single hashCode value per bin. > >>>>> > >>>>> So in this benchmark, treeifying with non-comparable keys is a > >>>>> waste of effort. > >>>> > >>>> On the other hand, the waste seems to only cost about 10% in > >>>> your runs. > >>>> I wonder why the original report using jdk7 vs jdk8 seemed > >>>> larger. > >>> > >>> I don't know. I ran the same benchmark with same VM options on > >>> JDK 7 too. Here > >>> are all results together: > >>> > >>> Original JDK 7 HashMap (and JVM): > >>> > >>> Benchmark (initialSize) Mode > >>> Samples Score Score error Units > >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >>> 2839.458 157.299 ms > >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >>> 2673.924 187.063 ms > >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >>> 686.972 32.928 ms > >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >>> 631.001 6.574 ms > >>> > >>> Original JDK 9 HashMap: > >>> > >>> Benchmark (initialSize) Mode > >>> Samples Score Score error Units > >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >>> 3011.738 78.249 ms > >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >>> 2984.280 48.315 ms > >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >>> 682.060 52.341 ms > >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >>> 685.705 55.183 ms > >>> > >>> Original JDK 9 HashMap with TREEIFY_THRESHOLD = 1 << 20: > >>> > >>> Benchmark (initialSize) Mode > >>> Samples Score Score error Units > >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >>> 2780.771 236.647 ms > >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >>> 2541.740 233.429 ms > >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >>> 757.364 67.869 ms > >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >>> 671.617 54.943 ms > >>> > >>> Caching of comparableClassFor (in ClassRepository - good for > >>> heterogeneous > >>> keys too): > >>> > >>> Benchmark (initialSize) Mode > >>> Samples Score Score error Units > >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >>> 3014.888 71.778 ms > >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >>> 2279.757 54.159 ms > >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >>> 760.743 70.674 ms > >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >>> 725.188 67.853 ms > >>> > >>> Caching of comparableClassFor (internally - good for homogeneous > >>> keys only): > >>> > >>> Benchmark (initialSize) Mode > >>> Samples Score Score error Units > >>> j.t.HashMapCollision.badDistNoComp 16 ss 60 > >>> 3026.707 84.571 ms > >>> j.t.HashMapCollision.badDistWithComp 16 ss 60 > >>> 2137.296 66.140 ms > >>> j.t.HashMapCollision.goodDistNoComp 16 ss 60 > >>> 635.964 8.213 ms > >>> j.t.HashMapCollision.goodDistWithComp 16 ss 60 > >>> 685.129 46.783 ms > >>> > >>>> > >>>>> > >>>>> Are there (non-forged) sets of non-comparable keys with > >>>>> hashCodes where > >>>>> treeifying makes sense? > >>>> > >>>> Try using a class like: > >>>> class FHC { float f; int hashCode() { return > >>>> Float.floatToIntBits(f); } } > >>>> and populate with instances with integral values for f. > >>>> > >>>> Similarly for doubles. > >>>> > >>>> Pre-jdk8, we devised a bit-smearing function that (among other > >>>> constraints) did OK for float/double keys with integral values, > >>>> that are not all that rare. With treeification, we don't need to > >>>> penalize classes with decent hashCodes by bit-smearing to still > >>>> get OK performance for these kinds of cases where the tree-based > >>>> hashCode comparison helps more than Comparability per se. > >>> > >>> I see. These keys actually have unique or near unique hashCodes > >>> but which are > >>> not good for power of two length tables without bit-smearing. > >>> With tree bins > >>> we don't need heavy bit-smearing to get decent performance in > >>> speed, but the > >>> table gets quite sparse anyway (although this is the smaller of > >>> the space > >>> overheads - tree nodes are bigger). For example, for 1M integral > >>> Floats, we > >>> get the following: > >>> > >>> >>> Float ... > >>> Capacity: 2097152 > >>> Load factor: 0.75 > >>> Size: 1000000 > >>> Bin sizes: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 > >>> 6*0 7*0 8*0 > >>> 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 > >>> Empty bins: 93.8 % > >>> Unique hash codes per bin: 0*1966080 1*0 2*0 3*24288 4*41248 5*0 > >>> 6*0 7*0 8*0 > >>> 9*0 10*4456 11*22963 12*30554 13*7539 14*24 total=1000000 > >>> > >>> > >>>> > >>>> Also... > >>>> > >>>> It looks like the simplest path to a minor improvement is > >>>> just to cache internally (your fourth test below). But I now > >>>> recall not doing this because it adds to footprint and > >>>> the field could prevent class unloading, for only a small > >>>> benefit. > >>> > >>> Footprint, yes (one reference field in HM instance), while class > >>> unloading is > >>> taken care of using WeakReference: > >>> > >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/HomogeneousKeysCache/webrev.01/ > >>> > >>> > >>>> > >>>> (Every time HashMap has changed, there have been reports of > >>>> performance regressions even though typical performance > >>>> generally improves.) > >>>> > >>>> -Doug > >>> > >>> Regards, Peter > >>> > >>>> > >>>> > >>>>>> Original JDK9 HashMap: > >>>>>> > >>>>>> Benchmark (initialSize) Mode > >>>>>> Samples Score Score error Units > >>>>>> j.t.HashMapCollision.badDistNoComp 16 > >>>>>> ss 60 3011.738 78.249 ms > >>>>>> j.t.HashMapCollision.badDistWithComp 16 > >>>>>> ss 60 2984.280 48.315 ms > >>>>>> j.t.HashMapCollision.goodDistNoComp 16 > >>>>>> ss 60 682.060 52.341 ms > >>>>>> j.t.HashMapCollision.goodDistWithComp 16 > >>>>>> ss 60 685.705 55.183 ms > >>>>>> > >>>>>> Original JDK9 HashMap with TREEIFY_THRESHOLD = 1 << 20: > >>>>>> > >>>>>> Benchmark (initialSize) Mode > >>>>>> Samples Score Score error Units > >>>>>> j.t.HashMapCollision.badDistNoComp 16 > >>>>>> ss 60 2780.771 236.647 ms > >>>>>> j.t.HashMapCollision.badDistWithComp 16 > >>>>>> ss 60 2541.740 233.429 ms > >>>>>> j.t.HashMapCollision.goodDistNoComp 16 > >>>>>> ss 60 757.364 67.869 ms > >>>>>> j.t.HashMapCollision.goodDistWithComp 16 > >>>>>> ss 60 671.617 54.943 ms > >>>>>> > >>>>>> Caching of comparableClassFor (in ClassRepository - good for > >>>>>> heterogeneous > >>>>>> keys too): > >>>>>> > >>>>>> Benchmark (initialSize) Mode > >>>>>> Samples Score Score error Units > >>>>>> j.t.HashMapCollision.badDistNoComp 16 > >>>>>> ss 60 3014.888 71.778 ms > >>>>>> j.t.HashMapCollision.badDistWithComp 16 > >>>>>> ss 60 2279.757 54.159 ms > >>>>>> j.t.HashMapCollision.goodDistNoComp 16 > >>>>>> ss 60 760.743 70.674 ms > >>>>>> j.t.HashMapCollision.goodDistWithComp 16 > >>>>>> ss 60 725.188 67.853 ms > >>>>>> > >>>>>> Caching of comparableClassFor (internally - good for > >>>>>> homogeneous keys only): > >>>>>> > >>>>>> Benchmark (initialSize) Mode > >>>>>> Samples Score Score error Units > >>>>>> j.t.HashMapCollision.badDistNoComp 16 > >>>>>> ss 60 3026.707 84.571 ms > >>>>>> j.t.HashMapCollision.badDistWithComp 16 > >>>>>> ss 60 2137.296 66.140 ms > >>>>>> j.t.HashMapCollision.goodDistNoComp 16 > >>>>>> ss 60 635.964 8.213 ms > >>>>>> j.t.HashMapCollision.goodDistWithComp 16 > >>>>>> ss 60 685.129 46.783 ms > >>>>>> > >>>>>> > >>>> > >>> > >> > > > From joe.darcy at oracle.com Wed Jan 14 02:46:59 2015 From: joe.darcy at oracle.com (joe darcy) Date: Tue, 13 Jan 2015 18:46:59 -0800 Subject: JDK 9 RFR of JDK-8068948: Update java.base module to use new try-with-resources statement Message-ID: <54B5D8A3.1020000@oracle.com> Hello, Earlier in JDK 9, a language change was made so that if a resource for a try-with-resources statement is already final or effectively final, a new resource variable does *not* need to be declared to manage the variable. The java.base module should take advantage of this language feature. I've run an experimental checker for such locations over the base module and found one candidate where the new language feature can be used; please review this patch for it: diff -r d873f6a7d16b src/java.base/unix/classes/java/io/FileDescriptor.java --- a/src/java.base/unix/classes/java/io/FileDescriptor.java Tue Jan 13 14:33:54 2015 -0800 +++ b/src/java.base/unix/classes/java/io/FileDescriptor.java Tue Jan 13 18:40:54 2015 -0800 @@ -214,7 +214,7 @@ if (!closed) { closed = true; IOException ioe = null; - try (Closeable c = releaser) { + try (releaser) { if (otherParents != null) { for (Closeable referent : otherParents) { try { The build succeeds with this change and the java.io.FileDescriptor regression tests pass. Thanks, -Joe From ivan.gerasimov at oracle.com Wed Jan 14 07:53:42 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Wed, 14 Jan 2015 10:53:42 +0300 Subject: JDK 9 RFR of JDK-8068948: Update java.base module to use new try-with-resources statement In-Reply-To: <54B5D8A3.1020000@oracle.com> References: <54B5D8A3.1020000@oracle.com> Message-ID: <54B62086.8060200@oracle.com> Hi! A useful change! Might it also make sense to allow anonymous variables in the try-with-resource statement? So that something like `try (() -> System.out.println("closed")) {}` would work... Would it make sense to also patch jdk/src/java.base/windows/classes/java/io/FileDescriptor.java in a similar manner, as it appears to contain the exactly same code? Sincerely yours, Ivan On 14.01.2015 5:46, joe darcy wrote: > Hello, > > Earlier in JDK 9, a language change was made so that if a resource for > a try-with-resources statement is already final or effectively final, > a new resource variable does *not* need to be declared to manage the > variable. > > The java.base module should take advantage of this language feature. > > I've run an experimental checker for such locations over the base > module and found one candidate where the new language feature can be > used; please review this patch for it: > > diff -r d873f6a7d16b > src/java.base/unix/classes/java/io/FileDescriptor.java > --- a/src/java.base/unix/classes/java/io/FileDescriptor.java Tue Jan > 13 14:33:54 2015 -0800 > +++ b/src/java.base/unix/classes/java/io/FileDescriptor.java Tue Jan > 13 18:40:54 2015 -0800 > @@ -214,7 +214,7 @@ > if (!closed) { > closed = true; > IOException ioe = null; > - try (Closeable c = releaser) { > + try (releaser) { > if (otherParents != null) { > for (Closeable referent : otherParents) { > try { > > The build succeeds with this change and the java.io.FileDescriptor > regression tests pass. > > Thanks, > > -Joe > > From Alan.Bateman at oracle.com Wed Jan 14 08:23:10 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 14 Jan 2015 08:23:10 +0000 Subject: JDK 9 RFR of JDK-8068948: Update java.base module to use new try-with-resources statement In-Reply-To: <54B5D8A3.1020000@oracle.com> References: <54B5D8A3.1020000@oracle.com> Message-ID: <54B6276E.8080803@oracle.com> On 14/01/2015 02:46, joe darcy wrote: > Hello, > > Earlier in JDK 9, a language change was made so that if a resource for > a try-with-resources statement is already final or effectively final, > a new resource variable does *not* need to be declared to manage the > variable. > > The java.base module should take advantage of this language feature. > > I've run an experimental checker for such locations over the base > module and found one candidate where the new language feature can be > used; please review this patch for it: > > diff -r d873f6a7d16b > src/java.base/unix/classes/java/io/FileDescriptor.java > --- a/src/java.base/unix/classes/java/io/FileDescriptor.java Tue Jan > 13 14:33:54 2015 -0800 > +++ b/src/java.base/unix/classes/java/io/FileDescriptor.java Tue Jan > 13 18:40:54 2015 -0800 > @@ -214,7 +214,7 @@ > if (!closed) { > closed = true; > IOException ioe = null; > - try (Closeable c = releaser) { > + try (releaser) { > if (otherParents != null) { > for (Closeable referent : otherParents) { > try { > > The build succeeds with this change and the java.io.FileDescriptor > regression tests pass. This looks to me and as Ivan noted, you can update the Windows implementation too. For the tests then make sure to run the :jdk_io and :jdk_nio tests as they will exercise this code well. -Alan From konstantin.shefov at oracle.com Wed Jan 14 11:14:05 2015 From: konstantin.shefov at oracle.com (Konstantin Shefov) Date: Wed, 14 Jan 2015 14:14:05 +0300 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54B3DC1E.1060900@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <19036153-CA73-435C-B1EA-7335D632927E@oracle.com> <54B3DC1E.1060900@oracle.com> Message-ID: <54B64F7D.6030109@oracle.com> Kindly reminder On 12.01.2015 17:37, Konstantin Shefov wrote: > Hello, Chris > > You have asked "Have you seen NetworkInterface.getName() return names > with ?_?, ?.?, or ?:? ,or is this theoretical?" > > I can answer your question. I have named one of network interfaces on > my Linux desktop like "eth0.1_55". > > ifconfig says: > eth0.1_55 Link encap:Ethernet HWaddr 08:00:27:10:c0:3e > inet addr:10.162.82.115 Bcast:10.162.83.255 Mask:255.255.254.0 > inet6 addr: 2606:b400:81c:1050:6568:8bee:d39b:68aa/64 > Scope:Global > inet6 addr: 2606:b400:81c:1044:a00:27ff:fe10:c03e/64 > Scope:Global > inet6 addr: 2606:b400:81c:1050:a00:27ff:fe10:c03e/64 > Scope:Global > inet6 addr: fe80::a00:27ff:fe10:c03e/64 Scope:Link > inet6 addr: 2606:b400:81c:1044:6568:8bee:d39b:68aa/64 > Scope:Global > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:1198 errors:0 dropped:0 overruns:0 frame:0 > TX packets:255 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:144325 (144.3 KB) TX bytes:31843 (31.8 KB) > > Simple java code > > Enumeration nis = > NetworkInterface.getNetworkInterfaces(); > System.out.println(nis.nextElement().getName()); > > returns exactly "eth0.1_55". > > This means we can include dot "." and underscore "_" symbols to URI > scope id. > > -Konstantin > > On 11.12.2014 18:29, Chris Hegarty wrote: >> On 11 Dec 2014, at 11:09, Konstantin Shefov >> wrote: >> >>> CC'ed core-libs-dev at openjdk.java.net >>> >>> On 10.12.2014 18:21, Konstantin Shefov wrote: >>>> Hello, >>>> >>>> Please, review the bug fix: >>>> https://bugs.openjdk.java.net/browse/JDK-6933879 >>>> Webrev: http://cr.openjdk.java.net/~kshefov/6933879/webrev.00 >>>> >>>> It is suggested to add some more symbols allowed for scope id in >>>> IPv6 URI, namely "_", "." and ":", because these symbols may be >>>> included in network interface names. Now only alphanumeric >>>> characters are allowed to be in a scope id. >> From Inet6Address, "Textual representation of IPv6 scoped addresses? >> [1] : >> >> ?? >> As a string. This must be the exact string that is returned by >> NetworkInterface.getName() for the particular interface in >> question. When an Inet6Address is created in this way, the >> numeric scope-id is determined at the time the object is >> created by querying the relevant NetworkInterface." >> >> Have you seen NetworkInterface.getName() return names with ?_?, ?.?, >> or ?:? ,or is this theoretical ? >> >> -Chris. >> >> [1] >> http://docs.oracle.com/javase/8/docs/api/java/net/Inet6Address.html#scoped >> >> >>>> Thanks >>>> -Konstantin > From chris.hegarty at oracle.com Wed Jan 14 11:37:26 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 14 Jan 2015 11:37:26 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <1421144693.7530.20.camel@Nokia-N900> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> Message-ID: <54B654F6.6060207@oracle.com> Peter F, I am still struggling with the basic concept of you proposal. Let me see if I understand it correctly. Does the following describe a similar scenario as you envisage: 1) For each Serializable type, T, in the deserialized types hierarchy, starting with the top most ( closest to j.l.Object ), 1a) Read T's fields from the stream, fields 1b) validate(t, fields) // t will be null first time 1c) allocate a new instance of T, and assign to t 1d) set fields in t 2) Return t; So for each level in the hierarchy, an instance of a type is created only after its invariants have been checked. This instance is then passed to the next level so it can participate in that levels invariants validation. If this scenario is along the same lines as yours, then I just don't see how 1c above will always be possible. If we could somehow make the object caller sensitive until after deserialization completes, then could avoid having to try to allocate multiple instance down the hierarchy. -Chris. On 13/01/15 10:24, Peter Firmstone wrote: > Could we use a static validator method and generate bytecode for > constructors dynamically? > > The developer can optionally implement the constructors. > > static GetField invariantCheck(GetField f); > > Create a caller sensitive GetField implementation and add a two new > methods to GetField: > > abstract Object createSuper(); // to access superclass object methods > for inavariant checking. > > abstract Class getType(String name); > > Set fields from within constructors. > > The generated constructors are straight forward: > > 1. Call static method. > 2. Call super class constructor with result from static method. > 3. Set final fields > 4. How to set transient fields, implement a private method called from > within the constructor? > > Require a permission to extend GetField? > From chris.hegarty at oracle.com Wed Jan 14 11:37:29 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 14 Jan 2015 11:37:29 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54B4ECA6.8050308@gmail.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <54B4ECA6.8050308@gmail.com> Message-ID: <54B654F9.1000709@oracle.com> On 13/01/15 10:00, Peter Levart wrote: > Hi Chris, > > I stepped back and asked myself what problem(s) we are searching > solution(s) for and tried to see how existing API solves or doesn't > solve them. I think that existing readObject() + GetFields API already > solves most of the problems discussed so far as it: > - provides encapsulation (using GetFields API we only have access to > deserialized values of the class where readObject() method being invoked > is declared) > - provides a means to check intra-object invariants in an "encapsulated" > way - the superclass state is already initialized/deserialized when > readObject() is called, so we can check GetFields provided values > against superclass state in a way that preserves encapsulation. > > What readObject()/GetFields combo doesn't provide is a relatively nice > solution for final fields (we have to resort to clumsy reflection). So > why not adding just that to the API? For example, extend > ObjectInputStream with the following method: > > ObjectInputStream { > ... > FieldAccess fields(); > ... > } > > > where FieldAccess is something like: > > public interface FieldAccess { > FieldAccess set(String fieldName, boolean value); > FieldAccess set(String fieldName, byte value); > FieldAccess set(String fieldName, char value); > FieldAccess set(String fieldName, short value); > FieldAccess set(String fieldName, int value); > FieldAccess set(String fieldName, long value); > FieldAccess set(String fieldName, float value); > FieldAccess set(String fieldName, double value); > FieldAccess set(String fieldName, T value); > } > > ...which would confine access to just those fields declared in the class > where readObject() being invoked is declared and just for the instance > and time of readObject() call-back. > > > Simple, flexible and stright-forward. It solves the problem with final > fields. It doesn't provide any auto-magic, which, although it might seem > pretty at first, can sometimes just be in the way because of > inflexibility. Sugar can be added though (in the form of > GetFields.defaultReadFields() or even special check() methods), but > flexibility is a necessary ingredient, I think. > > What do you think? I agree completely. An API at the level that you are proposing will provide the necessary functionality and flexibility that is required to do validation in readObject. As you clearly stated, and is already the case, validation in this way can depend on supertype state. Failure automicity of the whole type hierarchy is the only thing that is missing, but I think that could possibly be built on top, or solved in a different way. I don't see that as being a blocker for moving this forward. This proposal stands on its own merits. Peter, Is this something that you want to actively flesh out? If not, I can try to help move this forward. -Chris. > Regards, Peter > > On 01/12/2015 04:56 PM, Chris Hegarty wrote: >> On 10/01/15 07:00, Peter Firmstone wrote: >>> Again, thank you all for engaging in discussion of this very difficult >>> topic. >>> >>> While we can't presently check intra object dependencies during >>> deserialization with readObject(), the examples I provide can do this. >> >> I have replied to Davids mail with a small change to GetField ( added >> superTypeFields() ) to return the deserialized supertypes fields. This >> gives subtypes the ability to check values of the supertypes >> persistent state. >> >> As with your original proposal, it is limited to the persistent state >> as read off the stream, and not the transient state, but I think it >> gets us most of the way there. >> >> In your original proposal, it looks quite cumbersome to hook up the >> static validator method in the constructor hierarchy ( as it is to do >> this with standard constructors too ). It also relies on the fact that >> the subtype has to create an instance of the supertype. I just wonder >> if we can push on the alternative static validator proposal to come up >> with something a but more attractive ( maybe not! ). >> >>> ObjectInputValidation is sufficient for enforcing business rules, >>> however it can't stop an attacker. By the time the validator runs, the >>> attackers object of choice has been instantiated and it's game over. >>> That's right, the attacker may choose any Serializable class from the >>> classpath, and may even extend non serializable classes, with a zero arg >>> constructor, such as ClassLoader. >>> >>> To trust deserialization of an Object, as opposed to a String or >>> primitive, the ObjectInputStream (or overriding subclass) must limit the >>> classes allowed to be deserialized. >> >> Agreed. I am working on fleshing out a more concrete proposal on >> confinement, that I mentioned in previous mails. Hopefully, I will >> have something in the next few days. >> >>> For a class to be trusted, it must be trusted to check its parameters >>> and enforce its invariants during object deserialization and to not >>> deserialize with priviliged context. >>> >>> Validating an entire object graph's invariants, requires each class in >>> the graph to take responsibility for validating and enforcing its own >>> invariants. >> >> Agreed, and I think that the revised static validator method gives us >> this. >> >>> As shown in my earlier example, intra class invariants can be enforced >>> using Serialization constructors, from which static methods are called >>> to check invariants prior to super class constructors being called. >> >> Yes, but it is cumbersome and easy to make mistakes. >> >>> Presently, I override ObjectInputStream and use a Permission called >>> DeserializationPermission to limit classes that can be deserialized. >>> Untrusted connections are run from unprivileged context to limit classes >>> that can be instantiated, while those with trusted connections are run >>> with a Subject that allows any class to be deserialized. >> >> Interesting. I think there is overlap here with confinement. >> >>> ReadSerial, could do this: >>> Class c = rs.getType("foo"); >>> >>> And >>> >>> Foo f = rs.getObject("foo", Foo.class); >>> >>> Which performs instanceof type check, (prior to Object deserialization, >>> if first time deserialized, otherwise after) and generic complile time >>> type checked method return. Thus we must restrict the classes that can >>> be deserialized with ObjectInputStream. >> >> Have you seen the changes I proposing for failure atomicity, >> preliminary webrev >> http://cr.openjdk.java.net/~chegar/failureAtomicity/ >> >> .. and I think we can go further than this, creating the containing >> object lazily, if there are no readObjectXXX methods in the hierarchy. >> >>> If Foo is mutable and we want a private copy, the caller needs to copy >>> or clone it before checking invariants, as it would any mutable >>> constructor parameter. >>> >>> Each level of validation is the responsibility of the component with the >>> most knowledge. >>> >>> 1. Trusted class lists - administration, it might change. >>> 2. Object invariants and intra object invariants - Classes, not >>> objects. >>> 3. Business rules, but not security - ObjectInputValidation. >>> >>> So summing up, in order to secure deserialization we must validate all >>> data input, preferably before allowing object instantiation. >> >> Yes, where possible. If we can phase out readObjectXXX and replace it >> with a static validator, then I think this is possible. >> >>> Once an object has been constructed an attacker can gain a reference, >>> whether by a finalizer attack or some other cleverly crafted stream, the >>> attacker cannot obtain a reference to an object that doesn't exist. >>> >>> Circular links can allow an attacker to obtain an object reference >>> ,prior to its invariants being checked, when we rely on readObject() to >>> throw an exception. Delaying finalizer registration won't save you. >> >> Yes, circular references are nasty. >> >>> Classes with circular links should be final and use a transient boolean >>> "initialized" field, that every method checks to prevent an attacker >>> doing anything useful, should they gain a reference to an incorrectly >>> constructed object. >>> >>> The real question is, do we continue to plaster over the issues with the >>> Serialization framework's api, and continue to create special cases such >>> as a second final field freeze after a constructor completes? To be >>> honest, I don't like this second final field freeze, because invariants >>> haven't been checked. >> >> I think some of the discussion on this mailing list can provide APIs >> that avoid this, in the common case. >> >>> Don't get me wrong, Serialization is quite clever, but implementing >>> Serializable properly can consume a lot of time due to api complexity >>> and is presently a security problem. >> >> Yes, we too have spent too much time on this already. :-( I hope this >> discussion will result in concrete changes to help make Serialization >> more secure by default, and provide additional tools, confinement, >> etc, when this is not possible. >> >>> In my examples all constructors share the same invariant checks and >>> because it's public API, the invariants can be tested easily with unit >>> tests. >>> >>> It seems like we're trying to give constructor properties to a private >>> instance method at the expense of increasing complexity, wouldn't it be >>> simpler in the long term to provide a Serialization constructor? >> >> I think long term we will need a replacement for >> Unsafe.allocateInstance. There are a number of issues in JIRA about >> possibly adding support, with suitable permissions, to reflection, to >> create an instance without running its constructor. I need to track >> down the bugs and details, not sure if this is even a realistic >> possibility. >> >>> P.S. David, I like your suggestion of using a protected method, for >>> limiting array size. >> >> Agreed. This looks like it may have potential. I thought some work >> was done in this area in the recent past, restricting the size of the >> graph or objects. Let me see if I can find out more. >> >> -Chris. >> >>> Thank you, >>> >>> Peter. >>>> On 01/08/2015 02:10 PM, Brian Goetz wrote: >>>>>> >> 1) Validate invariants >>>>>> >> >>>>>> >> A clear and easy to understand mechanism that can validate >>>>>> the >>>>>> >> deserialized >>>>>> >> fields. Does not prevent the use of final fields, as the >>>>>> >> serialization framework >>>>>> >> will be responsible for setting them. Something along the >>>>>> lines >>>>>> >> of what David >>>>>> >> suggested: >>>>>> >> >>>>>> >> private static void validate(GetField fields) { >>>>>> >> if (fields.getInt("lo")> fields.getInt("hi")) { ... } >>>>>> >> } >>>>>> >> >>>>>> >> This could be a ?special? method, or annotation driven. >>>>>> TBD. >>>>>> >> >>>>>> >> Note: the validate method is static, so the object >>>>>> instance is >>>>>> >> not required to >>>>>> >> be created before running the validation. >>>>> > >>>>> > Sort of... >>>>> > >>>>> > This is true if the fields participating in the invariant are >>>>> > primitives. But if they're refs, what do you do? What if you >>>>> want to >>>>> > validate something like >>>>> > >>>>> > count == list.size() // fields are int count, List list >>>>> > >>>>> > ? Then wouldn't GetField.getObject have to deserialize the object >>>>> > referred to by that field? >>>> In fact you cannot validate invariants across multiple objects at all >>>> using this method*or* readObject() (existing or enhanced) unless the >>>> object in question is an enum, Class, or String (and a few other >>>> special >>>> cases) because you can't rely on initialization order during >>>> deserialization. That's the very reason why OIS#registerValidation() >>>> even exists - inter-object validation is not possible until after the >>>> root-most readObject has completed, which is the time when validations >>>> are executed. Robust validation of a given object class may require >>>> two >>>> stages - "near" validation and "spanning" validation - to fully >>>> validate. However the readObject() approach and its proposed >>>> variations >>>> (including the static validate() version) can still be useful for >>>> fail-fast and non-complex validations; you just have to understand that >>>> (just as today) any Object you examine might not be fully initialized >>>> yet. >>>> >>>> -- - DML > From chris.hegarty at oracle.com Wed Jan 14 11:39:26 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 14 Jan 2015 11:39:26 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <1421110269.7217.4.camel@Nokia-N900> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421110269.7217.4.camel@Nokia-N900> Message-ID: <54B6556E.4060306@oracle.com> On 13/01/15 00:51, Peter Firmstone wrote: > ----- Original message ----- > > On 10/01/15 07:00, Peter Firmstone wrote: > > > Again, thank you all for engaging in discussion of this very difficult > > > topic. > > > > > > While we can't presently check intra object dependencies during > > > deserialization with readObject(), the examples I provide can do this. > > > > I have replied to Davids mail with a small change to GetField ( added > > superTypeFields() ) to return the deserialized supertypes fields. This > > gives subtypes the ability to check values of the supertypes persistent > > state. > > Unfortunately this breaks encapsulation, a class is then locked into > using that serial form as public api forever. Yes, of course. Withdrawn. -Chris. From Alan.Bateman at oracle.com Wed Jan 14 12:56:28 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 14 Jan 2015 12:56:28 +0000 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54B64F7D.6030109@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <19036153-CA73-435C-B1EA-7335D632927E@oracle.com> <54B3DC1E.1060900@oracle.com> <54B64F7D.6030109@oracle.com> Message-ID: <54B6677C.1010303@oracle.com> On 14/01/2015 11:14, Konstantin Shefov wrote: > Kindly reminder > Did we establish if this requires a specification change to clarify the characters allowed in the host component? -Alan From peter.firmstone at zeus.net.au Wed Jan 14 12:58:54 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Wed, 14 Jan 2015 22:58:54 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54B654F6.6060207@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> Message-ID: <1421240334.8856.17.camel@Nokia-N900> Hi Chris, Sorry, no. Currently when an ObjectStreamClass is read in from the stream, the framework searches for the first zero arg constructor of a non serializable class and creates and instance of the class read and resolved from the stream, however it does so using a super class constructor. Then from the super class down, fields are read in and set in order for each class in the object's inheritance hierarchy. The alternative I propose, doesn't create the instance, instead it reads the fields from the stream, one by one and without instantiating them, if they are newly read objects, stores them temporarily into byte [] arrays in a Map with reference handle keys, otherwise it just holds the reference handle. What it does next is wrap this information into a caller sensitive api, GetFields or ReadSerial instance, that is passed as a constructor parameter to the child class serial constructor. The child class checks invariants and reads each field it needs using a static method prior to calling a superclass constructor, each class in the inheritance hierarchy for the object then checks its invariants until it gets to the first non serializable superclass. The benefit of this order is that each class is present in the thread security context, so protection domain security and invariants are enforced before instantiating an object. Hope this helps illuminate it a little better, regards, Peter. ----- Original message ----- > Peter F, > > I am still struggling with the basic concept of you proposal. Let me see > if I understand it correctly. Does the following describe a similar > scenario as you envisage: > >? ? 1) For each Serializable type, T, in the deserialized types >? ? ? ? ? hierarchy, starting with the top most ( closest to j.l.Object ), > >? ? ? ? ? 1a) Read T's fields from the stream, fields > >? ? ? ? ? 1b) validate(t, fields)? // t will be null first time > >? ? ? ? ? 1c) allocate a new instance of T, and assign to t > >? ? ? ? ? 1d) set fields in t > >? ? 2) Return t; > > So for each level in the hierarchy, an instance of a type is created > only after its invariants have been checked. This instance is then > passed to the next level so it can participate in that levels invariants > validation. > > If this scenario is along the same lines as yours, then I just don't see > how 1c above will always be possible. > > If we could somehow make the object caller sensitive until after > deserialization completes, then could avoid having to try to allocate > multiple instance down the hierarchy. > > -Chris. > > On 13/01/15 10:24, Peter Firmstone wrote: > > Could we use a static validator method and generate bytecode for > > constructors dynamically? > > > > The developer can optionally implement the constructors. > > > > static GetField invariantCheck(GetField f); > > > > Create a caller sensitive GetField implementation and add a two new > > methods to GetField: > > > > abstract Object createSuper(); // to access superclass object methods > > for inavariant checking. > > > > abstract Class getType(String name); > > > > Set fields from within constructors. > > > > The generated constructors are straight forward: > > > > 1. Call static method. > > 2. Call super class constructor with result from static method. > > 3. Set final fields > > 4. How to set transient fields, implement a private method called from > > within the constructor? > > > > Require a permission to extend GetField? > > From chris.hegarty at oracle.com Wed Jan 14 13:55:31 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 14 Jan 2015 13:55:31 +0000 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem Message-ID: <54B67553.8030401@oracle.com> This test currently completes successfully, but indicates that it has "Parsed 0 classfiles". It doesn't do anything useful. The modular image no longer contains jar files. The test should use the JRT FileSystem to iterate over the image classfiles. http://cr.openjdk.java.net/~chegar/8061297/webrev.00/webrev/ -Chris. From chris.hegarty at oracle.com Wed Jan 14 14:15:53 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 14 Jan 2015 14:15:53 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <1421240334.8856.17.camel@Nokia-N900> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> Message-ID: <54B67A19.1050507@oracle.com> On 14/01/15 12:58, Peter Firmstone wrote: > Hi Chris, > > Sorry, no. > > Currently when an ObjectStreamClass is read in from the stream, the > framework searches for the first zero arg constructor of a non > serializable class and creates and instance of the class read and > resolved from the stream, however it does so using a super class > constructor. > > Then from the super class down, fields are read in and set in order for > each class in the object's inheritance hierarchy. This is my understanding also. > The alternative I propose, doesn't create the instance, instead it reads > the fields from the stream, one by one and without instantiating them, > if they are newly read objects, stores them temporarily into byte [] > arrays in a Map with reference handle keys, otherwise it just holds the > reference handle. GetField.get(...), or ReadSerial, will create an object before returning it. So the invariant checker methods will turn the stream field values into objects at some point. Also, from your previous examples, I though I seen the invariant checker method explicitly create superclass instance, so that it could be used in the validation, right? What I was trying to do was to reduce your examples, and description, into a sequence of steps, to try to better understand how this would work. What I came up with, I believe, is equivalent to what was done in this example [1]. If you intend to chain the "serial constructors" up with static check methods that create superclass instances, then I think is equates to top-down checking as I have described it. > What it does next is wrap this information into a caller sensitive api, > GetFields or ReadSerial instance, that is passed as a constructor > parameter to the child class serial constructor. > > The child class checks invariants and reads each field it needs using a > static method prior to calling a superclass constructor, each class in > the inheritance hierarchy for the object then checks its invariants > until it gets to the first non serializable superclass. > > The benefit of this order is that each class is present in the thread > security context, so protection domain security and invariants are > enforced before instantiating an object. > > Hope this helps illuminate it a little better, regards, -Chris. [1] http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-January/030550.html > > Peter. > > ----- Original message ----- > > Peter F, > > > > I am still struggling with the basic concept of you proposal. Let me see > > if I understand it correctly. Does the following describe a similar > > scenario as you envisage: > > > > 1) For each Serializable type, T, in the deserialized types > > hierarchy, starting with the top most ( closest to > j.l.Object ), > > > > 1a) Read T's fields from the stream, fields > > > > 1b) validate(t, fields) // t will be null first time > > > > 1c) allocate a new instance of T, and assign to t > > > > 1d) set fields in t > > > > 2) Return t; > > > > So for each level in the hierarchy, an instance of a type is created > > only after its invariants have been checked. This instance is then > > passed to the next level so it can participate in that levels invariants > > validation. > > > > If this scenario is along the same lines as yours, then I just don't see > > how 1c above will always be possible. > > > > If we could somehow make the object caller sensitive until after > > deserialization completes, then could avoid having to try to allocate > > multiple instance down the hierarchy. > > > > -Chris. > > > > On 13/01/15 10:24, Peter Firmstone wrote: > > > Could we use a static validator method and generate bytecode for > > > constructors dynamically? > > > > > > The developer can optionally implement the constructors. > > > > > > static GetField invariantCheck(GetField f); > > > > > > Create a caller sensitive GetField implementation and add a two new > > > methods to GetField: > > > > > > abstract Object createSuper(); // to access superclass object methods > > > for inavariant checking. > > > > > > abstract Class getType(String name); > > > > > > Set fields from within constructors. > > > > > > The generated constructors are straight forward: > > > > > > 1. Call static method. > > > 2. Call super class constructor with result from static method. > > > 3. Set final fields > > > 4. How to set transient fields, implement a private method called from > > > within the constructor? > > > > > > Require a permission to extend GetField? > > > > From Alan.Bateman at oracle.com Wed Jan 14 14:29:03 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 14 Jan 2015 14:29:03 +0000 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem In-Reply-To: <54B67553.8030401@oracle.com> References: <54B67553.8030401@oracle.com> Message-ID: <54B67D2F.9@oracle.com> On 14/01/2015 13:55, Chris Hegarty wrote: > This test currently completes successfully, but indicates that it has > "Parsed 0 classfiles". It doesn't do anything useful. > > The modular image no longer contains jar files. The test should use > the JRT FileSystem to iterate over the image classfiles. > > http://cr.openjdk.java.net/~chegar/8061297/webrev.00/webrev/ > > -Chris. This looks okay to me. One thing to simplify it a bit is to drop the roots from JrtClasses as there is only a single root in the jrt file system. The exists check can also be dropped at L199 as isDirectory can only return true if the directory exists. -Alan. From paul.sandoz at oracle.com Wed Jan 14 14:32:54 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 14 Jan 2015 15:32:54 +0100 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem In-Reply-To: <54B67553.8030401@oracle.com> References: <54B67553.8030401@oracle.com> Message-ID: <4A8495AA-BF25-4117-85C9-9EE4DBBE9661@oracle.com> On Jan 14, 2015, at 2:55 PM, Chris Hegarty wrote: > This test currently completes successfully, but indicates that it has "Parsed 0 classfiles". It doesn't do anything useful. > > The modular image no longer contains jar files. The test should use the JRT FileSystem to iterate over the image classfiles. > > http://cr.openjdk.java.net/~chegar/8061297/webrev.00/webrev/ > 191 static class PlatformClassPath { 192 static List getClasses() throws IOException { 193 List result = new ArrayList(); 194 Path home = Paths.get(System.getProperty("java.home")); 195 196 if (home.resolve("lib").toFile().exists()) { 197 // either an exploded build or an image 198 File classes = home.resolve("modules").toFile(); 199 if (classes.exists() && classes.isDirectory()) { 200 result.add(classes.toPath()); 201 } else { 202 JrtClasses jrt = new JrtClasses(); 203 result.addAll(jrt.paths()); 204 } There is no need to copy the list. At line 200 one can return a singleton list. The list is copied yet again if there are no explicit classes added: 73 if (classes.isEmpty()) { 74 classes.addAll(PlatformClassPath.getClasses()); 75 } You should be able to just copy the reference if classes is changed to List: classes = PlatformClassPath.getClasses(); (IIUC there is just one pass over the classes so you could probably modify to be totally lazy and process on the Stream directly but that would require slightly more modification to the run method) 230 private Stream toStream(Path root) { 231 try { 232 return Files.walk(root); 233 } catch(IOException x) { 234 x.printStackTrace(System.err); 235 } 236 return Collections.emptyList().stream(); Is there a reason why the exception is swallowed? If not it would be better to wrap in an UncheckedIOException and re-throw. In general is this test thread safe if multiple tasks are submitted in a thread pool? i.e. can the visit method be called concurrently and update the non-synchronized csMethodsMissingAnnotation? Paul. From konstantin.shefov at oracle.com Wed Jan 14 15:12:07 2015 From: konstantin.shefov at oracle.com (Konstantin Shefov) Date: Wed, 14 Jan 2015 18:12:07 +0300 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54B6677C.1010303@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <19036153-CA73-435C-B1EA-7335D632927E@oracle.com> <54B3DC1E.1060900@oracle.com> <54B64F7D.6030109@oracle.com> <54B6677C.1010303@oracle.com> Message-ID: <54B68747.2010508@oracle.com> On 14.01.2015 15:56, Alan Bateman wrote: > Did we establish if this requires a specification change to clarify > the characters allowed in the host component? Here http://docs.oracle.com/javase/7/docs/api/java/net/Inet6Address.html#scoped we can see that "The scope_id refers to an interface on the local system, and it can be specified in two ways. 1. As a numeric identifier........ 2. As a string. This must be the exact string that is returned by NetworkInterface.getName() for the particular interface in question. When an Inet6Address is created in this way, the numeric scope-id is determined at the time the object is created by querying the relevant NetworkInterface." I have configured a host where NetworkInterface.getName() returns "eth0.1_55", i.e. "dot" and "underscore" should be added to allowed chars. And it seems we do not have to change that specification. -Konstantin From peter.levart at gmail.com Wed Jan 14 15:50:46 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 14 Jan 2015 16:50:46 +0100 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B50345.4020808@cs.oswego.edu> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> <54B306AB.8010509@gmail.com> <54B446BD.7050306@gmail.com> <54B50345.4020808@cs.oswego.edu> Message-ID: <54B69056.7000307@gmail.com> Hi, Loosely related to this topic, there is some local variable caching of comparableClassFor() result already being performed inside iterative/recursive methods of TreeNode, but this caching is just positive caching, meaning that null return is not cached. For keys that happen to implement Comparable, but are not simple-self-comparable (their class is not of the form: class C implements Comparable), reflection is invoked each time comparison of the search key with Node key is attempted. This can be improved: http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/InCallNegativeCache/webrev.01/ Using modified version of Bernd's benchmark (adding 3rd variant of keys - FalseComp): http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/InCallNegativeCache/HashMapCollision.java ...and and only running for badHash shows almost 25% improvement for FalseComp keys (skipping reflection invocations) and barely measurable improvement for NoComp keys (skipping instanceof checks), while Comp keys are not affected. Original JDK9 HashMap: Benchmark (initialSize) Mode Samples Score Error Units j.t.HashMapCollision.badDistFalseComp 16 ss 20 4221.438 +- 250.774 ms j.t.HashMapCollision.badDistNoComp 16 ss 20 2868.605 +- 40.754 ms j.t.HashMapCollision.badDistWithComp 16 ss 20 3030.780 +- 111.315 ms Patched: Benchmark (initialSize) Mode Samples Score Error Units j.t.HashMapCollision.badDistFalseComp 16 ss 20 3237.953 +- 143.608 ms j.t.HashMapCollision.badDistNoComp 16 ss 20 2643.024 +- 137.067 ms j.t.HashMapCollision.badDistWithComp 16 ss 20 3087.902 +- 122.041 ms Regards, Peter From chris.hegarty at oracle.com Wed Jan 14 16:04:53 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 14 Jan 2015 16:04:53 +0000 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem In-Reply-To: <4A8495AA-BF25-4117-85C9-9EE4DBBE9661@oracle.com> References: <54B67553.8030401@oracle.com> <4A8495AA-BF25-4117-85C9-9EE4DBBE9661@oracle.com> Message-ID: <54B693A5.4060801@oracle.com> Alan, Paul, Sometimes with these changes you do not want to cloud the significant body of the change with other refactoring. That said, we are on round two, so it seems reasonable. I think I have captured/addressed all comments: http://cr.openjdk.java.net/~chegar/8061297/webrev.01/webrev/ -Chris. On 14/01/15 14:32, Paul Sandoz wrote: > > On Jan 14, 2015, at 2:55 PM, Chris Hegarty wrote: > >> This test currently completes successfully, but indicates that it has "Parsed 0 classfiles". It doesn't do anything useful. >> >> The modular image no longer contains jar files. The test should use the JRT FileSystem to iterate over the image classfiles. >> >> http://cr.openjdk.java.net/~chegar/8061297/webrev.00/webrev/ >> > > 191 static class PlatformClassPath { > 192 static List getClasses() throws IOException { > 193 List result = new ArrayList(); > 194 Path home = Paths.get(System.getProperty("java.home")); > 195 > 196 if (home.resolve("lib").toFile().exists()) { > 197 // either an exploded build or an image > 198 File classes = home.resolve("modules").toFile(); > 199 if (classes.exists() && classes.isDirectory()) { > 200 result.add(classes.toPath()); > 201 } else { > 202 JrtClasses jrt = new JrtClasses(); > 203 result.addAll(jrt.paths()); > 204 } > > There is no need to copy the list. At line 200 one can return a singleton list. > > The list is copied yet again if there are no explicit classes added: > > 73 if (classes.isEmpty()) { > 74 classes.addAll(PlatformClassPath.getClasses()); > 75 } > > You should be able to just copy the reference if classes is changed to List: > > classes = PlatformClassPath.getClasses(); > > (IIUC there is just one pass over the classes so you could probably modify to be totally lazy and process on the Stream directly but that would require slightly more modification to the run method) > > > 230 private Stream toStream(Path root) { > 231 try { > 232 return Files.walk(root); > 233 } catch(IOException x) { > 234 x.printStackTrace(System.err); > 235 } > 236 return Collections.emptyList().stream(); > > Is there a reason why the exception is swallowed? If not it would be better to wrap in an UncheckedIOException and re-throw. > > In general is this test thread safe if multiple tasks are submitted in a thread pool? i.e. can the visit method be called concurrently and update the non-synchronized csMethodsMissingAnnotation? > > Paul. > From mandy.chung at oracle.com Wed Jan 14 16:03:08 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 14 Jan 2015 08:03:08 -0800 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem In-Reply-To: <54B67553.8030401@oracle.com> References: <54B67553.8030401@oracle.com> Message-ID: <54B6933C.5030304@oracle.com> On 1/14/2015 5:55 AM, Chris Hegarty wrote: > This test currently completes successfully, but indicates that it has > "Parsed 0 classfiles". It doesn't do anything useful. > > The modular image no longer contains jar files. The test should use > the JRT FileSystem to iterate over the image classfiles. > > http://cr.openjdk.java.net/~chegar/8061297/webrev.00/webrev/ Thanks for taking this on. This looks okay. line 196: checking if $JAVA_HOME/lib doesn't seem necessary as the current runtime is valid, otherwise it won't startup. It may be good to change getClasses() to return Stream. As Paul points out the thread-safe issue, csMethodsMissingAnnotation should be CopyOnWriteArrayList. Mandy From mandy.chung at oracle.com Wed Jan 14 16:18:56 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 14 Jan 2015 08:18:56 -0800 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem In-Reply-To: <54B693A5.4060801@oracle.com> References: <54B67553.8030401@oracle.com> <4A8495AA-BF25-4117-85C9-9EE4DBBE9661@oracle.com> <54B693A5.4060801@oracle.com> Message-ID: <54B696F0.7010701@oracle.com> On 1/14/2015 8:04 AM, Chris Hegarty wrote: > > http://cr.openjdk.java.net/~chegar/8061297/webrev.01/webrev/ Looks okay to me. In CallerSensitiveFinder.java line 57, 80, 137 - not sure if you were thinking to pass "pool" to the CallerSensitiveFinder constructor. Looks like some editing you meant to cleanup. nit: line 76 a space between "classes" and "==" line 207 can be removed. Mandy From chris.hegarty at oracle.com Wed Jan 14 16:31:44 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 14 Jan 2015 16:31:44 +0000 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem In-Reply-To: <54B696F0.7010701@oracle.com> References: <54B67553.8030401@oracle.com> <4A8495AA-BF25-4117-85C9-9EE4DBBE9661@oracle.com> <54B693A5.4060801@oracle.com> <54B696F0.7010701@oracle.com> Message-ID: <54B699F0.2020209@oracle.com> On 14/01/15 16:18, Mandy Chung wrote: > On 1/14/2015 8:04 AM, Chris Hegarty wrote: >> >> http://cr.openjdk.java.net/~chegar/8061297/webrev.01/webrev/ > > Looks okay to me. > > In CallerSensitiveFinder.java line 57, 80, 137 - not sure if you were > thinking to pass "pool" to the CallerSensitiveFinder constructor. Looks > like some editing you meant to cleanup. > > nit: line 76 a space between "classes" and "==" > line 207 can be removed. D'oh! Fixed both of these. Webrev updated in place. -Chris. > Mandy > > From joe.darcy at oracle.com Wed Jan 14 16:29:41 2015 From: joe.darcy at oracle.com (joe darcy) Date: Wed, 14 Jan 2015 08:29:41 -0800 Subject: JDK 9 RFR of JDK-8068948: Update java.base module to use new try-with-resources statement In-Reply-To: <54B62086.8060200@oracle.com> References: <54B5D8A3.1020000@oracle.com> <54B62086.8060200@oracle.com> Message-ID: <54B69975.3010802@oracle.com> Hello, On 1/13/2015 11:53 PM, Ivan Gerasimov wrote: > Hi! > > A useful change! I agree :-) We haven't publicized the change very much yet, but it is part of JEP 213, Milling Project Coin, (http://openjdk.java.net/jeps/213) and went into JDK 9's javac late in 2014. > Might it also make sense to allow anonymous variables in the > try-with-resource statement? > So that something like `try (() -> System.out.println("closed")) {}` > would work... I don't think that is necessary. In JDK 7, we started out allowing a general AutoCloseable expression in a try-with-resources statement and that proved problematic. Supporting a final / effectively final variable seems to be the right balance. The rationale is written up in the JSR 334 materials. > > Would it make sense to also patch > jdk/src/java.base/windows/classes/java/io/FileDescriptor.java in a > similar manner, as it appears to contain the exactly same code? Good catch. Thanks, -Joe > > Sincerely yours, > Ivan > > On 14.01.2015 5:46, joe darcy wrote: >> Hello, >> >> Earlier in JDK 9, a language change was made so that if a resource >> for a try-with-resources statement is already final or effectively >> final, a new resource variable does *not* need to be declared to >> manage the variable. >> >> The java.base module should take advantage of this language feature. >> >> I've run an experimental checker for such locations over the base >> module and found one candidate where the new language feature can be >> used; please review this patch for it: >> >> diff -r d873f6a7d16b >> src/java.base/unix/classes/java/io/FileDescriptor.java >> --- a/src/java.base/unix/classes/java/io/FileDescriptor.java Tue Jan >> 13 14:33:54 2015 -0800 >> +++ b/src/java.base/unix/classes/java/io/FileDescriptor.java Tue Jan >> 13 18:40:54 2015 -0800 >> @@ -214,7 +214,7 @@ >> if (!closed) { >> closed = true; >> IOException ioe = null; >> - try (Closeable c = releaser) { >> + try (releaser) { >> if (otherParents != null) { >> for (Closeable referent : otherParents) { >> try { >> >> The build succeeds with this change and the java.io.FileDescriptor >> regression tests pass. >> >> Thanks, >> >> -Joe >> >> > From ivan.st.ivanov at gmail.com Wed Jan 14 19:25:38 2015 From: ivan.st.ivanov at gmail.com (Ivan St. Ivanov) Date: Wed, 14 Jan 2015 21:25:38 +0200 Subject: RFR: 5050783: Throwable convenience method: String getStackTraceString() In-Reply-To: <54A3FAEB.7010709@oracle.com> References: <54A3FAEB.7010709@oracle.com> Message-ID: Hi Claes, core libs developers, As promised, we did yesterday our hackathon in the Bulgarian Java User Group and incorporated the feedback that we got from you. Here is our next webrev: http://dmitryalexandrov.net/~bgjug/5050783/webrev.01/ As you can see: - We got rid of the inner classes altogether in favor of java.util.function.Consumer - The javadoc was updated to follow the standards [1] - In the tests we use an exception that we create rather than relying on one from the JDK that potentially could change We also ran some microbenchmarks to compare the performance of the new functionality with the Apache commons solution mentioned in the bug. Here is the benchmark test: @State(Scope.Benchmark) public class ThrowableStacktraceBenchmark { Throwable chainedThrowable; Throwable simpleThrowable; @Setup public void prepare() { chainedThrowable = createThrowable(64); simpleThrowable = new Throwable("Simple Throwable"); } Throwable createThrowable(int depthLevel) { Throwable t = null; for (int i = 0; i < depthLevel; ++i) { t = new Throwable("My dummy Throwable " + i, t); } return t; } @Benchmark public void benchmarkThrowableGetStackTraceString() { chainedThrowable.getStackTraceString(); } @Benchmark public void benchmarkThrowableGetStackTraceStringSmall() { simpleThrowable.getStackTraceString(); } @Benchmark public void benchmarkThrowablePrintStackTraceToStringWriter() { StringWriter sw = new StringWriter(); chainedThrowable.printStackTrace(new PrintWriter(sw)); sw.toString(); } @Benchmark public void benchmarkThrowablePrintStackTraceToStringWriterSmall() { StringWriter sw = new StringWriter(); simpleThrowable.printStackTrace(new PrintWriter(sw)); sw.toString(); } } And here are the results (test was run with the following options -wi 5 -i 5 -t 1): Benchmark Mode Samples Score Error Units o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTraceString thrpt 50 25351.743 ? 347.326 ops/s o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTraceStringSmall thrpt 50 134990.643 ? 760.225 ops/s o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTraceToStringWriter thrpt 50 22730.786 ? 167.378 ops/s o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTraceToStringWriterSmall thrpt 50 115223.025 ? 936.770 ops/s Just in any case we ran this test, actually just the last two methods, with JDK 1.9 ea build 45. We wanted to compare the result of the Apache commons solution with inner classes (as it is now) to the same, but with java.util.function.Consumer. There are the results with the inner classes: o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTraceToStringWriter thrpt 50 22974.324 ? 391.564 ops/s o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTraceToStringWriterSmall thrpt 50 114973.452 ? 1051.276 ops/s So you can see that the performance is nearly the same (I guess that the difference is ignorable). Please share with us the feedback about this proposal! Thanks, Ivan [1] http://www.oracle.com/technetwork/articles/java/index-137868.html On Wed, Dec 31, 2014 at 3:32 PM, Claes Redestad wrote: > Hi, > > seems like a reasonable convenience method. :-) > > Process stuff: OCA/JSPA signing aside, the patch needs to be submitted in > its > entirety to some part of the OpenJDK infrastructure before it can be > accepted. > Attaching the patch in an e-mail to this mailing list is acceptable for a > small > patch like this when lacking upload privileges to cr.openjdk.java.net. A > sponsor > could help you with the uploading etc. > > This request adds a new public method, which means some internal approval > will be necessary. This is something you'll need help with from an Oracle > sponsor. I could volunteer to initiate the requests internally. > > Code feedback: > > The test relies on the message generated when provoking a division by > zero. I'm not sure this is a good idea nor the place to enforce this > message format with a test. Perhaps just throwing a RuntimeException > with some specified message and test for that? > > May I suggest to also add a test to ensure the method behaves well for an > exception with fillInStackTrace overridden to produce stackless exceptions? > > Just a thought... I noticed PrintWriter, PrintStream and StringBuilder all > inherit from Appendable, so the PrintStreamOrWriter/StackTracePrinter > classes > could be replaced with a single StackTraceAppendable taking an Appendable. > One > static class instead of 1 abstract and 3 concrete static inner classes > could work > out to be a good deal, but there's some odd mechanics in > BufferedWriter/PrintWriter > to use the value of the line.separator property at object creation time > which > would be hard to support without exposing the lineSeparator fields to > Throwable. > Perhaps a future consideration. > > Performance characteristics might change ever so slightly either way, so it > would be nice with some microbenchmarks to quantify. > > /Claes > > > On 2014-12-31 11:51, Ivan St. Ivanov wrote: > >> Hello, >> >> I am Ivan from the Bulgarian Java User Group. As part of our Adopt OpenJDK >> efforts, we organized hackathon beginning of this month in Sofia. There we >> worked on the following JIRA issue: >> >> https://bugs.openjdk.java.net/browse/JDK-5050783 >> >> Unfortunately none of us has author privileges, so the issue is still >> unassigned. But, nevertheless, our JUG has already signed JSPA and some of >> our members (including myself) have signed OCA. And we informed in a >> special email this mailing list about our plans prior to the meeting. >> >> Our hackathon was already documented by our JUG member Mihail Stoynov: >> >> http://mihail.stoynov.com/2014/12/12/building-openjdk- >> and-submitting-a-solution-on-a-feature-request-with-the- >> bulgarian-java-user-group/ >> >> We used the days after the meeting for offline polishing our contribution >> and adding some more unit tests. Here is the result of our work: >> >> http://dmitryalexandrov.net/~bgjug/5050783/webrev.00/ >> >> We would be very happy if you are able to review it, share your feedback >> and guide us through the contribution process as this is our first >> attempt. >> >> Thanks and have a very successful year! >> Ivan >> > > From forax at univ-mlv.fr Wed Jan 14 19:44:40 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 14 Jan 2015 20:44:40 +0100 Subject: RFR: 5050783: Throwable convenience method: String getStackTraceString() In-Reply-To: References: <54A3FAEB.7010709@oracle.com> Message-ID: <54B6C728.2030203@univ-mlv.fr> While using a lambda is cool in that context, if someone tries to do a printStackTrace in the LambdaMetafactory (for debugging purpose by example), it will be fun to debug. cheers, R?mi On 01/14/2015 08:25 PM, Ivan St. Ivanov wrote: > Hi Claes, core libs developers, > > As promised, we did yesterday our hackathon in the Bulgarian Java User > Group and incorporated the feedback that we got from you. Here is our next > webrev: > > http://dmitryalexandrov.net/~bgjug/5050783/webrev.01/ > > As you can see: > > - We got rid of the inner classes altogether in favor of > java.util.function.Consumer > - The javadoc was updated to follow the standards [1] > - In the tests we use an exception that we create rather than relying on > one from the JDK that potentially could change > > We also ran some microbenchmarks to compare the performance of the new > functionality with the Apache commons solution mentioned in the bug. Here > is the benchmark test: > > @State(Scope.Benchmark) > public class ThrowableStacktraceBenchmark { > Throwable chainedThrowable; > Throwable simpleThrowable; > > @Setup > public void prepare() { > chainedThrowable = createThrowable(64); > simpleThrowable = new Throwable("Simple Throwable"); > } > > Throwable createThrowable(int depthLevel) { > Throwable t = null; > for (int i = 0; i < depthLevel; ++i) { > t = new Throwable("My dummy Throwable " + i, t); > } > return t; > } > > @Benchmark > public void benchmarkThrowableGetStackTraceString() { > chainedThrowable.getStackTraceString(); > } > > @Benchmark > public void benchmarkThrowableGetStackTraceStringSmall() { > simpleThrowable.getStackTraceString(); > } > > @Benchmark > public void benchmarkThrowablePrintStackTraceToStringWriter() { > StringWriter sw = new StringWriter(); > chainedThrowable.printStackTrace(new PrintWriter(sw)); > sw.toString(); > } > > @Benchmark > public void benchmarkThrowablePrintStackTraceToStringWriterSmall() { > StringWriter sw = new StringWriter(); > simpleThrowable.printStackTrace(new PrintWriter(sw)); > sw.toString(); > } > } > > And here are the results (test was run with the following options -wi 5 -i > 5 -t 1): > > Benchmark Mode Samples Score Error Units > o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTraceString > thrpt 50 25351.743 ? 347.326 ops/s > o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTraceStringSmall > thrpt 50 134990.643 ? 760.225 ops/s > o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTraceToStringWriter > thrpt 50 22730.786 ? 167.378 ops/s > o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTraceToStringWriterSmall > thrpt 50 115223.025 ? 936.770 ops/s > > Just in any case we ran this test, actually just the last two methods, with > JDK 1.9 ea build 45. We wanted to compare the result of the Apache commons > solution with inner classes (as it is now) to the same, but with > java.util.function.Consumer. There are the results with the inner classes: > > o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTraceToStringWriter > thrpt 50 22974.324 ? 391.564 ops/s > o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTraceToStringWriterSmall > thrpt 50 114973.452 ? 1051.276 ops/s > > So you can see that the performance is nearly the same (I guess that the > difference is ignorable). > > Please share with us the feedback about this proposal! > > Thanks, > Ivan > > [1] http://www.oracle.com/technetwork/articles/java/index-137868.html > > On Wed, Dec 31, 2014 at 3:32 PM, Claes Redestad > wrote: > >> Hi, >> >> seems like a reasonable convenience method. :-) >> >> Process stuff: OCA/JSPA signing aside, the patch needs to be submitted in >> its >> entirety to some part of the OpenJDK infrastructure before it can be >> accepted. >> Attaching the patch in an e-mail to this mailing list is acceptable for a >> small >> patch like this when lacking upload privileges to cr.openjdk.java.net. A >> sponsor >> could help you with the uploading etc. >> >> This request adds a new public method, which means some internal approval >> will be necessary. This is something you'll need help with from an Oracle >> sponsor. I could volunteer to initiate the requests internally. >> >> Code feedback: >> >> The test relies on the message generated when provoking a division by >> zero. I'm not sure this is a good idea nor the place to enforce this >> message format with a test. Perhaps just throwing a RuntimeException >> with some specified message and test for that? >> >> May I suggest to also add a test to ensure the method behaves well for an >> exception with fillInStackTrace overridden to produce stackless exceptions? >> >> Just a thought... I noticed PrintWriter, PrintStream and StringBuilder all >> inherit from Appendable, so the PrintStreamOrWriter/StackTracePrinter >> classes >> could be replaced with a single StackTraceAppendable taking an Appendable. >> One >> static class instead of 1 abstract and 3 concrete static inner classes >> could work >> out to be a good deal, but there's some odd mechanics in >> BufferedWriter/PrintWriter >> to use the value of the line.separator property at object creation time >> which >> would be hard to support without exposing the lineSeparator fields to >> Throwable. >> Perhaps a future consideration. >> >> Performance characteristics might change ever so slightly either way, so it >> would be nice with some microbenchmarks to quantify. >> >> /Claes >> >> >> On 2014-12-31 11:51, Ivan St. Ivanov wrote: >> >>> Hello, >>> >>> I am Ivan from the Bulgarian Java User Group. As part of our Adopt OpenJDK >>> efforts, we organized hackathon beginning of this month in Sofia. There we >>> worked on the following JIRA issue: >>> >>> https://bugs.openjdk.java.net/browse/JDK-5050783 >>> >>> Unfortunately none of us has author privileges, so the issue is still >>> unassigned. But, nevertheless, our JUG has already signed JSPA and some of >>> our members (including myself) have signed OCA. And we informed in a >>> special email this mailing list about our plans prior to the meeting. >>> >>> Our hackathon was already documented by our JUG member Mihail Stoynov: >>> >>> http://mihail.stoynov.com/2014/12/12/building-openjdk- >>> and-submitting-a-solution-on-a-feature-request-with-the- >>> bulgarian-java-user-group/ >>> >>> We used the days after the meeting for offline polishing our contribution >>> and adding some more unit tests. Here is the result of our work: >>> >>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.00/ >>> >>> We would be very happy if you are able to review it, share your feedback >>> and guide us through the contribution process as this is our first >>> attempt. >>> >>> Thanks and have a very successful year! >>> Ivan >>> >> From ivan.st.ivanov at gmail.com Wed Jan 14 20:34:00 2015 From: ivan.st.ivanov at gmail.com (Ivan St. Ivanov) Date: Wed, 14 Jan 2015 22:34:00 +0200 Subject: RFR: 5050783: Throwable convenience method: String getStackTraceString() In-Reply-To: <54B6C728.2030203@univ-mlv.fr> References: <54A3FAEB.7010709@oracle.com> <54B6C728.2030203@univ-mlv.fr> Message-ID: Hi Remi, I tried it and, well, the JDK image now does not even compile classes that contain lambda expressions: javac -cp . ./LambdaMetaFactoryTester.java Exception in thread "main" Exception: java.lang.BootstrapMethodError thrown from the UncaughtExceptionHandler in thread "main" With the old Throwable implementation it printed a long stack trace of exceptions, but at least my sample class was compiled. So, I have two questions: 1) Does anyone dump stack traces in such location, even for debugging purposes? 2) If yes, what are the other JDK classes and methods where we should avoid using lambdas? Thanks, Ivan On Wed, Jan 14, 2015 at 9:44 PM, Remi Forax wrote: > While using a lambda is cool in that context, > if someone tries to do a printStackTrace in the LambdaMetafactory (for > debugging purpose by example), > it will be fun to debug. > > cheers, > R?mi > > > On 01/14/2015 08:25 PM, Ivan St. Ivanov wrote: > >> Hi Claes, core libs developers, >> >> As promised, we did yesterday our hackathon in the Bulgarian Java User >> Group and incorporated the feedback that we got from you. Here is our next >> webrev: >> >> http://dmitryalexandrov.net/~bgjug/5050783/webrev.01/ >> >> As you can see: >> >> - We got rid of the inner classes altogether in favor of >> java.util.function.Consumer >> - The javadoc was updated to follow the standards [1] >> - In the tests we use an exception that we create rather than relying on >> one from the JDK that potentially could change >> >> We also ran some microbenchmarks to compare the performance of the new >> functionality with the Apache commons solution mentioned in the bug. Here >> is the benchmark test: >> >> @State(Scope.Benchmark) >> public class ThrowableStacktraceBenchmark { >> Throwable chainedThrowable; >> Throwable simpleThrowable; >> >> @Setup >> public void prepare() { >> chainedThrowable = createThrowable(64); >> simpleThrowable = new Throwable("Simple Throwable"); >> } >> >> Throwable createThrowable(int depthLevel) { >> Throwable t = null; >> for (int i = 0; i < depthLevel; ++i) { >> t = new Throwable("My dummy Throwable " + i, t); >> } >> return t; >> } >> >> @Benchmark >> public void benchmarkThrowableGetStackTraceString() { >> chainedThrowable.getStackTraceString(); >> } >> >> @Benchmark >> public void benchmarkThrowableGetStackTraceStringSmall() { >> simpleThrowable.getStackTraceString(); >> } >> >> @Benchmark >> public void benchmarkThrowablePrintStackTraceToStringWriter() { >> StringWriter sw = new StringWriter(); >> chainedThrowable.printStackTrace(new PrintWriter(sw)); >> sw.toString(); >> } >> >> @Benchmark >> public void benchmarkThrowablePrintStackTraceToStringWriterSmall() { >> StringWriter sw = new StringWriter(); >> simpleThrowable.printStackTrace(new PrintWriter(sw)); >> sw.toString(); >> } >> } >> >> And here are the results (test was run with the following options -wi 5 -i >> 5 -t 1): >> >> Benchmark Mode Samples Score Error Units >> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTraceString >> thrpt 50 25351.743 ? 347.326 ops/s >> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTrac >> eStringSmall >> thrpt 50 134990.643 ? 760.225 ops/s >> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >> aceToStringWriter >> thrpt 50 22730.786 ? 167.378 ops/s >> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >> aceToStringWriterSmall >> thrpt 50 115223.025 ? 936.770 ops/s >> >> Just in any case we ran this test, actually just the last two methods, >> with >> JDK 1.9 ea build 45. We wanted to compare the result of the Apache commons >> solution with inner classes (as it is now) to the same, but with >> java.util.function.Consumer. There are the results with the inner classes: >> >> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >> aceToStringWriter >> thrpt 50 22974.324 ? 391.564 ops/s >> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >> aceToStringWriterSmall >> thrpt 50 114973.452 ? 1051.276 ops/s >> >> So you can see that the performance is nearly the same (I guess that the >> difference is ignorable). >> >> Please share with us the feedback about this proposal! >> >> Thanks, >> Ivan >> >> [1] http://www.oracle.com/technetwork/articles/java/index-137868.html >> >> On Wed, Dec 31, 2014 at 3:32 PM, Claes Redestad < >> claes.redestad at oracle.com> >> wrote: >> >> Hi, >>> >>> seems like a reasonable convenience method. :-) >>> >>> Process stuff: OCA/JSPA signing aside, the patch needs to be submitted in >>> its >>> entirety to some part of the OpenJDK infrastructure before it can be >>> accepted. >>> Attaching the patch in an e-mail to this mailing list is acceptable for a >>> small >>> patch like this when lacking upload privileges to cr.openjdk.java.net. A >>> sponsor >>> could help you with the uploading etc. >>> >>> This request adds a new public method, which means some internal approval >>> will be necessary. This is something you'll need help with from an Oracle >>> sponsor. I could volunteer to initiate the requests internally. >>> >>> Code feedback: >>> >>> The test relies on the message generated when provoking a division by >>> zero. I'm not sure this is a good idea nor the place to enforce this >>> message format with a test. Perhaps just throwing a RuntimeException >>> with some specified message and test for that? >>> >>> May I suggest to also add a test to ensure the method behaves well for an >>> exception with fillInStackTrace overridden to produce stackless >>> exceptions? >>> >>> Just a thought... I noticed PrintWriter, PrintStream and StringBuilder >>> all >>> inherit from Appendable, so the PrintStreamOrWriter/StackTracePrinter >>> classes >>> could be replaced with a single StackTraceAppendable taking an >>> Appendable. >>> One >>> static class instead of 1 abstract and 3 concrete static inner classes >>> could work >>> out to be a good deal, but there's some odd mechanics in >>> BufferedWriter/PrintWriter >>> to use the value of the line.separator property at object creation time >>> which >>> would be hard to support without exposing the lineSeparator fields to >>> Throwable. >>> Perhaps a future consideration. >>> >>> Performance characteristics might change ever so slightly either way, so >>> it >>> would be nice with some microbenchmarks to quantify. >>> >>> /Claes >>> >>> >>> On 2014-12-31 11:51, Ivan St. Ivanov wrote: >>> >>> Hello, >>>> >>>> I am Ivan from the Bulgarian Java User Group. As part of our Adopt >>>> OpenJDK >>>> efforts, we organized hackathon beginning of this month in Sofia. There >>>> we >>>> worked on the following JIRA issue: >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-5050783 >>>> >>>> Unfortunately none of us has author privileges, so the issue is still >>>> unassigned. But, nevertheless, our JUG has already signed JSPA and some >>>> of >>>> our members (including myself) have signed OCA. And we informed in a >>>> special email this mailing list about our plans prior to the meeting. >>>> >>>> Our hackathon was already documented by our JUG member Mihail Stoynov: >>>> >>>> http://mihail.stoynov.com/2014/12/12/building-openjdk- >>>> and-submitting-a-solution-on-a-feature-request-with-the- >>>> bulgarian-java-user-group/ >>>> >>>> We used the days after the meeting for offline polishing our >>>> contribution >>>> and adding some more unit tests. Here is the result of our work: >>>> >>>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.00/ >>>> >>>> We would be very happy if you are able to review it, share your feedback >>>> and guide us through the contribution process as this is our first >>>> attempt. >>>> >>>> Thanks and have a very successful year! >>>> Ivan >>>> >>>> >>> > From peter.firmstone at zeus.net.au Wed Jan 14 20:41:32 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Thu, 15 Jan 2015 06:41:32 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54B67A19.1050507@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54B67A19.1050507@oracle.com> Message-ID: <1421268092.8958.5.camel@Nokia-N900> It's only top down if there are intra class invariants, in this case invariants are checked more than once. This done simply because we can't move up the constructor call chain untill all child class invariants have been satisfied. For simpler class relationships, invariant check order is bottom up. The child class cannot allow a super class constructor to be called until its own invariants have been satisfied, in order to prevent an illegal instance of itself being created. If we check invariants after the object has been created, unless we introduce a check initialized method into every method, we loose. Unfortunately we can't share this check with child classes, so the class really should be declared final, or all its methods should be, including any superclass methods, including Object equals and hashcode. Unfortunately enforcing security by enforcing invariants after calling a superclass constructor, is far more complicated. Using a serial constructor, is less evil. Regards, Peter. ----- Original message ----- > On 14/01/15 12:58, Peter Firmstone wrote: > > Hi Chris, > > > > Sorry, no. > > > > Currently when an ObjectStreamClass is read in from the stream, the > > framework searches for the first zero arg constructor of a non > > serializable class and creates and instance of the class read and > > resolved from the stream, however it does so using a super class > > constructor. > > > > Then from the super class down, fields are read in and set in order for > > each class in the object's inheritance hierarchy. > > This is my understanding also. > > > The alternative I propose, doesn't create the instance, instead it > > reads the fields from the stream, one by one and without instantiating > > them, if they are newly read objects, stores them temporarily into > > byte [] arrays in a Map with reference handle keys, otherwise it just > > holds the reference handle. > > GetField.get(...), or ReadSerial, will create an object before returning > it. So the invariant checker methods will turn the stream field values > into objects at some point. > > Also, from your previous examples, I though I seen the invariant checker > method explicitly create superclass instance, so that it could be used > in the validation, right? > > What I was trying to do was to reduce your examples, and description, > into a sequence of steps, to try to better understand how this would > work. > > What I came up with, I believe, is equivalent to what was done in this > example [1]. If you intend to chain the "serial constructors" up with > static check methods that create superclass instances, then I think is > equates to top-down checking as I have described it. > > > > What it does next is wrap this information into a caller sensitive api, > > GetFields or ReadSerial instance, that is passed as a constructor > > parameter to the child class serial constructor. > > > > The child class checks invariants and reads each field it needs using a > > static method prior to calling a superclass constructor, each class in > > the inheritance hierarchy for the object then checks its invariants > > until it gets to the first non serializable superclass. > > > > The benefit of this order is that each class is present in the thread > > security context, so protection domain security and invariants are > > enforced before instantiating an object. > > > > Hope this helps illuminate it a little better, regards, > > > -Chris. > > [1] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-January/030550.html > > > > > Peter. > > > > ----- Original message ----- > > Peter F, > > > > I am still struggling with the basic concept of you proposal. Let me > > see if I understand it correctly. Does the following describe a similar > > scenario as you envisage: > > > > 1) For each Serializable type, T, in the deserialized types > > hierarchy, starting with the top most ( closest to > > j.l.Object ), > > > > 1a) Read T's fields from the stream, fields > > > > 1b) validate(t, fields)? // t will be null first time > > > > 1c) allocate a new instance of T, and assign to t > > > > 1d) set fields in t > > > > 2) Return t; > > > > So for each level in the hierarchy, an instance of a type is created > > only after its invariants have been checked. This instance is then > > passed to the next level so it can participate in that levels > > invariants validation. > > > > If this scenario is along the same lines as yours, then I just don't > > see how 1c above will always be possible. > > > > If we could somehow make the object caller sensitive until after > > deserialization completes, then could avoid having to try to allocate > > multiple instance down the hierarchy. > > > > -Chris. > > > > On 13/01/15 10:24, Peter Firmstone wrote: > > Could we use a static validator method and generate bytecode for > > constructors dynamically? > > > > The developer can optionally implement the constructors. > > > > static GetField invariantCheck(GetField f); > > > > Create a caller sensitive GetField implementation and add a two new > > methods to GetField: > > > > abstract Object createSuper(); // to access superclass object methods > > for inavariant checking. > > > > abstract Class getType(String name); > > > > Set fields from within constructors. > > > > The generated constructors are straight forward: > > > > 1. Call static method. > > 2. Call super class constructor with result from static method. > > 3. Set final fields > > 4. How to set transient fields, implement a private method called from > > within the constructor? > > > > Require a permission to extend GetField? > > > > From peter.levart at gmail.com Wed Jan 14 20:56:24 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 14 Jan 2015 21:56:24 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <54B654F9.1000709@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <54B4ECA6.8050308@gmail.com> <54B654F9.1000709@oracle.com> Message-ID: <54B6D7F8.7020004@gmail.com> On 01/14/2015 12:37 PM, Chris Hegarty wrote: >> What do you think? > > I agree completely. > > An API at the level that you are proposing will provide the necessary > functionality and flexibility that is required to do validation in > readObject. As you clearly stated, and is already the case, validation > in this way can depend on supertype state. > > Failure automicity of the whole type hierarchy is the only thing that > is missing, but I think that could possibly be built on top, or solved > in a different way. I don't see that as being a blocker for moving > this forward. This proposal stands on its own merits. > > Peter, > Is this something that you want to actively flesh out? If not, I can > try to help move this forward. Hi Chris, I can prepare a prototype, yes. Just give me a couple of days. Regards, Peter > > -Chris. From mandy.chung at oracle.com Wed Jan 14 21:51:20 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 14 Jan 2015 13:51:20 -0800 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem In-Reply-To: <54B699F0.2020209@oracle.com> References: <54B67553.8030401@oracle.com> <4A8495AA-BF25-4117-85C9-9EE4DBBE9661@oracle.com> <54B693A5.4060801@oracle.com> <54B696F0.7010701@oracle.com> <54B699F0.2020209@oracle.com> Message-ID: <54B6E4D8.2000704@oracle.com> On 1/14/2015 8:31 AM, Chris Hegarty wrote: > On 14/01/15 16:18, Mandy Chung wrote: >> On 1/14/2015 8:04 AM, Chris Hegarty wrote: >>> >>> http://cr.openjdk.java.net/~chegar/8061297/webrev.01/webrev/ >> >> Looks okay to me. >> >> In CallerSensitiveFinder.java line 57, 80, 137 - not sure if you were >> thinking to pass "pool" to the CallerSensitiveFinder constructor. Looks >> like some editing you meant to cleanup. >> >> nit: line 76 a space between "classes" and "==" >> line 207 can be removed. > > D'oh! Fixed both of these. Webrev updated in place. Thumbs up. Mandy From peter.firmstone at zeus.net.au Wed Jan 14 22:47:15 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Thu, 15 Jan 2015 08:47:15 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54B67A19.1050507@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54B67A19.1050507@oracle.com> Message-ID: <1421275635.9067.5.camel@Nokia-N900> The following is a summary of our exploration so far: We have narrowed down to two options: A. constructors with static invariant check methods. B. Continue to use readObject(), we have a proposed solution to the final field problem and could modify the jvm to register for finalization late to avoid finalizer attacks. Question: How can a programmer protect against reference stealing attacks? Once this is taken into consideration, B's complexity exceeds A. This is a good guideline for constructors, but deserialization offers more attack vectors: https://www.securecoding.cert.org/confluence/display/java/TSM01-J.+Do+not+let+the+this+reference+escape+during+object+construction Item A.: * Prevents reference stealing attacks. * A subclass checking its invariants can only create a superclass instance for intra invariant checking, if superclass invariants are satisfied. Logic Steps for item A: 1. Framework calls child class serial constructor 2. Child class constructor calls static invariant check. 3. For each class in hierarchy: a. If current class has no intra class invariants: * Check invariants and return result to super class constructor or throw Exception. b. Else if the child class has intra class invariants: * Create a superclass instance (invariants are checked before it can be created). * Check invariants and return result to super class constructor or throw Exception. 7. Each Superclass checks its invariants, repeating step 3, until Object's constructor is called, or an Exception is thrown. 8. Constructors return, each class descending down the object hierarchy sets fields. Process is complete and new object is published. Remember that when satisfying invariants, class types can be checked before deserializing an object. For flexibility this could be in the form of a permission check. Regards, Peter. From joe.darcy at oracle.com Thu Jan 15 01:03:16 2015 From: joe.darcy at oracle.com (joe darcy) Date: Wed, 14 Jan 2015 17:03:16 -0800 Subject: JDK 9 RFR of 8060077: Class.toGenericString specification doesn't mention array types Message-ID: <54B711D4.7070902@oracle.com> Hello, Please review these changes to address JDK-8060077 : Class.toGenericString specification doesn't mention array types http://cr.openjdk.java.net/~darcy/8060077.0/ Thanks, -Joe From felix.yang at oracle.com Thu Jan 15 07:15:50 2015 From: felix.yang at oracle.com (FELIX YANG) Date: Thu, 15 Jan 2015 15:15:50 +0800 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem In-Reply-To: <54B699F0.2020209@oracle.com> References: <54B67553.8030401@oracle.com> <4A8495AA-BF25-4117-85C9-9EE4DBBE9661@oracle.com> <54B693A5.4060801@oracle.com> <54B696F0.7010701@oracle.com> <54B699F0.2020209@oracle.com> Message-ID: <54B76926.9040405@oracle.com> Chris, excuse me, I'm not a reviewer but have some questions. at line 209, classes is actually trying to resolve a path like "c:\jdk1.9.0\modules", which never exists in JDK directory. It should be "c:\jdk1.9.0\lib\modules". 207 if (home.resolve("lib").toFile().exists()) { 208 // either an exploded build or an image *209 File classes = home.resolve("modules").toFile();* 210 if (classes.isDirectory()) { 211 return Stream.of(classes.toPath()); 212 } else { 213 return JrtPaths(); 214 } The test works because it always go to JrtPaths method. The jimages have been mounted at initializing of Jrt file system. There may be two issues here: 1. the naming*J*rtPaths doesn't follow common rules. Is it more suitable with*j*rtPaths()? 2. Even when changing classes to "lib/modules", it is still probably unable to process class files. I tried last code and failed. *File classes = home.resolve("lib/modules").toFile();* Because, com.sun.tools.jdeps.ClassFileReader hasn't been upgraded to correctly handle Jrt image files. So this logic is either unnecessary (as stated above, the jimages have been mounted at initializing of Jrt file system) or unable to work now. Thanks, -Felix On 1/15/2015 12:31 AM, Chris Hegarty wrote: > On 14/01/15 16:18, Mandy Chung wrote: >> On 1/14/2015 8:04 AM, Chris Hegarty wrote: >>> >>> http://cr.openjdk.java.net/~chegar/8061297/webrev.01/webrev/ >> >> Looks okay to me. >> >> In CallerSensitiveFinder.java line 57, 80, 137 - not sure if you were >> thinking to pass "pool" to the CallerSensitiveFinder constructor. Looks >> like some editing you meant to cleanup. >> >> nit: line 76 a space between "classes" and "==" >> line 207 can be removed. > > D'oh! Fixed both of these. Webrev updated in place. > > -Chris. > >> Mandy >> >> From chris.hegarty at oracle.com Thu Jan 15 09:28:17 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 15 Jan 2015 09:28:17 +0000 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem In-Reply-To: <54B76926.9040405@oracle.com> References: <54B67553.8030401@oracle.com> <4A8495AA-BF25-4117-85C9-9EE4DBBE9661@oracle.com> <54B693A5.4060801@oracle.com> <54B696F0.7010701@oracle.com> <54B699F0.2020209@oracle.com> <54B76926.9040405@oracle.com> Message-ID: <07FD783A-E3BD-4C0C-BFB8-8DDFBA930A4B@oracle.com> Felix, On 15 Jan 2015, at 07:15, FELIX YANG wrote: > Chris, > excuse me, I'm not a reviewer but have some questions. All comments and questions are welcome. > at line 209, classes is actually trying to resolve a path like "c:\jdk1.9.0\modules", which never exists in JDK directory. It should be "c:\jdk1.9.0\lib\modules?. The 'jdk/modules' directory will only exist when running with an exploded build. It is useful to be able to run tests against both exploded builds and images. > 207 if (home.resolve("lib").toFile().exists()) { > 208 // either an exploded build or an image The check for 'lib' is not strictly necessary, I will remove it, as it is confusing. > 209 File classes = home.resolve("modules").toFile(); > > 210 if (classes.isDirectory()) { > 211 return Stream.of(classes.toPath()); > 212 } else { > 213 return JrtPaths(); > 214 } > > The test works because it always go to JrtPaths method. The jimages have been mounted at initializing of Jrt file system. When running with a binary image, then yes it will always use the JRT code path. > There may be two issues here: > 1. the naming > JrtPaths doesn't follow common rules. Is it more suitable with j > rtPaths()? I?ll make this change before pushing. > 2. Even when changing classes to "lib/modules", it is still probably unable to process class files. I tried last code and failed. > > File classes = home.resolve("lib/modules").toFile(); ?lib/modules? is not a path that is search for classes by the runtime. Previous comments should have already covered the requirement for ?modules? Thanks for looking at this felix., -Chris. > Because, com.sun.tools.jdeps.ClassFileReader hasn't been upgraded to correctly handle Jrt image files. > > So this logic is either unnecessary (as stated above, the jimages have been mounted at initializing of Jrt file system) or unable to work now. > > Thanks, > > -Felix > On 1/15/2015 12:31 AM, Chris Hegarty wrote: >> On 14/01/15 16:18, Mandy Chung wrote: >>> On 1/14/2015 8:04 AM, Chris Hegarty wrote: >>>> >>>> http://cr.openjdk.java.net/~chegar/8061297/webrev.01/webrev/ >>> >>> Looks okay to me. >>> >>> In CallerSensitiveFinder.java line 57, 80, 137 - not sure if you were >>> thinking to pass "pool" to the CallerSensitiveFinder constructor. Looks >>> like some editing you meant to cleanup. >>> >>> nit: line 76 a space between "classes" and "==" >>> line 207 can be removed. >> >> D'oh! Fixed both of these. Webrev updated in place. >> >> -Chris. >> >>> Mandy >>> >>> > From paul.sandoz at oracle.com Thu Jan 15 10:11:06 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 15 Jan 2015 11:11:06 +0100 Subject: RFR 8061297 : sun/reflect/CallerSensitive/CallerSensitiveFinder.java should use the JRT FileSystem In-Reply-To: <54B699F0.2020209@oracle.com> References: <54B67553.8030401@oracle.com> <4A8495AA-BF25-4117-85C9-9EE4DBBE9661@oracle.com> <54B693A5.4060801@oracle.com> <54B696F0.7010701@oracle.com> <54B699F0.2020209@oracle.com> Message-ID: <5BDDC56F-ECF1-4D37-9B50-08D699013FE6@oracle.com> On Jan 14, 2015, at 5:31 PM, Chris Hegarty wrote: > On 14/01/15 16:18, Mandy Chung wrote: >> On 1/14/2015 8:04 AM, Chris Hegarty wrote: >>> >>> http://cr.openjdk.java.net/~chegar/8061297/webrev.01/webrev/ >> >> Looks okay to me. >> >> In CallerSensitiveFinder.java line 57, 80, 137 - not sure if you were >> thinking to pass "pool" to the CallerSensitiveFinder constructor. Looks >> like some editing you meant to cleanup. >> >> nit: line 76 a space between "classes" and "==" >> line 207 can be removed. > > D'oh! Fixed both of these. Webrev updated in place. > Looks good. Paul. From peter.firmstone at zeus.net.au Thu Jan 15 12:25:36 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Thu, 15 Jan 2015 22:25:36 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54B67A19.1050507@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54B67A19.1050507@oracle.com> Message-ID: <54B7B1C0.4010802@zeus.net.au> Chris, Can you explain some of the detail in your failure atomicity work? Thanks, Peter. From claes.redestad at oracle.com Thu Jan 15 12:24:34 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Thu, 15 Jan 2015 13:24:34 +0100 Subject: RFR: 5050783: Throwable convenience method: String getStackTraceString() In-Reply-To: References: <54A3FAEB.7010709@oracle.com> <54B6C728.2030203@univ-mlv.fr> Message-ID: <54B7B182.5060206@oracle.com> Hi, anything directly used by the LambdaMetafactory itself would of course be impossible to add lambdas to, and as Remi points out, this makes lambdaified utilities unusable from that subsystem unless there are ways around them, e.g., exposing a desugared method along with the lambdaified API (which would defeat the potential for footprint optimizations). I think this patch/review should focus on the need for the utility method itself. As I suggested privately, I think a minimal implementation of this convenience method using existing code would be a good starting point: public String getStackTraceString() { StringWriter sw = new StringWriter(); printStackTrace(new PrintWriter(sw)); return sw.toString(); } Your benchmark indicates this would have a 10-15% overhead over a StringBuilder-based implementation, which I think is OK for a convenience method such as this. If accepted we can start to think about ways to optimize the internals. On that note, I'm thinking loudly about whether or not it'd be appropriate to extract common methods in PrintStream and PrintWriter to an interface. java.io.Printable? This would allow getting rid of most inner classes in j.l.Throwable along with some duplicated code across the JDK (I've noticed at least a few places where there are identical methods which differ only in that one takes a PrintStream, the other a PrintWriter). This could grow into a larger effort, but maybe it's worth it since the compatibility risks should be minimal. /Claes On 01/14/2015 09:34 PM, Ivan St. Ivanov wrote: > Hi Remi, > > I tried it and, well, the JDK image now does not even compile classes that > contain lambda expressions: > > javac -cp . ./LambdaMetaFactoryTester.java > Exception in thread "main" > Exception: java.lang.BootstrapMethodError thrown from the > UncaughtExceptionHandler in thread "main" > > With the old Throwable implementation it printed a long stack trace of > exceptions, but at least my sample class was compiled. > > So, I have two questions: > 1) Does anyone dump stack traces in such location, even for debugging > purposes? > 2) If yes, what are the other JDK classes and methods where we should avoid > using lambdas? > > Thanks, > Ivan > > On Wed, Jan 14, 2015 at 9:44 PM, Remi Forax wrote: > >> While using a lambda is cool in that context, >> if someone tries to do a printStackTrace in the LambdaMetafactory (for >> debugging purpose by example), >> it will be fun to debug. >> >> cheers, >> R?mi >> >> >> On 01/14/2015 08:25 PM, Ivan St. Ivanov wrote: >> >>> Hi Claes, core libs developers, >>> >>> As promised, we did yesterday our hackathon in the Bulgarian Java User >>> Group and incorporated the feedback that we got from you. Here is our next >>> webrev: >>> >>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.01/ >>> >>> As you can see: >>> >>> - We got rid of the inner classes altogether in favor of >>> java.util.function.Consumer >>> - The javadoc was updated to follow the standards [1] >>> - In the tests we use an exception that we create rather than relying on >>> one from the JDK that potentially could change >>> >>> We also ran some microbenchmarks to compare the performance of the new >>> functionality with the Apache commons solution mentioned in the bug. Here >>> is the benchmark test: >>> >>> @State(Scope.Benchmark) >>> public class ThrowableStacktraceBenchmark { >>> Throwable chainedThrowable; >>> Throwable simpleThrowable; >>> >>> @Setup >>> public void prepare() { >>> chainedThrowable = createThrowable(64); >>> simpleThrowable = new Throwable("Simple Throwable"); >>> } >>> >>> Throwable createThrowable(int depthLevel) { >>> Throwable t = null; >>> for (int i = 0; i < depthLevel; ++i) { >>> t = new Throwable("My dummy Throwable " + i, t); >>> } >>> return t; >>> } >>> >>> @Benchmark >>> public void benchmarkThrowableGetStackTraceString() { >>> chainedThrowable.getStackTraceString(); >>> } >>> >>> @Benchmark >>> public void benchmarkThrowableGetStackTraceStringSmall() { >>> simpleThrowable.getStackTraceString(); >>> } >>> >>> @Benchmark >>> public void benchmarkThrowablePrintStackTraceToStringWriter() { >>> StringWriter sw = new StringWriter(); >>> chainedThrowable.printStackTrace(new PrintWriter(sw)); >>> sw.toString(); >>> } >>> >>> @Benchmark >>> public void benchmarkThrowablePrintStackTraceToStringWriterSmall() { >>> StringWriter sw = new StringWriter(); >>> simpleThrowable.printStackTrace(new PrintWriter(sw)); >>> sw.toString(); >>> } >>> } >>> >>> And here are the results (test was run with the following options -wi 5 -i >>> 5 -t 1): >>> >>> Benchmark Mode Samples Score Error Units >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTraceString >>> thrpt 50 25351.743 ? 347.326 ops/s >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTrac >>> eStringSmall >>> thrpt 50 134990.643 ? 760.225 ops/s >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>> aceToStringWriter >>> thrpt 50 22730.786 ? 167.378 ops/s >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>> aceToStringWriterSmall >>> thrpt 50 115223.025 ? 936.770 ops/s >>> >>> Just in any case we ran this test, actually just the last two methods, >>> with >>> JDK 1.9 ea build 45. We wanted to compare the result of the Apache commons >>> solution with inner classes (as it is now) to the same, but with >>> java.util.function.Consumer. There are the results with the inner classes: >>> >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>> aceToStringWriter >>> thrpt 50 22974.324 ? 391.564 ops/s >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>> aceToStringWriterSmall >>> thrpt 50 114973.452 ? 1051.276 ops/s >>> >>> So you can see that the performance is nearly the same (I guess that the >>> difference is ignorable). >>> >>> Please share with us the feedback about this proposal! >>> >>> Thanks, >>> Ivan >>> >>> [1] http://www.oracle.com/technetwork/articles/java/index-137868.html >>> >>> On Wed, Dec 31, 2014 at 3:32 PM, Claes Redestad < >>> claes.redestad at oracle.com> >>> wrote: >>> >>> Hi, >>>> seems like a reasonable convenience method. :-) >>>> >>>> Process stuff: OCA/JSPA signing aside, the patch needs to be submitted in >>>> its >>>> entirety to some part of the OpenJDK infrastructure before it can be >>>> accepted. >>>> Attaching the patch in an e-mail to this mailing list is acceptable for a >>>> small >>>> patch like this when lacking upload privileges to cr.openjdk.java.net. A >>>> sponsor >>>> could help you with the uploading etc. >>>> >>>> This request adds a new public method, which means some internal approval >>>> will be necessary. This is something you'll need help with from an Oracle >>>> sponsor. I could volunteer to initiate the requests internally. >>>> >>>> Code feedback: >>>> >>>> The test relies on the message generated when provoking a division by >>>> zero. I'm not sure this is a good idea nor the place to enforce this >>>> message format with a test. Perhaps just throwing a RuntimeException >>>> with some specified message and test for that? >>>> >>>> May I suggest to also add a test to ensure the method behaves well for an >>>> exception with fillInStackTrace overridden to produce stackless >>>> exceptions? >>>> >>>> Just a thought... I noticed PrintWriter, PrintStream and StringBuilder >>>> all >>>> inherit from Appendable, so the PrintStreamOrWriter/StackTracePrinter >>>> classes >>>> could be replaced with a single StackTraceAppendable taking an >>>> Appendable. >>>> One >>>> static class instead of 1 abstract and 3 concrete static inner classes >>>> could work >>>> out to be a good deal, but there's some odd mechanics in >>>> BufferedWriter/PrintWriter >>>> to use the value of the line.separator property at object creation time >>>> which >>>> would be hard to support without exposing the lineSeparator fields to >>>> Throwable. >>>> Perhaps a future consideration. >>>> >>>> Performance characteristics might change ever so slightly either way, so >>>> it >>>> would be nice with some microbenchmarks to quantify. >>>> >>>> /Claes >>>> >>>> >>>> On 2014-12-31 11:51, Ivan St. Ivanov wrote: >>>> >>>> Hello, >>>>> I am Ivan from the Bulgarian Java User Group. As part of our Adopt >>>>> OpenJDK >>>>> efforts, we organized hackathon beginning of this month in Sofia. There >>>>> we >>>>> worked on the following JIRA issue: >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-5050783 >>>>> >>>>> Unfortunately none of us has author privileges, so the issue is still >>>>> unassigned. But, nevertheless, our JUG has already signed JSPA and some >>>>> of >>>>> our members (including myself) have signed OCA. And we informed in a >>>>> special email this mailing list about our plans prior to the meeting. >>>>> >>>>> Our hackathon was already documented by our JUG member Mihail Stoynov: >>>>> >>>>> http://mihail.stoynov.com/2014/12/12/building-openjdk- >>>>> and-submitting-a-solution-on-a-feature-request-with-the- >>>>> bulgarian-java-user-group/ >>>>> >>>>> We used the days after the meeting for offline polishing our >>>>> contribution >>>>> and adding some more unit tests. Here is the result of our work: >>>>> >>>>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.00/ >>>>> >>>>> We would be very happy if you are able to review it, share your feedback >>>>> and guide us through the contribution process as this is our first >>>>> attempt. >>>>> >>>>> Thanks and have a very successful year! >>>>> Ivan >>>>> >>>>> From david.holmes at oracle.com Thu Jan 15 12:34:21 2015 From: david.holmes at oracle.com (David Holmes) Date: Thu, 15 Jan 2015 22:34:21 +1000 Subject: RFR: 5050783: Throwable convenience method: String getStackTraceString() In-Reply-To: References: <54A3FAEB.7010709@oracle.com> <54B6C728.2030203@univ-mlv.fr> Message-ID: <54B7B3CD.4000009@oracle.com> On 15/01/2015 6:34 AM, Ivan St. Ivanov wrote: > Hi Remi, > > I tried it and, well, the JDK image now does not even compile classes that > contain lambda expressions: > > javac -cp . ./LambdaMetaFactoryTester.java > Exception in thread "main" > Exception: java.lang.BootstrapMethodError thrown from the > UncaughtExceptionHandler in thread "main" > > With the old Throwable implementation it printed a long stack trace of > exceptions, but at least my sample class was compiled. > > So, I have two questions: > 1) Does anyone dump stack traces in such location, even for debugging > purposes? > 2) If yes, what are the other JDK classes and methods where we should avoid > using lambdas? It is hard to be definitive, but the initialization sequence for the JDK classes is extremely fragile. So anything that gets initialized very early in the VM initialization sequence needs to avoid using features that would require that the initialization sequence change. Using -verbose:class shows the order in which classes get loaded. David > Thanks, > Ivan > > On Wed, Jan 14, 2015 at 9:44 PM, Remi Forax wrote: > >> While using a lambda is cool in that context, >> if someone tries to do a printStackTrace in the LambdaMetafactory (for >> debugging purpose by example), >> it will be fun to debug. >> >> cheers, >> R?mi >> >> >> On 01/14/2015 08:25 PM, Ivan St. Ivanov wrote: >> >>> Hi Claes, core libs developers, >>> >>> As promised, we did yesterday our hackathon in the Bulgarian Java User >>> Group and incorporated the feedback that we got from you. Here is our next >>> webrev: >>> >>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.01/ >>> >>> As you can see: >>> >>> - We got rid of the inner classes altogether in favor of >>> java.util.function.Consumer >>> - The javadoc was updated to follow the standards [1] >>> - In the tests we use an exception that we create rather than relying on >>> one from the JDK that potentially could change >>> >>> We also ran some microbenchmarks to compare the performance of the new >>> functionality with the Apache commons solution mentioned in the bug. Here >>> is the benchmark test: >>> >>> @State(Scope.Benchmark) >>> public class ThrowableStacktraceBenchmark { >>> Throwable chainedThrowable; >>> Throwable simpleThrowable; >>> >>> @Setup >>> public void prepare() { >>> chainedThrowable = createThrowable(64); >>> simpleThrowable = new Throwable("Simple Throwable"); >>> } >>> >>> Throwable createThrowable(int depthLevel) { >>> Throwable t = null; >>> for (int i = 0; i < depthLevel; ++i) { >>> t = new Throwable("My dummy Throwable " + i, t); >>> } >>> return t; >>> } >>> >>> @Benchmark >>> public void benchmarkThrowableGetStackTraceString() { >>> chainedThrowable.getStackTraceString(); >>> } >>> >>> @Benchmark >>> public void benchmarkThrowableGetStackTraceStringSmall() { >>> simpleThrowable.getStackTraceString(); >>> } >>> >>> @Benchmark >>> public void benchmarkThrowablePrintStackTraceToStringWriter() { >>> StringWriter sw = new StringWriter(); >>> chainedThrowable.printStackTrace(new PrintWriter(sw)); >>> sw.toString(); >>> } >>> >>> @Benchmark >>> public void benchmarkThrowablePrintStackTraceToStringWriterSmall() { >>> StringWriter sw = new StringWriter(); >>> simpleThrowable.printStackTrace(new PrintWriter(sw)); >>> sw.toString(); >>> } >>> } >>> >>> And here are the results (test was run with the following options -wi 5 -i >>> 5 -t 1): >>> >>> Benchmark Mode Samples Score Error Units >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTraceString >>> thrpt 50 25351.743 ? 347.326 ops/s >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTrac >>> eStringSmall >>> thrpt 50 134990.643 ? 760.225 ops/s >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>> aceToStringWriter >>> thrpt 50 22730.786 ? 167.378 ops/s >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>> aceToStringWriterSmall >>> thrpt 50 115223.025 ? 936.770 ops/s >>> >>> Just in any case we ran this test, actually just the last two methods, >>> with >>> JDK 1.9 ea build 45. We wanted to compare the result of the Apache commons >>> solution with inner classes (as it is now) to the same, but with >>> java.util.function.Consumer. There are the results with the inner classes: >>> >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>> aceToStringWriter >>> thrpt 50 22974.324 ? 391.564 ops/s >>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>> aceToStringWriterSmall >>> thrpt 50 114973.452 ? 1051.276 ops/s >>> >>> So you can see that the performance is nearly the same (I guess that the >>> difference is ignorable). >>> >>> Please share with us the feedback about this proposal! >>> >>> Thanks, >>> Ivan >>> >>> [1] http://www.oracle.com/technetwork/articles/java/index-137868.html >>> >>> On Wed, Dec 31, 2014 at 3:32 PM, Claes Redestad < >>> claes.redestad at oracle.com> >>> wrote: >>> >>> Hi, >>>> >>>> seems like a reasonable convenience method. :-) >>>> >>>> Process stuff: OCA/JSPA signing aside, the patch needs to be submitted in >>>> its >>>> entirety to some part of the OpenJDK infrastructure before it can be >>>> accepted. >>>> Attaching the patch in an e-mail to this mailing list is acceptable for a >>>> small >>>> patch like this when lacking upload privileges to cr.openjdk.java.net. A >>>> sponsor >>>> could help you with the uploading etc. >>>> >>>> This request adds a new public method, which means some internal approval >>>> will be necessary. This is something you'll need help with from an Oracle >>>> sponsor. I could volunteer to initiate the requests internally. >>>> >>>> Code feedback: >>>> >>>> The test relies on the message generated when provoking a division by >>>> zero. I'm not sure this is a good idea nor the place to enforce this >>>> message format with a test. Perhaps just throwing a RuntimeException >>>> with some specified message and test for that? >>>> >>>> May I suggest to also add a test to ensure the method behaves well for an >>>> exception with fillInStackTrace overridden to produce stackless >>>> exceptions? >>>> >>>> Just a thought... I noticed PrintWriter, PrintStream and StringBuilder >>>> all >>>> inherit from Appendable, so the PrintStreamOrWriter/StackTracePrinter >>>> classes >>>> could be replaced with a single StackTraceAppendable taking an >>>> Appendable. >>>> One >>>> static class instead of 1 abstract and 3 concrete static inner classes >>>> could work >>>> out to be a good deal, but there's some odd mechanics in >>>> BufferedWriter/PrintWriter >>>> to use the value of the line.separator property at object creation time >>>> which >>>> would be hard to support without exposing the lineSeparator fields to >>>> Throwable. >>>> Perhaps a future consideration. >>>> >>>> Performance characteristics might change ever so slightly either way, so >>>> it >>>> would be nice with some microbenchmarks to quantify. >>>> >>>> /Claes >>>> >>>> >>>> On 2014-12-31 11:51, Ivan St. Ivanov wrote: >>>> >>>> Hello, >>>>> >>>>> I am Ivan from the Bulgarian Java User Group. As part of our Adopt >>>>> OpenJDK >>>>> efforts, we organized hackathon beginning of this month in Sofia. There >>>>> we >>>>> worked on the following JIRA issue: >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-5050783 >>>>> >>>>> Unfortunately none of us has author privileges, so the issue is still >>>>> unassigned. But, nevertheless, our JUG has already signed JSPA and some >>>>> of >>>>> our members (including myself) have signed OCA. And we informed in a >>>>> special email this mailing list about our plans prior to the meeting. >>>>> >>>>> Our hackathon was already documented by our JUG member Mihail Stoynov: >>>>> >>>>> http://mihail.stoynov.com/2014/12/12/building-openjdk- >>>>> and-submitting-a-solution-on-a-feature-request-with-the- >>>>> bulgarian-java-user-group/ >>>>> >>>>> We used the days after the meeting for offline polishing our >>>>> contribution >>>>> and adding some more unit tests. Here is the result of our work: >>>>> >>>>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.00/ >>>>> >>>>> We would be very happy if you are able to review it, share your feedback >>>>> and guide us through the contribution process as this is our first >>>>> attempt. >>>>> >>>>> Thanks and have a very successful year! >>>>> Ivan >>>>> >>>>> >>>> >> From claes.redestad at oracle.com Thu Jan 15 12:32:16 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Thu, 15 Jan 2015 13:32:16 +0100 Subject: RFR: 8068736: Avoid synchronization on Executable/Field.declaredAnnotations Message-ID: <54B7B350.1080409@oracle.com> Hi, please review this patch to use double-checked locking to avoid synchronization overhead on repeated access to the declaredAnnotations field in Executable and Field. webrev: http://cr.openjdk.java.net/~redestad/8068736/webrev.00 bug: https://bugs.openjdk.java.net/browse/JDK-8068736 Testing: jtreg java/lang Benchmarking: - no significant changes in a small set of SPEC benchmarks - up to 4x in micros focused on repeated access to the declaredAnnotations field, both in single- and multithreaded setups - less than 5% cost (not statistically significant) on micros accessing the declaredAnnotation map of a method only once. /Claes From peter.firmstone at zeus.net.au Thu Jan 15 13:01:03 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Thu, 15 Jan 2015 23:01:03 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54B3EEC5.5040805@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> Message-ID: <54B7BA0F.1020807@zeus.net.au> On 13/01/2015 1:56 AM, Chris Hegarty wrote: > On 10/01/15 07:00, Peter Firmstone wrote: > >> As shown in my earlier example, intra class invariants can be enforced >> using Serialization constructors, from which static methods are called >> to check invariants prior to super class constructors being called. > > Yes, but it is cumbersome and easy to make mistakes. I think if the user has unit tests for the invariants, then the liklihood of mistakes is very low. Cumbersome, I don't think we need to worry about that. I've had to do so much work, implementing work arounds, it would be like all my prayers answered just to be able to prevent reference stealing attacks and enforce invariants with failure atomicity. Did you know that .NET uses de-serialization constructors? A third option that hasn't yet been considered would be to investigate an api that various already existing frameworks can implement providers for, so they no longer need to find creative ways to construct instances of java platform classes when unmarshalling. > >> Presently, I override ObjectInputStream and use a Permission called >> DeserializationPermission to limit classes that can be deserialized. >> Untrusted connections are run from unprivileged context to limit classes >> that can be instantiated, while those with trusted connections are run >> with a Subject that allows any class to be deserialized. > > Interesting. I think there is overlap here with confinement. The good thing about using a permission check is it allows administrative based changed when a new vulnerability is discovered. Regards, Peter. From ivan.st.ivanov at gmail.com Thu Jan 15 13:05:15 2015 From: ivan.st.ivanov at gmail.com (Ivan St. Ivanov) Date: Thu, 15 Jan 2015 15:05:15 +0200 Subject: RFR: 5050783: Throwable convenience method: String getStackTraceString() In-Reply-To: <54B7B3CD.4000009@oracle.com> References: <54A3FAEB.7010709@oracle.com> <54B6C728.2030203@univ-mlv.fr> <54B7B3CD.4000009@oracle.com> Message-ID: Thanks everybody for the feedback! :) Claes, we were deliberately trying to avoid the "Apache commons" solution. It looks perfectly fine outside of the JDK. But our wild guess was that as we have access to the Throwable private methods, we can do something more elegant. The fact is (and I shared it with you in our private conversation) that we now understood that most elegant is not always the best when it comes to the core of the core of Java :) On that note, I'm thinking loudly about whether or not it'd be appropriate to extract common methods in PrintStream and PrintWriter to an interface. java.io.Printable? This would allow getting rid of most inner classes in j.l.Throwable along with some duplicated code across the JDK (I've noticed at least a few places where there are identical methods which differ only in that one takes a PrintStream, the other a PrintWriter). Do you think that we can experiment with that as part of the effort for this issue, or you think it is better to have it as a separate work item (possibly done by someone with more experience) and apply it here once it is ready? Regards, Ivan On Thu, Jan 15, 2015 at 2:34 PM, David Holmes wrote: > On 15/01/2015 6:34 AM, Ivan St. Ivanov wrote: > >> Hi Remi, >> >> I tried it and, well, the JDK image now does not even compile classes that >> contain lambda expressions: >> >> javac -cp . ./LambdaMetaFactoryTester.java >> Exception in thread "main" >> Exception: java.lang.BootstrapMethodError thrown from the >> UncaughtExceptionHandler in thread "main" >> >> With the old Throwable implementation it printed a long stack trace of >> exceptions, but at least my sample class was compiled. >> >> So, I have two questions: >> 1) Does anyone dump stack traces in such location, even for debugging >> purposes? >> 2) If yes, what are the other JDK classes and methods where we should >> avoid >> using lambdas? >> > > It is hard to be definitive, but the initialization sequence for the JDK > classes is extremely fragile. So anything that gets initialized very early > in the VM initialization sequence needs to avoid using features that would > require that the initialization sequence change. Using -verbose:class shows > the order in which classes get loaded. > > David > > > Thanks, >> Ivan >> >> On Wed, Jan 14, 2015 at 9:44 PM, Remi Forax wrote: >> >> While using a lambda is cool in that context, >>> if someone tries to do a printStackTrace in the LambdaMetafactory (for >>> debugging purpose by example), >>> it will be fun to debug. >>> >>> cheers, >>> R?mi >>> >>> >>> On 01/14/2015 08:25 PM, Ivan St. Ivanov wrote: >>> >>> Hi Claes, core libs developers, >>>> >>>> As promised, we did yesterday our hackathon in the Bulgarian Java User >>>> Group and incorporated the feedback that we got from you. Here is our >>>> next >>>> webrev: >>>> >>>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.01/ >>>> >>>> As you can see: >>>> >>>> - We got rid of the inner classes altogether in favor of >>>> java.util.function.Consumer >>>> - The javadoc was updated to follow the standards [1] >>>> - In the tests we use an exception that we create rather than relying on >>>> one from the JDK that potentially could change >>>> >>>> We also ran some microbenchmarks to compare the performance of the new >>>> functionality with the Apache commons solution mentioned in the bug. >>>> Here >>>> is the benchmark test: >>>> >>>> @State(Scope.Benchmark) >>>> public class ThrowableStacktraceBenchmark { >>>> Throwable chainedThrowable; >>>> Throwable simpleThrowable; >>>> >>>> @Setup >>>> public void prepare() { >>>> chainedThrowable = createThrowable(64); >>>> simpleThrowable = new Throwable("Simple Throwable"); >>>> } >>>> >>>> Throwable createThrowable(int depthLevel) { >>>> Throwable t = null; >>>> for (int i = 0; i < depthLevel; ++i) { >>>> t = new Throwable("My dummy Throwable " + i, t); >>>> } >>>> return t; >>>> } >>>> >>>> @Benchmark >>>> public void benchmarkThrowableGetStackTraceString() { >>>> chainedThrowable.getStackTraceString(); >>>> } >>>> >>>> @Benchmark >>>> public void benchmarkThrowableGetStackTraceStringSmall() { >>>> simpleThrowable.getStackTraceString(); >>>> } >>>> >>>> @Benchmark >>>> public void benchmarkThrowablePrintStackTraceToStringWriter() { >>>> StringWriter sw = new StringWriter(); >>>> chainedThrowable.printStackTrace(new PrintWriter(sw)); >>>> sw.toString(); >>>> } >>>> >>>> @Benchmark >>>> public void benchmarkThrowablePrintStackTraceToStringWriterSmall() >>>> { >>>> StringWriter sw = new StringWriter(); >>>> simpleThrowable.printStackTrace(new PrintWriter(sw)); >>>> sw.toString(); >>>> } >>>> } >>>> >>>> And here are the results (test was run with the following options -wi 5 >>>> -i >>>> 5 -t 1): >>>> >>>> Benchmark Mode Samples Score Error Units >>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTrac >>>> eString >>>> thrpt 50 25351.743 ? 347.326 ops/s >>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTrac >>>> eStringSmall >>>> thrpt 50 134990.643 ? 760.225 ops/s >>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>> aceToStringWriter >>>> thrpt 50 22730.786 ? 167.378 ops/s >>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>> aceToStringWriterSmall >>>> thrpt 50 115223.025 ? 936.770 ops/s >>>> >>>> Just in any case we ran this test, actually just the last two methods, >>>> with >>>> JDK 1.9 ea build 45. We wanted to compare the result of the Apache >>>> commons >>>> solution with inner classes (as it is now) to the same, but with >>>> java.util.function.Consumer. There are the results with the inner >>>> classes: >>>> >>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>> aceToStringWriter >>>> thrpt 50 22974.324 ? 391.564 ops/s >>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>> aceToStringWriterSmall >>>> thrpt 50 114973.452 ? 1051.276 ops/s >>>> >>>> So you can see that the performance is nearly the same (I guess that the >>>> difference is ignorable). >>>> >>>> Please share with us the feedback about this proposal! >>>> >>>> Thanks, >>>> Ivan >>>> >>>> [1] http://www.oracle.com/technetwork/articles/java/index-137868.html >>>> >>>> On Wed, Dec 31, 2014 at 3:32 PM, Claes Redestad < >>>> claes.redestad at oracle.com> >>>> wrote: >>>> >>>> Hi, >>>> >>>>> >>>>> seems like a reasonable convenience method. :-) >>>>> >>>>> Process stuff: OCA/JSPA signing aside, the patch needs to be submitted >>>>> in >>>>> its >>>>> entirety to some part of the OpenJDK infrastructure before it can be >>>>> accepted. >>>>> Attaching the patch in an e-mail to this mailing list is acceptable >>>>> for a >>>>> small >>>>> patch like this when lacking upload privileges to cr.openjdk.java.net. >>>>> A >>>>> sponsor >>>>> could help you with the uploading etc. >>>>> >>>>> This request adds a new public method, which means some internal >>>>> approval >>>>> will be necessary. This is something you'll need help with from an >>>>> Oracle >>>>> sponsor. I could volunteer to initiate the requests internally. >>>>> >>>>> Code feedback: >>>>> >>>>> The test relies on the message generated when provoking a division by >>>>> zero. I'm not sure this is a good idea nor the place to enforce this >>>>> message format with a test. Perhaps just throwing a RuntimeException >>>>> with some specified message and test for that? >>>>> >>>>> May I suggest to also add a test to ensure the method behaves well for >>>>> an >>>>> exception with fillInStackTrace overridden to produce stackless >>>>> exceptions? >>>>> >>>>> Just a thought... I noticed PrintWriter, PrintStream and StringBuilder >>>>> all >>>>> inherit from Appendable, so the PrintStreamOrWriter/StackTracePrinter >>>>> classes >>>>> could be replaced with a single StackTraceAppendable taking an >>>>> Appendable. >>>>> One >>>>> static class instead of 1 abstract and 3 concrete static inner classes >>>>> could work >>>>> out to be a good deal, but there's some odd mechanics in >>>>> BufferedWriter/PrintWriter >>>>> to use the value of the line.separator property at object creation time >>>>> which >>>>> would be hard to support without exposing the lineSeparator fields to >>>>> Throwable. >>>>> Perhaps a future consideration. >>>>> >>>>> Performance characteristics might change ever so slightly either way, >>>>> so >>>>> it >>>>> would be nice with some microbenchmarks to quantify. >>>>> >>>>> /Claes >>>>> >>>>> >>>>> On 2014-12-31 11:51, Ivan St. Ivanov wrote: >>>>> >>>>> Hello, >>>>> >>>>>> >>>>>> I am Ivan from the Bulgarian Java User Group. As part of our Adopt >>>>>> OpenJDK >>>>>> efforts, we organized hackathon beginning of this month in Sofia. >>>>>> There >>>>>> we >>>>>> worked on the following JIRA issue: >>>>>> >>>>>> https://bugs.openjdk.java.net/browse/JDK-5050783 >>>>>> >>>>>> Unfortunately none of us has author privileges, so the issue is still >>>>>> unassigned. But, nevertheless, our JUG has already signed JSPA and >>>>>> some >>>>>> of >>>>>> our members (including myself) have signed OCA. And we informed in a >>>>>> special email this mailing list about our plans prior to the meeting. >>>>>> >>>>>> Our hackathon was already documented by our JUG member Mihail Stoynov: >>>>>> >>>>>> http://mihail.stoynov.com/2014/12/12/building-openjdk- >>>>>> and-submitting-a-solution-on-a-feature-request-with-the- >>>>>> bulgarian-java-user-group/ >>>>>> >>>>>> We used the days after the meeting for offline polishing our >>>>>> contribution >>>>>> and adding some more unit tests. Here is the result of our work: >>>>>> >>>>>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.00/ >>>>>> >>>>>> We would be very happy if you are able to review it, share your >>>>>> feedback >>>>>> and guide us through the contribution process as this is our first >>>>>> attempt. >>>>>> >>>>>> Thanks and have a very successful year! >>>>>> Ivan >>>>>> >>>>>> >>>>>> >>>>> >>> From paul.sandoz at oracle.com Thu Jan 15 13:12:50 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 15 Jan 2015 14:12:50 +0100 Subject: JDK 9 RFR of 8060077: Class.toGenericString specification doesn't mention array types In-Reply-To: <54B711D4.7070902@oracle.com> References: <54B711D4.7070902@oracle.com> Message-ID: <8B1E6309-CBE3-4C8D-BD4F-D2C8B82E1EC3@oracle.com> On Jan 15, 2015, at 2:03 AM, joe darcy wrote: > Hello, > > Please review these changes to address > > JDK-8060077 : Class.toGenericString specification doesn't mention array types > http://cr.openjdk.java.net/~darcy/8060077.0/ > +1. Paul. From paul.sandoz at oracle.com Thu Jan 15 13:19:09 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 15 Jan 2015 14:19:09 +0100 Subject: RFR: 8068736: Avoid synchronization on Executable/Field.declaredAnnotations In-Reply-To: <54B7B350.1080409@oracle.com> References: <54B7B350.1080409@oracle.com> Message-ID: <831EA9E3-7F42-4680-B316-6B0B7223B6BF@oracle.com> On Jan 15, 2015, at 1:32 PM, Claes Redestad wrote: > Hi, > > please review this patch to use double-checked locking to avoid > synchronization overhead on repeated access to the > declaredAnnotations field in Executable and Field. > > webrev: http://cr.openjdk.java.net/~redestad/8068736/webrev.00 > bug: https://bugs.openjdk.java.net/browse/JDK-8068736 > +1 Paul. > Testing: jtreg java/lang > > Benchmarking: > - no significant changes in a small set of SPEC benchmarks > - up to 4x in micros focused on repeated access to the > declaredAnnotations field, both in single- and multithreaded setups > - less than 5% cost (not statistically significant) on micros accessing > the declaredAnnotation map of a method only once. > > /Claes From joel.franck at oracle.com Thu Jan 15 13:46:13 2015 From: joel.franck at oracle.com (=?utf-8?Q?Joel_Borggr=C3=A9n-Franck?=) Date: Thu, 15 Jan 2015 14:46:13 +0100 Subject: RFR: 8068736: Avoid synchronization on Executable/Field.declaredAnnotations In-Reply-To: <54B7B350.1080409@oracle.com> References: <54B7B350.1080409@oracle.com> Message-ID: <76493188-D3BE-48BF-8EC2-C7CC5FE5C313@oracle.com> Looks good. cheers /Joel > On 15 jan 2015, at 13:32, Claes Redestad wrote: > > Hi, > > please review this patch to use double-checked locking to avoid > synchronization overhead on repeated access to the > declaredAnnotations field in Executable and Field. > > webrev: http://cr.openjdk.java.net/~redestad/8068736/webrev.00 > bug: https://bugs.openjdk.java.net/browse/JDK-8068736 > > Testing: jtreg java/lang > > Benchmarking: > - no significant changes in a small set of SPEC benchmarks > - up to 4x in micros focused on repeated access to the > declaredAnnotations field, both in single- and multithreaded setups > - less than 5% cost (not statistically significant) on micros accessing > the declaredAnnotation map of a method only once. > > /Claes From joel.franck at oracle.com Thu Jan 15 13:53:10 2015 From: joel.franck at oracle.com (=?utf-8?Q?Joel_Borggr=C3=A9n-Franck?=) Date: Thu, 15 Jan 2015 14:53:10 +0100 Subject: JDK 9 RFR of 8060077: Class.toGenericString specification doesn't mention array types In-Reply-To: <54B711D4.7070902@oracle.com> References: <54B711D4.7070902@oracle.com> Message-ID: <7B8AB251-37EF-42D5-BC30-CE2A955D5207@oracle.com> Hi Joe, Nits, 152 * "void". If this {@code Class} object presents an array type, 153 * this method returns "class " followed by {@code getName}. line 152, presents or represents? Personally I would add a test for a 2-dim array of a generic type. cheers /Joel > On 15 jan 2015, at 02:03, joe darcy wrote: > > Hello, > > Please review these changes to address > > JDK-8060077 : Class.toGenericString specification doesn't mention array types > http://cr.openjdk.java.net/~darcy/8060077.0/ > > Thanks, > > -Joe > From scolebourne at joda.org Thu Jan 15 14:27:44 2015 From: scolebourne at joda.org (Stephen Colebourne) Date: Thu, 15 Jan 2015 14:27:44 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54B7BA0F.1020807@zeus.net.au> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <54B7BA0F.1020807@zeus.net.au> Message-ID: On 15 January 2015 at 13:01, Peter Firmstone wrote: > A third option that hasn't yet been considered would be to investigate an > api that various already existing frameworks can implement providers for, so > they no longer need to find creative ways to construct instances of java > platform classes when unmarshalling. Which is part of what this project is trying to build (for the JDK) https://github.com/jodastephen/property-alliance Stephen From chris.hegarty at oracle.com Thu Jan 15 16:37:17 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 15 Jan 2015 16:37:17 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54B7B1C0.4010802@zeus.net.au> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54B67A19.1050507@oracle.com> <54B7B1C0.4010802@zeus.net.au> Message-ID: <5D7FF52D-8E47-45ED-A738-911A2888A6DB@oracle.com> On 15 Jan 2015, at 12:25, Peter Firmstone wrote: > Chris, > > Can you explain some of the detail in your failure atomicity work? Certainly. The current implementation of defaultReadObject sets non-primitive field values one at a time, without first checking that all their types are assignable. If, for example, the assignment of the last non-primitive value fails, a CCE is thrown, and the previously set fields remain set. The setting of field values can be deferred until after they have been "validated", at a minimum that the non-primitive types are assignable. This is achievable per Class in the hierarchy, and provides some additional level of confidence. A change along the lines of this should not be controversial ( either all fields will be set, or contain their default values ), but it is only per Class in the hierarchy. Pushing on this a little, and ignoring cyclic references for now. For the first Serializable type in the hierarchy ( closest to j.l.Object ), there is no need to eagerly create the instance before ?validating? the field values. If the field values are assignable, then the object can created and the values assigned. If a field value is found to be unassignable, then CCE is thrown, and there is no dangling object. Pushing a little more. If there are no readObjectXXX methods in the hierarchy, then it would seem possible to read the field values for all the types in the hierarchy, and ?validate? them ( ensure they are assignable ), before creating the object and assigning the values. I do not believe that there are any observable affects from doing this. Cyclic references: Assumption, we do not want to leak our secure deserialized objects. It seems reasonable to not share the reference in the stream. This would rule out cyclic references. If you want sharing, then the implementation would create the referenced object, and assign the already read field values, when it encounters a cyclic reference. But you open the object to anything in the stream referencing it. readObjectXX: Assumption, many readObject methods exist to check invariants of values read from the stream. If we had another mechanism, similar to the static validate(GetFields) method we discussed previously, then may readObject methods could be eliminated. In which case we can achieve whole hierarchy failure atomicity ( as described above ). Given this, it seems like the natural place for a static invariant checker is between the reading and reconstitution of field values, and the assigning of them. If the invariant checker could be called by the serialization mechanism then the user code can use final fields, without needing to muck around with reflection, and also have the added benefit of, in many cases, not potential leaving the object in an inconsistent state. There is no support in this proposal for subclass to superclass invariant checking. I also believe that we can resolve the Finalization issue separately. -Chris. From dl at cs.oswego.edu Thu Jan 15 19:09:36 2015 From: dl at cs.oswego.edu (Doug Lea) Date: Thu, 15 Jan 2015 14:09:36 -0500 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B69056.7000307@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> <54B306AB.8010509@gmail.com> <54B446BD.7050306@gmail.com> <54B50345.4020808@cs.oswego.edu> <54B69056.7000307@gmail.com> Message-ID: <54B81070.9070102@cs.oswego.edu> On 01/14/2015 10:50 AM, Peter Levart wrote: > Hi, > > Loosely related to this topic, there is some local variable caching of > comparableClassFor() result already being performed inside iterative/recursive > methods of TreeNode, but this caching is just positive caching, meaning that > null return is not cached. For keys that happen to implement Comparable, but are > not simple-self-comparable (their class is not of the form: class C implements > Comparable), reflection is invoked each time comparison of the search key > with Node key is attempted. This can be improved: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/InCallNegativeCache/webrev.01/ > > Thanks! This seems like a definite improvement, that we ought to put in place. -Doug From peter.firmstone at zeus.net.au Thu Jan 15 20:33:41 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Fri, 16 Jan 2015 06:33:41 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <5D7FF52D-8E47-45ED-A738-911A2888A6DB@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54B67A19.1050507@oracle.com> <54B7B1C0.4010802@zeus.net.au> <5D7FF52D-8E47-45ED-A738-911A2888A6DB@oracle.com> Message-ID: <1421354021.1609.5.camel@Nokia-N900> Thanks Chris, WRT circular references, is it possible to detect and delay setting these until after all verifiers run? An option might be to provide a protected method in ObjectInputStream to opt out or switch off support for circular references. Apart from Serialization, another common use for readObject is for class evolution. Our projects uses Serialization more than most, it was originally written by the same people who earlier implemented Java's Serialization framework. It's also the root cause of our security problems, I'm presently working on a complete reimplimentation of ObjectInputStream using Apache Harmony's code, as you might imagine I'm running into a lot of trouble instantiating objects in a secure and cross platform compatible fashion. Can I suggest we have the static verifier return GetFields, to leave the door open to supporting a constructor in future? Regards, Peter. ----- Original message ----- > On 15 Jan 2015, at 12:25, Peter Firmstone > wrote: > > > Chris, > > > > Can you explain some of the detail in your failure atomicity work? > > Certainly. > > The current implementation of defaultReadObject sets non-primitive field > values one at a time, without first checking that all their types are > assignable. If, for example, the assignment of the last non-primitive > value fails, a CCE is thrown, and the previously set fields remain set. > The setting of field values can be deferred until after they have been > "validated", at a minimum that the non-primitive types are assignable. > This is achievable per Class in the hierarchy, and provides some > additional level of confidence. A change along the lines of this should > not be controversial ( either all fields will be set, or contain their > default values ), but it is only per Class in the hierarchy. > > Pushing on this a little, and ignoring cyclic references for now.? For > the first Serializable type in the hierarchy ( closest to j.l.Object ), > there is no need to eagerly create the instance before ?validating? the > field values. If the field values are assignable, then the object can > created and the values assigned. If a field value is found to be > unassignable, then CCE is thrown, and there is no dangling object. > > Pushing a little more. If there are no readObjectXXX methods in the > hierarchy, then it would seem possible to read the field values for all > the types in the hierarchy, and ?validate? them ( ensure they are > assignable ), before creating the object and assigning the values. I do > not believe that there are any observable affects from doing this. > > > Cyclic references: Assumption, we do not want to leak our secure > deserialized objects. It seems reasonable to not share the reference in > the stream. This would rule out cyclic references.? If you want sharing, > then the implementation would create the referenced object, and assign > the already read field values, when it encounters a cyclic reference. > But you open the object to anything in the stream referencing it. > > readObjectXX:? Assumption, many readObject methods exist to check > invariants of values read from the stream. If we had another mechanism, > similar to the static validate(GetFields) method we discussed > previously, then may readObject methods could be eliminated. In which > case we can achieve whole hierarchy failure atomicity ( as described > above ). > > Given this, it seems like the natural place for a static invariant > checker is between the reading and reconstitution of field values, and > the assigning of them. If the invariant checker could be called by the > serialization mechanism then the user code can use final fields, without > needing to muck around with reflection, and also have the added benefit > of, in many cases, not potential leaving the object in an inconsistent > state. > > There is no support in this proposal for subclass to superclass > invariant checking. > > I also believe that we can resolve the Finalization issue separately. > > -Chris. From chris.hegarty at oracle.com Fri Jan 16 10:49:59 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 16 Jan 2015 10:49:59 +0000 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <548ECDA4.9070002@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <548EC2D7.6000604@oracle.com> <548EC534.2090803@oracle.com> <548ECDA4.9070002@oracle.com> Message-ID: <54B8ECD7.3010703@oracle.com> On 15/12/14 12:01, Alan Bateman wrote: > On 15/12/2014 11:25, Chris Hegarty wrote: >> Konstantin, >> >> I did reply to this RFR [1], with a question, that is still unanswered. >> >> -Chris. >> >> [1] >> http://mail.openjdk.java.net/pipermail/net-dev/2014-December/008782.html >> > I can think of configurations where the scenario could arise. It make me > wonder if this will require a spec clarification as scope_id is defined > in Inet6Address to allow any String and so raises the question as to the > characters permitted in the host component. I think that we can add '_' and '.' . ':' I am less sure about. I don't see any reason to update the spec here, given that the set of allowable character is not clearly defined in the relevant RFC's. The change, as proposed in the webrev, is reasonable ( with ':' removed ), but you should also include an update to jdk/test/java/net/URI/Test.java, to cover this. -Chris. From Alan.Bateman at oracle.com Fri Jan 16 11:29:52 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 16 Jan 2015 11:29:52 +0000 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54B8ECD7.3010703@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <548EC2D7.6000604@oracle.com> <548EC534.2090803@oracle.com> <548ECDA4.9070002@oracle.com> <54B8ECD7.3010703@oracle.com> Message-ID: <54B8F630.4040608@oracle.com> On 16/01/2015 10:49, Chris Hegarty wrote: > : > > I don't see any reason to update the spec here, given that the set of > allowable character is not clearly defined in the relevant RFC's. > I think we need to create a bug to look into this more. Inet6Address allows the scope of be any String but URI puts restrictions on what is legal. The proposed change updates the list of allowed characters but I assume it will need to re-visited again as systems with more exotic interface names are encountered. -Alan From chris.hegarty at oracle.com Fri Jan 16 11:42:02 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 16 Jan 2015 11:42:02 +0000 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54B8F630.4040608@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <548EC2D7.6000604@oracle.com> <548EC534.2090803@oracle.com> <548ECDA4.9070002@oracle.com> <54B8ECD7.3010703@oracle.com> <54B8F630.4040608@oracle.com> Message-ID: <54B8F90A.7010402@oracle.com> On 16/01/15 11:29, Alan Bateman wrote: > On 16/01/2015 10:49, Chris Hegarty wrote: >> : >> >> I don't see any reason to update the spec here, given that the set of >> allowable character is not clearly defined in the relevant RFC's. >> > I think we need to create a bug to look into this more. Inet6Address > allows the scope of be any String but URI puts restrictions on what is > legal. The proposed change updates the list of allowed characters but I > assume it will need to re-visited again as systems with more exotic > interface names are encountered. Yes that is possible, but to date this is the first time I've seen this come up. Given the lack of clear specification in the relevant RFC's (relating to scope ids ), I don't think we should be making too many specification changes in this area. But I agree with your comment, URI has a restriction that it not clearly specified in the Java spec. -Chris. From staffan.larsen at oracle.com Fri Jan 16 11:46:47 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Fri, 16 Jan 2015 12:46:47 +0100 Subject: RFR 8068730: Increase the precision of the implementation of java.time.Clock.systemUTC() In-Reply-To: <54B50EFF.9070308@oracle.com> References: <54B0083C.1090409@oracle.com> <54B32F8A.5020006@oracle.com> <54B3B762.9060508@oracle.com> <54B49670.9080905@oracle.com> <54B4EB36.9010909@oracle.com> <54B50EFF.9070308@oracle.com> Message-ID: > On 13 jan 2015, at 13:26, David Holmes wrote: > > Hi Daniel, > > On 13/01/2015 7:53 PM, Daniel Fuchs wrote: >> On 13/01/15 04:52, David Holmes wrote: >>>> Here is the new webrev with Stephen & your feedback included >>>> http://cr.openjdk.java.net/~dfuchs/webrev_8068730/webrev.01/ >>>> >>>> (windows_time_stamp not renamed yet) >>> >>> I checked the updated webrev.02 and everything looks good on the hotspot >>> side. >>> >>> What are your plans for pushing this? >> >> Thanks David, >> >> My understanding is that I'll need to push that through hs-rt - since >> it has changes for hotspot. I believe I'll need two reviewers - is that >> right? Roger Riggs has also reviewed the changes on core-libs-dev, so >> is this good to go, or do both reviewers need to be members of the hsx >> project? > > hsx project is defunct, so technically we're all just jdk9 reviewers, but we still like two reviews from hotspot team members. I?ve looked at the hotspot changes: Looks good! Thanks, /Staffan > >> Also - I am not sure whether I need a sponsor (I have never pushed >> anything to hs-rt - I usually push in jdk9-dev) or whether I am >> allowed to push this myself (I'll use jprt in that case). > > If you are a jdk9 committer then you can push yourself via jprt - full forest submission from top-level: "jprt submit -forest -testset hotspot" > > David > >> best regards, >> >> -- daniel >> >>> >>> David >>> ----- From claes.redestad at oracle.com Fri Jan 16 12:18:20 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Fri, 16 Jan 2015 13:18:20 +0100 Subject: RFR: 8068736: Avoid synchronization on Executable/Field.declaredAnnotations In-Reply-To: <76493188-D3BE-48BF-8EC2-C7CC5FE5C313@oracle.com> References: <54B7B350.1080409@oracle.com> <76493188-D3BE-48BF-8EC2-C7CC5FE5C313@oracle.com> Message-ID: <54B9018C.1080609@oracle.com> Joel, Paul, thanks for reviewing! Pushed. /Claes On 01/15/2015 02:46 PM, Joel Borggr?n-Franck wrote: > Looks good. > > cheers > /Joel > >> On 15 jan 2015, at 13:32, Claes Redestad wrote: >> >> Hi, >> >> please review this patch to use double-checked locking to avoid >> synchronization overhead on repeated access to the >> declaredAnnotations field in Executable and Field. >> >> webrev: http://cr.openjdk.java.net/~redestad/8068736/webrev.00 >> bug: https://bugs.openjdk.java.net/browse/JDK-8068736 >> >> Testing: jtreg java/lang >> >> Benchmarking: >> - no significant changes in a small set of SPEC benchmarks >> - up to 4x in micros focused on repeated access to the >> declaredAnnotations field, both in single- and multithreaded setups >> - less than 5% cost (not statistically significant) on micros accessing >> the declaredAnnotation map of a method only once. >> >> /Claes From konstantin.shefov at oracle.com Fri Jan 16 12:50:02 2015 From: konstantin.shefov at oracle.com (Konstantin Shefov) Date: Fri, 16 Jan 2015 15:50:02 +0300 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54B8F90A.7010402@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <548EC2D7.6000604@oracle.com> <548EC534.2090803@oracle.com> <548ECDA4.9070002@oracle.com> <54B8ECD7.3010703@oracle.com> <54B8F630.4040608@oracle.com> <54B8F90A.7010402@oracle.com> Message-ID: <54B908FA.9020405@oracle.com> Hi Chris, Alan, thank you for reviewing. I have made a new webrev http://cr.openjdk.java.net/~kshefov/6933879/webrev.01 I have removed ":" and added a test case. -Konstantin 16.01.2015 14:42, Chris Hegarty ?????: > On 16/01/15 11:29, Alan Bateman wrote: >> On 16/01/2015 10:49, Chris Hegarty wrote: >>> : >>> >>> I don't see any reason to update the spec here, given that the set of >>> allowable character is not clearly defined in the relevant RFC's. >>> >> I think we need to create a bug to look into this more. Inet6Address >> allows the scope of be any String but URI puts restrictions on what is >> legal. The proposed change updates the list of allowed characters but I >> assume it will need to re-visited again as systems with more exotic >> interface names are encountered. > > Yes that is possible, but to date this is the first time I've seen > this come up. > > Given the lack of clear specification in the relevant RFC's (relating > to scope ids ), I don't think we should be making too many > specification changes in this area. But I agree with your comment, URI > has a restriction that it not clearly specified in the Java spec. > > -Chris. From chris.hegarty at oracle.com Fri Jan 16 14:02:21 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 16 Jan 2015 14:02:21 +0000 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54B908FA.9020405@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <548EC2D7.6000604@oracle.com> <548EC534.2090803@oracle.com> <548ECDA4.9070002@oracle.com> <54B8ECD7.3010703@oracle.com> <54B8F630.4040608@oracle.com> <54B8F90A.7010402@oracle.com> <54B908FA.9020405@oracle.com> Message-ID: <54B919ED.8080403@oracle.com> On 16/01/15 12:50, Konstantin Shefov wrote: > Hi Chris, Alan, thank you for reviewing. > > I have made a new webrev > http://cr.openjdk.java.net/~kshefov/6933879/webrev.01 > I have removed ":" and added a test case. This looks ok to me. -Chris. > -Konstantin > > 16.01.2015 14:42, Chris Hegarty ?????: >> On 16/01/15 11:29, Alan Bateman wrote: >>> On 16/01/2015 10:49, Chris Hegarty wrote: >>>> : >>>> >>>> I don't see any reason to update the spec here, given that the set of >>>> allowable character is not clearly defined in the relevant RFC's. >>>> >>> I think we need to create a bug to look into this more. Inet6Address >>> allows the scope of be any String but URI puts restrictions on what is >>> legal. The proposed change updates the list of allowed characters but I >>> assume it will need to re-visited again as systems with more exotic >>> interface names are encountered. >> >> Yes that is possible, but to date this is the first time I've seen >> this come up. >> >> Given the lack of clear specification in the relevant RFC's (relating >> to scope ids ), I don't think we should be making too many >> specification changes in this area. But I agree with your comment, URI >> has a restriction that it not clearly specified in the Java spec. >> >> -Chris. > From chris.hegarty at oracle.com Fri Jan 16 14:15:58 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 16 Jan 2015 14:15:58 +0000 Subject: HashMap collision speed (regression 7->8) In-Reply-To: <54B69056.7000307@gmail.com> References: <20150108173811.00003a76.ecki@zusammenkunft.net> <54AED8E5.5080109@gmail.com> <20150108204120.0000323b.ecki@zusammenkunft.net> <54AF0042.3080304@gmail.com> <20150109002413.00000678.ecki@zusammenkunft.net> <20150109011035.000000f5.ecki@zusammenkunft.net> <54B01E98.308@oracle.com> <20150109202230.00000e85.ecki@zusammenkunft.net> <54B03127.8080706@cs.oswego.edu> <54B07051.5050409@cs.oswego.edu> <54B1229E.5060904@gmail.com> <54B264C4.90506@gmail.com> <54B2920E.4000803@gmail.com> <54B2CE69.4050300@gmail.com> <54B2E471.3090905@cs.oswego.edu> <54B306AB.8010509@gmail.com> <54B446BD.7050306@gmail.com> <54B50345.4020808@cs.oswego.edu> <54B69056.7000307@gmail.com> Message-ID: <54B91D1E.5080508@oracle.com> On 14/01/15 15:50, Peter Levart wrote: > Hi, > > Loosely related to this topic, there is some local variable caching of > comparableClassFor() result already being performed inside > iterative/recursive methods of TreeNode, but this caching is just > positive caching, meaning that null return is not cached. For keys that > happen to implement Comparable, but are not simple-self-comparable > (their class is not of the form: class C implements Comparable), > reflection is invoked each time comparison of the search key with Node > key is attempted. This can be improved: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/InCallNegativeCache/webrev.01/ This seems like a good change. -Chris > Using modified version of Bernd's benchmark (adding 3rd variant of keys > - FalseComp): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/HM.comparableClassFor/InCallNegativeCache/HashMapCollision.java > > > ...and and only running for badHash shows almost 25% improvement for > FalseComp keys (skipping reflection invocations) and barely measurable > improvement for NoComp keys (skipping instanceof checks), while Comp > keys are not affected. > > Original JDK9 HashMap: > > Benchmark (initialSize) Mode Samples > Score Error Units > j.t.HashMapCollision.badDistFalseComp 16 ss 20 4221.438 > +- 250.774 ms > j.t.HashMapCollision.badDistNoComp 16 ss 20 2868.605 > +- 40.754 ms > j.t.HashMapCollision.badDistWithComp 16 ss 20 3030.780 > +- 111.315 ms > > Patched: > > Benchmark (initialSize) Mode Samples > Score Error Units > j.t.HashMapCollision.badDistFalseComp 16 ss 20 3237.953 > +- 143.608 ms > j.t.HashMapCollision.badDistNoComp 16 ss 20 2643.024 > +- 137.067 ms > j.t.HashMapCollision.badDistWithComp 16 ss 20 3087.902 > +- 122.041 ms > > > > Regards, Peter > From lev.priima at oracle.com Fri Jan 16 17:02:10 2015 From: lev.priima at oracle.com (Lev Priima) Date: Fri, 16 Jan 2015 20:02:10 +0300 Subject: RFR : 8068795: Missing trailing blank in a HttpServer response leads to client crashes Message-ID: <54B94412.8@oracle.com> The second space should not be omitted in first line(status-line) of a http response message: http://tools.ietf.org/html/rfc7230#section-3.1.2 . Issue: http://bugs.openjdk.java.net/browse/JDK-8068795 Patch: http://cr.openjdk.java.net/~lpriima/8068795/webrev.00/ Testing: $ jtreg -jdk:jdk9/dev/build/linux-x86_64-normal-server-release/jdk/ jdk9/dev/jdk/test/com/sun/net/httpserver/ Test results: passed: 41 -- Best Regards, Lev From vladimir.x.ivanov at oracle.com Fri Jan 16 17:16:22 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Fri, 16 Jan 2015 20:16:22 +0300 Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when GWT LambdaForms are shared Message-ID: <54B94766.2080102@oracle.com> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/hotspot/ http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/jdk/ https://bugs.openjdk.java.net/browse/JDK-8063137 After GuardWithTest (GWT) LambdaForms became shared, profile pollution significantly distorted compilation decisions. It affected inlining and hindered some optimizations. It causes significant performance regressions for Nashorn (on Octane benchmarks). Inlining was fixed by 8059877 [1], but it didn't cover the case when a branch is never taken. It can cause missed optimization opportunity, and not just increase in code size. For example, non-pruned branch can break escape analysis. Currently, there are 2 problems: - branch frequencies profile pollution - deoptimization counts pollution Branch frequency pollution hides from JIT the fact that a branch is never taken. Since GWT LambdaForms (and hence their bytecode) are heavily shared, but the behavior is specific to MethodHandle, there's no way for JIT to understand how particular GWT instance behaves. The solution I propose is to do profiling in Java code and feed it to JIT. Every GWT MethodHandle holds an auxiliary array (int[2]) where profiling info is stored. Once JIT kicks in, it can retrieve these counts, if corresponding MethodHandle is a compile-time constant (and it is usually the case). To communicate the profile data from Java code to JIT, MethodHandleImpl::profileBranch() is used. If GWT MethodHandle isn't a compile-time constant, profiling should proceed. It happens when corresponding LambdaForm is already shared, for newly created GWT MethodHandles profiling can occur only in native code (dedicated nmethod for a single LambdaForm). So, when compilation of the whole MethodHandle chain is triggered, the profile should be already gathered. Overriding branch frequencies is not enough. Statistics on deoptimization events is also polluted. Even if a branch is never taken, JIT doesn't issue an uncommon trap there unless corresponding bytecode doesn't trap too much and doesn't cause too many recompiles. I added @IgnoreProfile and place it only on GWT LambdaForms. When JIT sees it on some method, Compile::too_many_traps & Compile::too_many_recompiles for that method always return false. It allows JIT to prune the branch based on custom profile and recompile the method, if the branch is visited. For now, I wanted to keep the fix very focused. The next thing I plan to do is to experiment with ignoring deoptimization counts for other LambdaForms which are heavily shared. I already saw problems caused by deoptimization counts pollution (see JDK-8068915 [2]). I plan to backport the fix into 8u40, once I finish extensive performance testing. Testing: JPRT, java/lang/invoke tests, nashorn (nashorn testsuite, Octane). Thanks! PS: as a summary, my experiments show that fixes for 8063137 & 8068915 [2] almost completely recovers peak performance after LambdaForm sharing [3]. There's one more problem left (non-inlined MethodHandle invocations are more expensive when LFs are shared), but it's a story for another day. Best regards, Vladimir Ivanov [1] https://bugs.openjdk.java.net/browse/JDK-8059877 8059877: GWT branch frequencies pollution due to LF sharing [2] https://bugs.openjdk.java.net/browse/JDK-8068915 [3] https://bugs.openjdk.java.net/browse/JDK-8046703 JEP 210: LambdaForm Reduction and Caching From ivan.gerasimov at oracle.com Fri Jan 16 17:46:37 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Fri, 16 Jan 2015 20:46:37 +0300 Subject: JDK 9 RFR of JDK-8068948: Update java.base module to use new try-with-resources statement In-Reply-To: <54B69975.3010802@oracle.com> References: <54B5D8A3.1020000@oracle.com> <54B62086.8060200@oracle.com> <54B69975.3010802@oracle.com> Message-ID: <54B94E7D.10400@oracle.com> >> Might it also make sense to allow anonymous variables in the >> try-with-resource statement? >> So that something like `try (() -> System.out.println("closed")) {}` >> would work... > > I don't think that is necessary. In JDK 7, we started out allowing a > general AutoCloseable expression in a try-with-resources statement and > that proved problematic. Supporting a final / effectively final > variable seems to be the right balance. The rationale is written up in > the JSR 334 materials. > Yes, I agree. And it can easily be moved to the finally block, of course. However, this: ExecutorService executor = ...; try (executor::shutdown) { } still looks a bit more attractive to me, comparing to `try (Closeable c = executor::shutdown) {` Maybe one day ... :-) Sincerely yours, Ivan From chris.hegarty at oracle.com Fri Jan 16 18:14:19 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 16 Jan 2015 18:14:19 +0000 Subject: RFR : 8068795: Missing trailing blank in a HttpServer response leads to client crashes In-Reply-To: <54B94412.8@oracle.com> References: <54B94412.8@oracle.com> Message-ID: <853EA566-EC31-44F3-A133-8A37D5B13666@oracle.com> This looks ok to me Lev. -Chris. On 16 Jan 2015, at 17:02, Lev Priima wrote: > The second space should not be omitted in first line(status-line) of a http response message: http://tools.ietf.org/html/rfc7230#section-3.1.2 . > > Issue: http://bugs.openjdk.java.net/browse/JDK-8068795 > Patch: http://cr.openjdk.java.net/~lpriima/8068795/webrev.00/ > Testing: > $ jtreg -jdk:jdk9/dev/build/linux-x86_64-normal-server-release/jdk/ jdk9/dev/jdk/test/com/sun/net/httpserver/ > Test results: passed: 41 > > -- > Best Regards, > Lev > From chris.hegarty at oracle.com Fri Jan 16 18:32:24 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 16 Jan 2015 18:32:24 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <1421354021.1609.5.camel@Nokia-N900> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54B67A19.1050507@oracle.com> <54B7B1C0.4010802@zeus.net.au> <5D7FF52D-8E47-45ED-A738-911A2888A6DB@oracle.com> <1421354021.1609.5.camel@Nokia-N900> Message-ID: <1D115AAD-B2D3-490A-A69D-229D29BDFB2C@oracle.com> On 15/01/15 20:33, Peter Firmstone wrote: > Thanks Chris, > > WRT circular references, is it possible to detect and delay setting > these until after all verifiers run? It is possible to detect the circular reference. Currently you can retrieve a circular reference ( in readObject ) from readFields/GetFields, and all its fields will be their default value, null, 0, etc. How useful is it to be able to retrieve this reference? Probably not very useful. Could a static validator(GetField) API throw, or just return null, for a not yet created circular reference field value? It doesn?t seem useful for validation to be able to retrieve the circular reference. > An option might be to provide a protected method in ObjectInputStream to > opt out or switch off support for circular references. As of today, you can explicitly call readUnshared ( rather then readObject ), and if there is a cyclic reference to the object being deserializing will fail. Alternatively, if you want to protect individual fields, you can use serialPersistentFields and explicitly mark the field as unshared. Both mechanisms are inflexible. Note: unshared above is not limited to cyclic references, but all references in the stream. Maybe there is scope for such a switch, or similar, to disable circular references, for part of the graph, without completely disabling other references? > Apart from Serialization, another common use for readObject is for class > evolution. True, validation and sometimes evolution. > Our projects uses Serialization more than most, it was originally > written by the same people who earlier implemented Java's Serialization > framework. > > It's also the root cause of our security problems, I'm presently working > on a complete reimplimentation of ObjectInputStream using Apache > Harmony's code, as you might imagine I'm running into a lot of trouble > instantiating objects in a secure and cross platform compatible fashion. > > Can I suggest we have the static verifier return GetFields, to leave the > door open to supporting a constructor in future? If we finally agree upon a static validate(GetField) API, are you suggesting that the return type be GetField, and that an implementation of it would typically return the given arg? Or something else. -Chris. > Regards, > > Peter. > > ----- Original message ----- > > On 15 Jan 2015, at 12:25, Peter Firmstone > > > > wrote: > > > > > Chris, > > > > > > Can you explain some of the detail in your failure atomicity work? > > > > Certainly. > > > > The current implementation of defaultReadObject sets non-primitive field > > values one at a time, without first checking that all their types are > > assignable. If, for example, the assignment of the last non-primitive > > value fails, a CCE is thrown, and the previously set fields remain set. > > The setting of field values can be deferred until after they have been > > "validated", at a minimum that the non-primitive types are assignable. > > This is achievable per Class in the hierarchy, and provides some > > additional level of confidence. A change along the lines of this should > > not be controversial ( either all fields will be set, or contain their > > default values ), but it is only per Class in the hierarchy. > > > > Pushing on this a little, and ignoring cyclic references for now. For > > the first Serializable type in the hierarchy ( closest to j.l.Object ), > > there is no need to eagerly create the instance before ?validating? the > > field values. If the field values are assignable, then the object can > > created and the values assigned. If a field value is found to be > > unassignable, then CCE is thrown, and there is no dangling object. > > > > Pushing a little more. If there are no readObjectXXX methods in the > > hierarchy, then it would seem possible to read the field values for all > > the types in the hierarchy, and ?validate? them ( ensure they are > > assignable ), before creating the object and assigning the values. I do > > not believe that there are any observable affects from doing this. > > > > > > Cyclic references: Assumption, we do not want to leak our secure > > deserialized objects. It seems reasonable to not share the reference in > > the stream. This would rule out cyclic references. If you want sharing, > > then the implementation would create the referenced object, and assign > > the already read field values, when it encounters a cyclic reference. > > But you open the object to anything in the stream referencing it. > > > > readObjectXX: Assumption, many readObject methods exist to check > > invariants of values read from the stream. If we had another mechanism, > > similar to the static validate(GetFields) method we discussed > > previously, then may readObject methods could be eliminated. In which > > case we can achieve whole hierarchy failure atomicity ( as described > > above ). > > > > Given this, it seems like the natural place for a static invariant > > checker is between the reading and reconstitution of field values, and > > the assigning of them. If the invariant checker could be called by the > > serialization mechanism then the user code can use final fields, without > > needing to muck around with reflection, and also have the added benefit > > of, in many cases, not potential leaving the object in an inconsistent > > state. > > > > There is no support in this proposal for subclass to superclass > > invariant checking. > > > > I also believe that we can resolve the Finalization issue separately. > > > > -Chris. > From lev.priima at oracle.com Fri Jan 16 19:35:50 2015 From: lev.priima at oracle.com (Lev Priima) Date: Fri, 16 Jan 2015 22:35:50 +0300 Subject: RFR : 8068795: Missing trailing blank in a HttpServer response leads to client crashes In-Reply-To: <853EA566-EC31-44F3-A133-8A37D5B13666@oracle.com> References: <54B94412.8@oracle.com> <853EA566-EC31-44F3-A133-8A37D5B13666@oracle.com> Message-ID: <54B96816.6040206@oracle.com> Thanks Chris, Could you please push it? Best Regards, Lev On 01/16/2015 09:14 PM, Chris Hegarty wrote: > This looks ok to me Lev. > > -Chris. > > On 16 Jan 2015, at 17:02, Lev Priima wrote: > >> The second space should not be omitted in first line(status-line) of a http response message: http://tools.ietf.org/html/rfc7230#section-3.1.2 . >> >> Issue: http://bugs.openjdk.java.net/browse/JDK-8068795 >> Patch: http://cr.openjdk.java.net/~lpriima/8068795/webrev.00/ >> Testing: >> $ jtreg -jdk:jdk9/dev/build/linux-x86_64-normal-server-release/jdk/ jdk9/dev/jdk/test/com/sun/net/httpserver/ >> Test results: passed: 41 >> >> -- >> Best Regards, >> Lev >> From huizhe.wang at oracle.com Fri Jan 16 19:51:59 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Fri, 16 Jan 2015 11:51:59 -0800 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: <54934601.4010104@oracle.com> References: <54934601.4010104@oracle.com> Message-ID: <54B96BDF.7090204@oracle.com> Hi all, Could you review the change? Thanks, Joe On 12/18/2014 1:24 PM, huizhe wang wrote: > Hi, > > This is to add support for any type and improvement with new features > reflected in the new evaluateExpression methods, XPathEvaluationResult > and XPathNodes. > > https://bugs.openjdk.java.net/browse/JDK-8054196 > http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ > > Thanks, > Joe From nikolov.javor at gmail.com Fri Jan 16 20:14:33 2015 From: nikolov.javor at gmail.com (Yavor Nikolov) Date: Fri, 16 Jan 2015 22:14:33 +0200 Subject: [bg-jug] Re: RFR: 5050783: Throwable convenience method: String getStackTraceString() In-Reply-To: References: <54A3FAEB.7010709@oracle.com> <54B6C728.2030203@univ-mlv.fr> <54B7B3CD.4000009@oracle.com> Message-ID: Hi, On that note, I'm thinking loudly about whether or not it'd be appropriate > to extract common methods in PrintStream and PrintWriter to an interface. > java.io.Printable? > Thinking a bit further in a similar direction: a thinner interface gives more clarity about the API contract, it's also easier to override, especially if eligible for lambdas. I speculatively guess that logging implementations might also benefit from that: I mean simpler API like printStackTrace(Printable). Regarding the Appendable idea: one concern has been that println(s) and append(x + NL) have different behaviour in case of auto-flush-enabled output. Do we need to bother preserving printStackTrace auto-flush behaviour? (The Javadoc doesn't specify it, yet in real world there might have been assumptions that it auto-flushes...). Best regards, Yavor On Thu, Jan 15, 2015 at 3:05 PM, Ivan St. Ivanov wrote: > Thanks everybody for the feedback! :) > > Claes, we were deliberately trying to avoid the "Apache commons" solution. > It looks perfectly fine outside of the JDK. But our wild guess was that as > we have access to the Throwable private methods, we can do something more > elegant. The fact is (and I shared it with you in our private conversation) > that we now understood that most elegant is not always the best when it > comes to the core of the core of Java :) > > On that note, I'm thinking loudly about whether or not it'd be > appropriate to extract common methods in PrintStream and > PrintWriter to an interface. java.io.Printable? > > This would allow getting rid of most inner classes in j.l.Throwable > along with some duplicated code across the JDK (I've noticed > at least a few places where there are identical methods which > differ only in that one takes a PrintStream, the other a PrintWriter). > > Do you think that we can experiment with that as part of the effort for > this issue, or you think it is better to have it as a separate work item > (possibly done by someone with more experience) and apply it here once it > is ready? > > Regards, > Ivan > > > On Thu, Jan 15, 2015 at 2:34 PM, David Holmes > wrote: > >> On 15/01/2015 6:34 AM, Ivan St. Ivanov wrote: >> >>> Hi Remi, >>> >>> I tried it and, well, the JDK image now does not even compile classes >>> that >>> contain lambda expressions: >>> >>> javac -cp . ./LambdaMetaFactoryTester.java >>> Exception in thread "main" >>> Exception: java.lang.BootstrapMethodError thrown from the >>> UncaughtExceptionHandler in thread "main" >>> >>> With the old Throwable implementation it printed a long stack trace of >>> exceptions, but at least my sample class was compiled. >>> >>> So, I have two questions: >>> 1) Does anyone dump stack traces in such location, even for debugging >>> purposes? >>> 2) If yes, what are the other JDK classes and methods where we should >>> avoid >>> using lambdas? >>> >> >> It is hard to be definitive, but the initialization sequence for the JDK >> classes is extremely fragile. So anything that gets initialized very early >> in the VM initialization sequence needs to avoid using features that would >> require that the initialization sequence change. Using -verbose:class shows >> the order in which classes get loaded. >> >> David >> >> >> Thanks, >>> Ivan >>> >>> On Wed, Jan 14, 2015 at 9:44 PM, Remi Forax wrote: >>> >>> While using a lambda is cool in that context, >>>> if someone tries to do a printStackTrace in the LambdaMetafactory (for >>>> debugging purpose by example), >>>> it will be fun to debug. >>>> >>>> cheers, >>>> R?mi >>>> >>>> >>>> On 01/14/2015 08:25 PM, Ivan St. Ivanov wrote: >>>> >>>> Hi Claes, core libs developers, >>>>> >>>>> As promised, we did yesterday our hackathon in the Bulgarian Java User >>>>> Group and incorporated the feedback that we got from you. Here is our >>>>> next >>>>> webrev: >>>>> >>>>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.01/ >>>>> >>>>> As you can see: >>>>> >>>>> - We got rid of the inner classes altogether in favor of >>>>> java.util.function.Consumer >>>>> - The javadoc was updated to follow the standards [1] >>>>> - In the tests we use an exception that we create rather than relying >>>>> on >>>>> one from the JDK that potentially could change >>>>> >>>>> We also ran some microbenchmarks to compare the performance of the new >>>>> functionality with the Apache commons solution mentioned in the bug. >>>>> Here >>>>> is the benchmark test: >>>>> >>>>> @State(Scope.Benchmark) >>>>> public class ThrowableStacktraceBenchmark { >>>>> Throwable chainedThrowable; >>>>> Throwable simpleThrowable; >>>>> >>>>> @Setup >>>>> public void prepare() { >>>>> chainedThrowable = createThrowable(64); >>>>> simpleThrowable = new Throwable("Simple Throwable"); >>>>> } >>>>> >>>>> Throwable createThrowable(int depthLevel) { >>>>> Throwable t = null; >>>>> for (int i = 0; i < depthLevel; ++i) { >>>>> t = new Throwable("My dummy Throwable " + i, t); >>>>> } >>>>> return t; >>>>> } >>>>> >>>>> @Benchmark >>>>> public void benchmarkThrowableGetStackTraceString() { >>>>> chainedThrowable.getStackTraceString(); >>>>> } >>>>> >>>>> @Benchmark >>>>> public void benchmarkThrowableGetStackTraceStringSmall() { >>>>> simpleThrowable.getStackTraceString(); >>>>> } >>>>> >>>>> @Benchmark >>>>> public void benchmarkThrowablePrintStackTraceToStringWriter() { >>>>> StringWriter sw = new StringWriter(); >>>>> chainedThrowable.printStackTrace(new PrintWriter(sw)); >>>>> sw.toString(); >>>>> } >>>>> >>>>> @Benchmark >>>>> public void benchmarkThrowablePrintStackTraceToStringWriterSmall() >>>>> { >>>>> StringWriter sw = new StringWriter(); >>>>> simpleThrowable.printStackTrace(new PrintWriter(sw)); >>>>> sw.toString(); >>>>> } >>>>> } >>>>> >>>>> And here are the results (test was run with the following options -wi >>>>> 5 -i >>>>> 5 -t 1): >>>>> >>>>> Benchmark Mode Samples Score Error Units >>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTrac >>>>> eString >>>>> thrpt 50 25351.743 ? 347.326 ops/s >>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTrac >>>>> eStringSmall >>>>> thrpt 50 134990.643 ? 760.225 ops/s >>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>>> aceToStringWriter >>>>> thrpt 50 22730.786 ? 167.378 ops/s >>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>>> aceToStringWriterSmall >>>>> thrpt 50 115223.025 ? 936.770 ops/s >>>>> >>>>> Just in any case we ran this test, actually just the last two methods, >>>>> with >>>>> JDK 1.9 ea build 45. We wanted to compare the result of the Apache >>>>> commons >>>>> solution with inner classes (as it is now) to the same, but with >>>>> java.util.function.Consumer. There are the results with the inner >>>>> classes: >>>>> >>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>>> aceToStringWriter >>>>> thrpt 50 22974.324 ? 391.564 ops/s >>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>>> aceToStringWriterSmall >>>>> thrpt 50 114973.452 ? 1051.276 ops/s >>>>> >>>>> So you can see that the performance is nearly the same (I guess that >>>>> the >>>>> difference is ignorable). >>>>> >>>>> Please share with us the feedback about this proposal! >>>>> >>>>> Thanks, >>>>> Ivan >>>>> >>>>> [1] http://www.oracle.com/technetwork/articles/java/index-137868.html >>>>> >>>>> On Wed, Dec 31, 2014 at 3:32 PM, Claes Redestad < >>>>> claes.redestad at oracle.com> >>>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>>> >>>>>> seems like a reasonable convenience method. :-) >>>>>> >>>>>> Process stuff: OCA/JSPA signing aside, the patch needs to be >>>>>> submitted in >>>>>> its >>>>>> entirety to some part of the OpenJDK infrastructure before it can be >>>>>> accepted. >>>>>> Attaching the patch in an e-mail to this mailing list is acceptable >>>>>> for a >>>>>> small >>>>>> patch like this when lacking upload privileges to cr.openjdk.java.net. >>>>>> A >>>>>> sponsor >>>>>> could help you with the uploading etc. >>>>>> >>>>>> This request adds a new public method, which means some internal >>>>>> approval >>>>>> will be necessary. This is something you'll need help with from an >>>>>> Oracle >>>>>> sponsor. I could volunteer to initiate the requests internally. >>>>>> >>>>>> Code feedback: >>>>>> >>>>>> The test relies on the message generated when provoking a division by >>>>>> zero. I'm not sure this is a good idea nor the place to enforce this >>>>>> message format with a test. Perhaps just throwing a RuntimeException >>>>>> with some specified message and test for that? >>>>>> >>>>>> May I suggest to also add a test to ensure the method behaves well >>>>>> for an >>>>>> exception with fillInStackTrace overridden to produce stackless >>>>>> exceptions? >>>>>> >>>>>> Just a thought... I noticed PrintWriter, PrintStream and StringBuilder >>>>>> all >>>>>> inherit from Appendable, so the PrintStreamOrWriter/StackTracePrinter >>>>>> classes >>>>>> could be replaced with a single StackTraceAppendable taking an >>>>>> Appendable. >>>>>> One >>>>>> static class instead of 1 abstract and 3 concrete static inner classes >>>>>> could work >>>>>> out to be a good deal, but there's some odd mechanics in >>>>>> BufferedWriter/PrintWriter >>>>>> to use the value of the line.separator property at object creation >>>>>> time >>>>>> which >>>>>> would be hard to support without exposing the lineSeparator fields to >>>>>> Throwable. >>>>>> Perhaps a future consideration. >>>>>> >>>>>> Performance characteristics might change ever so slightly either way, >>>>>> so >>>>>> it >>>>>> would be nice with some microbenchmarks to quantify. >>>>>> >>>>>> /Claes >>>>>> >>>>>> >>>>>> On 2014-12-31 11:51, Ivan St. Ivanov wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>>> >>>>>>> I am Ivan from the Bulgarian Java User Group. As part of our Adopt >>>>>>> OpenJDK >>>>>>> efforts, we organized hackathon beginning of this month in Sofia. >>>>>>> There >>>>>>> we >>>>>>> worked on the following JIRA issue: >>>>>>> >>>>>>> https://bugs.openjdk.java.net/browse/JDK-5050783 >>>>>>> >>>>>>> Unfortunately none of us has author privileges, so the issue is still >>>>>>> unassigned. But, nevertheless, our JUG has already signed JSPA and >>>>>>> some >>>>>>> of >>>>>>> our members (including myself) have signed OCA. And we informed in a >>>>>>> special email this mailing list about our plans prior to the meeting. >>>>>>> >>>>>>> Our hackathon was already documented by our JUG member Mihail >>>>>>> Stoynov: >>>>>>> >>>>>>> http://mihail.stoynov.com/2014/12/12/building-openjdk- >>>>>>> and-submitting-a-solution-on-a-feature-request-with-the- >>>>>>> bulgarian-java-user-group/ >>>>>>> >>>>>>> We used the days after the meeting for offline polishing our >>>>>>> contribution >>>>>>> and adding some more unit tests. Here is the result of our work: >>>>>>> >>>>>>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.00/ >>>>>>> >>>>>>> We would be very happy if you are able to review it, share your >>>>>>> feedback >>>>>>> and guide us through the contribution process as this is our first >>>>>>> attempt. >>>>>>> >>>>>>> Thanks and have a very successful year! >>>>>>> Ivan >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> > -- > You received this message because you are subscribed to the Google Groups > "Bulgarian Java Users Group" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to bg-jug+unsubscribe at googlegroups.com. > To post to this group, send email to bg-jug at googlegroups.com. > Visit this group at http://groups.google.com/group/bg-jug. > For more options, visit https://groups.google.com/d/optout. > From vladimir.kozlov at oracle.com Fri Jan 16 20:34:50 2015 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Fri, 16 Jan 2015 12:34:50 -0800 Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when GWT LambdaForms are shared In-Reply-To: <54B94766.2080102@oracle.com> References: <54B94766.2080102@oracle.com> Message-ID: <54B975EA.6040005@oracle.com> Nice! At least Hotspot part since I don't understand jdk part :) I would suggest to add more detailed comment (instead of simple "Stop profiling") to inline_profileBranch() intrinsic explaining what it is doing because it is not strictly "intrinsic" - it does not implement profileBranch() java code when counts is constant. You forgot to mark Opaque4Node as macro node. I would suggest to base it on Opaque2Node then you will get some methods from it. Thanks, Vladimir On 1/16/15 9:16 AM, Vladimir Ivanov wrote: > http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/hotspot/ > http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/jdk/ > https://bugs.openjdk.java.net/browse/JDK-8063137 > > After GuardWithTest (GWT) LambdaForms became shared, profile pollution > significantly distorted compilation decisions. It affected inlining and > hindered some optimizations. It causes significant performance > regressions for Nashorn (on Octane benchmarks). > > Inlining was fixed by 8059877 [1], but it didn't cover the case when a > branch is never taken. It can cause missed optimization opportunity, and > not just increase in code size. For example, non-pruned branch can break > escape analysis. > > Currently, there are 2 problems: > - branch frequencies profile pollution > - deoptimization counts pollution > > Branch frequency pollution hides from JIT the fact that a branch is > never taken. Since GWT LambdaForms (and hence their bytecode) are > heavily shared, but the behavior is specific to MethodHandle, there's no > way for JIT to understand how particular GWT instance behaves. > > The solution I propose is to do profiling in Java code and feed it to > JIT. Every GWT MethodHandle holds an auxiliary array (int[2]) where > profiling info is stored. Once JIT kicks in, it can retrieve these > counts, if corresponding MethodHandle is a compile-time constant (and it > is usually the case). To communicate the profile data from Java code to > JIT, MethodHandleImpl::profileBranch() is used. > > If GWT MethodHandle isn't a compile-time constant, profiling should > proceed. It happens when corresponding LambdaForm is already shared, for > newly created GWT MethodHandles profiling can occur only in native code > (dedicated nmethod for a single LambdaForm). So, when compilation of the > whole MethodHandle chain is triggered, the profile should be already > gathered. > > Overriding branch frequencies is not enough. Statistics on > deoptimization events is also polluted. Even if a branch is never taken, > JIT doesn't issue an uncommon trap there unless corresponding bytecode > doesn't trap too much and doesn't cause too many recompiles. > > I added @IgnoreProfile and place it only on GWT LambdaForms. When JIT > sees it on some method, Compile::too_many_traps & > Compile::too_many_recompiles for that method always return false. It > allows JIT to prune the branch based on custom profile and recompile the > method, if the branch is visited. > > For now, I wanted to keep the fix very focused. The next thing I plan to > do is to experiment with ignoring deoptimization counts for other > LambdaForms which are heavily shared. I already saw problems caused by > deoptimization counts pollution (see JDK-8068915 [2]). > > I plan to backport the fix into 8u40, once I finish extensive > performance testing. > > Testing: JPRT, java/lang/invoke tests, nashorn (nashorn testsuite, Octane). > > Thanks! > > PS: as a summary, my experiments show that fixes for 8063137 & 8068915 > [2] almost completely recovers peak performance after LambdaForm sharing > [3]. There's one more problem left (non-inlined MethodHandle invocations > are more expensive when LFs are shared), but it's a story for another day. > > Best regards, > Vladimir Ivanov > > [1] https://bugs.openjdk.java.net/browse/JDK-8059877 > 8059877: GWT branch frequencies pollution due to LF sharing > [2] https://bugs.openjdk.java.net/browse/JDK-8068915 > [3] https://bugs.openjdk.java.net/browse/JDK-8046703 > JEP 210: LambdaForm Reduction and Caching > _______________________________________________ > mlvm-dev mailing list > mlvm-dev at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev From brian.burkhalter at oracle.com Fri Jan 16 21:18:18 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 16 Jan 2015 13:18:18 -0800 Subject: [9] RFR of 8066842: java.math.BigDecimal.divide(BigDecimal, RoundingMode) produces incorrect result Message-ID: <27F1BCB8-8160-4E82-A12C-2932C33EE9D5@oracle.com> Hello, Please review at your convenience. Issue: https://bugs.openjdk.java.net/browse/JDK-8066842 Patch: http://cr.openjdk.java.net/~bpb/8066842/webrev.00/ The problem appears to have been that at line 4941 of the old source, in the divWord() method, one or both of the long variables ?r? and ?q? overflowed the range of int so that information was lost when these variables were truncated to 32 bits. The code of divWord() seems to have been ported from a method of the same name in MutableBigInteger wherein its constraints were made explicit. In the patch, divWord() is replaced by divRemNegativeLong(), and the use of the former in divideAndRound128() replaced with inline code for the non-negative dividend cases, and by divRemNegativeLong() for the negative dividend cases. Thanks, Brian From xueming.shen at oracle.com Fri Jan 16 21:28:50 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Fri, 16 Jan 2015 13:28:50 -0800 Subject: RFR JDK-8042262: Javadoc typo in java.util.Formatter Message-ID: <54B98292.9080807@oracle.com> Please help review the typo in the Formatter javadoc. https://bugs.openjdk.java.net/browse/JDK-8042262 http://cr.openjdk.java.net/~sherman/8042262/ thanks! -Sherman From lance.andersen at oracle.com Fri Jan 16 21:29:50 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 16 Jan 2015 16:29:50 -0500 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: <54B96BDF.7090204@oracle.com> References: <54934601.4010104@oracle.com> <54B96BDF.7090204@oracle.com> Message-ID: Hi Joe, Overall it is OK, a few minor comments - Is there a reason that XPathExpressionImpl is no longer public and XPathImpl is public? - I think you can leverage {@inheritdoc} in your impl classes to avoid comment duplication possibly? - please remember to make the copyright year 2015 - XPathTestBase, can the static block be moved to a @BeforeClass - XPathNodes, should that have an @since 1.9? - Given you are not including @param, etc for your test comments, you might want to consider /* */ vs /** */ style comments. That is consider if you really want doc comments (really a style choice but some IDEs will issue a warning for missing tags HTH Best, Lance On Jan 16, 2015, at 2:51 PM, huizhe wang wrote: > Hi all, > > Could you review the change? > > Thanks, > Joe > > On 12/18/2014 1:24 PM, huizhe wang wrote: >> Hi, >> >> This is to add support for any type and improvement with new features reflected in the new evaluateExpression methods, XPathEvaluationResult and XPathNodes. >> >> https://bugs.openjdk.java.net/browse/JDK-8054196 >> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >> >> Thanks, >> Joe > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From lance.andersen at oracle.com Fri Jan 16 21:30:31 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 16 Jan 2015 16:30:31 -0500 Subject: RFR JDK-8042262: Javadoc typo in java.util.Formatter In-Reply-To: <54B98292.9080807@oracle.com> References: <54B98292.9080807@oracle.com> Message-ID: <2FD91507-25B4-4038-8CD9-C66FB4EC454B@oracle.com> +1 On Jan 16, 2015, at 4:28 PM, Xueming Shen wrote: > Please help review the typo in the Formatter javadoc. > > https://bugs.openjdk.java.net/browse/JDK-8042262 > http://cr.openjdk.java.net/~sherman/8042262/ > > thanks! > -Sherman Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From Roger.Riggs at Oracle.com Fri Jan 16 21:30:35 2015 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Fri, 16 Jan 2015 16:30:35 -0500 Subject: RFR JDK-8042262: Javadoc typo in java.util.Formatter In-Reply-To: <54B98292.9080807@oracle.com> References: <54B98292.9080807@oracle.com> Message-ID: <54B982FB.4070505@Oracle.com> Hi Sherman, Looks fine. Roger On 1/16/2015 4:28 PM, Xueming Shen wrote: > Please help review the typo in the Formatter javadoc. > > https://bugs.openjdk.java.net/browse/JDK-8042262 > http://cr.openjdk.java.net/~sherman/8042262/ > > thanks! > -Sherman From brian.burkhalter at oracle.com Fri Jan 16 22:55:28 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 16 Jan 2015 14:55:28 -0800 Subject: JDK 9 RFR of 8067669: Documentation for methods in Number incomplete regarding too large values. In-Reply-To: <20150108151355.GA11334@e6430> References: <9D55A146-D76D-4E91-B02D-2BC646451CE2@oracle.com> <20141217102612.GE4911@e6430> <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com> <5494B41B.8060003@oracle.com> <0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com> <3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com> <54AB3DBE.5040806@oracle.com> <20150107080837.GB2073@e6430> <54ADC94E.4090607@oracle.com> <20150108151355.GA11334@e6430> Message-ID: <46FD3812-626A-40AE-8305-10D6B1CA7B4C@oracle.com> On Jan 8, 2015, at 7:13 AM, Andreas Lundblad wrote: > Although I was the one who brought it up, I think the whole Comparator discussion is slightly off topic. I still think that explicitly mentioning rounding and truncation is a bit confusing, as even the JDK implementations resort to other methods. I don?t think we quite agreed on the resolution of this issue. I don?t have a strong opinion about it, but I do think it should be resolved either as Fixed or Not An Issue. One alternative which removes the seemingly contentious verbiage is here: http://cr.openjdk.java.net/~bpb/8067669/webrev.01/ Note that the change at line 40 should be made even if the other diffs are rejected. Thanks, Brian From john.r.rose at oracle.com Fri Jan 16 23:13:48 2015 From: john.r.rose at oracle.com (John Rose) Date: Fri, 16 Jan 2015 15:13:48 -0800 Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when GWT LambdaForms are shared In-Reply-To: <54B94766.2080102@oracle.com> References: <54B94766.2080102@oracle.com> Message-ID: <7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com> On Jan 16, 2015, at 9:16 AM, Vladimir Ivanov wrote: > > http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/hotspot/ > http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/jdk/ > https://bugs.openjdk.java.net/browse/JDK-8063137 > ... > PS: as a summary, my experiments show that fixes for 8063137 & 8068915 [2] almost completely recovers peak performance after LambdaForm sharing [3]. There's one more problem left (non-inlined MethodHandle invocations are more expensive when LFs are shared), but it's a story for another day. This performance bump is excellent news. LFs are supposed to express emergently common behaviors, like hidden classes. We are much closer to that goal now. I'm glad to see that the library-assisted profiling turns out to be relatively clean. In effect this restores the pre-LF CountingMethodHandle logic from 2011, which was so beneficial in JDK 7: http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/02de5cdbef21/src/share/classes/java/lang/invoke/CountingMethodHandle.java I have some suggestions to make this version a little cleaner; see below. Starting with the JDK changes: In LambdaForm.java, I'm feeling flag pressure from all the little boolean fields and constructor parameters. (Is it time to put in a bit-encoded field "private byte LambdaForm.flags", or do we wait for another boolean to come along? But see next questions, which are more important.) What happens when a GWT LF gets inlined into a larger LF? Then there might be two or more selectAlternative calls. Will this confuse anything or will it Just Work? The combined LF will get profiled as usual, and the selectAlternative calls will also collect profile (or not?). This leads to another question: Why have a boolean 'isGWT' at all? Why not just check for one or more occurrence of selectAlternative, and declare that those guys override (some of) the profiling. Something like: -+ if (PROFILE_GWT && lambdaForm.isGWT) ++ if (PROFILE_GWT && lambdaForm.containsFunction(NF_selectAlternative)) (...where LF.containsFunction(NamedFunction) is a variation of LF.contains(Name).) I suppose the answer may be that you want to inline GWTs (if ever) into customized code where the JVM profiling should get maximum benefit. In that case case you might want to set the boolean to "false" to distinguish "immature" GWT combinators from customized ones. If that's the case, perhaps the real boolean flag you want is not 'isGWT' but 'sharedProfile' or 'immature' or some such, or (inverting) 'customized'. (I like the feel of a 'customized' flag.) Then @IgnoreProfile would get attached to a LF that (a ) contains selectAlternative and (b ) is marked as non-customized/immature/shared. You might also want to adjust the call to 'profileBranch' based on whether the containing LF was shared or customized. What I'm mainly poking at here is that 'isGWT' is not informative about the intended use of the flag. In 'updateCounters', if the counter overflows, you'll get continuous creation of ArithmeticExceptions. Will that optimize or will it cause a permanent slowdown? Consider a hack like this on the exception path: counters[idx] = Integer.MAX_VALUE / 2; On the Name Bikeshed: It looks like @IgnoreProfile (ignore_profile in the VM) promises too much "ignorance", since it suppresses branch counts and traps, but allows type profiles to be consulted. Maybe something positive like "@ManyTraps" or "@SharedMegamorphic"? (It's just a name, and this is just a suggestion.) Going to the JVM: In library_call.cpp, I think you should change the assert to a guard: -+ assert(aobj->length() == 2, ""); ++ && aobj->length() == 2) { In Parse::dynamic_branch_prediction, the mere presence of the Opaque4 node is enough to trigger replacement of profiling. I think there should *not* be a test of method()->ignore_profile(). That should provide better integration between the two sources of profile data to JVM profiling? Also, I think the name 'Opaque4Node' is way too? opaque. Suggest 'ProfileBranchNode', since that's exactly what it does. Suggest changing the log element "profile_branch" to "observe source='profileBranch'", to make a better hint as to the source of the info. ? John From brian.burkhalter at oracle.com Sat Jan 17 00:02:02 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 16 Jan 2015 16:02:02 -0800 Subject: [9] RFR of 6880737: (fs) FileLock constructors don't throw NPE if the channel argument is null Message-ID: <3B73EA6D-467E-4035-B3DD-949129D07E99@oracle.com> Hello, Please review at your convenience. Issue: https://bugs.openjdk.java.net/browse/JDK-6880737 Patch: http://cr.openjdk.java.net/~bpb/6880737/webrev.00/ This is a minor change to check the channel parameter of FileLock(). Although this class is public, a non-JDK subclass does not appear to be able to be used via any JDK API so the effect of the change should be innocuous. The minor change to the channel parameter specification should not, I would think, merit a CCC request. Thanks, Brian From huizhe.wang at oracle.com Sat Jan 17 02:33:14 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Fri, 16 Jan 2015 18:33:14 -0800 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: References: <54934601.4010104@oracle.com> <54B96BDF.7090204@oracle.com> Message-ID: <54B9C9EA.7050407@oracle.com> On 1/16/2015 1:29 PM, Lance Andersen wrote: > Hi Joe, > > > > Overall it is OK, a few minor comments > > - Is there a reason that XPathExpressionImpl is no longer public and > XPathImpl is public? Ok, I'll keep both public, may be useful in the future. > - I think you can leverage {@inheritdoc} in your impl classes to avoid > comment duplication possibly? Looks like javadoc "Automatically inherit comment ". So @Override is good enough. I've removed the javadocs for methods that override, including the existing methods. It's good to avoid the duplication, and potential copy n paste errors. > - please remember to make the copyright year 2015 Done. > - XPathTestBase, can the static block be moved to a @BeforeClass Moved to within the Dataprovider. > - XPathNodes, should that have an @since 1.9? Fixed. > - Given you are not including @param, etc for your test comments, you > might want to consider /* */ vs /** */ style comments. Looks like s/\/**/\/* worked the trick. > That is consider if you really want doc comments (really a style > choice but some IDEs will issue a warning for missing tags Yeh, it's good to turn off the warning "light" :-) http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ Thanks, Joe > > > HTH > > Best, > Lance > On Jan 16, 2015, at 2:51 PM, huizhe wang > wrote: > >> Hi all, >> >> Could you review the change? >> >> Thanks, >> Joe >> >> On 12/18/2014 1:24 PM, huizhe wang wrote: >>> Hi, >>> >>> This is to add support for any type and improvement with new >>> features reflected in the new evaluateExpression methods, >>> XPathEvaluationResult and XPathNodes. >>> >>> https://bugs.openjdk.java.net/browse/JDK-8054196 >>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>> >>> Thanks, >>> Joe >> > > > > Lance > Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > From Alan.Bateman at oracle.com Sat Jan 17 07:33:48 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 17 Jan 2015 07:33:48 +0000 Subject: [9] RFR of 6880737: (fs) FileLock constructors don't throw NPE if the channel argument is null In-Reply-To: <3B73EA6D-467E-4035-B3DD-949129D07E99@oracle.com> References: <3B73EA6D-467E-4035-B3DD-949129D07E99@oracle.com> Message-ID: <54BA105C.8030809@oracle.com> On 17/01/2015 00:02, Brian Burkhalter wrote: > Hello, > > Please review at your convenience. > > Issue: https://bugs.openjdk.java.net/browse/JDK-6880737 > Patch: http://cr.openjdk.java.net/~bpb/6880737/webrev.00/ > > This is a minor change to check the channel parameter of FileLock(). Although this class is public, a non-JDK subclass does not appear to be able to be used via any JDK API so the effect of the change should be innocuous. The minor change to the channel parameter specification should not, I would think, merit a CCC request. > The change to the javadoc isn't strictly required as there is a general statement in the package description that NPE be thrown when methods or constructors are called with null. So the implementation should throw NPE rather than IAE and as it's an incompatible change then it should be tracked. It's important to add a test case too. -Alan From peter.firmstone at zeus.net.au Sun Jan 18 00:24:55 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Sun, 18 Jan 2015 10:24:55 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <1D115AAD-B2D3-490A-A69D-229D29BDFB2C@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54B67A19.1050507@oracle.com> <54B7B1C0.4010802@zeus.net.au> <5D7FF52D-8E47-45ED-A738-911A2888A6DB@oracle.com> <1421354021.1609.5.camel@Nokia-N900> <1D115AAD-B2D3-490A-A69D-229D29BDFB2C@oracle.com> Message-ID: <1421540695.1520.14.camel@Nokia-N900> ----- Original message ----- > On 15/01/15 20:33, Peter Firmstone wrote: > > Thanks Chris, > > > > WRT circular references, is it possible to detect and delay setting > > these until after all verifiers run? > > It is possible to detect the circular reference. > > Currently you can retrieve a circular reference ( in readObject ) from > readFields/GetFields, and all its fields will be their default value, > null, 0, etc. How useful is it to be able to retrieve this reference? > Probably not very useful. You're right, circular links can't be checked until after construction. A class with one of these links will have to check its invariants on every method invocation (perhaps lazily). > > Could a static validator(GetField) API throw, or just return null, for a > not yet created circular reference field value? Probably better to fail early and throw an exception, if the developer is aware of the circularity, they won't attempt to retrieve it before their object has been constructed. It doesn?t seem useful > for validation to be able to retrieve the circular reference. I agree, not much use for validation. > > > An option might be to provide a protected method in ObjectInputStream > > to opt out or switch off support for circular references. > > As of today, you can explicitly call readUnshared ( rather then > readObject ), and if there is a cyclic reference to the object being > deserializing will fail. Alternatively, if you want to protect > individual fields, you can use serialPersistentFields and explicitly > mark the field as unshared. Both mechanisms are inflexible. Unfortunately read unshared has some issues too. > > Note: unshared above is not limited to cyclic references, but all > references in the stream. > > Maybe there is scope for such a switch, or similar, to disable circular > references, for part of the graph, without completely disabling other > references? > > > Apart from Serialization, another common use for readObject is for > > class evolution. > > True, validation and sometimes evolution. > > > Our projects uses Serialization more than most, it was originally > > written by the same people who earlier implemented Java's Serialization > > framework. > > > > It's also the root cause of our security problems, I'm presently > > working on a complete reimplimentation of ObjectInputStream using > > Apache Harmony's code, as you might imagine I'm running into a lot of > > trouble instantiating objects in a secure and cross platform > > compatible fashion. > > > > Can I suggest we have the static verifier return GetFields, to leave > > the door open to supporting a constructor in future? > > If we finally agree upon a static validate(GetField) API, are you > suggesting that the return type be GetField, and that an implementation > of it would typically return the given arg? Yes. One problem with our validator, is we can only verify "fields" there's nothing we can do with readObject methods that read values directly from the stream. Interacting directly with the stream in readObject or writeObject methods is brittle, it requires class bytecode. Also browsing readObject implementations in openjdk, I've found dos vulnerabilities. For our project, I'm looking at implementing a stream that sacrifices compatibility on the alter of security. This will be constraint based and only used for unauthenticated network connections. I'll be using the serialization protocol, but with a public api that requires caller sensitive GetField and PutField as parameters, no direct access to the stream and no magic. The interface will have a different name, so as not to conflict with Serializable. Hindsight and learning from mistakes is 20:20. The Objects that can be serialized will be minimal: arrays String, Enum, Object primitives. Stateless Objects Proxy Maps and Collections will be converted to an immutable safe implementation that the receiver can use as an argument to their preferred map or collection, preserving Comparator's if they exist. All other Object types will have to implement the new interface. Implementers will be able to implement Serializable as well as the new interface, for compatibility. I'm not sure of what name, perhaps AtomicSerial. The receiver will be able to place size constriants on the stream, arrays, Collections and Maps. This will need to be performed over non blocking sockets as well. Sctp looks very promising for this. Peter. Or something else. > > -Chris. > > > Regards, > > > > Peter. > > > > ----- Original message ----- > > > On 15 Jan 2015, at 12:25, Peter Firmstone > > > > > > wrote: > > > > > > > Chris, > > > > > > > > Can you explain some of the detail in your failure atomicity work? > > > > > > Certainly. > > > > > > The current implementation of defaultReadObject sets non-primitive > > > field values one at a time, without first checking that all their > > > types are assignable. If, for example, the assignment of the last > > > non-primitive value fails, a CCE is thrown, and the previously set > > > fields remain set. The setting of field values can be deferred until > > > after they have been "validated", at a minimum that the > > > non-primitive types are assignable. This is achievable per Class in > > > the hierarchy, and provides some additional level of confidence. A > > > change along the lines of this should not be controversial ( either > > > all fields will be set, or contain their default values ), but it is > > > only per Class in the hierarchy. > > > > > > Pushing on this a little, and ignoring cyclic references for now. > > > For the first Serializable type in the hierarchy ( closest to > > > j.l.Object ), there is no need to eagerly create the instance before > > > ?validating? the field values. If the field values are assignable, > > > then the object can created and the values assigned. If a field > > > value is found to be unassignable, then CCE is thrown, and there is > > > no dangling object. > > > > > > Pushing a little more. If there are no readObjectXXX methods in the > > > hierarchy, then it would seem possible to read the field values for > > > all the types in the hierarchy, and ?validate? them ( ensure they are > > > assignable ), before creating the object and assigning the values. I > > > do not believe that there are any observable affects from doing this. > > > > > > > > > Cyclic references: Assumption, we do not want to leak our secure > > > deserialized objects. It seems reasonable to not share the reference > > > in the stream. This would rule out cyclic references.? If you want > > > sharing, then the implementation would create the referenced object, > > > and assign the already read field values, when it encounters a > > > cyclic reference. But you open the object to anything in the stream > > > referencing it. > > > > > > readObjectXX:? Assumption, many readObject methods exist to check > > > invariants of values read from the stream. If we had another > > > mechanism, similar to the static validate(GetFields) method we > > > discussed previously, then may readObject methods could be > > > eliminated. In which case we can achieve whole hierarchy failure > > > atomicity ( as described above ). > > > > > > Given this, it seems like the natural place for a static invariant > > > checker is between the reading and reconstitution of field values, > > > and the assigning of them. If the invariant checker could be called > > > by the serialization mechanism then the user code can use final > > > fields, without needing to muck around with reflection, and also > > > have the added benefit of, in many cases, not potential leaving the > > > object in an inconsistent state. > > > > > > There is no support in this proposal for subclass to superclass > > > invariant checking. > > > > > > I also believe that we can resolve the Finalization issue separately. > > > > > > -Chris. > > From claes.redestad at oracle.com Sun Jan 18 22:20:50 2015 From: claes.redestad at oracle.com (Claes Redestad) Date: Sun, 18 Jan 2015 23:20:50 +0100 Subject: [bg-jug] Re: RFR: 5050783: Throwable convenience method: String getStackTraceString() In-Reply-To: References: <54A3FAEB.7010709@oracle.com> <54B6C728.2030203@univ-mlv.fr> <54B7B3CD.4000009@oracle.com> Message-ID: <54BC31C2.8030202@oracle.com> Hi, On 2015-01-16 21:14, Yavor Nikolov wrote: > Hi, > >> On that note, I'm thinking loudly about whether or not it'd be appropriate >> to extract common methods in PrintStream and PrintWriter to an interface. >> java.io.Printable? >> > Thinking a bit further in a similar direction: a thinner interface gives > more clarity about the API contract, it's also easier to override, > especially if eligible for lambdas. I speculatively guess that logging > implementations might also benefit from that: I mean simpler API like > printStackTrace(Printable). Agreed, but sadly we wouldn't able to remove any existing methods (breaks binary compatibility), thus adding methods taking Printable to legacy classes like Throwable might cost more complexity in the API than it will help to reduce. A more thorough analysis might be needed to prove if it's worth it. The behavior here should be pretty well-defined, so I've so far not seen any purely theoretical reasons not to do it (should not violate the Liskov substition principle etc). Related discussions have occurred on core-libs before, for example adding methods with ones taking CharSequence next to or even instead of ones taking String[1]. Typically this hasn't happened for various reasons, but it seems easier to use the more generic interface when adding new functionality[2]. > Regarding the Appendable idea: one concern has been that println(s) and > append(x + NL) have different behaviour in case of auto-flush-enabled > output. Do we need to bother preserving printStackTrace auto-flush > behaviour? (The Javadoc doesn't specify it, yet in real world there might > have been assumptions that it auto-flushes...). Yes, sufficient reason to scrap that approach. > Best regards, > Yavor > > On Thu, Jan 15, 2015 at 3:05 PM, Ivan St. Ivanov > wrote: > >> Thanks everybody for the feedback! :) >> >> Claes, we were deliberately trying to avoid the "Apache commons" solution. >> It looks perfectly fine outside of the JDK. But our wild guess was that as >> we have access to the Throwable private methods, we can do something more >> elegant. The fact is (and I shared it with you in our private conversation) >> that we now understood that most elegant is not always the best when it >> comes to the core of the core of Java :) >> >> On that note, I'm thinking loudly about whether or not it'd be >> appropriate to extract common methods in PrintStream and >> PrintWriter to an interface. java.io.Printable? >> >> This would allow getting rid of most inner classes in j.l.Throwable >> along with some duplicated code across the JDK (I've noticed >> at least a few places where there are identical methods which >> differ only in that one takes a PrintStream, the other a PrintWriter). >> >> Do you think that we can experiment with that as part of the effort for >> this issue, or you think it is better to have it as a separate work item >> (possibly done by someone with more experience) and apply it here once it >> is ready? For this particular issue, no, this is most definitely out of scope. Working on larger API changes (cleanups?) like this should probably be done, planned and worked on as a (smallish) JEP[3], so that the pros and cons can be made clear. Whether or not this is interesting: I don't really know. I think it could be of interest if you're willing to spend the time, but I'm far from the only one who needs to be convinced. There might be more approachable issues to work on to get more experience and contribute more directly to the project. Hint: anything which doesn't require a public API change is typically orders of magnitude easier to get accepted. /Claes [1] http://mail.openjdk.java.net/pipermail/core-libs-dev/2012-April/009819.html [2] https://bugs.openjdk.java.net/browse/JDK-8041972 [3] http://cr.openjdk.java.net/~mr/jep/jep-2.0.html >> >> Regards, >> Ivan >> >> >> On Thu, Jan 15, 2015 at 2:34 PM, David Holmes >> wrote: >> >>> On 15/01/2015 6:34 AM, Ivan St. Ivanov wrote: >>> >>>> Hi Remi, >>>> >>>> I tried it and, well, the JDK image now does not even compile classes >>>> that >>>> contain lambda expressions: >>>> >>>> javac -cp . ./LambdaMetaFactoryTester.java >>>> Exception in thread "main" >>>> Exception: java.lang.BootstrapMethodError thrown from the >>>> UncaughtExceptionHandler in thread "main" >>>> >>>> With the old Throwable implementation it printed a long stack trace of >>>> exceptions, but at least my sample class was compiled. >>>> >>>> So, I have two questions: >>>> 1) Does anyone dump stack traces in such location, even for debugging >>>> purposes? >>>> 2) If yes, what are the other JDK classes and methods where we should >>>> avoid >>>> using lambdas? >>>> >>> It is hard to be definitive, but the initialization sequence for the JDK >>> classes is extremely fragile. So anything that gets initialized very early >>> in the VM initialization sequence needs to avoid using features that would >>> require that the initialization sequence change. Using -verbose:class shows >>> the order in which classes get loaded. >>> >>> David >>> >>> >>> Thanks, >>>> Ivan >>>> >>>> On Wed, Jan 14, 2015 at 9:44 PM, Remi Forax wrote: >>>> >>>> While using a lambda is cool in that context, >>>>> if someone tries to do a printStackTrace in the LambdaMetafactory (for >>>>> debugging purpose by example), >>>>> it will be fun to debug. >>>>> >>>>> cheers, >>>>> R?mi >>>>> >>>>> >>>>> On 01/14/2015 08:25 PM, Ivan St. Ivanov wrote: >>>>> >>>>> Hi Claes, core libs developers, >>>>>> As promised, we did yesterday our hackathon in the Bulgarian Java User >>>>>> Group and incorporated the feedback that we got from you. Here is our >>>>>> next >>>>>> webrev: >>>>>> >>>>>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.01/ >>>>>> >>>>>> As you can see: >>>>>> >>>>>> - We got rid of the inner classes altogether in favor of >>>>>> java.util.function.Consumer >>>>>> - The javadoc was updated to follow the standards [1] >>>>>> - In the tests we use an exception that we create rather than relying >>>>>> on >>>>>> one from the JDK that potentially could change >>>>>> >>>>>> We also ran some microbenchmarks to compare the performance of the new >>>>>> functionality with the Apache commons solution mentioned in the bug. >>>>>> Here >>>>>> is the benchmark test: >>>>>> >>>>>> @State(Scope.Benchmark) >>>>>> public class ThrowableStacktraceBenchmark { >>>>>> Throwable chainedThrowable; >>>>>> Throwable simpleThrowable; >>>>>> >>>>>> @Setup >>>>>> public void prepare() { >>>>>> chainedThrowable = createThrowable(64); >>>>>> simpleThrowable = new Throwable("Simple Throwable"); >>>>>> } >>>>>> >>>>>> Throwable createThrowable(int depthLevel) { >>>>>> Throwable t = null; >>>>>> for (int i = 0; i < depthLevel; ++i) { >>>>>> t = new Throwable("My dummy Throwable " + i, t); >>>>>> } >>>>>> return t; >>>>>> } >>>>>> >>>>>> @Benchmark >>>>>> public void benchmarkThrowableGetStackTraceString() { >>>>>> chainedThrowable.getStackTraceString(); >>>>>> } >>>>>> >>>>>> @Benchmark >>>>>> public void benchmarkThrowableGetStackTraceStringSmall() { >>>>>> simpleThrowable.getStackTraceString(); >>>>>> } >>>>>> >>>>>> @Benchmark >>>>>> public void benchmarkThrowablePrintStackTraceToStringWriter() { >>>>>> StringWriter sw = new StringWriter(); >>>>>> chainedThrowable.printStackTrace(new PrintWriter(sw)); >>>>>> sw.toString(); >>>>>> } >>>>>> >>>>>> @Benchmark >>>>>> public void benchmarkThrowablePrintStackTraceToStringWriterSmall() >>>>>> { >>>>>> StringWriter sw = new StringWriter(); >>>>>> simpleThrowable.printStackTrace(new PrintWriter(sw)); >>>>>> sw.toString(); >>>>>> } >>>>>> } >>>>>> >>>>>> And here are the results (test was run with the following options -wi >>>>>> 5 -i >>>>>> 5 -t 1): >>>>>> >>>>>> Benchmark Mode Samples Score Error Units >>>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTrac >>>>>> eString >>>>>> thrpt 50 25351.743 ? 347.326 ops/s >>>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowableGetStackTrac >>>>>> eStringSmall >>>>>> thrpt 50 134990.643 ? 760.225 ops/s >>>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>>>> aceToStringWriter >>>>>> thrpt 50 22730.786 ? 167.378 ops/s >>>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>>>> aceToStringWriterSmall >>>>>> thrpt 50 115223.025 ? 936.770 ops/s >>>>>> >>>>>> Just in any case we ran this test, actually just the last two methods, >>>>>> with >>>>>> JDK 1.9 ea build 45. We wanted to compare the result of the Apache >>>>>> commons >>>>>> solution with inner classes (as it is now) to the same, but with >>>>>> java.util.function.Consumer. There are the results with the inner >>>>>> classes: >>>>>> >>>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>>>> aceToStringWriter >>>>>> thrpt 50 22974.324 ? 391.564 ops/s >>>>>> o.m.t.ThrowableStacktraceBenchmark.benchmarkThrowablePrintStackTr >>>>>> aceToStringWriterSmall >>>>>> thrpt 50 114973.452 ? 1051.276 ops/s >>>>>> >>>>>> So you can see that the performance is nearly the same (I guess that >>>>>> the >>>>>> difference is ignorable). >>>>>> >>>>>> Please share with us the feedback about this proposal! >>>>>> >>>>>> Thanks, >>>>>> Ivan >>>>>> >>>>>> [1]http://www.oracle.com/technetwork/articles/java/index-137868.html >>>>>> >>>>>> On Wed, Dec 31, 2014 at 3:32 PM, Claes Redestad < >>>>>> claes.redestad at oracle.com> >>>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>>> seems like a reasonable convenience method. :-) >>>>>>> >>>>>>> Process stuff: OCA/JSPA signing aside, the patch needs to be >>>>>>> submitted in >>>>>>> its >>>>>>> entirety to some part of the OpenJDK infrastructure before it can be >>>>>>> accepted. >>>>>>> Attaching the patch in an e-mail to this mailing list is acceptable >>>>>>> for a >>>>>>> small >>>>>>> patch like this when lacking upload privileges to cr.openjdk.java.net. >>>>>>> A >>>>>>> sponsor >>>>>>> could help you with the uploading etc. >>>>>>> >>>>>>> This request adds a new public method, which means some internal >>>>>>> approval >>>>>>> will be necessary. This is something you'll need help with from an >>>>>>> Oracle >>>>>>> sponsor. I could volunteer to initiate the requests internally. >>>>>>> >>>>>>> Code feedback: >>>>>>> >>>>>>> The test relies on the message generated when provoking a division by >>>>>>> zero. I'm not sure this is a good idea nor the place to enforce this >>>>>>> message format with a test. Perhaps just throwing a RuntimeException >>>>>>> with some specified message and test for that? >>>>>>> >>>>>>> May I suggest to also add a test to ensure the method behaves well >>>>>>> for an >>>>>>> exception with fillInStackTrace overridden to produce stackless >>>>>>> exceptions? >>>>>>> >>>>>>> Just a thought... I noticed PrintWriter, PrintStream and StringBuilder >>>>>>> all >>>>>>> inherit from Appendable, so the PrintStreamOrWriter/StackTracePrinter >>>>>>> classes >>>>>>> could be replaced with a single StackTraceAppendable taking an >>>>>>> Appendable. >>>>>>> One >>>>>>> static class instead of 1 abstract and 3 concrete static inner classes >>>>>>> could work >>>>>>> out to be a good deal, but there's some odd mechanics in >>>>>>> BufferedWriter/PrintWriter >>>>>>> to use the value of the line.separator property at object creation >>>>>>> time >>>>>>> which >>>>>>> would be hard to support without exposing the lineSeparator fields to >>>>>>> Throwable. >>>>>>> Perhaps a future consideration. >>>>>>> >>>>>>> Performance characteristics might change ever so slightly either way, >>>>>>> so >>>>>>> it >>>>>>> would be nice with some microbenchmarks to quantify. >>>>>>> >>>>>>> /Claes >>>>>>> >>>>>>> >>>>>>> On 2014-12-31 11:51, Ivan St. Ivanov wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>>> I am Ivan from the Bulgarian Java User Group. As part of our Adopt >>>>>>>> OpenJDK >>>>>>>> efforts, we organized hackathon beginning of this month in Sofia. >>>>>>>> There >>>>>>>> we >>>>>>>> worked on the following JIRA issue: >>>>>>>> >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-5050783 >>>>>>>> >>>>>>>> Unfortunately none of us has author privileges, so the issue is still >>>>>>>> unassigned. But, nevertheless, our JUG has already signed JSPA and >>>>>>>> some >>>>>>>> of >>>>>>>> our members (including myself) have signed OCA. And we informed in a >>>>>>>> special email this mailing list about our plans prior to the meeting. >>>>>>>> >>>>>>>> Our hackathon was already documented by our JUG member Mihail >>>>>>>> Stoynov: >>>>>>>> >>>>>>>> http://mihail.stoynov.com/2014/12/12/building-openjdk- >>>>>>>> and-submitting-a-solution-on-a-feature-request-with-the- >>>>>>>> bulgarian-java-user-group/ >>>>>>>> >>>>>>>> We used the days after the meeting for offline polishing our >>>>>>>> contribution >>>>>>>> and adding some more unit tests. Here is the result of our work: >>>>>>>> >>>>>>>> http://dmitryalexandrov.net/~bgjug/5050783/webrev.00/ >>>>>>>> >>>>>>>> We would be very happy if you are able to review it, share your >>>>>>>> feedback >>>>>>>> and guide us through the contribution process as this is our first >>>>>>>> attempt. >>>>>>>> >>>>>>>> Thanks and have a very successful year! >>>>>>>> Ivan >>>>>>>> >>>>>>>> >>>>>>>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Bulgarian Java Users Group" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email tobg-jug+unsubscribe at googlegroups.com. >> To post to this group, send email tobg-jug at googlegroups.com. >> Visit this group athttp://groups.google.com/group/bg-jug. >> For more options, visithttps://groups.google.com/d/optout. >> From amy.lu at oracle.com Mon Jan 19 06:31:01 2015 From: amy.lu at oracle.com (Amy Lu) Date: Mon, 19 Jan 2015 14:31:01 +0800 Subject: JDK 9 RFR of JDK-8069255: Suppress deprecation warnings in jdk.rmic module Message-ID: <54BCA4A5.30202@oracle.com> Please review these changes to address a few stray deprecation warnings in the jdk.rmic module: JDK-8069255: Suppress deprecation warnings in jdk.rmic module http://cr.openjdk.java.net/~amlu/8069255/webrev.00/ I also need a sponsor for this. Thanks, Amy From andreas.lundblad at oracle.com Mon Jan 19 08:32:03 2015 From: andreas.lundblad at oracle.com (Andreas Lundblad) Date: Mon, 19 Jan 2015 09:32:03 +0100 Subject: JDK 9 RFR of 8067669: Documentation for methods in Number incomplete regarding too large values. In-Reply-To: <46FD3812-626A-40AE-8305-10D6B1CA7B4C@oracle.com> References: <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com> <5494B41B.8060003@oracle.com> <0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com> <3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com> <54AB3DBE.5040806@oracle.com> <20150107080837.GB2073@e6430> <54ADC94E.4090607@oracle.com> <20150108151355.GA11334@e6430> <46FD3812-626A-40AE-8305-10D6B1CA7B4C@oracle.com> Message-ID: <20150119083203.GA8241@e6430> On Fri, Jan 16, 2015 at 02:55:28PM -0800, Brian Burkhalter wrote: > > On Jan 8, 2015, at 7:13 AM, Andreas Lundblad wrote: > > > Although I was the one who brought it up, I think the whole Comparator discussion is slightly off topic. I still think that explicitly mentioning rounding and truncation is a bit confusing, as even the JDK implementations resort to other methods. > > I don?t think we quite agreed on the resolution of this issue. I don?t have a strong opinion about it, but I do think it should be resolved either as Fixed or Not An Issue. > > One alternative which removes the seemingly contentious verbiage is here: > > http://cr.openjdk.java.net/~bpb/8067669/webrev.01/ > > Note that the change at line 40 should be made even if the other diffs are rejected. This patch is an improvement in my opinion since it does not indicate that any effort is made to round or truncate the number. The obvious place to look for further documentation would be in the implementing class. I am however still in favour of making this explicit by adding "The specific semantics of the conversion is defined by the subclass in question." which is also in line with the class level documentation. -- Andreas From amy.lu at oracle.com Mon Jan 19 09:34:20 2015 From: amy.lu at oracle.com (Amy Lu) Date: Mon, 19 Jan 2015 17:34:20 +0800 Subject: JDK 9 RFR of JDK-8069262: Doclint regression in java.nio.channels.Channels Message-ID: <54BCCF9C.2080209@oracle.com> Please review the fix for Doclint regression in java.nio.channels.Channels, removed the empty

tag. bug: https://bugs.openjdk.java.net/browse/JDK-8069262 webrev: http://cr.openjdk.java.net/~amlu/8069262/webrev.00/ I also need a sponsor for this. Thanks, Amy From peter.levart at gmail.com Mon Jan 19 10:02:34 2015 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 19 Jan 2015 11:02:34 +0100 Subject: JDK 9 RFR of JDK-8068948: Update java.base module to use new try-with-resources statement In-Reply-To: <54B94E7D.10400@oracle.com> References: <54B5D8A3.1020000@oracle.com> <54B62086.8060200@oracle.com> <54B69975.3010802@oracle.com> <54B94E7D.10400@oracle.com> Message-ID: <54BCD63A.90305@gmail.com> On 01/16/2015 06:46 PM, Ivan Gerasimov wrote: > >>> Might it also make sense to allow anonymous variables in the >>> try-with-resource statement? >>> So that something like `try (() -> System.out.println("closed")) {}` >>> would work... >> >> I don't think that is necessary. In JDK 7, we started out allowing a >> general AutoCloseable expression in a try-with-resources statement >> and that proved problematic. Supporting a final / effectively final >> variable seems to be the right balance. The rationale is written up >> in the JSR 334 materials. >> > > Yes, I agree. And it can easily be moved to the finally block, of course. > However, this: > > ExecutorService executor = ...; > try (executor::shutdown) { > } > > still looks a bit more attractive to me, comparing to `try (Closeable > c = executor::shutdown) {` > Maybe one day ... :-) > > Sincerely yours, > Ivan > If try-with resources was designed after lambdas, various other forms could be considered. For example, the following: try (SomeType var = expression; expression with (target) type Consumer) { ... } The example with ExecutorService would then read: try (ExecutorService executor = Executors.newCachedThreadPool(); ExecutorService::shutdown) { ... use executor ... } This could even be done in the future as the 2nd form of try-with-resources. Maybe one day ... :-) Peter From felix.yang at oracle.com Mon Jan 19 10:39:10 2015 From: felix.yang at oracle.com (FELIX YANG) Date: Mon, 19 Jan 2015 18:39:10 +0800 Subject: RFR 8061448, Update sun/misc/JarIndex tests to cleanup the check for the "jre" directory Message-ID: <54BCDECE.1050902@oracle.com> Hi all, please review the fix for 8061448. It cleans up dependency with "jre" directory and tools.jar, which have been removed by Jigsaw feature. Bug: https://bugs.openjdk.java.net/browse/JDK-8061448 Webrev: http://cr.openjdk.java.net/~xiaofeya/8061448/webrev.00/ I needs a sponsor for this change. Thanks, Felix From aleksej.efimov at oracle.com Mon Jan 19 11:25:42 2015 From: aleksej.efimov at oracle.com (Aleksej Efimov) Date: Mon, 19 Jan 2015 14:25:42 +0300 Subject: RFR: 8062923: XSL: Run-time internal error in 'substring()' Message-ID: <54BCE9B6.7090506@oracle.com> Hi, Please, review the fix for the XSL substring function failures. Two issues were reported during usage of this function: 1. Runtime internal error with negative length: https://bugs.openjdk.java.net/browse/JDK-8062923 2. Wrong answer when -Inf length is used: https://bugs.openjdk.java.net/browse/JDK-8062924 These two issues can be solved by fix proposed for the review: http://cr.openjdk.java.net/~aefimov/8062923/webrev.00 Testing: xml related regression tests (with new test) - no failures (from jdk and jaxp repos). JCK xml related tests - no failures. With Best Regards, Aleksej From aleksej.efimov at oracle.com Mon Jan 19 12:26:25 2015 From: aleksej.efimov at oracle.com (Aleksej Efimov) Date: Mon, 19 Jan 2015 15:26:25 +0300 Subject: RFR: 8062518: AIOBE occurs when accessing to document function in extended function in JAXP Message-ID: <54BCF7F1.3010404@oracle.com> Please, review the fix for the failure observed while accessing external document during xsl transformation: https://bugs.openjdk.java.net/browse/JDK-8062518 The jaxp code limits the access to external documents, but there is a possibility to access such documents from the xsl code - new regression test checks this functionality. The fix removes this restriction and AIOBE: http://cr.openjdk.java.net/~aefimov/8062518/9/webrev.00/ Testing: xml related regression tests (with new test) - no failures (from jdk and jaxp repos). JCK xml related tests - no failures. With Best Regards, Aleksej From aleksej.efimov at oracle.com Mon Jan 19 16:48:30 2015 From: aleksej.efimov at oracle.com (Aleksej Efimov) Date: Mon, 19 Jan 2015 19:48:30 +0300 Subject: RFR: 8046817: JDK 8 schemagen tool does not generate xsd files for enum types In-Reply-To: <543E55A1.8060903@oracle.com> References: <5434FF2D.104@oracle.com> <543CD0E0.7060109@oracle.com> <543E55A1.8060903@oracle.com> Message-ID: <54BD355E.6070707@oracle.com> Hi, Returning back to this issue: The fix will be integrated soon to standalone JAXB project. Local build of standalone JAXB succeeded. No issues with jaxb tests were observed with applied fix. Can I ask reviewers to look at this issue now? Thank you and Best Regards, Aleksej On 10/15/2014 03:08 PM, Miroslav Kos wrote: > Hi Aleksej, > > the change looks ok to me, but it would be good to integrate it into > upstream (standalone JAX-B RI project) first to be on safe side - > there are much more tests then in OpenJDK. > > Copying Iaroslav who is JAX-B RI lead and can help you with > integration and running tests. > > Thanks > Miran > > > On 14/10/14 09:29, Aleksej Efimov wrote: >> Hi XML experts, >> Can I humbly ask to review this simple fix. >> >> Thank you, >> Aleksej >> >> On 08.10.2014 13:09, Aleksej Efimov wrote: >>> Hello, >>> >>> Please, review the fix [1] for the 8046817 [2]. >>> Problem: schemagen tool doesn't generate schema file for the enum >>> types [3]. >>> SchemaGenerator class gets a list of annotated elements and filters >>> out only the class variables, but annotation parser >>> (AnnotationParser.java) filters out the Class and Enum. The proposed >>> fix solves the problem: SchemaGenerator filters like AnnotationParser. >>> Testing: JPRT build and test passes on platforms (with new >>> regression test). JTREG tests: javax/xml - no failures. >>> >>> Thanks, >>> Aleksej >>> >>> [1] Webrev: http://cr.openjdk.java.net/~aefimov/8046817/9/webrev.00/ >>> [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8046817 >>> [3] Enum type: >>> http://cr.openjdk.java.net/~aefimov/8046817/9/webrev.00/jdk/test/javax/xml/ws/8046817/TestEnumType.java.html >> > From vladimir.x.ivanov at oracle.com Mon Jan 19 17:05:49 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Mon, 19 Jan 2015 20:05:49 +0300 Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when GWT LambdaForms are shared In-Reply-To: <54B975EA.6040005@oracle.com> References: <54B94766.2080102@oracle.com> <54B975EA.6040005@oracle.com> Message-ID: <54BD396D.2050907@oracle.com> Thanks, Vladimir! > I would suggest to add more detailed comment (instead of simple "Stop > profiling") to inline_profileBranch() intrinsic explaining what it is > doing because it is not strictly "intrinsic" - it does not implement > profileBranch() java code when counts is constant. Sure, will do. > You forgot to mark Opaque4Node as macro node. I would suggest to base it > on Opaque2Node then you will get some methods from it. Do I really need to do so? I expect it to go away during IGVN pass right after parsing is over. That's why I register the node for igvn in LibraryCallKit::inline_profileBranch(). Changes in macro.cpp & compile.cpp are leftovers from the version when Opaque4 was macro node. I plan to remove them. Best regards, Vladimir Ivanov > On 1/16/15 9:16 AM, Vladimir Ivanov wrote: >> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/hotspot/ >> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/jdk/ >> https://bugs.openjdk.java.net/browse/JDK-8063137 >> >> After GuardWithTest (GWT) LambdaForms became shared, profile pollution >> significantly distorted compilation decisions. It affected inlining and >> hindered some optimizations. It causes significant performance >> regressions for Nashorn (on Octane benchmarks). >> >> Inlining was fixed by 8059877 [1], but it didn't cover the case when a >> branch is never taken. It can cause missed optimization opportunity, and >> not just increase in code size. For example, non-pruned branch can break >> escape analysis. >> >> Currently, there are 2 problems: >> - branch frequencies profile pollution >> - deoptimization counts pollution >> >> Branch frequency pollution hides from JIT the fact that a branch is >> never taken. Since GWT LambdaForms (and hence their bytecode) are >> heavily shared, but the behavior is specific to MethodHandle, there's no >> way for JIT to understand how particular GWT instance behaves. >> >> The solution I propose is to do profiling in Java code and feed it to >> JIT. Every GWT MethodHandle holds an auxiliary array (int[2]) where >> profiling info is stored. Once JIT kicks in, it can retrieve these >> counts, if corresponding MethodHandle is a compile-time constant (and it >> is usually the case). To communicate the profile data from Java code to >> JIT, MethodHandleImpl::profileBranch() is used. >> >> If GWT MethodHandle isn't a compile-time constant, profiling should >> proceed. It happens when corresponding LambdaForm is already shared, for >> newly created GWT MethodHandles profiling can occur only in native code >> (dedicated nmethod for a single LambdaForm). So, when compilation of the >> whole MethodHandle chain is triggered, the profile should be already >> gathered. >> >> Overriding branch frequencies is not enough. Statistics on >> deoptimization events is also polluted. Even if a branch is never taken, >> JIT doesn't issue an uncommon trap there unless corresponding bytecode >> doesn't trap too much and doesn't cause too many recompiles. >> >> I added @IgnoreProfile and place it only on GWT LambdaForms. When JIT >> sees it on some method, Compile::too_many_traps & >> Compile::too_many_recompiles for that method always return false. It >> allows JIT to prune the branch based on custom profile and recompile the >> method, if the branch is visited. >> >> For now, I wanted to keep the fix very focused. The next thing I plan to >> do is to experiment with ignoring deoptimization counts for other >> LambdaForms which are heavily shared. I already saw problems caused by >> deoptimization counts pollution (see JDK-8068915 [2]). >> >> I plan to backport the fix into 8u40, once I finish extensive >> performance testing. >> >> Testing: JPRT, java/lang/invoke tests, nashorn (nashorn testsuite, >> Octane). >> >> Thanks! >> >> PS: as a summary, my experiments show that fixes for 8063137 & 8068915 >> [2] almost completely recovers peak performance after LambdaForm sharing >> [3]. There's one more problem left (non-inlined MethodHandle invocations >> are more expensive when LFs are shared), but it's a story for another >> day. >> >> Best regards, >> Vladimir Ivanov >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8059877 >> 8059877: GWT branch frequencies pollution due to LF sharing >> [2] https://bugs.openjdk.java.net/browse/JDK-8068915 >> [3] https://bugs.openjdk.java.net/browse/JDK-8046703 >> JEP 210: LambdaForm Reduction and Caching >> _______________________________________________ >> mlvm-dev mailing list >> mlvm-dev at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev > _______________________________________________ > mlvm-dev mailing list > mlvm-dev at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev From paul.sandoz at oracle.com Mon Jan 19 17:46:22 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 19 Jan 2015 17:46:22 +0000 Subject: RFR 8068432: Inconsistent exception handling in CompletableFuture.thenCompose Message-ID: <4D0311DB-C957-491A-9102-9DD539F10322@oracle.com> Hi, http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8068432-CompletableFuture-thenCompose-Exception/webrev/ This is a cherry-picked fix from the 166 repo with a JDK test added. Once approved i will request a back-port to 8u60. Paul. From kissziszi at gmail.com Mon Jan 19 19:13:24 2015 From: kissziszi at gmail.com (Zoltan Sziladi) Date: Mon, 19 Jan 2015 11:13:24 -0800 Subject: String.indexOf optimization In-Reply-To: References: <534D68B8.3040002@oracle.com> <5350E316.2050406@oracle.com> <54ABAAC2.2090502@redhat.com> Message-ID: Martin, yes, we're talking about that method. Other than tightening that code some... if we find some algorithm that outperforms the naive implementation under certain conditions (let's say when the searched pattern is longer than 10 characters), would it be worth including it as a special case? (For example naive would run normally but if pattern is longer than 10 characters then the other algorithm would run). Or do you think that would just make the code too complicated without enough benefits? Regards, Zoltan On Mon, Jan 12, 2015 at 12:11 PM, Martin Buchholz wrote: > If there's a clean improvement in the java code, it's worth putting in. > You can try benchmarking with -Xint. > Are we talking about this method? > > static int indexOf(char[] source, int sourceOffset, int sourceCount, > char[] target, int targetOffset, int targetCount, > int fromIndex) { > > It does look like we can tighten the code up a little... > > > On Thu, Jan 8, 2015 at 3:05 PM, Zoltan Sziladi > wrote: > >> Thanks for the info. >> So that basically means we have 2 implementations of indexOf currently, >> one >> is in HotSpot, the other is in the JDK itself, which rarely gets executed. >> Aside from this later fact, isn't it still worth improving the JDK >> implementation if it is possible? I know that the intrinsified method gets >> executed most of the time, but still, if we can improve the JDK >> implementation also, why not? I don't know much about other JVMs but maybe >> a few don't intrinsify it? >> >> Is there any existing test suite which is considered conclusive enough >> that >> if an implementation beats the naive algorithm in those testcases then it >> could be considered as a replacement in the JDK? >> >> Regards, >> Zoltan >> >> >> On Thu, Jan 8, 2015 at 12:42 PM, Vitaly Davidovich >> wrote: >> >> > The java impl you saw would be called by (a) interpreter, (b) if you >> > explicitly disable intrinsification of this function, or (c) some other >> JVM >> > that doesn't intrinsify this method (or any method). >> > >> > People don't usually disable intrinsics; if they do, it's because they >> hit >> > some JIT bug and may disable it. >> > >> > On Thu, Jan 8, 2015 at 3:34 PM, Zoltan Sziladi >> > wrote: >> > >> >> Hi, >> >> >> >> Thanks everyone for all the info. >> >> So, just to go step by step in understanding this. >> >> Andrew said HotSpot would ignore my implementation. So why is there an >> >> implementation of indexOf at all in the JDK, if that's not the code >> that's >> >> executed? Is it just a default fallback? When is the indexOf function >> not >> >> intrinsified? When do people usually disable intrinsification? >> >> Sorry if these are newbie questions, I'm new to this part of Java. >> >> >> >> Regards, >> >> Zoltan >> >> >> >> On Tue, Jan 6, 2015 at 1:28 AM, Andrew Haley wrote: >> >> >> >> > Hi, >> >> > >> >> > On 05/01/15 18:59, Zoltan Sziladi wrote: >> >> > >> >> > > This discussion was a long time ago, I was just reading through it >> to >> >> > check >> >> > > again what was the last state of the discussion about the >> >> String.indexOf. >> >> > > There is one part which I still do not understand, hopefully >> someone >> >> > could >> >> > > shed some light on it. A few emails ago Martin mentioned >> >> > > >> >> > > "Hotspot seems to have some intrinsification of String.indexOf, >> which >> >> > > confuses me. >> >> > > Hotspot seems the right place to provide more optimizations for >> this, >> >> > since >> >> > > there has been a fair amount of work creating high-performance >> >> low-level >> >> > > implementations of this idea in C." >> >> > > >> >> > > Then Ivan asked what that actually meant, whether hotspot actually >> >> > replaced >> >> > > the jdk implementation with a low level optimized C implementation, >> >> but I >> >> > > never saw an answer to that. >> >> > >> >> > You can have a look at an implementation of >> >> MacroAssembler::string_indexof >> >> > in >> >> > >> >> > >> >> > >> >> >> http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b6b89b8f8531/src/cpu/x86/vm/macroAssembler_x86.cpp >> >> > >> >> > > Can someone please explain this? If we somehow found an algorithm >> that >> >> > beat >> >> > > the naive implementation in the average case, would it be possible >> to >> >> > just >> >> > > implement it in the JDK? >> >> > >> >> > No, because HotSpot would ignore it. Any speed improvements have to >> be >> >> > done in the architecture-dependent files. >> >> > >> >> > Andrew. >> >> > >> >> > >> >> >> > >> > >> > > From chris.hegarty at oracle.com Tue Jan 20 09:11:33 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 20 Jan 2015 09:11:33 +0000 Subject: RFR 8068432: Inconsistent exception handling in CompletableFuture.thenCompose In-Reply-To: <4D0311DB-C957-491A-9102-9DD539F10322@oracle.com> References: <4D0311DB-C957-491A-9102-9DD539F10322@oracle.com> Message-ID: Looks fine to me PAul. -Chris. On 19 Jan 2015, at 17:46, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8068432-CompletableFuture-thenCompose-Exception/webrev/ > > This is a cherry-picked fix from the 166 repo with a JDK test added. Once approved i will request a back-port to 8u60. > > Paul. From lange.fabian at gmail.com Tue Jan 20 09:49:12 2015 From: lange.fabian at gmail.com (Fabian Lange) Date: Tue, 20 Jan 2015 10:49:12 +0100 Subject: Support for StringBuilder / Appendable in java.text APIs Message-ID: Hi all, not sure if this is the appropriate audience, but I noticed that when I have a StringBuilder, I cannot use any of the optimized format() methods in the java.text APIs (NumberFormat, DateFormat, ChoiceFormat etc) All those methods only accept a StringBuffer. The obvious change would be to make them accept "Appendable", but that would break to much reflective use of them, wouldn't it? Could an API be added which accepts Appendable? Or is that effort not worth the benefit of using StringBuilder over StringBuffer? Well the JIT might be able to eliminate the locking overhead, however, there is a lot of code out there using StringBuilders, and I would guess they should be able to use the java.text APIs? The sun misc internal APIs support Appendable already (like FloatingDecimal.appendTo) I searched the JDK Bug System and mailing list for this topic, but could not find a suitable hit. If there is such a discussion somewhere, let me know and I will weight in there. Fabian From chris.hegarty at oracle.com Tue Jan 20 11:03:50 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 20 Jan 2015 11:03:50 +0000 Subject: RFR 8061448, Update sun/misc/JarIndex tests to cleanup the check for the "jre" directory In-Reply-To: <54BCDECE.1050902@oracle.com> References: <54BCDECE.1050902@oracle.com> Message-ID: <65A61E8F-294C-4D42-B807-2CDC7ADF12CA@oracle.com> On 19 Jan 2015, at 10:39, FELIX YANG wrote: > Hi all, > please review the fix for 8061448. It cleans up dependency with "jre" directory and tools.jar, which have been removed by Jigsaw feature. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8061448 > Webrev: > http://cr.openjdk.java.net/~xiaofeya/8061448/webrev.00/ Looks ok to me Felix. > I needs a sponsor for this change. I can sponsor this for you. -Chris. > Thanks, > Felix From vladimir.x.ivanov at oracle.com Tue Jan 20 12:40:50 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Tue, 20 Jan 2015 15:40:50 +0300 Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when GWT LambdaForms are shared In-Reply-To: References: <54B94766.2080102@oracle.com> Message-ID: <54BE4CD2.30805@oracle.com> Duncan, thanks a lot for giving it a try! If you plan to spend more time on it, please, apply 8068915 as well. I saw huge intermittent performance regressions due to continuous deoptimization storm. You can look into -XX:+LogCompilation output and look for repeated deoptimization events in steady state w/ Action_none. Also, there's deoptimization statistics in the log (at least, in jdk9). It's located right before compilation_log tag. Thanks again for the valuable feedback! Best regards, Vladimir Ivanov [1] http://cr.openjdk.java.net/~vlivanov/8068915/webrev.00 On 1/19/15 11:21 PM, MacGregor, Duncan (GE Energy Management) wrote: > Okay, I?ve done some tests of this with the micro benchmarks for our > language & runtime which show pretty much no change except for one test > which is now almost 3x slower. It uses nested loops to iterate over an > array and concatenate the string-like objects it contains, and replaces > elements with these new longer string-llike objects. It?s a bit of a > pathological case, and I haven?t seen the same sort of degradation in the > other benchmarks or in real applications, but I haven?t done serious > benchmarking of them with this change. > > I shall see if the test case can be reduced down to anything simpler while > still showing the same performance behaviour, and try add some compilation > logging options to narrow down what?s going on. > > Duncan. > > On 16/01/2015 17:16, "Vladimir Ivanov" > wrote: > >> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/hotspot/ >> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/jdk/ >> https://bugs.openjdk.java.net/browse/JDK-8063137 >> >> After GuardWithTest (GWT) LambdaForms became shared, profile pollution >> significantly distorted compilation decisions. It affected inlining and >> hindered some optimizations. It causes significant performance >> regressions for Nashorn (on Octane benchmarks). >> >> Inlining was fixed by 8059877 [1], but it didn't cover the case when a >> branch is never taken. It can cause missed optimization opportunity, and >> not just increase in code size. For example, non-pruned branch can break >> escape analysis. >> >> Currently, there are 2 problems: >> - branch frequencies profile pollution >> - deoptimization counts pollution >> >> Branch frequency pollution hides from JIT the fact that a branch is >> never taken. Since GWT LambdaForms (and hence their bytecode) are >> heavily shared, but the behavior is specific to MethodHandle, there's no >> way for JIT to understand how particular GWT instance behaves. >> >> The solution I propose is to do profiling in Java code and feed it to >> JIT. Every GWT MethodHandle holds an auxiliary array (int[2]) where >> profiling info is stored. Once JIT kicks in, it can retrieve these >> counts, if corresponding MethodHandle is a compile-time constant (and it >> is usually the case). To communicate the profile data from Java code to >> JIT, MethodHandleImpl::profileBranch() is used. >> >> If GWT MethodHandle isn't a compile-time constant, profiling should >> proceed. It happens when corresponding LambdaForm is already shared, for >> newly created GWT MethodHandles profiling can occur only in native code >> (dedicated nmethod for a single LambdaForm). So, when compilation of the >> whole MethodHandle chain is triggered, the profile should be already >> gathered. >> >> Overriding branch frequencies is not enough. Statistics on >> deoptimization events is also polluted. Even if a branch is never taken, >> JIT doesn't issue an uncommon trap there unless corresponding bytecode >> doesn't trap too much and doesn't cause too many recompiles. >> >> I added @IgnoreProfile and place it only on GWT LambdaForms. When JIT >> sees it on some method, Compile::too_many_traps & >> Compile::too_many_recompiles for that method always return false. It >> allows JIT to prune the branch based on custom profile and recompile the >> method, if the branch is visited. >> >> For now, I wanted to keep the fix very focused. The next thing I plan to >> do is to experiment with ignoring deoptimization counts for other >> LambdaForms which are heavily shared. I already saw problems caused by >> deoptimization counts pollution (see JDK-8068915 [2]). >> >> I plan to backport the fix into 8u40, once I finish extensive >> performance testing. >> >> Testing: JPRT, java/lang/invoke tests, nashorn (nashorn testsuite, >> Octane). >> >> Thanks! >> >> PS: as a summary, my experiments show that fixes for 8063137 & 8068915 >> [2] almost completely recovers peak performance after LambdaForm sharing >> [3]. There's one more problem left (non-inlined MethodHandle invocations >> are more expensive when LFs are shared), but it's a story for another day. >> >> Best regards, >> Vladimir Ivanov >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8059877 >> 8059877: GWT branch frequencies pollution due to LF sharing >> [2] https://bugs.openjdk.java.net/browse/JDK-8068915 >> [3] https://bugs.openjdk.java.net/browse/JDK-8046703 >> JEP 210: LambdaForm Reduction and Caching >> _______________________________________________ >> mlvm-dev mailing list >> mlvm-dev at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev > > _______________________________________________ > mlvm-dev mailing list > mlvm-dev at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev > From paul.sandoz at oracle.com Tue Jan 20 14:50:20 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Jan 2015 14:50:20 +0000 Subject: 8062901: Iterators is spelled incorrectly in the Javadoc for Spliterator Message-ID: <84940F7B-25E2-4C43-922E-8695D8FC781A@oracle.com> Hi, See the patch below that fixes a type on the Spliterator docs. Thanks, Paul. diff -r 1daa66d4a028 src/java.base/share/classes/java/util/Spliterator.java --- a/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 14:48:55 2015 +0000 +++ b/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 14:49:38 2015 +0000 @@ -125,7 +125,7 @@ * are encountered. * * @apiNote - *

Spliterators, like {@code Iterators}s, are for traversing the elements of + *

Spliterators, like {@code Iterator}s, are for traversing the elements of * a source. The {@code Spliterator} API was designed to support efficient * parallel traversal in addition to sequential traversal, by supporting * decomposition as well as single-element iteration. In addition, the From lance.andersen at oracle.com Tue Jan 20 14:51:55 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Tue, 20 Jan 2015 09:51:55 -0500 Subject: 8062901: Iterators is spelled incorrectly in the Javadoc for Spliterator In-Reply-To: <84940F7B-25E2-4C43-922E-8695D8FC781A@oracle.com> References: <84940F7B-25E2-4C43-922E-8695D8FC781A@oracle.com> Message-ID: +1 On Jan 20, 2015, at 9:50 AM, Paul Sandoz wrote: > Hi, > > See the patch below that fixes a type on the Spliterator docs. > > Thanks, > Paul. > > diff -r 1daa66d4a028 src/java.base/share/classes/java/util/Spliterator.java > --- a/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 14:48:55 2015 +0000 > +++ b/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 14:49:38 2015 +0000 > @@ -125,7 +125,7 @@ > * are encountered. > * > * @apiNote > - *

Spliterators, like {@code Iterators}s, are for traversing the elements of > + *

Spliterators, like {@code Iterator}s, are for traversing the elements of > * a source. The {@code Spliterator} API was designed to support efficient > * parallel traversal in addition to sequential traversal, by supporting > * decomposition as well as single-element iteration. In addition, the > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From chris.hegarty at oracle.com Tue Jan 20 14:51:49 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 20 Jan 2015 14:51:49 +0000 Subject: 8062901: Iterators is spelled incorrectly in the Javadoc for Spliterator In-Reply-To: <84940F7B-25E2-4C43-922E-8695D8FC781A@oracle.com> References: <84940F7B-25E2-4C43-922E-8695D8FC781A@oracle.com> Message-ID: <6FDB2A3D-6F4F-4A6C-B032-9E02535AB2FB@oracle.com> Reviewed. -Chris. On 20 Jan 2015, at 14:50, Paul Sandoz wrote: > Hi, > > See the patch below that fixes a type on the Spliterator docs. > > Thanks, > Paul. > > diff -r 1daa66d4a028 src/java.base/share/classes/java/util/Spliterator.java > --- a/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 14:48:55 2015 +0000 > +++ b/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 14:49:38 2015 +0000 > @@ -125,7 +125,7 @@ > * are encountered. > * > * @apiNote > - *

Spliterators, like {@code Iterators}s, are for traversing the elements of > + *

Spliterators, like {@code Iterator}s, are for traversing the elements of > * a source. The {@code Spliterator} API was designed to support efficient > * parallel traversal in addition to sequential traversal, by supporting > * decomposition as well as single-element iteration. In addition, the > From Alan.Bateman at oracle.com Tue Jan 20 14:58:51 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Jan 2015 14:58:51 +0000 Subject: 8062901: Iterators is spelled incorrectly in the Javadoc for Spliterator In-Reply-To: <84940F7B-25E2-4C43-922E-8695D8FC781A@oracle.com> References: <84940F7B-25E2-4C43-922E-8695D8FC781A@oracle.com> Message-ID: <54BE6D2B.3000300@oracle.com> On 20/01/2015 14:50, Paul Sandoz wrote: > Hi, > > See the patch below that fixes a type on the Spliterator docs. > > Thanks, > Paul. > > diff -r 1daa66d4a028 src/java.base/share/classes/java/util/Spliterator.java > --- a/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 14:48:55 2015 +0000 > +++ b/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 14:49:38 2015 +0000 > @@ -125,7 +125,7 @@ > * are encountered. > * > * @apiNote > - *

Spliterators, like {@code Iterators}s, are for traversing the elements of > + *

Spliterators, like {@code Iterator}s, are for traversing the elements of > * a source. The {@code Spliterator} API was designed to support efficient > * parallel traversal in addition to sequential traversal, by supporting > * decomposition as well as single-element iteration. In addition, the > Looks okay. From lance.andersen at oracle.com Tue Jan 20 15:02:05 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Tue, 20 Jan 2015 10:02:05 -0500 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: <54B9C9EA.7050407@oracle.com> References: <54934601.4010104@oracle.com> <54B96BDF.7090204@oracle.com> <54B9C9EA.7050407@oracle.com> Message-ID: <5A817A94-0176-4613-91F2-9B298C1DA6AA@oracle.com> Hi Joe, I think the changes look fine. I am wondering if we have any suggested standard for the use of @Override as I see it is inconsistent in its usage with methods implementing an interface. Is this something we should add to existing code? I don't see Netbeans asking for it to be done except in the case of an overridden method? Thank you for the extra clean up Joe Best Lance On Jan 16, 2015, at 9:33 PM, huizhe wang wrote: > > On 1/16/2015 1:29 PM, Lance Andersen wrote: >> Hi Joe, >> >> >> >> Overall it is OK, a few minor comments >> >> - Is there a reason that XPathExpressionImpl is no longer public and XPathImpl is public? > > Ok, I'll keep both public, may be useful in the future. > >> - I think you can leverage {@inheritdoc} in your impl classes to avoid comment duplication possibly? > > Looks like javadoc "Automatically inherit comment ". So @Override is good enough. I've removed the javadocs for methods that override, including the existing methods. It's good to avoid the duplication, and potential copy n paste errors. > >> - please remember to make the copyright year 2015 > > Done. > >> - XPathTestBase, can the static block be moved to a @BeforeClass > > Moved to within the Dataprovider. > >> - XPathNodes, should that have an @since 1.9? > > Fixed. > >> - Given you are not including @param, etc for your test comments, you might want to consider /* */ vs /** */ style comments. > > Looks like s/\/**/\/* worked the trick. > >> That is consider if you really want doc comments (really a style choice but some IDEs will issue a warning for missing tags > > Yeh, it's good to turn off the warning "light" :-) > > http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ > > Thanks, > Joe > >> >> >> HTH >> >> Best, >> Lance >> On Jan 16, 2015, at 2:51 PM, huizhe wang wrote: >> >>> Hi all, >>> >>> Could you review the change? >>> >>> Thanks, >>> Joe >>> >>> On 12/18/2014 1:24 PM, huizhe wang wrote: >>>> Hi, >>>> >>>> This is to add support for any type and improvement with new features reflected in the new evaluateExpression methods, XPathEvaluationResult and XPathNodes. >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8054196 >>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>> >>>> Thanks, >>>> Joe >>> >> >> >> >> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> >> >> > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From paul.sandoz at oracle.com Tue Jan 20 16:17:47 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Jan 2015 16:17:47 +0000 Subject: 8069325: Pattern.splitAsStream does not return input if it is empty and there is no match Message-ID: <722C43ED-09C0-4C77-9C6F-98A05D1134EE@oracle.com> Hi, http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8069325-Pattern-splitAsStream-emptyInput/webrev/ This patch fixes an edge case in Pattern.splitAsStream for matching against an empty input string, which deviated from the behaviour of Pattern.split. When there are no matches a stream containing the input string should be returned rather than an empty stream. -- I have kept compatibility with Pattern.split(String ) but i noticed another an edge case. What should the following return: Pattern.compile("").split("") [] or [""]? There is a zero-width match at the beginning and an empty remaining segment both of which should be discarded, as such i would expect the result to be [] rather than as [""], as currently produced result. If people agree that this is an issue i suggest we log a new one independent of fixing 8069325. Thanks, Paul. From paul.sandoz at oracle.com Tue Jan 20 16:27:03 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Jan 2015 16:27:03 +0000 Subject: RFR 8032513: The Spliterator characteristics CONCURRENT and IMMUTABLE are mutually exclusive Message-ID: <64FDDDBA-449B-4C5D-86DD-926F73FF5747@oracle.com> Hi http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8032513-Spliterator-immutable-concurrent/webrev/ This is really just a clarification making what was implicitly obvious explicitly so. I will file a CCC. Paul. diff -r ff20291021b3 src/java.base/share/classes/java/util/Spliterator.java --- a/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 16:19:16 2015 +0000 +++ b/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 16:23:23 2015 +0000 @@ -553,6 +553,12 @@ * sub-split size is known and additions or removals to the source are not * reflected when traversing. * + *

A top-level Spliterator should not report both {@code CONCURRENT} and + * {@code IMMUTABLE}, since they are mutually exclusive. Such a Spliterator + * is inconsistent and no guarantees can be made about any computation using + * that Spliterator. Sub-spliterators may report {@code IMMUTABLE} if + * additions or removals to the source are not reflected when traversing. + * * @apiNote Most concurrent collections maintain a consistency policy * guaranteeing accuracy with respect to elements present at the point of * Spliterator construction, but possibly not reflecting subsequent From chris.hegarty at oracle.com Tue Jan 20 16:36:11 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 20 Jan 2015 16:36:11 +0000 Subject: RFR 8032513: The Spliterator characteristics CONCURRENT and IMMUTABLE are mutually exclusive In-Reply-To: <64FDDDBA-449B-4C5D-86DD-926F73FF5747@oracle.com> References: <64FDDDBA-449B-4C5D-86DD-926F73FF5747@oracle.com> Message-ID: <7050F08F-9073-4B50-84E9-D87BF8D5DA4B@oracle.com> Looks fine to me Paul. -Chris. On 20 Jan 2015, at 16:27, Paul Sandoz wrote: > Hi > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8032513-Spliterator-immutable-concurrent/webrev/ > > This is really just a clarification making what was implicitly obvious explicitly so. > > I will file a CCC. > > Paul. > > diff -r ff20291021b3 src/java.base/share/classes/java/util/Spliterator.java > --- a/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 16:19:16 2015 +0000 > +++ b/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 16:23:23 2015 +0000 > @@ -553,6 +553,12 @@ > * sub-split size is known and additions or removals to the source are not > * reflected when traversing. > * > + *

A top-level Spliterator should not report both {@code CONCURRENT} and > + * {@code IMMUTABLE}, since they are mutually exclusive. Such a Spliterator > + * is inconsistent and no guarantees can be made about any computation using > + * that Spliterator. Sub-spliterators may report {@code IMMUTABLE} if > + * additions or removals to the source are not reflected when traversing. > + * > * @apiNote Most concurrent collections maintain a consistency policy > * guaranteeing accuracy with respect to elements present at the point of > * Spliterator construction, but possibly not reflecting subsequent > From lance.andersen at oracle.com Tue Jan 20 16:37:58 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Tue, 20 Jan 2015 11:37:58 -0500 Subject: RFR 8032513: The Spliterator characteristics CONCURRENT and IMMUTABLE are mutually exclusive In-Reply-To: <64FDDDBA-449B-4C5D-86DD-926F73FF5747@oracle.com> References: <64FDDDBA-449B-4C5D-86DD-926F73FF5747@oracle.com> Message-ID: <6B3B714F-A45D-49E8-B273-915361AC502D@oracle.com> Looks OK assuming the CCC is approved On Jan 20, 2015, at 11:27 AM, Paul Sandoz wrote: > Hi > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8032513-Spliterator-immutable-concurrent/webrev/ > > This is really just a clarification making what was implicitly obvious explicitly so. > > I will file a CCC. > > Paul. > > diff -r ff20291021b3 src/java.base/share/classes/java/util/Spliterator.java > --- a/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 16:19:16 2015 +0000 > +++ b/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 16:23:23 2015 +0000 > @@ -553,6 +553,12 @@ > * sub-split size is known and additions or removals to the source are not > * reflected when traversing. > * > + *

A top-level Spliterator should not report both {@code CONCURRENT} and > + * {@code IMMUTABLE}, since they are mutually exclusive. Such a Spliterator > + * is inconsistent and no guarantees can be made about any computation using > + * that Spliterator. Sub-spliterators may report {@code IMMUTABLE} if > + * additions or removals to the source are not reflected when traversing. > + * > * @apiNote Most concurrent collections maintain a consistency policy > * guaranteeing accuracy with respect to elements present at the point of > * Spliterator construction, but possibly not reflecting subsequent > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From daniel.fuchs at oracle.com Tue Jan 20 16:38:56 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 20 Jan 2015 17:38:56 +0100 Subject: RFR 8032513: The Spliterator characteristics CONCURRENT and IMMUTABLE are mutually exclusive In-Reply-To: <64FDDDBA-449B-4C5D-86DD-926F73FF5747@oracle.com> References: <64FDDDBA-449B-4C5D-86DD-926F73FF5747@oracle.com> Message-ID: <54BE84A0.60702@oracle.com> Hi Paul, Your clarification looks good and I'm glad for it! best regards, -- daniel On 20/01/15 17:27, Paul Sandoz wrote: > Hi > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8032513-Spliterator-immutable-concurrent/webrev/ > > This is really just a clarification making what was implicitly obvious explicitly so. > > I will file a CCC. > > Paul. > > diff -r ff20291021b3 src/java.base/share/classes/java/util/Spliterator.java > --- a/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 16:19:16 2015 +0000 > +++ b/src/java.base/share/classes/java/util/Spliterator.java Tue Jan 20 16:23:23 2015 +0000 > @@ -553,6 +553,12 @@ > * sub-split size is known and additions or removals to the source are not > * reflected when traversing. > * > + *

A top-level Spliterator should not report both {@code CONCURRENT} and > + * {@code IMMUTABLE}, since they are mutually exclusive. Such a Spliterator > + * is inconsistent and no guarantees can be made about any computation using > + * that Spliterator. Sub-spliterators may report {@code IMMUTABLE} if > + * additions or removals to the source are not reflected when traversing. > + * > * @apiNote Most concurrent collections maintain a consistency policy > * guaranteeing accuracy with respect to elements present at the point of > * Spliterator construction, but possibly not reflecting subsequent > From duncan.macgregor at ge.com Mon Jan 19 20:21:59 2015 From: duncan.macgregor at ge.com (MacGregor, Duncan (GE Energy Management)) Date: Mon, 19 Jan 2015 20:21:59 +0000 Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when GWT LambdaForms are shared In-Reply-To: <54B94766.2080102@oracle.com> References: <54B94766.2080102@oracle.com> Message-ID: Okay, I?ve done some tests of this with the micro benchmarks for our language & runtime which show pretty much no change except for one test which is now almost 3x slower. It uses nested loops to iterate over an array and concatenate the string-like objects it contains, and replaces elements with these new longer string-llike objects. It?s a bit of a pathological case, and I haven?t seen the same sort of degradation in the other benchmarks or in real applications, but I haven?t done serious benchmarking of them with this change. I shall see if the test case can be reduced down to anything simpler while still showing the same performance behaviour, and try add some compilation logging options to narrow down what?s going on. Duncan. On 16/01/2015 17:16, "Vladimir Ivanov" wrote: >http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/hotspot/ >http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/jdk/ >https://bugs.openjdk.java.net/browse/JDK-8063137 > >After GuardWithTest (GWT) LambdaForms became shared, profile pollution >significantly distorted compilation decisions. It affected inlining and >hindered some optimizations. It causes significant performance >regressions for Nashorn (on Octane benchmarks). > >Inlining was fixed by 8059877 [1], but it didn't cover the case when a >branch is never taken. It can cause missed optimization opportunity, and >not just increase in code size. For example, non-pruned branch can break >escape analysis. > >Currently, there are 2 problems: > - branch frequencies profile pollution > - deoptimization counts pollution > >Branch frequency pollution hides from JIT the fact that a branch is >never taken. Since GWT LambdaForms (and hence their bytecode) are >heavily shared, but the behavior is specific to MethodHandle, there's no >way for JIT to understand how particular GWT instance behaves. > >The solution I propose is to do profiling in Java code and feed it to >JIT. Every GWT MethodHandle holds an auxiliary array (int[2]) where >profiling info is stored. Once JIT kicks in, it can retrieve these >counts, if corresponding MethodHandle is a compile-time constant (and it >is usually the case). To communicate the profile data from Java code to >JIT, MethodHandleImpl::profileBranch() is used. > >If GWT MethodHandle isn't a compile-time constant, profiling should >proceed. It happens when corresponding LambdaForm is already shared, for >newly created GWT MethodHandles profiling can occur only in native code >(dedicated nmethod for a single LambdaForm). So, when compilation of the >whole MethodHandle chain is triggered, the profile should be already >gathered. > >Overriding branch frequencies is not enough. Statistics on >deoptimization events is also polluted. Even if a branch is never taken, >JIT doesn't issue an uncommon trap there unless corresponding bytecode >doesn't trap too much and doesn't cause too many recompiles. > >I added @IgnoreProfile and place it only on GWT LambdaForms. When JIT >sees it on some method, Compile::too_many_traps & >Compile::too_many_recompiles for that method always return false. It >allows JIT to prune the branch based on custom profile and recompile the >method, if the branch is visited. > >For now, I wanted to keep the fix very focused. The next thing I plan to >do is to experiment with ignoring deoptimization counts for other >LambdaForms which are heavily shared. I already saw problems caused by >deoptimization counts pollution (see JDK-8068915 [2]). > >I plan to backport the fix into 8u40, once I finish extensive >performance testing. > >Testing: JPRT, java/lang/invoke tests, nashorn (nashorn testsuite, >Octane). > >Thanks! > >PS: as a summary, my experiments show that fixes for 8063137 & 8068915 >[2] almost completely recovers peak performance after LambdaForm sharing >[3]. There's one more problem left (non-inlined MethodHandle invocations >are more expensive when LFs are shared), but it's a story for another day. > >Best regards, >Vladimir Ivanov > >[1] https://bugs.openjdk.java.net/browse/JDK-8059877 > 8059877: GWT branch frequencies pollution due to LF sharing >[2] https://bugs.openjdk.java.net/browse/JDK-8068915 >[3] https://bugs.openjdk.java.net/browse/JDK-8046703 > JEP 210: LambdaForm Reduction and Caching >_______________________________________________ >mlvm-dev mailing list >mlvm-dev at openjdk.java.net >http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev From paul.sandoz at oracle.com Tue Jan 20 16:51:13 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Jan 2015 16:51:13 +0000 Subject: RFR 8048689: Clarify documentation on BaseStream.spliterator Message-ID: <748EFA84-3146-478D-A8FE-75641159FA7E@oracle.com> Hi, http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8048689-BaseStream-spliterator/webrev/ Some clarifications to the characteristics of the Spliterator returned from BaseStream.spliterator() (otherwise known as the "escape-hatch" spliterator). Implementations are free to report fewer characteristics than all those that could be derived from the stream pipeline. For example, the current implementation does not support the propagation of NONNULL, IMMUTABLE or CONCURRENT through the stream pipeline. It could, we have room in the bit field for characteristic/flag propagation, but we did not think it worth the effort. Paul. diff -r 66508b45ff35 src/java.base/share/classes/java/util/stream/BaseStream.java --- a/src/java.base/share/classes/java/util/stream/BaseStream.java Tue Jan 20 16:22:05 2015 +0000 +++ b/src/java.base/share/classes/java/util/stream/BaseStream.java Tue Jan 20 16:35:32 2015 +0000 @@ -79,6 +79,14 @@ *

This is a terminal * operation. * + *

+ * The returned spliterator should report the set of characteristics derived + * from the stream pipeline (namely the characteristics derived from the + * stream source spliterator and the intermediate operations). + * Implementations may report a sub-set of those characteristics. For + * example, it may be too expensive to compute the entire set for some or + * all possible stream pipelines. + * * @return the element spliterator for this stream */ Spliterator spliterator(); From paul.sandoz at oracle.com Tue Jan 20 16:56:14 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Jan 2015 16:56:14 +0000 Subject: RFR 8059324: orElseThrow has different signatures for OptionalPrimitive and Optional Message-ID: Hi, http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8059324-Optional-orElseThrow-signatures/webrev/ The patch fixes up the signatures of Optional{Int, Long, Double}.orElseThrow to be consistent with Optional.orElseThrow. I believe this is a compatible change since the classes are final. I will also file a CCC. Paul. diff -r 8b8b84397c5a src/java.base/share/classes/java/util/OptionalDouble.java --- a/src/java.base/share/classes/java/util/OptionalDouble.java Tue Jan 20 16:35:19 2015 +0000 +++ b/src/java.base/share/classes/java/util/OptionalDouble.java Tue Jan 20 16:54:07 2015 +0000 @@ -182,7 +182,7 @@ * @throws NullPointerException if no value is present and * {@code exceptionSupplier} is null */ - public double orElseThrow(Supplier exceptionSupplier) throws X { + public double orElseThrow(Supplier exceptionSupplier) throws X { if (isPresent) { return value; } else { diff -r 8b8b84397c5a src/java.base/share/classes/java/util/OptionalInt.java --- a/src/java.base/share/classes/java/util/OptionalInt.java Tue Jan 20 16:35:19 2015 +0000 +++ b/src/java.base/share/classes/java/util/OptionalInt.java Tue Jan 20 16:54:07 2015 +0000 @@ -182,7 +182,7 @@ * @throws NullPointerException if no value is present and * {@code exceptionSupplier} is null */ - public int orElseThrow(Supplier exceptionSupplier) throws X { + public int orElseThrow(Supplier exceptionSupplier) throws X { if (isPresent) { return value; } else { diff -r 8b8b84397c5a src/java.base/share/classes/java/util/OptionalLong.java --- a/src/java.base/share/classes/java/util/OptionalLong.java Tue Jan 20 16:35:19 2015 +0000 +++ b/src/java.base/share/classes/java/util/OptionalLong.java Tue Jan 20 16:54:07 2015 +0000 @@ -182,7 +182,7 @@ * @throws NullPointerException if no value is present and * {@code exceptionSupplier} is null */ - public long orElseThrow(Supplier exceptionSupplier) throws X { + public long orElseThrow(Supplier exceptionSupplier) throws X { if (isPresent) { return value; } else { From lance.andersen at oracle.com Tue Jan 20 17:28:41 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Tue, 20 Jan 2015 12:28:41 -0500 Subject: RFR 8059324: orElseThrow has different signatures for OptionalPrimitive and Optional In-Reply-To: References: Message-ID: <435D4358-6273-41DD-BA1D-A6A12AC0142E@oracle.com> looks ok paul On Jan 20, 2015, at 11:56 AM, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8059324-Optional-orElseThrow-signatures/webrev/ > > The patch fixes up the signatures of Optional{Int, Long, Double}.orElseThrow to be consistent with Optional.orElseThrow. > > I believe this is a compatible change since the classes are final. > > I will also file a CCC. > > Paul. > > diff -r 8b8b84397c5a src/java.base/share/classes/java/util/OptionalDouble.java > --- a/src/java.base/share/classes/java/util/OptionalDouble.java Tue Jan 20 16:35:19 2015 +0000 > +++ b/src/java.base/share/classes/java/util/OptionalDouble.java Tue Jan 20 16:54:07 2015 +0000 > @@ -182,7 +182,7 @@ > * @throws NullPointerException if no value is present and > * {@code exceptionSupplier} is null > */ > - public double orElseThrow(Supplier exceptionSupplier) throws X { > + public double orElseThrow(Supplier exceptionSupplier) throws X { > if (isPresent) { > return value; > } else { > diff -r 8b8b84397c5a src/java.base/share/classes/java/util/OptionalInt.java > --- a/src/java.base/share/classes/java/util/OptionalInt.java Tue Jan 20 16:35:19 2015 +0000 > +++ b/src/java.base/share/classes/java/util/OptionalInt.java Tue Jan 20 16:54:07 2015 +0000 > @@ -182,7 +182,7 @@ > * @throws NullPointerException if no value is present and > * {@code exceptionSupplier} is null > */ > - public int orElseThrow(Supplier exceptionSupplier) throws X { > + public int orElseThrow(Supplier exceptionSupplier) throws X { > if (isPresent) { > return value; > } else { > diff -r 8b8b84397c5a src/java.base/share/classes/java/util/OptionalLong.java > --- a/src/java.base/share/classes/java/util/OptionalLong.java Tue Jan 20 16:35:19 2015 +0000 > +++ b/src/java.base/share/classes/java/util/OptionalLong.java Tue Jan 20 16:54:07 2015 +0000 > @@ -182,7 +182,7 @@ > * @throws NullPointerException if no value is present and > * {@code exceptionSupplier} is null > */ > - public long orElseThrow(Supplier exceptionSupplier) throws X { > + public long orElseThrow(Supplier exceptionSupplier) throws X { > if (isPresent) { > return value; > } else { > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From lance.andersen at oracle.com Tue Jan 20 17:33:27 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Tue, 20 Jan 2015 12:33:27 -0500 Subject: RFR 8048689: Clarify documentation on BaseStream.spliterator In-Reply-To: <748EFA84-3146-478D-A8FE-75641159FA7E@oracle.com> References: <748EFA84-3146-478D-A8FE-75641159FA7E@oracle.com> Message-ID: Hi Paul, I think this is OK though probably should have a CCC as the javadoc is being updated which providing more detail vs just a couple of typos. Best Lance On Jan 20, 2015, at 11:51 AM, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8048689-BaseStream-spliterator/webrev/ > > Some clarifications to the characteristics of the Spliterator returned from BaseStream.spliterator() (otherwise known as the "escape-hatch" spliterator). > > Implementations are free to report fewer characteristics than all those that could be derived from the stream pipeline. > > For example, the current implementation does not support the propagation of NONNULL, IMMUTABLE or CONCURRENT through the stream pipeline. It could, we have room in the bit field for characteristic/flag propagation, but we did not think it worth the effort. > > Paul. > > diff -r 66508b45ff35 src/java.base/share/classes/java/util/stream/BaseStream.java > --- a/src/java.base/share/classes/java/util/stream/BaseStream.java Tue Jan 20 16:22:05 2015 +0000 > +++ b/src/java.base/share/classes/java/util/stream/BaseStream.java Tue Jan 20 16:35:32 2015 +0000 > @@ -79,6 +79,14 @@ > *

This is a terminal > * operation. > * > + *

> + * The returned spliterator should report the set of characteristics derived > + * from the stream pipeline (namely the characteristics derived from the > + * stream source spliterator and the intermediate operations). > + * Implementations may report a sub-set of those characteristics. For > + * example, it may be too expensive to compute the entire set for some or > + * all possible stream pipelines. > + * > * @return the element spliterator for this stream > */ > Spliterator spliterator(); > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From xueming.shen at oracle.com Tue Jan 20 17:35:27 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Tue, 20 Jan 2015 09:35:27 -0800 Subject: 8069325: Pattern.splitAsStream does not return input if it is empty and there is no match In-Reply-To: <722C43ED-09C0-4C77-9C6F-98A05D1134EE@oracle.com> References: <722C43ED-09C0-4C77-9C6F-98A05D1134EE@oracle.com> Message-ID: <54BE91DF.8010802@oracle.com> On 1/20/15 8:17 AM, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8069325-Pattern-splitAsStream-emptyInput/webrev/ > > This patch fixes an edge case in Pattern.splitAsStream for matching against an empty input string, which deviated from the behaviour of Pattern.split. When there are no matches a stream containing the input string should be returned rather than an empty stream. > > -- > > I have kept compatibility with Pattern.split(String ) but i noticed another an edge case. > > What should the following return: > > Pattern.compile("").split("") > > [] or [""]? > > There is a zero-width match at the beginning and an empty remaining segment both of which should be discarded, as such i would expect the result to be [] rather than as [""], as currently produced result. It may depend on how the "trailing empty string" gets interpreted. Is it possible to interpret it as the empty string is the result of the "substring from the beginning 0-width match and the end of the input sequence", any thing after that is "trailing"? It would be clear if the spec explicitly said, the result of splitting an empty input is an empty string. I would assume someone, mostly the user of String.split(), will get hit by this "incompatible" change. -Sherman > > If people agree that this is an issue i suggest we log a new one independent of fixing 8069325. > > Thanks, > Paul. From martinrb at google.com Tue Jan 20 17:59:44 2015 From: martinrb at google.com (Martin Buchholz) Date: Tue, 20 Jan 2015 09:59:44 -0800 Subject: RFR 8068432: Inconsistent exception handling in CompletableFuture.thenCompose In-Reply-To: References: <4D0311DB-C957-491A-9102-9DD539F10322@oracle.com> Message-ID: >From the point of view of back-port-ability, backporting the fix in isolation is good. We added tck tests for this, so could use label noreg-tck, but you've already writetn the test, and more testing (and more styles of testing) is always good! Pedantically, those static functions should be final to justify use of ALL_CAPS On Tue, Jan 20, 2015 at 1:11 AM, Chris Hegarty wrote: > Looks fine to me PAul. > > -Chris. > > On 19 Jan 2015, at 17:46, Paul Sandoz wrote: > > > Hi, > > > > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8068432-CompletableFuture-thenCompose-Exception/webrev/ > > > > This is a cherry-picked fix from the 166 repo with a JDK test added. > Once approved i will request a back-port to 8u60. > > > > Paul. > > From huizhe.wang at oracle.com Tue Jan 20 18:51:50 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Tue, 20 Jan 2015 10:51:50 -0800 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: <5A817A94-0176-4613-91F2-9B298C1DA6AA@oracle.com> References: <54934601.4010104@oracle.com> <54B96BDF.7090204@oracle.com> <54B9C9EA.7050407@oracle.com> <5A817A94-0176-4613-91F2-9B298C1DA6AA@oracle.com> Message-ID: <54BEA3C6.6090501@oracle.com> On 1/20/2015 7:02 AM, Lance Andersen wrote: > Hi Joe, > > I think the changes look fine. Thanks. > > I am wondering if we have any suggested standard for the use of > @Override as I see it is inconsistent in its usage with methods > implementing an interface. Is this something we should add to > existing code? I don't see Netbeans asking for it to be done except > in the case of an overridden method? NetBeans does make a suggestion such as: "Add @Override Annotation". I think you're right about avoiding comment duplication. In case of existing "evaluate" methods, I previously updated* the javadocs for both the interface and impl. It makes sense to take advantage of the "automatically inherit" feature of the javadoc tool to avoid the duplication. *Note that the update was only on format and styles, and some re-wording, no changes on definition or semantics. > > Thank you for the extra clean up Joe Thank you, now it looks better and cleaner. Best, Joe > > Best > Lance > On Jan 16, 2015, at 9:33 PM, huizhe wang > wrote: > >> >> On 1/16/2015 1:29 PM, Lance Andersen wrote: >>> Hi Joe, >>> >>> >>> >>> Overall it is OK, a few minor comments >>> >>> - Is there a reason that XPathExpressionImpl is no longer public >>> and XPathImpl is public? >> >> Ok, I'll keep both public, may be useful in the future. >> >>> - I think you can leverage {@inheritdoc} in your impl classes to >>> avoid comment duplication possibly? >> >> Looks like javadoc "Automatically inherit comment ". So @Override is >> good enough. I've removed the javadocs for methods that override, >> including the existing methods. It's good to avoid the duplication, >> and potential copy n paste errors. >> >>> - please remember to make the copyright year 2015 >> >> Done. >> >>> - XPathTestBase, can the static block be moved to a @BeforeClass >> >> Moved to within the Dataprovider. >> >>> - XPathNodes, should that have an @since 1.9? >> >> Fixed. >> >>> - Given you are not including @param, etc for your test comments, >>> you might want to consider /* */ vs /** */ style comments. >> >> Looks like s/\/**/\/* worked the trick. >> >>> That is consider if you really want doc comments (really a style >>> choice but some IDEs will issue a warning for missing tags >> >> Yeh, it's good to turn off the warning "light" :-) >> >> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >> >> Thanks, >> Joe >> >>> >>> >>> HTH >>> >>> Best, >>> Lance >>> On Jan 16, 2015, at 2:51 PM, huizhe wang >> > wrote: >>> >>>> Hi all, >>>> >>>> Could you review the change? >>>> >>>> Thanks, >>>> Joe >>>> >>>> On 12/18/2014 1:24 PM, huizhe wang wrote: >>>>> Hi, >>>>> >>>>> This is to add support for any type and improvement with new >>>>> features reflected in the new evaluateExpression methods, >>>>> XPathEvaluationResult and XPathNodes. >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-8054196 >>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>> >>>>> Thanks, >>>>> Joe >>>> >>> >>> >>> >>> >>> Lance >>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>> Oracle Java Engineering >>> 1 Network Drive >>> Burlington, MA 01803 >>> Lance.Andersen at oracle.com >>> >>> >>> >> > > > > Lance > Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > From paul.sandoz at oracle.com Tue Jan 20 18:59:52 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Jan 2015 18:59:52 +0000 Subject: 8050820: Please add java.util.Optional.stream() to convert Optional to Stream Message-ID: Hi, http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050820-Optional-stream/webrev/ This is one of those cases where i think adding a new method to Optional carries enough weight. It can be really awkward to use Optional with Stream.flatMap to map to a stream of present values. It's not at all obvious that one can do "op.map(Stream::of).orElse(Stream.empty()))" so often one sees some contorted code of there in the wild. Hopefully adding Optional.stream (and to the primitive variants) will help reduce such contortions. A CCC will be filed. Thanks, Paul. From paul.sandoz at oracle.com Tue Jan 20 19:05:34 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Jan 2015 19:05:34 +0000 Subject: RFR 8050819: Please add java.util.Stream.ofNullable(T object) Message-ID: <93EC212E-69DA-4874-8255-E73746404461@oracle.com> Hi, http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050819-Stream-ofNullable/webrev/ Stream.ofNullable can make it easier to work fluently when streams are produced, for example within flatMap. A CCC will be filed. Thanks, Paul. From paul.sandoz at oracle.com Tue Jan 20 19:06:08 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Jan 2015 19:06:08 +0000 Subject: RFR 8048689: Clarify documentation on BaseStream.spliterator In-Reply-To: References: <748EFA84-3146-478D-A8FE-75641159FA7E@oracle.com> Message-ID: <31B82002-2906-4934-A0B9-7D9CB54FCA43@oracle.com> On Jan 20, 2015, at 5:33 PM, Lance Andersen wrote: > Hi Paul, > > I think this is OK though probably should have a CCC as the javadoc is being updated which providing more detail vs just a couple of typos. > Yes, thanks for the reminder, i neglected to mention that, Paul. From paul.sandoz at oracle.com Tue Jan 20 19:08:35 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Jan 2015 19:08:35 +0000 Subject: RFR 8068432: Inconsistent exception handling in CompletableFuture.thenCompose In-Reply-To: References: <4D0311DB-C957-491A-9102-9DD539F10322@oracle.com> Message-ID: <94B48203-442B-4B37-A6B7-C8771C4D7360@oracle.com> On Jan 20, 2015, at 5:59 PM, Martin Buchholz wrote: > From the point of view of back-port-ability, backporting the fix in isolation is good. > > We added tck tests for this, so could use label noreg-tck, but you've already writetn the test, and more testing (and more styles of testing) is always good! I nearly did not bother :-) > Pedantically, those static functions should be final to justify use of ALL_CAPS > Updated locally. Thanks, Paul. From vladimir.x.ivanov at oracle.com Tue Jan 20 19:09:11 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Tue, 20 Jan 2015 22:09:11 +0300 Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when GWT LambdaForms are shared In-Reply-To: <7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com> References: <54B94766.2080102@oracle.com> <7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com> Message-ID: <54BEA7D7.6080008@oracle.com> John, thanks for the review! Updated webrev: http://cr.openjdk.java.net/~vlivanov/8063137/webrev.01/hotspot http://cr.openjdk.java.net/~vlivanov/8063137/webrev.01/jdk See my answers inline. On 1/17/15 2:13 AM, John Rose wrote: > On Jan 16, 2015, at 9:16 AM, Vladimir Ivanov > > wrote: >> >> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/hotspot/ >> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/jdk/ >> https://bugs.openjdk.java.net/browse/JDK-8063137 >> ... >> PS: as a summary, my experiments show that fixes for 8063137 & 8068915 >> [2] almost completely recovers peak performance after LambdaForm >> sharing [3]. There's one more problem left (non-inlined MethodHandle >> invocations are more expensive when LFs are shared), but it's a story >> for another day. > > This performance bump is excellent news. LFs are supposed to express > emergently common behaviors, like hidden classes. We are much closer to > that goal now. > > I'm glad to see that the library-assisted profiling turns out to be > relatively clean. > > In effect this restores the pre-LF CountingMethodHandle logic from 2011, > which was so beneficial in JDK 7: > http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/02de5cdbef21/src/share/classes/java/lang/invoke/CountingMethodHandle.java > > I have some suggestions to make this version a little cleaner; see below. > > Starting with the JDK changes: > > In LambdaForm.java, I'm feeling flag pressure from all the little > boolean fields and constructor parameters. > > (Is it time to put in a bit-encoded field "private byte > LambdaForm.flags", or do we wait for another boolean to come along? But > see next questions, which are more important.) > > What happens when a GWT LF gets inlined into a larger LF? Then there > might be two or more selectAlternative calls. > Will this confuse anything or will it Just Work? The combined LF will > get profiled as usual, and the selectAlternative calls will also collect > profile (or not?). > > This leads to another question: Why have a boolean 'isGWT' at all? Why > not just check for one or more occurrence of selectAlternative, and > declare that those guys override (some of) the profiling. Something like: > > -+ if (PROFILE_GWT && lambdaForm.isGWT) > ++ if (PROFILE_GWT && lambdaForm.containsFunction(NF_selectAlternative)) > (...where LF.containsFunction(NamedFunction) is a variation of > LF.contains(Name).) > > I suppose the answer may be that you want to inline GWTs (if ever) into > customized code where the JVM profiling should get maximum benefit. In > that case case you might want to set the boolean to "false" to > distinguish "immature" GWT combinators from customized ones. > > If that's the case, perhaps the real boolean flag you want is not > 'isGWT' but 'sharedProfile' or 'immature' or some such, or (inverting) > 'customized'. (I like the feel of a 'customized' flag.) Then > @IgnoreProfile would get attached to a LF that (a ) contains > selectAlternative and (b ) is marked as non-customized/immature/shared. > You might also want to adjust the call to 'profileBranch' based on > whether the containing LF was shared or customized. > > What I'm mainly poking at here is that 'isGWT' is not informative about > the intended use of the flag. I agree. It was an interim solution. Initially, I planned to introduce customization and guide the logic based on that property. But it's not there yet and I needed something for GWT case. Unfortunately, I missed the case when GWT is edited. In that case, isGWT flag is missed and no annotation is set. So, I removed isGWT flag and introduced a check for selectAlternative occurence in LambdaForm shape, as you suggested. > In 'updateCounters', if the counter overflows, you'll get continuous > creation of ArithmeticExceptions. Will that optimize or will it cause a > permanent slowdown? Consider a hack like this on the exception path: > counters[idx] = Integer.MAX_VALUE / 2; I had an impression that VM optimizes overflows in Math.exact* intrinsics, but it's not the case - it always inserts an uncommon trap. I used the workaround you proposed. > On the Name Bikeshed: It looks like @IgnoreProfile (ignore_profile in > the VM) promises too much "ignorance", since it suppresses branch counts > and traps, but allows type profiles to be consulted. Maybe something > positive like "@ManyTraps" or "@SharedMegamorphic"? (It's just a name, > and this is just a suggestion.) What do you think about @LambdaForm.Shared? > Going to the JVM: > > In library_call.cpp, I think you should change the assert to a guard: > -+ assert(aobj->length() == 2, ""); > ++ && aobj->length() == 2) { Done. > In Parse::dynamic_branch_prediction, the mere presence of the Opaque4 > node is enough to trigger replacement of profiling. I think there > should *not* be a test of method()->ignore_profile(). That should > provide better integration between the two sources of profile data to > JVM profiling? Done. > Also, I think the name 'Opaque4Node' is way too? opaque. Suggest > 'ProfileBranchNode', since that's exactly what it does. Done. > Suggest changing the log element "profile_branch" to "observe > source='profileBranch'", to make a better hint as to the source of the info. Done. Best regards, Vladimir Ivanov From paul.sandoz at oracle.com Tue Jan 20 19:31:01 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Jan 2015 19:31:01 +0000 Subject: 8069325: Pattern.splitAsStream does not return input if it is empty and there is no match In-Reply-To: <54BE91DF.8010802@oracle.com> References: <722C43ED-09C0-4C77-9C6F-98A05D1134EE@oracle.com> <54BE91DF.8010802@oracle.com> Message-ID: On Jan 20, 2015, at 5:35 PM, Xueming Shen wrote: > On 1/20/15 8:17 AM, Paul Sandoz wrote: >> Hi, >> >> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8069325-Pattern-splitAsStream-emptyInput/webrev/ >> >> This patch fixes an edge case in Pattern.splitAsStream for matching against an empty input string, which deviated from the behaviour of Pattern.split. When there are no matches a stream containing the input string should be returned rather than an empty stream. >> >> -- >> >> I have kept compatibility with Pattern.split(String ) but i noticed another an edge case. >> >> What should the following return: >> >> Pattern.compile("").split("") >> >> [] or [""]? >> >> There is a zero-width match at the beginning and an empty remaining segment both of which should be discarded, as such i would expect the result to be [] rather than as [""], as currently produced result. > > It may depend on how the "trailing empty string" gets interpreted. Is it possible to interpret it as the > empty string is the result of the "substring from the beginning 0-width match and the end of the input > sequence", any thing after that is "trailing"? > Seems a stretch to me. Consider the following which returns []: Pattern.compile("x").split("x"); Replace "x" with "" and intuitively i would expect the same behaviour. > It would be clear if the spec explicitly said, the result of splitting an empty input is an empty string. > > I would assume someone, mostly the user of String.split(), will get hit by this "incompatible" change. > Yeah, there is some risk in that. Paul. From peter.levart at gmail.com Tue Jan 20 20:22:50 2015 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 20 Jan 2015 21:22:50 +0100 Subject: Explicit Serialization API and Security In-Reply-To: <1421240334.8856.17.camel@Nokia-N900> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> Message-ID: <54BEB91A.4020603@gmail.com> Hi Chris and Peter, It just occurred to me that the following scheme would provide failure atomicity for the whole Object regardless of whether readObject methods are used or not for various classes in hierarchy: - allocate uninitialized instance - call default accessible constructor of the most specific non-Serializable class - deserialize (by calling readObject methods where provided) the fields of all classes in hierarchy like normally (up to this point, nothing is changed from what we have now) - if deserialization fails anywhere, undo everything by setting all the fields in the Serializable part of the hierarchy to default values (null for references, 0 for primitives), abandon the object and propagate failure. While deserializing, the object is in inconsistent state. If deserialization fails, this state is rolled-back to uninitialized state. finalize() can still get to the instance, but it will be uninitialized. Peter On 01/14/2015 01:58 PM, Peter Firmstone wrote: > > Hi Chris, > > Sorry, no. > > Currently when an ObjectStreamClass is read in from the stream, the > framework searches for the first zero arg constructor of a non > serializable class and creates and instance of the class read and > resolved from the stream, however it does so using a super class > constructor. > > Then from the super class down, fields are read in and set in order > for each class in the object's inheritance hierarchy. > > The alternative I propose, doesn't create the instance, instead it > reads the fields from the stream, one by one and without instantiating > them, if they are newly read objects, stores them temporarily into > byte [] arrays in a Map with reference handle keys, otherwise it just > holds the reference handle. > > What it does next is wrap this information into a caller sensitive > api, GetFields or ReadSerial instance, that is passed as a constructor > parameter to the child class serial constructor. > > The child class checks invariants and reads each field it needs using > a static method prior to calling a superclass constructor, each class > in the inheritance hierarchy for the object then checks its invariants > until it gets to the first non serializable superclass. > > The benefit of this order is that each class is present in the thread > security context, so protection domain security and invariants are > enforced before instantiating an object. > > Hope this helps illuminate it a little better, regards, > > Peter. > > ----- Original message ----- > > Peter F, > > > > I am still struggling with the basic concept of you proposal. Let me > see > > if I understand it correctly. Does the following describe a similar > > scenario as you envisage: > > > > 1) For each Serializable type, T, in the deserialized types > > hierarchy, starting with the top most ( closest to > j.l.Object ), > > > > 1a) Read T's fields from the stream, fields > > > > 1b) validate(t, fields) // t will be null first time > > > > 1c) allocate a new instance of T, and assign to t > > > > 1d) set fields in t > > > > 2) Return t; > > > > So for each level in the hierarchy, an instance of a type is created > > only after its invariants have been checked. This instance is then > > passed to the next level so it can participate in that levels > invariants > > validation. > > > > If this scenario is along the same lines as yours, then I just don't > see > > how 1c above will always be possible. > > > > If we could somehow make the object caller sensitive until after > > deserialization completes, then could avoid having to try to allocate > > multiple instance down the hierarchy. > > > > -Chris. > > > > On 13/01/15 10:24, Peter Firmstone wrote: > > > Could we use a static validator method and generate bytecode for > > > constructors dynamically? > > > > > > The developer can optionally implement the constructors. > > > > > > static GetField invariantCheck(GetField f); > > > > > > Create a caller sensitive GetField implementation and add a two new > > > methods to GetField: > > > > > > abstract Object createSuper(); // to access superclass object methods > > > for inavariant checking. > > > > > > abstract Class getType(String name); > > > > > > Set fields from within constructors. > > > > > > The generated constructors are straight forward: > > > > > > 1. Call static method. > > > 2. Call super class constructor with result from static method. > > > 3. Set final fields > > > 4. How to set transient fields, implement a private method called > from > > > within the constructor? > > > > > > Require a permission to extend GetField? > > > > From joe.darcy at oracle.com Tue Jan 20 22:52:26 2015 From: joe.darcy at oracle.com (Joseph D. Darcy) Date: Tue, 20 Jan 2015 14:52:26 -0800 Subject: JDK 9 RFR of JDK-8069262: Doclint regression in java.nio.channels.Channels In-Reply-To: <54BCCF9C.2080209@oracle.com> References: <54BCCF9C.2080209@oracle.com> Message-ID: <54BEDC2A.9010903@oracle.com> Hi Amy, The fix looks fine; approved to go back, Thanks, -Joe On 1/19/2015 1:34 AM, Amy Lu wrote: > Please review the fix for Doclint regression in > java.nio.channels.Channels, removed the empty

tag. > > bug: https://bugs.openjdk.java.net/browse/JDK-8069262 > webrev: http://cr.openjdk.java.net/~amlu/8069262/webrev.00/ > > I also need a sponsor for this. > > Thanks, > Amy From weijun.wang at oracle.com Wed Jan 21 08:29:25 2015 From: weijun.wang at oracle.com (Wang Weijun) Date: Wed, 21 Jan 2015 16:29:25 +0800 Subject: [9] request for review: 8049171: Additional tests for jarsigner's warnings In-Reply-To: <54BF48BF.9090505@oracle.com> References: <54BF48BF.9090505@oracle.com> Message-ID: Thanks for adding so many tests. Some suggestions: - JarUtils.java You can use the new InputStream.transferTo() method. I am not sure if I understand updateJar correctly. It looks like srcJarFile is opened multiple times so its entries are duplicated a lot in the destination. Or is there a secret break? - Utils.java The mixed using of File and Files is strange, but you are free to keep it. - TimestampCheck.java You can make Handler Autocloseable to use try-with-resources in tests. - Various tests I am not a fan of calling Utils.cleanup() in final block. Unless you have created huge garbages, those files are precious when the test fails (given you provide -retain in jtreg, which I always do). Thanks Max > On Jan 21, 2015, at 14:35, Artem Smotrakov wrote: > > Hello, > > Please review a couple of new tests for jarsigner's warnings. Basically tests run jarsigner and check warning/error messages and exit codes according to [1]. > > https://bugs.openjdk.java.net/browse/JDK-8049171 > http://cr.openjdk.java.net/~asmotrak/8049171/webrev.00 > > [1] http://docs.oracle.com/javase/7/docs/technotes/tools/windows/jarsigner.html > > Artem > > From andreas.lundblad at oracle.com Wed Jan 21 10:00:00 2015 From: andreas.lundblad at oracle.com (Andreas Lundblad) Date: Wed, 21 Jan 2015 11:00:00 +0100 Subject: 8050820: Please add java.util.Optional.stream() to convert Optional to Stream In-Reply-To: References: Message-ID: <20150121100000.GA11988@e6430> On Tue, Jan 20, 2015 at 06:59:52PM +0000, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050820-Optional-stream/webrev/ > > This is one of those cases where i think adding a new method to Optional carries enough weight. > > It can be really awkward to use Optional with Stream.flatMap to map to a stream of present values. > > It's not at all obvious that one can do "op.map(Stream::of).orElse(Stream.empty()))" so often one sees some contorted code of there in the wild. Hopefully adding Optional.stream (and to the primitive variants) will help reduce such contortions. > > A CCC will be filed. > > Thanks, > Paul. Hi Paul, My name is Andreas Lundblad, and I joined the langtools team a little more than a year ago. I'm not a Reviewer or anything but I casually read through your patch out of curiosity. You don't seem to be a fan of the ?: operator. I would have written return isPresent() ? Stream.of(value) : Stream.empty(); Any reason for if/else in this case? Also, I find it strange to include { ... } only on the else-branch: + if (!isPresent()) + return Stream.empty(); + else { + return Stream.of(value); + } It's in all of the if-statements in the patch so perhaps it's intentional. I think it looks a bit peculiar. -- Andreas From paul.sandoz at oracle.com Wed Jan 21 10:25:49 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 21 Jan 2015 10:25:49 +0000 Subject: 8050820: Please add java.util.Optional.stream() to convert Optional to Stream In-Reply-To: <20150121100000.GA11988@e6430> References: <20150121100000.GA11988@e6430> Message-ID: <3CED45C1-A695-44E9-8E9A-F1D0990A9393@oracle.com> Hi Andreas, On Jan 21, 2015, at 10:00 AM, Andreas Lundblad wrote: > Hi Paul, > > My name is Andreas Lundblad, and I joined the langtools team a little more than a year ago. I'm not a Reviewer or anything but I casually read through your patch out of curiosity. > > You don't seem to be a fan of the ?: operator. I would have written > > return isPresent() ? Stream.of(value) : Stream.empty(); > > Any reason for if/else in this case? Also, I find it strange to include { ... } only on the else-branch: > > + if (!isPresent()) > + return Stream.empty(); > + else { > + return Stream.of(value); > + } > > It's in all of the if-statements in the patch so perhaps it's intentional. I think it looks a bit peculiar. > I copied the same style used in the existing methods. I personally would have used the ternary operator if i wrote this class :-) You are right, the inconsistent use of braces is odd, i will fix throughout to be consistent but will resist the urge to use the ternary operator. Even though you are not an official reviewer i can still add you as a reviewer of this patch. Paul. From paul.sandoz at oracle.com Wed Jan 21 10:35:16 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 21 Jan 2015 10:35:16 +0000 Subject: [8u60] RFR 8069302 Deprecate Unsafe monitor methods in JDK 8u release Message-ID: <7BA76D88-2443-4DF2-AC64-439FF0911CDA@oracle.com> Hi, The Unsafe monitor methods monitorEnter. monitorExit and tryMonitorEnter were removed from JDK 9 [1]. I would like to mark those methods as deprecated in 8u60. I doubt no one will notice :-) but it's the correct thing to do. Paul. [1] http://hg.openjdk.java.net/jdk9/hs-rt/jdk/rev/0a0a0986400e From forax at univ-mlv.fr Wed Jan 21 10:42:10 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 21 Jan 2015 11:42:10 +0100 Subject: [8u60] RFR 8069302 Deprecate Unsafe monitor methods in JDK 8u release In-Reply-To: <7BA76D88-2443-4DF2-AC64-439FF0911CDA@oracle.com> References: <7BA76D88-2443-4DF2-AC64-439FF0911CDA@oracle.com> Message-ID: <54BF8282.6060903@univ-mlv.fr> On 01/21/2015 11:35 AM, Paul Sandoz wrote: > Hi, > > The Unsafe monitor methods monitorEnter. monitorExit and tryMonitorEnter were removed from JDK 9 [1]. > > I would like to mark those methods as deprecated in 8u60. I doubt no one will notice :-) but it's the correct thing to do. > > Paul. > > [1] http://hg.openjdk.java.net/jdk9/hs-rt/jdk/rev/0a0a0986400e a good idea. R?mi From paul.sandoz at oracle.com Wed Jan 21 11:01:50 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 21 Jan 2015 11:01:50 +0000 Subject: 8050820: Please add java.util.Optional.stream() to convert Optional to Stream In-Reply-To: <3CED45C1-A695-44E9-8E9A-F1D0990A9393@oracle.com> References: <20150121100000.GA11988@e6430> <3CED45C1-A695-44E9-8E9A-F1D0990A9393@oracle.com> Message-ID: On Jan 21, 2015, at 10:25 AM, Paul Sandoz wrote: > Hi Andreas, > > On Jan 21, 2015, at 10:00 AM, Andreas Lundblad wrote: >> Hi Paul, >> >> My name is Andreas Lundblad, and I joined the langtools team a little more than a year ago. I'm not a Reviewer or anything but I casually read through your patch out of curiosity. >> >> You don't seem to be a fan of the ?: operator. I would have written >> >> return isPresent() ? Stream.of(value) : Stream.empty(); >> >> Any reason for if/else in this case? Also, I find it strange to include { ... } only on the else-branch: >> >> + if (!isPresent()) >> + return Stream.empty(); >> + else { >> + return Stream.of(value); >> + } >> >> It's in all of the if-statements in the patch so perhaps it's intentional. I think it looks a bit peculiar. >> > > I copied the same style used in the existing methods. > > I personally would have used the ternary operator if i wrote this class :-) You are right, the inconsistent use of braces is odd, i will fix throughout to be consistent but will resist the urge to use the ternary operator. > > Even though you are not an official reviewer i can still add you as a reviewer of this patch. > I updated the webrev in place to be more consistent in the use of braces and better consistency for the primitive specializations: http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050820-Optional-stream/webrev/ I don't wanna make any more syntax-related changes unless i done something silly. Paul. From forax at univ-mlv.fr Wed Jan 21 11:12:06 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 21 Jan 2015 12:12:06 +0100 Subject: 8050820: Please add java.util.Optional.stream() to convert Optional to Stream In-Reply-To: References: <20150121100000.GA11988@e6430> <3CED45C1-A695-44E9-8E9A-F1D0990A9393@oracle.com> Message-ID: <54BF8986.7000309@univ-mlv.fr> On 01/21/2015 12:01 PM, Paul Sandoz wrote: > On Jan 21, 2015, at 10:25 AM, Paul Sandoz wrote: > >> Hi Andreas, >> >> On Jan 21, 2015, at 10:00 AM, Andreas Lundblad wrote: >>> Hi Paul, >>> >>> My name is Andreas Lundblad, and I joined the langtools team a little more than a year ago. I'm not a Reviewer or anything but I casually read through your patch out of curiosity. >>> >>> You don't seem to be a fan of the ?: operator. I would have written >>> >>> return isPresent() ? Stream.of(value) : Stream.empty(); >>> >>> Any reason for if/else in this case? Also, I find it strange to include { ... } only on the else-branch: >>> >>> + if (!isPresent()) >>> + return Stream.empty(); >>> + else { >>> + return Stream.of(value); >>> + } >>> >>> It's in all of the if-statements in the patch so perhaps it's intentional. I think it looks a bit peculiar. >>> >> I copied the same style used in the existing methods. >> >> I personally would have used the ternary operator if i wrote this class :-) You are right, the inconsistent use of braces is odd, i will fix throughout to be consistent but will resist the urge to use the ternary operator. >> >> Even though you are not an official reviewer i can still add you as a reviewer of this patch. >> > I updated the webrev in place to be more consistent in the use of braces and better consistency for the primitive specializations: > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050820-Optional-stream/webrev/ > > I don't wanna make any more syntax-related changes unless i done something silly. > > Paul. This patch is good for me. I'm glad to see that feature integrated into OptionalX. R?mi From vladimir.x.ivanov at oracle.com Wed Jan 21 11:41:15 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Wed, 21 Jan 2015 14:41:15 +0300 Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when GWT LambdaForms are shared In-Reply-To: References: <54B94766.2080102@oracle.com> Message-ID: <54BF905B.7020407@oracle.com> Duncan, sorry for that. Updated webrev inplace. Best regards, Vladimir Ivanov On 1/21/15 1:39 PM, MacGregor, Duncan (GE Energy Management) wrote: > This version seems to have inconsistent removal of ignore profile in the > hotspot patch. It?s no longer added to vmSymbols but is still referenced > in classFileParser. > > On 19/01/2015 20:21, "MacGregor, Duncan (GE Energy Management)" > wrote: > >> Okay, I?ve done some tests of this with the micro benchmarks for our >> language & runtime which show pretty much no change except for one test >> which is now almost 3x slower. It uses nested loops to iterate over an >> array and concatenate the string-like objects it contains, and replaces >> elements with these new longer string-llike objects. It?s a bit of a >> pathological case, and I haven?t seen the same sort of degradation in the >> other benchmarks or in real applications, but I haven?t done serious >> benchmarking of them with this change. >> >> I shall see if the test case can be reduced down to anything simpler while >> still showing the same performance behaviour, and try add some compilation >> logging options to narrow down what?s going on. >> >> Duncan. >> >> On 16/01/2015 17:16, "Vladimir Ivanov" >> wrote: >> >>> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/hotspot/ >>> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/jdk/ >>> https://bugs.openjdk.java.net/browse/JDK-8063137 >>> >>> After GuardWithTest (GWT) LambdaForms became shared, profile pollution >>> significantly distorted compilation decisions. It affected inlining and >>> hindered some optimizations. It causes significant performance >>> regressions for Nashorn (on Octane benchmarks). >>> >>> Inlining was fixed by 8059877 [1], but it didn't cover the case when a >>> branch is never taken. It can cause missed optimization opportunity, and >>> not just increase in code size. For example, non-pruned branch can break >>> escape analysis. >>> >>> Currently, there are 2 problems: >>> - branch frequencies profile pollution >>> - deoptimization counts pollution >>> >>> Branch frequency pollution hides from JIT the fact that a branch is >>> never taken. Since GWT LambdaForms (and hence their bytecode) are >>> heavily shared, but the behavior is specific to MethodHandle, there's no >>> way for JIT to understand how particular GWT instance behaves. >>> >>> The solution I propose is to do profiling in Java code and feed it to >>> JIT. Every GWT MethodHandle holds an auxiliary array (int[2]) where >>> profiling info is stored. Once JIT kicks in, it can retrieve these >>> counts, if corresponding MethodHandle is a compile-time constant (and it >>> is usually the case). To communicate the profile data from Java code to >>> JIT, MethodHandleImpl::profileBranch() is used. >>> >>> If GWT MethodHandle isn't a compile-time constant, profiling should >>> proceed. It happens when corresponding LambdaForm is already shared, for >>> newly created GWT MethodHandles profiling can occur only in native code >>> (dedicated nmethod for a single LambdaForm). So, when compilation of the >>> whole MethodHandle chain is triggered, the profile should be already >>> gathered. >>> >>> Overriding branch frequencies is not enough. Statistics on >>> deoptimization events is also polluted. Even if a branch is never taken, >>> JIT doesn't issue an uncommon trap there unless corresponding bytecode >>> doesn't trap too much and doesn't cause too many recompiles. >>> >>> I added @IgnoreProfile and place it only on GWT LambdaForms. When JIT >>> sees it on some method, Compile::too_many_traps & >>> Compile::too_many_recompiles for that method always return false. It >>> allows JIT to prune the branch based on custom profile and recompile the >>> method, if the branch is visited. >>> >>> For now, I wanted to keep the fix very focused. The next thing I plan to >>> do is to experiment with ignoring deoptimization counts for other >>> LambdaForms which are heavily shared. I already saw problems caused by >>> deoptimization counts pollution (see JDK-8068915 [2]). >>> >>> I plan to backport the fix into 8u40, once I finish extensive >>> performance testing. >>> >>> Testing: JPRT, java/lang/invoke tests, nashorn (nashorn testsuite, >>> Octane). >>> >>> Thanks! >>> >>> PS: as a summary, my experiments show that fixes for 8063137 & 8068915 >>> [2] almost completely recovers peak performance after LambdaForm sharing >>> [3]. There's one more problem left (non-inlined MethodHandle invocations >>> are more expensive when LFs are shared), but it's a story for another >>> day. >>> >>> Best regards, >>> Vladimir Ivanov >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8059877 >>> 8059877: GWT branch frequencies pollution due to LF sharing >>> [2] https://bugs.openjdk.java.net/browse/JDK-8068915 >>> [3] https://bugs.openjdk.java.net/browse/JDK-8046703 >>> JEP 210: LambdaForm Reduction and Caching >>> _______________________________________________ >>> mlvm-dev mailing list >>> mlvm-dev at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev >> >> _______________________________________________ >> mlvm-dev mailing list >> mlvm-dev at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev > From peter.levart at gmail.com Wed Jan 21 11:52:36 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 21 Jan 2015 12:52:36 +0100 Subject: RFR: JDK-8047769 SecureRandom should be more frugal with file descriptors In-Reply-To: <54AC3CEA.6000702@oracle.com> References: <5492D557.2000807@gmail.com> <5498C5A0.50104@oracle.com> <549AA573.2060801@gmail.com> <54A20475.3040206@oracle.com> <54A5AC7C.5010806@gmail.com> <54A7228D.4040801@oracle.com> <54A9696A.3070406@gmail.com> <54AC3CEA.6000702@oracle.com> Message-ID: <54BF9304.10906@gmail.com> Hi, I have pushed this change. Thanks Brad, Alan and Chris for reviews. Regards, Peter On 01/06/2015 08:52 PM, Bradford Wetmore wrote: > I only looked at test, looks good to me. > > I'd rarely ask to remove extra prints in tests. It adds initial > debugging data points in case something breaks down the road. > > Brad > > > On 1/4/2015 8:25 AM, Peter Levart wrote: >> Hi Brad, >> >> So I created another webrev (just removed the unneeded import and >> left-over System.out.println from test): >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.06/ >> >> >> I'll leave it here to rest for a couple of days and if no one objects, >> I'll push it to jdk9-dev. >> >> Thanks everybody for reviews and happy new year! >> >> Regards, Peter >> >> On 01/02/2015 11:58 PM, Bradford Wetmore wrote: >>> >>> On 1/1/2015 12:22 PM, Peter Levart wrote: >>>> Hi Brad, >>>> >>>> Here's next webrev which tries to cover all your comments: >>>> >>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/FileInputStreamPool.8047769/webrev.04/ >>>> >>>> >>> >>> I downloaded the webrev.05 (Chris' later followup email) and ran it >>> through JPRT. The only error was the previously seen -Dseed which is >>> not your problem. >>> >>> Again, I only ran: >>> >>> jdk_lang, jdk_math, jdk_util, jdk_io, jdk_net, jdk_nio, >>> jdk_security*, jdk_rmi, jdk_text, jdk_time, jdk_other, core_tools. >>> >>> If you need anything else run, let me know. >>> >>>>>>> Looks like you have a committer status, will you be pushing this? >>>>>> >>>>>> I can, yes. As soon as we clear-out the remaining questions, right? >>>>> >>>>> Yes. The comments below are minor and shouldn't need another review >>>>> cycle. >>>> >>>> I'm worried about the failure of the test you observed while running >>>> from NetBeans. Perhaps a 0.5s wait is sometimes not enough for >>>> ReferenceHandler thread to process (enqueue) a WeakReference. Since >>>> there is already a facility in place to help ReferenceHandler thread >>>> instead of wait for it, I used it in new version of the test. >>> >>> BTW, there's now an unnecessary import from java.lang.AssertionError >>> added in webrev.04. >>> >>>>> TEST RESULT: Failed. Compilation failed: Compilation failed >>>> >>>> I changed the test to be self-contained now so one can run it without >>>> testlib in classpath. >>> >>> Thanks. It's compiling fine now. >>> >>>>> Two minor nits? SeedGenerator.java: Lines 507/518 >>>> >>>> Done that too. >>> >>> Thanks. >>> >>>>> Maybe issue multiple reads to exercise in1 and in2? e.g. 2 bytes of >>>>> in1, 4 bytes of in2, then 2 bytes of in1? >>>> >>>> The 1st assert makes sure in1 == in2, so there's no point in invoking >>>> the same instance via two aliases. >>> >>> True. >>> >>>>> IIRC, when I ran this under NetBeans last week, the second >>>>> testCaching >>>>> didn't clear the WeakReference. >>>> >>>> This should not happen any more now that the test is helping to >>>> enqueue >>>> the WeakReferences instead of waiting for ReferenceHandler to enqueue >>>> them. >>> >>> Yes, that hit the refQueue.poll(). >>> >>> It's always interesting to work with core-libs folks, learn something >>> new everyday. Mixing Lambdas/try-with. >>> >>> I need a time-machine for your CFV/jdk8 Committer: >>> >>> http://mail.openjdk.java.net/pipermail/jdk8-dev/2013-August/002896.html >>> >>> I vote yes. >>> >>>> The test can now fail only if System.gc() does not trigger >>>> WeakReference processing in the VM. Can you give it a try on your >>>> NetBeans environment? >>> >>> One last comment. It's now 2015. ;) >>> >>> Brad >>> >> From chris.hegarty at oracle.com Wed Jan 21 12:26:42 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 21 Jan 2015 12:26:42 +0000 Subject: RFR 8048689: Clarify documentation on BaseStream.spliterator In-Reply-To: <748EFA84-3146-478D-A8FE-75641159FA7E@oracle.com> References: <748EFA84-3146-478D-A8FE-75641159FA7E@oracle.com> Message-ID: <54BF9B02.904@oracle.com> Looks good to me Paul. -Chris. On 20/01/15 16:51, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8048689-BaseStream-spliterator/webrev/ > > Some clarifications to the characteristics of the Spliterator returned from BaseStream.spliterator() (otherwise known as the "escape-hatch" spliterator). > > Implementations are free to report fewer characteristics than all those that could be derived from the stream pipeline. > > For example, the current implementation does not support the propagation of NONNULL, IMMUTABLE or CONCURRENT through the stream pipeline. It could, we have room in the bit field for characteristic/flag propagation, but we did not think it worth the effort. > > Paul. > > diff -r 66508b45ff35 src/java.base/share/classes/java/util/stream/BaseStream.java > --- a/src/java.base/share/classes/java/util/stream/BaseStream.java Tue Jan 20 16:22:05 2015 +0000 > +++ b/src/java.base/share/classes/java/util/stream/BaseStream.java Tue Jan 20 16:35:32 2015 +0000 > @@ -79,6 +79,14 @@ > *

This is a terminal > * operation. > * > + *

> + * The returned spliterator should report the set of characteristics derived > + * from the stream pipeline (namely the characteristics derived from the > + * stream source spliterator and the intermediate operations). > + * Implementations may report a sub-set of those characteristics. For > + * example, it may be too expensive to compute the entire set for some or > + * all possible stream pipelines. > + * > * @return the element spliterator for this stream > */ > Spliterator spliterator(); > From peter.levart at gmail.com Wed Jan 21 13:04:31 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 21 Jan 2015 14:04:31 +0100 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table with wrong capacity In-Reply-To: <54AFAA61.30008@gmail.com> References: <7b2072fe0674e465f86518a9f25cef6d@sonic.net> <54AFAA61.30008@gmail.com> Message-ID: <54BFA3DF.4090201@gmail.com> Hi Martin and others, I have also modified the test per Martin's suggestion to factor out the serialClone() method. Here's the latest webrev: http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.03/ Is this ok now? Regards, Peter On 01/09/2015 11:16 AM, Peter Levart wrote: > On 01/05/2015 05:52 PM, Mike Duigou wrote: >> Well spotted Peter. The change looks good though I wonder if it >> should be: >> >> int length = (int)((elements + elements / 20) / loadFactor) + 3; >> >> FYI, regarding Daniel's suggestion: When similar invariant checks >> were added to the HashMap deserialization method we found code which >> relied upon the illegal values. In some cases the serialized HashMaps >> were created by alternative serialization implementations which >> included illegal, but until the checks were added, "harmless" values. >> >> The invariant checks should still be added though. It might even be >> worth adding checks that the other deserialized values are in valid >> ranges. >> >> Mike > > Hi Mike and others, > > Yes, your suggested length computation is more in "spirit" with the > comment above that states: "Compute new length with a bit of room 5% > to grow...", since it takes loadFactor into account for that > additional 5% too. I changed it to your suggested expression. > > Here's new webrev: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.02/ > > In addition to valid loadFactor, # of elements is checked to be > non-negative. The original length is just "repaired" if it appears to > be less than the enforced auto-growth invariant of Hashtable. > > I also expanded the test to exercise more combinations of # of > elements and loadFactor. Here's what gets printed with current > Hashtable implementation: > > ser. deser. > size load lentgh length valid range ok? > ------- ----- ------- ------- ----------------- ------ > 10 0.15 127 4 67... 134 NOT-OK > 10 0.50 31 8 21... 42 NOT-OK > 10 0.75 15 10 14... 28 NOT-OK > 10 1.00 15 13 11... 22 OK > 10 2.50 7 7 5... 10 OK > 50 0.15 511 12 334... 668 NOT-OK > 50 0.50 127 30 101... 202 NOT-OK > 50 0.75 127 42 67... 134 NOT-OK > 50 1.00 63 55 51... 102 OK > 50 2.50 31 31 21... 42 OK > 500 0.15 4095 103 3334... 6668 NOT-OK > 500 0.50 1023 278 1001... 2002 NOT-OK > 500 0.75 1023 403 667... 1334 NOT-OK > 500 1.00 511 511 501... 1002 OK > 500 2.50 255 255 201... 402 OK > 5000 0.15 65535 1003 33334... 66668 NOT-OK > 5000 0.50 16383 2753 10001... 20002 NOT-OK > 5000 0.75 8191 4003 6667... 13334 NOT-OK > 5000 1.00 8191 5253 5001... 10002 OK > 5000 2.50 2047 2047 2001... 4002 OK > > > With patched Hashtable, the test passes: > > ser. deser. > size load lentgh length valid range ok? > ------- ----- ------- ------- ----------------- ------ > 10 0.15 127 69 67... 134 OK > 10 0.50 31 23 21... 42 OK > 10 0.75 15 15 14... 28 OK > 10 1.00 15 13 11... 22 OK > 10 2.50 7 7 5... 10 OK > 50 0.15 511 349 334... 668 OK > 50 0.50 127 107 101... 202 OK > 50 0.75 127 71 67... 134 OK > 50 1.00 63 55 51... 102 OK > 50 2.50 31 23 21... 42 OK > 500 0.15 4095 3501 3334... 6668 OK > 500 0.50 1023 1023 1001... 2002 OK > 500 0.75 1023 703 667... 1334 OK > 500 1.00 511 511 501... 1002 OK > 500 2.50 255 213 201... 402 OK > 5000 0.15 65535 35003 33334... 66668 OK > 5000 0.50 16383 10503 10001... 20002 OK > 5000 0.75 8191 7003 6667... 13334 OK > 5000 1.00 8191 5253 5001... 10002 OK > 5000 2.50 2047 2047 2001... 4002 OK > > > > Regards, Peter > >> >> On 2015-01-05 07:48, core-libs-dev-request at openjdk.java.net wrote: >>> >>> Today's Topics: >>> >>> 2. Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes >>> table with wrong capacity (Daniel Fuchs) >> >> >>> Message: 2 >>> Date: Mon, 05 Jan 2015 15:52:55 +0100 >>> From: Daniel Fuchs >>> To: Peter Levart , core-libs-dev >>> >>> Subject: Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes >>> table with wrong capacity >>> Message-ID: <54AAA547.8070804 at oracle.com> >>> Content-Type: text/plain; charset=utf-8; format=flowed >>> >>> On 04/01/15 18:58, Peter Levart wrote: >>>> Hi, >>>> >>>> While investigating the following issue: >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8029891 >>>> >>>> I noticed there's a bug in deserialization code of java.util.Hashtable >>>> (from day one probably): >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8068427 >>>> >>>> The fix is a trivial one-character replacement: '*' -> '/', but I also >>>> corrected some untruthful comments in the neighbourhood (which might >>>> have been true from day one, but are not any more): >>>> >>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ >>>> >>> >>> Hi Peter, >>> >>> I wonder whether there should be a guard against loadFactor being >>> zero/negative/NaN after line 1173, like in the constructor e.g. as >>> in lines >>> >>> 188 if (loadFactor <= 0 || Float.isNaN(loadFactor)) >>> 189 throw new IllegalArgumentException("Illegal Load: >>> "+loadFactor); >>> >>> if only to avoid division by zero. >>> >>> best regards, >>> >>> -- daniel >>> >>> >>>> >>>> >>>> Regards, Peter >>>> >>> >> > From konstantin.shefov at oracle.com Wed Jan 21 13:06:59 2015 From: konstantin.shefov at oracle.com (Konstantin Shefov) Date: Wed, 21 Jan 2015 16:06:59 +0300 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54B919ED.8080403@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <548EC2D7.6000604@oracle.com> <548EC534.2090803@oracle.com> <548ECDA4.9070002@oracle.com> <54B8ECD7.3010703@oracle.com> <54B8F630.4040608@oracle.com> <54B8F90A.7010402@oracle.com> <54B908FA.9020405@oracle.com> <54B919ED.8080403@oracle.com> Message-ID: <54BFA473.5070905@oracle.com> Hi, Chris I have done a pre-integration testing for this fix on Linux, Windows, Mac and Solaris x64 and sparc. I have run all regression tests for core libraries, and no new failures appeared. Can I push this change now or should I wait for an approve by one more person? Thanks -Konstantin On 16.01.2015 17:02, Chris Hegarty wrote: > On 16/01/15 12:50, Konstantin Shefov wrote: >> Hi Chris, Alan, thank you for reviewing. >> >> I have made a new webrev >> http://cr.openjdk.java.net/~kshefov/6933879/webrev.01 >> I have removed ":" and added a test case. > > This looks ok to me. > > -Chris. > >> -Konstantin >> >> 16.01.2015 14:42, Chris Hegarty ?????: >>> On 16/01/15 11:29, Alan Bateman wrote: >>>> On 16/01/2015 10:49, Chris Hegarty wrote: >>>>> : >>>>> >>>>> I don't see any reason to update the spec here, given that the set of >>>>> allowable character is not clearly defined in the relevant RFC's. >>>>> >>>> I think we need to create a bug to look into this more. Inet6Address >>>> allows the scope of be any String but URI puts restrictions on what is >>>> legal. The proposed change updates the list of allowed characters >>>> but I >>>> assume it will need to re-visited again as systems with more exotic >>>> interface names are encountered. >>> >>> Yes that is possible, but to date this is the first time I've seen >>> this come up. >>> >>> Given the lack of clear specification in the relevant RFC's (relating >>> to scope ids ), I don't think we should be making too many >>> specification changes in this area. But I agree with your comment, URI >>> has a restriction that it not clearly specified in the Java spec. >>> >>> -Chris. >> From chris.hegarty at oracle.com Wed Jan 21 13:19:04 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 21 Jan 2015 13:19:04 +0000 Subject: RFR 8059324: orElseThrow has different signatures for OptionalPrimitive and Optional In-Reply-To: References: Message-ID: <54BFA748.4040900@oracle.com> On 20/01/15 16:56, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8059324-Optional-orElseThrow-signatures/webrev/ Looks ok to me. > The patch fixes up the signatures of Optional{Int, Long, Double}.orElseThrow to be consistent with Optional.orElseThrow. > > I believe this is a compatible change since the classes are final. That is my understanding also. And you are expanding the set of acceptable suppliers. -Chris. > I will also file a CCC. > > Paul. > > diff -r 8b8b84397c5a src/java.base/share/classes/java/util/OptionalDouble.java > --- a/src/java.base/share/classes/java/util/OptionalDouble.java Tue Jan 20 16:35:19 2015 +0000 > +++ b/src/java.base/share/classes/java/util/OptionalDouble.java Tue Jan 20 16:54:07 2015 +0000 > @@ -182,7 +182,7 @@ > * @throws NullPointerException if no value is present and > * {@code exceptionSupplier} is null > */ > - public double orElseThrow(Supplier exceptionSupplier) throws X { > + public double orElseThrow(Supplier exceptionSupplier) throws X { > if (isPresent) { > return value; > } else { > diff -r 8b8b84397c5a src/java.base/share/classes/java/util/OptionalInt.java > --- a/src/java.base/share/classes/java/util/OptionalInt.java Tue Jan 20 16:35:19 2015 +0000 > +++ b/src/java.base/share/classes/java/util/OptionalInt.java Tue Jan 20 16:54:07 2015 +0000 > @@ -182,7 +182,7 @@ > * @throws NullPointerException if no value is present and > * {@code exceptionSupplier} is null > */ > - public int orElseThrow(Supplier exceptionSupplier) throws X { > + public int orElseThrow(Supplier exceptionSupplier) throws X { > if (isPresent) { > return value; > } else { > diff -r 8b8b84397c5a src/java.base/share/classes/java/util/OptionalLong.java > --- a/src/java.base/share/classes/java/util/OptionalLong.java Tue Jan 20 16:35:19 2015 +0000 > +++ b/src/java.base/share/classes/java/util/OptionalLong.java Tue Jan 20 16:54:07 2015 +0000 > @@ -182,7 +182,7 @@ > * @throws NullPointerException if no value is present and > * {@code exceptionSupplier} is null > */ > - public long orElseThrow(Supplier exceptionSupplier) throws X { > + public long orElseThrow(Supplier exceptionSupplier) throws X { > if (isPresent) { > return value; > } else { > From chris.hegarty at oracle.com Wed Jan 21 15:11:48 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 21 Jan 2015 15:11:48 +0000 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table with wrong capacity In-Reply-To: <54BFA3DF.4090201@gmail.com> References: <7b2072fe0674e465f86518a9f25cef6d@sonic.net> <54AFAA61.30008@gmail.com> <54BFA3DF.4090201@gmail.com> Message-ID: <54BFC1B4.2040509@oracle.com> This looks good to me Peter. -Chris. On 21/01/15 13:04, Peter Levart wrote: > Hi Martin and others, > > I have also modified the test per Martin's suggestion to factor out the > serialClone() method. Here's the latest webrev: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.03/ > > Is this ok now? > > Regards, Peter > > On 01/09/2015 11:16 AM, Peter Levart wrote: >> On 01/05/2015 05:52 PM, Mike Duigou wrote: >>> Well spotted Peter. The change looks good though I wonder if it >>> should be: >>> >>> int length = (int)((elements + elements / 20) / loadFactor) + 3; >>> >>> FYI, regarding Daniel's suggestion: When similar invariant checks >>> were added to the HashMap deserialization method we found code which >>> relied upon the illegal values. In some cases the serialized HashMaps >>> were created by alternative serialization implementations which >>> included illegal, but until the checks were added, "harmless" values. >>> >>> The invariant checks should still be added though. It might even be >>> worth adding checks that the other deserialized values are in valid >>> ranges. >>> >>> Mike >> >> Hi Mike and others, >> >> Yes, your suggested length computation is more in "spirit" with the >> comment above that states: "Compute new length with a bit of room 5% >> to grow...", since it takes loadFactor into account for that >> additional 5% too. I changed it to your suggested expression. >> >> Here's new webrev: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.02/ >> >> In addition to valid loadFactor, # of elements is checked to be >> non-negative. The original length is just "repaired" if it appears to >> be less than the enforced auto-growth invariant of Hashtable. >> >> I also expanded the test to exercise more combinations of # of >> elements and loadFactor. Here's what gets printed with current >> Hashtable implementation: >> >> ser. deser. >> size load lentgh length valid range ok? >> ------- ----- ------- ------- ----------------- ------ >> 10 0.15 127 4 67... 134 NOT-OK >> 10 0.50 31 8 21... 42 NOT-OK >> 10 0.75 15 10 14... 28 NOT-OK >> 10 1.00 15 13 11... 22 OK >> 10 2.50 7 7 5... 10 OK >> 50 0.15 511 12 334... 668 NOT-OK >> 50 0.50 127 30 101... 202 NOT-OK >> 50 0.75 127 42 67... 134 NOT-OK >> 50 1.00 63 55 51... 102 OK >> 50 2.50 31 31 21... 42 OK >> 500 0.15 4095 103 3334... 6668 NOT-OK >> 500 0.50 1023 278 1001... 2002 NOT-OK >> 500 0.75 1023 403 667... 1334 NOT-OK >> 500 1.00 511 511 501... 1002 OK >> 500 2.50 255 255 201... 402 OK >> 5000 0.15 65535 1003 33334... 66668 NOT-OK >> 5000 0.50 16383 2753 10001... 20002 NOT-OK >> 5000 0.75 8191 4003 6667... 13334 NOT-OK >> 5000 1.00 8191 5253 5001... 10002 OK >> 5000 2.50 2047 2047 2001... 4002 OK >> >> >> With patched Hashtable, the test passes: >> >> ser. deser. >> size load lentgh length valid range ok? >> ------- ----- ------- ------- ----------------- ------ >> 10 0.15 127 69 67... 134 OK >> 10 0.50 31 23 21... 42 OK >> 10 0.75 15 15 14... 28 OK >> 10 1.00 15 13 11... 22 OK >> 10 2.50 7 7 5... 10 OK >> 50 0.15 511 349 334... 668 OK >> 50 0.50 127 107 101... 202 OK >> 50 0.75 127 71 67... 134 OK >> 50 1.00 63 55 51... 102 OK >> 50 2.50 31 23 21... 42 OK >> 500 0.15 4095 3501 3334... 6668 OK >> 500 0.50 1023 1023 1001... 2002 OK >> 500 0.75 1023 703 667... 1334 OK >> 500 1.00 511 511 501... 1002 OK >> 500 2.50 255 213 201... 402 OK >> 5000 0.15 65535 35003 33334... 66668 OK >> 5000 0.50 16383 10503 10001... 20002 OK >> 5000 0.75 8191 7003 6667... 13334 OK >> 5000 1.00 8191 5253 5001... 10002 OK >> 5000 2.50 2047 2047 2001... 4002 OK >> >> >> >> Regards, Peter >> >>> >>> On 2015-01-05 07:48, core-libs-dev-request at openjdk.java.net wrote: >>>> >>>> Today's Topics: >>>> >>>> 2. Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes >>>> table with wrong capacity (Daniel Fuchs) >>> >>> >>>> Message: 2 >>>> Date: Mon, 05 Jan 2015 15:52:55 +0100 >>>> From: Daniel Fuchs >>>> To: Peter Levart , core-libs-dev >>>> >>>> Subject: Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes >>>> table with wrong capacity >>>> Message-ID: <54AAA547.8070804 at oracle.com> >>>> Content-Type: text/plain; charset=utf-8; format=flowed >>>> >>>> On 04/01/15 18:58, Peter Levart wrote: >>>>> Hi, >>>>> >>>>> While investigating the following issue: >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-8029891 >>>>> >>>>> I noticed there's a bug in deserialization code of java.util.Hashtable >>>>> (from day one probably): >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-8068427 >>>>> >>>>> The fix is a trivial one-character replacement: '*' -> '/', but I also >>>>> corrected some untruthful comments in the neighbourhood (which might >>>>> have been true from day one, but are not any more): >>>>> >>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ >>>>> >>>> >>>> Hi Peter, >>>> >>>> I wonder whether there should be a guard against loadFactor being >>>> zero/negative/NaN after line 1173, like in the constructor e.g. as >>>> in lines >>>> >>>> 188 if (loadFactor <= 0 || Float.isNaN(loadFactor)) >>>> 189 throw new IllegalArgumentException("Illegal Load: >>>> "+loadFactor); >>>> >>>> if only to avoid division by zero. >>>> >>>> best regards, >>>> >>>> -- daniel >>>> >>>> >>>>> >>>>> >>>>> Regards, Peter >>>>> >>>> >>> >> > From konstantin.shefov at oracle.com Wed Jan 21 15:29:51 2015 From: konstantin.shefov at oracle.com (Konstantin Shefov) Date: Wed, 21 Jan 2015 18:29:51 +0300 Subject: [8u60] Request for approval: JDK-8067344: Adjust java/lang/invoke/LFCaching/LFGarbageCollectedTest.java for recent changes in java.lang.invoke Message-ID: <54BFC5EF.6000708@oracle.com> Hello, Please approve the direct backport of the test bug fix to 8u60 The webrev is slightly different from that for JDK 9, but only in line numbers, all the rest is just the same. The bug: https://bugs.openjdk.java.net/browse/JDK-8067344 The 8u60 webrev: http://cr.openjdk.java.net/~kshefov/8067344_8u/webrev.00/ JDK 9 changeset: http://hg.openjdk.java.net/jdk9/dev/jdk/rev/75c351a01d3c Review thread: http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-December/030452.html Thanks -Konstantin From chris.hegarty at oracle.com Wed Jan 21 16:27:32 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 21 Jan 2015 16:27:32 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54BEB91A.4020603@gmail.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54BEB91A.4020603@gmail.com> Message-ID: <54BFD374.5080803@oracle.com> On 20/01/15 20:22, Peter Levart wrote: > Hi Chris and Peter, > > It just occurred to me that the following scheme would provide failure > atomicity for the whole Object regardless of whether readObject methods > are used or not for various classes in hierarchy: > > > - allocate uninitialized instance > - call default accessible constructor of the most specific > non-Serializable class > - deserialize (by calling readObject methods where provided) the fields > of all classes in hierarchy like normally > (up to this point, nothing is changed from what we have now) > - if deserialization fails anywhere, undo everything by setting all the > fields in the Serializable part of the hierarchy to default values (null > for references, 0 for primitives), abandon the object and propagate failure. I think this is a good idea, and I can prototype something to this affect. -Chris. > While deserializing, the object is in inconsistent state. If > deserialization fails, this state is rolled-back to uninitialized state. > finalize() can still get to the instance, but it will be uninitialized. > > > Peter > > On 01/14/2015 01:58 PM, Peter Firmstone wrote: >> >> Hi Chris, >> >> Sorry, no. >> >> Currently when an ObjectStreamClass is read in from the stream, the >> framework searches for the first zero arg constructor of a non >> serializable class and creates and instance of the class read and >> resolved from the stream, however it does so using a super class >> constructor. >> >> Then from the super class down, fields are read in and set in order >> for each class in the object's inheritance hierarchy. >> >> The alternative I propose, doesn't create the instance, instead it >> reads the fields from the stream, one by one and without instantiating >> them, if they are newly read objects, stores them temporarily into >> byte [] arrays in a Map with reference handle keys, otherwise it just >> holds the reference handle. >> >> What it does next is wrap this information into a caller sensitive >> api, GetFields or ReadSerial instance, that is passed as a constructor >> parameter to the child class serial constructor. >> >> The child class checks invariants and reads each field it needs using >> a static method prior to calling a superclass constructor, each class >> in the inheritance hierarchy for the object then checks its invariants >> until it gets to the first non serializable superclass. >> >> The benefit of this order is that each class is present in the thread >> security context, so protection domain security and invariants are >> enforced before instantiating an object. >> >> Hope this helps illuminate it a little better, regards, >> >> Peter. >> >> ----- Original message ----- >> > Peter F, >> > >> > I am still struggling with the basic concept of you proposal. Let me >> see >> > if I understand it correctly. Does the following describe a similar >> > scenario as you envisage: >> > >> > 1) For each Serializable type, T, in the deserialized types >> > hierarchy, starting with the top most ( closest to >> j.l.Object ), >> > >> > 1a) Read T's fields from the stream, fields >> > >> > 1b) validate(t, fields) // t will be null first time >> > >> > 1c) allocate a new instance of T, and assign to t >> > >> > 1d) set fields in t >> > >> > 2) Return t; >> > >> > So for each level in the hierarchy, an instance of a type is created >> > only after its invariants have been checked. This instance is then >> > passed to the next level so it can participate in that levels >> invariants >> > validation. >> > >> > If this scenario is along the same lines as yours, then I just don't >> see >> > how 1c above will always be possible. >> > >> > If we could somehow make the object caller sensitive until after >> > deserialization completes, then could avoid having to try to allocate >> > multiple instance down the hierarchy. >> > >> > -Chris. >> > >> > On 13/01/15 10:24, Peter Firmstone wrote: >> > > Could we use a static validator method and generate bytecode for >> > > constructors dynamically? >> > > >> > > The developer can optionally implement the constructors. >> > > >> > > static GetField invariantCheck(GetField f); >> > > >> > > Create a caller sensitive GetField implementation and add a two new >> > > methods to GetField: >> > > >> > > abstract Object createSuper(); // to access superclass object methods >> > > for inavariant checking. >> > > >> > > abstract Class getType(String name); >> > > >> > > Set fields from within constructors. >> > > >> > > The generated constructors are straight forward: >> > > >> > > 1. Call static method. >> > > 2. Call super class constructor with result from static method. >> > > 3. Set final fields >> > > 4. How to set transient fields, implement a private method called >> from >> > > within the constructor? >> > > >> > > Require a permission to extend GetField? >> > > >> > From vladimir.x.ivanov at oracle.com Wed Jan 21 16:25:18 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Wed, 21 Jan 2015 19:25:18 +0300 Subject: [9] RFR (M): 8069591: Customize LambdaForms which are invoked using MH.invoke/invokeExact Message-ID: <54BFD2EE.3060909@oracle.com> http://cr.openjdk.java.net/~vlivanov/8069591/webrev.00/ https://bugs.openjdk.java.net/browse/JDK-8069591 Overhead of non-inlined MH.invoke/invokeExact calls significantly increased with LambdaForm sharing. The cause is JIT compiler can't produce a single nmethod for the whole MethodHandle chain, so the execution is spread around numerous nmethods (1 per each MethodHandle in the chain). The longer the chain the larger overhead. The fix is to customize LambdaForms (create a dedicated LambdaForm for a MethodHandle). Per-MethodHandle count is introduced, which is incremented every time a MethodHandle is invoked using MethodHandle.invoke/invokeExact. Once CUSTOMIZE_THRESHOLD is reached for a particular MethodHandle, it's LambdaForm is substituted with a customized one, which has it's MethodHandle embedded. It allows JIT to see actual MethodHandle during compilation and produce more efficient code. This fix completely recovers Gbemu peak performance to pre-LambdaForm sharing level. Testing: jck (api/java_lang/invoke), jdk/java/lang/invoke, nashorn tests, nashorn/octane Thanks! Best regards, Vladimir Ivanov From martinrb at google.com Wed Jan 21 16:44:50 2015 From: martinrb at google.com (Martin Buchholz) Date: Wed, 21 Jan 2015 08:44:50 -0800 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table with wrong capacity In-Reply-To: <54BFA3DF.4090201@gmail.com> References: <7b2072fe0674e465f86518a9f25cef6d@sonic.net> <54AFAA61.30008@gmail.com> <54BFA3DF.4090201@gmail.com> Message-ID: Looks good to me! On Wed, Jan 21, 2015 at 5:04 AM, Peter Levart wrote: > Hi Martin and others, > > I have also modified the test per Martin's suggestion to factor out the > serialClone() method. Here's the latest webrev: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.03/ > > Is this ok now? > > Regards, Peter > > > On 01/09/2015 11:16 AM, Peter Levart wrote: > >> On 01/05/2015 05:52 PM, Mike Duigou wrote: >> >>> Well spotted Peter. The change looks good though I wonder if it should >>> be: >>> >>> int length = (int)((elements + elements / 20) / loadFactor) + 3; >>> >>> FYI, regarding Daniel's suggestion: When similar invariant checks were >>> added to the HashMap deserialization method we found code which relied upon >>> the illegal values. In some cases the serialized HashMaps were created by >>> alternative serialization implementations which included illegal, but until >>> the checks were added, "harmless" values. >>> >>> The invariant checks should still be added though. It might even be >>> worth adding checks that the other deserialized values are in valid ranges. >>> >>> Mike >>> >> >> Hi Mike and others, >> >> Yes, your suggested length computation is more in "spirit" with the >> comment above that states: "Compute new length with a bit of room 5% to >> grow...", since it takes loadFactor into account for that additional 5% >> too. I changed it to your suggested expression. >> >> Here's new webrev: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.02/ >> >> In addition to valid loadFactor, # of elements is checked to be >> non-negative. The original length is just "repaired" if it appears to be >> less than the enforced auto-growth invariant of Hashtable. >> >> I also expanded the test to exercise more combinations of # of elements >> and loadFactor. Here's what gets printed with current Hashtable >> implementation: >> >> ser. deser. >> size load lentgh length valid range ok? >> ------- ----- ------- ------- ----------------- ------ >> 10 0.15 127 4 67... 134 NOT-OK >> 10 0.50 31 8 21... 42 NOT-OK >> 10 0.75 15 10 14... 28 NOT-OK >> 10 1.00 15 13 11... 22 OK >> 10 2.50 7 7 5... 10 OK >> 50 0.15 511 12 334... 668 NOT-OK >> 50 0.50 127 30 101... 202 NOT-OK >> 50 0.75 127 42 67... 134 NOT-OK >> 50 1.00 63 55 51... 102 OK >> 50 2.50 31 31 21... 42 OK >> 500 0.15 4095 103 3334... 6668 NOT-OK >> 500 0.50 1023 278 1001... 2002 NOT-OK >> 500 0.75 1023 403 667... 1334 NOT-OK >> 500 1.00 511 511 501... 1002 OK >> 500 2.50 255 255 201... 402 OK >> 5000 0.15 65535 1003 33334... 66668 NOT-OK >> 5000 0.50 16383 2753 10001... 20002 NOT-OK >> 5000 0.75 8191 4003 6667... 13334 NOT-OK >> 5000 1.00 8191 5253 5001... 10002 OK >> 5000 2.50 2047 2047 2001... 4002 OK >> >> >> With patched Hashtable, the test passes: >> >> ser. deser. >> size load lentgh length valid range ok? >> ------- ----- ------- ------- ----------------- ------ >> 10 0.15 127 69 67... 134 OK >> 10 0.50 31 23 21... 42 OK >> 10 0.75 15 15 14... 28 OK >> 10 1.00 15 13 11... 22 OK >> 10 2.50 7 7 5... 10 OK >> 50 0.15 511 349 334... 668 OK >> 50 0.50 127 107 101... 202 OK >> 50 0.75 127 71 67... 134 OK >> 50 1.00 63 55 51... 102 OK >> 50 2.50 31 23 21... 42 OK >> 500 0.15 4095 3501 3334... 6668 OK >> 500 0.50 1023 1023 1001... 2002 OK >> 500 0.75 1023 703 667... 1334 OK >> 500 1.00 511 511 501... 1002 OK >> 500 2.50 255 213 201... 402 OK >> 5000 0.15 65535 35003 33334... 66668 OK >> 5000 0.50 16383 10503 10001... 20002 OK >> 5000 0.75 8191 7003 6667... 13334 OK >> 5000 1.00 8191 5253 5001... 10002 OK >> 5000 2.50 2047 2047 2001... 4002 OK >> >> >> >> Regards, Peter >> >> >>> On 2015-01-05 07:48, core-libs-dev-request at openjdk.java.net wrote: >>> >>>> >>>> Today's Topics: >>>> >>>> 2. Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes >>>> table with wrong capacity (Daniel Fuchs) >>>> >>> >>> >>> Message: 2 >>>> Date: Mon, 05 Jan 2015 15:52:55 +0100 >>>> From: Daniel Fuchs >>>> To: Peter Levart , core-libs-dev >>>> >>>> Subject: Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes >>>> table with wrong capacity >>>> Message-ID: <54AAA547.8070804 at oracle.com> >>>> Content-Type: text/plain; charset=utf-8; format=flowed >>>> >>>> On 04/01/15 18:58, Peter Levart wrote: >>>> >>>>> Hi, >>>>> >>>>> While investigating the following issue: >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-8029891 >>>>> >>>>> I noticed there's a bug in deserialization code of java.util.Hashtable >>>>> (from day one probably): >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-8068427 >>>>> >>>>> The fix is a trivial one-character replacement: '*' -> '/', but I also >>>>> corrected some untruthful comments in the neighbourhood (which might >>>>> have been true from day one, but are not any more): >>>>> >>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable. >>>>> 8068427/webrev.01/ >>>>> >>>> >>>> Hi Peter, >>>> >>>> I wonder whether there should be a guard against loadFactor being >>>> zero/negative/NaN after line 1173, like in the constructor e.g. as >>>> in lines >>>> >>>> 188 if (loadFactor <= 0 || Float.isNaN(loadFactor)) >>>> 189 throw new IllegalArgumentException("Illegal Load: >>>> "+loadFactor); >>>> >>>> if only to avoid division by zero. >>>> >>>> best regards, >>>> >>>> -- daniel >>>> >>>> >>>> >>>>> >>>>> Regards, Peter >>>>> >>>>> >>>> >>> >> > From forax at univ-mlv.fr Wed Jan 21 17:31:05 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 21 Jan 2015 18:31:05 +0100 Subject: [9] RFR (M): 8069591: Customize LambdaForms which are invoked using MH.invoke/invokeExact In-Reply-To: <54BFD2EE.3060909@oracle.com> References: <54BFD2EE.3060909@oracle.com> Message-ID: <54BFE259.1090402@univ-mlv.fr> Hi Vladimir, in Invokers.java, I think that checkCustomized should take an Object and not a MethodHandle exactly like getCallSiteTarget takes an Object and not a CallSite. in MethodHandle.java, customizationCount is declared as a byte and there is no check that the CUSTOMIZE_THRESHOLD is not greater than 127. cheers, R?mi On 01/21/2015 05:25 PM, Vladimir Ivanov wrote: > http://cr.openjdk.java.net/~vlivanov/8069591/webrev.00/ > https://bugs.openjdk.java.net/browse/JDK-8069591 > > Overhead of non-inlined MH.invoke/invokeExact calls significantly > increased with LambdaForm sharing. The cause is JIT compiler can't > produce a single nmethod for the whole MethodHandle chain, so the > execution is spread around numerous nmethods (1 per each MethodHandle > in the chain). The longer the chain the larger overhead. > > The fix is to customize LambdaForms (create a dedicated LambdaForm for > a MethodHandle). Per-MethodHandle count is introduced, which is > incremented every time a MethodHandle is invoked using > MethodHandle.invoke/invokeExact. Once CUSTOMIZE_THRESHOLD is reached > for a particular MethodHandle, it's LambdaForm is substituted with a > customized one, which has it's MethodHandle embedded. It allows JIT to > see actual MethodHandle during compilation and produce more efficient > code. > > This fix completely recovers Gbemu peak performance to pre-LambdaForm > sharing level. > > Testing: jck (api/java_lang/invoke), jdk/java/lang/invoke, nashorn > tests, nashorn/octane > > Thanks! > > Best regards, > Vladimir Ivanov > _______________________________________________ > mlvm-dev mailing list > mlvm-dev at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev From ariel at weisberg.ws Wed Jan 21 17:34:19 2015 From: ariel at weisberg.ws (Ariel Weisberg) Date: Wed, 21 Jan 2015 12:34:19 -0500 Subject: Expected vs. observed performance of java.util.zip.CRC32 in Java 7 and 8 Message-ID: <1421861659.3934810.216893077.7CB17890@webmail.messagingengine.com> Hi, Hopefully this is the right mailing list. I have some questions about the performance of java.util.zip.CRC32 in OpenJDK. I heard that CRC32 became an intrinsic in Java 8 that uses hardware support if available. I tried it out, but got some odd numbers and I am having trouble nailing down the difference between Java 7 and 8. I saw some mailing list traffic mentioning that Adler32 got the intrinsic treatment as well, but looking at the source I don't see any mention in C1. Maybe it is just non-obvious? I compared the JDK CRC32 and Adler32 with a pure java slicing by 8 implementation. I was not expecting to see 13 gigabytes/second per core. Certainly not in Java 7 or 8 without hardware support. I also was not expecting to see the performance so close for small sizes, but so far for large sizes. I would expect the throughput to level out quickly as you start checksumming larger sizes. The parameter is the number of bytes being checksummed. JMH code is at http://pastebin.com/y96EFwcL I ran on a Haswell quad-core macbook pro. I can tell the correct JDK is running by having setup print the java.version property. Apologies for the formatting disaster. I attached the output as a text file so the columns line up. If you could shed any light on what the performance should be it would help when choosing whether to use the JDK CRC32 implementation or another depending on the currently runtime. Thanks, Ariel jdk1.7.0_71 [java] Benchmark (byteSize) Mode Samples Score Error Units [java] o.a.c.t.m.Sample.Adler32Array 128 thrpt 6 9484470.705 ? 3544496.362 ops/s [java] o.a.c.t.m.Sample.Adler32Array 512 thrpt 6 7553107.572 ? 2017788.822 ops/s [java] o.a.c.t.m.Sample.Adler32Array 1024 thrpt 6 5925103.324 ? 1237581.263 ops/s [java] o.a.c.t.m.Sample.Adler32Array 1048576 thrpt 6 12958.857 ? 313.405 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 128 thrpt 6 8675457.907 ? 2920818.797 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 512 thrpt 6 6906837.280 ? 949573.737 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 1024 thrpt 6 5537421.656 ? 658220.086 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 1048576 thrpt 6 13103.481 ? 400.833 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 128 thrpt 6 11013067.959 ? 688587.910 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 512 thrpt 6 2991944.703 ? 72920.216 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 1024 thrpt 6 1516586.386 ? 68061.147 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 1048576 thrpt 6 1483.413 ? 88.246 ops/s jdk1.8.0_25 [java] Benchmark (byteSize) Mode Samples Score Error Units [java] o.a.c.t.m.Sample.Adler32Array 128 thrpt 6 9216616.134 ? 4037583.644 ops/s [java] o.a.c.t.m.Sample.Adler32Array 512 thrpt 6 7470492.783 ? 2059216.459 ops/s [java] o.a.c.t.m.Sample.Adler32Array 1024 thrpt 6 5792188.710 ? 1363845.066 ops/s [java] o.a.c.t.m.Sample.Adler32Array 1048576 thrpt 6 12273.582 ? 1077.855 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 128 thrpt 6 25619960.160 ? 34966219.926 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 512 thrpt 6 13858414.869 ? 9113069.866 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 1024 thrpt 6 8781575.118 ? 3524581.216 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 1048576 thrpt 6 12631.141 ? 1340.355 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 128 thrpt 6 10772800.588 ? 398939.396 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 512 thrpt 6 2917829.545 ? 96841.740 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 1024 thrpt 6 1541478.127 ? 27234.072 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 1048576 thrpt 6 1526.471 ? 27.515 ops/s jdk1.8.0_25 with -XX:-UseCLMUL -XX:-UseCRC32Intrinsics [java] Benchmark (byteSize) Mode Samples Score Error Units [java] o.a.c.t.m.Sample.Adler32Array 128 thrpt 6 9469461.434 ? 3770031.916 ops/s [java] o.a.c.t.m.Sample.Adler32Array 512 thrpt 6 7675500.241 ? 2044377.592 ops/s [java] o.a.c.t.m.Sample.Adler32Array 1024 thrpt 6 5693849.558 ? 1477732.642 ops/s [java] o.a.c.t.m.Sample.Adler32Array 1048576 thrpt 6 13110.626 ? 386.136 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 128 thrpt 6 8850789.721 ? 3204385.552 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 512 thrpt 6 7118956.038 ? 1004576.517 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 1024 thrpt 6 5590757.571 ? 448188.497 ops/s [java] o.a.c.t.m.Sample.CRC32OriginalArray 1048576 thrpt 6 13270.839 ? 598.086 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 128 thrpt 6 11351558.878 ? 546114.618 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 512 thrpt 6 3011605.270 ? 217000.560 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 1024 thrpt 6 1529659.010 ? 70247.359 ops/s [java] o.a.c.t.m.Sample.PureJavaCrc32Array 1048576 thrpt 6 1466.362 ? 211.293 ops/s -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: benchmark.txt URL: From aph at redhat.com Wed Jan 21 17:40:48 2015 From: aph at redhat.com (Andrew Haley) Date: Wed, 21 Jan 2015 17:40:48 +0000 Subject: Expected vs. observed performance of java.util.zip.CRC32 in Java 7 and 8 In-Reply-To: <1421861659.3934810.216893077.7CB17890@webmail.messagingengine.com> References: <1421861659.3934810.216893077.7CB17890@webmail.messagingengine.com> Message-ID: <54BFE4A0.5080208@redhat.com> On 01/21/2015 05:34 PM, Ariel Weisberg wrote: > If you could shed any light on what the performance should be it would > help when choosing whether to use the JDK CRC32 implementation or > another depending on the currently runtime. Please normalize the output to bytes/s; it's very confusing. Andrew. From john.r.rose at oracle.com Wed Jan 21 19:30:28 2015 From: john.r.rose at oracle.com (John Rose) Date: Wed, 21 Jan 2015 11:30:28 -0800 Subject: [9] RFR (M): 8069591: Customize LambdaForms which are invoked using MH.invoke/invokeExact In-Reply-To: <54BFE259.1090402@univ-mlv.fr> References: <54BFD2EE.3060909@oracle.com> <54BFE259.1090402@univ-mlv.fr> Message-ID: <3B4D19E0-8DA6-4FE1-BD77-E12E8BCF15EC@oracle.com> On Jan 21, 2015, at 9:31 AM, Remi Forax wrote: > > in Invokers.java, I think that checkCustomized should take an Object and not a MethodHandle > exactly like getCallSiteTarget takes an Object and not a CallSite. The use of erased types (any ref => Object) in the MH runtime is an artifact of bootstrapping difficulties, early in the project. I hope it is not necessary any more. That said, I agree that the pattern should be consistent. Vladimir, would you please file a tracking bug for this cleanup, to change MH library functions to use stronger types instead of Object? > in MethodHandle.java, customizationCount is declared as a byte and there is no check that > the CUSTOMIZE_THRESHOLD is not greater than 127. Yes. Also, the maybeCustomize method has a race condition that could cause the counter to wrap. It shouldn't use "+=1" to increment; it should load the old counter value, test it, increment it (in a local), and then store the updated value. That is also one possible place to deal with jumbo CUSTOMIZE_THRESHOLD values. ? John From john.r.rose at oracle.com Wed Jan 21 19:42:36 2015 From: john.r.rose at oracle.com (John Rose) Date: Wed, 21 Jan 2015 11:42:36 -0800 Subject: 8050820: Please add java.util.Optional.stream() to convert Optional to Stream In-Reply-To: References: <20150121100000.GA11988@e6430> <3CED45C1-A695-44E9-8E9A-F1D0990A9393@oracle.com> Message-ID: <3005029A-25AF-402B-9993-34754BFE54FB@oracle.com> On Jan 21, 2015, at 3:01 AM, Paul Sandoz wrote: > > I updated the webrev in place to be more consistent in the use of braces and better consistency for the primitive specializations: > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050820-Optional-stream/webrev/ > > I don't wanna make any more syntax-related changes unless i done something silly. It's often a good idea to make cleanups to code you touch, but there is a natural limit. If cleanups proliferate far away from the semantic changes, the two sorts of changes will begin to confuse each other, both during review and later on during backports or other change analysis. I think what you have done is acceptable from that viewpoint, since the changes can be separated "by eye" easily enough. You can count me as a Reviewer for this. ? John From john.r.rose at oracle.com Wed Jan 21 19:43:55 2015 From: john.r.rose at oracle.com (John Rose) Date: Wed, 21 Jan 2015 11:43:55 -0800 Subject: [8u60] RFR 8069302 Deprecate Unsafe monitor methods in JDK 8u release In-Reply-To: <7BA76D88-2443-4DF2-AC64-439FF0911CDA@oracle.com> References: <7BA76D88-2443-4DF2-AC64-439FF0911CDA@oracle.com> Message-ID: <48C8A30A-6393-4D6F-8ED6-E767AC93FFF8@oracle.com> +1; do it. > On Jan 21, 2015, at 2:35 AM, Paul Sandoz wrote: > > Hi, > > The Unsafe monitor methods monitorEnter. monitorExit and tryMonitorEnter were removed from JDK 9 [1]. > > I would like to mark those methods as deprecated in 8u60. I doubt no one will notice :-) but it's the correct thing to do. > > Paul. > > [1] http://hg.openjdk.java.net/jdk9/hs-rt/jdk/rev/0a0a0986400e From ariel at weisberg.ws Wed Jan 21 20:15:34 2015 From: ariel at weisberg.ws (Ariel Weisberg) Date: Wed, 21 Jan 2015 15:15:34 -0500 Subject: Expected vs. observed performance of java.util.zip.CRC32 in Java 7 and 8 In-Reply-To: <54BFE4A0.5080208@redhat.com> References: <1421861659.3934810.216893077.7CB17890@webmail.messagingengine.com> <54BFE4A0.5080208@redhat.com> Message-ID: <1421871334.180989.217049125.4E87A292@webmail.messagingengine.com> Hi, I cleaned up the results and converted to bytes and megabytes/second by multiplying byteSize with ops/sec. https://docs.google.com/spreadsheets/d/1SirRnW7JOqUvAZhx0b2HjJug8lQGHAUiDU8-6Zcb9Os/edit?usp=sharing Thanks, Ariel On Wed, Jan 21, 2015, at 12:40 PM, Andrew Haley wrote: > On 01/21/2015 05:34 PM, Ariel Weisberg wrote: > > If you could shed any light on what the performance should be it would > > help when choosing whether to use the JDK CRC32 implementation or > > another depending on the currently runtime. > > Please normalize the output to bytes/s; it's very confusing. > > Andrew. > From peter.firmstone at zeus.net.au Wed Jan 21 20:26:14 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Thu, 22 Jan 2015 06:26:14 +1000 Subject: Explicit Serialization API and Security In-Reply-To: <54BFD374.5080803@oracle.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54BEB91A.4020603@gmail.com> <54BFD374.5080803@oracle.com> Message-ID: <54C00B66.5090302@zeus.net.au> Don't forget that "null" may also be an invalid state. What I have learnt so far: 1. An attacker can craft a stream to obtain a reference to any object created during deserialization, using finalizers or circular links (there may be yet other undiscovered methods). 2. An attacker can craft a stream that deliberately doesn't satisfy invariants, in order to use an object to perform a function that wasn't intended by its developer. 3. Objects that interact with the stream directly using readObject et al, are often prone to DOS. Example, many objects read a length integer from the stream when creating an array or collection, without first validating it. 4. Objects that interact directly with the stream become an implicit part of the stream protocol. 5. Once you allow an object to be created, it's too late to invalidate invariants, unless the class is final and invariants are checked in every method call. 6. We need to be able to restrict classes used for deserialization to those we trust to check invariants properly (but we haven't provided a way for them to avoid object construction yet). 7. A static validator method can ONLY be used to check field invariants, not other objects and primitives that are read directly from the stream by an arbitrary Object during the process of deserialization. 8. The jvm can be modified to delay finalizer registration for deserialization. 9. Circular links can be disallowed. Ultimately however, all proposed changes add complexity, but when an object has been created with invalid invariants, an attacker will find a way. Thank you all for your time, this has been a very good discussion. Regards, Peter. On 22/01/2015 2:27 AM, Chris Hegarty wrote: > On 20/01/15 20:22, Peter Levart wrote: >> Hi Chris and Peter, >> >> It just occurred to me that the following scheme would provide failure >> atomicity for the whole Object regardless of whether readObject methods >> are used or not for various classes in hierarchy: >> >> >> - allocate uninitialized instance >> - call default accessible constructor of the most specific >> non-Serializable class >> - deserialize (by calling readObject methods where provided) the fields >> of all classes in hierarchy like normally >> (up to this point, nothing is changed from what we have now) >> - if deserialization fails anywhere, undo everything by setting all the >> fields in the Serializable part of the hierarchy to default values (null >> for references, 0 for primitives), abandon the object and propagate >> failure. > > I think this is a good idea, and I can prototype something to this > affect. > > -Chris. > > >> While deserializing, the object is in inconsistent state. If >> deserialization fails, this state is rolled-back to uninitialized state. >> finalize() can still get to the instance, but it will be uninitialized. >> >> >> Peter >> >> On 01/14/2015 01:58 PM, Peter Firmstone wrote: >>> >>> Hi Chris, >>> >>> Sorry, no. >>> >>> Currently when an ObjectStreamClass is read in from the stream, the >>> framework searches for the first zero arg constructor of a non >>> serializable class and creates and instance of the class read and >>> resolved from the stream, however it does so using a super class >>> constructor. >>> >>> Then from the super class down, fields are read in and set in order >>> for each class in the object's inheritance hierarchy. >>> >>> The alternative I propose, doesn't create the instance, instead it >>> reads the fields from the stream, one by one and without instantiating >>> them, if they are newly read objects, stores them temporarily into >>> byte [] arrays in a Map with reference handle keys, otherwise it just >>> holds the reference handle. >>> >>> What it does next is wrap this information into a caller sensitive >>> api, GetFields or ReadSerial instance, that is passed as a constructor >>> parameter to the child class serial constructor. >>> >>> The child class checks invariants and reads each field it needs using >>> a static method prior to calling a superclass constructor, each class >>> in the inheritance hierarchy for the object then checks its invariants >>> until it gets to the first non serializable superclass. >>> >>> The benefit of this order is that each class is present in the thread >>> security context, so protection domain security and invariants are >>> enforced before instantiating an object. >>> >>> Hope this helps illuminate it a little better, regards, >>> >>> Peter. >>> >>> ----- Original message ----- >>> > Peter F, >>> > >>> > I am still struggling with the basic concept of you proposal. Let me >>> see >>> > if I understand it correctly. Does the following describe a similar >>> > scenario as you envisage: >>> > >>> > 1) For each Serializable type, T, in the deserialized types >>> > hierarchy, starting with the top most ( closest to >>> j.l.Object ), >>> > >>> > 1a) Read T's fields from the stream, fields >>> > >>> > 1b) validate(t, fields) // t will be null first time >>> > >>> > 1c) allocate a new instance of T, and assign to t >>> > >>> > 1d) set fields in t >>> > >>> > 2) Return t; >>> > >>> > So for each level in the hierarchy, an instance of a type is created >>> > only after its invariants have been checked. This instance is then >>> > passed to the next level so it can participate in that levels >>> invariants >>> > validation. >>> > >>> > If this scenario is along the same lines as yours, then I just don't >>> see >>> > how 1c above will always be possible. >>> > >>> > If we could somehow make the object caller sensitive until after >>> > deserialization completes, then could avoid having to try to allocate >>> > multiple instance down the hierarchy. >>> > >>> > -Chris. >>> > >>> > On 13/01/15 10:24, Peter Firmstone wrote: >>> > > Could we use a static validator method and generate bytecode for >>> > > constructors dynamically? >>> > > >>> > > The developer can optionally implement the constructors. >>> > > >>> > > static GetField invariantCheck(GetField f); >>> > > >>> > > Create a caller sensitive GetField implementation and add a two new >>> > > methods to GetField: >>> > > >>> > > abstract Object createSuper(); // to access superclass object >>> methods >>> > > for inavariant checking. >>> > > >>> > > abstract Class getType(String name); >>> > > >>> > > Set fields from within constructors. >>> > > >>> > > The generated constructors are straight forward: >>> > > >>> > > 1. Call static method. >>> > > 2. Call super class constructor with result from static method. >>> > > 3. Set final fields >>> > > 4. How to set transient fields, implement a private method called >>> from >>> > > within the constructor? >>> > > >>> > > Require a permission to extend GetField? >>> > > >>> >> From peter.levart at gmail.com Wed Jan 21 20:43:07 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 21 Jan 2015 21:43:07 +0100 Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table with wrong capacity In-Reply-To: References: <7b2072fe0674e465f86518a9f25cef6d@sonic.net> <54AFAA61.30008@gmail.com> <54BFA3DF.4090201@gmail.com> Message-ID: <54C00F5B.6060005@gmail.com> Thanks Martin, Mike, Chris and Daniel, This has been pushed. Regards, Peter On 01/21/2015 05:44 PM, Martin Buchholz wrote: > Looks good to me! > > On Wed, Jan 21, 2015 at 5:04 AM, Peter Levart > wrote: > > Hi Martin and others, > > I have also modified the test per Martin's suggestion to factor > out the serialClone() method. Here's the latest webrev: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.03/ > > > Is this ok now? > > Regards, Peter > > > On 01/09/2015 11:16 AM, Peter Levart wrote: > > On 01/05/2015 05:52 PM, Mike Duigou wrote: > > Well spotted Peter. The change looks good though I wonder > if it should be: > > int length = (int)((elements + elements / 20) / > loadFactor) + 3; > > FYI, regarding Daniel's suggestion: When similar invariant > checks were added to the HashMap deserialization method we > found code which relied upon the illegal values. In some > cases the serialized HashMaps were created by alternative > serialization implementations which included illegal, but > until the checks were added, "harmless" values. > > The invariant checks should still be added though. It > might even be worth adding checks that the other > deserialized values are in valid ranges. > > Mike > > > Hi Mike and others, > > Yes, your suggested length computation is more in "spirit" > with the comment above that states: "Compute new length with a > bit of room 5% to grow...", since it takes loadFactor into > account for that additional 5% too. I changed it to your > suggested expression. > > Here's new webrev: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.02/ > > > In addition to valid loadFactor, # of elements is checked to > be non-negative. The original length is just "repaired" if it > appears to be less than the enforced auto-growth invariant of > Hashtable. > > I also expanded the test to exercise more combinations of # of > elements and loadFactor. Here's what gets printed with current > Hashtable implementation: > > ser. deser. > size load lentgh length valid range ok? > ------- ----- ------- ------- ----------------- ------ > 10 0.15 127 4 67... 134 NOT-OK > 10 0.50 31 8 21... 42 NOT-OK > 10 0.75 15 10 14... 28 NOT-OK > 10 1.00 15 13 11... 22 OK > 10 2.50 7 7 5... 10 OK > 50 0.15 511 12 334... 668 NOT-OK > 50 0.50 127 30 101... 202 NOT-OK > 50 0.75 127 42 67... 134 NOT-OK > 50 1.00 63 55 51... 102 OK > 50 2.50 31 31 21... 42 OK > 500 0.15 4095 103 3334... 6668 NOT-OK > 500 0.50 1023 278 1001... 2002 NOT-OK > 500 0.75 1023 403 667... 1334 NOT-OK > 500 1.00 511 511 501... 1002 OK > 500 2.50 255 255 201... 402 OK > 5000 0.15 65535 1003 33334... 66668 NOT-OK > 5000 0.50 16383 2753 10001... 20002 NOT-OK > 5000 0.75 8191 4003 6667... 13334 NOT-OK > 5000 1.00 8191 5253 5001... 10002 OK > 5000 2.50 2047 2047 2001... 4002 OK > > > With patched Hashtable, the test passes: > > ser. deser. > size load lentgh length valid range ok? > ------- ----- ------- ------- ----------------- ------ > 10 0.15 127 69 67... 134 OK > 10 0.50 31 23 21... 42 OK > 10 0.75 15 15 14... 28 OK > 10 1.00 15 13 11... 22 OK > 10 2.50 7 7 5... 10 OK > 50 0.15 511 349 334... 668 OK > 50 0.50 127 107 101... 202 OK > 50 0.75 127 71 67... 134 OK > 50 1.00 63 55 51... 102 OK > 50 2.50 31 23 21... 42 OK > 500 0.15 4095 3501 3334... 6668 OK > 500 0.50 1023 1023 1001... 2002 OK > 500 0.75 1023 703 667... 1334 OK > 500 1.00 511 511 501... 1002 OK > 500 2.50 255 213 201... 402 OK > 5000 0.15 65535 35003 33334... 66668 OK > 5000 0.50 16383 10503 10001... 20002 OK > 5000 0.75 8191 7003 6667... 13334 OK > 5000 1.00 8191 5253 5001... 10002 OK > 5000 2.50 2047 2047 2001... 4002 OK > > > > Regards, Peter > > > On 2015-01-05 07:48, > core-libs-dev-request at openjdk.java.net > wrote: > > > Today's Topics: > > 2. Re: RFR: JDK-8068427 Hashtable deserialization > reconstitutes > table with wrong capacity (Daniel Fuchs) > > > > Message: 2 > Date: Mon, 05 Jan 2015 15:52:55 +0100 > From: Daniel Fuchs > > To: Peter Levart >, core-libs-dev > > > Subject: Re: RFR: JDK-8068427 Hashtable > deserialization reconstitutes > table with wrong capacity > Message-ID: <54AAA547.8070804 at oracle.com > > > Content-Type: text/plain; charset=utf-8; format=flowed > > On 04/01/15 18:58, Peter Levart wrote: > > Hi, > > While investigating the following issue: > > https://bugs.openjdk.java.net/browse/JDK-8029891 > > I noticed there's a bug in deserialization code of > java.util.Hashtable > (from day one probably): > > https://bugs.openjdk.java.net/browse/JDK-8068427 > > The fix is a trivial one-character replacement: > '*' -> '/', but I also > corrected some untruthful comments in the > neighbourhood (which might > have been true from day one, but are not any more): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ > > > > > Hi Peter, > > I wonder whether there should be a guard against > loadFactor being > zero/negative/NaN after line 1173, like in the > constructor e.g. as > in lines > > 188 if (loadFactor <= 0 || > Float.isNaN(loadFactor)) > 189 throw new > IllegalArgumentException("Illegal Load: > "+loadFactor); > > if only to avoid division by zero. > > best regards, > > -- daniel > > > > > Regards, Peter > > > > > > From chris.hegarty at oracle.com Wed Jan 21 21:34:26 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 21 Jan 2015 21:34:26 +0000 Subject: 8050820: Please add java.util.Optional.stream() to convert Optional to Stream In-Reply-To: References: Message-ID: <2F58FA26-0D16-4119-831C-301C5C8E34F0@oracle.com> > On 20 Jan 2015, at 18:59, Paul Sandoz wrote: > > Hi, > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050820-Optional-stream/webrev/ > > This is one of those cases where i think adding a new method to Optional carries enough weight. Agreed. The changes look good to me. -Chris. > It can be really awkward to use Optional with Stream.flatMap to map to a stream of present values. > > It's not at all obvious that one can do "op.map(Stream::of).orElse(Stream.empty()))" so often one sees some contorted code of there in the wild. Hopefully adding Optional.stream (and to the primitive variants) will help reduce such contortions. > > A CCC will be filed. > > Thanks, > Paul. From david.lloyd at redhat.com Wed Jan 21 21:43:13 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 21 Jan 2015 15:43:13 -0600 Subject: Explicit Serialization API and Security In-Reply-To: <54C00B66.5090302@zeus.net.au> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54BEB91A.4020603@gmail.com> <54BFD374.5080803@oracle.com> <54C00B66.5090302@zeus.net.au> Message-ID: <54C01D71.9010901@redhat.com> At some point, the responsibility *must* fall on the author of the serializable class in question to avoid constructs which are exploitable - like finalizers, and classes that can have side-effects before validation can be completed. On 01/21/2015 02:26 PM, Peter Firmstone wrote: > Don't forget that "null" may also be an invalid state. > > What I have learnt so far: > > 1. An attacker can craft a stream to obtain a reference to any object > created during deserialization, using finalizers or circular links > (there may be yet other undiscovered methods). > 2. An attacker can craft a stream that deliberately doesn't satisfy > invariants, in order to use an object to perform a function that > wasn't intended by its developer. > 3. Objects that interact with the stream directly using readObject et > al, are often prone to DOS. Example, many objects read a length > integer from the stream when creating an array or collection, > without first validating it. > 4. Objects that interact directly with the stream become an implicit > part of the stream protocol. > 5. Once you allow an object to be created, it's too late to > invalidate invariants, unless the class is final and invariants > are checked in every method call. > 6. We need to be able to restrict classes used for deserialization to > those we trust to check invariants properly (but we haven't > provided a way for them to avoid object construction yet). > 7. A static validator method can ONLY be used to check field > invariants, not other objects and primitives that are read > directly from the stream by an arbitrary Object during the process > of deserialization. > 8. The jvm can be modified to delay finalizer registration for > deserialization. > 9. Circular links can be disallowed. > > Ultimately however, all proposed changes add complexity, but when an > object has been created with invalid invariants, an attacker will find a > way. > > Thank you all for your time, this has been a very good discussion. > > Regards, > > Peter. > > On 22/01/2015 2:27 AM, Chris Hegarty wrote: >> On 20/01/15 20:22, Peter Levart wrote: >>> Hi Chris and Peter, >>> >>> It just occurred to me that the following scheme would provide failure >>> atomicity for the whole Object regardless of whether readObject methods >>> are used or not for various classes in hierarchy: >>> >>> >>> - allocate uninitialized instance >>> - call default accessible constructor of the most specific >>> non-Serializable class >>> - deserialize (by calling readObject methods where provided) the fields >>> of all classes in hierarchy like normally >>> (up to this point, nothing is changed from what we have now) >>> - if deserialization fails anywhere, undo everything by setting all the >>> fields in the Serializable part of the hierarchy to default values (null >>> for references, 0 for primitives), abandon the object and propagate >>> failure. >> >> I think this is a good idea, and I can prototype something to this >> affect. >> >> -Chris. >> >> >>> While deserializing, the object is in inconsistent state. If >>> deserialization fails, this state is rolled-back to uninitialized state. >>> finalize() can still get to the instance, but it will be uninitialized. >>> >>> >>> Peter >>> >>> On 01/14/2015 01:58 PM, Peter Firmstone wrote: >>>> >>>> Hi Chris, >>>> >>>> Sorry, no. >>>> >>>> Currently when an ObjectStreamClass is read in from the stream, the >>>> framework searches for the first zero arg constructor of a non >>>> serializable class and creates and instance of the class read and >>>> resolved from the stream, however it does so using a super class >>>> constructor. >>>> >>>> Then from the super class down, fields are read in and set in order >>>> for each class in the object's inheritance hierarchy. >>>> >>>> The alternative I propose, doesn't create the instance, instead it >>>> reads the fields from the stream, one by one and without instantiating >>>> them, if they are newly read objects, stores them temporarily into >>>> byte [] arrays in a Map with reference handle keys, otherwise it just >>>> holds the reference handle. >>>> >>>> What it does next is wrap this information into a caller sensitive >>>> api, GetFields or ReadSerial instance, that is passed as a constructor >>>> parameter to the child class serial constructor. >>>> >>>> The child class checks invariants and reads each field it needs using >>>> a static method prior to calling a superclass constructor, each class >>>> in the inheritance hierarchy for the object then checks its invariants >>>> until it gets to the first non serializable superclass. >>>> >>>> The benefit of this order is that each class is present in the thread >>>> security context, so protection domain security and invariants are >>>> enforced before instantiating an object. >>>> >>>> Hope this helps illuminate it a little better, regards, >>>> >>>> Peter. >>>> >>>> ----- Original message ----- >>>> > Peter F, >>>> > >>>> > I am still struggling with the basic concept of you proposal. Let me >>>> see >>>> > if I understand it correctly. Does the following describe a similar >>>> > scenario as you envisage: >>>> > >>>> > 1) For each Serializable type, T, in the deserialized types >>>> > hierarchy, starting with the top most ( closest to >>>> j.l.Object ), >>>> > >>>> > 1a) Read T's fields from the stream, fields >>>> > >>>> > 1b) validate(t, fields) // t will be null first time >>>> > >>>> > 1c) allocate a new instance of T, and assign to t >>>> > >>>> > 1d) set fields in t >>>> > >>>> > 2) Return t; >>>> > >>>> > So for each level in the hierarchy, an instance of a type is created >>>> > only after its invariants have been checked. This instance is then >>>> > passed to the next level so it can participate in that levels >>>> invariants >>>> > validation. >>>> > >>>> > If this scenario is along the same lines as yours, then I just don't >>>> see >>>> > how 1c above will always be possible. >>>> > >>>> > If we could somehow make the object caller sensitive until after >>>> > deserialization completes, then could avoid having to try to allocate >>>> > multiple instance down the hierarchy. >>>> > >>>> > -Chris. >>>> > >>>> > On 13/01/15 10:24, Peter Firmstone wrote: >>>> > > Could we use a static validator method and generate bytecode for >>>> > > constructors dynamically? >>>> > > >>>> > > The developer can optionally implement the constructors. >>>> > > >>>> > > static GetField invariantCheck(GetField f); >>>> > > >>>> > > Create a caller sensitive GetField implementation and add a two new >>>> > > methods to GetField: >>>> > > >>>> > > abstract Object createSuper(); // to access superclass objectmethods >>>> > > for inavariant checking. >>>> > > >>>> > > abstract Class getType(String name); >>>> > > >>>> > > Set fields from within constructors. >>>> > > >>>> > > The generated constructors are straight forward: >>>> > > >>>> > > 1. Call static method. >>>> > > 2. Call super class constructor with result from static method. >>>> > > 3. Set final fields >>>> > > 4. How to set transient fields, implement a private method called >>>> from >>>> > > within the constructor? >>>> > > >>>> > > Require a permission to extend GetField? >>>> > > >>>> >>> > -- - DML From nathan.a.clement at hotmail.com Wed Jan 21 22:37:20 2015 From: nathan.a.clement at hotmail.com (Nathan Clement) Date: Thu, 22 Jan 2015 09:37:20 +1100 Subject: [PATCH] JDK-8054565: FilterOutputStream.close may throw IOException if called twice and underlying flush or close fails In-Reply-To: <549826FE.1060600@oracle.com> References: , <54807258.4040303@oracle.com> , <20141205002430.000060c9.ecki@zusammenkunft.net>, <2C4F3368-64DC-4F8B-9923-0C08999C8C5B@oracle.com>, <20141205015827.00007ac6.ecki@zusammenkunft.net>, , <8F896CE2-E268-418F-96CA-0557C1C258B3@oracle.com>, <5481BB6C.1040700@oracle.com> <5481C3DB.70303@gmail.com>, <5481CA8C.7050100@oracle.com> <5481EF01.6060009@gmail.com>, , , <548EB053.2040908@oracle.com>,<549826FE.1060600@oracle.com> Message-ID: Hi Chris, Thanks for your help on this one! Will this be backported to JDK 8, or is that not allowed because of the spec change? Thanks, Nathan > Date: Mon, 22 Dec 2014 14:13:18 +0000 > From: chris.hegarty at oracle.com > To: core-libs-dev at openjdk.java.net; nathan.a.clement at hotmail.com > Subject: Re: [PATCH] JDK-8054565: FilterOutputStream.close may throw IOException if called twice and underlying flush or close fails > > To close the loop; find below the complete reviewed changes. > > http://cr.openjdk.java.net/~chegar/8054565/webrev.00/webrev/ > > Since there is a minor spec change, I will submit a CCC to cover it. > Unless any unforeseen issues arise from the CCC, you can assume that > this is final. > > -Chris. From huizhe.wang at oracle.com Thu Jan 22 00:45:37 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Wed, 21 Jan 2015 16:45:37 -0800 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: <54BEA3C6.6090501@oracle.com> References: <54934601.4010104@oracle.com> <54B96BDF.7090204@oracle.com> <54B9C9EA.7050407@oracle.com> <5A817A94-0176-4613-91F2-9B298C1DA6AA@oracle.com> <54BEA3C6.6090501@oracle.com> Message-ID: <54C04831.3060105@oracle.com> Thanks Lance for pointing me to Joe's -Xlint:all email thread. I re-compiled with -Xlint:all and noticed 7 warnings in this webrev. I've fixed them with a new webrev: New webrev: http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev01/ Below are the details. Refer to the previous webrev: http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ 1. XPathExpressionImpl.java:143 and XPathImpl.java:213 These warnings are fixed by having the getXPathResult method in XPathImpUtil returning the type instead 2. XPathExpressionImpl.java:165: warning: [rawtypes] found raw type: XPathEvaluationResult similarly, XPathImpl.java:222 and XPathImpl.java:235 Changed to return XPathEvaluationResult 3. XPath.java:359, XPath.java:454, XPathExpression.java:252 and XPathExpression.java:344 These are warnings from the default methods, casting the result of existing (old) methods. Use type.cast instead. 4. XPathNodesImpl.java:97: warning: [cast] redundant cast to Node Removed the cast 5. XPathResultImpl.java:160: warning: [fallthrough] possible fall-through into case Added break; Thanks, Joe On 1/20/2015 10:51 AM, huizhe wang wrote: > > On 1/20/2015 7:02 AM, Lance Andersen wrote: >> Hi Joe, >> >> I think the changes look fine. > > Thanks. >> >> I am wondering if we have any suggested standard for the use of >> @Override as I see it is inconsistent in its usage with methods >> implementing an interface. Is this something we should add to >> existing code? I don't see Netbeans asking for it to be done except >> in the case of an overridden method? > > NetBeans does make a suggestion such as: "Add @Override Annotation". > I think you're right about avoiding comment duplication. In case of > existing "evaluate" methods, I previously updated* the javadocs for > both the interface and impl. It makes sense to take advantage of the > "automatically inherit" feature > > of the javadoc tool to avoid the duplication. > > *Note that the update was only on format and styles, and some > re-wording, no changes on definition or semantics. > >> >> Thank you for the extra clean up Joe > > Thank you, now it looks better and cleaner. > > Best, > Joe > >> >> Best >> Lance >> On Jan 16, 2015, at 9:33 PM, huizhe wang > > wrote: >> >>> >>> On 1/16/2015 1:29 PM, Lance Andersen wrote: >>>> Hi Joe, >>>> >>>> >>>> >>>> Overall it is OK, a few minor comments >>>> >>>> - Is there a reason that XPathExpressionImpl is no longer public >>>> and XPathImpl is public? >>> >>> Ok, I'll keep both public, may be useful in the future. >>> >>>> - I think you can leverage {@inheritdoc} in your impl classes to >>>> avoid comment duplication possibly? >>> >>> Looks like javadoc "Automatically inherit comment ". So @Override is >>> good enough. I've removed the javadocs for methods that override, >>> including the existing methods. It's good to avoid the duplication, >>> and potential copy n paste errors. >>> >>>> - please remember to make the copyright year 2015 >>> >>> Done. >>> >>>> - XPathTestBase, can the static block be moved to a @BeforeClass >>> >>> Moved to within the Dataprovider. >>> >>>> - XPathNodes, should that have an @since 1.9? >>> >>> Fixed. >>> >>>> - Given you are not including @param, etc for your test comments, >>>> you might want to consider /* */ vs /** */ style comments. >>> >>> Looks like s/\/**/\/* worked the trick. >>> >>>> That is consider if you really want doc comments (really a style >>>> choice but some IDEs will issue a warning for missing tags >>> >>> Yeh, it's good to turn off the warning "light" :-) >>> >>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>> >>> Thanks, >>> Joe >>> >>>> >>>> >>>> HTH >>>> >>>> Best, >>>> Lance >>>> On Jan 16, 2015, at 2:51 PM, huizhe wang >>> > wrote: >>>> >>>>> Hi all, >>>>> >>>>> Could you review the change? >>>>> >>>>> Thanks, >>>>> Joe >>>>> >>>>> On 12/18/2014 1:24 PM, huizhe wang wrote: >>>>>> Hi, >>>>>> >>>>>> This is to add support for any type and improvement with new >>>>>> features reflected in the new evaluateExpression methods, >>>>>> XPathEvaluationResult and XPathNodes. >>>>>> >>>>>> https://bugs.openjdk.java.net/browse/JDK-8054196 >>>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>>> >>>>>> Thanks, >>>>>> Joe >>>>> >>>> >>>> >>>> >>>> >>>> >>>> Lance >>>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>> Oracle Java Engineering >>>> 1 Network Drive >>>> Burlington, MA 01803 >>>> Lance.Andersen at oracle.com >>>> >>>> >>>> >>> >> >> >> >> >> Lance >> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> >> >> > From lance.andersen at oracle.com Thu Jan 22 01:09:08 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Wed, 21 Jan 2015 20:09:08 -0500 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: <54C04831.3060105@oracle.com> References: <54934601.4010104@oracle.com> <54B96BDF.7090204@oracle.com> <54B9C9EA.7050407@oracle.com> <5A817A94-0176-4613-91F2-9B298C1DA6AA@oracle.com> <54BEA3C6.6090501@oracle.com> <54C04831.3060105@oracle.com> Message-ID: <7B9200B5-D8C2-4070-957D-A97B32525BAF@oracle.com> Hi Joe, I think this is OK (as we discussed offline), one minor comment/suggestion below Best Lance On Jan 21, 2015, at 7:45 PM, huizhe wang wrote: > Thanks Lance for pointing me to Joe's -Xlint:all email thread. I re-compiled with -Xlint:all and noticed 7 warnings in this webrev. I've fixed them with a new webrev: > New webrev: http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev01/ > > Below are the details. > Refer to the previous webrev: http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ > > 1. XPathExpressionImpl.java:143 and XPathImpl.java:213 > > These warnings are fixed by having the getXPathResult method in XPathImpUtil returning the type instead > > 2. XPathExpressionImpl.java:165: warning: [rawtypes] found raw type: XPathEvaluationResult > similarly, XPathImpl.java:222 and XPathImpl.java:235 > > Changed to return XPathEvaluationResult > > 3. XPath.java:359, XPath.java:454, XPathExpression.java:252 and XPathExpression.java:344 > These are warnings from the default methods, casting the result of existing (old) methods. Use type.cast instead. > > 4. XPathNodesImpl.java:97: warning: [cast] redundant cast to Node > Removed the cast > > 5. XPathResultImpl.java:160: warning: [fallthrough] possible fall-through into case > Added break; I would add a comment or probably use @SuppressWarnings("fallthrough") instead > > Thanks, > Joe > > On 1/20/2015 10:51 AM, huizhe wang wrote: >> >> On 1/20/2015 7:02 AM, Lance Andersen wrote: >>> Hi Joe, >>> >>> I think the changes look fine. >> >> Thanks. >>> >>> I am wondering if we have any suggested standard for the use of @Override as I see it is inconsistent in its usage with methods implementing an interface. Is this something we should add to existing code? I don't see Netbeans asking for it to be done except in the case of an overridden method? >> >> NetBeans does make a suggestion such as: "Add @Override Annotation". I think you're right about avoiding comment duplication. In case of existing "evaluate" methods, I previously updated* the javadocs for both the interface and impl. It makes sense to take advantage of the "automatically inherit" feature of the javadoc tool to avoid the duplication. >> >> *Note that the update was only on format and styles, and some re-wording, no changes on definition or semantics. >> >>> >>> Thank you for the extra clean up Joe >> >> Thank you, now it looks better and cleaner. >> >> Best, >> Joe >> >>> >>> Best >>> Lance >>> On Jan 16, 2015, at 9:33 PM, huizhe wang > wrote: >>> >>>> >>>> On 1/16/2015 1:29 PM, Lance Andersen wrote: >>>>> Hi Joe, >>>>> >>>>> >>>>> >>>>> Overall it is OK, a few minor comments >>>>> >>>>> - Is there a reason that XPathExpressionImpl is no longer public and XPathImpl is public? >>>> >>>> Ok, I'll keep both public, may be useful in the future. >>>> >>>>> - I think you can leverage {@inheritdoc} in your impl classes to avoid comment duplication possibly? >>>> >>>> Looks like javadoc "Automatically inherit comment ". So @Override is good enough. I've removed the javadocs for methods that override, including the existing methods. It's good to avoid the duplication, and potential copy n paste errors. >>>> >>>>> - please remember to make the copyright year 2015 >>>> >>>> Done. >>>> >>>>> - XPathTestBase, can the static block be moved to a @BeforeClass >>>> >>>> Moved to within the Dataprovider. >>>> >>>>> - XPathNodes, should that have an @since 1.9? >>>> >>>> Fixed. >>>> >>>>> - Given you are not including @param, etc for your test comments, you might want to consider /* */ vs /** */ style comments. >>>> >>>> Looks like s/\/**/\/* worked the trick. >>>> >>>>> That is consider if you really want doc comments (really a style choice but some IDEs will issue a warning for missing tags >>>> >>>> Yeh, it's good to turn off the warning "light" :-) >>>> >>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>> >>>> Thanks, >>>> Joe >>>> >>>>> >>>>> >>>>> HTH >>>>> >>>>> Best, >>>>> Lance >>>>> On Jan 16, 2015, at 2:51 PM, huizhe wang > wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> Could you review the change? >>>>>> >>>>>> Thanks, >>>>>> Joe >>>>>> >>>>>> On 12/18/2014 1:24 PM, huizhe wang wrote: >>>>>>> Hi, >>>>>>> >>>>>>> This is to add support for any type and improvement with new features reflected in the new evaluateExpression methods, XPathEvaluationResult and XPathNodes. >>>>>>> >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8054196 >>>>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>>>> >>>>>>> Thanks, >>>>>>> Joe >>>>>> >>>>> >>>>> >>>>> >>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>> Oracle Java Engineering >>>>> 1 Network Drive >>>>> Burlington, MA 01803 >>>>> Lance.Andersen at oracle.com >>>>> >>>>> >>>>> >>>> >>> >>> >>> >>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>> Oracle Java Engineering >>> 1 Network Drive >>> Burlington, MA 01803 >>> Lance.Andersen at oracle.com >>> >>> >>> >> > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From huizhe.wang at oracle.com Thu Jan 22 05:18:02 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Wed, 21 Jan 2015 21:18:02 -0800 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: <7B9200B5-D8C2-4070-957D-A97B32525BAF@oracle.com> References: <54934601.4010104@oracle.com> <54B96BDF.7090204@oracle.com> <54B9C9EA.7050407@oracle.com> <5A817A94-0176-4613-91F2-9B298C1DA6AA@oracle.com> <54BEA3C6.6090501@oracle.com> <54C04831.3060105@oracle.com> <7B9200B5-D8C2-4070-957D-A97B32525BAF@oracle.com> Message-ID: <54C0880A.6090808@oracle.com> On 1/21/2015 5:09 PM, Lance Andersen wrote: > Hi Joe, > > I think this is OK (as we discussed offline), one minor > comment/suggestion below > > Best > Lance > On Jan 21, 2015, at 7:45 PM, huizhe wang > wrote: > >> Thanks Lance for pointing me to Joe's -Xlint:all email thread. I >> re-compiled with -Xlint:all and noticed 7 warnings in this webrev. >> I've fixed them with a new webrev: >> New webrev: http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev01/ >> >> >> Below are the details. >> Refer to the previous webrev: >> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >> >> >> 1. XPathExpressionImpl.java:143 and XPathImpl.java:213 >> >> These warnings are fixed by having the getXPathResult method in >> XPathImpUtil returning the type instead >> >> 2. XPathExpressionImpl.java:165: warning: [rawtypes] found raw type: >> XPathEvaluationResult >> similarly, XPathImpl.java:222 and XPathImpl.java:235 >> >> Changed to return XPathEvaluationResult >> >> 3. XPath.java:359, XPath.java:454, XPathExpression.java:252 and >> XPathExpression.java:344 >> These are warnings from the default methods, casting the result of >> existing (old) methods. Use type.cast instead. >> >> 4. XPathNodesImpl.java:97: warning: [cast] redundant cast to Node >> Removed the cast >> >> 5. XPathResultImpl.java:160: warning: [fallthrough] possible >> fall-through into case >> Added break; > > I would add a comment or probably > use @SuppressWarnings("fallthrough") instead Thanks! As I look at adding a comment, I realized I needed to be explicit on the numeric types as the spec stated. I've replaced the abstract Number type in XPathImplUtil::isSupportedClassType with Double/Integer/Long, and also in XPathResultImpl::getValue. I also added a test case for Long in XPathAnyTypeTest::test05, and a new test test06 to verify that the numeric types other than the supported can not be used as the type. http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev01/ Thanks, Joe >> >> Thanks, >> Joe >> >> On 1/20/2015 10:51 AM, huizhe wang wrote: >>> >>> On 1/20/2015 7:02 AM, Lance Andersen wrote: >>>> Hi Joe, >>>> >>>> I think the changes look fine. >>> >>> Thanks. >>>> >>>> I am wondering if we have any suggested standard for the use of >>>> @Override as I see it is inconsistent in its usage with methods >>>> implementing an interface. Is this something we should add to >>>> existing code? I don't see Netbeans asking for it to be done >>>> except in the case of an overridden method? >>> >>> NetBeans does make a suggestion such as: "Add @Override Annotation". >>> I think you're right about avoiding comment duplication. In case of >>> existing "evaluate" methods, I previously updated* the javadocs for >>> both the interface and impl. It makes sense to take advantage of the >>> "automatically inherit" feature >>> >>> of the javadoc tool to avoid the duplication. >>> >>> *Note that the update was only on format and styles, and some >>> re-wording, no changes on definition or semantics. >>> >>>> >>>> Thank you for the extra clean up Joe >>> >>> Thank you, now it looks better and cleaner. >>> >>> Best, >>> Joe >>> >>>> >>>> Best >>>> Lance >>>> On Jan 16, 2015, at 9:33 PM, huizhe wang >>> > wrote: >>>> >>>>> >>>>> On 1/16/2015 1:29 PM, Lance Andersen wrote: >>>>>> Hi Joe, >>>>>> >>>>>> >>>>>> >>>>>> Overall it is OK, a few minor comments >>>>>> >>>>>> - Is there a reason that XPathExpressionImpl is no longer >>>>>> public and XPathImpl is public? >>>>> >>>>> Ok, I'll keep both public, may be useful in the future. >>>>> >>>>>> - I think you can leverage {@inheritdoc} in your impl classes to >>>>>> avoid comment duplication possibly? >>>>> >>>>> Looks like javadoc "Automatically inherit comment ". So @Override >>>>> is good enough. I've removed the javadocs for methods that >>>>> override, including the existing methods. It's good to avoid the >>>>> duplication, and potential copy n paste errors. >>>>> >>>>>> - please remember to make the copyright year 2015 >>>>> >>>>> Done. >>>>> >>>>>> - XPathTestBase, can the static block be moved to a @BeforeClass >>>>> >>>>> Moved to within the Dataprovider. >>>>> >>>>>> - XPathNodes, should that have an @since 1.9? >>>>> >>>>> Fixed. >>>>> >>>>>> - Given you are not including @param, etc for your test comments, >>>>>> you might want to consider /* */ vs /** */ style comments. >>>>> >>>>> Looks like s/\/**/\/* worked the trick. >>>>> >>>>>> That is consider if you really want doc comments (really a style >>>>>> choice but some IDEs will issue a warning for missing tags >>>>> >>>>> Yeh, it's good to turn off the warning "light" :-) >>>>> >>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>> >>>>> >>>>> Thanks, >>>>> Joe >>>>> >>>>>> >>>>>> >>>>>> HTH >>>>>> >>>>>> Best, >>>>>> Lance >>>>>> On Jan 16, 2015, at 2:51 PM, huizhe wang >>>>> > wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Could you review the change? >>>>>>> >>>>>>> Thanks, >>>>>>> Joe >>>>>>> >>>>>>> On 12/18/2014 1:24 PM, huizhe wang wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> This is to add support for any type and improvement with new >>>>>>>> features reflected in the new evaluateExpression methods, >>>>>>>> XPathEvaluationResult and XPathNodes. >>>>>>>> >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8054196 >>>>>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Joe >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Lance >>>>>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>> Oracle Java Engineering >>>>>> 1 Network Drive >>>>>> Burlington, MA 01803 >>>>>> Lance.Andersen at oracle.com >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>>> >>>> Lance >>>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>> Oracle Java Engineering >>>> 1 Network Drive >>>> Burlington, MA 01803 >>>> Lance.Andersen at oracle.com >>>> >>>> >>>> >>>> >>> >> > > > > Lance > Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > From chris.hegarty at oracle.com Thu Jan 22 09:01:10 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 22 Jan 2015 09:01:10 +0000 Subject: [PATCH] JDK-8054565: FilterOutputStream.close may throw IOException if called twice and underlying flush or close fails In-Reply-To: References: , <54807258.4040303@oracle.com> , <20141205002430.000060c9.ecki@zusammenkunft.net>, <2C4F3368-64DC-4F8B-9923-0C08999C8C5B@oracle.com>, <20141205015827.00007ac6.ecki@zusammenkunft.net>, , <8F896CE2-E268-418F-96CA-0557C1C258B3@oracle.com>, <5481BB6C.1040700@oracle.com> <5481C3DB.70303@gmail.com>, <5481CA8C.7050100@oracle.com> <5481EF01.6060009@gmail.com>, , , <548EB053.2040908@oracle.com>, <549826FE.1060600@oracle.com> Message-ID: <2EFEF8AB-61A4-4031-92CE-DC14843DF83D@oracle.com> On 21 Jan 2015, at 22:37, Nathan Clement wrote: > Hi Chris, > > Thanks for your help on this one! Will this be backported to JDK 8, or is that not allowed because of the spec change? It is possible that the implementation only, not the spec changes, could be backported to 8u. I currently do not have any plans to do this. Is this something that really need in an 8 Update ? -Chris. > Thanks, > > Nathan > > > Date: Mon, 22 Dec 2014 14:13:18 +0000 > > From: chris.hegarty at oracle.com > > To: core-libs-dev at openjdk.java.net; nathan.a.clement at hotmail.com > > Subject: Re: [PATCH] JDK-8054565: FilterOutputStream.close may throw IOException if called twice and underlying flush or close fails > > > > To close the loop; find below the complete reviewed changes. > > > > http://cr.openjdk.java.net/~chegar/8054565/webrev.00/webrev/ > > > > Since there is a minor spec change, I will submit a CCC to cover it. > > Unless any unforeseen issues arise from the CCC, you can assume that > > this is final. > > > > -Chris. From sean.coffey at oracle.com Thu Jan 22 09:22:23 2015 From: sean.coffey at oracle.com (=?windows-1252?Q?Se=E1n_Coffey?=) Date: Thu, 22 Jan 2015 09:22:23 +0000 Subject: [8u60] - Request for Approval : RFR 8069302 Deprecate Unsafe monitor methods in JDK 8u release In-Reply-To: <48C8A30A-6393-4D6F-8ED6-E767AC93FFF8@oracle.com> References: <7BA76D88-2443-4DF2-AC64-439FF0911CDA@oracle.com> <48C8A30A-6393-4D6F-8ED6-E767AC93FFF8@oracle.com> Message-ID: <54C0C14F.6040800@oracle.com> Paul, Approved for jdk8u-dev. Can you add a subcomponent to the bug, add the 9-na label and add a suitable noreg- label. regards, Sean. On 21/01/2015 19:43, John Rose wrote: > +1; do it. > >> On Jan 21, 2015, at 2:35 AM, Paul Sandoz wrote: >> >> Hi, >> >> The Unsafe monitor methods monitorEnter. monitorExit and tryMonitorEnter were removed from JDK 9 [1]. >> >> I would like to mark those methods as deprecated in 8u60. I doubt no one will notice :-) but it's the correct thing to do. >> >> Paul. >> >> [1] http://hg.openjdk.java.net/jdk9/hs-rt/jdk/rev/0a0a0986400e From konstantin.shefov at oracle.com Thu Jan 22 09:43:05 2015 From: konstantin.shefov at oracle.com (Konstantin Shefov) Date: Thu, 22 Jan 2015 12:43:05 +0300 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54BFA473.5070905@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <548EC2D7.6000604@oracle.com> <548EC534.2090803@oracle.com> <548ECDA4.9070002@oracle.com> <54B8ECD7.3010703@oracle.com> <54B8F630.4040608@oracle.com> <54B8F90A.7010402@oracle.com> <54B908FA.9020405@oracle.com> <54B919ED.8080403@oracle.com> <54BFA473.5070905@oracle.com> Message-ID: <54C0C629.6010500@oracle.com> Kindly reminder On 21.01.2015 16:06, Konstantin Shefov wrote: > Hi, Chris > > I have done a pre-integration testing for this fix on Linux, Windows, > Mac and Solaris x64 and sparc. I have run all regression tests for > core libraries, and no new failures appeared. > > Can I push this change now or should I wait for an approve by one more > person? > > Thanks > -Konstantin > > On 16.01.2015 17:02, Chris Hegarty wrote: >> On 16/01/15 12:50, Konstantin Shefov wrote: >>> Hi Chris, Alan, thank you for reviewing. >>> >>> I have made a new webrev >>> http://cr.openjdk.java.net/~kshefov/6933879/webrev.01 >>> I have removed ":" and added a test case. >> >> This looks ok to me. >> >> -Chris. >> >>> -Konstantin >>> >>> 16.01.2015 14:42, Chris Hegarty ?????: >>>> On 16/01/15 11:29, Alan Bateman wrote: >>>>> On 16/01/2015 10:49, Chris Hegarty wrote: >>>>>> : >>>>>> >>>>>> I don't see any reason to update the spec here, given that the >>>>>> set of >>>>>> allowable character is not clearly defined in the relevant RFC's. >>>>>> >>>>> I think we need to create a bug to look into this more. Inet6Address >>>>> allows the scope of be any String but URI puts restrictions on >>>>> what is >>>>> legal. The proposed change updates the list of allowed characters >>>>> but I >>>>> assume it will need to re-visited again as systems with more exotic >>>>> interface names are encountered. >>>> >>>> Yes that is possible, but to date this is the first time I've seen >>>> this come up. >>>> >>>> Given the lack of clear specification in the relevant RFC's (relating >>>> to scope ids ), I don't think we should be making too many >>>> specification changes in this area. But I agree with your comment, URI >>>> has a restriction that it not clearly specified in the Java spec. >>>> >>>> -Chris. >>> > From artem.smotrakov at oracle.com Thu Jan 22 11:40:19 2015 From: artem.smotrakov at oracle.com (Artem Smotrakov) Date: Thu, 22 Jan 2015 14:40:19 +0300 Subject: [9] request for review: 8049171: Additional tests for jarsigner's warnings In-Reply-To: References: <54BF48BF.9090505@oracle.com> Message-ID: <54C0E1A3.8070600@oracle.com> Thanks, Max. Please see inline. On 01/21/2015 11:29 AM, Wang Weijun wrote: > Thanks for adding so many tests. Some suggestions: > > - JarUtils.java > > You can use the new InputStream.transferTo() method. Sure, I didn't know about that, thanks. > > I am not sure if I understand updateJar correctly. It looks like srcJarFile is opened multiple times so its entries are duplicated a lot in the destination. Or is there a secret break? There is no any secret, just a bug. It is not necessary to open srcJarFile multiple times. I have updated the webrev, updateJar() method does the following: - creates a new jar file (destJarFilename) - puts files which are specified in files parameter to destJarFilename - copies files from srcJarFilename to destJarFilename if they are no files with the same names in destJarFilename > > - Utils.java > > The mixed using of File and Files is strange, but you are free to keep it. Agree. I removed this method since tests don't need it anymore after I removed cleanup() methods (please see below). > > - TimestampCheck.java > > You can make Handler Autocloseable to use try-with-resources in tests. Good idea. > > - Various tests > > I am not a fan of calling Utils.cleanup() in final block. Unless you have created huge garbages, those files are precious when the test fails (given you provide -retain in jtreg, which I always do). Agree, those files may be helpful, they are not huge. I have removed a cleanup() methods. Here is an updated webrev with your suggested changes and a couple of others: - added @ignore tag for BadKeyUsageTest since it fails due to a bug in JDK - updated MultipleWarningsTest test to check ExtendedKeyUsage case instead of KeyUsage http://cr.openjdk.java.net/~asmotrak/8049171/webrev.01/ Artem > Thanks > Max > >> On Jan 21, 2015, at 14:35, Artem Smotrakov wrote: >> >> Hello, >> >> Please review a couple of new tests for jarsigner's warnings. Basically tests run jarsigner and check warning/error messages and exit codes according to [1]. >> >> https://bugs.openjdk.java.net/browse/JDK-8049171 >> http://cr.openjdk.java.net/~asmotrak/8049171/webrev.00 >> >> [1] http://docs.oracle.com/javase/7/docs/technotes/tools/windows/jarsigner.html >> >> Artem >> >> From ivan.gerasimov at oracle.com Thu Jan 22 14:37:08 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Thu, 22 Jan 2015 17:37:08 +0300 Subject: RFR 8067748: (process) Child is terminated when parent's console is closed [win] Message-ID: <54C10B14.3080908@oracle.com> Hello! It was reported that the fix for JDK-8023130 ((process) ProcessBuilder#inheritIO does not work on Windows) had caused a regression. It manifests itself in the following situation: 1) A console java application starts a child process. 2) The parent then exits without waiting for the child to complete, the child is running. 3) The console window, the parent app was started from, is closed. 4) The child process is unexpectedly terminated. The solution is to tighten the cases in which we drop CREATE_NO_WINDOW flag: it is dropped only any IO stream of the parent is going to be inherited by the child. Only in that later case keeping the child attached to console makes sense. The fix was tested manually: the inheritIO behavior is preserved, the regression is gone. Unfortunately, no regression test is attached. It's hard to create a test, which involves closing the console window and manipulating the detached child application. Would you please help review the fix? BUGURL: https://bugs.openjdk.java.net/browse/JDK-8067748 WEBREV: http://cr.openjdk.java.net/~igerasim/8067748/0/webrev/ Sincerely yours, Ivan From paul.sandoz at oracle.com Thu Jan 22 14:53:20 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 22 Jan 2015 14:53:20 +0000 Subject: 8050820: Please add java.util.Optional.stream() to convert Optional to Stream In-Reply-To: <3005029A-25AF-402B-9993-34754BFE54FB@oracle.com> References: <20150121100000.GA11988@e6430> <3CED45C1-A695-44E9-8E9A-F1D0990A9393@oracle.com> <3005029A-25AF-402B-9993-34754BFE54FB@oracle.com> Message-ID: <6297580F-F12F-4240-B543-6727CB4E632B@oracle.com> On Jan 21, 2015, at 7:42 PM, John Rose wrote: > On Jan 21, 2015, at 3:01 AM, Paul Sandoz wrote: >> >> I updated the webrev in place to be more consistent in the use of braces and better consistency for the primitive specializations: >> >> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050820-Optional-stream/webrev/ >> >> I don't wanna make any more syntax-related changes unless i done something silly. > > It's often a good idea to make cleanups to code you touch, but there is a natural limit. > > If cleanups proliferate far away from the semantic changes, the two sorts of changes will begin to confuse each other, both during review and later on during backports or other change analysis. > Yes, it's very tempting to make the source conform to one's own "cultural of code format" (and it's all to easy to hit the reformat hot key in the IDE) [*]. > I think what you have done is acceptable from that viewpoint, since the changes can be separated "by eye" easily enough. > > You can count me as a Reviewer for this. > Thanks. I updated the webrev to add "@since 1.9" to the docs. Paul. [*] I would be quite happy with a Java endorsed "culture of code format" and just configure the IDE to format to that (one less thing to think about). I am rather skeptical that any consensus driven approach can produce such a format before the heat death of the universe. Someone with trusted taste and leadership needs to make a proposal, one is allowed to comment during a period of time, but one is not allowed to comment on other people's comments :-) From forax at univ-mlv.fr Thu Jan 22 15:08:48 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 22 Jan 2015 16:08:48 +0100 Subject: 8050820: Please add java.util.Optional.stream() to convert Optional to Stream In-Reply-To: <6297580F-F12F-4240-B543-6727CB4E632B@oracle.com> References: <20150121100000.GA11988@e6430> <3CED45C1-A695-44E9-8E9A-F1D0990A9393@oracle.com> <3005029A-25AF-402B-9993-34754BFE54FB@oracle.com> <6297580F-F12F-4240-B543-6727CB4E632B@oracle.com> Message-ID: <54C11280.6020002@univ-mlv.fr> On 01/22/2015 03:53 PM, Paul Sandoz wrote: [...] > Yes, it's very tempting to make the source conform to one's own "cultural of code format" (and it's all to easy to hit the reformat hot key in the IDE) [*]. [...] > > >> I think what you have done is acceptable from that viewpoint, since the changes can be separated "by eye" easily enough. >> >> You can count me as a Reviewer for this. >> > Thanks. I updated the webrev to add "@since 1.9" to the docs. [...] > > Paul. > > [*] I would be quite happy with a Java endorsed "culture of code format" and just configure the IDE to format to that (one less thing to think about). I am rather skeptical that any consensus driven approach can produce such a format before the heat death of the universe. Someone with trusted taste and leadership needs to make a proposal, I vote for you :) > one is allowed to comment during a period of time, but one is not allowed to comment on other people's comments :-) cheers, R?mi From lance.andersen at oracle.com Thu Jan 22 17:11:08 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Thu, 22 Jan 2015 12:11:08 -0500 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: <54C0880A.6090808@oracle.com> References: <54934601.4010104@oracle.com> <54B96BDF.7090204@oracle.com> <54B9C9EA.7050407@oracle.com> <5A817A94-0176-4613-91F2-9B298C1DA6AA@oracle.com> <54BEA3C6.6090501@oracle.com> <54C04831.3060105@oracle.com> <7B9200B5-D8C2-4070-957D-A97B32525BAF@oracle.com> <54C0880A.6090808@oracle.com> Message-ID: <2A097154-9F50-463F-8506-769C9DFE7B73@oracle.com> Hi Joe, I just looked at the changes below, I looked at the changes below? see minor comments On Jan 22, 2015, at 12:18 AM, huizhe wang wrote: > > On 1/21/2015 5:09 PM, Lance Andersen wrote: >> Hi Joe, >> >> I think this is OK (as we discussed offline), one minor comment/suggestion below >> >> Best >> Lance >> On Jan 21, 2015, at 7:45 PM, huizhe wang wrote: >> >>> Thanks Lance for pointing me to Joe's -Xlint:all email thread. I re-compiled with -Xlint:all and noticed 7 warnings in this webrev. I've fixed them with a new webrev: >>> New webrev: http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev01/ >>> >>> Below are the details. >>> Refer to the previous webrev: http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>> >>> 1. XPathExpressionImpl.java:143 and XPathImpl.java:213 >>> >>> These warnings are fixed by having the getXPathResult method in XPathImpUtil returning the type instead >>> >>> 2. XPathExpressionImpl.java:165: warning: [rawtypes] found raw type: XPathEvaluationResult >>> similarly, XPathImpl.java:222 and XPathImpl.java:235 >>> >>> Changed to return XPathEvaluationResult >>> >>> 3. XPath.java:359, XPath.java:454, XPathExpression.java:252 and XPathExpression.java:344 >>> These are warnings from the default methods, casting the result of existing (old) methods. Use type.cast instead. >>> >>> 4. XPathNodesImpl.java:97: warning: [cast] redundant cast to Node >>> Removed the cast >>> >>> 5. XPathResultImpl.java:160: warning: [fallthrough] possible fall-through into case >>> Added break; >> >> I would add a comment or probably use @SuppressWarnings("fallthrough") instead > > Thanks! As I look at adding a comment, I realized I needed to be explicit on the numeric types as the spec stated. I've replaced the abstract Number type in XPathImplUtil::isSupportedClassType with Double/Integer/Long, and also in XPathResultImpl::getValue. Can we document why XPathImplUti.getXPathResult() should return null, same for XPathResultImpl.getValue() > > I also added a test case for Long in XPathAnyTypeTest::test05, and a new test test06 to verify that the numeric types other than the supported can not be used as the type. This looks good > > http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev01/ > > Thanks, > Joe > >>> >>> Thanks, >>> Joe >>> >>> On 1/20/2015 10:51 AM, huizhe wang wrote: >>>> >>>> On 1/20/2015 7:02 AM, Lance Andersen wrote: >>>>> Hi Joe, >>>>> >>>>> I think the changes look fine. >>>> >>>> Thanks. >>>>> >>>>> I am wondering if we have any suggested standard for the use of @Override as I see it is inconsistent in its usage with methods implementing an interface. Is this something we should add to existing code? I don't see Netbeans asking for it to be done except in the case of an overridden method? >>>> >>>> NetBeans does make a suggestion such as: "Add @Override Annotation". I think you're right about avoiding comment duplication. In case of existing "evaluate" methods, I previously updated* the javadocs for both the interface and impl. It makes sense to take advantage of the "automatically inherit" feature of the javadoc tool to avoid the duplication. >>>> >>>> *Note that the update was only on format and styles, and some re-wording, no changes on definition or semantics. >>>> >>>>> >>>>> Thank you for the extra clean up Joe >>>> >>>> Thank you, now it looks better and cleaner. >>>> >>>> Best, >>>> Joe >>>> >>>>> >>>>> Best >>>>> Lance >>>>> On Jan 16, 2015, at 9:33 PM, huizhe wang > wrote: >>>>> >>>>>> >>>>>> On 1/16/2015 1:29 PM, Lance Andersen wrote: >>>>>>> Hi Joe, >>>>>>> >>>>>>> >>>>>>> >>>>>>> Overall it is OK, a few minor comments >>>>>>> >>>>>>> - Is there a reason that XPathExpressionImpl is no longer public and XPathImpl is public? >>>>>> >>>>>> Ok, I'll keep both public, may be useful in the future. >>>>>> >>>>>>> - I think you can leverage {@inheritdoc} in your impl classes to avoid comment duplication possibly? >>>>>> >>>>>> Looks like javadoc "Automatically inherit comment ". So @Override is good enough. I've removed the javadocs for methods that override, including the existing methods. It's good to avoid the duplication, and potential copy n paste errors. >>>>>> >>>>>>> - please remember to make the copyright year 2015 >>>>>> >>>>>> Done. >>>>>> >>>>>>> - XPathTestBase, can the static block be moved to a @BeforeClass >>>>>> >>>>>> Moved to within the Dataprovider. >>>>>> >>>>>>> - XPathNodes, should that have an @since 1.9? >>>>>> >>>>>> Fixed. >>>>>> >>>>>>> - Given you are not including @param, etc for your test comments, you might want to consider /* */ vs /** */ style comments. >>>>>> >>>>>> Looks like s/\/**/\/* worked the trick. >>>>>> >>>>>>> That is consider if you really want doc comments (really a style choice but some IDEs will issue a warning for missing tags >>>>>> >>>>>> Yeh, it's good to turn off the warning "light" :-) >>>>>> >>>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>>> >>>>>> Thanks, >>>>>> Joe >>>>>> >>>>>>> >>>>>>> >>>>>>> HTH >>>>>>> >>>>>>> Best, >>>>>>> Lance >>>>>>> On Jan 16, 2015, at 2:51 PM, huizhe wang > wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Could you review the change? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Joe >>>>>>>> >>>>>>>> On 12/18/2014 1:24 PM, huizhe wang wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> This is to add support for any type and improvement with new features reflected in the new evaluateExpression methods, XPathEvaluationResult and XPathNodes. >>>>>>>>> >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8054196 >>>>>>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Joe >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>> Oracle Java Engineering >>>>>>> 1 Network Drive >>>>>>> Burlington, MA 01803 >>>>>>> Lance.Andersen at oracle.com >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>> Oracle Java Engineering >>>>> 1 Network Drive >>>>> Burlington, MA 01803 >>>>> Lance.Andersen at oracle.com >>>>> >>>>> >>>>> >>>> >>> >> >> >> >> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> >> >> > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From vladimir.x.ivanov at oracle.com Thu Jan 22 17:56:30 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Thu, 22 Jan 2015 20:56:30 +0300 Subject: [9] RFR (M): 8069591: Customize LambdaForms which are invoked using MH.invoke/invokeExact In-Reply-To: <3B4D19E0-8DA6-4FE1-BD77-E12E8BCF15EC@oracle.com> References: <54BFD2EE.3060909@oracle.com> <54BFE259.1090402@univ-mlv.fr> <3B4D19E0-8DA6-4FE1-BD77-E12E8BCF15EC@oracle.com> Message-ID: <54C139CE.4000005@oracle.com> Remi, John, thanks for review! Updated webrev: http://cr.openjdk.java.net/~vlivanov/8069591/webrev.01/ This time I did additional testing (COMPILE_THRESHOLD > 0) and spotted a problem with MethodHandle.copyWith(): a MethodHandle can inherit customized LambdaForm this way. I could have added LambdaForm::uncustomize() call in evey Species_*::copyWith() method, but I decided to add it into MethodHandle constructor. Let me know if you think it's too intrusive. Also, I made DirectMethodHandles a special-case, since I don't see any benefit in customizing them. Best regards, Vladimir Ivanov On 1/21/15 10:30 PM, John Rose wrote: > On Jan 21, 2015, at 9:31 AM, Remi Forax wrote: >> >> in Invokers.java, I think that checkCustomized should take an Object and not a MethodHandle >> exactly like getCallSiteTarget takes an Object and not a CallSite. > > The use of erased types (any ref => Object) in the MH runtime is an artifact of bootstrapping difficulties, early in the project. I hope it is not necessary any more. That said, I agree that the pattern should be consistent. > > Vladimir, would you please file a tracking bug for this cleanup, to change MH library functions to use stronger types instead of Object? > >> in MethodHandle.java, customizationCount is declared as a byte and there is no check that >> the CUSTOMIZE_THRESHOLD is not greater than 127. > > Yes. Also, the maybeCustomize method has a race condition that could cause the counter to wrap. It shouldn't use "+=1" to increment; it should load the old counter value, test it, increment it (in a local), and then store the updated value. That is also one possible place to deal with jumbo CUSTOMIZE_THRESHOLD values. > > ? John > From Alan.Bateman at oracle.com Thu Jan 22 18:12:13 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 22 Jan 2015 18:12:13 +0000 Subject: RFR 8067748: (process) Child is terminated when parent's console is closed [win] In-Reply-To: <54C10B14.3080908@oracle.com> References: <54C10B14.3080908@oracle.com> Message-ID: <54C13D7D.7030701@oracle.com> On 22/01/2015 14:37, Ivan Gerasimov wrote: > : > It's hard to create a test, which involves closing the console window > and manipulating the detached child application. > > Would you please help review the fix? > > BUGURL: https://bugs.openjdk.java.net/browse/JDK-8067748 > WEBREV: http://cr.openjdk.java.net/~igerasim/8067748/0/webrev/ Yet again, the Process code bites back :-) I think this looks okay and we should probably have looked at redirect and inheritance more when discussing the console window pop-up issue. I don't personally like references to bug numbers in comments, others will have different opinions. You have a good case for noreg-hard as it's not clear that it would be easy to create an automated test for this. -Alan From vladimir.x.ivanov at oracle.com Thu Jan 22 18:21:52 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Thu, 22 Jan 2015 21:21:52 +0300 Subject: [9] RFR (M): 8069591: Customize LambdaForms which are invoked using MH.invoke/invokeExact In-Reply-To: <3B4D19E0-8DA6-4FE1-BD77-E12E8BCF15EC@oracle.com> References: <54BFD2EE.3060909@oracle.com> <54BFE259.1090402@univ-mlv.fr> <3B4D19E0-8DA6-4FE1-BD77-E12E8BCF15EC@oracle.com> Message-ID: <54C13FC0.1030608@oracle.com> >> in Invokers.java, I think that checkCustomized should take an Object and not a MethodHandle >> exactly like getCallSiteTarget takes an Object and not a CallSite. > > The use of erased types (any ref => Object) in the MH runtime is an artifact of bootstrapping difficulties, early in the project. I hope it is not necessary any more. That said, I agree that the pattern should be consistent. Sure. Here is it [1] Best regards, Vladimir Ivanov [1] https://bugs.openjdk.java.net/browse/JDK-8071368 From martinrb at google.com Thu Jan 22 20:01:54 2015 From: martinrb at google.com (Martin Buchholz) Date: Thu, 22 Jan 2015 12:01:54 -0800 Subject: RFR 8067748: (process) Child is terminated when parent's console is closed [win] In-Reply-To: <54C10B14.3080908@oracle.com> References: <54C10B14.3080908@oracle.com> Message-ID: Not really reviewing, but cheerleading - this area is really hard! Historically I tried to avoid touching windows api and avoid changing any aspect of process termination, since whichever way you go, someone will be relying on the historic behavior. On Thu, Jan 22, 2015 at 6:37 AM, Ivan Gerasimov wrote: > Hello! > > It was reported that the fix for JDK-8023130 ((process) > ProcessBuilder#inheritIO does not work on Windows) had caused a regression. > > It manifests itself in the following situation: > 1) A console java application starts a child process. > 2) The parent then exits without waiting for the child to complete, the > child is running. > 3) The console window, the parent app was started from, is closed. > 4) The child process is unexpectedly terminated. > > The solution is to tighten the cases in which we drop CREATE_NO_WINDOW > flag: it is dropped only any IO stream of the parent is going to be > inherited by the child. > Only in that later case keeping the child attached to console makes sense. > > The fix was tested manually: the inheritIO behavior is preserved, the > regression is gone. > > Unfortunately, no regression test is attached. > It's hard to create a test, which involves closing the console window and > manipulating the detached child application. > > Would you please help review the fix? > > BUGURL: https://bugs.openjdk.java.net/browse/JDK-8067748 > WEBREV: http://cr.openjdk.java.net/~igerasim/8067748/0/webrev/ > > Sincerely yours, > Ivan > From xueming.shen at oracle.com Thu Jan 22 21:48:14 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Thu, 22 Jan 2015 13:48:14 -0800 Subject: RFR: JDK-8064736: Part of java.util.jar.JarFile spec looks confusing with references to Zip Message-ID: <54C1701E.3030702@oracle.com> Hi Please help review issue: https://bugs.openjdk.java.net/browse/JDK-8064736 webrev: http://cr.openjdk.java.net/~sherman/8064736 The goal is to have the api doc for these two methods to be consistent with the rest of the apis, especially the ISE exception part. Thanks! -Sherman From nathan.a.clement at hotmail.com Thu Jan 22 21:57:11 2015 From: nathan.a.clement at hotmail.com (Nathan Clement) Date: Fri, 23 Jan 2015 08:57:11 +1100 Subject: [PATCH] JDK-8054565: FilterOutputStream.close may throw IOException if called twice and underlying flush or close fails In-Reply-To: <2EFEF8AB-61A4-4031-92CE-DC14843DF83D@oracle.com> References: , <54807258.4040303@oracle.com> , <20141205002430.000060c9.ecki@zusammenkunft.net>, <2C4F3368-64DC-4F8B-9923-0C08999C8C5B@oracle.com>, <20141205015827.00007ac6.ecki@zusammenkunft.net>, , <8F896CE2-E268-418F-96CA-0557C1C258B3@oracle.com>, <5481BB6C.1040700@oracle.com> <5481C3DB.70303@gmail.com>, <5481CA8C.7050100@oracle.com> <5481EF01.6060009@gmail.com>, , , <548EB053.2040908@oracle.com>,<549826FE.1060600@oracle.com> , <2EFEF8AB-61A4-4031-92CE-DC14843DF83D@oracle.com> Message-ID: Hi Chris, We have worked around the problem by using a small jar in our bootstrap classpath (now containing your updated code). From our perspective there's probably no pressing need to have the fix backported. Thanks, Nathan > Subject: Re: [PATCH] JDK-8054565: FilterOutputStream.close may throw IOException if called twice and underlying flush or close fails > From: chris.hegarty at oracle.com > Date: Thu, 22 Jan 2015 09:01:10 +0000 > CC: core-libs-dev at openjdk.java.net > To: nathan.a.clement at hotmail.com > > On 21 Jan 2015, at 22:37, Nathan Clement wrote: > > > Hi Chris, > > > > Thanks for your help on this one! Will this be backported to JDK 8, or is that not allowed because of the spec change? > > It is possible that the implementation only, not the spec changes, could be backported to 8u. I currently do not have any plans to do this. Is this something that really need in an 8 Update ? > > -Chris. > > > Thanks, > > > > Nathan > > > > > Date: Mon, 22 Dec 2014 14:13:18 +0000 > > > From: chris.hegarty at oracle.com > > > To: core-libs-dev at openjdk.java.net; nathan.a.clement at hotmail.com > > > Subject: Re: [PATCH] JDK-8054565: FilterOutputStream.close may throw IOException if called twice and underlying flush or close fails > > > > > > To close the loop; find below the complete reviewed changes. > > > > > > http://cr.openjdk.java.net/~chegar/8054565/webrev.00/webrev/ > > > > > > Since there is a minor spec change, I will submit a CCC to cover it. > > > Unless any unforeseen issues arise from the CCC, you can assume that > > > this is final. > > > > > > -Chris. > From huizhe.wang at oracle.com Thu Jan 22 23:04:29 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Thu, 22 Jan 2015 15:04:29 -0800 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: <2A097154-9F50-463F-8506-769C9DFE7B73@oracle.com> References: <54934601.4010104@oracle.com> <54B96BDF.7090204@oracle.com> <54B9C9EA.7050407@oracle.com> <5A817A94-0176-4613-91F2-9B298C1DA6AA@oracle.com> <54BEA3C6.6090501@oracle.com> <54C04831.3060105@oracle.com> <7B9200B5-D8C2-4070-957D-A97B32525BAF@oracle.com> <54C0880A.6090808@oracle.com> <2A097154-9F50-463F-8506-769C9DFE7B73@oracle.com> Message-ID: <54C181FD.7000203@oracle.com> Thanks Lance. On 1/22/2015 9:11 AM, Lance Andersen wrote: > Hi Joe, > > I just looked at the changes below, > > I looked at the changes below? see minor comments > On Jan 22, 2015, at 12:18 AM, huizhe wang > wrote: > >> >> On 1/21/2015 5:09 PM, Lance Andersen wrote: >>> Hi Joe, >>> >>> I think this is OK (as we discussed offline), one minor >>> comment/suggestion below >>> >>> Best >>> Lance >>> On Jan 21, 2015, at 7:45 PM, huizhe wang >> > wrote: >>> >>>> Thanks Lance for pointing me to Joe's -Xlint:all email thread. I >>>> re-compiled with -Xlint:all and noticed 7 warnings in this webrev. >>>> I've fixed them with a new webrev: >>>> New webrev: >>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev01/ >>>> >>>> >>>> Below are the details. >>>> Refer to the previous webrev: >>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>> >>>> >>>> 1. XPathExpressionImpl.java:143 and XPathImpl.java:213 >>>> >>>> These warnings are fixed by having the getXPathResult method in >>>> XPathImpUtil returning the type instead >>>> >>>> 2. XPathExpressionImpl.java:165: warning: [rawtypes] found raw >>>> type: XPathEvaluationResult >>>> similarly, XPathImpl.java:222 and XPathImpl.java:235 >>>> >>>> Changed to return XPathEvaluationResult >>>> >>>> 3. XPath.java:359, XPath.java:454, XPathExpression.java:252 and >>>> XPathExpression.java:344 >>>> These are warnings from the default methods, casting the result >>>> of existing (old) methods. Use type.cast instead. >>>> >>>> 4. XPathNodesImpl.java:97: warning: [cast] redundant cast to Node >>>> Removed the cast >>>> >>>> 5. XPathResultImpl.java:160: warning: [fallthrough] possible >>>> fall-through into case >>>> Added break; >>> >>> I would add a comment or probably >>> use @SuppressWarnings("fallthrough") instead >> >> Thanks! As I look at adding a comment, I realized I needed to be >> explicit on the numeric types as the spec stated. I've replaced the >> abstract Number type in XPathImplUtil::isSupportedClassType with >> Double/Integer/Long, and also in XPathResultImpl::getValue. > > Can we document why XPathImplUti.getXPathResult() should return null, > same for XPathResultImpl.getValue() Done. While I was there, I also removed the 'type' parameter from the constructor of XPathNodesImpl and then XPathResultImpl::classToType method as it was only used to for constructing the XPathNodes object. >> >> I also added a test case for Long in XPathAnyTypeTest::test05, and a >> new test test06 to verify that the numeric types other than the >> supported can not be used as the type. > This looks good http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev02/ Thanks, Joe >> >> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev01/ >> >> >> Thanks, >> Joe >> >>>> >>>> Thanks, >>>> Joe >>>> >>>> On 1/20/2015 10:51 AM, huizhe wang wrote: >>>>> >>>>> On 1/20/2015 7:02 AM, Lance Andersen wrote: >>>>>> Hi Joe, >>>>>> >>>>>> I think the changes look fine. >>>>> >>>>> Thanks. >>>>>> >>>>>> I am wondering if we have any suggested standard for the use of >>>>>> @Override as I see it is inconsistent in its usage with methods >>>>>> implementing an interface. Is this something we should add to >>>>>> existing code? I don't see Netbeans asking for it to be done >>>>>> except in the case of an overridden method? >>>>> >>>>> NetBeans does make a suggestion such as: "Add @Override >>>>> Annotation". I think you're right about avoiding comment >>>>> duplication. In case of existing "evaluate" methods, I previously >>>>> updated* the javadocs for both the interface and impl. It makes >>>>> sense to take advantage of the "automatically inherit" feature >>>>> >>>>> of the javadoc tool to avoid the duplication. >>>>> >>>>> *Note that the update was only on format and styles, and some >>>>> re-wording, no changes on definition or semantics. >>>>> >>>>>> >>>>>> Thank you for the extra clean up Joe >>>>> >>>>> Thank you, now it looks better and cleaner. >>>>> >>>>> Best, >>>>> Joe >>>>> >>>>>> >>>>>> Best >>>>>> Lance >>>>>> On Jan 16, 2015, at 9:33 PM, huizhe wang >>>>> > >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> On 1/16/2015 1:29 PM, Lance Andersen wrote: >>>>>>>> Hi Joe, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Overall it is OK, a few minor comments >>>>>>>> >>>>>>>> - Is there a reason that XPathExpressionImpl is no longer >>>>>>>> public and XPathImpl is public? >>>>>>> >>>>>>> Ok, I'll keep both public, may be useful in the future. >>>>>>> >>>>>>>> - I think you can leverage {@inheritdoc} in your impl classes >>>>>>>> to avoid comment duplication possibly? >>>>>>> >>>>>>> Looks like javadoc "Automatically inherit comment ". So >>>>>>> @Override is good enough. I've removed the javadocs for methods >>>>>>> that override, including the existing methods. It's good to >>>>>>> avoid the duplication, and potential copy n paste errors. >>>>>>> >>>>>>>> - please remember to make the copyright year 2015 >>>>>>> >>>>>>> Done. >>>>>>> >>>>>>>> - XPathTestBase, can the static block be moved to a @BeforeClass >>>>>>> >>>>>>> Moved to within the Dataprovider. >>>>>>> >>>>>>>> - XPathNodes, should that have an @since 1.9? >>>>>>> >>>>>>> Fixed. >>>>>>> >>>>>>>> - Given you are not including @param, etc for your test >>>>>>>> comments, you might want to consider /* */ vs /** */ style >>>>>>>> comments. >>>>>>> >>>>>>> Looks like s/\/**/\/* worked the trick. >>>>>>> >>>>>>>> That is consider if you really want doc comments (really a >>>>>>>> style choice but some IDEs will issue a warning for missing tags >>>>>>> >>>>>>> Yeh, it's good to turn off the warning "light" :-) >>>>>>> >>>>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Joe >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> HTH >>>>>>>> >>>>>>>> Best, >>>>>>>> Lance >>>>>>>> On Jan 16, 2015, at 2:51 PM, huizhe wang >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> Could you review the change? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Joe >>>>>>>>> >>>>>>>>> On 12/18/2014 1:24 PM, huizhe wang wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> This is to add support for any type and improvement with new >>>>>>>>>> features reflected in the new evaluateExpression methods, >>>>>>>>>> XPathEvaluationResult and XPathNodes. >>>>>>>>>> >>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8054196 >>>>>>>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Joe >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Lance >>>>>>>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>>> Oracle Java Engineering >>>>>>>> 1 Network Drive >>>>>>>> Burlington, MA 01803 >>>>>>>> Lance.Andersen at oracle.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Lance >>>>>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>> Oracle Java Engineering >>>>>> 1 Network Drive >>>>>> Burlington, MA 01803 >>>>>> Lance.Andersen at oracle.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> >>> >>> Lance >>> Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>> Oracle Java Engineering >>> 1 Network Drive >>> Burlington, MA 01803 >>> Lance.Andersen at oracle.com >>> >>> >>> >> > > > > Lance > Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > From john.r.rose at oracle.com Thu Jan 22 23:30:59 2015 From: john.r.rose at oracle.com (John Rose) Date: Thu, 22 Jan 2015 15:30:59 -0800 Subject: [9] RFR (M): 8069591: Customize LambdaForms which are invoked using MH.invoke/invokeExact In-Reply-To: <54C139CE.4000005@oracle.com> References: <54BFD2EE.3060909@oracle.com> <54BFE259.1090402@univ-mlv.fr> <3B4D19E0-8DA6-4FE1-BD77-E12E8BCF15EC@oracle.com> <54C139CE.4000005@oracle.com> Message-ID: On Jan 22, 2015, at 9:56 AM, Vladimir Ivanov wrote: > > Remi, John, thanks for review! > > Updated webrev: > http://cr.openjdk.java.net/~vlivanov/8069591/webrev.01/ > > This time I did additional testing (COMPILE_THRESHOLD > 0) and spotted a problem with MethodHandle.copyWith(): a MethodHandle can inherit customized LambdaForm this way. I could have added LambdaForm::uncustomize() call in evey Species_*::copyWith() method, but I decided to add it into MethodHandle constructor. Let me know if you think it's too intrusive. It's OK to put it there. Now I'm worried that the new customization logic will defeat code sharing for invoked MHs, since uncustomize creates a new LF that is a duplicate of the original LF. That breaks the genetic link for children of the invoked MH, doesn't it? (I like the compileToBytecode call, if it is done on the original.) In fact, that is also a potential problem for the first version of your patch, also. Suggestion: Have every customized LF contain a direct link to its uncustomized original. Have uncustomize just return that same original, every time. Then, when using LF editor operations to derive new LFs, always have them extract the original before making a derivation. (Alternatively, have the LF editor caches be shared between original LFs and all their customized versions. But that doesn't save all the genetic links.) > Also, I made DirectMethodHandles a special-case, since I don't see any benefit in customizing them. The overriding method in DHM should be marked @Override, so that we know all the bits fit together. ? John From john.r.rose at oracle.com Fri Jan 23 01:31:59 2015 From: john.r.rose at oracle.com (John Rose) Date: Thu, 22 Jan 2015 17:31:59 -0800 Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when GWT LambdaForms are shared In-Reply-To: <54BEA7D7.6080008@oracle.com> References: <54B94766.2080102@oracle.com> <7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com> <54BEA7D7.6080008@oracle.com> Message-ID: <5BA1E369-ED87-4EBD-8408-B73B726D91BD@oracle.com> On Jan 20, 2015, at 11:09 AM, Vladimir Ivanov wrote: > >> What I'm mainly poking at here is that 'isGWT' is not informative about >> the intended use of the flag. > I agree. It was an interim solution. Initially, I planned to introduce customization and guide the logic based on that property. But it's not there yet and I needed something for GWT case. Unfortunately, I missed the case when GWT is edited. In that case, isGWT flag is missed and no annotation is set. > So, I removed isGWT flag and introduced a check for selectAlternative occurence in LambdaForm shape, as you suggested. Good. I think there is a sweeter spot just a little further on. Make profileBranch be an LF intrinsic and expose it like this: GWT(p,t,f;S) := let(a=new int[3]) in lambda(*: S) { selectAlternative(profileBranch(p.invoke( *), a), t, f).invoke( *); } Then selectAlternative triggers branchy bytecodes in the IBGen, and profileBranch injects profiling in C2. The presence of profileBranch would then trigger the @Shared annotation, if you still need it. After thinking about it some more, I still believe it would be better to detect the use of profileBranch during a C2 compile task, and feed that to the too_many_traps logic. I agree it is much easier to stick the annotation on in the IBGen; the problem is that because of a minor phase ordering problem you are introducing an annotation which flows from the JDK to the VM. Here's one more suggestion at reducing this coupling? Note that C->set_trap_count is called when each Parse phase processes a whole method. This means that information about the contents of the nmethod accumulates during the parse. Likewise, add a flag method C->{has,set}_injected_profile, and set the flag whenever the parser sees a profileBranch intrinsic (with or without a constant profile array; your call). Then consult that flag from too_many_traps. It is true that code which is parsed upstream of the very first profileBranch will potentially issue a non-trapping fallback, but by definition that code would be unrelated to the injected profile, so I don't see a harm in that. If this approach works, then you can remove the annotation altogether, which is clearly preferable. We understand the annotation now, but it has the danger of becoming a maintainer's puzzlement. > >> In 'updateCounters', if the counter overflows, you'll get continuous >> creation of ArithmeticExceptions. Will that optimize or will it cause a >> permanent slowdown? Consider a hack like this on the exception path: >> counters[idx] = Integer.MAX_VALUE / 2; > I had an impression that VM optimizes overflows in Math.exact* intrinsics, but it's not the case - it always inserts an uncommon trap. I used the workaround you proposed. Good. > >> On the Name Bikeshed: It looks like @IgnoreProfile (ignore_profile in >> the VM) promises too much "ignorance", since it suppresses branch counts >> and traps, but allows type profiles to be consulted. Maybe something >> positive like "@ManyTraps" or "@SharedMegamorphic"? (It's just a name, >> and this is just a suggestion.) > What do you think about @LambdaForm.Shared? That's fine. Suggest changing the JVM accessor to is_lambda_form_shared, because the term "shared" is already overused in the VM. Or, to be much more accurate, s/@Shared/@CollectiveProfile/. Better yet, get rid of it, as suggested above. (I just realized that profile pollution looks logically parallel to the http://en.wikipedia.org/wiki/Tragedy_of_the_commons .) Also, in the comment explaining the annotation: s/mostly useless/probably polluted by conflicting behavior from multiple call sites/ I very much like the fact that profileBranch is the VM intrinsic, not selectAlternative. A VM intrinsic should be nice and narrow like that. In fact, you can delete selectAlternative from vmSymbols while you are at it. (We could do profileInteger and profileClass in a similar way, if that turned out to be useful.) ? John From weijun.wang at oracle.com Fri Jan 23 02:18:35 2015 From: weijun.wang at oracle.com (Wang Weijun) Date: Fri, 23 Jan 2015 10:18:35 +0800 Subject: [9] request for review: 8049171: Additional tests for jarsigner's warnings In-Reply-To: <54C0E1A3.8070600@oracle.com> References: <54BF48BF.9090505@oracle.com> <54C0E1A3.8070600@oracle.com> Message-ID: > On Jan 22, 2015, at 19:40, Artem Smotrakov wrote: > >> I am not sure if I understand updateJar correctly. It looks like srcJarFile is opened multiple times so its entries are duplicated a lot in the destination. Or is there a secret break? > There is no any secret, just a bug. It is not necessary to open srcJarFile multiple times. > > I have updated the webrev, updateJar() method does the following: > - creates a new jar file (destJarFilename) > - puts files which are specified in files parameter to destJarFilename > - copies files from srcJarFilename to destJarFilename if they are no files with the same names in destJarFilename The process above means the new files are added at the beginning. While jarsigner is able to verify such a file (it uses JarFile) the output is actually invalid because the MANIFEST and the signature files must be at the beginning (otherwise a JarInputStream cannot verify it). The "jar u" way is to copy each old entry into destination unless the entry name is in the updated list where the new file will be read. Finally the untouched files in the updated list are appended. > > Here is an updated webrev with your suggested changes and a couple of others: > - added @ignore tag for BadKeyUsageTest since it fails due to a bug in JDK > - updated MultipleWarningsTest test to check ExtendedKeyUsage case instead of KeyUsage Great. Thanks Max > > http://cr.openjdk.java.net/~asmotrak/8049171/webrev.01/ > From peter.firmstone at zeus.net.au Fri Jan 23 07:10:51 2015 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Fri, 23 Jan 2015 17:10:51 +1000 Subject: Explicit Serialization API and Security In-Reply-To: References: Message-ID: <1421997051.2425.5.camel@Nokia-N900> And that is the point of this discussion, to determine how to do just that. But considering it's such a difficult problem, we haven't figured out how to yet. Peter. ----- Original message ----- > > Message: 2 > Date: Wed, 21 Jan 2015 15:43:13 -0600 > From: "David M. Lloyd" > > At some point, the responsibility *must* fall on the author of the > serializable class in question to avoid constructs which are exploitable > - like finalizers, and classes that can have side-effects before > validation can be completed. > > On 01/21/2015 02:26 PM, Peter Firmstone wrote: > > Don't forget that "null" may also be an invalid state. > > > > What I have learnt so far: > > > > 1. An attacker can craft a stream to obtain a reference to any object > > created during deserialization, using finalizers or circular links > > (there may be yet other undiscovered methods). > > 2. An attacker can craft a stream that deliberately doesn't satisfy > > invariants, in order to use an object to perform a function that > > wasn't intended by its developer. > > 3. Objects that interact with the stream directly using readObject et > > al, are often prone to DOS.? Example, many objects read a length > > integer from the stream when creating an array or collection, > > without first validating it. > > 4. Objects that interact directly with the stream become an implicit > > part of the stream protocol. > > 5. Once you allow an object to be created, it's too late to > > invalidate invariants, unless the class is final and invariants > > are checked in every method call. > > 6. We need to be able to restrict classes used for deserialization to > > those we trust to check invariants properly (but we haven't > > provided a way for them to avoid object construction yet). > > 7. A static validator method can ONLY be used to check field > > invariants, not other objects and primitives that are read > > directly from the stream by an arbitrary Object during the process > > of deserialization. > > 8. The jvm can be modified to delay finalizer registration for > > deserialization. > > 9. Circular links can be disallowed. > > > > Ultimately however, all proposed changes add complexity, but when an > > object has been created with invalid invariants, an attacker will find > > a way. > > > > Thank you all for your time, this has been a very good discussion. > > > > Regards, > > > > Peter. > > > > On 22/01/2015 2:27 AM, Chris Hegarty wrote: > > > On 20/01/15 20:22, Peter Levart wrote: > > > > Hi Chris and Peter, > > > > > > > > It just occurred to me that the following scheme would provide > > > > failure atomicity for the whole Object regardless of whether > > > > readObject methods are used or not for various classes in > > > > hierarchy: > > > > > > > > > > > > - allocate uninitialized instance > > > > - call default accessible constructor of the most specific > > > > non-Serializable class > > > > - deserialize (by calling readObject methods where provided) the > > > > fields of all classes in hierarchy like normally > > > > (up to this point, nothing is changed from what we have now) > > > > - if deserialization fails anywhere, undo everything by setting > > > > all the fields in the Serializable part of the hierarchy to > > > > default values (null for references, 0 for primitives), abandon > > > > the object and propagate failure. > > > > > > I think this is a good idea, and I can prototype something to this > > > affect. > > > > > > -Chris. > > > > > > > > > > While deserializing, the object is in inconsistent state. If > > > > deserialization fails, this state is rolled-back to uninitialized > > > > state. finalize() can still get to the instance, but it will be > > > > uninitialized. > > > > > > > > > > > > Peter > > > > > > > > On 01/14/2015 01:58 PM, Peter Firmstone wrote: > > > > > > > > > > Hi Chris, > > > > > > > > > > Sorry, no. > > > > > > > > > > Currently when an ObjectStreamClass is read in from the stream, > > > > > the framework searches for the first zero arg constructor of a > > > > > non serializable class and creates and instance of the class > > > > > read and resolved from the stream, however it does so using a > > > > > super class constructor. > > > > > > > > > > Then from the super class down, fields are read in and set in > > > > > order for each class in the object's inheritance hierarchy. > > > > > > > > > > The alternative I propose, doesn't create the instance, instead > > > > > it reads the fields from the stream, one by one and without > > > > > instantiating them, if they are newly read objects, stores them > > > > > temporarily into byte [] arrays in a Map with reference handle > > > > > keys, otherwise it just holds the reference handle. > > > > > > > > > > What it does next is wrap this information into a caller > > > > > sensitive api, GetFields or ReadSerial instance, that is passed > > > > > as a constructor parameter to the child class serial constructor. > > > > > > > > > > The child class checks invariants and reads each field it needs > > > > > using a static method prior to calling a superclass constructor, > > > > > each class in the inheritance hierarchy for the object then > > > > > checks its invariants until it gets to the first non > > > > > serializable superclass. > > > > > > > > > > The benefit of this order is that each class is present in the > > > > > thread security context, so protection domain security and > > > > > invariants are enforced before instantiating an object. > > > > > > > > > > Hope this helps illuminate it a little better, regards, > > > > > > > > > > Peter. > > > > > > > > > > ----- Original message ----- > > > > > > Peter F, > > > > > > > > > > > > I am still struggling with the basic concept of you proposal. > > > > > > Let me > > > > > see > > > > > > if I understand it correctly. Does the following describe a > > > > > > similar scenario as you envisage: > > > > > > > > > > > > 1) For each Serializable type, T, in the deserialized types > > > > > > hierarchy, starting with the top most ( closest to > > > > > j.l.Object ), > > > > > > > > > > > > 1a) Read T's fields from the stream, fields > > > > > > > > > > > > 1b) validate(t, fields)? // t will be null first time > > > > > > > > > > > > 1c) allocate a new instance of T, and assign to t > > > > > > > > > > > > 1d) set fields in t > > > > > > > > > > > > 2) Return t; > > > > > > > > > > > > So for each level in the hierarchy, an instance of a type is > > > > > > created only after its invariants have been checked. This > > > > > > instance is then passed to the next level so it can > > > > > > participate in that levels > > > > > invariants > > > > > > validation. > > > > > > > > > > > > If this scenario is along the same lines as yours, then I just > > > > > > don't > > > > > see > > > > > > how 1c above will always be possible. > > > > > > > > > > > > If we could somehow make the object caller sensitive until > > > > > > after deserialization completes, then could avoid having to > > > > > > try to allocate multiple instance down the hierarchy. > > > > > > > > > > > > -Chris. > > > > > > > > > > > > On 13/01/15 10:24, Peter Firmstone wrote: > > > > > > > Could we use a static validator method and generate bytecode > > > > > > > for constructors dynamically? > > > > > > > > > > > > > > The developer can optionally implement the constructors. > > > > > > > > > > > > > > static GetField invariantCheck(GetField f); > > > > > > > > > > > > > > Create a caller sensitive GetField implementation and add a > > > > > > > two new methods to GetField: > > > > > > > > > > > > > > abstract Object createSuper(); // to access superclass > > > > > > > objectmethods for inavariant checking. > > > > > > > > > > > > > > abstract Class getType(String name); > > > > > > > > > > > > > > Set fields from within constructors. > > > > > > > > > > > > > > The generated constructors are straight forward: > > > > > > > > > > > > > > 1. Call static method. > > > > > > > 2. Call super class constructor with result from static > > > > > > > method. 3. Set final fields > > > > > > > 4. How to set transient fields, implement a private method > > > > > > > called > > > > > from > > > > > > > within the constructor? > > > > > > > > > > > > > > Require a permission to extend GetField? > > > > > > > > > > > > > > > > > > > > -- > - DML > > > -------------- From ivan.gerasimov at oracle.com Fri Jan 23 08:10:55 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Fri, 23 Jan 2015 11:10:55 +0300 Subject: RFR 8067748: (process) Child is terminated when parent's console is closed [win] In-Reply-To: <54C13D7D.7030701@oracle.com> References: <54C10B14.3080908@oracle.com> <54C13D7D.7030701@oracle.com> Message-ID: <54C2020F.6060908@oracle.com> Thank you Alan! On 22.01.2015 21:12, Alan Bateman wrote: > On 22/01/2015 14:37, Ivan Gerasimov wrote: >> : >> It's hard to create a test, which involves closing the console window >> and manipulating the detached child application. >> >> Would you please help review the fix? >> >> BUGURL: https://bugs.openjdk.java.net/browse/JDK-8067748 >> WEBREV: http://cr.openjdk.java.net/~igerasim/8067748/0/webrev/ > Yet again, the Process code bites back :-) > Yes. I remember you warned about that :-) > I think this looks okay and we should probably have looked at redirect > and inheritance more when discussing the console window pop-up issue. > I don't personally like references to bug numbers in comments, others > will have different opinions. Okay I removed the reference. I agree that it can easily be found in the hg history anyway. > You have a good case for noreg-hard as it's not clear that it would be > easy to create an automated test for this. > I'll provide the textual instructions for the SQE in the comments to the bug. Sincerely yours, Ivan From artem.smotrakov at oracle.com Fri Jan 23 08:12:54 2015 From: artem.smotrakov at oracle.com (Artem Smotrakov) Date: Fri, 23 Jan 2015 11:12:54 +0300 Subject: [9] request for review: 8049171: Additional tests for jarsigner's warnings In-Reply-To: References: <54BF48BF.9090505@oracle.com> <54C0E1A3.8070600@oracle.com> Message-ID: <54C20286.1060106@oracle.com> Hi Max, Please see inline. On 01/23/2015 05:18 AM, Wang Weijun wrote: >> I have updated the webrev, updateJar() method does the following: >> >- creates a new jar file (destJarFilename) >> >- puts files which are specified in files parameter to destJarFilename >> >- copies files from srcJarFilename to destJarFilename if they are no files with the same names in destJarFilename > The process above means the new files are added at the beginning. Yes, "jar -tvf" shows the following for an original signed jar (before updateJar() is called) jar -tvf JTwork/scratch/0/signed.jar 83 Fri Jan 23 10:06:16 MSK 2015 META-INF/MANIFEST.MF 327 Fri Jan 23 10:06:16 MSK 2015 META-INF/ALIAS.SF 1091 Fri Jan 23 10:06:16 MSK 2015 META-INF/ALIAS.RSA 0 Fri Jan 23 10:06:10 MSK 2015 first.txt After pdateJar() is called, "jar -tvf" shows the following for modified jar jar -tvf JTwork/scratch/0/updated_signed.jar 0 Fri Jan 23 10:06:16 MSK 2015 second.txt 83 Fri Jan 23 10:06:16 MSK 2015 META-INF/MANIFEST.MF 327 Fri Jan 23 10:06:16 MSK 2015 META-INF/ALIAS.SF 1091 Fri Jan 23 10:06:16 MSK 2015 META-INF/ALIAS.RSA 0 Fri Jan 23 10:06:10 MSK 2015 first.txt > While jarsigner is able to verify such a file (it uses JarFile) the output is actually invalid because the MANIFEST and the signature files must be at the beginning (otherwise a JarInputStream cannot verify it). jarsigner successfully verifies a jar file that was modified by updateJar() method, please see a piece of log for sun/security/tools/jarsigner/warnings/HasUnsignedEntryTest.java test Copy signed.jar to updated_signed.jar, and add second.txt.class, so it contains unsigned entry Updating updated_signed.jar with second.txt Copying META-INF/MANIFEST.MF to updated_signed.jar Copying META-INF/ALIAS.SF to updated_signed.jar Copying META-INF/ALIAS.RSA to updated_signed.jar Copying first.txt to updated_signed.jar Command line: [/home/artem/ws/jdk/jdk9_dev/build/linux-x86_64-normal-server-fastdebug/jdk/bin/jarsigner -verify -verbose -keystore keystore.jks -storepass password -keypass password updated_signed.jar] 0 Fri Jan 23 10:06:16 MSK 2015 second.txt s k 83 Fri Jan 23 10:06:16 MSK 2015 META-INF/MANIFEST.MF 327 Fri Jan 23 10:06:16 MSK 2015 META-INF/ALIAS.SF 1091 Fri Jan 23 10:06:16 MSK 2015 META-INF/ALIAS.RSA smk 0 Fri Jan 23 10:06:10 MSK 2015 first.txt s = signature was verified m = entry is listed in manifest k = at least one certificate was found in keystore i = at least one certificate was found in identity scope jar verified. Warning: This jar contains unsigned entries which have not been integrity-checked. This jar contains signatures that does not include a timestamp. Without a timestamp, users may not be able to validate this jar after the signer certificate's expiration date (2016-01-23) or after any future revocation date. Re-run with the -verbose and -certs options for more details. If the MANIFEST and the signature files must be at the beginning, should it be considered as a bug in jarsigner? Should it reject such files? > > The "jar u" way is to copy each old entry into destination unless the entry name is in the updated list where the new file will be read. Finally the untouched files in the updated list are appended. Since tests were not originally for checking some unusual ways for updating jars, I think they need to be updated to use the "jar u" way for adding unsigned entry. Artem > From chris.hegarty at oracle.com Fri Jan 23 10:28:58 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 23 Jan 2015 10:28:58 +0000 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54C0C629.6010500@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <548EC2D7.6000604@oracle.com> <548EC534.2090803@oracle.com> <548ECDA4.9070002@oracle.com> <54B8ECD7.3010703@oracle.com> <54B8F630.4040608@oracle.com> <54B8F90A.7010402@oracle.com> <54B908FA.9020405@oracle.com> <54B919ED.8080403@oracle.com> <54BFA473.5070905@oracle.com> <54C0C629.6010500@oracle.com> Message-ID: <54C2226A.9070608@oracle.com> > On 21.01.2015 16:06, Konstantin Shefov wrote: >> Hi, Chris >> >> I have done a pre-integration testing for this fix on Linux, Windows, >> Mac and Solaris x64 and sparc. I have run all regression tests for >> core libraries, and no new failures appeared. >> >> Can I push this change now or should I wait for an approve by one more >> person? You have a Reviewer, and I see no objections. So you are good to push. -Chris. >> Thanks >> -Konstantin >> >> On 16.01.2015 17:02, Chris Hegarty wrote: >>> On 16/01/15 12:50, Konstantin Shefov wrote: >>>> Hi Chris, Alan, thank you for reviewing. >>>> >>>> I have made a new webrev >>>> http://cr.openjdk.java.net/~kshefov/6933879/webrev.01 >>>> I have removed ":" and added a test case. >>> >>> This looks ok to me. >>> >>> -Chris. >>> >>>> -Konstantin >>>> >>>> 16.01.2015 14:42, Chris Hegarty ?????: >>>>> On 16/01/15 11:29, Alan Bateman wrote: >>>>>> On 16/01/2015 10:49, Chris Hegarty wrote: >>>>>>> : >>>>>>> >>>>>>> I don't see any reason to update the spec here, given that the >>>>>>> set of >>>>>>> allowable character is not clearly defined in the relevant RFC's. >>>>>>> >>>>>> I think we need to create a bug to look into this more. Inet6Address >>>>>> allows the scope of be any String but URI puts restrictions on >>>>>> what is >>>>>> legal. The proposed change updates the list of allowed characters >>>>>> but I >>>>>> assume it will need to re-visited again as systems with more exotic >>>>>> interface names are encountered. >>>>> >>>>> Yes that is possible, but to date this is the first time I've seen >>>>> this come up. >>>>> >>>>> Given the lack of clear specification in the relevant RFC's (relating >>>>> to scope ids ), I don't think we should be making too many >>>>> specification changes in this area. But I agree with your comment, URI >>>>> has a restriction that it not clearly specified in the Java spec. >>>>> >>>>> -Chris. >>>> >> > From Alan.Bateman at oracle.com Fri Jan 23 12:49:45 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 23 Jan 2015 12:49:45 +0000 Subject: [9] Review request : JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54C2226A.9070608@oracle.com> References: <54886510.1010704@oracle.com> <54897B63.30801@oracle.com> <548EC2D7.6000604@oracle.com> <548EC534.2090803@oracle.com> <548ECDA4.9070002@oracle.com> <54B8ECD7.3010703@oracle.com> <54B8F630.4040608@oracle.com> <54B8F90A.7010402@oracle.com> <54B908FA.9020405@oracle.com> <54B919ED.8080403@oracle.com> <54BFA473.5070905@oracle.com> <54C0C629.6010500@oracle.com> <54C2226A.9070608@oracle.com> Message-ID: <54C24369.5070608@oracle.com> On 23/01/2015 10:28, Chris Hegarty wrote: >> On 21.01.2015 16:06, Konstantin Shefov wrote: >>> Hi, Chris >>> >>> I have done a pre-integration testing for this fix on Linux, Windows, >>> Mac and Solaris x64 and sparc. I have run all regression tests for >>> core libraries, and no new failures appeared. >>> >>> Can I push this change now or should I wait for an approve by one more >>> person? > > You have a Reviewer, and I see no objections. So you are good to push. > Can you make sure to link this to a new issue to look at the spec issue that this exposes? -Alan From konstantin.shefov at oracle.com Fri Jan 23 12:56:49 2015 From: konstantin.shefov at oracle.com (Konstantin Shefov) Date: Fri, 23 Jan 2015 15:56:49 +0300 Subject: [8u-dev] Request for approval: JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id Message-ID: <54C24511.4020902@oracle.com> Hello, Please approve the direct backport of the bug fix to 8u-dev Patch applies cleanly to JDK 8u. The bug:https://bugs.openjdk.java.net/browse/JDK-6933879 The webrev: http://cr.openjdk.java.net/~kshefov/6933879/webrev.01/ JDK 9 changeset: http://hg.openjdk.java.net/jdk9/dev/jdk/rev/801eb37fc6c4 Review thread: http://mail.openjdk.java.net/pipermail/net-dev/2015-January/008834.html Thanks -Konstantin From sean.coffey at oracle.com Fri Jan 23 13:47:36 2015 From: sean.coffey at oracle.com (=?UTF-8?B?U2XDoW4gQ29mZmV5?=) Date: Fri, 23 Jan 2015 13:47:36 +0000 Subject: [8u-dev] Request for approval: JDK-6933879: URISyntaxException when non-alphanumeric characters are present in scope_id In-Reply-To: <54C24511.4020902@oracle.com> References: <54C24511.4020902@oracle.com> Message-ID: <54C250F8.70808@oracle.com> Konstantin, can you hold off pushing this fix to jdk8u for the moment ? It's a P4 and could have behavioural consequences (something we try and avoid in update releases). I see JDK-8071458 was logged to track IPv6 scope specifications. Let's see how this soaks into JDK 9 over coming days and let's see how JDK-8071458 progresses. regards, Sean. On 23/01/15 12:56, Konstantin Shefov wrote: > Hello, > > Please approve the direct backport of the bug fix to 8u-dev > > Patch applies cleanly to JDK 8u. > > The bug:https://bugs.openjdk.java.net/browse/JDK-6933879 > The webrev: http://cr.openjdk.java.net/~kshefov/6933879/webrev.01/ > > JDK 9 changeset: http://hg.openjdk.java.net/jdk9/dev/jdk/rev/801eb37fc6c4 > Review thread: > http://mail.openjdk.java.net/pipermail/net-dev/2015-January/008834.html > > Thanks > -Konstantin From erik.joelsson at oracle.com Fri Jan 23 14:35:42 2015 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Fri, 23 Jan 2015 15:35:42 +0100 Subject: RFR: JDK-8055190: Cleanup include and exclude of core-libs native libraries after source code restructure Message-ID: <54C25C3E.4030501@oracle.com> Hello, Please review this build patch, cleaning up some in the native compilation of some core libraries. After the source code restructure into modules, the need for explicit exclude and include of source files was reduced. This patch moves some OS specific files to OS specific source dirs and adjusts the makefiles. It's based on Magnus' review comments on the source code restructure change. Bug: https://bugs.openjdk.java.net/browse/JDK-8055190 Webrev: http://cr.openjdk.java.net/~erikj/8055190/webrev.01/ /Erik From chris.hegarty at oracle.com Fri Jan 23 14:41:52 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 23 Jan 2015 14:41:52 +0000 Subject: Explicit Serialization API and Security In-Reply-To: <54C01D71.9010901@redhat.com> References: <54B0CE06.2020408@zeus.net.au> <54B3EEC5.5040805@oracle.com> <1421144693.7530.20.camel@Nokia-N900> <54B654F6.6060207@oracle.com> <1421240334.8856.17.camel@Nokia-N900> <54BEB91A.4020603@gmail.com> <54BFD374.5080803@oracle.com> <54C00B66.5090302@zeus.net.au> <54C01D71.9010901@redhat.com> Message-ID: <54C25DB0.4080207@oracle.com> I have attempted to capture some of the ideas that we have discussed so far. https://bugs.openjdk.java.net/browse/JDK-8071471 -Chris. On 21/01/15 21:43, David M. Lloyd wrote: > At some point, the responsibility *must* fall on the author of the > serializable class in question to avoid constructs which are exploitable > - like finalizers, and classes that can have side-effects before > validation can be completed. > > On 01/21/2015 02:26 PM, Peter Firmstone wrote: >> Don't forget that "null" may also be an invalid state. >> >> What I have learnt so far: >> >> 1. An attacker can craft a stream to obtain a reference to any object >> created during deserialization, using finalizers or circular links >> (there may be yet other undiscovered methods). >> 2. An attacker can craft a stream that deliberately doesn't satisfy >> invariants, in order to use an object to perform a function that >> wasn't intended by its developer. >> 3. Objects that interact with the stream directly using readObject et >> al, are often prone to DOS. Example, many objects read a length >> integer from the stream when creating an array or collection, >> without first validating it. >> 4. Objects that interact directly with the stream become an implicit >> part of the stream protocol. >> 5. Once you allow an object to be created, it's too late to >> invalidate invariants, unless the class is final and invariants >> are checked in every method call. >> 6. We need to be able to restrict classes used for deserialization to >> those we trust to check invariants properly (but we haven't >> provided a way for them to avoid object construction yet). >> 7. A static validator method can ONLY be used to check field >> invariants, not other objects and primitives that are read >> directly from the stream by an arbitrary Object during the process >> of deserialization. >> 8. The jvm can be modified to delay finalizer registration for >> deserialization. >> 9. Circular links can be disallowed. >> >> Ultimately however, all proposed changes add complexity, but when an >> object has been created with invalid invariants, an attacker will find a >> way. >> >> Thank you all for your time, this has been a very good discussion. >> >> Regards, >> >> Peter. >> >> On 22/01/2015 2:27 AM, Chris Hegarty wrote: >>> On 20/01/15 20:22, Peter Levart wrote: >>>> Hi Chris and Peter, >>>> >>>> It just occurred to me that the following scheme would provide failure >>>> atomicity for the whole Object regardless of whether readObject methods >>>> are used or not for various classes in hierarchy: >>>> >>>> >>>> - allocate uninitialized instance >>>> - call default accessible constructor of the most specific >>>> non-Serializable class >>>> - deserialize (by calling readObject methods where provided) the fields >>>> of all classes in hierarchy like normally >>>> (up to this point, nothing is changed from what we have now) >>>> - if deserialization fails anywhere, undo everything by setting all the >>>> fields in the Serializable part of the hierarchy to default values >>>> (null >>>> for references, 0 for primitives), abandon the object and propagate >>>> failure. >>> >>> I think this is a good idea, and I can prototype something to this >>> affect. >>> >>> -Chris. >>> >>> >>>> While deserializing, the object is in inconsistent state. If >>>> deserialization fails, this state is rolled-back to uninitialized >>>> state. >>>> finalize() can still get to the instance, but it will be uninitialized. >>>> >>>> >>>> Peter >>>> >>>> On 01/14/2015 01:58 PM, Peter Firmstone wrote: >>>>> >>>>> Hi Chris, >>>>> >>>>> Sorry, no. >>>>> >>>>> Currently when an ObjectStreamClass is read in from the stream, the >>>>> framework searches for the first zero arg constructor of a non >>>>> serializable class and creates and instance of the class read and >>>>> resolved from the stream, however it does so using a super class >>>>> constructor. >>>>> >>>>> Then from the super class down, fields are read in and set in order >>>>> for each class in the object's inheritance hierarchy. >>>>> >>>>> The alternative I propose, doesn't create the instance, instead it >>>>> reads the fields from the stream, one by one and without instantiating >>>>> them, if they are newly read objects, stores them temporarily into >>>>> byte [] arrays in a Map with reference handle keys, otherwise it just >>>>> holds the reference handle. >>>>> >>>>> What it does next is wrap this information into a caller sensitive >>>>> api, GetFields or ReadSerial instance, that is passed as a constructor >>>>> parameter to the child class serial constructor. >>>>> >>>>> The child class checks invariants and reads each field it needs using >>>>> a static method prior to calling a superclass constructor, each class >>>>> in the inheritance hierarchy for the object then checks its invariants >>>>> until it gets to the first non serializable superclass. >>>>> >>>>> The benefit of this order is that each class is present in the thread >>>>> security context, so protection domain security and invariants are >>>>> enforced before instantiating an object. >>>>> >>>>> Hope this helps illuminate it a little better, regards, >>>>> >>>>> Peter. >>>>> >>>>> ----- Original message ----- >>>>> > Peter F, >>>>> > >>>>> > I am still struggling with the basic concept of you proposal. Let me >>>>> see >>>>> > if I understand it correctly. Does the following describe a similar >>>>> > scenario as you envisage: >>>>> > >>>>> > 1) For each Serializable type, T, in the deserialized types >>>>> > hierarchy, starting with the top most ( closest to >>>>> j.l.Object ), >>>>> > >>>>> > 1a) Read T's fields from the stream, fields >>>>> > >>>>> > 1b) validate(t, fields) // t will be null first time >>>>> > >>>>> > 1c) allocate a new instance of T, and assign to t >>>>> > >>>>> > 1d) set fields in t >>>>> > >>>>> > 2) Return t; >>>>> > >>>>> > So for each level in the hierarchy, an instance of a type is created >>>>> > only after its invariants have been checked. This instance is then >>>>> > passed to the next level so it can participate in that levels >>>>> invariants >>>>> > validation. >>>>> > >>>>> > If this scenario is along the same lines as yours, then I just don't >>>>> see >>>>> > how 1c above will always be possible. >>>>> > >>>>> > If we could somehow make the object caller sensitive until after >>>>> > deserialization completes, then could avoid having to try to >>>>> allocate >>>>> > multiple instance down the hierarchy. >>>>> > >>>>> > -Chris. >>>>> > >>>>> > On 13/01/15 10:24, Peter Firmstone wrote: >>>>> > > Could we use a static validator method and generate bytecode for >>>>> > > constructors dynamically? >>>>> > > >>>>> > > The developer can optionally implement the constructors. >>>>> > > >>>>> > > static GetField invariantCheck(GetField f); >>>>> > > >>>>> > > Create a caller sensitive GetField implementation and add a two >>>>> new >>>>> > > methods to GetField: >>>>> > > >>>>> > > abstract Object createSuper(); // to access superclass >>>>> objectmethods >>>>> > > for inavariant checking. >>>>> > > >>>>> > > abstract Class getType(String name); >>>>> > > >>>>> > > Set fields from within constructors. >>>>> > > >>>>> > > The generated constructors are straight forward: >>>>> > > >>>>> > > 1. Call static method. >>>>> > > 2. Call super class constructor with result from static method. >>>>> > > 3. Set final fields >>>>> > > 4. How to set transient fields, implement a private method called >>>>> from >>>>> > > within the constructor? >>>>> > > >>>>> > > Require a permission to extend GetField? >>>>> > > >>>>> >>>> >> > From peter.levart at gmail.com Fri Jan 23 14:38:58 2015 From: peter.levart at gmail.com (Peter Levart) Date: Fri, 23 Jan 2015 15:38:58 +0100 Subject: [9] RFR (M): 8069591: Customize LambdaForms which are invoked using MH.invoke/invokeExact In-Reply-To: References: <54BFD2EE.3060909@oracle.com> <54BFE259.1090402@univ-mlv.fr> <3B4D19E0-8DA6-4FE1-BD77-E12E8BCF15EC@oracle.com> <54C139CE.4000005@oracle.com> Message-ID: <54C25D02.2020209@gmail.com> On 01/23/2015 12:30 AM, John Rose wrote: > On Jan 22, 2015, at 9:56 AM, Vladimir Ivanov wrote: >> Remi, John, thanks for review! >> >> Updated webrev: >> http://cr.openjdk.java.net/~vlivanov/8069591/webrev.01/ >> >> This time I did additional testing (COMPILE_THRESHOLD > 0) and spotted a problem with MethodHandle.copyWith(): a MethodHandle can inherit customized LambdaForm this way. I could have added LambdaForm::uncustomize() call in evey Species_*::copyWith() method, but I decided to add it into MethodHandle constructor. Let me know if you think it's too intrusive. > It's OK to put it there. > > Now I'm worried that the new customization logic will defeat code sharing for invoked MHs, since uncustomize creates a new LF that is a duplicate of the original LF. That breaks the genetic link for children of the invoked MH, doesn't it? (I like the compileToBytecode call, if it is done on the original.) In fact, that is also a potential problem for the first version of your patch, also. > > Suggestion: Have every customized LF contain a direct link to its uncustomized original. Have uncustomize just return that same original, every time. Then, when using LF editor operations to derive new LFs, always have them extract the original before making a derivation. The customized LF then don't need 'transformCache' field. It could be re-used to point to original uncustomized LF. That would also be a signal for LF editor (the 4th type of payload attached to transformCache field) to follow the link to get to the uncustomized LF... Peter > > (Alternatively, have the LF editor caches be shared between original LFs and all their customized versions. But that doesn't save all the genetic links.) > >> Also, I made DirectMethodHandles a special-case, since I don't see any benefit in customizing them. > The overriding method in DHM should be marked @Override, so that we know all the bits fit together. > > ? John > _______________________________________________ > mlvm-dev mailing list > mlvm-dev at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev From Alan.Bateman at oracle.com Fri Jan 23 14:41:32 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 23 Jan 2015 14:41:32 +0000 Subject: RFR: JDK-8055190: Cleanup include and exclude of core-libs native libraries after source code restructure In-Reply-To: <54C25C3E.4030501@oracle.com> References: <54C25C3E.4030501@oracle.com> Message-ID: <54C25D9C.2040408@oracle.com> On 23/01/2015 14:35, Erik Joelsson wrote: > Hello, > > Please review this build patch, cleaning up some in the native > compilation of some core libraries. After the source code restructure > into modules, the need for explicit exclude and include of source > files was reduced. This patch moves some OS specific files to OS > specific source dirs and adjusts the makefiles. It's based on Magnus' > review comments on the source code restructure change. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8055190 > Webrev: http://cr.openjdk.java.net/~erikj/8055190/webrev.01/ > > /Erik This looks a good clean-up and demonstrates the usefulness of the new source layout. Can refs.allowed be removed? I thought we removed the JDK 8 dependency checker early in JDK 9 so this file shouldn't be needed. -Alan From erik.joelsson at oracle.com Fri Jan 23 14:59:18 2015 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Fri, 23 Jan 2015 15:59:18 +0100 Subject: RFR: JDK-8055190: Cleanup include and exclude of core-libs native libraries after source code restructure In-Reply-To: <54C25D9C.2040408@oracle.com> References: <54C25C3E.4030501@oracle.com> <54C25D9C.2040408@oracle.com> Message-ID: <54C261C6.5030705@oracle.com> On 2015-01-23 15:41, Alan Bateman wrote: > On 23/01/2015 14:35, Erik Joelsson wrote: >> Hello, >> >> Please review this build patch, cleaning up some in the native >> compilation of some core libraries. After the source code restructure >> into modules, the need for explicit exclude and include of source >> files was reduced. This patch moves some OS specific files to OS >> specific source dirs and adjusts the makefiles. It's based on Magnus' >> review comments on the source code restructure change. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8055190 >> Webrev: http://cr.openjdk.java.net/~erikj/8055190/webrev.01/ >> >> /Erik > This looks a good clean-up and demonstrates the usefulness of the new > source layout. > Thanks > Can refs.allowed be removed? I thought we removed the JDK 8 dependency > checker early in JDK 9 so this file shouldn't be needed. > I was wondering what it was and could not find any usages. With your confirmation I have opted to remove it completely instead. Updated webrev: http://cr.openjdk.java.net/~erikj/8055190/webrev.02/ /Erik From vladimir.x.ivanov at oracle.com Fri Jan 23 16:00:53 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Fri, 23 Jan 2015 19:00:53 +0300 Subject: [9] RFR (M): 8069591: Customize LambdaForms which are invoked using MH.invoke/invokeExact In-Reply-To: <54C25D02.2020209@gmail.com> References: <54BFD2EE.3060909@oracle.com> <54BFE259.1090402@univ-mlv.fr> <3B4D19E0-8DA6-4FE1-BD77-E12E8BCF15EC@oracle.com> <54C139CE.4000005@oracle.com> <54C25D02.2020209@gmail.com> Message-ID: <54C27035.5050103@oracle.com> Good idea, Peter! Updated version: http://cr.openjdk.java.net/~vlivanov/8069591/webrev.02/ Best regards, Vladimir Ivanov On 1/23/15 5:38 PM, Peter Levart wrote: > On 01/23/2015 12:30 AM, John Rose wrote: >> On Jan 22, 2015, at 9:56 AM, Vladimir Ivanov >> wrote: >>> Remi, John, thanks for review! >>> >>> Updated webrev: >>> http://cr.openjdk.java.net/~vlivanov/8069591/webrev.01/ >>> >>> This time I did additional testing (COMPILE_THRESHOLD > 0) and >>> spotted a problem with MethodHandle.copyWith(): a MethodHandle can >>> inherit customized LambdaForm this way. I could have added >>> LambdaForm::uncustomize() call in evey Species_*::copyWith() method, >>> but I decided to add it into MethodHandle constructor. Let me know if >>> you think it's too intrusive. >> It's OK to put it there. >> >> Now I'm worried that the new customization logic will defeat code >> sharing for invoked MHs, since uncustomize creates a new LF that is a >> duplicate of the original LF. That breaks the genetic link for >> children of the invoked MH, doesn't it? (I like the compileToBytecode >> call, if it is done on the original.) In fact, that is also a >> potential problem for the first version of your patch, also. >> >> Suggestion: Have every customized LF contain a direct link to its >> uncustomized original. Have uncustomize just return that same >> original, every time. Then, when using LF editor operations to derive >> new LFs, always have them extract the original before making a >> derivation. > > The customized LF then don't need 'transformCache' field. It could be > re-used to point to original uncustomized LF. That would also be a > signal for LF editor (the 4th type of payload attached to transformCache > field) to follow the link to get to the uncustomized LF... > > Peter > >> >> (Alternatively, have the LF editor caches be shared between original >> LFs and all their customized versions. But that doesn't save all the >> genetic links.) >> >>> Also, I made DirectMethodHandles a special-case, since I don't see >>> any benefit in customizing them. >> The overriding method in DHM should be marked @Override, so that we >> know all the bits fit together. >> >> ? John >> _______________________________________________ >> mlvm-dev mailing list >> mlvm-dev at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev > From Roger.Riggs at Oracle.com Fri Jan 23 16:11:54 2015 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Fri, 23 Jan 2015 11:11:54 -0500 Subject: RFR 9: 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java Message-ID: <54C272CA.5030102@Oracle.com> Please review this merge of UNIXProcess.java with ProcessImpl.java. For Unix variants of Process, the implementation is split between src/java.base/unix/classes/java/lang/ProcessImpl.java and UNIXProcess.java. ProcessImpl.java contains only static methods and delegates to UNIXProcess.java. Merging UNIXProcess.java into ProcessImpl.java removes a redundant method and aligns the source files with the Windows implementation that contains only ProcessImpl.java. Webrev: http://cr.openjdk.java.net/~rriggs/webrev-unixprocess-rename-8071481/ Issue: 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java [1] https://bugs.openjdk.java.net/browse/JDK-8071481 From paul.sandoz at oracle.com Fri Jan 23 17:00:53 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 23 Jan 2015 17:00:53 +0000 Subject: RFR 8071477: Better Spliterator implementations for String.chars() and String.codePoints() Message-ID: Hi, http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8071477-String-spliterators/webrev/ This patch implements better spliterators for String/Buffer/Builder.chars/codePoints than those provided by the default methods on CharSequence. The test java/lang/CharSequence/DefaultTest.java is removed as i now pass the spliterators through the grinder of the spliterator-and-traversing tests. Thanks, Paul. From peter.levart at gmail.com Fri Jan 23 17:18:15 2015 From: peter.levart at gmail.com (Peter Levart) Date: Fri, 23 Jan 2015 18:18:15 +0100 Subject: RFR 9: 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java In-Reply-To: <54C272CA.5030102@Oracle.com> References: <54C272CA.5030102@Oracle.com> Message-ID: <54C28257.2090902@gmail.com> Hi Roger, Nice move/rename/merge. I compared old UNIXProcess.java with new ProcessImpl.java using KDiff3 and they match perfectly as well as including the old ProcessImpl.java code. Regards, Peter On 01/23/2015 05:11 PM, Roger Riggs wrote: > Please review this merge of UNIXProcess.java with ProcessImpl.java. > > For Unix variants of Process, the implementation is split between > src/java.base/unix/classes/java/lang/ProcessImpl.java and > UNIXProcess.java. > ProcessImpl.java contains only static methods and delegates to > UNIXProcess.java. > Merging UNIXProcess.java into ProcessImpl.java removes a redundant method > and aligns the source files with the Windows implementation that > contains only ProcessImpl.java. > > Webrev: > http://cr.openjdk.java.net/~rriggs/webrev-unixprocess-rename-8071481/ > > Issue: > 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java > > [1] https://bugs.openjdk.java.net/browse/JDK-8071481 From chris.hegarty at oracle.com Fri Jan 23 17:53:53 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 23 Jan 2015 17:53:53 +0000 Subject: RFR 9: 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java In-Reply-To: <54C28257.2090902@gmail.com> References: <54C272CA.5030102@Oracle.com> <54C28257.2090902@gmail.com> Message-ID: <823A6941-ABF8-43F8-9CBC-BBC6A19472D2@oracle.com> On 23 Jan 2015, at 17:18, Peter Levart wrote: > Hi Roger, > > Nice move/rename/merge. > > I compared old UNIXProcess.java with new ProcessImpl.java using KDiff3 and they match perfectly as well as including the old ProcessImpl.java code. Nice. I did the same just to see it for myself. The other parts of the webrev look good too. -Chris. > Regards, Peter > > On 01/23/2015 05:11 PM, Roger Riggs wrote: >> Please review this merge of UNIXProcess.java with ProcessImpl.java. >> >> For Unix variants of Process, the implementation is split between >> src/java.base/unix/classes/java/lang/ProcessImpl.java and UNIXProcess.java. >> ProcessImpl.java contains only static methods and delegates to UNIXProcess.java. >> Merging UNIXProcess.java into ProcessImpl.java removes a redundant method >> and aligns the source files with the Windows implementation that contains only ProcessImpl.java. >> >> Webrev: >> http://cr.openjdk.java.net/~rriggs/webrev-unixprocess-rename-8071481/ >> >> Issue: >> 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8071481 > From john.r.rose at oracle.com Fri Jan 23 18:13:40 2015 From: john.r.rose at oracle.com (John Rose) Date: Fri, 23 Jan 2015 10:13:40 -0800 Subject: [9] RFR (M): 8069591: Customize LambdaForms which are invoked using MH.invoke/invokeExact In-Reply-To: <54C27035.5050103@oracle.com> References: <54BFD2EE.3060909@oracle.com> <54BFE259.1090402@univ-mlv.fr> <3B4D19E0-8DA6-4FE1-BD77-E12E8BCF15EC@oracle.com> <54C139CE.4000005@oracle.com> <54C25D02.2020209@gmail.com> <54C27035.5050103@oracle.com> Message-ID: On Jan 23, 2015, at 8:00 AM, Vladimir Ivanov wrote: > > Good idea, Peter! +1 > Updated version: > http://cr.openjdk.java.net/~vlivanov/8069591/webrev.02/ Yes, that's good, and you can count me as a reviewer. ? John P.S. One could also get rid of the LF.customized field by stuffing both that value and the original LF in the transformCache (as a 2-array), but that's overkill. P.P.S. A possible generalization to the LF.customized field would be an optional list of type, value, and/or structure constraints for one or more arguments to the LF. Then we could (a) customize on additional arguments if we thought that were useful, and/or (b) produce semi-custom code that could be shared by more than one MH, if we thought there was an interesting equivalence class of MHs to speed up with common code. From xueming.shen at oracle.com Fri Jan 23 18:22:10 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Fri, 23 Jan 2015 10:22:10 -0800 Subject: RFR 8071477: Better Spliterator implementations for String.chars() and String.codePoints() In-Reply-To: References: Message-ID: <54C29152.7090608@oracle.com> On 01/23/2015 09:00 AM, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8071477-String-spliterators/webrev/ > > This patch implements better spliterators for String/Buffer/Builder.chars/codePoints than those provided by the default methods on CharSequence. > > The test java/lang/CharSequence/DefaultTest.java is removed as i now pass the spliterators through the grinder of the spliterator-and-traversing tests. > > Thanks, > Paul. I'm a little confused at following logic. 2997 // Mid-point is a high-surrogate 2998 // Or mid-point and the previous are low-surrogates 2999 if (Character.isHighSurrogate(array[mid]) || 3000 Character.isLowSurrogate(array[midOneLess = (mid - 1)])) 3001 return new CodePointsSpliterator(array, lo, index = mid, cs); Shouldn't it be something like if (!Character.isLowSurrogate(array[mid]) || !Character.isHighSurrogate(array[midOneLess = (mid -1)])) { return new CodePointsSpliterator(array, lo, index = mid, cs); } For example, in case both "mid" and "midOneLess" are normal non-surrogate character, I would assume the trySplit() should return [lo, index=mid) as wekk? or something like ... if (Character.isLowSurrogate(array[mid]) || Character.isHighSurrogate(array[midOneLess = (mid -1)])) { if (lo >= midOneLess) return null; return new CodePointsSpliterator(array, lo, index = midOneless, cs); } return new CodePointsSpliterator(array, lo, index = mid, cs); ... means, we only return [lo, midOneless), if mid is in the middle of a surrogate pair (midOneLess is hiSurr, mid is hoSurr)? btw, is it worth having a "nextCodePoint()" to be shared by forEachRemaining and tryAdvance()? -sherman From huizhe.wang at oracle.com Fri Jan 23 18:29:56 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Fri, 23 Jan 2015 10:29:56 -0800 Subject: RFR: 8062923: XSL: Run-time internal error in 'substring()' In-Reply-To: <54BCE9B6.7090506@oracle.com> References: <54BCE9B6.7090506@oracle.com> Message-ID: <54C29324.8050904@oracle.com> Hi Aleksej, Looks good. There is a minor format issue at line 299. Note that, for JDK 9, we now have functional/unit tests under jaxp/test. This test would fix in javax/xml/jaxp/unittest/javax/xml/transform. For test format, please take a look at a functional test (javax/xml/jaxp/functional) since Tristan has refactored them, and SQE plans to do the same with the unit tests. For a testng test, the @run tag is not necessary. The jaxp tests are defined in its own testset "jaxp". Thanks, Joe On 1/19/2015 3:25 AM, Aleksej Efimov wrote: > Hi, > > Please, review the fix for the XSL substring function failures. Two > issues were reported during usage of this function: > 1. Runtime internal error with negative length: > https://bugs.openjdk.java.net/browse/JDK-8062923 > 2. Wrong answer when -Inf length is used: > https://bugs.openjdk.java.net/browse/JDK-8062924 > > These two issues can be solved by fix proposed for the review: > http://cr.openjdk.java.net/~aefimov/8062923/webrev.00 > > Testing: > xml related regression tests (with new test) - no failures (from jdk > and jaxp repos). > JCK xml related tests - no failures. > > With Best Regards, > Aleksej From Roger.Riggs at Oracle.com Fri Jan 23 18:57:11 2015 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Fri, 23 Jan 2015 13:57:11 -0500 Subject: RFR 9: 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java In-Reply-To: <823A6941-ABF8-43F8-9CBC-BBC6A19472D2@oracle.com> References: <54C272CA.5030102@Oracle.com> <54C28257.2090902@gmail.com> <823A6941-ABF8-43F8-9CBC-BBC6A19472D2@oracle.com> Message-ID: <54C29987.3060003@Oracle.com> Hi Peter, Chris, Thanks for the reviews. I found one more reference to UNIXProcess in a comment in childproc.h and will wait for a jprt run to finish. Webrev: http://cr.openjdk.java.net/~rriggs/webrev-unixprocess-rename-8071481/ Thanks, Roger On 1/23/2015 12:53 PM, Chris Hegarty wrote: > On 23 Jan 2015, at 17:18, Peter Levart wrote: > >> Hi Roger, >> >> Nice move/rename/merge. >> >> I compared old UNIXProcess.java with new ProcessImpl.java using KDiff3 and they match perfectly as well as including the old ProcessImpl.java code. > Nice. I did the same just to see it for myself. The other parts of the webrev look good too. > > -Chris. > >> Regards, Peter >> >> On 01/23/2015 05:11 PM, Roger Riggs wrote: >>> Please review this merge of UNIXProcess.java with ProcessImpl.java. >>> >>> For Unix variants of Process, the implementation is split between >>> src/java.base/unix/classes/java/lang/ProcessImpl.java and UNIXProcess.java. >>> ProcessImpl.java contains only static methods and delegates to UNIXProcess.java. >>> Merging UNIXProcess.java into ProcessImpl.java removes a redundant method >>> and aligns the source files with the Windows implementation that contains only ProcessImpl.java. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~rriggs/webrev-unixprocess-rename-8071481/ >>> >>> Issue: >>> 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8071481 From chris.hegarty at oracle.com Fri Jan 23 19:03:48 2015 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 23 Jan 2015 19:03:48 +0000 Subject: RFR 9: 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java In-Reply-To: <54C29987.3060003@Oracle.com> References: <54C272CA.5030102@Oracle.com> <54C28257.2090902@gmail.com> <823A6941-ABF8-43F8-9CBC-BBC6A19472D2@oracle.com> <54C29987.3060003@Oracle.com> Message-ID: <4F2E81BA-6E18-480D-A687-B8D529CD7D85@oracle.com> On 23 Jan 2015, at 18:57, Roger Riggs wrote: > Hi Peter, Chris, > > Thanks for the reviews. > > I found one more reference to UNIXProcess in a comment in childproc.h Ah, good find. Another reference in comments in BigFork.java, but probably doesn?t make sense to change at this point. -Chris. > and will wait for a jprt run to finish. > > Webrev: > > http://cr.openjdk.java.net/~rriggs/webrev-unixprocess-rename-8071481/ > > Thanks, Roger > > On 1/23/2015 12:53 PM, Chris Hegarty wrote: >> On 23 Jan 2015, at 17:18, Peter Levart wrote: >> >>> Hi Roger, >>> >>> Nice move/rename/merge. >>> >>> I compared old UNIXProcess.java with new ProcessImpl.java using KDiff3 and they match perfectly as well as including the old ProcessImpl.java code. >> Nice. I did the same just to see it for myself. The other parts of the webrev look good too. >> >> -Chris. >> >>> Regards, Peter >>> >>> On 01/23/2015 05:11 PM, Roger Riggs wrote: >>>> Please review this merge of UNIXProcess.java with ProcessImpl.java. >>>> >>>> For Unix variants of Process, the implementation is split between >>>> src/java.base/unix/classes/java/lang/ProcessImpl.java and UNIXProcess.java. >>>> ProcessImpl.java contains only static methods and delegates to UNIXProcess.java. >>>> Merging UNIXProcess.java into ProcessImpl.java removes a redundant method >>>> and aligns the source files with the Windows implementation that contains only ProcessImpl.java. >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~rriggs/webrev-unixprocess-rename-8071481/ >>>> >>>> Issue: >>>> 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java >>>> >>>> [1] https://bugs.openjdk.java.net/browse/JDK-8071481 > From lance.andersen at oracle.com Fri Jan 23 19:04:50 2015 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 23 Jan 2015 14:04:50 -0500 Subject: RFR (JAXP): 8054196: XPath: support any type In-Reply-To: <54C181FD.7000203@oracle.com> References: <54934601.4010104@oracle.com> <54B96BDF.7090204@oracle.com> <54B9C9EA.7050407@oracle.com> <5A817A94-0176-4613-91F2-9B298C1DA6AA@oracle.com> <54BEA3C6.6090501@oracle.com> <54C04831.3060105@oracle.com> <7B9200B5-D8C2-4070-957D-A97B32525BAF@oracle.com> <54C0880A.6090808@oracle.com> <2A097154-9F50-463F-8506-769C9DFE7B73@oracle.com> <54C181FD.7000203@oracle.com> Message-ID: Looks good Joe, ship it :-) On Jan 22, 2015, at 6:04 PM, huizhe wang wrote: > Thanks Lance. > > On 1/22/2015 9:11 AM, Lance Andersen wrote: >> Hi Joe, >> >> I just looked at the changes below, >> >> I looked at the changes below? see minor comments >> On Jan 22, 2015, at 12:18 AM, huizhe wang wrote: >> >>> >>> On 1/21/2015 5:09 PM, Lance Andersen wrote: >>>> Hi Joe, >>>> >>>> I think this is OK (as we discussed offline), one minor comment/suggestion below >>>> >>>> Best >>>> Lance >>>> On Jan 21, 2015, at 7:45 PM, huizhe wang wrote: >>>> >>>>> Thanks Lance for pointing me to Joe's -Xlint:all email thread. I re-compiled with -Xlint:all and noticed 7 warnings in this webrev. I've fixed them with a new webrev: >>>>> New webrev: http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev01/ >>>>> >>>>> Below are the details. >>>>> Refer to the previous webrev: http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>> >>>>> 1. XPathExpressionImpl.java:143 and XPathImpl.java:213 >>>>> >>>>> These warnings are fixed by having the getXPathResult method in XPathImpUtil returning the type instead >>>>> >>>>> 2. XPathExpressionImpl.java:165: warning: [rawtypes] found raw type: XPathEvaluationResult >>>>> similarly, XPathImpl.java:222 and XPathImpl.java:235 >>>>> >>>>> Changed to return XPathEvaluationResult >>>>> >>>>> 3. XPath.java:359, XPath.java:454, XPathExpression.java:252 and XPathExpression.java:344 >>>>> These are warnings from the default methods, casting the result of existing (old) methods. Use type.cast instead. >>>>> >>>>> 4. XPathNodesImpl.java:97: warning: [cast] redundant cast to Node >>>>> Removed the cast >>>>> >>>>> 5. XPathResultImpl.java:160: warning: [fallthrough] possible fall-through into case >>>>> Added break; >>>> >>>> I would add a comment or probably use @SuppressWarnings("fallthrough") instead >>> >>> Thanks! As I look at adding a comment, I realized I needed to be explicit on the numeric types as the spec stated. I've replaced the abstract Number type in XPathImplUtil::isSupportedClassType with Double/Integer/Long, and also in XPathResultImpl::getValue. >> >> Can we document why XPathImplUti.getXPathResult() should return null, same for XPathResultImpl.getValue() > > Done. > > While I was there, I also removed the 'type' parameter from the constructor of XPathNodesImpl and then XPathResultImpl::classToType method as it was only used to for constructing the XPathNodes object. >>> >>> I also added a test case for Long in XPathAnyTypeTest::test05, and a new test test06 to verify that the numeric types other than the supported can not be used as the type. >> This looks good > > http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev02/ > > Thanks, > Joe > >>> >>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev01/ >>> >>> Thanks, >>> Joe >>> >>>>> >>>>> Thanks, >>>>> Joe >>>>> >>>>> On 1/20/2015 10:51 AM, huizhe wang wrote: >>>>>> >>>>>> On 1/20/2015 7:02 AM, Lance Andersen wrote: >>>>>>> Hi Joe, >>>>>>> >>>>>>> I think the changes look fine. >>>>>> >>>>>> Thanks. >>>>>>> >>>>>>> I am wondering if we have any suggested standard for the use of @Override as I see it is inconsistent in its usage with methods implementing an interface. Is this something we should add to existing code? I don't see Netbeans asking for it to be done except in the case of an overridden method? >>>>>> >>>>>> NetBeans does make a suggestion such as: "Add @Override Annotation". I think you're right about avoiding comment duplication. In case of existing "evaluate" methods, I previously updated* the javadocs for both the interface and impl. It makes sense to take advantage of the "automatically inherit" feature of the javadoc tool to avoid the duplication. >>>>>> >>>>>> *Note that the update was only on format and styles, and some re-wording, no changes on definition or semantics. >>>>>> >>>>>>> >>>>>>> Thank you for the extra clean up Joe >>>>>> >>>>>> Thank you, now it looks better and cleaner. >>>>>> >>>>>> Best, >>>>>> Joe >>>>>> >>>>>>> >>>>>>> Best >>>>>>> Lance >>>>>>> On Jan 16, 2015, at 9:33 PM, huizhe wang > wrote: >>>>>>> >>>>>>>> >>>>>>>> On 1/16/2015 1:29 PM, Lance Andersen wrote: >>>>>>>>> Hi Joe, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Overall it is OK, a few minor comments >>>>>>>>> >>>>>>>>> - Is there a reason that XPathExpressionImpl is no longer public and XPathImpl is public? >>>>>>>> >>>>>>>> Ok, I'll keep both public, may be useful in the future. >>>>>>>> >>>>>>>>> - I think you can leverage {@inheritdoc} in your impl classes to avoid comment duplication possibly? >>>>>>>> >>>>>>>> Looks like javadoc "Automatically inherit comment ". So @Override is good enough. I've removed the javadocs for methods that override, including the existing methods. It's good to avoid the duplication, and potential copy n paste errors. >>>>>>>> >>>>>>>>> - please remember to make the copyright year 2015 >>>>>>>> >>>>>>>> Done. >>>>>>>> >>>>>>>>> - XPathTestBase, can the static block be moved to a @BeforeClass >>>>>>>> >>>>>>>> Moved to within the Dataprovider. >>>>>>>> >>>>>>>>> - XPathNodes, should that have an @since 1.9? >>>>>>>> >>>>>>>> Fixed. >>>>>>>> >>>>>>>>> - Given you are not including @param, etc for your test comments, you might want to consider /* */ vs /** */ style comments. >>>>>>>> >>>>>>>> Looks like s/\/**/\/* worked the trick. >>>>>>>> >>>>>>>>> That is consider if you really want doc comments (really a style choice but some IDEs will issue a warning for missing tags >>>>>>>> >>>>>>>> Yeh, it's good to turn off the warning "light" :-) >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Joe >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> HTH >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Lance >>>>>>>>> On Jan 16, 2015, at 2:51 PM, huizhe wang > wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> Could you review the change? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Joe >>>>>>>>>> >>>>>>>>>> On 12/18/2014 1:24 PM, huizhe wang wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> This is to add support for any type and improvement with new features reflected in the new evaluateExpression methods, XPathEvaluationResult and XPathNodes. >>>>>>>>>>> >>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8054196 >>>>>>>>>>> http://cr.openjdk.java.net/~joehw/jdk9/8054196/webrev/ >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Joe >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>>>> Oracle Java Engineering >>>>>>>>> 1 Network Drive >>>>>>>>> Burlington, MA 01803 >>>>>>>>> Lance.Andersen at oracle.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>>>>> Oracle Java Engineering >>>>>>> 1 Network Drive >>>>>>> Burlington, MA 01803 >>>>>>> Lance.Andersen at oracle.com >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>>> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >>>> Oracle Java Engineering >>>> 1 Network Drive >>>> Burlington, MA 01803 >>>> Lance.Andersen at oracle.com >>>> >>>> >>>> >>> >> >> >> >> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> >> >> > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From brian.burkhalter at oracle.com Fri Jan 23 19:17:01 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 23 Jan 2015 11:17:01 -0800 Subject: [9] RFR of 6880737: (fs) FileLock constructors don't throw NPE if the channel argument is null In-Reply-To: <54BA105C.8030809@oracle.com> References: <3B73EA6D-467E-4035-B3DD-949129D07E99@oracle.com> <54BA105C.8030809@oracle.com> Message-ID: <4976B3C8-81F4-4CDE-A9CF-62DE7F38B8CF@oracle.com> On Jan 16, 2015, at 11:33 PM, Alan Bateman wrote: > The change to the javadoc isn't strictly required as there is a general statement in the package description that NPE be thrown when methods or constructors are called with null. Thanks, I missed that. > So the implementation should throw NPE rather than IAE and as it's an incompatible change then it should be tracked. It's important to add a test case too. Here?s an updated patch including a (perhaps overkill) test: http://cr.openjdk.java.net/~bpb/6880737/webrev.01/ Thanks, Brian From paul.sandoz at oracle.com Fri Jan 23 19:19:06 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 23 Jan 2015 19:19:06 +0000 Subject: RFR 8071477: Better Spliterator implementations for String.chars() and String.codePoints() In-Reply-To: <54C29152.7090608@oracle.com> References: <54C29152.7090608@oracle.com> Message-ID: On Jan 23, 2015, at 6:22 PM, Xueming Shen wrote: > On 01/23/2015 09:00 AM, Paul Sandoz wrote: >> Hi, >> >> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8071477-String-spliterators/webrev/ >> >> This patch implements better spliterators for String/Buffer/Builder.chars/codePoints than those provided by the default methods on CharSequence. >> >> The test java/lang/CharSequence/DefaultTest.java is removed as i now pass the spliterators through the grinder of the spliterator-and-traversing tests. >> >> Thanks, >> Paul. > > I'm a little confused at following logic. > > 2997 // Mid-point is a high-surrogate > 2998 // Or mid-point and the previous are low-surrogates > 2999 if (Character.isHighSurrogate(array[mid]) || > 3000 Character.isLowSurrogate(array[midOneLess = (mid - 1)])) > 3001 return new CodePointsSpliterator(array, lo, index = mid, cs); > > > Shouldn't it be something like > > if (!Character.isLowSurrogate(array[mid]) || > !Character.isHighSurrogate(array[midOneLess = (mid -1)])) { > return new CodePointsSpliterator(array, lo, index = mid, cs); > } > > For example, in case both "mid" and "midOneLess" are normal non-surrogate > character, I would assume the trySplit() should return [lo, index=mid) as wekk? > > or something like > > ... > if (Character.isLowSurrogate(array[mid]) || && > Character.isHighSurrogate(array[midOneLess = (mid -1)])) { > if (lo >= midOneLess) > return null; > return new CodePointsSpliterator(array, lo, index = midOneless, cs); > } > return new CodePointsSpliterator(array, lo, index = mid, cs); > ... > > means, we only return [lo, midOneless), if mid is in the middle of a surrogate > pair (midOneLess is hiSurr, mid is hoSurr)? > Doh! thanks i dunno what i was thinking (isLowSurrogate was the negation of isHighSurrogate perhaps...) > btw, is it worth having a "nextCodePoint()" to be shared by forEachRemaining > and tryAdvance()? > Nice suggestion. I created a static method "advance". I also noticed that test/TEST.groups needs updating to remove a reference to the removed char sequence test. Webrev updated in place. Thanks, Paul. From Roger.Riggs at Oracle.com Fri Jan 23 20:14:56 2015 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Fri, 23 Jan 2015 15:14:56 -0500 Subject: RFR 9: 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java In-Reply-To: <4F2E81BA-6E18-480D-A687-B8D529CD7D85@oracle.com> References: <54C272CA.5030102@Oracle.com> <54C28257.2090902@gmail.com> <823A6941-ABF8-43F8-9CBC-BBC6A19472D2@oracle.com> <54C29987.3060003@Oracle.com> <4F2E81BA-6E18-480D-A687-B8D529CD7D85@oracle.com> Message-ID: <54C2ABC0.2040000@Oracle.com> Hi Chris, I considered updating BigFork but would have had to update the stacktrace lines numbers, etc. And I think it will be recognized that it is a dated reference. Roger On 1/23/2015 2:03 PM, Chris Hegarty wrote: > On 23 Jan 2015, at 18:57, Roger Riggs wrote: > >> Hi Peter, Chris, >> >> Thanks for the reviews. >> >> I found one more reference to UNIXProcess in a comment in childproc.h > Ah, good find. > > Another reference in comments in BigFork.java, but probably doesn?t make sense to change at this point. > > -Chris. > >> and will wait for a jprt run to finish. >> >> Webrev: >> >> http://cr.openjdk.java.net/~rriggs/webrev-unixprocess-rename-8071481/ >> >> Thanks, Roger >> >> On 1/23/2015 12:53 PM, Chris Hegarty wrote: >>> On 23 Jan 2015, at 17:18, Peter Levart wrote: >>> >>>> Hi Roger, >>>> >>>> Nice move/rename/merge. >>>> >>>> I compared old UNIXProcess.java with new ProcessImpl.java using KDiff3 and they match perfectly as well as including the old ProcessImpl.java code. >>> Nice. I did the same just to see it for myself. The other parts of the webrev look good too. >>> >>> -Chris. >>> >>>> Regards, Peter >>>> >>>> On 01/23/2015 05:11 PM, Roger Riggs wrote: >>>>> Please review this merge of UNIXProcess.java with ProcessImpl.java. >>>>> >>>>> For Unix variants of Process, the implementation is split between >>>>> src/java.base/unix/classes/java/lang/ProcessImpl.java and UNIXProcess.java. >>>>> ProcessImpl.java contains only static methods and delegates to UNIXProcess.java. >>>>> Merging UNIXProcess.java into ProcessImpl.java removes a redundant method >>>>> and aligns the source files with the Windows implementation that contains only ProcessImpl.java. >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~rriggs/webrev-unixprocess-rename-8071481/ >>>>> >>>>> Issue: >>>>> 8071481: (Process) Merge UNIXProcess.java into ProcessImpl.java >>>>> >>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8071481 From xueming.shen at oracle.com Fri Jan 23 20:26:49 2015 From: xueming.shen at oracle.com (Xueming Shen) Date: Fri, 23 Jan 2015 12:26:49 -0800 Subject: RFR 8071477: Better Spliterator implementations for String.chars() and String.codePoints() In-Reply-To: References: <54C29152.7090608@oracle.com> Message-ID: <54C2AE89.1090808@oracle.com> The webrev looks good. However I have a question regarding the spec of CharsSequence.chars/codePoints(). It says * *

If the sequence is mutated while the stream is being read, the * result is undefined. But it looks like ABS.chars/codePoints() "predefine/fix" the size of the resulting stream when the stream is being created/initialized. The possible corner case is that the "size/length" of ABS is changed between the stream is created and read? I would assume the CharSequence default implementation will go after the real size, even a "length()" is passed in as an estimated size during creation, but the optimized implementation will only return the length/size of char/cp when the stream is created. A behavior change? -Sherman On 01/23/2015 11:19 AM, Paul Sandoz wrote: > On Jan 23, 2015, at 6:22 PM, Xueming Shen wrote: > >> On 01/23/2015 09:00 AM, Paul Sandoz wrote: >>> Hi, >>> >>> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8071477-String-spliterators/webrev/ >>> >>> This patch implements better spliterators for String/Buffer/Builder.chars/codePoints than those provided by the default methods on CharSequence. >>> >>> The test java/lang/CharSequence/DefaultTest.java is removed as i now pass the spliterators through the grinder of the spliterator-and-traversing tests. >>> >>> Thanks, >>> Paul. >> I'm a little confused at following logic. >> >> 2997 // Mid-point is a high-surrogate >> 2998 // Or mid-point and the previous are low-surrogates >> 2999 if (Character.isHighSurrogate(array[mid]) || >> 3000 Character.isLowSurrogate(array[midOneLess = (mid - 1)])) >> 3001 return new CodePointsSpliterator(array, lo, index = mid, cs); >> >> >> Shouldn't it be something like >> >> if (!Character.isLowSurrogate(array[mid]) || >> !Character.isHighSurrogate(array[midOneLess = (mid -1)])) { >> return new CodePointsSpliterator(array, lo, index = mid, cs); >> } >> >> For example, in case both "mid" and "midOneLess" are normal non-surrogate >> character, I would assume the trySplit() should return [lo, index=mid) as wekk? >> >> or something like >> >> ... >> if (Character.isLowSurrogate(array[mid]) || > && > >> Character.isHighSurrogate(array[midOneLess = (mid -1)])) { >> if (lo>= midOneLess) >> return null; >> return new CodePointsSpliterator(array, lo, index = midOneless, cs); >> } >> return new CodePointsSpliterator(array, lo, index = mid, cs); >> ... >> >> means, we only return [lo, midOneless), if mid is in the middle of a surrogate >> pair (midOneLess is hiSurr, mid is hoSurr)? >> > Doh! thanks i dunno what i was thinking (isLowSurrogate was the negation of isHighSurrogate perhaps...) > > >> btw, is it worth having a "nextCodePoint()" to be shared by forEachRemaining >> and tryAdvance()? >> > Nice suggestion. I created a static method "advance". > > I also noticed that test/TEST.groups needs updating to remove a reference to the removed char sequence test. > > Webrev updated in place. > > Thanks, > Paul. > From paul.sandoz at oracle.com Fri Jan 23 20:51:23 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 23 Jan 2015 20:51:23 +0000 Subject: RFR 8071477: Better Spliterator implementations for String.chars() and String.codePoints() In-Reply-To: <54C2AE89.1090808@oracle.com> References: <54C29152.7090608@oracle.com> <54C2AE89.1090808@oracle.com> Message-ID: <3388BFFF-FEDB-4E68-9BCD-1CF740B57D74@oracle.com> On Jan 23, 2015, at 8:26 PM, Xueming Shen wrote: > The webrev looks good. > Thanks. > However I have a question regarding the spec of CharsSequence.chars/codePoints(). It says > * > *

If the sequence is mutated while the stream is being read, the > * result is undefined. > > But it looks like ABS.chars/codePoints() "predefine/fix" the size of the resulting stream when > the stream is being created/initialized. No, because a supplier of a spliterator is used (as is the case for the CharSequence implementations): return StreamSupport.intStream( () -> new String.IntCharArraySpliterator(value, 0, count, 0), Spliterator.ORDERED | Spliterator.SIZED | Spliterator.SUBSIZED, false); the "value" and the "count" will be obtained when the terminal operation executed not when the Stream is created and returned. > The possible corner case is that the "size/length" of > ABS is changed between the stream is created and read? > > I would assume the CharSequence default implementation will go after the real size, even > a "length()" is passed in as an estimated size during creation, but the optimized implementation > will only return the length/size of char/cp when the stream is created. A behavior change? > For CharSequence.chars() the length() is taken as the exact size: return StreamSupport.intStream(() -> Spliterators.spliterator( new CharIterator(), length(), Spliterator.ORDERED), Spliterator.SUBSIZED | Spliterator.SIZED | Spliterator.ORDERED, false); and (like above) is called when the terminal operation executes. So it's equivalent behaviour regarding when the snapshot of the sequence length is obtained. Paul. From brian.burkhalter at oracle.com Fri Jan 23 20:56:51 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 23 Jan 2015 12:56:51 -0800 Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec Message-ID: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com> Please review at your convenience: Issue: https://bugs.openjdk.java.net/browse/JDK-8069269 Patch: http://cr.openjdk.java.net/~bpb/8069269/webrev.00/ The effective change is line 391 -> 393; the remainder is reformatting. Consider these hypothetical nanoTime values: long t0 = Long.MAX_VALUE - 41; long t1 = t0 + 42; Ignoring numerical overflow, the second value is mathematically equal to 2^63 which is greater than the first value. The results of the old and new comparison methodologies applied to the two long values are: old: (t1 - t0) < 0: false t1 < t0: true new: (t0 - t1) < 0: true t0 < t1: false This exhibits that the old documentation is incorrect regarding the methodology to be used when comparing two nanoTime values in the case of numerical overflow, and that the new documentation is accurate in this case. Thanks, Brian From martinrb at google.com Fri Jan 23 21:10:51 2015 From: martinrb at google.com (Martin Buchholz) Date: Fri, 23 Jan 2015 13:10:51 -0800 Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec In-Reply-To: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com> References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com> Message-ID: We have struggled for years with formatting for code samples. If you want to change it, get authoritative statement on how to do it with latest javadoc, publish it somewhere, and change it everywhere. Putting the } on a line by itself did not produce the most readable output IIRC. --- I don't see why switching t0 and t1 makes things more readable. A real improvement might be: since one almost never compares two nanoTime values against each other, compare against "timeout" instead of against zero. E.g. long startTime = System.nanoTime(); ... long elapsedNanos = System.nanoTime() - startTime; if (elapsedNanos < timeout) ... using currently popular conventions in j.u.c. On Fri, Jan 23, 2015 at 12:56 PM, Brian Burkhalter < brian.burkhalter at oracle.com> wrote: > Please review at your convenience: > > Issue: https://bugs.openjdk.java.net/browse/JDK-8069269 > Patch: http://cr.openjdk.java.net/~bpb/8069269/webrev.00/ > > The effective change is line 391 -> 393; the remainder is reformatting. > > Consider these hypothetical nanoTime values: > > long t0 = Long.MAX_VALUE - 41; > long t1 = t0 + 42; > > Ignoring numerical overflow, the second value is mathematically equal to > 2^63 which is greater than the first value. > > The results of the old and new comparison methodologies applied to the two > long values are: > > old: > > (t1 - t0) < 0: false > t1 < t0: true > > new: > > (t0 - t1) < 0: true > t0 < t1: false > > This exhibits that the old documentation is incorrect regarding the > methodology to be used when comparing two nanoTime values in the case of > numerical overflow, and that the new documentation is accurate in this case. > > Thanks, > > Brian From huizhe.wang at oracle.com Fri Jan 23 21:18:11 2015 From: huizhe.wang at oracle.com (huizhe wang) Date: Fri, 23 Jan 2015 13:18:11 -0800 Subject: RFR: 8046817: JDK 8 schemagen tool does not generate xsd files for enum types In-Reply-To: <54BD355E.6070707@oracle.com> References: <5434FF2D.104@oracle.com> <543CD0E0.7060109@oracle.com> <543E55A1.8060903@oracle.com> <54BD355E.6070707@oracle.com> Message-ID: <54C2BA93.2090900@oracle.com> Looks good, Aleksej. Thanks, Joe On 1/19/2015 8:48 AM, Aleksej Efimov wrote: > Hi, > Returning back to this issue: The fix will be integrated soon to > standalone JAXB project. Local build of standalone JAXB succeeded. No > issues with jaxb tests were observed with applied fix. > Can I ask reviewers to look at this issue now? > > Thank you and Best Regards, > Aleksej > > On 10/15/2014 03:08 PM, Miroslav Kos wrote: >> Hi Aleksej, >> >> the change looks ok to me, but it would be good to integrate it into >> upstream (standalone JAX-B RI project) first to be on safe side - >> there are much more tests then in OpenJDK. >> >> Copying Iaroslav who is JAX-B RI lead and can help you with >> integration and running tests. >> >> Thanks >> Miran >> >> >> On 14/10/14 09:29, Aleksej Efimov wrote: >>> Hi XML experts, >>> Can I humbly ask to review this simple fix. >>> >>> Thank you, >>> Aleksej >>> >>> On 08.10.2014 13:09, Aleksej Efimov wrote: >>>> Hello, >>>> >>>> Please, review the fix [1] for the 8046817 [2]. >>>> Problem: schemagen tool doesn't generate schema file for the enum >>>> types [3]. >>>> SchemaGenerator class gets a list of annotated elements and filters >>>> out only the class variables, but annotation parser >>>> (AnnotationParser.java) filters out the Class and Enum. The >>>> proposed fix solves the problem: SchemaGenerator filters like >>>> AnnotationParser. >>>> Testing: JPRT build and test passes on platforms (with new >>>> regression test). JTREG tests: javax/xml - no failures. >>>> >>>> Thanks, >>>> Aleksej >>>> >>>> [1] Webrev: http://cr.openjdk.java.net/~aefimov/8046817/9/webrev.00/ >>>> [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8046817 >>>> [3] Enum type: >>>> http://cr.openjdk.java.net/~aefimov/8046817/9/webrev.00/jdk/test/javax/xml/ws/8046817/TestEnumType.java.html >>> >> > From brian.burkhalter at oracle.com Fri Jan 23 21:20:58 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 23 Jan 2015 13:20:58 -0800 Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec In-Reply-To: References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com> Message-ID: On Jan 23, 2015, at 1:10 PM, Martin Buchholz wrote: > We have struggled for years with formatting for code samples. If you want to change it, get authoritative statement on how to do it with latest javadoc, publish it somewhere, and change it everywhere. Putting the } on a line by itself did not produce the most readable output IIRC. Any official guidance here would be welcome. I actually investigated the formatting style in other classes, e.g., AsynchronousServerSocketChannel, ServiceLoader, etc., and there is in fact a lack of consistency aside from the use of 

. I did however build the docs before and after this change and the revised one looks better, at least on my system.

> ---
> 
> I don't see why switching t0 and t1 makes things more readable.

It?s about accuracy. The previous version is incorrect in the overflow case (if we even care).

> A real improvement might be: since one almost never compares two nanoTime values against each other, compare against "timeout" instead of against zero.  E.g.
> 
> long startTime = System.nanoTime();
> ...
> long elapsedNanos = System.nanoTime() - startTime;
> if (elapsedNanos < timeout) ...
> 
> using currently popular conventions in j.u.c.

That?s a thought ?

Brian

From xueming.shen at oracle.com  Fri Jan 23 21:51:26 2015
From: xueming.shen at oracle.com (Xueming Shen)
Date: Fri, 23 Jan 2015 13:51:26 -0800
Subject: RFR 8071477: Better Spliterator implementations for String.chars()
	and String.codePoints()
In-Reply-To: <3388BFFF-FEDB-4E68-9BCD-1CF740B57D74@oracle.com>
References: 	<54C29152.7090608@oracle.com>		<54C2AE89.1090808@oracle.com>
	<3388BFFF-FEDB-4E68-9BCD-1CF740B57D74@oracle.com>
Message-ID: <54C2C25E.4080206@oracle.com>

On 01/23/2015 12:51 PM, Paul Sandoz wrote:
>
> No, because a supplier of a spliterator is used (as is the case for the CharSequence implementations):
>
> return StreamSupport.intStream(
>          () ->  new String.IntCharArraySpliterator(value, 0, count, 0),
>          Spliterator.ORDERED | Spliterator.SIZED | Spliterator.SUBSIZED,
>          false);
>
> the "value" and the "count" will be obtained when the terminal operation executed not when the Stream is created and returned.

thanks! I missed it's a supplier of spliterator is being passed in:-)

-Sherman


From martinrb at google.com  Fri Jan 23 21:58:54 2015
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 23 Jan 2015 13:58:54 -0800
Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec
In-Reply-To: 
References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com>
	
	
Message-ID: 

[+javadoc-dev]

On Fri, Jan 23, 2015 at 1:20 PM, Brian Burkhalter <
brian.burkhalter at oracle.com> wrote:

> On Jan 23, 2015, at 1:10 PM, Martin Buchholz  wrote:
>
> We have struggled for years with formatting for code samples.  If you want
> to change it, get authoritative statement on how to do it with latest
> javadoc, publish it somewhere, and change it everywhere.  Putting the
> } on a line by itself did not produce the most readable output IIRC.
>
>
> Any official guidance here would be welcome. I actually investigated the
> formatting style in other classes, e.g., AsynchronousServerSocketChannel,
> ServiceLoader, etc., and there is in fact a lack of consistency aside from
> the use of 
. I did however build the docs before and after this
> change and the revised one looks better, at least on my system.
>

Here's my view of the history.  javadoc does not provide a baked in
{@codesnippet ...} (PLEEEASE javadoc maintainers, add it!) so people have
come up with different local styles combining {@code and 
.  But
javadoc's output changes every release, so people have changed their
blessed style based on both personal preference and the target version of
javadoc used to generate their docs.  In java.util.concurrent, we've
currently standardized on

 * 
 {@code
...
 * }}



From martinrb at google.com  Fri Jan 23 22:16:04 2015
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 23 Jan 2015 14:16:04 -0800
Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec
In-Reply-To: 
References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com>
	
	
Message-ID: 

On Fri, Jan 23, 2015 at 1:20 PM, Brian Burkhalter <
brian.burkhalter at oracle.com> wrote:

> On Jan 23, 2015, at 1:10 PM, Martin Buchholz  wrote:
>
> ---
>
> I don't see why switching t0 and t1 makes things more readable.
>
>
> It?s about accuracy. The previous version is incorrect in the overflow
> case (if we even care).
>
>
I don't get it.  t0 and t1 are supposed to be symmetrical (perhaps obtained
in different threads).
Switching them around doesn't change anything.  And correctness in the face
of two's complement numerical overflow is the whole point of the example!


From brent.christian at oracle.com  Sat Jan 24 00:15:22 2015
From: brent.christian at oracle.com (Brent Christian)
Date: Fri, 23 Jan 2015 16:15:22 -0800
Subject: RFR 9: 8068578: test/java/io/Serializable/subclassGC/SubclassGC.java
	assumes app class loader is a URLClassLoader
Message-ID: <54C2E41A.9090609@oracle.com>

Hi,

Please review this test case update.

Bug:
https://bugs.openjdk.java.net/browse/JDK-8068578
Webrev:
http://cr.openjdk.java.net/~bchristi/8068578/webrev.0/

The test relies on creating a new URLClassLoader and using it to load a 
subclass of ObjectOutputStream.  The system classloader is cast to a 
URLClassLoader in order to call getURLs(), but the test can be made to 
work without this URL[].  We can instead create a URL based on the value 
of the "test.classes" system property.

I have confirmed that the test still detects the original bug (6232010) 
in 5u6, and passes with later JDKs.

Thanks,
-Brent



From brian.burkhalter at oracle.com  Sat Jan 24 00:27:31 2015
From: brian.burkhalter at oracle.com (Brian Burkhalter)
Date: Fri, 23 Jan 2015 16:27:31 -0800
Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec
In-Reply-To: 
References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com>
	
	
	
Message-ID: <1D643418-534A-4448-B229-50BCFEB64F2F@oracle.com>


On Jan 23, 2015, at 2:16 PM, Martin Buchholz  wrote:

> I don't get it.  t0 and t1 are supposed to be symmetrical (perhaps obtained in different threads).
> Switching them around doesn't change anything.  And correctness in the face of two's complement numerical overflow is the whole point of the example!

My take on it is, for example, given the (non-overflowing) mathematical quantities

T0 = 2^63 - 42
T1 = 2^63

and the Java variables

long t0 = Long.MAX_VALUE - 41;
long t1 = t0 + 42;

then the Java logical expression (t0 - t1) < 0 is equivalent to the mathematical inequality T0 < T1 which is what would want when comparing two such values.

From martinrb at google.com  Sat Jan 24 02:16:27 2015
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 23 Jan 2015 18:16:27 -0800
Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec
In-Reply-To: <1D643418-534A-4448-B229-50BCFEB64F2F@oracle.com>
References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com>
	
	
	
	<1D643418-534A-4448-B229-50BCFEB64F2F@oracle.com>
Message-ID: 

I'm agreeing with what you say here, but t0 and t1 are symmetrical, so
still cannot understand why you want to switch them in the javadoc.

On Fri, Jan 23, 2015 at 4:27 PM, Brian Burkhalter <
brian.burkhalter at oracle.com> wrote:

>
> On Jan 23, 2015, at 2:16 PM, Martin Buchholz  wrote:
>
> I don't get it.  t0 and t1 are supposed to be symmetrical (perhaps
> obtained in different threads).
> Switching them around doesn't change anything.  And correctness in the
> face of two's complement numerical overflow is the whole point of the
> example!
>
>
> My take on it is, for example, given the (non-overflowing) mathematical
> quantities
>
> T0 = 2^63 - 42
> T1 = 2^63
>
> and the Java variables
>
> long t0 = Long.MAX_VALUE - 41;
> long t1 = t0 + 42;
>
> then the Java logical expression (t0 - t1) < 0 is equivalent to the
> mathematical inequality T0 < T1 which is what would want when comparing two
> such values.
>


From Alan.Bateman at oracle.com  Sat Jan 24 08:55:40 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Sat, 24 Jan 2015 08:55:40 +0000
Subject: RFR: JDK-8055190: Cleanup include and exclude of core-libs native
	libraries after source code restructure
In-Reply-To: <54C261C6.5030705@oracle.com>
References: <54C25C3E.4030501@oracle.com> <54C25D9C.2040408@oracle.com>
	<54C261C6.5030705@oracle.com>
Message-ID: <54C35E0C.8080406@oracle.com>

On 23/01/2015 14:59, Erik Joelsson wrote:
> :
>> Can refs.allowed be removed? I thought we removed the JDK 8 
>> dependency checker early in JDK 9 so this file shouldn't be needed.
>>
> I was wondering what it was and could not find any usages. With your 
> confirmation I have opted to remove it completely instead.
>
> Updated webrev: http://cr.openjdk.java.net/~erikj/8055190/webrev.02/
Thanks, this looks good to me.

-Alan


From Alan.Bateman at oracle.com  Sat Jan 24 08:57:16 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Sat, 24 Jan 2015 08:57:16 +0000
Subject: RFR 9: 8068578:
	test/java/io/Serializable/subclassGC/SubclassGC.java
	assumes app class loader is a URLClassLoader
In-Reply-To: <54C2E41A.9090609@oracle.com>
References: <54C2E41A.9090609@oracle.com>
Message-ID: <54C35E6C.9050905@oracle.com>

On 24/01/2015 00:15, Brent Christian wrote:
> Hi,
>
> Please review this test case update.
>
> Bug:
> https://bugs.openjdk.java.net/browse/JDK-8068578
> Webrev:
> http://cr.openjdk.java.net/~bchristi/8068578/webrev.0/
>
> The test relies on creating a new URLClassLoader and using it to load 
> a subclass of ObjectOutputStream.  The system classloader is cast to a 
> URLClassLoader in order to call getURLs(), but the test can be made to 
> work without this URL[].  We can instead create a URL based on the 
> value of the "test.classes" system property.
>
> I have confirmed that the test still detects the original bug 
> (6232010) in 5u6, and passes with later JDKs.
It's good to have this assumption removed from the test. The change 
looks okay to me and thanks for confirming the original issue.

-Alan


From martinrb at google.com  Sat Jan 24 17:55:23 2015
From: martinrb at google.com (Martin Buchholz)
Date: Sat, 24 Jan 2015 09:55:23 -0800
Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec
In-Reply-To: 
References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com>
	
	
	
	<1D643418-534A-4448-B229-50BCFEB64F2F@oracle.com>
	
Message-ID: 

How about:

--- a/src/java.base/share/classes/java/lang/System.java
+++ b/src/java.base/share/classes/java/lang/System.java
@@ -376,19 +376,16 @@
      * the difference between two such values, obtained within the same
      * instance of a Java virtual machine, is computed.
      *
-     * 
 For example, to measure how long some code takes to execute:
+     * 
For example, to measure how long some code takes to execute:
      *  
 {@code
      * long startTime = System.nanoTime();
      * // ... the code being measured ...
-     * long estimatedTime = System.nanoTime() - startTime;}

+     * long elapsedNanos = System.nanoTime() - startTime;}

      *
-     * 
To compare two nanoTime values
-     *  
 {@code
-     * long t0 = System.nanoTime();
-     * ...
-     * long t1 = System.nanoTime();}

-     *
-     * one should use {@code t1 - t0 < 0}, not {@code t1 < t0},
+     * 
To compare elapsed time against a timeout, use 
 {@code
+     * if (System.nanoTime() - startTime >= timeoutNanos) ...}

+     * instead of 
 {@code
+     * if (System.nanoTime() >= startTime + timeoutNanos) ...}

      * because of the possibility of numerical overflow.
      *
      * @return the current value of the running Java Virtual Machine's


On Fri, Jan 23, 2015 at 6:16 PM, Martin Buchholz 
wrote:

> I'm agreeing with what you say here, but t0 and t1 are symmetrical, so
> still cannot understand why you want to switch them in the javadoc.
>
> On Fri, Jan 23, 2015 at 4:27 PM, Brian Burkhalter <
> brian.burkhalter at oracle.com> wrote:
>
>>
>> On Jan 23, 2015, at 2:16 PM, Martin Buchholz  wrote:
>>
>> I don't get it.  t0 and t1 are supposed to be symmetrical (perhaps
>> obtained in different threads).
>> Switching them around doesn't change anything.  And correctness in the
>> face of two's complement numerical overflow is the whole point of the
>> example!
>>
>>
>> My take on it is, for example, given the (non-overflowing) mathematical
>> quantities
>>
>> T0 = 2^63 - 42
>> T1 = 2^63
>>
>> and the Java variables
>>
>> long t0 = Long.MAX_VALUE - 41;
>> long t1 = t0 + 42;
>>
>> then the Java logical expression (t0 - t1) < 0 is equivalent to the
>> mathematical inequality T0 < T1 which is what would want when comparing two
>> such values.
>>
>
>


From martinrb at google.com  Sat Jan 24 21:13:12 2015
From: martinrb at google.com (Martin Buchholz)
Date: Sat, 24 Jan 2015 13:13:12 -0800
Subject: String.indexOf optimization
In-Reply-To: 
References: 
	
	<534D68B8.3040002@oracle.com>
	
	<5350E316.2050406@oracle.com>
	
	
	<54ABAAC2.2090502@redhat.com>
	
	
	
	
	
Message-ID: 

I took another look.  It seems to me that the java code implementing
String.indexOf still assumes the old String layout and so is full of dead
code, which should be excised just to be clean.  (I continue to think the
change to String layout came too late in the life of Java and is incomplete
in ways such as this one).  Is there an easy way to disable a hotspot
intrinsic for testing?

Here's a start at such a change:

diff --git
a/src/java.base/share/classes/java/lang/AbstractStringBuilder.java
b/src/java.base/share/classes/java/lang/AbstractStringBuilder.java
--- a/src/java.base/share/classes/java/lang/AbstractStringBuilder.java
+++ b/src/java.base/share/classes/java/lang/AbstractStringBuilder.java
@@ -1319,7 +1319,7 @@
      *          or {@code -1} if there is no such occurrence.
      */
     public int indexOf(String str, int fromIndex) {
-        return String.indexOf(value, 0, count, str, fromIndex);
+        return String.indexOf(value, str, fromIndex);
     }

     /**
diff --git a/src/java.base/share/classes/java/lang/String.java
b/src/java.base/share/classes/java/lang/String.java
--- a/src/java.base/share/classes/java/lang/String.java
+++ b/src/java.base/share/classes/java/lang/String.java
@@ -1703,8 +1703,7 @@
      *          or {@code -1} if there is no such occurrence.
      */
     public int indexOf(String str, int fromIndex) {
-        return indexOf(value, 0, value.length,
-                str.value, 0, str.value.length, fromIndex);
+        return indexOf(value, str.value, fromIndex);
     }

     /**
@@ -1713,16 +1712,11 @@
      * is the string being searched for.
      *
      * @param   source       the characters being searched.
-     * @param   sourceOffset offset of the source string.
-     * @param   sourceCount  count of the source string.
      * @param   target       the characters being searched for.
      * @param   fromIndex    the index to begin searching from.
      */
-    static int indexOf(char[] source, int sourceOffset, int sourceCount,
-            String target, int fromIndex) {
-        return indexOf(source, sourceOffset, sourceCount,
-                       target.value, 0, target.value.length,
-                       fromIndex);
+    static int indexOf(char[] source, String target, int fromIndex) {
+        return indexOf(source, target.value, fromIndex);
     }

     /**
@@ -1731,47 +1725,38 @@
      * is the string being searched for.
      *
      * @param   source       the characters being searched.
-     * @param   sourceOffset offset of the source string.
-     * @param   sourceCount  count of the source string.
      * @param   target       the characters being searched for.
-     * @param   targetOffset offset of the target string.
-     * @param   targetCount  count of the target string.
      * @param   fromIndex    the index to begin searching from.
      */
-    static int indexOf(char[] source, int sourceOffset, int sourceCount,
-            char[] target, int targetOffset, int targetCount,
-            int fromIndex) {
-        if (fromIndex >= sourceCount) {
-            return (targetCount == 0 ? sourceCount : -1);
+    static int indexOf(char[] source, char[] target, int fromIndex) {
+        final int slen = source.length;
+        final int tlen = target.length;
+        if (fromIndex >= slen) {
+            return (tlen == 0 ? slen : -1);
         }
         if (fromIndex < 0) {
             fromIndex = 0;
         }
-        if (targetCount == 0) {
+        if (tlen == 0) {
             return fromIndex;
         }

-        char first = target[targetOffset];
-        int max = sourceOffset + (sourceCount - targetCount);
+        char first = target[0];

-        for (int i = sourceOffset + fromIndex; i <= max; i++) {
+        outer: for (int i = fromIndex, max = slen - tlen;; i++) {
             /* Look for first character. */
-            if (source[i] != first) {
-                while (++i <= max && source[i] != first);
+            inner: for (;; i++) {
+                if (i >= max)
+                    break outer;
+                if (source[i] == first)
+                    break inner;
             }

-            /* Found first character, now look at the rest of v2 */
-            if (i <= max) {
-                int j = i + 1;
-                int end = j + targetCount - 1;
-                for (int k = targetOffset + 1; j < end && source[j]
-                        == target[k]; j++, k++);
-
-                if (j == end) {
-                    /* Found whole string. */
-                    return i - sourceOffset;
-                }
+            for (int k = 1, j = i + 1, end = i + tlen; j < end; j++, k++) {
+                if (source[j] != target[k])
+                    continue outer;
             }
+            return i;
         }
         return -1;
     }


On Mon, Jan 19, 2015 at 11:13 AM, Zoltan Sziladi 
wrote:

> Martin, yes, we're talking about that method.
>
> Other than tightening that code some... if we find some algorithm that
> outperforms the naive implementation under certain conditions (let's say
> when the searched pattern is longer than 10 characters), would it be worth
> including it as a special case? (For example naive would run normally but
> if pattern is longer than 10 characters then the other algorithm would
> run). Or do you think that would just make the code too complicated without
> enough benefits?
>
> Regards,
> Zoltan
>
> On Mon, Jan 12, 2015 at 12:11 PM, Martin Buchholz 
> wrote:
>
>> If there's a clean improvement in the java code, it's worth putting in.
>> You can try benchmarking with -Xint.
>> Are we talking about this method?
>>
>>     static int indexOf(char[] source, int sourceOffset, int sourceCount,
>>             char[] target, int targetOffset, int targetCount,
>>             int fromIndex) {
>>
>> It does look like we can tighten the code up a little...
>>
>>
>> On Thu, Jan 8, 2015 at 3:05 PM, Zoltan Sziladi 
>> wrote:
>>
>>> Thanks for the info.
>>> So that basically means we have 2 implementations of indexOf currently,
>>> one
>>> is in HotSpot, the other is in the JDK itself, which rarely gets
>>> executed.
>>> Aside from this later fact, isn't it still worth improving the JDK
>>> implementation if it is possible? I know that the intrinsified method
>>> gets
>>> executed most of the time, but still, if we can improve the JDK
>>> implementation also, why not? I don't know much about other JVMs but
>>> maybe
>>> a few don't intrinsify it?
>>>
>>> Is there any existing test suite which is considered conclusive enough
>>> that
>>> if an implementation beats the naive algorithm in those testcases then it
>>> could be considered as a replacement in the JDK?
>>>
>>> Regards,
>>> Zoltan
>>>
>>>
>>> On Thu, Jan 8, 2015 at 12:42 PM, Vitaly Davidovich 
>>> wrote:
>>>
>>> > The java impl you saw would be called by (a) interpreter, (b) if you
>>> > explicitly disable intrinsification of this function, or (c) some
>>> other JVM
>>> > that doesn't intrinsify this method (or any method).
>>> >
>>> > People don't usually disable intrinsics; if they do, it's because they
>>> hit
>>> > some JIT bug and may disable it.
>>> >
>>> > On Thu, Jan 8, 2015 at 3:34 PM, Zoltan Sziladi 
>>> > wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> Thanks everyone for all the info.
>>> >> So, just to go step by step in understanding this.
>>> >> Andrew said HotSpot would ignore my implementation. So why is there an
>>> >> implementation of indexOf at all in the JDK, if that's not the code
>>> that's
>>> >> executed? Is it just a default fallback? When is the indexOf function
>>> not
>>> >> intrinsified? When do people usually disable intrinsification?
>>> >> Sorry if these are newbie questions, I'm new to this part of Java.
>>> >>
>>> >> Regards,
>>> >> Zoltan
>>> >>
>>> >> On Tue, Jan 6, 2015 at 1:28 AM, Andrew Haley  wrote:
>>> >>
>>> >> > Hi,
>>> >> >
>>> >> > On 05/01/15 18:59, Zoltan Sziladi wrote:
>>> >> >
>>> >> > > This discussion was a long time ago, I was just reading through
>>> it to
>>> >> > check
>>> >> > > again what was the last state of the discussion about the
>>> >> String.indexOf.
>>> >> > > There is one part which I still do not understand, hopefully
>>> someone
>>> >> > could
>>> >> > > shed some light on it. A few emails ago Martin mentioned
>>> >> > >
>>> >> > > "Hotspot seems to have some intrinsification of String.indexOf,
>>> which
>>> >> > > confuses me.
>>> >> > > Hotspot seems the right place to provide more optimizations for
>>> this,
>>> >> > since
>>> >> > > there has been a fair amount of work creating high-performance
>>> >> low-level
>>> >> > > implementations of this idea in C."
>>> >> > >
>>> >> > > Then Ivan asked what that actually meant, whether hotspot actually
>>> >> > replaced
>>> >> > > the jdk implementation with a low level optimized C
>>> implementation,
>>> >> but I
>>> >> > > never saw an answer to that.
>>> >> >
>>> >> > You can have a look at an implementation of
>>> >> MacroAssembler::string_indexof
>>> >> > in
>>> >> >
>>> >> >
>>> >> >
>>> >>
>>> http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b6b89b8f8531/src/cpu/x86/vm/macroAssembler_x86.cpp
>>> >> >
>>> >> > > Can someone please explain this? If we somehow found an algorithm
>>> that
>>> >> > beat
>>> >> > > the naive implementation in the average case, would it be
>>> possible to
>>> >> > just
>>> >> > > implement it in the JDK?
>>> >> >
>>> >> > No, because HotSpot would ignore it.  Any speed improvements have
>>> to be
>>> >> > done in the architecture-dependent files.
>>> >> >
>>> >> > Andrew.
>>> >> >
>>> >> >
>>> >>
>>> >
>>> >
>>>
>>
>>
>


From vitalyd at gmail.com  Sat Jan 24 21:19:08 2015
From: vitalyd at gmail.com (Vitaly Davidovich)
Date: Sat, 24 Jan 2015 16:19:08 -0500
Subject: String.indexOf optimization
In-Reply-To: 
References: 
	
	<534D68B8.3040002@oracle.com>
	
	<5350E316.2050406@oracle.com>
	
	
	<54ABAAC2.2090502@redhat.com>
	
	
	
	
	
	
Message-ID: 

Try -XX:+UnlockDiagnosticVMOptions -XX:DisableIntrinsic:_indexOf

sent from my phone
On Jan 24, 2015 4:13 PM, "Martin Buchholz"  wrote:

> I took another look.  It seems to me that the java code implementing
> String.indexOf still assumes the old String layout and so is full of dead
> code, which should be excised just to be clean.  (I continue to think the
> change to String layout came too late in the life of Java and is incomplete
> in ways such as this one).  Is there an easy way to disable a hotspot
> intrinsic for testing?
>
> Here's a start at such a change:
>
> diff --git
> a/src/java.base/share/classes/java/lang/AbstractStringBuilder.java
> b/src/java.base/share/classes/java/lang/AbstractStringBuilder.java
> --- a/src/java.base/share/classes/java/lang/AbstractStringBuilder.java
> +++ b/src/java.base/share/classes/java/lang/AbstractStringBuilder.java
> @@ -1319,7 +1319,7 @@
>       *          or {@code -1} if there is no such occurrence.
>       */
>      public int indexOf(String str, int fromIndex) {
> -        return String.indexOf(value, 0, count, str, fromIndex);
> +        return String.indexOf(value, str, fromIndex);
>      }
>
>      /**
> diff --git a/src/java.base/share/classes/java/lang/String.java
> b/src/java.base/share/classes/java/lang/String.java
> --- a/src/java.base/share/classes/java/lang/String.java
> +++ b/src/java.base/share/classes/java/lang/String.java
> @@ -1703,8 +1703,7 @@
>       *          or {@code -1} if there is no such occurrence.
>       */
>      public int indexOf(String str, int fromIndex) {
> -        return indexOf(value, 0, value.length,
> -                str.value, 0, str.value.length, fromIndex);
> +        return indexOf(value, str.value, fromIndex);
>      }
>
>      /**
> @@ -1713,16 +1712,11 @@
>       * is the string being searched for.
>       *
>       * @param   source       the characters being searched.
> -     * @param   sourceOffset offset of the source string.
> -     * @param   sourceCount  count of the source string.
>       * @param   target       the characters being searched for.
>       * @param   fromIndex    the index to begin searching from.
>       */
> -    static int indexOf(char[] source, int sourceOffset, int sourceCount,
> -            String target, int fromIndex) {
> -        return indexOf(source, sourceOffset, sourceCount,
> -                       target.value, 0, target.value.length,
> -                       fromIndex);
> +    static int indexOf(char[] source, String target, int fromIndex) {
> +        return indexOf(source, target.value, fromIndex);
>      }
>
>      /**
> @@ -1731,47 +1725,38 @@
>       * is the string being searched for.
>       *
>       * @param   source       the characters being searched.
> -     * @param   sourceOffset offset of the source string.
> -     * @param   sourceCount  count of the source string.
>       * @param   target       the characters being searched for.
> -     * @param   targetOffset offset of the target string.
> -     * @param   targetCount  count of the target string.
>       * @param   fromIndex    the index to begin searching from.
>       */
> -    static int indexOf(char[] source, int sourceOffset, int sourceCount,
> -            char[] target, int targetOffset, int targetCount,
> -            int fromIndex) {
> -        if (fromIndex >= sourceCount) {
> -            return (targetCount == 0 ? sourceCount : -1);
> +    static int indexOf(char[] source, char[] target, int fromIndex) {
> +        final int slen = source.length;
> +        final int tlen = target.length;
> +        if (fromIndex >= slen) {
> +            return (tlen == 0 ? slen : -1);
>          }
>          if (fromIndex < 0) {
>              fromIndex = 0;
>          }
> -        if (targetCount == 0) {
> +        if (tlen == 0) {
>              return fromIndex;
>          }
>
> -        char first = target[targetOffset];
> -        int max = sourceOffset + (sourceCount - targetCount);
> +        char first = target[0];
>
> -        for (int i = sourceOffset + fromIndex; i <= max; i++) {
> +        outer: for (int i = fromIndex, max = slen - tlen;; i++) {
>              /* Look for first character. */
> -            if (source[i] != first) {
> -                while (++i <= max && source[i] != first);
> +            inner: for (;; i++) {
> +                if (i >= max)
> +                    break outer;
> +                if (source[i] == first)
> +                    break inner;
>              }
>
> -            /* Found first character, now look at the rest of v2 */
> -            if (i <= max) {
> -                int j = i + 1;
> -                int end = j + targetCount - 1;
> -                for (int k = targetOffset + 1; j < end && source[j]
> -                        == target[k]; j++, k++);
> -
> -                if (j == end) {
> -                    /* Found whole string. */
> -                    return i - sourceOffset;
> -                }
> +            for (int k = 1, j = i + 1, end = i + tlen; j < end; j++, k++)
> {
> +                if (source[j] != target[k])
> +                    continue outer;
>              }
> +            return i;
>          }
>          return -1;
>      }
>
>
> On Mon, Jan 19, 2015 at 11:13 AM, Zoltan Sziladi 
> wrote:
>
>> Martin, yes, we're talking about that method.
>>
>> Other than tightening that code some... if we find some algorithm that
>> outperforms the naive implementation under certain conditions (let's say
>> when the searched pattern is longer than 10 characters), would it be worth
>> including it as a special case? (For example naive would run normally but
>> if pattern is longer than 10 characters then the other algorithm would
>> run). Or do you think that would just make the code too complicated without
>> enough benefits?
>>
>> Regards,
>> Zoltan
>>
>> On Mon, Jan 12, 2015 at 12:11 PM, Martin Buchholz 
>> wrote:
>>
>>> If there's a clean improvement in the java code, it's worth putting in.
>>> You can try benchmarking with -Xint.
>>> Are we talking about this method?
>>>
>>>     static int indexOf(char[] source, int sourceOffset, int sourceCount,
>>>             char[] target, int targetOffset, int targetCount,
>>>             int fromIndex) {
>>>
>>> It does look like we can tighten the code up a little...
>>>
>>>
>>> On Thu, Jan 8, 2015 at 3:05 PM, Zoltan Sziladi 
>>> wrote:
>>>
>>>> Thanks for the info.
>>>> So that basically means we have 2 implementations of indexOf currently,
>>>> one
>>>> is in HotSpot, the other is in the JDK itself, which rarely gets
>>>> executed.
>>>> Aside from this later fact, isn't it still worth improving the JDK
>>>> implementation if it is possible? I know that the intrinsified method
>>>> gets
>>>> executed most of the time, but still, if we can improve the JDK
>>>> implementation also, why not? I don't know much about other JVMs but
>>>> maybe
>>>> a few don't intrinsify it?
>>>>
>>>> Is there any existing test suite which is considered conclusive enough
>>>> that
>>>> if an implementation beats the naive algorithm in those testcases then
>>>> it
>>>> could be considered as a replacement in the JDK?
>>>>
>>>> Regards,
>>>> Zoltan
>>>>
>>>>
>>>> On Thu, Jan 8, 2015 at 12:42 PM, Vitaly Davidovich 
>>>> wrote:
>>>>
>>>> > The java impl you saw would be called by (a) interpreter, (b) if you
>>>> > explicitly disable intrinsification of this function, or (c) some
>>>> other JVM
>>>> > that doesn't intrinsify this method (or any method).
>>>> >
>>>> > People don't usually disable intrinsics; if they do, it's because
>>>> they hit
>>>> > some JIT bug and may disable it.
>>>> >
>>>> > On Thu, Jan 8, 2015 at 3:34 PM, Zoltan Sziladi 
>>>> > wrote:
>>>> >
>>>> >> Hi,
>>>> >>
>>>> >> Thanks everyone for all the info.
>>>> >> So, just to go step by step in understanding this.
>>>> >> Andrew said HotSpot would ignore my implementation. So why is there
>>>> an
>>>> >> implementation of indexOf at all in the JDK, if that's not the code
>>>> that's
>>>> >> executed? Is it just a default fallback? When is the indexOf
>>>> function not
>>>> >> intrinsified? When do people usually disable intrinsification?
>>>> >> Sorry if these are newbie questions, I'm new to this part of Java.
>>>> >>
>>>> >> Regards,
>>>> >> Zoltan
>>>> >>
>>>> >> On Tue, Jan 6, 2015 at 1:28 AM, Andrew Haley  wrote:
>>>> >>
>>>> >> > Hi,
>>>> >> >
>>>> >> > On 05/01/15 18:59, Zoltan Sziladi wrote:
>>>> >> >
>>>> >> > > This discussion was a long time ago, I was just reading through
>>>> it to
>>>> >> > check
>>>> >> > > again what was the last state of the discussion about the
>>>> >> String.indexOf.
>>>> >> > > There is one part which I still do not understand, hopefully
>>>> someone
>>>> >> > could
>>>> >> > > shed some light on it. A few emails ago Martin mentioned
>>>> >> > >
>>>> >> > > "Hotspot seems to have some intrinsification of String.indexOf,
>>>> which
>>>> >> > > confuses me.
>>>> >> > > Hotspot seems the right place to provide more optimizations for
>>>> this,
>>>> >> > since
>>>> >> > > there has been a fair amount of work creating high-performance
>>>> >> low-level
>>>> >> > > implementations of this idea in C."
>>>> >> > >
>>>> >> > > Then Ivan asked what that actually meant, whether hotspot
>>>> actually
>>>> >> > replaced
>>>> >> > > the jdk implementation with a low level optimized C
>>>> implementation,
>>>> >> but I
>>>> >> > > never saw an answer to that.
>>>> >> >
>>>> >> > You can have a look at an implementation of
>>>> >> MacroAssembler::string_indexof
>>>> >> > in
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >>
>>>> http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b6b89b8f8531/src/cpu/x86/vm/macroAssembler_x86.cpp
>>>> >> >
>>>> >> > > Can someone please explain this? If we somehow found an
>>>> algorithm that
>>>> >> > beat
>>>> >> > > the naive implementation in the average case, would it be
>>>> possible to
>>>> >> > just
>>>> >> > > implement it in the JDK?
>>>> >> >
>>>> >> > No, because HotSpot would ignore it.  Any speed improvements have
>>>> to be
>>>> >> > done in the architecture-dependent files.
>>>> >> >
>>>> >> > Andrew.
>>>> >> >
>>>> >> >
>>>> >>
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>


From martinrb at google.com  Sat Jan 24 21:29:56 2015
From: martinrb at google.com (Martin Buchholz)
Date: Sat, 24 Jan 2015 13:29:56 -0800
Subject: String.indexOf optimization
In-Reply-To: 
References: 
	
	<534D68B8.3040002@oracle.com>
	
	<5350E316.2050406@oracle.com>
	
	
	<54ABAAC2.2090502@redhat.com>
	
	
	
	
	
	
	
Message-ID: 

On Sat, Jan 24, 2015 at 1:19 PM, Vitaly Davidovich 
wrote:

> Try -XX:+UnlockDiagnosticVMOptions -XX:DisableIntrinsic:_indexOf
>

Thank you very much!  Hard to find because -XX:+PrintFlagsFinal is
insufficient - also needs  -XX:+UnlockDiagnosticVMOptions


From ariel at weisberg.ws  Sun Jan 25 18:27:47 2015
From: ariel at weisberg.ws (Ariel Weisberg)
Date: Sun, 25 Jan 2015 13:27:47 -0500
Subject: Expected vs. observed performance of java.util.zip.CRC32 in Java 7
	and 8
In-Reply-To: <1421871334.180989.217049125.4E87A292@webmail.messagingengine.com>
References: <1421861659.3934810.216893077.7CB17890@webmail.messagingengine.com>
	<54BFE4A0.5080208@redhat.com>
	<1421871334.180989.217049125.4E87A292@webmail.messagingengine.com>
Message-ID: <1422210467.2780669.218577685.1C941BC3@webmail.messagingengine.com>

Hi,

Well to wrap this up for me at least. I tested with a very simple
benchmark on Linux and I see the expected behavior.

JDK 8  CRC32 is fast, but slows down with -XX:-UseCLMUL
-XX:-UseCRC32Intrinsics.
JDK 7 both implementations perform similarly

The JDK 8 non-intrinsic may have gotten slower. I didn't repeat enough
to confirm.

I repeated the same test on OS X and -XX:-UseCLMUL
-XX:-UseCRC32Intrinsics have no effect on JDK 8. The performance remains
high. JDK 7  had the same high performance as JDK 8 and this is with me
invoking javac/java from the CLI with an absolute path. The benchmark
also prints out the java.version property which confirms that JDK 7 is
really going that fast.

Regards,
Ariel

On Wed, Jan 21, 2015, at 03:15 PM, Ariel Weisberg wrote:
> Hi,
> 
> I cleaned up the results and converted to bytes and megabytes/second by
> multiplying byteSize with ops/sec.
> 
> https://docs.google.com/spreadsheets/d/1SirRnW7JOqUvAZhx0b2HjJug8lQGHAUiDU8-6Zcb9Os/edit?usp=sharing
> 
> Thanks,
> Ariel
> 
> On Wed, Jan 21, 2015, at 12:40 PM, Andrew Haley wrote:
> > On 01/21/2015 05:34 PM, Ariel Weisberg wrote:
> > > If you could shed any light on what the performance should be it would
> > > help when choosing whether to use the JDK CRC32 implementation or
> > > another depending on the currently runtime.
> > 
> > Please normalize the output to bytes/s; it's very confusing.
> > 
> > Andrew.
> > 


From otaviopolianasantana at gmail.com  Sun Jan 25 22:30:14 2015
From: otaviopolianasantana at gmail.com (=?UTF-8?Q?Ot=C3=A1vio_Gon=C3=A7alves_de_Santana?=)
Date: Sun, 25 Jan 2015 20:30:14 -0200
Subject: inefficient indexof when String has one length
In-Reply-To: 
References: 
	<8AEA68DB-8351-4EE1-8D3D-0EB9D475FA70@oracle.com>
	
Message-ID: 

Can anyone help me as sponsor?
On May 27, 2014 2:17 PM, "Ot?vio Gon?alves de Santana" <
otaviopolianasantana at gmail.com> wrote:

> Can anyone help me as sponsor?
> On May 12, 2014 1:57 PM, "Paul Sandoz"  wrote:
>
>> On Apr 26, 2014, at 12:56 PM, Ot?vio Gon?alves de Santana <
>> otaviojava at java.net> wrote:
>> > When a String has length just one, could be replaced by equivalent
>> > character literals, gaining some performance enhancement.
>> >
>> > I found 107 changes.
>> >
>>
>> This looks good to me, just one small issue:
>>
>> ---
>> a/src/share/classes/javax/security/auth/PrivateCredentialPermission.java
>>  Wed Apr 23 11:35:40 2014 -0700
>> +++
>> b/src/share/classes/javax/security/auth/PrivateCredentialPermission.java
>>  Sat Apr 26 07:31:04 2014 -0300
>> @@ -495,7 +495,7 @@
>>
>>          // perform new initialization from the permission name
>>
>> -        if (getName().indexOf(" ") == -1 && getName().indexOf("\"") ==
>> -1) {
>> +        if (getName().indexOf(' ') == -1 && getName().indexOf('\"') ==
>> -1) {
>>
>> You no longer need to escape the double quote.
>>
>> Can someone else sponsor this? unfortunately i do not have the time at
>> the moment.
>>
>> Paul.
>>
>


From claes.redestad at oracle.com  Sun Jan 25 22:47:12 2015
From: claes.redestad at oracle.com (Claes Redestad)
Date: Sun, 25 Jan 2015 23:47:12 +0100
Subject: inefficient indexof when String has one length
In-Reply-To: 
References: 	<8AEA68DB-8351-4EE1-8D3D-0EB9D475FA70@oracle.com>	
	
Message-ID: <54C57270.4060105@oracle.com>


On 2015-01-25 23:30, Ot?vio Gon?alves de Santana wrote:
> Can anyone help me as sponsor?

Sure, why not.

I'll remove the escaped double-quote as per Paul's request and run it 
through a
quick sanity test. Which e-mail do you want to get credited?

/Claes


From otaviopolianasantana at gmail.com  Mon Jan 26 00:53:27 2015
From: otaviopolianasantana at gmail.com (=?UTF-8?Q?Ot=C3=A1vio_Gon=C3=A7alves_de_Santana?=)
Date: Sun, 25 Jan 2015 22:53:27 -0200
Subject: inefficient indexof when String has one length
In-Reply-To: <54C57270.4060105@oracle.com>
References: 
	<8AEA68DB-8351-4EE1-8D3D-0EB9D475FA70@oracle.com>
	
	
	<54C57270.4060105@oracle.com>
Message-ID: 

Thank you.
My email: otaviojava at java.net

On Sun, Jan 25, 2015 at 8:47 PM, Claes Redestad 
wrote:

>
> On 2015-01-25 23:30, Ot?vio Gon?alves de Santana wrote:
>
>> Can anyone help me as sponsor?
>>
>
> Sure, why not.
>
> I'll remove the escaped double-quote as per Paul's request and run it
> through a
> quick sanity test. Which e-mail do you want to get credited?
>
> /Claes
>



-- 
Ot?vio Gon?alves de Santana

blog:     http://otaviosantana.blogspot.com.br/
twitter: http://twitter.com/otaviojava
site:     *http://about.me/otaviojava *
55 (11) 98255-3513


From weijun.wang at oracle.com  Mon Jan 26 02:03:33 2015
From: weijun.wang at oracle.com (Weijun Wang)
Date: Mon, 26 Jan 2015 10:03:33 +0800
Subject: [9] request for review: 8049171: Additional tests for jarsigner's
	warnings
In-Reply-To: <54C20286.1060106@oracle.com>
References: <54BF48BF.9090505@oracle.com>
	
	<54C0E1A3.8070600@oracle.com>
	
	<54C20286.1060106@oracle.com>
Message-ID: <54C5A075.4060304@oracle.com>



On 1/23/2015 16:12, Artem Smotrakov wrote:
> If the MANIFEST and the signature files must be at the beginning, should
> it be considered as a bug in jarsigner? Should it reject such files?

I think so. Will file a bug.

>>
>> The "jar u" way is to copy each old entry into destination unless the
>> entry name is in the updated list where the new file will be read.
>> Finally the untouched files in the updated list are appended.
> Since tests were not originally for checking some unusual ways for
> updating jars, I think they need to be updated to use the "jar u" way
> for adding unsigned entry.

Good.

Thanks
Max

>
> Artem
>>
>
>
>


From frank.yuan at oracle.com  Mon Jan 26 03:27:19 2015
From: frank.yuan at oracle.com (Frank Yuan)
Date: Mon, 26 Jan 2015 11:27:19 +0800
Subject: Review request for JDK-8051547: Convert JAXP function tests:
	javax.xml.validation.* to jtreg (testng) tests
Message-ID: <007601d03917$ff381180$fda83480$@oracle.com>

Hi, Joe and All

We are working on moving internal jaxp functional tests to open jdk repo.
This is the validation suite. Would you please review these test?  Any
comment will be appreciated.

bug: https://bugs.openjdk.java.net/browse/JDK-8051547
webrev: http://cr.openjdk.java.net/~fyuan/8051547/webrev.00/


Thanks,

Frank



From frank.yuan at oracle.com  Mon Jan 26 06:42:46 2015
From: frank.yuan at oracle.com (Frank Yuan)
Date: Mon, 26 Jan 2015 14:42:46 +0800
Subject: Review request for JDK-8051709: Convert JAXP function tests:
	javax.xml.datatype to jtreg (testng) tests
Message-ID: <00b801d03933$4d0d3190$e72794b0$@oracle.com>

Hi, Joe and All

We are working on moving internal jaxp functional tests to open jdk repo.
This is the datatype suite. Would you please review these test?  Any comment
will be appreciated.

bug: https://bugs.openjdk.java.net/browse/JDK-8051709
webrev: http://cr.openjdk.java.net/~fyuan/8051709/webrev.00/


Thanks,

Frank



From artem.smotrakov at oracle.com  Mon Jan 26 07:55:59 2015
From: artem.smotrakov at oracle.com (Artem Smotrakov)
Date: Mon, 26 Jan 2015 10:55:59 +0300
Subject: [9] request for review: 8049171: Additional tests for jarsigner's
	warnings
In-Reply-To: <54C5A075.4060304@oracle.com>
References: <54BF48BF.9090505@oracle.com>
	
	<54C0E1A3.8070600@oracle.com>
	
	<54C20286.1060106@oracle.com> <54C5A075.4060304@oracle.com>
Message-ID: <54C5F30F.7070207@oracle.com>

Hi Max,

Here is an updated webrev, please take a look.

http://cr.openjdk.java.net/~asmotrak/8049171/webrev.02/

Artem

On 01/26/2015 05:03 AM, Weijun Wang wrote:
>
>
> On 1/23/2015 16:12, Artem Smotrakov wrote:
>> If the MANIFEST and the signature files must be at the beginning, should
>> it be considered as a bug in jarsigner? Should it reject such files?
>
> I think so. Will file a bug.
>
>>>
>>> The "jar u" way is to copy each old entry into destination unless the
>>> entry name is in the updated list where the new file will be read.
>>> Finally the untouched files in the updated list are appended.
>> Since tests were not originally for checking some unusual ways for
>> updating jars, I think they need to be updated to use the "jar u" way
>> for adding unsigned entry.
>
> Good.
>
> Thanks
> Max
>
>>
>> Artem
>>>
>>
>>
>>



From weijun.wang at oracle.com  Mon Jan 26 08:11:50 2015
From: weijun.wang at oracle.com (Wang Weijun)
Date: Mon, 26 Jan 2015 16:11:50 +0800
Subject: [9] request for review: 8049171: Additional tests for jarsigner's
	warnings
In-Reply-To: <54C5F30F.7070207@oracle.com>
References: <54BF48BF.9090505@oracle.com>
	
	<54C0E1A3.8070600@oracle.com>
	
	<54C20286.1060106@oracle.com> <54C5A075.4060304@oracle.com>
	<54C5F30F.7070207@oracle.com>
Message-ID: <3A738B71-A464-46D7-BACC-16108A5311C8@oracle.com>

JarUtils:

You can break after line 83.

Otherwise very good.

Thanks
Max

> On Jan 26, 2015, at 15:55, Artem Smotrakov  wrote:
> 
> Hi Max,
> 
> Here is an updated webrev, please take a look.
> 
> http://cr.openjdk.java.net/~asmotrak/8049171/webrev.02/
> 
> Artem
> 
> On 01/26/2015 05:03 AM, Weijun Wang wrote:
>> 
>> 
>> On 1/23/2015 16:12, Artem Smotrakov wrote:
>>> If the MANIFEST and the signature files must be at the beginning, should
>>> it be considered as a bug in jarsigner? Should it reject such files?
>> 
>> I think so. Will file a bug.
>> 
>>>> 
>>>> The "jar u" way is to copy each old entry into destination unless the
>>>> entry name is in the updated list where the new file will be read.
>>>> Finally the untouched files in the updated list are appended.
>>> Since tests were not originally for checking some unusual ways for
>>> updating jars, I think they need to be updated to use the "jar u" way
>>> for adding unsigned entry.
>> 
>> Good.
>> 
>> Thanks
>> Max
>> 
>>> 
>>> Artem
>>>> 
>>> 
>>> 
>>> 
> 



From artem.smotrakov at oracle.com  Mon Jan 26 08:24:31 2015
From: artem.smotrakov at oracle.com (Artem Smotrakov)
Date: Mon, 26 Jan 2015 11:24:31 +0300
Subject: [9] request for review: 8049171: Additional tests for jarsigner's
	warnings
In-Reply-To: <3A738B71-A464-46D7-BACC-16108A5311C8@oracle.com>
References: <54BF48BF.9090505@oracle.com>
	
	<54C0E1A3.8070600@oracle.com>
	
	<54C20286.1060106@oracle.com> <54C5A075.4060304@oracle.com>
	<54C5F30F.7070207@oracle.com>
	<3A738B71-A464-46D7-BACC-16108A5311C8@oracle.com>
Message-ID: <54C5F9BF.3080903@oracle.com>

On 01/26/2015 11:11 AM, Wang Weijun wrote:
> JarUtils:
>
> You can break after line 83.
Sure, I have updated the webrev

http://cr.openjdk.java.net/~asmotrak/8049171/webrev.03/
>
> Otherwise very good.
Thanks for reviewing this.
>
> Thanks
> Max
>
>> On Jan 26, 2015, at 15:55, Artem Smotrakov  wrote:
>>
>> Hi Max,
>>
>> Here is an updated webrev, please take a look.
>>
>> http://cr.openjdk.java.net/~asmotrak/8049171/webrev.02/
>>
>> Artem
>>
>> On 01/26/2015 05:03 AM, Weijun Wang wrote:
>>>
>>> On 1/23/2015 16:12, Artem Smotrakov wrote:
>>>> If the MANIFEST and the signature files must be at the beginning, should
>>>> it be considered as a bug in jarsigner? Should it reject such files?
>>> I think so. Will file a bug.
>>>
>>>>> The "jar u" way is to copy each old entry into destination unless the
>>>>> entry name is in the updated list where the new file will be read.
>>>>> Finally the untouched files in the updated list are appended.
>>>> Since tests were not originally for checking some unusual ways for
>>>> updating jars, I think they need to be updated to use the "jar u" way
>>>> for adding unsigned entry.
>>> Good.
>>>
>>> Thanks
>>> Max
>>>
>>>> Artem
>>>>
>>>>



From aleksej.efimov at oracle.com  Mon Jan 26 09:09:45 2015
From: aleksej.efimov at oracle.com (Aleksej Efimov)
Date: Mon, 26 Jan 2015 12:09:45 +0300
Subject: RFR: 8062923: XSL: Run-time internal error in 'substring()'
In-Reply-To: <54C29324.8050904@oracle.com>
References: <54BCE9B6.7090506@oracle.com> <54C29324.8050904@oracle.com>
Message-ID: <54C60459.5070308@oracle.com>

Joe,
Thank you for the review. Will fix the format issue on line 299 before 
the push. About @run: If I recall it correctly, it is not needed when 
the test resides in a folder (or a tree) that contains the 
TEST.properties file with defined TestNG.dirs option inside it.

With Best Regards,
Aleksej

On 01/23/2015 09:29 PM, huizhe wang wrote:
> Hi Aleksej,
>
> Looks good. There is a minor format issue at line 299.
>
> Note that, for JDK 9, we now have functional/unit tests under 
> jaxp/test. This test would fix in 
> javax/xml/jaxp/unittest/javax/xml/transform. For test format, please 
> take a look at a functional test (javax/xml/jaxp/functional) since 
> Tristan has refactored them, and SQE plans to do the same with the 
> unit tests.  For a testng test, the @run tag is not necessary.
>
> The jaxp tests are defined in its own testset "jaxp".
>
> Thanks,
> Joe
>
> On 1/19/2015 3:25 AM, Aleksej Efimov wrote:
>> Hi,
>>
>> Please, review the fix for the XSL substring function failures. Two 
>> issues were reported during usage of this function:
>> 1. Runtime internal error with negative length: 
>> https://bugs.openjdk.java.net/browse/JDK-8062923
>> 2. Wrong answer when -Inf length is used: 
>> https://bugs.openjdk.java.net/browse/JDK-8062924
>>
>> These two issues can be solved by fix proposed for the review: 
>> http://cr.openjdk.java.net/~aefimov/8062923/webrev.00
>>
>> Testing:
>> xml related regression tests (with new test) - no failures (from jdk 
>> and jaxp repos).
>> JCK xml related tests - no failures.
>>
>> With Best Regards,
>> Aleksej
>



From erik.joelsson at oracle.com  Mon Jan 26 09:47:06 2015
From: erik.joelsson at oracle.com (Erik Joelsson)
Date: Mon, 26 Jan 2015 10:47:06 +0100
Subject: RFR: JDK-8071329: Stop exporting INCLUDE and LIB when building
	on windows
In-Reply-To: <54C10BC8.50805@oracle.com>
References: <54C10BC8.50805@oracle.com>
Message-ID: <54C60D1A.8090408@oracle.com>

Adding core-libs-dev looking for a reviewer for the unpack200 changes.

In the old build, unpack200.exe was linked with cl.exe instead of 
link.exe like all other executables and libraries. Since the formatting 
of options is completely different, the same linker flags were not used. 
In this change, I'm removing this special treatment of unpack200.exe so 
that it is linked just like all other executables.

/Erik

On 2015-01-22 15:40, Erik Joelsson wrote:
> Hello,
>
> Please review this patch, which makes it possible to take a compile 
> command line from the make debug log on Windows, and rerun it in a 
> normal cygwin environment, without the need for running vsvars*.bat 
> first.
>
> When building native code on windows, using Visual Studio, configure 
> extracts the build environment from the setup .bat file provided by VS 
> and sets 3 variables in spec.gmk: PATH, INCLUDE and LIB. These 3 
> variables are also exported into the environment in spec.gmk, so that 
> every tool run by the build will see them. While this is convenient, 
> it makes the command lines used by the build unusable outside of the 
> build, unless you also export these variables with the correct values.
>
> I have removed the need for INCLUDE and LIB to be exported, by 
> converting their contents into compiler and linker flags. These flags 
> conceptually fit well in the recent SYSROOT_CFLAGS and SYSROOT_LDFLAGS 
> variables.
>
> The PATH variable would be nice to not have to set, and while not 
> setting it seems to work most of the time, I suspect that there are 
> cases when it won't work. More specifically, in certain environments, 
> some dll needed by the compiler program might not be on the path 
> without it. So I left it being set for now.
>
> The new LDFLAGS requires unpack200.exe to stop being linked 
> differently to all other executables. There is no reason for this 
> discrepancy that I can find, it just seems like someone did a bit of a 
> quick hack getting it to build long ago in the old build, and we 
> wanted to keep it equivalent in build-infra.
>
> The hotspot build still requires the variables to be exported, so they 
> are still being defined in hotspot-spec.gmk.
>
> While working on this, I stumbled on a problem when running "make 
> reconfigure". The PATH variable value, since exported in make, would 
> get longer and longer for each time you run reconfigure. I fixed this 
> by saving the original path and resetting it before running configure 
> from make.
>
> Bug: https://bugs.openjdk.java.net/browse/JDK-8071329
> Webrevs:
> http://cr.openjdk.java.net/~erikj/8071329/webrev.root.01/
> http://cr.openjdk.java.net/~erikj/8071329/webrev.jdk.01/
>
> /Erik



From konstantin.shefov at oracle.com  Mon Jan 26 11:26:38 2015
From: konstantin.shefov at oracle.com (Konstantin Shefov)
Date: Mon, 26 Jan 2015 14:26:38 +0300
Subject: [8u60] Request for approval: JDK-8067344: Adjust
	java/lang/invoke/LFCaching/LFGarbageCollectedTest.java
	for recent changes in java.lang.invoke
In-Reply-To: <54BFC5EF.6000708@oracle.com>
References: <54BFC5EF.6000708@oracle.com>
Message-ID: <54C6246E.2080209@oracle.com>

Kindly reminder

On 21.01.2015 18:29, Konstantin Shefov wrote:
> Hello,
>
> Please approve the direct backport of the test bug fix to 8u60
>
> The webrev is slightly different from that for JDK 9, but only in line 
> numbers, all the rest is just the same.
>
> The bug: https://bugs.openjdk.java.net/browse/JDK-8067344
> The 8u60 webrev: 
> http://cr.openjdk.java.net/~kshefov/8067344_8u/webrev.00/
>
> JDK 9 changeset: http://hg.openjdk.java.net/jdk9/dev/jdk/rev/75c351a01d3c
> Review thread: 
> http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-December/030452.html
>
> Thanks
> -Konstantin



From sean.coffey at oracle.com  Mon Jan 26 11:32:37 2015
From: sean.coffey at oracle.com (=?UTF-8?B?U2XDoW4gQ29mZmV5?=)
Date: Mon, 26 Jan 2015 11:32:37 +0000
Subject: [8u60] Request for approval: JDK-8067344: Adjust
	java/lang/invoke/LFCaching/LFGarbageCollectedTest.java
	for recent changes in java.lang.invoke
In-Reply-To: <54C6246E.2080209@oracle.com>
References: <54BFC5EF.6000708@oracle.com> <54C6246E.2080209@oracle.com>
Message-ID: <54C625D5.7080707@oracle.com>

Approved. Might be good to add noreg-self label to bug report also.

regards,
Sean.

On 26/01/15 11:26, Konstantin Shefov wrote:
> Kindly reminder
>
> On 21.01.2015 18:29, Konstantin Shefov wrote:
>> Hello,
>>
>> Please approve the direct backport of the test bug fix to 8u60
>>
>> The webrev is slightly different from that for JDK 9, but only in 
>> line numbers, all the rest is just the same.
>>
>> The bug: https://bugs.openjdk.java.net/browse/JDK-8067344
>> The 8u60 webrev: 
>> http://cr.openjdk.java.net/~kshefov/8067344_8u/webrev.00/
>>
>> JDK 9 changeset: 
>> http://hg.openjdk.java.net/jdk9/dev/jdk/rev/75c351a01d3c
>> Review thread: 
>> http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-December/030452.html
>>
>> Thanks
>> -Konstantin
>



From Alan.Bateman at oracle.com  Mon Jan 26 11:52:35 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Mon, 26 Jan 2015 11:52:35 +0000
Subject: [9] RFR of 6880737: (fs) FileLock constructors don't throw NPE
	if the channel argument is null
In-Reply-To: <4976B3C8-81F4-4CDE-A9CF-62DE7F38B8CF@oracle.com>
References: <3B73EA6D-467E-4035-B3DD-949129D07E99@oracle.com>
	<54BA105C.8030809@oracle.com>
	<4976B3C8-81F4-4CDE-A9CF-62DE7F38B8CF@oracle.com>
Message-ID: <54C62A83.5040900@oracle.com>

On 23/01/2015 19:17, Brian Burkhalter wrote:
> Here?s an updated patch including a (perhaps overkill) test:
>
> http://cr.openjdk.java.net/~bpb/6880737/webrev.01/ 
> 
>
Thanks, the null check is fine. The webrev still has the javadoc change, 
it looks okay but not needed.

The test needs to be moved into a FileLock sub-directory to keep the 
current organization consistent.  As it's become a unit test for 
FileLock then I think it would be a bit clearer as a TestNG test but not 
strictly required. One thing that would be good is to switch the test to 
using FileChannel.open and to use try-with-resources. The catching of 
Exception can probably be dropped too, the test will fail if 
un-documented exceptions are throw.

-Alan


From vladimir.x.ivanov at oracle.com  Mon Jan 26 16:41:50 2015
From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov)
Date: Mon, 26 Jan 2015 19:41:50 +0300
Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when
	GWT LambdaForms are shared
In-Reply-To: <5BA1E369-ED87-4EBD-8408-B73B726D91BD@oracle.com>
References: <54B94766.2080102@oracle.com>
	<7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com>
	<54BEA7D7.6080008@oracle.com>
	<5BA1E369-ED87-4EBD-8408-B73B726D91BD@oracle.com>
Message-ID: <54C66E4E.9050805@oracle.com>

John,

What do you think about the following version?
http://cr.openjdk.java.net/~vlivanov/8063137/webrev.02

As you suggested, I reified MHI::profileBranch on LambdaForm level and 
removed @LambdaForm.Shared. My main concern about removing @Sharen was 
that profile pollution can affect the code before profileBranch call 
(akin to 8068915 [1]) and it seems it's the case: Gbemu (at least) is 
sensitive to that change (there's a 10% difference in peak performance 
between @Shared and has_injected_profile()).

I can leave @Shared as is for now or remove it and work on the fix to 
the deoptimization counts pollution. What do you prefer?

Best regards,
Vladimir Ivanov

[1] https://bugs.openjdk.java.net/browse/JDK-8068915

On 1/23/15 4:31 AM, John Rose wrote:
> On Jan 20, 2015, at 11:09 AM, Vladimir Ivanov
> > wrote:
>>
>>> What I'm mainly poking at here is that 'isGWT' is not informative about
>>> the intended use of the flag.
>> I agree. It was an interim solution. Initially, I planned to introduce
>> customization and guide the logic based on that property. But it's not
>> there yet and I needed something for GWT case. Unfortunately, I missed
>> the case when GWT is edited. In that case, isGWT flag is missed and no
>> annotation is set.
>> So, I removed isGWT flag and introduced a check for selectAlternative
>> occurence in LambdaForm shape, as you suggested.
>
> Good.
>
> I think there is a sweeter spot just a little further on.  Make
> profileBranch be an LF intrinsic and expose it like this:
>    GWT(p,t,f;S) := let(a=new int[3]) in lambda(*: S) {
> selectAlternative(profileBranch(p.invoke( *), a), t, f).invoke( *); }
>
> Then selectAlternative triggers branchy bytecodes in the IBGen, and
> profileBranch injects profiling in C2.
> The presence of profileBranch would then trigger the @Shared annotation,
> if you still need it.
>
> After thinking about it some more, I still believe it would be better to
> detect the use of profileBranch during a C2 compile task, and feed that
> to the too_many_traps logic.  I agree it is much easier to stick the
> annotation on in the IBGen; the problem is that because of a minor phase
> ordering problem you are introducing an annotation which flows from the
> JDK to the VM.  Here's one more suggestion at reducing this coupling?
>
> Note that C->set_trap_count is called when each Parse phase processes a
> whole method.  This means that information about the contents of the
> nmethod accumulates during the parse.  Likewise, add a flag method
> C->{has,set}_injected_profile, and set the flag whenever the parser sees
> a profileBranch intrinsic (with or without a constant profile array;
> your call).  Then consult that flag from too_many_traps.  It is true
> that code which is parsed upstream of the very first profileBranch will
> potentially issue a non-trapping fallback, but by definition that code
> would be unrelated to the injected profile, so I don't see a harm in
> that.  If this approach works, then you can remove the annotation
> altogether, which is clearly preferable.  We understand the annotation
> now, but it has the danger of becoming a maintainer's puzzlement.
>
>>
>>> In 'updateCounters', if the counter overflows, you'll get continuous
>>> creation of ArithmeticExceptions.  Will that optimize or will it cause a
>>> permanent slowdown?  Consider a hack like this on the exception path:
>>>    counters[idx] = Integer.MAX_VALUE / 2;
>> I had an impression that VM optimizes overflows in Math.exact*
>> intrinsics, but it's not the case - it always inserts an uncommon
>> trap. I used the workaround you proposed.
>
> Good.
>
>>
>>> On the Name Bikeshed:  It looks like @IgnoreProfile (ignore_profile in
>>> the VM) promises too much "ignorance", since it suppresses branch counts
>>> and traps, but allows type profiles to be consulted.  Maybe something
>>> positive like "@ManyTraps" or "@SharedMegamorphic"?  (It's just a name,
>>> and this is just a suggestion.)
>> What do you think about @LambdaForm.Shared?
>
> That's fine.  Suggest changing the JVM accessor to
> is_lambda_form_shared, because the term "shared" is already overused in
> the VM.
>
> Or, to be much more accurate, s/@Shared/@CollectiveProfile/.  Better
> yet, get rid of it, as suggested above.
>
> (I just realized that profile pollution looks logically parallel to the
> http://en.wikipedia.org/wiki/Tragedy_of_the_commons .)
>
> Also, in the comment explaining the annotation:
>    s/mostly useless/probably polluted by conflicting behavior from
> multiple call sites/
>
> I very much like the fact that profileBranch is the VM intrinsic, not
> selectAlternative.  A VM intrinsic should be nice and narrow like that.
>   In fact, you can delete selectAlternative from vmSymbols while you are
> at it.
>
> (We could do profileInteger and profileClass in a similar way, if that
> turned out to be useful.)
>
> ? John


From artem.smotrakov at oracle.com  Wed Jan 21 06:35:43 2015
From: artem.smotrakov at oracle.com (Artem Smotrakov)
Date: Wed, 21 Jan 2015 09:35:43 +0300
Subject: [9] request for review: 8049171: Additional tests for jarsigner's
	warnings
Message-ID: <54BF48BF.9090505@oracle.com>

Hello,

Please review a couple of new tests for jarsigner's warnings. Basically 
tests run jarsigner and check warning/error messages and exit codes 
according to [1].

https://bugs.openjdk.java.net/browse/JDK-8049171
http://cr.openjdk.java.net/~asmotrak/8049171/webrev.00

[1] 
http://docs.oracle.com/javase/7/docs/technotes/tools/windows/jarsigner.html

Artem




From duncan.macgregor at ge.com  Wed Jan 21 10:39:54 2015
From: duncan.macgregor at ge.com (MacGregor, Duncan (GE Energy Management))
Date: Wed, 21 Jan 2015 10:39:54 +0000
Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned
	when GWT LambdaForms are shared
In-Reply-To: 
References: <54B94766.2080102@oracle.com>
	
Message-ID: 

This version seems to have inconsistent removal of ignore profile in the
hotspot patch. It?s no longer added to vmSymbols but is still referenced
in classFileParser.

On 19/01/2015 20:21, "MacGregor, Duncan (GE Energy Management)"
 wrote:

>Okay, I?ve done some tests of this with the micro benchmarks for our
>language & runtime which show pretty much no change except for one test
>which is now almost 3x slower. It uses nested loops to iterate over an
>array and concatenate the string-like objects it contains, and replaces
>elements with these new longer string-llike objects. It?s a bit of a
>pathological case, and I haven?t seen the same sort of degradation in the
>other benchmarks or in real applications, but I haven?t done serious
>benchmarking of them with this change.
>
>I shall see if the test case can be reduced down to anything simpler while
>still showing the same performance behaviour, and try add some compilation
>logging options to narrow down what?s going on.
>
>Duncan.
>
>On 16/01/2015 17:16, "Vladimir Ivanov" 
>wrote:
>
>>http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/hotspot/
>>http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/jdk/
>>https://bugs.openjdk.java.net/browse/JDK-8063137
>>
>>After GuardWithTest (GWT) LambdaForms became shared, profile pollution
>>significantly distorted compilation decisions. It affected inlining and
>>hindered some optimizations. It causes significant performance
>>regressions for Nashorn (on Octane benchmarks).
>>
>>Inlining was fixed by 8059877 [1], but it didn't cover the case when a
>>branch is never taken. It can cause missed optimization opportunity, and
>>not just increase in code size. For example, non-pruned branch can break
>>escape analysis.
>>
>>Currently, there are 2 problems:
>>   - branch frequencies profile pollution
>>   - deoptimization counts pollution
>>
>>Branch frequency pollution hides from JIT the fact that a branch is
>>never taken. Since GWT LambdaForms (and hence their bytecode) are
>>heavily shared, but the behavior is specific to MethodHandle, there's no
>>way for JIT to understand how particular GWT instance behaves.
>>
>>The solution I propose is to do profiling in Java code and feed it to
>>JIT. Every GWT MethodHandle holds an auxiliary array (int[2]) where
>>profiling info is stored. Once JIT kicks in, it can retrieve these
>>counts, if corresponding MethodHandle is a compile-time constant (and it
>>is usually the case). To communicate the profile data from Java code to
>>JIT, MethodHandleImpl::profileBranch() is used.
>>
>>If GWT MethodHandle isn't a compile-time constant, profiling should
>>proceed. It happens when corresponding LambdaForm is already shared, for
>>newly created GWT MethodHandles profiling can occur only in native code
>>(dedicated nmethod for a single LambdaForm). So, when compilation of the
>>whole MethodHandle chain is triggered, the profile should be already
>>gathered.
>>
>>Overriding branch frequencies is not enough. Statistics on
>>deoptimization events is also polluted. Even if a branch is never taken,
>>JIT doesn't issue an uncommon trap there unless corresponding bytecode
>>doesn't trap too much and doesn't cause too many recompiles.
>>
>>I added @IgnoreProfile and place it only on GWT LambdaForms. When JIT
>>sees it on some method, Compile::too_many_traps &
>>Compile::too_many_recompiles for that method always return false. It
>>allows JIT to prune the branch based on custom profile and recompile the
>>method, if the branch is visited.
>>
>>For now, I wanted to keep the fix very focused. The next thing I plan to
>>do is to experiment with ignoring deoptimization counts for other
>>LambdaForms which are heavily shared. I already saw problems caused by
>>deoptimization counts pollution (see JDK-8068915 [2]).
>>
>>I plan to backport the fix into 8u40, once I finish extensive
>>performance testing.
>>
>>Testing: JPRT, java/lang/invoke tests, nashorn (nashorn testsuite,
>>Octane).
>>
>>Thanks!
>>
>>PS: as a summary, my experiments show that fixes for 8063137 & 8068915
>>[2] almost completely recovers peak performance after LambdaForm sharing
>>[3]. There's one more problem left (non-inlined MethodHandle invocations
>>are more expensive when LFs are shared), but it's a story for another
>>day.
>>
>>Best regards,
>>Vladimir Ivanov
>>
>>[1] https://bugs.openjdk.java.net/browse/JDK-8059877
>>     8059877: GWT branch frequencies pollution due to LF sharing
>>[2] https://bugs.openjdk.java.net/browse/JDK-8068915
>>[3] https://bugs.openjdk.java.net/browse/JDK-8046703
>>     JEP 210: LambdaForm Reduction and Caching
>>_______________________________________________
>>mlvm-dev mailing list
>>mlvm-dev at openjdk.java.net
>>http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
>
>_______________________________________________
>mlvm-dev mailing list
>mlvm-dev at openjdk.java.net
>http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


From pike630 at hotmail.com  Wed Jan 21 12:45:08 2015
From: pike630 at hotmail.com (pike)
Date: Wed, 21 Jan 2015 05:45:08 -0700 (MST)
Subject: Useful message about NullPointerException
Message-ID: <1421844308100-213240.post@n7.nabble.com>

We frequently see NullPointerException in our logs. It's really a big
headache when we see a NullPointerException and it is encapsulated in
another exception as we don't know which object is null and it is throwing
an Exception. Is there any way we can get to know the object type or the
object variable name where the object is null and it is throwing a
NullPointerException? i.e, instead of just saying there is a
NullPointerException, can we add some friendly message? 



--
View this message in context: http://openjdk.5641.n7.nabble.com/Useful-message-about-NullPointerException-tp213240.html
Sent from the OpenJDK Core Libraries mailing list archive at Nabble.com.


From mike at duigou.org  Thu Jan 22 06:37:33 2015
From: mike at duigou.org (Mike Duigou)
Date: Wed, 21 Jan 2015 22:37:33 -0800
Subject: RFR: JDK-8068427 Hashtable deserialization reconstitutes table
	with wrong capacity
In-Reply-To: <54C00F5B.6060005@gmail.com>
References: 
	<7b2072fe0674e465f86518a9f25cef6d@sonic.net>
	<54AFAA61.30008@gmail.com> <54BFA3DF.4090201@gmail.com>
	
	<54C00F5B.6060005@gmail.com>
Message-ID: <5ABA2F90-633D-4CA4-BABB-D352571A655E@duigou.org>

And post hoc looks fine to me as well.


Mike

> On Jan 21, 2015, at 12:43 PM, Peter Levart  wrote:
> 
> Thanks Martin, Mike, Chris and Daniel,
> 
> This has been pushed.
> 
> Regards, Peter
> 
>> On 01/21/2015 05:44 PM, Martin Buchholz wrote:
>> Looks good to me!
>> 
>>> On Wed, Jan 21, 2015 at 5:04 AM, Peter           Levart  wrote:
>>> Hi Martin and others,
>>> 
>>> I have also modified the test per Martin's suggestion to factor out the serialClone() method. Here's the latest webrev:
>>> 
>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.03/
>>> 
>>> Is this ok now?
>>> 
>>> Regards, Peter
>>> 
>>> 
>>>> On 01/09/2015 11:16 AM, Peter Levart wrote:
>>>>> On 01/05/2015 05:52 PM, Mike Duigou wrote:
>>>>> Well spotted Peter. The change looks good though I wonder if it should be:
>>>>> 
>>>>> int length = (int)((elements + elements / 20) / loadFactor) + 3;
>>>>> 
>>>>> FYI, regarding Daniel's suggestion: When similar invariant checks were added to the HashMap deserialization method we found code which relied upon the illegal values. In some cases the serialized HashMaps were created by alternative serialization implementations which included illegal, but until the checks were added, "harmless" values.
>>>>> 
>>>>> The invariant checks should still be added though. It might even be worth adding checks that the other deserialized values are in valid ranges.
>>>>> 
>>>>> Mike
>>>> 
>>>> Hi Mike and others,
>>>> 
>>>> Yes, your suggested length computation is more in "spirit" with the comment above that states: "Compute new length with a bit of room 5% to grow...", since it takes loadFactor into account for that additional 5% too. I changed it to your suggested expression.
>>>> 
>>>> Here's new webrev:
>>>> 
>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.02/
>>>> 
>>>> In addition to valid loadFactor, # of elements is checked to be non-negative. The original length is just "repaired" if it appears to be less than the enforced auto-growth invariant of Hashtable.
>>>> 
>>>> I also expanded the test to exercise more combinations of # of elements and loadFactor. Here's what gets printed with current Hashtable implementation:
>>>> 
>>>>                  ser.  deser.
>>>>    size  load  lentgh  length       valid range ok?
>>>> ------- ----- ------- ------- ----------------- ------
>>>>      10  0.15     127       4      67...    134 NOT-OK
>>>>      10  0.50      31       8      21...     42 NOT-OK
>>>>      10  0.75      15      10      14...     28 NOT-OK
>>>>      10  1.00      15      13      11...     22 OK
>>>>      10  2.50       7       7       5...     10 OK
>>>>      50  0.15     511      12     334...    668 NOT-OK
>>>>      50  0.50     127      30     101...    202 NOT-OK
>>>>      50  0.75     127      42      67...    134 NOT-OK
>>>>      50  1.00      63      55      51...    102 OK
>>>>      50  2.50      31      31      21...     42 OK
>>>>     500  0.15    4095     103    3334...   6668 NOT-OK
>>>>     500  0.50    1023     278    1001...   2002 NOT-OK
>>>>     500  0.75    1023     403     667...   1334 NOT-OK
>>>>     500  1.00     511     511     501...   1002 OK
>>>>     500  2.50     255     255     201...    402 OK
>>>>    5000  0.15   65535    1003   33334...  66668 NOT-OK
>>>>    5000  0.50   16383    2753   10001...  20002 NOT-OK
>>>>    5000  0.75    8191    4003    6667...  13334 NOT-OK
>>>>    5000  1.00    8191    5253    5001...  10002 OK
>>>>    5000  2.50    2047    2047    2001...   4002 OK
>>>> 
>>>> 
>>>> With patched Hashtable, the test passes:
>>>> 
>>>>                  ser.  deser.
>>>>    size  load  lentgh  length       valid range ok?
>>>> ------- ----- ------- ------- ----------------- ------
>>>>      10  0.15     127      69      67...    134 OK
>>>>      10  0.50      31      23      21...     42 OK
>>>>      10  0.75      15      15      14...     28 OK
>>>>      10  1.00      15      13      11...     22 OK
>>>>      10  2.50       7       7       5...     10 OK
>>>>      50  0.15     511     349     334...    668 OK
>>>>      50  0.50     127     107     101...    202 OK
>>>>      50  0.75     127      71      67...    134 OK
>>>>      50  1.00      63      55      51...    102 OK
>>>>      50  2.50      31      23      21...     42 OK
>>>>     500  0.15    4095    3501    3334...   6668 OK
>>>>     500  0.50    1023    1023    1001...   2002 OK
>>>>     500  0.75    1023     703     667...   1334 OK
>>>>     500  1.00     511     511     501...   1002 OK
>>>>     500  2.50     255     213     201...    402 OK
>>>>    5000  0.15   65535   35003   33334...  66668 OK
>>>>    5000  0.50   16383   10503   10001...  20002 OK
>>>>    5000  0.75    8191    7003    6667...  13334 OK
>>>>    5000  1.00    8191    5253    5001...  10002 OK
>>>>    5000  2.50    2047    2047    2001...   4002 OK
>>>> 
>>>> 
>>>> 
>>>> Regards, Peter
>>>> 
>>>>> 
>>>>>> On 2015-01-05 07:48, core-libs-dev-request at openjdk.java.net wrote:
>>>>>> 
>>>>>> Today's Topics:
>>>>>> 
>>>>>>    2. Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes
>>>>>>       table    with wrong capacity (Daniel Fuchs)
>>>>> 
>>>>> 
>>>>>> Message: 2
>>>>>> Date: Mon, 05 Jan 2015 15:52:55 +0100
>>>>>> From: Daniel Fuchs 
>>>>>> To: Peter Levart , core-libs-dev
>>>>>>     
>>>>>> Subject: Re: RFR: JDK-8068427 Hashtable deserialization reconstitutes
>>>>>>     table    with wrong capacity
>>>>>> Message-ID: <54AAA547.8070804 at oracle.com>
>>>>>> Content-Type: text/plain; charset=utf-8; format=flowed
>>>>>> 
>>>>>>> On 04/01/15 18:58, Peter Levart wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> While investigating the following issue:
>>>>>>> 
>>>>>>>      https://bugs.openjdk.java.net/browse/JDK-8029891
>>>>>>> 
>>>>>>> I noticed there's a bug in deserialization code of java.util.Hashtable
>>>>>>> (from day one probably):
>>>>>>> 
>>>>>>>      https://bugs.openjdk.java.net/browse/JDK-8068427
>>>>>>> 
>>>>>>> The fix is a trivial one-character replacement: '*' -> '/', but I also
>>>>>>> corrected some untruthful comments in the neighbourhood (which might
>>>>>>> have been true from day one, but are not any more):
>>>>>>> 
>>>>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/Hashtable.8068427/webrev.01/ 
>>>>>> 
>>>>>> Hi Peter,
>>>>>> 
>>>>>> I wonder whether there should be a guard against loadFactor being
>>>>>> zero/negative/NaN after line 1173, like in the constructor e.g. as
>>>>>> in lines
>>>>>> 
>>>>>>   188         if (loadFactor <= 0 || Float.isNaN(loadFactor))
>>>>>>   189             throw new IllegalArgumentException("Illegal Load:
>>>>>> "+loadFactor);
>>>>>> 
>>>>>> if only to avoid division by zero.
>>>>>> 
>>>>>> best regards,
>>>>>> 
>>>>>> -- daniel
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Regards, Peter
> 


From peter.firmstone at zeus.net.au  Mon Jan 26 14:16:38 2015
From: peter.firmstone at zeus.net.au (Peter Firmstone)
Date: Tue, 27 Jan 2015 00:16:38 +1000
Subject: Explicit Serialization API and Security
In-Reply-To: <54C25DB0.4080207@oracle.com>
References: <54B0CE06.2020408@zeus.net.au>
	<54B3EEC5.5040805@oracle.com>		<1421144693.7530.20.camel@Nokia-N900>
	<54B654F6.6060207@oracle.com>	<1421240334.8856.17.camel@Nokia-N900>	<54BEB91A.4020603@gmail.com>
	<54BFD374.5080803@oracle.com>	<54C00B66.5090302@zeus.net.au>
	<54C01D71.9010901@redhat.com> <54C25DB0.4080207@oracle.com>
Message-ID: <54C64C46.4040001@zeus.net.au>

Thanks Chris,

I've been playing around with a working prototype.

Attached is an example functional AtomicSerial implementation called 
Matches.java, (there are other implementations of AtomicSerial also in 
use) although in this case I'm using a serial constructor, it would also 
work with a static validator.

GetArg and PutArg are caller sensitive wrappers around GetField and 
PutField, that require privilege to instantiate, to prevent an attacker 
using them.

Looking at the very verbose output attached, there are three jvm's 
communicating using Jini Extensible Remote Invocation (JERI, was a 
replacement for RMI).

A side note: JERI scales and outperforms RMI considerably; all hotspots 
are native methods, mostly Sockets, without this level of verbosity of 
course.  ClassLoading is thread confined to avoid contended 
synchronization, its URLClassLoader's use RFC3986 URI normalization for 
URL identity in SecureClassLoader instead of DNS lookup (IP addresses) 
and the policy provider uses immutability to ensure good scalability 
with <1% impact on performance, plus a heap of other optimisations I've 
probably forgotten about.

In the output attached (Prototype ValidatingInputStream) you'll find 
CIRCULAR (ctrl-f), this is an enum Reference placeholder, returned 
through GetField when a circular reference exists.  Circular is replaced 
by the actual object after construction.  Other instances of this enum 
are UNSHARED and DISCARDED.

DISCARDED is experimental and used to mark objects that have been 
discarded from the stream without being instantiated.

Although in this case numerous Serializable object instances are being 
deserialized, classes that implement Serializable and AtomicSerial can 
have identical serial form, so can be cross compatible.  Block data 
written by writeObject is discarded from the stream when using 
AtomicSerial, only fields are consumed by AtomicSerial constructors.  
This ensures code isn't necessary for cross language compatibility.

The intent is to apply constraints via JERI and use a safer 
ObjectInputStream for untrusted endpoints, while using a standard 
ObjectInputStream with trusted authenticated endpoints.

DeSerializationPermission is granted to packages or individual classes, 
via a jvm wide default policy file grant statement:

grant {
     permission net.jini.io.DeSerializationPermission 
"com.sun.jini.test.share.*";
     permission net.jini.io.DeSerializationPermission 
"com.sun.jini.test.impl.norm.*";
     permission net.jini.io.DeSerializationPermission 
"com.sun.jini.lease.*";
     permission net.jini.io.DeSerializationPermission 
"com.sun.jini.landlord.*";
     permission net.jini.io.DeSerializationPermission "com.sun.jini.norm.*";
};


Another object in the stream PortableFactory (search the output 
attached), is a constructor / factory capable of creating any object and 
replacing itself during deserialization.  It does so from unprivileged 
context, to prevent it from creating a ClassLoader for example.  This 
stream is capable of transferring non Serializable objects by 
reconstructing them remotely, this will be provided to allow a user to 
use insecure Serializable instances without serializing them.

For example, deserializing an ArrayList that's been tampered with can 
cause an OutOfMemoryError that takes down the jvm, so it's wiser to 
serialize a size constrained array and recreate a new ArrayList from it 
during deserialization, ValidatingObjectInputStream places a size 
constraint on all deserialized arrays.

Cheers,

Peter.


On 24/01/2015 12:41 AM, Chris Hegarty wrote:
> I have attempted to capture some of the ideas that we have discussed 
> so far.
>
>  https://bugs.openjdk.java.net/browse/JDK-8071471
>
> -Chris.
>
> On 21/01/15 21:43, David M. Lloyd wrote:
>> At some point, the responsibility *must* fall on the author of the
>> serializable class in question to avoid constructs which are exploitable
>> - like finalizers, and classes that can have side-effects before
>> validation can be completed.
>>
>> On 01/21/2015 02:26 PM, Peter Firmstone wrote:
>>> Don't forget that "null" may also be an invalid state.
>>>
>>> What I have learnt so far:
>>>
>>>    1. An attacker can craft a stream to obtain a reference to any 
>>> object
>>>       created during deserialization, using finalizers or circular 
>>> links
>>>       (there may be yet other undiscovered methods).
>>>    2. An attacker can craft a stream that deliberately doesn't satisfy
>>>       invariants, in order to use an object to perform a function that
>>>       wasn't intended by its developer.
>>>    3. Objects that interact with the stream directly using 
>>> readObject et
>>>       al, are often prone to DOS.  Example, many objects read a length
>>>       integer from the stream when creating an array or collection,
>>>       without first validating it.
>>>    4. Objects that interact directly with the stream become an implicit
>>>       part of the stream protocol.
>>>    5. Once you allow an object to be created, it's too late to
>>>       invalidate invariants, unless the class is final and invariants
>>>       are checked in every method call.
>>>    6. We need to be able to restrict classes used for 
>>> deserialization to
>>>       those we trust to check invariants properly (but we haven't
>>>       provided a way for them to avoid object construction yet).
>>>    7. A static validator method can ONLY be used to check field
>>>       invariants, not other objects and primitives that are read
>>>       directly from the stream by an arbitrary Object during the 
>>> process
>>>       of deserialization.
>>>    8. The jvm can be modified to delay finalizer registration for
>>>       deserialization.
>>>    9. Circular links can be disallowed.
>>>
>>> Ultimately however, all proposed changes add complexity, but when an
>>> object has been created with invalid invariants, an attacker will 
>>> find a
>>> way.
>>>
>>> Thank you all for your time, this has been a very good discussion.
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>> On 22/01/2015 2:27 AM, Chris Hegarty wrote:
>>>> On 20/01/15 20:22, Peter Levart wrote:
>>>>> Hi Chris and Peter,
>>>>>
>>>>> It just occurred to me that the following scheme would provide 
>>>>> failure
>>>>> atomicity for the whole Object regardless of whether readObject 
>>>>> methods
>>>>> are used or not for various classes in hierarchy:
>>>>>
>>>>>
>>>>> - allocate uninitialized instance
>>>>> - call default accessible constructor of the most specific
>>>>> non-Serializable class
>>>>> - deserialize (by calling readObject methods where provided) the 
>>>>> fields
>>>>> of all classes in hierarchy like normally
>>>>> (up to this point, nothing is changed from what we have now)
>>>>> - if deserialization fails anywhere, undo everything by setting 
>>>>> all the
>>>>> fields in the Serializable part of the hierarchy to default values
>>>>> (null
>>>>> for references, 0 for primitives), abandon the object and propagate
>>>>> failure.
>>>>
>>>> I think this is a good idea, and I can prototype something to this
>>>> affect.
>>>>
>>>> -Chris.
>>>>
>>>>
>>>>> While deserializing, the object is in inconsistent state. If
>>>>> deserialization fails, this state is rolled-back to uninitialized
>>>>> state.
>>>>> finalize() can still get to the instance, but it will be 
>>>>> uninitialized.
>>>>>
>>>>>
>>>>> Peter
>>>>>
>>>>> On 01/14/2015 01:58 PM, Peter Firmstone wrote:
>>>>>>
>>>>>> Hi Chris,
>>>>>>
>>>>>> Sorry, no.
>>>>>>
>>>>>> Currently when an ObjectStreamClass is read in from the stream, the
>>>>>> framework searches for the first zero arg constructor of a non
>>>>>> serializable class and creates and instance of the class read and
>>>>>> resolved from the stream, however it does so using a super class
>>>>>> constructor.
>>>>>>
>>>>>> Then from the super class down, fields are read in and set in order
>>>>>> for each class in the object's inheritance hierarchy.
>>>>>>
>>>>>> The alternative I propose, doesn't create the instance, instead it
>>>>>> reads the fields from the stream, one by one and without 
>>>>>> instantiating
>>>>>> them, if they are newly read objects, stores them temporarily into
>>>>>> byte [] arrays in a Map with reference handle keys, otherwise it 
>>>>>> just
>>>>>> holds the reference handle.
>>>>>>
>>>>>> What it does next is wrap this information into a caller sensitive
>>>>>> api, GetFields or ReadSerial instance, that is passed as a 
>>>>>> constructor
>>>>>> parameter to the child class serial constructor.
>>>>>>
>>>>>> The child class checks invariants and reads each field it needs 
>>>>>> using
>>>>>> a static method prior to calling a superclass constructor, each 
>>>>>> class
>>>>>> in the inheritance hierarchy for the object then checks its 
>>>>>> invariants
>>>>>> until it gets to the first non serializable superclass.
>>>>>>
>>>>>> The benefit of this order is that each class is present in the 
>>>>>> thread
>>>>>> security context, so protection domain security and invariants are
>>>>>> enforced before instantiating an object.
>>>>>>
>>>>>> Hope this helps illuminate it a little better, regards,
>>>>>>
>>>>>> Peter.
>>>>>>
>>>>>> ----- Original message -----
>>>>>> > Peter F,
>>>>>> >
>>>>>> > I am still struggling with the basic concept of you proposal. 
>>>>>> Let me
>>>>>> see
>>>>>> > if I understand it correctly. Does the following describe a 
>>>>>> similar
>>>>>> > scenario as you envisage:
>>>>>> >
>>>>>> >    1) For each Serializable type, T, in the deserialized types
>>>>>> >          hierarchy, starting with the top most ( closest to
>>>>>> j.l.Object ),
>>>>>> >
>>>>>> >          1a) Read T's fields from the stream, fields
>>>>>> >
>>>>>> >          1b) validate(t, fields)  // t will be null first time
>>>>>> >
>>>>>> >          1c) allocate a new instance of T, and assign to t
>>>>>> >
>>>>>> >          1d) set fields in t
>>>>>> >
>>>>>> >    2) Return t;
>>>>>> >
>>>>>> > So for each level in the hierarchy, an instance of a type is 
>>>>>> created
>>>>>> > only after its invariants have been checked. This instance is then
>>>>>> > passed to the next level so it can participate in that levels
>>>>>> invariants
>>>>>> > validation.
>>>>>> >
>>>>>> > If this scenario is along the same lines as yours, then I just 
>>>>>> don't
>>>>>> see
>>>>>> > how 1c above will always be possible.
>>>>>> >
>>>>>> > If we could somehow make the object caller sensitive until after
>>>>>> > deserialization completes, then could avoid having to try to
>>>>>> allocate
>>>>>> > multiple instance down the hierarchy.
>>>>>> >
>>>>>> > -Chris.
>>>>>> >
>>>>>> > On 13/01/15 10:24, Peter Firmstone wrote:
>>>>>> > > Could we use a static validator method and generate bytecode for
>>>>>> > > constructors dynamically?
>>>>>> > >
>>>>>> > > The developer can optionally implement the constructors.
>>>>>> > >
>>>>>> > > static GetField invariantCheck(GetField f);
>>>>>> > >
>>>>>> > > Create a caller sensitive GetField implementation and add a two
>>>>>> new
>>>>>> > > methods to GetField:
>>>>>> > >
>>>>>> > > abstract Object createSuper(); // to access superclass
>>>>>> objectmethods
>>>>>> > > for inavariant checking.
>>>>>> > >
>>>>>> > > abstract Class getType(String name);
>>>>>> > >
>>>>>> > > Set fields from within constructors.
>>>>>> > >
>>>>>> > > The generated constructors are straight forward:
>>>>>> > >
>>>>>> > > 1. Call static method.
>>>>>> > > 2. Call super class constructor with result from static method.
>>>>>> > > 3. Set final fields
>>>>>> > > 4. How to set transient fields, implement a private method 
>>>>>> called
>>>>>> from
>>>>>> > > within the constructor?
>>>>>> > >
>>>>>> > > Require a permission to extend GetField?
>>>>>> > >
>>>>>>
>>>>>
>>>
>>

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Matches.java
URL: 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Prototype Validating Input Stream.txt
URL: 

From paul.sandoz at oracle.com  Mon Jan 26 18:30:07 2015
From: paul.sandoz at oracle.com (Paul Sandoz)
Date: Mon, 26 Jan 2015 18:30:07 +0000
Subject: 8069325: Pattern.splitAsStream does not return input if it is
	empty and there is no match
In-Reply-To: <54BE91DF.8010802@oracle.com>
References: <722C43ED-09C0-4C77-9C6F-98A05D1134EE@oracle.com>
	<54BE91DF.8010802@oracle.com>
Message-ID: 


On Jan 20, 2015, at 5:35 PM, Xueming Shen  wrote:

> On 1/20/15 8:17 AM, Paul Sandoz wrote:
>> Hi,
>> 
>> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8069325-Pattern-splitAsStream-emptyInput/webrev/
>> 
>> This patch fixes an edge case in Pattern.splitAsStream for matching against an empty input string, which deviated from the behaviour of Pattern.split. When there are no matches a stream containing the input string should be returned rather than an empty stream.
>> 

We got side-tracked by the discussion below. Did you have a chance to look at the patch?

Thanks,
Paul.


>> --
>> 
>> I have kept compatibility with Pattern.split(String ) but i noticed another an edge case.
>> 
>> What should the following return:
>> 
>>   Pattern.compile("").split("")
>> 
>> [] or [""]?
>> 
>> There is a zero-width match at the beginning and an empty remaining segment both of which should be discarded, as such i would expect the result to be [] rather than as [""], as currently produced result.
> 
> It may depend on how the "trailing empty string"  gets interpreted. Is it possible to interpret it as the
> empty string is the result of the "substring from the beginning 0-width match and the end of the input
> sequence", any thing after that is "trailing"?
> 
> It would be clear if the spec explicitly said, the result of splitting an empty input is an empty string.
> 
> I would assume someone, mostly the user of String.split(), will get hit by this "incompatible" change.
> 
> -Sherman
>> 
>> If people agree that this is an issue i suggest we log a new one independent of fixing 8069325.
>> 
>> Thanks,
>> Paul.
> 



From vladimir.x.ivanov at oracle.com  Mon Jan 26 18:31:30 2015
From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov)
Date: Mon, 26 Jan 2015 21:31:30 +0300
Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when
	GWT LambdaForms are shared
In-Reply-To: <54C66E4E.9050805@oracle.com>
References: <54B94766.2080102@oracle.com>
	<7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com>
	<54BEA7D7.6080008@oracle.com>
	<5BA1E369-ED87-4EBD-8408-B73B726D91BD@oracle.com>
	<54C66E4E.9050805@oracle.com>
Message-ID: <54C68802.7020105@oracle.com>

> As you suggested, I reified MHI::profileBranch on LambdaForm level and
> removed @LambdaForm.Shared. My main concern about removing @Sharen was
> that profile pollution can affect the code before profileBranch call
> (akin to 8068915 [1]) and it seems it's the case: Gbemu (at least) is
> sensitive to that change (there's a 10% difference in peak performance
> between @Shared and has_injected_profile()).
Ignore that. Additional runs don't prove there's a regression on Gbemu. 
There's some variance on Gbemu and it's present w/ and w/o @Shared.

Best regards,
Vladimir Ivanov

> I can leave @Shared as is for now or remove it and work on the fix to
> the deoptimization counts pollution. What do you prefer?
>
> Best regards,
> Vladimir Ivanov
>
> [1] https://bugs.openjdk.java.net/browse/JDK-8068915
>
> On 1/23/15 4:31 AM, John Rose wrote:
>> On Jan 20, 2015, at 11:09 AM, Vladimir Ivanov
>> >
>> wrote:
>>>
>>>> What I'm mainly poking at here is that 'isGWT' is not informative about
>>>> the intended use of the flag.
>>> I agree. It was an interim solution. Initially, I planned to introduce
>>> customization and guide the logic based on that property. But it's not
>>> there yet and I needed something for GWT case. Unfortunately, I missed
>>> the case when GWT is edited. In that case, isGWT flag is missed and no
>>> annotation is set.
>>> So, I removed isGWT flag and introduced a check for selectAlternative
>>> occurence in LambdaForm shape, as you suggested.
>>
>> Good.
>>
>> I think there is a sweeter spot just a little further on.  Make
>> profileBranch be an LF intrinsic and expose it like this:
>>    GWT(p,t,f;S) := let(a=new int[3]) in lambda(*: S) {
>> selectAlternative(profileBranch(p.invoke( *), a), t, f).invoke( *); }
>>
>> Then selectAlternative triggers branchy bytecodes in the IBGen, and
>> profileBranch injects profiling in C2.
>> The presence of profileBranch would then trigger the @Shared annotation,
>> if you still need it.
>>
>> After thinking about it some more, I still believe it would be better to
>> detect the use of profileBranch during a C2 compile task, and feed that
>> to the too_many_traps logic.  I agree it is much easier to stick the
>> annotation on in the IBGen; the problem is that because of a minor phase
>> ordering problem you are introducing an annotation which flows from the
>> JDK to the VM.  Here's one more suggestion at reducing this coupling?
>>
>> Note that C->set_trap_count is called when each Parse phase processes a
>> whole method.  This means that information about the contents of the
>> nmethod accumulates during the parse.  Likewise, add a flag method
>> C->{has,set}_injected_profile, and set the flag whenever the parser sees
>> a profileBranch intrinsic (with or without a constant profile array;
>> your call).  Then consult that flag from too_many_traps.  It is true
>> that code which is parsed upstream of the very first profileBranch will
>> potentially issue a non-trapping fallback, but by definition that code
>> would be unrelated to the injected profile, so I don't see a harm in
>> that.  If this approach works, then you can remove the annotation
>> altogether, which is clearly preferable.  We understand the annotation
>> now, but it has the danger of becoming a maintainer's puzzlement.
>>
>>>
>>>> In 'updateCounters', if the counter overflows, you'll get continuous
>>>> creation of ArithmeticExceptions.  Will that optimize or will it
>>>> cause a
>>>> permanent slowdown?  Consider a hack like this on the exception path:
>>>>    counters[idx] = Integer.MAX_VALUE / 2;
>>> I had an impression that VM optimizes overflows in Math.exact*
>>> intrinsics, but it's not the case - it always inserts an uncommon
>>> trap. I used the workaround you proposed.
>>
>> Good.
>>
>>>
>>>> On the Name Bikeshed:  It looks like @IgnoreProfile (ignore_profile in
>>>> the VM) promises too much "ignorance", since it suppresses branch
>>>> counts
>>>> and traps, but allows type profiles to be consulted.  Maybe something
>>>> positive like "@ManyTraps" or "@SharedMegamorphic"?  (It's just a name,
>>>> and this is just a suggestion.)
>>> What do you think about @LambdaForm.Shared?
>>
>> That's fine.  Suggest changing the JVM accessor to
>> is_lambda_form_shared, because the term "shared" is already overused in
>> the VM.
>>
>> Or, to be much more accurate, s/@Shared/@CollectiveProfile/.  Better
>> yet, get rid of it, as suggested above.
>>
>> (I just realized that profile pollution looks logically parallel to the
>> http://en.wikipedia.org/wiki/Tragedy_of_the_commons .)
>>
>> Also, in the comment explaining the annotation:
>>    s/mostly useless/probably polluted by conflicting behavior from
>> multiple call sites/
>>
>> I very much like the fact that profileBranch is the VM intrinsic, not
>> selectAlternative.  A VM intrinsic should be nice and narrow like that.
>>   In fact, you can delete selectAlternative from vmSymbols while you are
>> at it.
>>
>> (We could do profileInteger and profileClass in a similar way, if that
>> turned out to be useful.)
>>
>> ? John


From paul.sandoz at oracle.com  Mon Jan 26 18:32:04 2015
From: paul.sandoz at oracle.com (Paul Sandoz)
Date: Mon, 26 Jan 2015 18:32:04 +0000
Subject: RFR 8050819: Please add java.util.Stream.ofNullable(T object)
In-Reply-To: <93EC212E-69DA-4874-8255-E73746404461@oracle.com>
References: <93EC212E-69DA-4874-8255-E73746404461@oracle.com>
Message-ID: <3F562D05-6BD1-4278-BB12-47CEEEFA5CF2@oracle.com>

On Jan 20, 2015, at 7:05 PM, Paul Sandoz  wrote:
> Hi,
> 
> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050819-Stream-ofNullable/webrev/
> 
> Stream.ofNullable can make it easier to work fluently when streams are produced, for example within flatMap.
> 

So far there have been "null" reviews :-)

Paul.

> A CCC will be filed.
> 
> Thanks,
> Paul.



From Alan.Bateman at oracle.com  Mon Jan 26 18:42:53 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Mon, 26 Jan 2015 18:42:53 +0000
Subject: RFR 8050819: Please add java.util.Stream.ofNullable(T object)
In-Reply-To: <3F562D05-6BD1-4278-BB12-47CEEEFA5CF2@oracle.com>
References: <93EC212E-69DA-4874-8255-E73746404461@oracle.com>
	<3F562D05-6BD1-4278-BB12-47CEEEFA5CF2@oracle.com>
Message-ID: <54C68AAD.4060202@oracle.com>

On 26/01/2015 18:32, Paul Sandoz wrote:
> On Jan 20, 2015, at 7:05 PM, Paul Sandoz  wrote:
>> Hi,
>>
>> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050819-Stream-ofNullable/webrev/
>>
>> Stream.ofNullable can make it easier to work fluently when streams are produced, for example within flatMap.
>>
> So far there have been "null" reviews :-)
>
This looks okay to me, and the naming consistent with Optional.

-Alan.


From ecki at zusammenkunft.net  Mon Jan 26 19:20:11 2015
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Mon, 26 Jan 2015 20:20:11 +0100
Subject: Useful message about NullPointerException
In-Reply-To: <1421844308100-213240.post@n7.nabble.com>
References: <1421844308100-213240.post@n7.nabble.com>
Message-ID: <20150126202011.000049a2.ecki@zusammenkunft.net>

Am Wed, 21 Jan 2015 05:45:08 -0700 (MST)
schrieb pike :

> We frequently see NullPointerException in our logs. It's really a big
> headache when we see a NullPointerException and it is encapsulated in
> another exception as we don't know which object is null and it is
> throwing an Exception. Is there any way we can get to know the object
> type or the object variable name where the object is null and it is
> throwing a NullPointerException? i.e, instead of just saying there is
> a NullPointerException, can we add some friendly message? 

Note that if you keep the stack information in an exception it points
(most of the time) exactly to the location where the null access
happens.

I can imagine it is rather hard for the VM to add more informations.
Your best bet is to avoid the NPEs and log the exceptions properly.

Gruss
Bernd


From mandy.chung at oracle.com  Mon Jan 26 19:23:17 2015
From: mandy.chung at oracle.com (Mandy Chung)
Date: Mon, 26 Jan 2015 11:23:17 -0800
Subject: Review request 8057645: Deprivilege JAX-WS, JAXB, JAF to extension
	class loader
Message-ID: <54C69425.1040306@oracle.com>

http://cr.openjdk.java.net/~mchung/jdk9/webrevs/8057645/webrev.00/

This patch proposes to move java.xml.ws, java.xml.bind, java.activation 
out of the boot loader and be loaded by the extension class loader.  We 
grant java.xml.ws and java.xml.bind the minimum set of permissions.  
java.activation hasAllPermission for now and that can be revised in the 
future when JAF team identifies the permission set required java.activation.

Miroslav - can you confirm if the JAX-WS and JAXB standalone tests pass 
with this patch?

Existing code that assumes the defining class loader of JAX-WS, JAXB, 
JAF classes may be impacted by this change (e.g. the class loader 
delegation to the bootstrap class loader skipping the extension class 
loader).  They are standalone technologies that used to be loaded by 
non-null class loader before they were included in Java SE.  It should 
be rare of such dependency. Callbacks may assume java.xml.ws and 
java.xml.bind classes to have AllPermissions so that when running with 
security manager, if the permission required for callback is not part of 
the permission set granted to java.xml.ws and java.xml.bind, 
SecurityException will be thrown.  We need customer testings to identify 
this callback permission case and revisit if they should be granted with 
AllPermission for JDK 9.

Mandy


From lance.andersen at oracle.com  Mon Jan 26 19:24:11 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Mon, 26 Jan 2015 14:24:11 -0500
Subject: Review request for JDK-8051709: Convert JAXP function tests:
	javax.xml.datatype to jtreg (testng) tests
In-Reply-To: <00b801d03933$4d0d3190$e72794b0$@oracle.com>
References: <00b801d03933$4d0d3190$e72794b0$@oracle.com>
Message-ID: <5A84A3CE-440F-49BC-87F6-50AF44B8D80E@oracle.com>

Hi Frank,

I think this looks good. 


Not sure if you are going to add more tests in the future, but would be good to have tests such as

new Duration(x.toString()).equals(x)


Perhaps a few more checks on expected toString() output?.


For XMLGregorianCalendarTest.java,  I would consider at some point adding more permutations of some of the tests that are validating a bugs(now that you are adding this as a new test suite to openjdk)


Best
Lance

On Jan 26, 2015, at 1:42 AM, Frank Yuan  wrote:

> Hi, Joe and All
> 
> We are working on moving internal jaxp functional tests to open jdk repo.
> This is the datatype suite. Would you please review these test?  Any comment
> will be appreciated.
> 
> bug: https://bugs.openjdk.java.net/browse/JDK-8051709
> webrev: http://cr.openjdk.java.net/~fyuan/8051709/webrev.00/
> 
> 
> Thanks,
> 
> Frank
> 



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From lance.andersen at oracle.com  Mon Jan 26 19:48:55 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Mon, 26 Jan 2015 14:48:55 -0500
Subject: Review request 8057645: Deprivilege JAX-WS, JAXB,
	JAF to extension class loader
In-Reply-To: <54C69425.1040306@oracle.com>
References: <54C69425.1040306@oracle.com>
Message-ID: 

The changes look good Mandy.  Hopefully we won't have too many issues with tweaking permissions?

Best
Lance
On Jan 26, 2015, at 2:23 PM, Mandy Chung  wrote:

> http://cr.openjdk.java.net/~mchung/jdk9/webrevs/8057645/webrev.00/
> 
> This patch proposes to move java.xml.ws, java.xml.bind, java.activation out of the boot loader and be loaded by the extension class loader.  We grant java.xml.ws and java.xml.bind the minimum set of permissions.  java.activation hasAllPermission for now and that can be revised in the future when JAF team identifies the permission set required java.activation.
> 
> Miroslav - can you confirm if the JAX-WS and JAXB standalone tests pass with this patch?
> 
> Existing code that assumes the defining class loader of JAX-WS, JAXB, JAF classes may be impacted by this change (e.g. the class loader delegation to the bootstrap class loader skipping the extension class loader).  They are standalone technologies that used to be loaded by non-null class loader before they were included in Java SE.  It should be rare of such dependency. Callbacks may assume java.xml.ws and java.xml.bind classes to have AllPermissions so that when running with security manager, if the permission required for callback is not part of the permission set granted to java.xml.ws and java.xml.bind, SecurityException will be thrown.  We need customer testings to identify this callback permission case and revisit if they should be granted with AllPermission for JDK 9.
> 
> Mandy



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From 00pebuis at bsu.edu  Mon Jan 26 20:05:20 2015
From: 00pebuis at bsu.edu (Buis, Paul)
Date: Mon, 26 Jan 2015 20:05:20 +0000
Subject: java.util.DualPivotQuickSort does not guarantee NlogN time
Message-ID: 

DualPivotQuickSort is used by Arrays.sort() and provides NlogN average performance, but does not guarantee NlogN worst case performance. It would be relatively easy to incorporate the basic idea of IntroSort (see http://en.wikipedia.org/wiki/Introsort) to provide such a guarantee. To make this happen, in java.util.DualPivotQuicksort, for each primitive type one  need only keep track of the recursion depth relative to some constant times the log of the size of what is being sorted. So, for int[], one does something like:



// IntroSort normally uses a DEPTH_FACTOR of 2
private static final int DEPTH_FACTOR = 4;

private static void sort(int[] a, int left, int right, boolean leftmost){
    sort(a, left, right, leftmost 0, DEPTH_FACTOR*(31-Integer. numberOfLeadingZeros(right-left));
}


Then, modify the existing sort to take a couple of extra parameters to keep track of the recursion depth and if the recursion is going too deep, do a heapSort to prevent potential for quadratic time performance.

private static void sort(int[] a, int left, int right, boolean leftmost, int depth, int maxDepth){
   if (depth > maxDepth) {
      heapSort(a, left, right);
      return;
   }
   ...
   // replace recursive calls with
   sort(a, ... , depth+1, maxDepth)
}


I'd be happy to provide a complete DualPivotQuicksort.java source file with this implemented if the folks in charge think this is worthwhile.


From kedar.mhaswade at gmail.com  Mon Jan 26 20:16:44 2015
From: kedar.mhaswade at gmail.com (kedar mhaswade)
Date: Mon, 26 Jan 2015 12:16:44 -0800
Subject: Useful message about NullPointerException
In-Reply-To: <20150126202011.000049a2.ecki@zusammenkunft.net>
References: <1421844308100-213240.post@n7.nabble.com>
	<20150126202011.000049a2.ecki@zusammenkunft.net>
Message-ID: 

On Mon, Jan 26, 2015 at 11:20 AM, Bernd Eckenfels 
wrote:

> Am Wed, 21 Jan 2015 05:45:08 -0700 (MST)
> schrieb pike :
>
> > We frequently see NullPointerException in our logs. It's really a big
> > headache when we see a NullPointerException and it is encapsulated in
> > another exception as we don't know which object is null and it is
> > throwing an Exception. Is there any way we can get to know the object
> > type or the object variable name where the object is null and it is
> > throwing a NullPointerException? i.e, instead of just saying there is
> > a NullPointerException, can we add some friendly message?
>
> Note that if you keep the stack information in an exception it points
> (most of the time) exactly to the location where the null access
> happens.
>
>
?Maybe the objects in question are not talking only to their immediate
friends ?
This happens when we tend to do something like *a.getB().getC()* and we
can't tell by looking at the NPE stack trace (which works on lines of
source code) if *a* was null or what *a.getB()* returned was null.

I can imagine it is rather hard for the VM to add more informations.
> Your best bet is to avoid the NPEs and log the exceptions properly.


> Gruss
> Bernd
>


From 00pebuis at bsu.edu  Mon Jan 26 20:40:22 2015
From: 00pebuis at bsu.edu (Buis, Paul)
Date: Mon, 26 Jan 2015 20:40:22 +0000
Subject: Arrays.sort() inconsistent in which variant of
	DualPivotQuicksort.sort() is invoked
Message-ID: 

Arrays.sort(byte[]) and Array.sort(byte[], int, int) both invoke the 3 argument version of DualPivotQuickSort.sort() when  Arrays.sort() for all other primitive types invokes the 6 argument version.  This appears to be because there is no 6 argument version of DualPivotQuickSort.sort(byte[], ...). Is there a particular reason for this?


From 00pebuis at bsu.edu  Mon Jan 26 20:54:48 2015
From: 00pebuis at bsu.edu (Buis, Paul)
Date: Mon, 26 Jan 2015 20:54:48 +0000
Subject: No generic version of DualPivotQuickSort
Message-ID: 

The java.util.DualPivotQuicksort class implements sort() methods for the primitive types, has no methods that deal with generic arrays with methods like

static > void sort(T[] array, int iStart, int iEnd)

or

static > void sort(T[] a, int left, int right, T[] work, int workBase, int workLen)

Similarly, it contains no methods for Comparator-based sorting of generic arrays. Might it make sense to add such methods to DualPivotQucksort? The Arrays.sort() methods for generic arrays use TimSort which is likely to be slower. TimSort is stable and DualPivotQuickSort is not. Might it make sense to allow the user to pick a stable or an unstable version of a generic Arrays.sort() and use TimSort when stability is desired and DualPivotQuicksort when it is not?


From Alan.Bateman at oracle.com  Mon Jan 26 20:55:28 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Mon, 26 Jan 2015 20:55:28 +0000
Subject: Review request 8057645: Deprivilege JAX-WS, JAXB, JAF to extension
	class loader
In-Reply-To: <54C69425.1040306@oracle.com>
References: <54C69425.1040306@oracle.com>
Message-ID: <54C6A9C0.1030000@oracle.com>

On 26/01/2015 19:23, Mandy Chung wrote:
> http://cr.openjdk.java.net/~mchung/jdk9/webrevs/8057645/webrev.00/
>
> This patch proposes to move java.xml.ws, java.xml.bind, 
> java.activation out of the boot loader and be loaded by the extension 
> class loader.  We grant java.xml.ws and java.xml.bind the minimum set 
> of permissions.  java.activation hasAllPermission for now and that can 
> be revised in the future when JAF team identifies the permission set 
> required java.activation.
>
> Miroslav - can you confirm if the JAX-WS and JAXB standalone tests 
> pass with this patch?
>
> Existing code that assumes the defining class loader of JAX-WS, JAXB, 
> JAF classes may be impacted by this change (e.g. the class loader 
> delegation to the bootstrap class loader skipping the extension class 
> loader).  They are standalone technologies that used to be loaded by 
> non-null class loader before they were included in Java SE.  It should 
> be rare of such dependency. Callbacks may assume java.xml.ws and 
> java.xml.bind classes to have AllPermissions so that when running with 
> security manager, if the permission required for callback is not part 
> of the permission set granted to java.xml.ws and java.xml.bind, 
> SecurityException will be thrown.  We need customer testings to 
> identify this callback permission case and revisit if they should be 
> granted with AllPermission for JDK 9.
The changes looks good and will be interested to see if anyone is 
running JAX-WS and JAXB with a security manager willing to try out JDK 9 
builds.

-Alan.


From lowasser at google.com  Mon Jan 26 21:01:44 2015
From: lowasser at google.com (Louis Wasserman)
Date: Mon, 26 Jan 2015 21:01:44 +0000
Subject: No generic version of DualPivotQuickSort
References: 
Message-ID: 

My understanding was that the performance difference wasn't perceived to
outweigh the user confusion risks associated with multiple sorting
algorithm options. Right now, only one option is presented -- a stable sort
-- and that algorithm is intended to be *nearly* as fast as an unstable
sort, and unstable sorting algorithms are *subtly* unsuitable for some use
cases in a way that might be hard for users to debug or realize why their
code is buggy.

Is that an accurate assessment?

On Mon Jan 26 2015 at 12:55:18 PM Buis, Paul <00pebuis at bsu.edu> wrote:

> The java.util.DualPivotQuicksort class implements sort() methods for the
> primitive types, has no methods that deal with generic arrays with methods
> like
>
> static > void sort(T[] array, int iStart,
> int iEnd)
>
> or
>
> static > void sort(T[] a, int left, int
> right, T[] work, int workBase, int workLen)
>
> Similarly, it contains no methods for Comparator-based sorting of generic
> arrays. Might it make sense to add such methods to DualPivotQucksort? The
> Arrays.sort() methods for generic arrays use TimSort which is likely to be
> slower. TimSort is stable and DualPivotQuickSort is not. Might it make
> sense to allow the user to pick a stable or an unstable version of a
> generic Arrays.sort() and use TimSort when stability is desired and
> DualPivotQuicksort when it is not?
>


From lance.andersen at oracle.com  Mon Jan 26 21:07:39 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Mon, 26 Jan 2015 16:07:39 -0500
Subject: Review request for JDK-8051547: Convert JAXP function tests:
	javax.xml.validation.* to jtreg (testng) tests
In-Reply-To: <007601d03917$ff381180$fda83480$@oracle.com>
References: <007601d03917$ff381180$fda83480$@oracle.com>
Message-ID: <1EA0C710-63B0-4299-AAC1-8C711FB9E059@oracle.com>

Hi Frank,

This looks pretty good overall.

Best
Lance
On Jan 25, 2015, at 10:27 PM, Frank Yuan  wrote:

> Hi, Joe and All
> 
> We are working on moving internal jaxp functional tests to open jdk repo.
> This is the validation suite. Would you please review these test?  Any
> comment will be appreciated.
> 
> bug: https://bugs.openjdk.java.net/browse/JDK-8051547
> webrev: http://cr.openjdk.java.net/~fyuan/8051547/webrev.00/
> 
> 
> Thanks,
> 
> Frank
> 



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From jeffhain at rocketmail.com  Mon Jan 26 21:15:43 2015
From: jeffhain at rocketmail.com (Jeff Hain)
Date: Mon, 26 Jan 2015 21:15:43 +0000 (UTC)
Subject: java.util.DualPivotQuickSort does not guarantee NlogN time
In-Reply-To: 
References: 
Message-ID: <619749187.943332.1422306943878.JavaMail.yahoo@mail.yahoo.com>

Hi.

>It would be relatively easy to incorporate the basic idea of IntroSort (see http://en.wikipedia.org/wiki/Introsort)

For the record, I tried (not too hard :) to get it piggy-backed into DualPivotQuickSort-related modifications,
but without success :
http://mail.openjdk.java.net/pipermail/core-libs-dev/2010-September/004889.html

Note that those paranoid (like me) about quadratic hickups might also
be paranoid about memory and GC hickups due to Arrays.sort(...) creating
temporary arrays in some cases (big arrays especially hurt),
so they could still have to resort to a custom "QuietSort" class.

-Jeff



From 00pebuis at bsu.edu  Mon Jan 26 21:45:36 2015
From: 00pebuis at bsu.edu (Buis, Paul)
Date: Mon, 26 Jan 2015 21:45:36 +0000
Subject: No generic version of DualPivotQuickSort
In-Reply-To: 
References: 
	
Message-ID: 

Allowing the user to specify that they want an unstable sort or that they want a sort that takes minimal extra space at the price of being unstable, seems reasonable. I understand making the default behavior for sorting a generic type be doing a stable sort.

Arrays.sort(T[] a) should lead to a stable sort.

Adding either Arrays.sort(T[] a, boolean stable) or Arrays.unstableSort(T[] a) seems useful and forces the user to explicitly say they want an unstable sort for the marginal performance gain and/or the reduced memory requirement.

From: Louis Wasserman [mailto:lowasser at google.com]
Sent: Monday, January 26, 2015 4:02 PM
To: Buis, Paul; core-libs-dev at openjdk.java.net
Subject: Re: No generic version of DualPivotQuickSort

My understanding was that the performance difference wasn't perceived to outweigh the user confusion risks associated with multiple sorting algorithm options. Right now, only one option is presented -- a stable sort -- and that algorithm is intended to be *nearly* as fast as an unstable sort, and unstable sorting algorithms are *subtly* unsuitable for some use cases in a way that might be hard for users to debug or realize why their code is buggy.

Is that an accurate assessment?

On Mon Jan 26 2015 at 12:55:18 PM Buis, Paul <00pebuis at bsu.edu> wrote:
The java.util.DualPivotQuicksort class implements sort() methods for the primitive types, has no methods that deal with generic arrays with methods like

static > void sort(T[] array, int iStart, int iEnd)

or

static > void sort(T[] a, int left, int right, T[] work, int workBase, int workLen)

Similarly, it contains no methods for Comparator-based sorting of generic arrays. Might it make sense to add such methods to DualPivotQucksort? The Arrays.sort() methods for generic arrays use TimSort which is likely to be slower. TimSort is stable and DualPivotQuickSort is not. Might it make sense to allow the user to pick a stable or an unstable version of a generic Arrays.sort() and use TimSort when stability is desired and DualPivotQuicksort when it is not?

From forax at univ-mlv.fr  Mon Jan 26 22:15:09 2015
From: forax at univ-mlv.fr (Remi Forax)
Date: Mon, 26 Jan 2015 23:15:09 +0100
Subject: RFR 8050819: Please add java.util.Stream.ofNullable(T object)
In-Reply-To: <54C68AAD.4060202@oracle.com>
References: <93EC212E-69DA-4874-8255-E73746404461@oracle.com>	<3F562D05-6BD1-4278-BB12-47CEEEFA5CF2@oracle.com>
	<54C68AAD.4060202@oracle.com>
Message-ID: <54C6BC6D.202@univ-mlv.fr>


On 01/26/2015 07:42 PM, Alan Bateman wrote:
> On 26/01/2015 18:32, Paul Sandoz wrote:
>> On Jan 20, 2015, at 7:05 PM, Paul Sandoz  wrote:
>>> Hi,
>>>
>>> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050819-Stream-ofNullable/webrev/ 
>>>
>>>
>>> Stream.ofNullable can make it easier to work fluently when streams 
>>> are produced, for example within flatMap.
>>>
>> So far there have been "null" reviews :-)
>>
> This looks okay to me, and the naming consistent with Optional.
>
> -Alan.

I'm not sure this method pull it's own weight,
it can be written like this:
   Optional.ofNullable(t).stream()

I'm pretty sure that the JIT will remove the creation of the 
intermediary Optional
and adding a method in the JDK that considers that null is an acceptable 
value is in my opinion the wrong signal to send.

R?mi




From john.r.rose at oracle.com  Tue Jan 27 00:04:03 2015
From: john.r.rose at oracle.com (John Rose)
Date: Mon, 26 Jan 2015 16:04:03 -0800
Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when
	GWT LambdaForms are shared
In-Reply-To: <54C66E4E.9050805@oracle.com>
References: <54B94766.2080102@oracle.com>
	<7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com>
	<54BEA7D7.6080008@oracle.com>
	<5BA1E369-ED87-4EBD-8408-B73B726D91BD@oracle.com>
	<54C66E4E.9050805@oracle.com>
Message-ID: <915998BE-25E9-4196-BAC7-FE5527E10F83@oracle.com>

On Jan 26, 2015, at 8:41 AM, Vladimir Ivanov  wrote:
> 
> What do you think about the following version?
> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.02 
> 
> As you suggested, I reified MHI::profileBranch on LambdaForm level and removed @LambdaForm.Shared. My main concern about removing @Sharen was that profile pollution can affect the code before profileBranch call (akin to 8068915 [1]) and it seems it's the case: Gbemu (at least) is sensitive to that change (there's a 10% difference in peak performance between @Shared and has_injected_profile()).
> 
> I can leave @Shared as is for now or remove it and work on the fix to the deoptimization counts pollution. What do you prefer?

Generic advice here:  It's better to leave it out, if in doubt.  If it has a real benefit, and we don't have time to make it clean, put it in and file a tracking bug to clean it up.

I re-read the change.  It's simpler and more coherent now.

I see one more issue which we should fix now, while we can.  It's the sort of thing which is hard to clean up later.

The two fields of the profileBranch array have obscure and inconsistent labelings.  It took me some hard thought and the inspection of three files to decide what "taken" and "not taken" mean in the C2 code that injects the profile.  The problem is that, when you look at profileBranch, all you see is an integer (boolean) argument and an array, and no clear indication about which array element corresponds to which argument value.  It's made worse by the fact that "taken" and "not taken" are not mentioned at all in the JDK code, which instead wires together the branches of selectAlternative without much comment.

My preferred formulation, for making things clearer:  Decouple the idea of branching from the idea of profile injection.  Name the intrinsic (yes, one more bikeshed color) "profileBoolean" (or even "injectBooleanProfile"), and use the natural indexing of the array:  0 (Java false) is a[0], and 1 (Java true) is a[1].  We might later extend this to work with "booleans" (more generally, small-integer flags), of more than two possible values, klasses, etc.

This line then goes away, and 'result' is used directly as the profile index:
+        int idx = result ? 0 : 1;

The ProfileBooleanNode should have an embedded (or simply indirect) array of ints which is a simple copy of the profile array, so there's no doubt about which count is which.

The parsing of the predicate that contains "profileBoolean" should probably be more robust, at least allowing for 'eq' and 'ne' versions of the test.  (C2 freely flips comparison senses, in various places.)  The check for Op_AndI must be more precise; make sure n->in(2) is a constant of the expected value (1).  The most robust way to handle it (but try this another time, I think) would be to make two temp copies of the predicate, substituting the occurrence of ProfileBoolean with '0' and '1', respectively; if they both fold to '0' and '1' or '1' and '0', then you take the indicated action.

I suggest putting the new code in Parse::dynamic_branch_prediction, which pattern-matches for injected profiles, into its own subroutine.  Maybe:
   bool use_mdo = true;
   if (has_injected_profile(btest, test, &taken, ¬_taken)) {
     use_mdo = false;
   }
   if (use_mdo) { ... old code

I see why you used the opposite order in the existing code:  It mirrors the order of the second and third arguments to selectAlternative.  But the JVM knows nothing about selectAlternative, so it's just confusing when reading the VM code to know which profile array element means what.

? John

P.S.  Long experience with byte-order bugs in HotSpot convinces me that if you are not scrupulously clear in your terms, when working with equal and opposite configuration pairs, you will have a long bug tail, especially if you have to maintain agreement about the configurations through many layers of software.  This is one of those cases.  The best chance to fix such bugs is not to allow them in the first place.  In the case of byte-order, we have "first" vs. "second", "MSB" vs. "LSB", and "high" vs. "low" parts of values, for values in memory and in registers, and all possible misunderstandings about them and their relation have probably happened and caused bugs.

From brian.burkhalter at oracle.com  Tue Jan 27 00:49:10 2015
From: brian.burkhalter at oracle.com (Brian Burkhalter)
Date: Mon, 26 Jan 2015 16:49:10 -0800
Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec
In-Reply-To: 
References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com>
	
	
	
	<1D643418-534A-4448-B229-50BCFEB64F2F@oracle.com>
	
	
Message-ID: 

Thanks, I think that reads better than the somewhat obscure verbiage we were discussing.

It is formalized here:

http://cr.openjdk.java.net/~bpb/8069269/webrev.01/

On Jan 24, 2015, at 9:55 AM, Martin Buchholz  wrote:

> How about:
> 
> --- a/src/java.base/share/classes/java/lang/System.java
> +++ b/src/java.base/share/classes/java/lang/System.java
> @@ -376,19 +376,16 @@



From martinrb at google.com  Tue Jan 27 00:59:16 2015
From: martinrb at google.com (Martin Buchholz)
Date: Mon, 26 Jan 2015 16:59:16 -0800
Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec
In-Reply-To: 
References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com>
	
	
	
	<1D643418-534A-4448-B229-50BCFEB64F2F@oracle.com>
	
	
	
Message-ID: 

Looks good to me!  The best feature of the new code sample is that it not
only prevents overflow in the elapsed time computation, but also overflow
when adding to the timeout, and the latter is far more likely in practice
since users have good reasons to pick Integer.MAX_VALUE and MIN_VALUE as
"infinite" timeouts.  Of course, we paranoid folks know better and think
MAX_VALUE / 2 is plenty infinite enough!

On Mon, Jan 26, 2015 at 4:49 PM, Brian Burkhalter <
brian.burkhalter at oracle.com> wrote:

> Thanks, I think that reads better than the somewhat obscure verbiage we
> were discussing.
>
> It is formalized here:
>
> http://cr.openjdk.java.net/~bpb/8069269/webrev.01/
>
> On Jan 24, 2015, at 9:55 AM, Martin Buchholz  wrote:
>
> How about:
>
> --- a/src/java.base/share/classes/java/lang/System.java
> +++ b/src/java.base/share/classes/java/lang/System.java
> @@ -376,19 +376,16 @@
>
>
>


From brian.burkhalter at oracle.com  Tue Jan 27 01:03:28 2015
From: brian.burkhalter at oracle.com (Brian Burkhalter)
Date: Mon, 26 Jan 2015 17:03:28 -0800
Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec
In-Reply-To: 
References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com>
	
	
	
	<1D643418-534A-4448-B229-50BCFEB64F2F@oracle.com>
	
	
	
	
Message-ID: 


On Jan 26, 2015, at 4:59 PM, Martin Buchholz  wrote:

> Looks good to me!  The best feature of the new code sample is that it not only prevents overflow in the elapsed time computation, but also overflow when adding to the timeout, and the latter is far more likely in practice since users have good reasons to pick Integer.MAX_VALUE and MIN_VALUE as "infinite" timeouts.

Good point.

> Of course, we paranoid folks know better and think MAX_VALUE / 2 is plenty infinite enough!

Quite so!

From brian.burkhalter at oracle.com  Tue Jan 27 01:40:14 2015
From: brian.burkhalter at oracle.com (Brian Burkhalter)
Date: Mon, 26 Jan 2015 17:40:14 -0800
Subject: [9] RFR of 6880737: (fs) FileLock constructors don't throw NPE if
	the channel argument is null
In-Reply-To: <54C62A83.5040900@oracle.com>
References: <3B73EA6D-467E-4035-B3DD-949129D07E99@oracle.com>
	<54BA105C.8030809@oracle.com>
	<4976B3C8-81F4-4CDE-A9CF-62DE7F38B8CF@oracle.com>
	<54C62A83.5040900@oracle.com>
Message-ID: 


On Jan 26, 2015, at 3:52 AM, Alan Bateman  wrote:

> On 23/01/2015 19:17, Brian Burkhalter wrote:
>> Here?s an updated patch including a (perhaps overkill) test:
>> 
>> http://cr.openjdk.java.net/~bpb/6880737/webrev.01/
>> 

Here?s another updated patch:

http://cr.openjdk.java.net/~bpb/6880737/webrev.02/

> Thanks, the null check is fine. The webrev still has the javadoc change, it looks okay but not needed.

I removed the javadoc change.

> The test needs to be moved into a FileLock sub-directory to keep the current organization consistent.

Done. I had been unsure of adding a directory for just one test.

> As it's become a unit test for FileLock then I think it would be a bit clearer as a TestNG test but not strictly required.

Left as-is.

> One thing that would be good is to switch the test to using FileChannel.open and to use try-with-resources. The catching of Exception can probably be dropped too, the test will fail if un-documented exceptions are throw.

Both done.

Thanks,

Brian

From mandy.chung at oracle.com  Tue Jan 27 04:37:33 2015
From: mandy.chung at oracle.com (Mandy Chung)
Date: Mon, 26 Jan 2015 20:37:33 -0800
Subject: Time to retire System.runFinalizersOnExit?
Message-ID: <54C7160D.4000802@oracle.com>

System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
and this method is inherently unsafe.  I am thinking to propose this method
in JDK 9 to throw UnsupportedOperationException.

I believe it's rare for existing applications using System.runFinalizationOnExit.
My analysis on Maven Central ~315K artifacts that show about ~15 unique
artifacts calling System.runFinalizationOnExit while they all come from
only 5 classes.

Any thought/feedback?

Mandy



From mandy.chung at oracle.com  Tue Jan 27 05:14:59 2015
From: mandy.chung at oracle.com (Mandy Chung)
Date: Mon, 26 Jan 2015 21:14:59 -0800
Subject: Review Request 8068937: jdeps shows "not found" if target class has
	no reference other than its own package
Message-ID: <54C71ED3.9080005@oracle.com>

http://cr.openjdk.java.net/~mchung/jdk9/webrevs/8068937/webrev.00/

This simple patch fixes a couple jdeps issues.  It makes sure a parsed 
class is added to the class list even if there is no reference other 
than its own package which is filtered by default, exclude itself from 
the summary dependences and also remove duplicates from the input paths 
and -classpath option.

Mandy


From kissziszi at gmail.com  Tue Jan 27 06:50:59 2015
From: kissziszi at gmail.com (Zoltan Sziladi)
Date: Mon, 26 Jan 2015 22:50:59 -0800
Subject: String.indexOf optimization
In-Reply-To: 
References: 
	
	<534D68B8.3040002@oracle.com>
	
	<5350E316.2050406@oracle.com>
	
	
	<54ABAAC2.2090502@redhat.com>
	
	
	
	
	
	
	
	
Message-ID: 

Hi Martin,

Nice catches on the cleanup!

By the way, can you tell me why you used named loops in your code? Isn't it
considered bad practice as it is almost like a goto statement? Couldn't we
refactor it in a way that we do not use named loops?

Also, if there is a for loop that has no start statement and no condition
(like this: for (;; i++) ), then is a for loop a good choice for that code?

I'm just wondering, maybe there are some points of view that I'm not
considering...

Regards,
Zoltan

On Sat, Jan 24, 2015 at 1:29 PM, Martin Buchholz 
wrote:

>
>
> On Sat, Jan 24, 2015 at 1:19 PM, Vitaly Davidovich 
> wrote:
>
>> Try -XX:+UnlockDiagnosticVMOptions -XX:DisableIntrinsic:_indexOf
>>
>
> Thank you very much!  Hard to find because -XX:+PrintFlagsFinal is
> insufficient - also needs  -XX:+UnlockDiagnosticVMOptions
>


From chris.hegarty at oracle.com  Tue Jan 27 09:00:04 2015
From: chris.hegarty at oracle.com (Chris Hegarty)
Date: Tue, 27 Jan 2015 09:00:04 +0000
Subject: [9] RFR of 8069269: (spec) Defect in the System.nanoTime spec
In-Reply-To: 
References: <63853D01-BAF6-4B5E-BA67-6239FB99D878@oracle.com>
	
	
	
	<1D643418-534A-4448-B229-50BCFEB64F2F@oracle.com>
	
	
	
Message-ID: <2499D5AC-1249-4A82-B030-611FC86F14FF@oracle.com>

On 27 Jan 2015, at 00:49, Brian Burkhalter  wrote:

> Thanks, I think that reads better than the somewhat obscure verbiage we were discussing.
> 
> It is formalized here:
> 
> http://cr.openjdk.java.net/~bpb/8069269/webrev.01/

This looks good to me Brian.

-Chris.

> On Jan 24, 2015, at 9:55 AM, Martin Buchholz  wrote:
> 
>> How about:
>> 
>> --- a/src/java.base/share/classes/java/lang/System.java
>> +++ b/src/java.base/share/classes/java/lang/System.java
>> @@ -376,19 +376,16 @@
> 



From frank.yuan at oracle.com  Tue Jan 27 09:09:16 2015
From: frank.yuan at oracle.com (Frank Yuan)
Date: Tue, 27 Jan 2015 17:09:16 +0800
Subject: Review request for JDK-8051710: Convert JAXP function tests:
	javax.xml.jaxp14.* to jtreg (testng) tests
Message-ID: <03a201d03a10$ef463d70$cdd2b850$@oracle.com>

Hi, Joe, Lance and All

We are working on moving internal jaxp functional tests to open jdk repo.
This is the jaxp14 suite. Would you please review these test?  Any comment
will be appreciated.

bug: https://bugs.openjdk.java.net/browse/JDK-8051710
webrev: http://cr.openjdk.java.net/~fyuan/8051710/webrev.00/


Thanks,

Frank



From frank.yuan at oracle.com  Tue Jan 27 09:17:12 2015
From: frank.yuan at oracle.com (Frank Yuan)
Date: Tue, 27 Jan 2015 17:17:12 +0800
Subject: Review request for JDK-8052401: JAXP function gap tests conversion
Message-ID: <03af01d03a12$0ac947d0$205bd770$@oracle.com>

Hi, Joe, Lance and All

We are working on moving internal jaxp functional tests to open jdk repo.
This is the gaptest suite. Would you please review these test?  Any comment
will be appreciated.

bug: https://bugs.openjdk.java.net/browse/JDK-8052401
webrev: http://cr.openjdk.java.net/~fyuan/8052401/webrev.00/


Thanks,

Frank



From aleksey.shipilev at oracle.com  Tue Jan 27 09:49:55 2015
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 27 Jan 2015 12:49:55 +0300
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C7160D.4000802@oracle.com>
References: <54C7160D.4000802@oracle.com>
Message-ID: <54C75F43.2020103@oracle.com>

On 27.01.2015 07:37, Mandy Chung wrote:
> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
> and this method is inherently unsafe.  I am thinking to propose this method
> in JDK 9 to throw UnsupportedOperationException.
> 
> I believe it's rare for existing applications using
> System.runFinalizationOnExit.
> My analysis on Maven Central ~315K artifacts that show about ~15 unique
> artifacts calling System.runFinalizationOnExit while they all come from
> only 5 classes.

I think while the use case for runFinalizationOnExit(true) is weird to
begin with, there seems to be a valid reason to enforce the finalizers
to run at the end, e.g. to properly free the *critical* underlying
resources. At this point rFOE(true) trick is, while unsafe, still
operational, right? If we are to remove rFOE(true) escape hatch, what do
we propose users to do?

Thanks,
-Aleksey.



From pike630 at hotmail.com  Tue Jan 27 11:47:07 2015
From: pike630 at hotmail.com (pike)
Date: Tue, 27 Jan 2015 04:47:07 -0700 (MST)
Subject: Useful message about NullPointerException
In-Reply-To: <20150126202011.000049a2.ecki@zusammenkunft.net>
References: <1421844308100-213240.post@n7.nabble.com>
	<20150126202011.000049a2.ecki@zusammenkunft.net>
Message-ID: <1422359227102-213842.post@n7.nabble.com>

Bernd Eckenfels-4 wrote
> Am Wed, 21 Jan 2015 05:45:08 -0700 (MST)
> schrieb pike <

> pike630@

> >:
> 
>> We frequently see NullPointerException in our logs. It's really a big
>> headache when we see a NullPointerException and it is encapsulated in
>> another exception as we don't know which object is null and it is
>> throwing an Exception. Is there any way we can get to know the object
>> type or the object variable name where the object is null and it is
>> throwing a NullPointerException? i.e, instead of just saying there is
>> a NullPointerException, can we add some friendly message? 
> 
> Note that if you keep the stack information in an exception it points
> (most of the time) exactly to the location where the null access
> happens.
> 
> I can imagine it is rather hard for the VM to add more informations.
> Your best bet is to avoid the NPEs and log the exceptions properly.
> 
> Gruss
> Bernd

This is acually not helpful in some situations. For example, as what kedar
has mentioned, if there is a call "a.getB().getC()" and a NPE  is thrown.
There might be two situations:
1. a is null;
2. a.getB() is null

So it would be better if there can be one friendly message which indicates
what is null.



--
View this message in context: http://openjdk.5641.n7.nabble.com/Useful-message-about-NullPointerException-tp213240p213842.html
Sent from the OpenJDK Core Libraries mailing list archive at Nabble.com.


From dl at cs.oswego.edu  Tue Jan 27 11:54:05 2015
From: dl at cs.oswego.edu (Doug Lea)
Date: Tue, 27 Jan 2015 06:54:05 -0500
Subject: java.util.DualPivotQuickSort does not guarantee NlogN time
In-Reply-To: 
References: 
Message-ID: <54C77C5D.2000501@cs.oswego.edu>

On 01/26/2015 03:05 PM, Buis, Paul wrote:
> DualPivotQuickSort is used by Arrays.sort() and provides NlogN average
> performance, but does not guarantee NlogN worst case performance. It would be
> relatively easy to incorporate the basic idea of IntroSort (see
> http://en.wikipedia.org/wiki/Introsort) to provide such a guarantee.

I was only tangentially involved with this, but I believe that
this was considered and rejected because the patterns leading
to quadratic performance practically never occur -- why slow
down an algorithm to handle (nearly) non-existent cases.
But if there were ever any evidence otherwise, this would be
worth considering.

BTW, over the past few years, there have been some academic papers
investigating why DPQS is even faster than expected. (See for example
http://epubs.siam.org/doi/pdf/10.1137/1.9781611973198.6 and
http://arxiv.org/pdf/1412.0193v1.pdf) It seems that cache locality
is one factor. This would not be helped if heapSort were
unnecessarily called, since it has terrible cache locality.

-Doug


From pike630 at hotmail.com  Tue Jan 27 11:51:51 2015
From: pike630 at hotmail.com (pike)
Date: Tue, 27 Jan 2015 04:51:51 -0700 (MST)
Subject: Useful message about NullPointerException
In-Reply-To: 
References: <1421844308100-213240.post@n7.nabble.com>
	<20150126202011.000049a2.ecki@zusammenkunft.net>
	
Message-ID: <1422359511452-213845.post@n7.nabble.com>

kedar mhaswade wrote
> On Mon, Jan 26, 2015 at 11:20 AM, Bernd Eckenfels <

> ecki@

> >
> wrote:
> 
>> Am Wed, 21 Jan 2015 05:45:08 -0700 (MST)
>> schrieb pike <

> pike630@

> >:
>>
>> > We frequently see NullPointerException in our logs. It's really a big
>> > headache when we see a NullPointerException and it is encapsulated in
>> > another exception as we don't know which object is null and it is
>> > throwing an Exception. Is there any way we can get to know the object
>> > type or the object variable name where the object is null and it is
>> > throwing a NullPointerException? i.e, instead of just saying there is
>> > a NullPointerException, can we add some friendly message?
>>
>> Note that if you keep the stack information in an exception it points
>> (most of the time) exactly to the location where the null access
>> happens.
>>
>>
> ?Maybe the objects in question are not talking only to their immediate
> friends <https://en.wikipedia.org/wiki/Law_of_Demeter>?
> This happens when we tend to do something like *a.getB().getC()* and we
> can't tell by looking at the NPE stack trace (which works on lines of
> source code) if *a* was null or what *a.getB()* returned was null.
> 
> I can imagine it is rather hard for the VM to add more informations.
>> Your best bet is to avoid the NPEs and log the exceptions properly.
> 
> 
>> Gruss
>> Bernd
>>

Hmm, based on the JVM specification, there is Descriptor structure which can
be used to describe the name and type of a field or method in a class. I
think, the JVM can use this info to display the type of the object which is
null when NPE is thrown.

Furthermore, I think the constant pool also includes all the information
about the fields in a class. They can be used as well.



--
View this message in context: http://openjdk.5641.n7.nabble.com/Useful-message-about-NullPointerException-tp213240p213845.html
Sent from the OpenJDK Core Libraries mailing list archive at Nabble.com.


From Alan.Bateman at oracle.com  Tue Jan 27 12:17:42 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Tue, 27 Jan 2015 12:17:42 +0000
Subject: Review Request 8068937: jdeps shows "not found" if target class
	has no reference other than its own package
In-Reply-To: <54C71ED3.9080005@oracle.com>
References: <54C71ED3.9080005@oracle.com>
Message-ID: <54C781E6.4040309@oracle.com>

On 27/01/2015 05:14, Mandy Chung wrote:
> http://cr.openjdk.java.net/~mchung/jdk9/webrevs/8068937/webrev.00/
>
> This simple patch fixes a couple jdeps issues.  It makes sure a parsed 
> class is added to the class list even if there is no reference other 
> than its own package which is filtered by default, exclude itself from 
> the summary dependences and also remove duplicates from the input 
> paths and -classpath option.
This looks okay to me.

-Alan.


From dl at cs.oswego.edu  Tue Jan 27 12:18:38 2015
From: dl at cs.oswego.edu (Doug Lea)
Date: Tue, 27 Jan 2015 07:18:38 -0500
Subject: No generic version of DualPivotQuickSort
In-Reply-To: 
References: 
Message-ID: <54C7821E.7050707@cs.oswego.edu>

On 01/26/2015 03:54 PM, Buis, Paul wrote:
> The java.util.DualPivotQuicksort class implements sort() methods for the
> primitive types, has no methods that deal with generic arrays with methods
> like
>
> static > void sort(T[] array, int iStart, int
> iEnd)
>
> Similarly, it contains no methods for Comparator-based sorting of generic
> arrays. Might it make sense to add such methods to DualPivotQucksort? The
> Arrays.sort() methods for generic arrays use TimSort which is likely to be
> slower. TimSort is stable and DualPivotQuickSort is not. Might it make sense
> to allow the user to pick a stable or an unstable version of a generic
> Arrays.sort() and use TimSort when stability is desired and
> DualPivotQuicksort when it is not?
>

This was considered, including when introducing parallelSort
for which ensuring stability requires a bunch of precautions.
But after putting these into place, there was little or no
benefit over non-stable forms for parallel case (which requires
allocating a workspace of size N anyway.) And for the non-parallel
case, TimSort does not do very much allocation, and is faster
for the majority of cases seen in practice. (There is a big
suite of test cases around, including I think some in jtreg.)
So the empirical question remains whether there are enough cases
that would benefit to warrant adding code. Efforts to find this
out would be welcome.

-Doug



From Alan.Bateman at oracle.com  Tue Jan 27 12:19:04 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Tue, 27 Jan 2015 12:19:04 +0000
Subject: [9] RFR of 6880737: (fs) FileLock constructors don't throw NPE
	if the channel argument is null
In-Reply-To: 
References: <3B73EA6D-467E-4035-B3DD-949129D07E99@oracle.com>
	<54BA105C.8030809@oracle.com>
	<4976B3C8-81F4-4CDE-A9CF-62DE7F38B8CF@oracle.com>
	<54C62A83.5040900@oracle.com>
	
Message-ID: <54C78238.6020003@oracle.com>

On 27/01/2015 01:40, Brian Burkhalter wrote:
>
> Here?s another updated patch:
>
> http://cr.openjdk.java.net/~bpb/6880737/webrev.02/ 
> 
Thanks for the update and moving the test to the right place. I think 
this is okay for now.

-Alan


From peter.levart at gmail.com  Tue Jan 27 12:54:47 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Tue, 27 Jan 2015 13:54:47 +0100
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C75F43.2020103@oracle.com>
References: <54C7160D.4000802@oracle.com> <54C75F43.2020103@oracle.com>
Message-ID: <54C78A97.70001@gmail.com>

On 01/27/2015 10:49 AM, Aleksey Shipilev wrote:
> On 27.01.2015 07:37, Mandy Chung wrote:
>> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
>> and this method is inherently unsafe.  I am thinking to propose this method
>> in JDK 9 to throw UnsupportedOperationException.
>>
>> I believe it's rare for existing applications using
>> System.runFinalizationOnExit.
>> My analysis on Maven Central ~315K artifacts that show about ~15 unique
>> artifacts calling System.runFinalizationOnExit while they all come from
>> only 5 classes.
> I think while the use case for runFinalizationOnExit(true) is weird to
> begin with, there seems to be a valid reason to enforce the finalizers
> to run at the end, e.g. to properly free the *critical* underlying
> resources. At this point rFOE(true) trick is, while unsafe, still
> operational, right? If we are to remove rFOE(true) escape hatch, what do
> we propose users to do?
>
> Thanks,
> -Aleksey.
>
I think that finalizers as designed can not be used to properly shutdown 
the program running in a VM as a whole, since they are inherently 
limited to private iter-independent resources like primitive system 
resources (file handles, memory handles, etc...). And those don't need 
explicit "proper" shutdown as OS usually takes care of them when the 
process ends. Admittedly this does not happen if VM is used embedded in 
a process that does not end when the VM exits.

A poor-man's escape hatch is a shutdown hook that calls 
System.runFinalization(). Which might interfere with other shutdown 
hooks that run concurrently (runFinalizersOnExit runs finalizers after 
all shutdown hooks are finished).

To support orderly shutdown of a program running in a VM, other non-JDK 
solutions are available (app containers like Spring, etc...).

Regards, Peter



From cnewland at chrisnewland.com  Tue Jan 27 12:58:15 2015
From: cnewland at chrisnewland.com (Chris Newland)
Date: Tue, 27 Jan 2015 12:58:15 -0000
Subject: Useful message about NullPointerException
In-Reply-To: <1422359227102-213842.post@n7.nabble.com>
References: <1421844308100-213240.post@n7.nabble.com>
	<20150126202011.000049a2.ecki@zusammenkunft.net>
	<1422359227102-213842.post@n7.nabble.com>
Message-ID: 

Hi,

If the contract for a method allows a null return you should really handle
it defensively. Null checks are cheap in terms of performance.

If you *have* to chain method calls like that then you could drop each
call onto a new source line and the stack trace will show you which call
was on the null object:

public class NullTest
{
    class B {
        private String c = null;
        public String getC() { return c; }
    }

    class A {
        private B b = null;
        public B getB() { return b; }
    }

    public NullTest() {
        new A()
        .getB()
        .getC() // NPE here indicates getB() returned null
        .length();
    }

    public static void main(String[] args) {
        new NullTest();
    }
}

Regards,

Chris
@chriswhocodes

On Tue, January 27, 2015 11:47, pike wrote:
> Bernd Eckenfels-4 wrote
>
>> Am Wed, 21 Jan 2015 05:45:08 -0700 (MST)
>> schrieb pike <
>
>> pike630@
>
>> >:
>>
>>
>>> We frequently see NullPointerException in our logs. It's really a big
>>>  headache when we see a NullPointerException and it is encapsulated
>>> in another exception as we don't know which object is null and it is
>>> throwing an Exception. Is there any way we can get to know the object
>>>  type or the object variable name where the object is null and it is
>>> throwing a NullPointerException? i.e, instead of just saying there is
>>>  a NullPointerException, can we add some friendly message?
>>
>> Note that if you keep the stack information in an exception it points
>> (most of the time) exactly to the location where the null access
>> happens.
>>
>> I can imagine it is rather hard for the VM to add more informations.
>> Your best bet is to avoid the NPEs and log the exceptions properly.
>>
>>
>> Gruss
>> Bernd
>>
>
> This is acually not helpful in some situations. For example, as what
> kedar has mentioned, if there is a call "a.getB().getC()" and a NPE  is
> thrown. There might be two situations:
> 1. a is null;
> 2. a.getB() is null
>
>
> So it would be better if there can be one friendly message which
> indicates what is null.
>
>
>
> --
> View this message in context:
> http://openjdk.5641.n7.nabble.com/Useful-message-about-NullPointerExcepti
> on-tp213240p213842.html Sent from the OpenJDK Core Libraries mailing list
> archive at Nabble.com.
>




From pike630 at hotmail.com  Tue Jan 27 13:00:56 2015
From: pike630 at hotmail.com (pike)
Date: Tue, 27 Jan 2015 06:00:56 -0700 (MST)
Subject: Useful message about NullPointerException
In-Reply-To: 
References: <1421844308100-213240.post@n7.nabble.com>
	<20150126202011.000049a2.ecki@zusammenkunft.net>
	<1422359227102-213842.post@n7.nabble.com>
	
Message-ID: <1422363656532-213855.post@n7.nabble.com>

Chris Newland wrote
> Hi,
> 
> If the contract for a method allows a null return you should really handle
> it defensively. Null checks are cheap in terms of performance.
> 
> If you *have* to chain method calls like that then you could drop each
> call onto a new source line and the stack trace will show you which call
> was on the null object:
> 
> public class NullTest
> {
>     class B {
>         private String c = null;
>         public String getC() { return c; }
>     }
> 
>     class A {
>         private B b = null;
>         public B getB() { return b; }
>     }
> 
>     public NullTest() {
>         new A()
>         .getB()
>         .getC() // NPE here indicates getB() returned null
>         .length();
>     }
> 
>     public static void main(String[] args) {
>         new NullTest();
>     }
> }
> 
> Regards,
> 
> Chris
> @chriswhocodes
> 
> On Tue, January 27, 2015 11:47, pike wrote:
>> Bernd Eckenfels-4 wrote
>>
>>> Am Wed, 21 Jan 2015 05:45:08 -0700 (MST)
>>> schrieb pike <
>>
>>> pike630@
>>
>>> >:
>>>
>>>
>>>> We frequently see NullPointerException in our logs. It's really a big
>>>>  headache when we see a NullPointerException and it is encapsulated
>>>> in another exception as we don't know which object is null and it is
>>>> throwing an Exception. Is there any way we can get to know the object
>>>>  type or the object variable name where the object is null and it is
>>>> throwing a NullPointerException? i.e, instead of just saying there is
>>>>  a NullPointerException, can we add some friendly message?
>>>
>>> Note that if you keep the stack information in an exception it points
>>> (most of the time) exactly to the location where the null access
>>> happens.
>>>
>>> I can imagine it is rather hard for the VM to add more informations.
>>> Your best bet is to avoid the NPEs and log the exceptions properly.
>>>
>>>
>>> Gruss
>>> Bernd
>>>
>>
>> This is acually not helpful in some situations. For example, as what
>> kedar has mentioned, if there is a call "a.getB().getC()" and a NPE  is
>> thrown. There might be two situations:
>> 1. a is null;
>> 2. a.getB() is null
>>
>>
>> So it would be better if there can be one friendly message which
>> indicates what is null.
>>
>>
>>
>> --
>> View this message in context:
>> http://openjdk.5641.n7.nabble.com/Useful-message-about-NullPointerExcepti
>> on-tp213240p213842.html Sent from the OpenJDK Core Libraries mailing list
>> archive at Nabble.com.
>>

You are advising we should stick to the best coding style. But many times,
not everyone can stick to this. If you are handling something written by
others and you have no access to the source, you would appreciate if NPE can
provide more friendly message like "The object reference a.getB() is
null"... 



-----
Programmer
--
View this message in context: http://openjdk.5641.n7.nabble.com/Useful-message-about-NullPointerException-tp213240p213855.html
Sent from the OpenJDK Core Libraries mailing list archive at Nabble.com.


From fweimer at redhat.com  Tue Jan 27 13:14:44 2015
From: fweimer at redhat.com (Florian Weimer)
Date: Tue, 27 Jan 2015 14:14:44 +0100
Subject: Useful message about NullPointerException
In-Reply-To: <1421844308100-213240.post@n7.nabble.com>
References: <1421844308100-213240.post@n7.nabble.com>
Message-ID: <54C78F44.1030104@redhat.com>

On 01/21/2015 01:45 PM, pike wrote:
> We frequently see NullPointerException in our logs. It's really a big
> headache when we see a NullPointerException and it is encapsulated in
> another exception as we don't know which object is null and it is throwing
> an Exception. Is there any way we can get to know the object type or the
> object variable name where the object is null and it is throwing a
> NullPointerException?

The line number gives you the position in the source code, and from
that, you can usually figure out the static type.  If this is not
helpful in your case, you need to say why (no debugging information?
multiple candidates per line?).

The dynamic type is a different matter though, because null has no
specific type at run time.  It may be possible to provide type
information in theory, at a cost, but this would best be prototyped
through byte code rewriting.  Nullable annotations would also help to
pin-point location of the first leak, and you could record that
(including a stack trace) if you want something really fancy.  Whether
it is helpful for legacy code, I don't know.  There should be some
research projects out there covering this area.

-- 
Florian Weimer / Red Hat Product Security


From 00pebuis at bsu.edu  Tue Jan 27 13:44:54 2015
From: 00pebuis at bsu.edu (Buis, Paul)
Date: Tue, 27 Jan 2015 13:44:54 +0000
Subject: java.util.DualPivotQuickSort does not guarantee NlogN time
Message-ID: 

The slowdown would be passing a single extra integer parameter, decrementing it and comparing it to zero at the beginning of the function.

With an initial value of 4*logN, heapsort would hardly ever be called in practice, only after a long series of unfortundate pivot choices that indicate the data was pathological. The original paper of IntroSort suggested an initial value of 2*logN which triggers the invocation of heapsort just often enough to be noticeable. The proposed threshold should make invocation of heapsort so rare as to not be noticeable.

As the quicksort makes recursive calls, it starts piling on stack space. In a pathological case, taking quadratic time, it may also require linear stack space and throw an exception. Using code that is robust against rare problematic would seem to be appropriate for library code.

Paul Buis


From kedar.mhaswade at gmail.com  Tue Jan 27 14:34:28 2015
From: kedar.mhaswade at gmail.com (kedar mhaswade)
Date: Tue, 27 Jan 2015 06:34:28 -0800
Subject: Useful message about NullPointerException
In-Reply-To: <54C78F44.1030104@redhat.com>
References: <1421844308100-213240.post@n7.nabble.com>
	<54C78F44.1030104@redhat.com>
Message-ID: 

When the JVM executes instructions like getfield
,
getstatic, invokevirtual etc. with *objref* on the operand stack and if
*objref* is null, an NPE is thrown. It appears that the JVM could tell us
more about which *objref* was null at run-time. Candidate for an RFE?

That aside, (and Chris's trick is nice), but if you have no access to the
source for the offending code, life is hard in general, isn't it? Because
if you can't have control over the source, making that source run on a
platform where such an RFE would be perhaps fixed (a future release of the
JDK) would be even harder, no?

On Tue, Jan 27, 2015 at 5:14 AM, Florian Weimer  wrote:

> On 01/21/2015 01:45 PM, pike wrote:
> > We frequently see NullPointerException in our logs. It's really a big
> > headache when we see a NullPointerException and it is encapsulated in
> > another exception as we don't know which object is null and it is
> throwing
> > an Exception. Is there any way we can get to know the object type or the
> > object variable name where the object is null and it is throwing a
> > NullPointerException?
>
> The line number gives you the position in the source code, and from
> that, you can usually figure out the static type.  If this is not
> helpful in your case, you need to say why (no debugging information?
> multiple candidates per line?).
>
> The dynamic type is a different matter though, because null has no
> specific type at run time.  It may be possible to provide type
> information in theory, at a cost, but this would best be prototyped
> through byte code rewriting.  Nullable annotations would also help to
> pin-point location of the first leak, and you could record that
> (including a stack trace) if you want something really fancy.  Whether
> it is helpful for legacy code, I don't know.  There should be some
> research projects out there covering this area.
>
> --
> Florian Weimer / Red Hat Product Security
>


From peter.levart at gmail.com  Tue Jan 27 14:48:46 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Tue, 27 Jan 2015 15:48:46 +0100
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C78A97.70001@gmail.com>
References: <54C7160D.4000802@oracle.com> <54C75F43.2020103@oracle.com>
	<54C78A97.70001@gmail.com>
Message-ID: <54C7A54E.1040501@gmail.com>

On 01/27/2015 01:54 PM, Peter Levart wrote:
> A poor-man's escape hatch is a shutdown hook that calls 
> System.runFinalization(). Which might interfere with other shutdown 
> hooks that run concurrently (runFinalizersOnExit runs finalizers after 
> all shutdown hooks are finished). 

Not really. This only runs finalizers for Objects pending finalization, 
while runFinalizersOnExit runs it for all Objects with finalize() 
methods that have not been invoked yet. A big difference. So there's no 
alternative if this method is removed.

Peter



From pike630 at hotmail.com  Tue Jan 27 14:45:31 2015
From: pike630 at hotmail.com (pike)
Date: Tue, 27 Jan 2015 07:45:31 -0700 (MST)
Subject: Useful message about NullPointerException
In-Reply-To: 
References: <1421844308100-213240.post@n7.nabble.com>
	<54C78F44.1030104@redhat.com>
	
Message-ID: <1422369931858-213867.post@n7.nabble.com>

kedar mhaswade wrote
> When the JVM executes instructions like getfield
> <http://docs.oracle.com/javase/specs/jvms/se7/html/jvms-6.html#jvms-6.5.getfield>,
> getstatic, invokevirtual etc. with *objref* on the operand stack and if
> *objref* is null, an NPE is thrown. It appears that the JVM could tell us
> more about which *objref* was null at run-time. Candidate for an RFE?

Yes, agree with you on this. Actually the JVM has some information about
which reference is null when the NPE is thrown. It's reasonable if this kind
of information can be provided when NPE is thrown. This will save much time
for debugging.


kedar mhaswade wrote
> That aside, (and Chris's trick is nice), but if you have no access to the
> source for the offending code, life is hard in general, isn't it? Because
> if you can't have control over the source, making that source run on a
> platform where such an RFE would be perhaps fixed (a future release of the
> JDK) would be even harder, no? 

It's quite common that someone may have no access to the source code which
throws the NPE. For example, API developers may often not be able to access
user's source code, the only thing they can get are a bunch of logs.



-----
Programmer
--
View this message in context: http://openjdk.5641.n7.nabble.com/Useful-message-about-NullPointerException-tp213240p213867.html
Sent from the OpenJDK Core Libraries mailing list archive at Nabble.com.


From sean.coffey at oracle.com  Tue Jan 27 14:57:52 2015
From: sean.coffey at oracle.com (=?UTF-8?B?U2XDoW4gQ29mZmV5?=)
Date: Tue, 27 Jan 2015 14:57:52 +0000
Subject: Useful message about NullPointerException
In-Reply-To: 
References: <1421844308100-213240.post@n7.nabble.com>	<54C78F44.1030104@redhat.com>
	
Message-ID: <54C7A770.8080707@oracle.com>

Adding hotspot-dev to this mail thread also as it's relevant to hotspot.
(complete thread at 
http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-January/031015.html)

As one who often has to dig through application logs and JDK issues, I 
think this would certainly be a useful addition to the NPE handling 
process. If the VM has extra info on the exact object that caused the 
NPE, it should be output in that exception. "a.getB().getC()" is a good 
example of the different code paths one has to go down to determine 
where the NPE could have arisen from.

regards,
Sean.

On 27/01/2015 14:34, kedar mhaswade wrote:
> When the JVM executes instructions like getfield
> ,
> getstatic, invokevirtual etc. with *objref* on the operand stack and if
> *objref* is null, an NPE is thrown. It appears that the JVM could tell us
> more about which *objref* was null at run-time. Candidate for an RFE?
>
> That aside, (and Chris's trick is nice), but if you have no access to the
> source for the offending code, life is hard in general, isn't it? Because
> if you can't have control over the source, making that source run on a
> platform where such an RFE would be perhaps fixed (a future release of the
> JDK) would be even harder, no?
>
> On Tue, Jan 27, 2015 at 5:14 AM, Florian Weimer  wrote:
>
>> On 01/21/2015 01:45 PM, pike wrote:
>>> We frequently see NullPointerException in our logs. It's really a big
>>> headache when we see a NullPointerException and it is encapsulated in
>>> another exception as we don't know which object is null and it is
>> throwing
>>> an Exception. Is there any way we can get to know the object type or the
>>> object variable name where the object is null and it is throwing a
>>> NullPointerException?
>> The line number gives you the position in the source code, and from
>> that, you can usually figure out the static type.  If this is not
>> helpful in your case, you need to say why (no debugging information?
>> multiple candidates per line?).
>>
>> The dynamic type is a different matter though, because null has no
>> specific type at run time.  It may be possible to provide type
>> information in theory, at a cost, but this would best be prototyped
>> through byte code rewriting.  Nullable annotations would also help to
>> pin-point location of the first leak, and you could record that
>> (including a stack trace) if you want something really fancy.  Whether
>> it is helpful for legacy code, I don't know.  There should be some
>> research projects out there covering this area.
>>
>> --
>> Florian Weimer / Red Hat Product Security
>>



From fweimer at redhat.com  Tue Jan 27 15:04:22 2015
From: fweimer at redhat.com (Florian Weimer)
Date: Tue, 27 Jan 2015 16:04:22 +0100
Subject: Useful message about NullPointerException
In-Reply-To: <54C7A770.8080707@oracle.com>
References: <1421844308100-213240.post@n7.nabble.com>	<54C78F44.1030104@redhat.com>
	
	<54C7A770.8080707@oracle.com>
Message-ID: <54C7A8F6.5070606@redhat.com>

On 01/27/2015 03:57 PM, Se?n Coffey wrote:
> Adding hotspot-dev to this mail thread also as it's relevant to hotspot.
> (complete thread at
> http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-January/031015.html)
> 
> 
> As one who often has to dig through application logs and JDK issues, I
> think this would certainly be a useful addition to the NPE handling
> process. If the VM has extra info on the exact object that caused the
> NPE, it should be output in that exception. "a.getB().getC()" is a good
> example of the different code paths one has to go down to determine
> where the NPE could have arisen from.

Maybe it's possible to add a byte code office to the exception it self,
or the stack trace?  Especially the former wouldn't need much additional
allocation.

-- 
Florian Weimer / Red Hat Product Security


From lev.priima at oracle.com  Tue Jan 27 15:43:18 2015
From: lev.priima at oracle.com (Lev Priima)
Date: Tue, 27 Jan 2015 18:43:18 +0300
Subject: Q: 8071326: ThreadPoolExecutor in endless thread creation loop if
	workQueue.take() throws RuntimeException
Message-ID: <54C7B216.90404@oracle.com>

Using TPE w/ custom BlockingQueue and if RuntimeException happens in 
blocking BlockingQueue.take() method then this code

new ThreadPoolExecutor(1, 1, 0, TimeUnit.NANOSECONDS,
     new ArrayBlockingQueue(1) {
         public Runnable take() throws InterruptedException {
             throw new RuntimeException();
         }
     }
).prestartAllCoreThreads();

has an unbounded thread creation loop.

As a result there are many created unbounded threads in RUNNING state 
after printing stack trace to stderr by default 
UncaughtExceptionHandler. And these thread will be cleaned only when 
whole TPE finished.

Is this "Not an Issue"?

-- 
Lev



From peter.levart at gmail.com  Tue Jan 27 16:01:23 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Tue, 27 Jan 2015 17:01:23 +0100
Subject: Useful message about NullPointerException
In-Reply-To: 
References: <1421844308100-213240.post@n7.nabble.com>	<54C78F44.1030104@redhat.com>
	
Message-ID: <54C7B653.1070509@gmail.com>

On 01/27/2015 03:34 PM, kedar mhaswade wrote:
> When the JVM executes instructions like getfield
> ,
> getstatic, invokevirtual etc. with *objref* on the operand stack and if
> *objref* is null, an NPE is thrown. It appears that the JVM could tell us
> more about which *objref* was null at run-time. Candidate for an RFE?

In general it is hard to deduce the meaningfull source of null *objref* 
on the operand stack by analyzing the surrounding bytecodes. It could be 
a result of complex logic executed by bytecodes. Imagine the following:

int length(boolean first, String s1, String s2) {
     return (first ? s1 : s2).length();
}

...the analysis would have to trace the live execution so that it could 
be rolled-back to the meaningful source of null *objref*.

All VM might semi-realistically do is report the action VM was trying to 
perform when it dereferenced a null *objref*. Like 
"NullPointerException: while invoking method Xxxx.yyyy on a null 
target", or "NullPointerException: while de-referencing instance field 
Xxxx.yyyy of a null reference"

But what does this help if you don't have access to sources? Might be a 
hint, but not much.

If you have access to sources, then perhaps an easier solution would be 
for stack traces to include column number in addition to line number of 
location in source that resulted in bytecodes that include the one that 
triggered the NPE.

There is already a RFE for that:

     https://bugs.openjdk.java.net/browse/JDK-8020204

It seems that javac part is already there. The VM part and public API 
part (StackTraceElement) is not, though.

Regards, Peter

>
> That aside, (and Chris's trick is nice), but if you have no access to the
> source for the offending code, life is hard in general, isn't it? Because
> if you can't have control over the source, making that source run on a
> platform where such an RFE would be perhaps fixed (a future release of the
> JDK) would be even harder, no?
>
> On Tue, Jan 27, 2015 at 5:14 AM, Florian Weimer  wrote:
>
>> On 01/21/2015 01:45 PM, pike wrote:
>>> We frequently see NullPointerException in our logs. It's really a big
>>> headache when we see a NullPointerException and it is encapsulated in
>>> another exception as we don't know which object is null and it is
>> throwing
>>> an Exception. Is there any way we can get to know the object type or the
>>> object variable name where the object is null and it is throwing a
>>> NullPointerException?
>> The line number gives you the position in the source code, and from
>> that, you can usually figure out the static type.  If this is not
>> helpful in your case, you need to say why (no debugging information?
>> multiple candidates per line?).
>>
>> The dynamic type is a different matter though, because null has no
>> specific type at run time.  It may be possible to provide type
>> information in theory, at a cost, but this would best be prototyped
>> through byte code rewriting.  Nullable annotations would also help to
>> pin-point location of the first leak, and you could record that
>> (including a stack trace) if you want something really fancy.  Whether
>> it is helpful for legacy code, I don't know.  There should be some
>> research projects out there covering this area.
>>
>> --
>> Florian Weimer / Red Hat Product Security
>>



From vladimir.x.ivanov at oracle.com  Tue Jan 27 16:05:19 2015
From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov)
Date: Tue, 27 Jan 2015 19:05:19 +0300
Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when
	GWT LambdaForms are shared
In-Reply-To: <915998BE-25E9-4196-BAC7-FE5527E10F83@oracle.com>
References: <54B94766.2080102@oracle.com>
	<7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com>
	<54BEA7D7.6080008@oracle.com>
	<5BA1E369-ED87-4EBD-8408-B73B726D91BD@oracle.com>
	<54C66E4E.9050805@oracle.com>
	<915998BE-25E9-4196-BAC7-FE5527E10F83@oracle.com>
Message-ID: <54C7B73F.50404@oracle.com>

Thanks for the feedback, John!

Updated webrev:
http://cr.openjdk.java.net/~vlivanov/8063137/webrev.03/jdk
http://cr.openjdk.java.net/~vlivanov/8063137/webrev.03/hotspot

Changes:
   - renamed MHI::profileBranch to MHI::profileBoolean, and 
ProfileBranchNode to ProfileBooleanNode;
   - restructured profile layout ([0] => false_cnt, [1] => true_cnt)
   - factored out profile injection in a separate function 
(has_injected_profile() in parse2.cpp)
   - ProfileBooleanNode stores true/false counts instead of 
taken/not_taken counts
   - matching from value counts to taken/not_taken happens in 
has_injected_profile();
   - added BoolTest::ne support
   - sharpened test for AndI case: now it checks AndI (ProfileBoolean) 
(ConI 1) shape

Best regards,
Vladimir Ivanov

On 1/27/15 3:04 AM, John Rose wrote:
> On Jan 26, 2015, at 8:41 AM, Vladimir Ivanov
> > wrote:
>>
>> What do you think about the following version?
>> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.02
>>
>> As you suggested, I reified MHI::profileBranch on LambdaForm level and
>> removed @LambdaForm.Shared. My main concern about removing @Sharen was
>> that profile pollution can affect the code before profileBranch call
>> (akin to 8068915 [1]) and it seems it's the case: Gbemu (at least) is
>> sensitive to that change (there's a 10% difference in peak performance
>> between @Shared and has_injected_profile()).
>>
>> I can leave @Shared as is for now or remove it and work on the fix to
>> the deoptimization counts pollution. What do you prefer?
>
> Generic advice here:  It's better to leave it out, if in doubt.  If it
> has a real benefit, and we don't have time to make it clean, put it in
> and file a tracking bug to clean it up.
>
> I re-read the change.  It's simpler and more coherent now.
>
> I see one more issue which we should fix now, while we can.  It's the
> sort of thing which is hard to clean up later.
>
> The two fields of the profileBranch array have obscure and inconsistent
> labelings.  It took me some hard thought and the inspection of three
> files to decide what "taken" and "not taken" mean in the C2 code that
> injects the profile.  The problem is that, when you look at
> profileBranch, all you see is an integer (boolean) argument and an
> array, and no clear indication about which array element corresponds to
> which argument value.  It's made worse by the fact that "taken" and "not
> taken" are not mentioned at all in the JDK code, which instead wires
> together the branches of selectAlternative without much comment.
>
> My preferred formulation, for making things clearer:  Decouple the idea
> of branching from the idea of profile injection.  Name the intrinsic
> (yes, one more bikeshed color) "profileBoolean" (or even
> "injectBooleanProfile"), and use the natural indexing of the array:  0
> (Java false) is a[0], and 1 (Java true) is a[1].  We might later extend
> this to work with "booleans" (more generally, small-integer flags), of
> more than two possible values, klasses, etc.
>
> This line then goes away, and 'result' is used directly as the profile
> index:
> +        int idx = result ? 0 : 1;
>
> The ProfileBooleanNode should have an embedded (or simply indirect)
> array of ints which is a simple copy of the profile array, so there's no
> doubt about which count is which.
>
> The parsing of the predicate that contains "profileBoolean" should
> probably be more robust, at least allowing for 'eq' and 'ne' versions of
> the test.  (C2 freely flips comparison senses, in various places.)  The
> check for Op_AndI must be more precise; make sure n->in(2) is a constant
> of the expected value (1).  The most robust way to handle it (but try
> this another time, I think) would be to make two temp copies of the
> predicate, substituting the occurrence of ProfileBoolean with '0' and
> '1', respectively; if they both fold to '0' and '1' or '1' and '0', then
> you take the indicated action.
>
> I suggest putting the new code in Parse::dynamic_branch_prediction,
> which pattern-matches for injected profiles, into its own subroutine.
>   Maybe:
>     bool use_mdo = true;
>     if (has_injected_profile(btest, test, &taken, ¬_taken)) {
>       use_mdo = false;
>     }
>     if (use_mdo) { ... old code
>
> I see why you used the opposite order in the existing code:  It mirrors
> the order of the second and third arguments to selectAlternative.  But
> the JVM knows nothing about selectAlternative, so it's just confusing
> when reading the VM code to know which profile array element means what.
>
> ? John
>
> P.S.  Long experience with byte-order bugs in HotSpot convinces me that
> if you are not scrupulously clear in your terms, when working with equal
> and opposite configuration pairs, you will have a long bug tail,
> especially if you have to maintain agreement about the configurations
> through many layers of software.  This is one of those cases.  The best
> chance to fix such bugs is not to allow them in the first place.  In the
> case of byte-order, we have "first" vs. "second", "MSB" vs. "LSB", and
> "high" vs. "low" parts of values, for values in memory and in registers,
> and all possible misunderstandings about them and their relation have
> probably happened and caused bugs.


From brian.burkhalter at oracle.com  Tue Jan 27 17:00:10 2015
From: brian.burkhalter at oracle.com (Brian Burkhalter)
Date: Tue, 27 Jan 2015 09:00:10 -0800
Subject: [9] RFR of 6880737: (fs) FileLock constructors don't throw NPE if
	the channel argument is null
In-Reply-To: <54C78238.6020003@oracle.com>
References: <3B73EA6D-467E-4035-B3DD-949129D07E99@oracle.com>
	<54BA105C.8030809@oracle.com>
	<4976B3C8-81F4-4CDE-A9CF-62DE7F38B8CF@oracle.com>
	<54C62A83.5040900@oracle.com>
	
	<54C78238.6020003@oracle.com>
Message-ID: <4945418D-64C5-4117-AD4A-DD4B03CDFDB2@oracle.com>


On Jan 27, 2015, at 4:19 AM, Alan Bateman  wrote:

> On 27/01/2015 01:40, Brian Burkhalter wrote:
>> 
>> Here?s another updated patch:
>> 
>> http://cr.openjdk.java.net/~bpb/6880737/webrev.02/
> Thanks for the update and moving the test to the right place. I think this is okay for now.

This does not need a CCC request does it, as the spec has not changed, only its enforcement?

Brian

From marcins at microsoft.com  Tue Jan 27 17:21:12 2015
From: marcins at microsoft.com (Martin Sawicki (MS OPEN TECH))
Date: Tue, 27 Jan 2015 17:21:12 +0000
Subject: Java cmdline encoding challenges on Windows
Message-ID: 

Hello,
We're proposing an improvement to the OpenJDK intended to fix the currently existing problem with handling Unicode parameters on the command line in Windows (via cmd.exe), which prevents users for example in China from properly passing text strings in their own language via the java.exe command line.

We have a code submission figured out and tested internally. I've uploaded our webrev package here:

https://openjdkcontrib.blob.core.windows.net/unicodecmd/webrev-20150114.zip

The crux of the change lies in using the "W" (wide character) version of the Windows APIs for fetching the command line parameters, rather than the "A" (ascii) version. But this code path is taken only when the following options are set:

-Dwindows.UnicodeConsole=true - switches on Unicode support in the Windows console
-Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; If not specified, UTF-8 is used by default. Ignored when windows.UnicodeConsole is not set to true.

We'd appreciate a review and acceptance of this improvement.

And, as this is our first contribution to this sub-project within the OpenJDK, I apologize for any steps in the submission process that I may have missed here and would appreciate guidance as needed.

Best regards

Martin Sawicki (and Kirk Shoop, and Valeriy Kopylov)
Microsoft Open Technologies, Inc.
A subsidiary of Microsoft Corp.



From lance.andersen at oracle.com  Tue Jan 27 19:56:50 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Tue, 27 Jan 2015 14:56:50 -0500
Subject: Review request for JDK-8051709: Convert JAXP function tests:
	javax.xml.datatype to jtreg (testng) tests
In-Reply-To: <03d201d03a15$4bace290$e306a7b0$@oracle.com>
References: <00b801d03933$4d0d3190$e72794b0$@oracle.com>
	<5A84A3CE-440F-49BC-87F6-50AF44B8D80E@oracle.com>
	<03d201d03a15$4bace290$e306a7b0$@oracle.com>
Message-ID: <073FA30A-D79C-4F65-AB22-B0C23756C5AA@oracle.com>

Hi Frank,

On Jan 27, 2015, at 4:40 AM, Frank Yuan  wrote:

> Thank you, Lance!
>  
> I applied some experience from your previous comments:)

I a glad they were useful :-)
>  
> I have a question for your comment, could you check it below in line?

See below
>  
> Best Regards
> Frank
>  
> From: Lance Andersen [mailto:lance.andersen at oracle.com] 
> Sent: Tuesday, January 27, 2015 3:24 AM
> To: Frank Yuan
> Cc: 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> Subject: Re: Review request for JDK-8051709: Convert JAXP function tests: javax.xml.datatype to jtreg (testng) tests
>  
> Hi Frank,
>  
> I think this looks good. 
>  
>  
> Not sure if you are going to add more tests in the future, but would be good to have tests such as
>  
> new Duration(x.toString()).equals(x)
>  
>  
> Perhaps a few more checks on expected toString() output?.
>  
> //Frank: I will do it
>  
>  
> For XMLGregorianCalendarTest.java,  I would consider at some point adding more permutations of some of the tests that are validating a bugs(now that you are adding this as a new test suite to openjdk)
>  
> //Frank: I am not sure what you mean, which bug do you want me to add test for?

For example checkIsValid()

I would add a DataProvider and add more permutations to test so that you can reduce other potential errors..

Again, a nice to have for a next update.

The problem I always have when we add one off tests, it becomes very hard to manage your test suite and really understand the quality of your coverage.  Better to enhance existing tests to fill in weaknesses as this helps keep your test suite from getting out of control?



>  
> Best
> Lance
>  
> On Jan 26, 2015, at 1:42 AM, Frank Yuan  wrote:
> 
> 
> Hi, Joe and All
> 
> We are working on moving internal jaxp functional tests to open jdk repo.
> This is the datatype suite. Would you please review these test?  Any comment
> will be appreciated.
> 
> bug: https://bugs.openjdk.java.net/browse/JDK-8051709
> webrev: http://cr.openjdk.java.net/~fyuan/8051709/webrev.00/
> 
> 
> Thanks,
> 
> Frank
> 
>  
> 
> 
> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering 
> 1 Network Drive 
> Burlington, MA 01803
> Lance.Andersen at oracle.com
>  
> 
> 



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From dl at cs.oswego.edu  Tue Jan 27 20:15:33 2015
From: dl at cs.oswego.edu (Doug Lea)
Date: Tue, 27 Jan 2015 15:15:33 -0500
Subject: java.util.DualPivotQuickSort does not guarantee NlogN time
In-Reply-To: 
References: 
Message-ID: <54C7F1E5.6030808@cs.oswego.edu>

On 01/27/2015 08:44 AM, Buis, Paul wrote:
> The slowdown would be passing a single extra integer parameter, decrementing it and comparing it to zero at the beginning of the function.
>

Right. The question is whether even a small slowdown is warranted.
Does quadratic behavior arise more that once per trillion cases?
Or is there some scenario in which worst-case inputs can be crafted in advance?
It's possible that a good case can be made here, but I don't know it.
(So I'm not arguing against doing this, but it requires further justification.)

-Doug



From dl at cs.oswego.edu  Tue Jan 27 20:22:13 2015
From: dl at cs.oswego.edu (Doug Lea)
Date: Tue, 27 Jan 2015 15:22:13 -0500
Subject: Q: 8071326: ThreadPoolExecutor in endless thread creation loop
	if workQueue.take() throws RuntimeException
In-Reply-To: <54C7B216.90404@oracle.com>
References: <54C7B216.90404@oracle.com>
Message-ID: <54C7F375.7090403@cs.oswego.edu>


[Oops. I dropped core-libs-dev CC on reply. Resending...)

On 01/27/2015 10:43 AM, Lev Priima wrote:
> Using TPE w/ custom BlockingQueue and if RuntimeException happens in blocking
> BlockingQueue.take() method then this code
>
> new ThreadPoolExecutor(1, 1, 0, TimeUnit.NANOSECONDS,
>      new ArrayBlockingQueue(1) {
>          public Runnable take() throws InterruptedException {
>              throw new RuntimeException();
>          }
>      }
> ).prestartAllCoreThreads();
>
> has an unbounded thread creation loop.
>
> As a result there are many created unbounded threads in RUNNING state after
> printing stack trace to stderr by default UncaughtExceptionHandler. And these
> thread will be cleaned only when whole TPE finished.
>
> Is this "Not an Issue"?
>

It is apparently an issue for someone! But I don't see any spec
violation: If a supplied BlockingQueue not only does not block but also
continuously throws exceptions, then threads using it will be
continuously replaced, given the other TPE construction arguments
here and the call to prestartAllCoreThreads.

Any attempt to change TPE to somehow guess that it should stop trying
to replace threads seems unlikely to solve any actual problem, since
the TPE remains unusable either way.

-Doug



From martinrb at google.com  Tue Jan 27 20:31:23 2015
From: martinrb at google.com (Martin Buchholz)
Date: Tue, 27 Jan 2015 12:31:23 -0800
Subject: Q: 8071326: ThreadPoolExecutor in endless thread creation loop if
	workQueue.take() throws RuntimeException
In-Reply-To: <54C7B216.90404@oracle.com>
References: <54C7B216.90404@oracle.com>
Message-ID: 

On Tue, Jan 27, 2015 at 7:43 AM, Lev Priima  wrote:

> And these thread will be cleaned only when whole TPE finished.
>
>
Is this really true?  Each thread should be replaced while running and so
the total number of threads retained by the TPE at any one time should be
no more than core pool size.


From martinrb at google.com  Tue Jan 27 20:55:04 2015
From: martinrb at google.com (Martin Buchholz)
Date: Tue, 27 Jan 2015 12:55:04 -0800
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C7A54E.1040501@gmail.com>
References: <54C7160D.4000802@oracle.com> <54C75F43.2020103@oracle.com>
	<54C78A97.70001@gmail.com> <54C7A54E.1040501@gmail.com>
Message-ID: 

If you want to ensure something is run before VM exit, add a shutdown hook.

If you really want to run all the finalizers, you can always run a variant
of GcFinalization in your shutdown hook, although as always running during
shutdown increases the risk greatly.

On Tue, Jan 27, 2015 at 6:48 AM, Peter Levart 
wrote:

> On 01/27/2015 01:54 PM, Peter Levart wrote:
>
>> A poor-man's escape hatch is a shutdown hook that calls
>> System.runFinalization(). Which might interfere with other shutdown hooks
>> that run concurrently (runFinalizersOnExit runs finalizers after all
>> shutdown hooks are finished).
>>
>
> Not really. This only runs finalizers for Objects pending finalization,
> while runFinalizersOnExit runs it for all Objects with finalize() methods
> that have not been invoked yet. A big difference. So there's no alternative
> if this method is removed.
>
> Peter
>
>


From lev.priima at oracle.com  Tue Jan 27 21:03:36 2015
From: lev.priima at oracle.com (Lev Priima)
Date: Wed, 28 Jan 2015 00:03:36 +0300
Subject: Q: 8071326: ThreadPoolExecutor in endless thread creation loop
	if workQueue.take() throws RuntimeException
In-Reply-To: 
References: <54C7B216.90404@oracle.com>
	
Message-ID: <54C7FD28.8070708@oracle.com>

Yes. And if we have BlockingQueue w/ some amount of tasks which fail 
with exceptions, same amount of threads(not limited by neither 
maximumPoolSize/corePoolSize) will hang under TPE which takes tasks from 
this queue.

It may cause problems if queue has a big percentage of exception-fail 
tasks and we eventually get OOME while unable to create new native thread.

Lev

On 01/27/2015 11:31 PM, Martin Buchholz wrote:
>
>
> On Tue, Jan 27, 2015 at 7:43 AM, Lev Priima  > wrote:
>
>     And these thread will be cleaned only when whole TPE finished.
>
>
> Is this really true?  Each thread should be replaced while running and 
> so the total number of threads retained by the TPE at any one time 
> should be no more than core pool size.



From john.r.rose at oracle.com  Tue Jan 27 21:08:47 2015
From: john.r.rose at oracle.com (John Rose)
Date: Tue, 27 Jan 2015 13:08:47 -0800
Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when
	GWT LambdaForms are shared
In-Reply-To: <54C7B73F.50404@oracle.com>
References: <54B94766.2080102@oracle.com>
	<7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com>
	<54BEA7D7.6080008@oracle.com>
	<5BA1E369-ED87-4EBD-8408-B73B726D91BD@oracle.com>
	<54C66E4E.9050805@oracle.com>
	<915998BE-25E9-4196-BAC7-FE5527E10F83@oracle.com>
	<54C7B73F.50404@oracle.com>
Message-ID: <8AD9A8CC-E570-4DE6-ABB1-10B00FACB8AB@oracle.com>

Looking very good, thanks.  Ship it!

Actually, can you insert a comment why the injected counts are not scaled?  (Or perhaps they should be??)

Also, we may need a followup bug for the code with this comment:
  // Look for the following shape: AndI (ProfileBoolean) (ConI 1))

Since profileBoolean returns a TypeInt::BOOL, the AndI with (ConI 1) should fold up.
So there's some work to do in MulNode, which may allow that special pattern match to go away.
But I don't want to divert the present bug by a possibly complex dive into fixing AndI::Ideal.

(Generally speaking, pattern matching should assume strong normalization of its inputs.  Otherwise you end up duplicating pattern match code in many places, inconsistently.  Funny one-off idiom checks like this are evidence of incomplete IR normalization.  See http://en.wikipedia.org/wiki/Rewriting for some background on terms like "normalization" and "confluence" which are relevant to C2.)

? John

On Jan 27, 2015, at 8:05 AM, Vladimir Ivanov  wrote:
> 
> Thanks for the feedback, John!
> 
> Updated webrev:
> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.03/jdk
> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.03/hotspot
> 
> Changes:
>  - renamed MHI::profileBranch to MHI::profileBoolean, and ProfileBranchNode to ProfileBooleanNode;
>  - restructured profile layout ([0] => false_cnt, [1] => true_cnt)
>  - factored out profile injection in a separate function (has_injected_profile() in parse2.cpp)
>  - ProfileBooleanNode stores true/false counts instead of taken/not_taken counts
>  - matching from value counts to taken/not_taken happens in has_injected_profile();
>  - added BoolTest::ne support
>  - sharpened test for AndI case: now it checks AndI (ProfileBoolean) (ConI 1) shape
> 
> Best regards,
> Vladimir Ivanov



From peter.levart at gmail.com  Tue Jan 27 22:00:36 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Tue, 27 Jan 2015 23:00:36 +0100
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: 
References: <54C7160D.4000802@oracle.com>	<54C75F43.2020103@oracle.com>	<54C78A97.70001@gmail.com>	<54C7A54E.1040501@gmail.com>
	
Message-ID: <54C80A84.6050609@gmail.com>


On 01/27/2015 09:55 PM, Martin Buchholz wrote:
> If you want to ensure something is run before VM exit, add a shutdown 
> hook.
>
> If you really want to run all the finalizers, you can always run a 
> variant of GcFinalization in your shutdown hook, although as always 
> running during shutdown increases the risk greatly.

What do you mean by "GcFinalization" ?

When shutdown hooks run, there can be objects with finalize methods that 
are still reachable from GC roots, so they are not pending finalization 
yet. Invoking System.gc() and System.runFinalization() would not run 
finalizers for them. runFinalizersOnExit on the other hand, run the 
finalizers for all objects with finalize methods regardles of their 
"pending" status. This is dangerous of course. Even more than normal 
finalization. But there's no public API for that after 
runFinalizersOnExit is removed.

Practically, I think this only potentially affects environments where VM 
is embedded in a process that keeps running when a particular instance 
of VM already exits.

Peter

>
> On Tue, Jan 27, 2015 at 6:48 AM, Peter Levart  > wrote:
>
>     On 01/27/2015 01:54 PM, Peter Levart wrote:
>
>         A poor-man's escape hatch is a shutdown hook that calls
>         System.runFinalization(). Which might interfere with other
>         shutdown hooks that run concurrently (runFinalizersOnExit runs
>         finalizers after all shutdown hooks are finished).
>
>
>     Not really. This only runs finalizers for Objects pending
>     finalization, while runFinalizersOnExit runs it for all Objects
>     with finalize() methods that have not been invoked yet. A big
>     difference. So there's no alternative if this method is removed.
>
>     Peter
>
>



From martinrb at google.com  Tue Jan 27 22:07:50 2015
From: martinrb at google.com (Martin Buchholz)
Date: Tue, 27 Jan 2015 14:07:50 -0800
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C80A84.6050609@gmail.com>
References: <54C7160D.4000802@oracle.com> <54C75F43.2020103@oracle.com>
	<54C78A97.70001@gmail.com> <54C7A54E.1040501@gmail.com>
	
	<54C80A84.6050609@gmail.com>
Message-ID: 

I was referring to
http://code.google.com/p/guava-libraries/source/browse/guava-testlib/src/com/google/common/testing/GcFinalization.java
which I occasionally advertise here.
(but it's intended to be used for testing...)

On Tue, Jan 27, 2015 at 2:00 PM, Peter Levart 
wrote:

>
> On 01/27/2015 09:55 PM, Martin Buchholz wrote:
>
> If you want to ensure something is run before VM exit, add a shutdown
> hook.
>
> If you really want to run all the finalizers, you can always run a variant
> of GcFinalization in your shutdown hook, although as always running during
> shutdown increases the risk greatly.
>
>
> What do you mean by "GcFinalization" ?
>
> When shutdown hooks run, there can be objects with finalize methods that
> are still reachable from GC roots, so they are not pending finalization
> yet. Invoking System.gc() and System.runFinalization() would not run
> finalizers for them. runFinalizersOnExit on the other hand, run the
> finalizers for all objects with finalize methods regardles of their
> "pending" status. This is dangerous of course. Even more than normal
> finalization. But there's no public API for that after runFinalizersOnExit
> is removed.
>
> Practically, I think this only potentially affects environments where VM
> is embedded in a process that keeps running when a particular instance of
> VM already exits.
>
> Peter
>
>
> On Tue, Jan 27, 2015 at 6:48 AM, Peter Levart 
> wrote:
>
>> On 01/27/2015 01:54 PM, Peter Levart wrote:
>>
>>> A poor-man's escape hatch is a shutdown hook that calls
>>> System.runFinalization(). Which might interfere with other shutdown hooks
>>> that run concurrently (runFinalizersOnExit runs finalizers after all
>>> shutdown hooks are finished).
>>>
>>
>>  Not really. This only runs finalizers for Objects pending finalization,
>> while runFinalizersOnExit runs it for all Objects with finalize() methods
>> that have not been invoked yet. A big difference. So there's no alternative
>> if this method is removed.
>>
>> Peter
>>
>>
>
>


From stuart.marks at oracle.com  Tue Jan 27 22:49:49 2015
From: stuart.marks at oracle.com (Stuart Marks)
Date: Tue, 27 Jan 2015 14:49:49 -0800
Subject: RFR 8050819: Please add java.util.Stream.ofNullable(T object)
In-Reply-To: <54C6BC6D.202@univ-mlv.fr>
References: <93EC212E-69DA-4874-8255-E73746404461@oracle.com>	<3F562D05-6BD1-4278-BB12-47CEEEFA5CF2@oracle.com>	<54C68AAD.4060202@oracle.com>
	<54C6BC6D.202@univ-mlv.fr>
Message-ID: <54C8160D.9030707@oracle.com>


On 1/26/15 2:15 PM, Remi Forax wrote:
>
> On 01/26/2015 07:42 PM, Alan Bateman wrote:
>> On 26/01/2015 18:32, Paul Sandoz wrote:
>>> On Jan 20, 2015, at 7:05 PM, Paul Sandoz  wrote:
>>>> Hi,
>>>>
>>>> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050819-Stream-ofNullable/webrev/
>>>>
>>>> Stream.ofNullable can make it easier to work fluently when streams are
>>>> produced, for example within flatMap.
>>>>
>>> So far there have been "null" reviews :-)

I think I saw an empty review fly by. :-)

One quick note on the tests: the test data factory is

     TestData.Factory.ofSupplier("[0, 1)", ...)

which would seem to me to contain a single value zero, but they all use one as 
the value. Not that it really matters, but I was puzzling over it.

>> This looks okay to me, and the naming consistent with Optional.
>>
>> -Alan.
>
> I'm not sure this method pull it's own weight,
> it can be written like this:
>    Optional.ofNullable(t).stream()

It's certainly possible, but it seems like an unnecessary level of indirection. 
The need for something like this has come up multiple times, and it seems 
logical for it to be next to Stream.of() and Stream.empty().

> I'm pretty sure that the JIT will remove the creation of the intermediary Optional
> and adding a method in the JDK that considers that null is an acceptable value
> is in my opinion the wrong signal to send.

I guess having a method like this makes it easier for people to deal with their 
possibly-null-producing code in a new streams world. Such null-producing code 
already exists. I guess you could say that providing this method encourages 
people to continue producing nulls, but everybody is already doing this today, 
so....

s'marks


From forax at univ-mlv.fr  Tue Jan 27 23:09:33 2015
From: forax at univ-mlv.fr (Remi Forax)
Date: Wed, 28 Jan 2015 00:09:33 +0100
Subject: RFR 8050819: Please add java.util.Stream.ofNullable(T object)
In-Reply-To: <54C8160D.9030707@oracle.com>
References: <93EC212E-69DA-4874-8255-E73746404461@oracle.com>	<3F562D05-6BD1-4278-BB12-47CEEEFA5CF2@oracle.com>	<54C68AAD.4060202@oracle.com>
	<54C6BC6D.202@univ-mlv.fr> <54C8160D.9030707@oracle.com>
Message-ID: <54C81AAD.9020508@univ-mlv.fr>


On 01/27/2015 11:49 PM, Stuart Marks wrote:
>
> On 1/26/15 2:15 PM, Remi Forax wrote:
>>
>> On 01/26/2015 07:42 PM, Alan Bateman wrote:
>>> On 26/01/2015 18:32, Paul Sandoz wrote:
>>>> On Jan 20, 2015, at 7:05 PM, Paul Sandoz  
>>>> wrote:
>>>>> Hi,
>>>>>
>>>>> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050819-Stream-ofNullable/webrev/ 
>>>>>
>>>>>
>>>>> Stream.ofNullable can make it easier to work fluently when streams 
>>>>> are
>>>>> produced, for example within flatMap.
>>>>>
>>>> So far there have been "null" reviews :-)
>
> I think I saw an empty review fly by. :-)
>
> One quick note on the tests: the test data factory is
>
>     TestData.Factory.ofSupplier("[0, 1)", ...)
>
> which would seem to me to contain a single value zero, but they all 
> use one as the value. Not that it really matters, but I was puzzling 
> over it.
>
>>> This looks okay to me, and the naming consistent with Optional.
>>>
>>> -Alan.
>>
>> I'm not sure this method pull it's own weight,
>> it can be written like this:
>>    Optional.ofNullable(t).stream()
>
> It's certainly possible, but it seems like an unnecessary level of 
> indirection. The need for something like this has come up multiple 
> times, and it seems logical for it to be next to Stream.of() and 
> Stream.empty().
>
>> I'm pretty sure that the JIT will remove the creation of the 
>> intermediary Optional
>> and adding a method in the JDK that considers that null is an 
>> acceptable value
>> is in my opinion the wrong signal to send.
>
> I guess having a method like this makes it easier for people to deal 
> with their possibly-null-producing code in a new streams world. Such 
> null-producing code already exists. I guess you could say that 
> providing this method encourages people to continue producing nulls, 
> but everybody is already doing this today, so....

I prefer to be optimistic and think that at some point in the future,
people will think twice before returning null or storing null in a 
collection.

Anyway, if you have a code that already produce a null value, you can 
wrap it using Optional
as I've suggested or just let the value be streamed and filter it out, 
that's why we have Objects.nonNull() after all.
   Stream.of(t).filter(Objects::nonNull)

>
> s'marks

R?mi



From joe.darcy at oracle.com  Tue Jan 27 23:22:30 2015
From: joe.darcy at oracle.com (Joseph D. Darcy)
Date: Tue, 27 Jan 2015 15:22:30 -0800
Subject: java.util.DualPivotQuickSort does not guarantee NlogN time
In-Reply-To: <54C7F1E5.6030808@cs.oswego.edu>
References: 
	<54C7F1E5.6030808@cs.oswego.edu>
Message-ID: <54C81DB6.5090201@oracle.com>

On 1/27/2015 12:15 PM, Doug Lea wrote:
> On 01/27/2015 08:44 AM, Buis, Paul wrote:
>> The slowdown would be passing a single extra integer parameter, 
>> decrementing it and comparing it to zero at the beginning of the 
>> function.
>>
>
> Right. The question is whether even a small slowdown is warranted.
> Does quadratic behavior arise more that once per trillion cases?
> Or is there some scenario in which worst-case inputs can be crafted in 
> advance?
> It's possible that a good case can be made here, but I don't know it.
> (So I'm not arguing against doing this, but it requires further 
> justification.)
>

FWIW, long predating the dual pivot work in the JDK, back in 2002 I did 
some work looking into quicksort variants

     http://www.sonic.net/~jddarcy/Research/cs339-quicksort.pdf

which included looking over a sample of the many, many quicksort 
variations people have explored.

Germane to the worst case behavior discussion is another paper I recall 
reading at the time, McIlroy's "A Killer Adversary for Quicksort" [1] 
which uses an adversarial comparator in C to force quadratic behavior 
regardless of the pivoting strategy, including randomized pivots.

I assume a similar approach could be used to get worst-case behavior 
with analogous Java library constructs.

-Joe

[1] SOFTWARE?PRACTICE AND EXPERIENCE, VOL. 29(0), 1?4 (0 1999)



From jeffhain at rocketmail.com  Tue Jan 27 23:52:16 2015
From: jeffhain at rocketmail.com (Jeff Hain)
Date: Tue, 27 Jan 2015 23:52:16 +0000 (UTC)
Subject: java.util.DualPivotQuickSort does not guarantee NlogN time
In-Reply-To: <54C81DB6.5090201@oracle.com>
References: <54C81DB6.5090201@oracle.com>
Message-ID: <1880090362.1634804.1422402736253.JavaMail.yahoo@mail.yahoo.com>


An ugly but non-intrusive workaround, that would not add much overhead for usual cases,taking advantage of quadraticness blowing up in stack overflow before long:

public class ParanoidSort {??? public static void sort(Object[] a) {
??????? try {
??????????? Arrays.sort(a);
??????? } catch (StackOverflowError e) {
??????????? // Gone quadratic and array was too long.
??????????? // Falling back to slower but safer sort,
??????????? // hoping that array is not damaged.
??????????? HeapSort.sort(a);
??????? }
??? }
}

Could also catch OOME there but I don't know if it's wise.

-Jeff


From huizhe.wang at oracle.com  Wed Jan 28 02:26:53 2015
From: huizhe.wang at oracle.com (huizhe wang)
Date: Tue, 27 Jan 2015 18:26:53 -0800
Subject: Review request for JDK-8051710: Convert JAXP function tests:
	javax.xml.jaxp14.* to jtreg (testng) tests
In-Reply-To: <03a201d03a10$ef463d70$cdd2b850$@oracle.com>
References: <03a201d03a10$ef463d70$cdd2b850$@oracle.com>
Message-ID: <54C848ED.7090108@oracle.com>

Hi Frank,

Nice refactoring the original tests, esp. the TransformerTest!

jaxp14 is legacy in the jaxp standalone world. While we are at this, you 
may want to move these tests to their relevant packages since in the JDK 
world, jaxp14 is no longer relevant (jaxp 1.4 was integrated into JDK 
6).  As you've already split FactoryTest into various Factory tests, you 
may find them a bit thin in terms of test coverage now that they are 
named *FactoryTest since they cover just one of the two newInstance 
methods. I would think it makes sense to move them into / combine with 
the Factory tests of their own packages.

Thanks,
Joe

On 1/27/2015 1:09 AM, Frank Yuan wrote:
> Hi, Joe, Lance and All
>
> We are working on moving internal jaxp functional tests to open jdk repo.
> This is the jaxp14 suite. Would you please review these test?  Any comment
> will be appreciated.
>
> bug: https://bugs.openjdk.java.net/browse/JDK-8051710
> webrev: http://cr.openjdk.java.net/~fyuan/8051710/webrev.00/
>
>
> Thanks,
>
> Frank
>



From david.holmes at oracle.com  Wed Jan 28 02:30:09 2015
From: david.holmes at oracle.com (David Holmes)
Date: Wed, 28 Jan 2015 12:30:09 +1000
Subject: Useful message about NullPointerException
In-Reply-To: <54C7B653.1070509@gmail.com>
References: <1421844308100-213240.post@n7.nabble.com>	<54C78F44.1030104@redhat.com>	
	<54C7B653.1070509@gmail.com>
Message-ID: <54C849B1.80403@oracle.com>

Adding back in hotspot-dev to this reply

There have been previous RFEs for this dating back to 2006 - most closed 
as duplicates and the main one eventually closed as "will not fix" 
simply due to it being a low priority RFE for 8 years.

Also see:

https://bugs.openjdk.java.net/browse/JDK-6717558

On 28/01/2015 2:01 AM, Peter Levart wrote:
> On 01/27/2015 03:34 PM, kedar mhaswade wrote:
>> When the JVM executes instructions like getfield
>> ,
>>
>> getstatic, invokevirtual etc. with *objref* on the operand stack and if
>> *objref* is null, an NPE is thrown. It appears that the JVM could tell us
>> more about which *objref* was null at run-time. Candidate for an RFE?
>
> In general it is hard to deduce the meaningfull source of null *objref*
> on the operand stack by analyzing the surrounding bytecodes. It could be
> a result of complex logic executed by bytecodes. Imagine the following:
>
> int length(boolean first, String s1, String s2) {
>      return (first ? s1 : s2).length();
> }
>
> ...the analysis would have to trace the live execution so that it could
> be rolled-back to the meaningful source of null *objref*.

Exactly right - all the VM knows at that exact point is that the thing 
on the stack is an object reference - if its value is null that doesn't 
tell you where that came from. You have to look around to see what 
pushed that objRef onto the stack. Might be doable from the interpreter, 
but from compiled code?

> All VM might semi-realistically do is report the action VM was trying to
> perform when it dereferenced a null *objref*. Like
> "NullPointerException: while invoking method Xxxx.yyyy on a null
> target", or "NullPointerException: while de-referencing instance field
> Xxxx.yyyy of a null reference"
>
> But what does this help if you don't have access to sources? Might be a
> hint, but not much.

As I said there may be some context to assist with how the null was 
introduced, but in general it isn't something that is readily apparent - 
especially in compiled code.

> If you have access to sources, then perhaps an easier solution would be
> for stack traces to include column number in addition to line number of
> location in source that resulted in bytecodes that include the one that
> triggered the NPE.
>
> There is already a RFE for that:
>
>      https://bugs.openjdk.java.net/browse/JDK-8020204

Once past suggestion has been to include the ByteCode Index (BCI) as 
part of the exception stacktrace information:

https://bugs.openjdk.java.net/browse/JDK-4185378

David
-----

> It seems that javac part is already there. The VM part and public API
> part (StackTraceElement) is not, though.
>
> Regards, Peter
>
>>
>> That aside, (and Chris's trick is nice), but if you have no access to the
>> source for the offending code, life is hard in general, isn't it? Because
>> if you can't have control over the source, making that source run on a
>> platform where such an RFE would be perhaps fixed (a future release of
>> the
>> JDK) would be even harder, no?
>>
>> On Tue, Jan 27, 2015 at 5:14 AM, Florian Weimer 
>> wrote:
>>
>>> On 01/21/2015 01:45 PM, pike wrote:
>>>> We frequently see NullPointerException in our logs. It's really a big
>>>> headache when we see a NullPointerException and it is encapsulated in
>>>> another exception as we don't know which object is null and it is
>>> throwing
>>>> an Exception. Is there any way we can get to know the object type or
>>>> the
>>>> object variable name where the object is null and it is throwing a
>>>> NullPointerException?
>>> The line number gives you the position in the source code, and from
>>> that, you can usually figure out the static type.  If this is not
>>> helpful in your case, you need to say why (no debugging information?
>>> multiple candidates per line?).
>>>
>>> The dynamic type is a different matter though, because null has no
>>> specific type at run time.  It may be possible to provide type
>>> information in theory, at a cost, but this would best be prototyped
>>> through byte code rewriting.  Nullable annotations would also help to
>>> pin-point location of the first leak, and you could record that
>>> (including a stack trace) if you want something really fancy.  Whether
>>> it is helpful for legacy code, I don't know.  There should be some
>>> research projects out there covering this area.
>>>
>>> --
>>> Florian Weimer / Red Hat Product Security
>>>
>


From mandy.chung at oracle.com  Wed Jan 28 02:44:58 2015
From: mandy.chung at oracle.com (Mandy Chung)
Date: Tue, 27 Jan 2015 18:44:58 -0800
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C75F43.2020103@oracle.com>
References: <54C7160D.4000802@oracle.com> <54C75F43.2020103@oracle.com>
Message-ID: <54C84D2A.60202@oracle.com>


On 1/27/2015 1:49 AM, Aleksey Shipilev wrote:
> On 27.01.2015 07:37, Mandy Chung wrote:
>> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
>> and this method is inherently unsafe.  I am thinking to propose this method
>> in JDK 9 to throw UnsupportedOperationException.
>>
>> I believe it's rare for existing applications using
>> System.runFinalizationOnExit.
>> My analysis on Maven Central ~315K artifacts that show about ~15 unique
>> artifacts calling System.runFinalizationOnExit while they all come from
>> only 5 classes.
> I think while the use case for runFinalizationOnExit(true) is weird to
> begin with, there seems to be a valid reason to enforce the finalizers
> to run at the end, e.g. to properly free the *critical* underlying
> resources. At this point rFOE(true) trick is, while unsafe, still
> operational, right? If we are to remove rFOE(true) escape hatch, what do
> we propose users to do?

The strong references would have to convert to use WeakReference 
together with System.runFinalization() or System.gc() - 
GcFinalization-like mechanism as Martin suggests.

Mandy


From frank.yuan at oracle.com  Wed Jan 28 03:06:39 2015
From: frank.yuan at oracle.com (Frank Yuan)
Date: Wed, 28 Jan 2015 11:06:39 +0800
Subject: Review request for JDK-8051710: Convert JAXP function tests:
	javax.xml.jaxp14.* to jtreg (testng) tests
In-Reply-To: <54C848ED.7090108@oracle.com>
References: <03a201d03a10$ef463d70$cdd2b850$@oracle.com>
	<54C848ED.7090108@oracle.com>
Message-ID: <016401d03aa7$71428a40$53c79ec0$@oracle.com>

Thank you, Joe.

I will sort the tests as your suggestion, and have 3 questions to confirm
with you:
1. Should I also sort the gap tests?
2. Should I put astro suite at side of auctionportal?
3. If a package has very few test, e.g. I may put XMLEventFactoryTest and
something else in javax.xml.stream.ptest, is it ok? (I would rename
XMLEventFactoryTest as its small coverage)

Best Regards
Frank


-----Original Message-----
From: huizhe wang [mailto:huizhe.wang at oracle.com] 
Sent: Wednesday, January 28, 2015 10:27 AM
To: Frank Yuan
Cc: 'Lance Andersen'; 'Core-Libs-Dev'; 'jibing chen'; 'Gustavo Galimberti';
sandeep.konchady at oracle.com; 'Alexandre (Shura) Iline'
Subject: Re: Review request for JDK-8051710: Convert JAXP function tests:
javax.xml.jaxp14.* to jtreg (testng) tests

Hi Frank,

Nice refactoring the original tests, esp. the TransformerTest!

jaxp14 is legacy in the jaxp standalone world. While we are at this, you may
want to move these tests to their relevant packages since in the JDK world,
jaxp14 is no longer relevant (jaxp 1.4 was integrated into JDK 6).  As
you've already split FactoryTest into various Factory tests, you may find
them a bit thin in terms of test coverage now that they are named
*FactoryTest since they cover just one of the two newInstance methods. I
would think it makes sense to move them into / combine with the Factory
tests of their own packages.

Thanks,
Joe

On 1/27/2015 1:09 AM, Frank Yuan wrote:
> Hi, Joe, Lance and All
>
> We are working on moving internal jaxp functional tests to open jdk
repo.
> This is the jaxp14 suite. Would you please review these test?  Any
comment
> will be appreciated.
>
> bug: https://bugs.openjdk.java.net/browse/JDK-8051710
> webrev: http://cr.openjdk.java.net/~fyuan/8051710/webrev.00/
>
>
> Thanks,
>
> Frank
>




From david.holmes at oracle.com  Wed Jan 28 04:17:49 2015
From: david.holmes at oracle.com (David Holmes)
Date: Wed, 28 Jan 2015 14:17:49 +1000
Subject: Q: 8071326: ThreadPoolExecutor in endless thread creation loop
	if workQueue.take() throws RuntimeException
In-Reply-To: <54C7FD28.8070708@oracle.com>
References: <54C7B216.90404@oracle.com>	
	<54C7FD28.8070708@oracle.com>
Message-ID: <54C862ED.3040404@oracle.com>

On 28/01/2015 7:03 AM, Lev Priima wrote:
> Yes. And if we have BlockingQueue w/ some amount of tasks which fail
> with exceptions, same amount of threads(not limited by neither
> maximumPoolSize/corePoolSize) will hang under TPE which takes tasks from
> this queue.
>
> It may cause problems if queue has a big percentage of exception-fail
> tasks and we eventually get OOME while unable to create new native thread.

If you use your pathological example then of course you can get into a 
situation where the thread creation outpaces the thread termination - it 
takes time and CPU cycles for a thread to actually complete.

A BlockingQueue implementation should not have an expected failure mode 
that results in regularly throwing Errors or RuntimeExceptions. Such a 
BQ implementation would need to be fixed in my opinion.

The TPE is working as designed - if errors/runtime-exceptions are 
encountered the worker thread will terminate and be replaced by a fresh 
worker. If you keep feeding the worker threads such exceptions then you 
incur a high rate of thread churn. So don't do that. :)

Cheers,
David

> Lev
>
> On 01/27/2015 11:31 PM, Martin Buchholz wrote:
>>
>>
>> On Tue, Jan 27, 2015 at 7:43 AM, Lev Priima > > wrote:
>>
>>     And these thread will be cleaned only when whole TPE finished.
>>
>>
>> Is this really true?  Each thread should be replaced while running and
>> so the total number of threads retained by the TPE at any one time
>> should be no more than core pool size.
>


From huizhe.wang at oracle.com  Wed Jan 28 04:35:53 2015
From: huizhe.wang at oracle.com (huizhe wang)
Date: Tue, 27 Jan 2015 20:35:53 -0800
Subject: Review request for JDK-8052401: JAXP function gap tests conversion
In-Reply-To: <03af01d03a12$0ac947d0$205bd770$@oracle.com>
References: <03af01d03a12$0ac947d0$205bd770$@oracle.com>
Message-ID: <54C86729.20706@oracle.com>

Looks good.

Joe

On 1/27/2015 1:17 AM, Frank Yuan wrote:
> Hi, Joe, Lance and All
>
> We are working on moving internal jaxp functional tests to open jdk repo.
> This is the gaptest suite. Would you please review these test?  Any comment
> will be appreciated.
>
> bug: https://bugs.openjdk.java.net/browse/JDK-8052401
> webrev: http://cr.openjdk.java.net/~fyuan/8052401/webrev.00/
>
>
> Thanks,
>
> Frank
>



From huizhe.wang at oracle.com  Wed Jan 28 04:43:52 2015
From: huizhe.wang at oracle.com (huizhe wang)
Date: Tue, 27 Jan 2015 20:43:52 -0800
Subject: Review request for JDK-8051710: Convert JAXP function tests:
	javax.xml.jaxp14.* to jtreg (testng) tests
In-Reply-To: <016401d03aa7$71428a40$53c79ec0$@oracle.com>
References: <03a201d03a10$ef463d70$cdd2b850$@oracle.com>
	<54C848ED.7090108@oracle.com>
	<016401d03aa7$71428a40$53c79ec0$@oracle.com>
Message-ID: <54C86908.7090207@oracle.com>


On 1/27/2015 7:06 PM, Frank Yuan wrote:
> Thank you, Joe.
>
> I will sort the tests as your suggestion, and have 3 questions to confirm
> with you:
> 1. Should I also sort the gap tests?
> 2. Should I put astro suite at side of auctionportal?

Ah, in that case, you may put gap tests alongside the auctionportal as well.
> 3. If a package has very few test, e.g. I may put XMLEventFactoryTest and
> something else in javax.xml.stream.ptest, is it ok? (I would rename
> XMLEventFactoryTest as its small coverage)

Sounds good to me.

Thanks,
Joe

>
> Best Regards
> Frank
>
>
> -----Original Message-----
> From: huizhe wang [mailto:huizhe.wang at oracle.com]
> Sent: Wednesday, January 28, 2015 10:27 AM
> To: Frank Yuan
> Cc: 'Lance Andersen'; 'Core-Libs-Dev'; 'jibing chen'; 'Gustavo Galimberti';
> sandeep.konchady at oracle.com; 'Alexandre (Shura) Iline'
> Subject: Re: Review request for JDK-8051710: Convert JAXP function tests:
> javax.xml.jaxp14.* to jtreg (testng) tests
>
> Hi Frank,
>
> Nice refactoring the original tests, esp. the TransformerTest!
>
> jaxp14 is legacy in the jaxp standalone world. While we are at this, you may
> want to move these tests to their relevant packages since in the JDK world,
> jaxp14 is no longer relevant (jaxp 1.4 was integrated into JDK 6).  As
> you've already split FactoryTest into various Factory tests, you may find
> them a bit thin in terms of test coverage now that they are named
> *FactoryTest since they cover just one of the two newInstance methods. I
> would think it makes sense to move them into / combine with the Factory
> tests of their own packages.
>
> Thanks,
> Joe
>
> On 1/27/2015 1:09 AM, Frank Yuan wrote:
>> Hi, Joe, Lance and All
>>
>> We are working on moving internal jaxp functional tests to open jdk
> repo.
>> This is the jaxp14 suite. Would you please review these test?  Any
> comment
>> will be appreciated.
>>
>> bug: https://bugs.openjdk.java.net/browse/JDK-8051710
>> webrev: http://cr.openjdk.java.net/~fyuan/8051710/webrev.00/
>>
>>
>> Thanks,
>>
>> Frank
>>
>



From vladimir.x.ivanov at oracle.com  Wed Jan 28 09:00:55 2015
From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov)
Date: Wed, 28 Jan 2015 12:00:55 +0300
Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when
	GWT LambdaForms are shared
In-Reply-To: <8AD9A8CC-E570-4DE6-ABB1-10B00FACB8AB@oracle.com>
References: <54B94766.2080102@oracle.com>
	<7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com>
	<54BEA7D7.6080008@oracle.com>
	<5BA1E369-ED87-4EBD-8408-B73B726D91BD@oracle.com>
	<54C66E4E.9050805@oracle.com>
	<915998BE-25E9-4196-BAC7-FE5527E10F83@oracle.com>
	<54C7B73F.50404@oracle.com>
	<8AD9A8CC-E570-4DE6-ABB1-10B00FACB8AB@oracle.com>
Message-ID: <54C8A547.6050607@oracle.com>

> Looking very good, thanks.  Ship it!
Thanks, John!

> Actually, can you insert a comment why the injected counts are not scaled?  (Or perhaps they should be??)
Sure! I intentionally don't scale the counts because I don't see any 
reason to do so. Profiling is done on per-MethodHandle basis, so the 
counts should be very close (considering racy updates) to the actual 
behavior.

> Also, we may need a followup bug for the code with this comment:
>    // Look for the following shape: AndI (ProfileBoolean) (ConI 1))
>
> Since profileBoolean returns a TypeInt::BOOL, the AndI with (ConI 1) should fold up.
> So there's some work to do in MulNode, which may allow that special pattern match to go away.
> But I don't want to divert the present bug by a possibly complex dive into fixing AndI::Ideal.
Good catch! It's an overlook on my side.
The following change for ProfileBooleanNode solves the problem:
-  virtual const Type *bottom_type() const { return TypeInt::INT; }
+  virtual const Type *bottom_type() const { return TypeInt::BOOL; }

I polished the change a little according to your comments (diff against 
v03):
http://cr.openjdk.java.net/~vlivanov/8063137/webrev.03-04/hotspot

Changes:
   - added short explanation why injected counts aren't scaled

   - adjusted ProfileBooleanNode type to TypeInt::BOOL and removed 
excessive pattern matching in has_injected_profile()

   - added an assert when ProfileBooleanNode is removed to catch the 
cases when injected profile isn't used: if we decide to generalize the 
API, I'd be happy to remove it, but current usages assumes that injected 
counts are always consumed during parsing and missing cases can cause 
hard-to-diagnose performance problems.

Best regards,
Vladimir Ivanov

>
> (Generally speaking, pattern matching should assume strong normalization of its inputs.  Otherwise you end up duplicating pattern match code in many places, inconsistently.  Funny one-off idiom checks like this are evidence of incomplete IR normalization.  See http://en.wikipedia.org/wiki/Rewriting for some background on terms like "normalization" and "confluence" which are relevant to C2.)
>
> ? John
>
> On Jan 27, 2015, at 8:05 AM, Vladimir Ivanov  wrote:
>>
>> Thanks for the feedback, John!
>>
>> Updated webrev:
>> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.03/jdk
>> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.03/hotspot
>>
>> Changes:
>>   - renamed MHI::profileBranch to MHI::profileBoolean, and ProfileBranchNode to ProfileBooleanNode;
>>   - restructured profile layout ([0] => false_cnt, [1] => true_cnt)
>>   - factored out profile injection in a separate function (has_injected_profile() in parse2.cpp)
>>   - ProfileBooleanNode stores true/false counts instead of taken/not_taken counts
>>   - matching from value counts to taken/not_taken happens in has_injected_profile();
>>   - added BoolTest::ne support
>>   - sharpened test for AndI case: now it checks AndI (ProfileBoolean) (ConI 1) shape
>>
>> Best regards,
>> Vladimir Ivanov
>


From frank.yuan at oracle.com  Wed Jan 28 10:50:08 2015
From: frank.yuan at oracle.com (Frank Yuan)
Date: Wed, 28 Jan 2015 18:50:08 +0800
Subject: Review request for JDK-8052401: JAXP function gap tests conversion
In-Reply-To: <54C86729.20706@oracle.com>
References: <03af01d03a12$0ac947d0$205bd770$@oracle.com>
	<54C86729.20706@oracle.com>
Message-ID: <025901d03ae8$312ef8d0$938cea70$@oracle.com>

Thank you, Joe.

I have moved gatest to test.gatest, would you like to check again?
http://cr.openjdk.java.net/~fyuan/8052401/webrev.01/

If the code has no problem, would you like to be the sponsor?

Best Regards
Frank

-----Original Message-----
From: huizhe wang [mailto:huizhe.wang at oracle.com] 
Sent: Wednesday, January 28, 2015 12:36 PM
To: Frank Yuan
Cc: 'Lance Andersen'; 'Core-Libs-Dev'; 'jibing chen'; 'Gustavo Galimberti';
sandeep.konchady at oracle.com; 'Alexandre (Shura) Iline'
Subject: Re: Review request for JDK-8052401: JAXP function gap tests
conversion

Looks good.

Joe

On 1/27/2015 1:17 AM, Frank Yuan wrote:
> Hi, Joe, Lance and All
>
> We are working on moving internal jaxp functional tests to open jdk
repo.
> This is the gaptest suite. Would you please review these test?  Any
comment
> will be appreciated.
>
> bug: https://bugs.openjdk.java.net/browse/JDK-8052401
> webrev: http://cr.openjdk.java.net/~fyuan/8052401/webrev.00/
>
>
> Thanks,
>
> Frank
>




From frank.yuan at oracle.com  Wed Jan 28 11:18:53 2015
From: frank.yuan at oracle.com (Frank Yuan)
Date: Wed, 28 Jan 2015 19:18:53 +0800
Subject: Review request for JDK-8051710: Convert JAXP function tests:
	javax.xml.jaxp14.* to jtreg (testng) tests
In-Reply-To: <54C86908.7090207@oracle.com>
References: <03a201d03a10$ef463d70$cdd2b850$@oracle.com>
	<54C848ED.7090108@oracle.com>
	<016401d03aa7$71428a40$53c79ec0$@oracle.com>
	<54C86908.7090207@oracle.com>
Message-ID: <027e01d03aec$353ff060$9fbfd120$@oracle.com>

Hi Joe

I have moved the jaxp14 tests to the corresponding packages, would you like
to have a check? http://cr.openjdk.java.net/~fyuan/8051710/webrev.01/

There is an issue, SchemaFactoryTest is merged to
validation.SchemaFactoryTest, but the validation package(Lance reviewed it)
is waiting to push. I am not sure how to handle for your convenience, so
this webrev also includes validation package. Maybe we can wait for pushing
validation, or push this patch for both jaxp14 and validation if you think
these code change is ok.

Best Regards
Frank

-----Original Message-----
From: huizhe wang [mailto:huizhe.wang at oracle.com] 
Sent: Wednesday, January 28, 2015 12:44 PM
To: Frank Yuan
Cc: 'Lance Andersen'; 'Core-Libs-Dev'; 'jibing chen'; 'Gustavo Galimberti';
sandeep.konchady at oracle.com; 'Alexandre (Shura) Iline'
Subject: Re: Review request for JDK-8051710: Convert JAXP function tests:
javax.xml.jaxp14.* to jtreg (testng) tests


On 1/27/2015 7:06 PM, Frank Yuan wrote:
> Thank you, Joe.
>
> I will sort the tests as your suggestion, and have 3 questions to
confirm
> with you:
> 1. Should I also sort the gap tests?
> 2. Should I put astro suite at side of auctionportal?

Ah, in that case, you may put gap tests alongside the auctionportal as well.
> 3. If a package has very few test, e.g. I may put XMLEventFactoryTest
and
> something else in javax.xml.stream.ptest, is it ok? (I would rename 
> XMLEventFactoryTest as its small coverage)

Sounds good to me.

Thanks,
Joe

>
> Best Regards
> Frank
>
>
> -----Original Message-----
> From: huizhe wang [mailto:huizhe.wang at oracle.com]
> Sent: Wednesday, January 28, 2015 10:27 AM
> To: Frank Yuan
> Cc: 'Lance Andersen'; 'Core-Libs-Dev'; 'jibing chen'; 'Gustavo
Galimberti';
> sandeep.konchady at oracle.com; 'Alexandre (Shura) Iline'
> Subject: Re: Review request for JDK-8051710: Convert JAXP function
tests:
> javax.xml.jaxp14.* to jtreg (testng) tests
>
> Hi Frank,
>
> Nice refactoring the original tests, esp. the TransformerTest!
>
> jaxp14 is legacy in the jaxp standalone world. While we are at this, 
> you
may
> want to move these tests to their relevant packages since in the JDK
world,
> jaxp14 is no longer relevant (jaxp 1.4 was integrated into JDK 6).  As 
> you've already split FactoryTest into various Factory tests, you may
find
> them a bit thin in terms of test coverage now that they are named 
> *FactoryTest since they cover just one of the two newInstance methods. 
> I would think it makes sense to move them into / combine with the 
> Factory tests of their own packages.
>
> Thanks,
> Joe
>
> On 1/27/2015 1:09 AM, Frank Yuan wrote:
>> Hi, Joe, Lance and All
>>
>> We are working on moving internal jaxp functional tests to open jdk
> repo.
>> This is the jaxp14 suite. Would you please review these test?  Any
> comment
>> will be appreciated.
>>
>> bug: https://bugs.openjdk.java.net/browse/JDK-8051710
>> webrev: http://cr.openjdk.java.net/~fyuan/8051710/webrev.00/
>>
>>
>> Thanks,
>>
>> Frank
>>
>




From lev.priima at oracle.com  Wed Jan 28 12:27:34 2015
From: lev.priima at oracle.com (Lev Priima)
Date: Wed, 28 Jan 2015 15:27:34 +0300
Subject: Q: 8071326: ThreadPoolExecutor in endless thread creation loop
	if workQueue.take() throws RuntimeException
In-Reply-To: <54C862ED.3040404@oracle.com>
References: <54C7B216.90404@oracle.com>	
	<54C7FD28.8070708@oracle.com> <54C862ED.3040404@oracle.com>
Message-ID: <54C8D5B6.6070504@oracle.com>

Thanks Doug, David, Martin, especially Martin.
Is it worth to update javadoc of  ThreadPoolExecutor#Queuing section 
with this caveat ?

The original confusion in custom queue implementation raise up from 
javadoc, because BlockingQueue.take() interface specification does not 
prohibit explicitly to throw uncaught runtime exception/errors (as any 
other casual java code). But using this method in an exhaustive resource 
allocation loop obliges to deal with exceptional situations in 
work-producing methods more carefully.

Best Regards,
Lev

On 28.01.2015 7:17, David Holmes wrote:
> On 28/01/2015 7:03 AM, Lev Priima wrote:
>> Yes. And if we have BlockingQueue w/ some amount of tasks which fail
>> with exceptions, same amount of threads(not limited by neither
>> maximumPoolSize/corePoolSize) will hang under TPE which takes tasks from
>> this queue.
>>
>> It may cause problems if queue has a big percentage of exception-fail
>> tasks and we eventually get OOME while unable to create new native 
>> thread.
>
> If you use your pathological example then of course you can get into a 
> situation where the thread creation outpaces the thread termination - 
> it takes time and CPU cycles for a thread to actually complete.
>
> A BlockingQueue implementation should not have an expected failure 
> mode that results in regularly throwing Errors or RuntimeExceptions. 
> Such a BQ implementation would need to be fixed in my opinion.
>
> The TPE is working as designed - if errors/runtime-exceptions are 
> encountered the worker thread will terminate and be replaced by a 
> fresh worker. If you keep feeding the worker threads such exceptions 
> then you incur a high rate of thread churn. So don't do that. :)
>
> Cheers,
> David
>
>> Lev
>>
>> On 01/27/2015 11:31 PM, Martin Buchholz wrote:
>>>
>>>
>>> On Tue, Jan 27, 2015 at 7:43 AM, Lev Priima >> > wrote:
>>>
>>>     And these thread will be cleaned only when whole TPE finished.
>>>
>>>
>>> Is this really true?  Each thread should be replaced while running and
>>> so the total number of threads retained by the TPE at any one time
>>> should be no more than core pool size.
>>



From paul.sandoz at oracle.com  Wed Jan 28 14:14:07 2015
From: paul.sandoz at oracle.com (Paul Sandoz)
Date: Wed, 28 Jan 2015 15:14:07 +0100
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C7160D.4000802@oracle.com>
References: <54C7160D.4000802@oracle.com>
Message-ID: <4E2E98F4-0B5A-41F4-9BF7-F598CBE6B66C@oracle.com>


On Jan 27, 2015, at 5:37 AM, Mandy Chung  wrote:

> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
> and this method is inherently unsafe.  I am thinking to propose this method
> in JDK 9 to throw UnsupportedOperationException.
> 
> I believe it's rare for existing applications using System.runFinalizationOnExit.
> My analysis on Maven Central ~315K artifacts that show about ~15 unique
> artifacts calling System.runFinalizationOnExit while they all come from
> only 5 classes.
> 
> Any thought/feedback?
> 

If we do decide to do something perhaps we should remove it from 9 and back port the USO in 8? Why keep it around if it's been deprecated for over 10 years and from your analysis it's used by 0.005% of maven artifacts.

Paul.



From Alan.Bateman at oracle.com  Wed Jan 28 15:07:33 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Wed, 28 Jan 2015 15:07:33 +0000
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C7160D.4000802@oracle.com>
References: <54C7160D.4000802@oracle.com>
Message-ID: <54C8FB35.1010400@oracle.com>

On 27/01/2015 04:37, Mandy Chung wrote:
> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
> and this method is inherently unsafe.  I am thinking to propose this 
> method
> in JDK 9 to throw UnsupportedOperationException.
>
> I believe it's rare for existing applications using 
> System.runFinalizationOnExit.
> My analysis on Maven Central ~315K artifacts that show about ~15 unique
> artifacts calling System.runFinalizationOnExit while they all come from
> only 5 classes.
>
> Any thought/feedback?
It's broken in other ways beyond what is in the @deprecated note so I 
don't think it's much of a loss to finally disable it. I don't know if 
you have come up with candidate wording to replace the existing wording 
but having it reference the shutdown hooks and the ref API might be useful.

-Alan


From volker.simonis at gmail.com  Wed Jan 28 15:28:13 2015
From: volker.simonis at gmail.com (Volker Simonis)
Date: Wed, 28 Jan 2015 16:28:13 +0100
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <4E2E98F4-0B5A-41F4-9BF7-F598CBE6B66C@oracle.com>
References: <54C7160D.4000802@oracle.com>
	<4E2E98F4-0B5A-41F4-9BF7-F598CBE6B66C@oracle.com>
Message-ID: 

I understand that the benefits of calling System.runFinalizersOnExit()
may be questionable and I don?t have any objections to remove it in
Java 9. But I'm strongly against changing it to throw an
UnsupportedOperationException and downport that change to Java 8. In
my eyes that?s really a no-go.

I've quickly scanned our SAP-internal Java repositories and found
several occurrences of System.runFinalizersOnExit() altough most of
them seem to be in testing/example code. It's perfectly fine if that
code won't run out of the box in Java 9.

But it will be totally unacceptable if a product which has been
certified and runs on Java 8 will suddenly fail with an
UnsupportedOperationException just because of a new Java 8 update.

Regards,
Volker


On Wed, Jan 28, 2015 at 3:14 PM, Paul Sandoz  wrote:
>
> On Jan 27, 2015, at 5:37 AM, Mandy Chung  wrote:
>
>> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
>> and this method is inherently unsafe.  I am thinking to propose this method
>> in JDK 9 to throw UnsupportedOperationException.
>>
>> I believe it's rare for existing applications using System.runFinalizationOnExit.
>> My analysis on Maven Central ~315K artifacts that show about ~15 unique
>> artifacts calling System.runFinalizationOnExit while they all come from
>> only 5 classes.
>>
>> Any thought/feedback?
>>
>
> If we do decide to do something perhaps we should remove it from 9 and back port the USO in 8? Why keep it around if it's been deprecated for over 10 years and from your analysis it's used by 0.005% of maven artifacts.
>
> Paul.
>


From peter.levart at gmail.com  Wed Jan 28 15:59:38 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Wed, 28 Jan 2015 16:59:38 +0100
Subject: Useful message about NullPointerException
In-Reply-To: <54C849B1.80403@oracle.com>
References: <1421844308100-213240.post@n7.nabble.com>	<54C78F44.1030104@redhat.com>	
	<54C7B653.1070509@gmail.com> <54C849B1.80403@oracle.com>
Message-ID: <54C9076A.2060903@gmail.com>

On 01/28/2015 03:30 AM, David Holmes wrote:
>> If you have access to sources, then perhaps an easier solution would be
>> for stack traces to include column number in addition to line number of
>> location in source that resulted in bytecodes that include the one that
>> triggered the NPE.
>>
>> There is already a RFE for that:
>>
>> https://bugs.openjdk.java.net/browse/JDK-8020204
>
> Once past suggestion has been to include the ByteCode Index (BCI) as 
> part of the exception stacktrace information:
>
> https://bugs.openjdk.java.net/browse/JDK-4185378
>
> David 

Right,

I checked per-method CharacterRangeTable that gets emitted by javac 
-Xjcov option, and unfortunately it is of limited use. The 
CharacterRangeTable contains mappings from byte code index ranges 
(start-bci, end-bci) -> character ranges (start-line, start-column, 
end-line, end-column) of adequate code in source file. The ranges I have 
observed have 2 granularities: "statement" and "block". For example, the 
following program:


public class CharRangeTest {

     String str() {
         return "ABC";
     }

     public static void main(String[] args) {
         int i = new CharRangeTest().str().substring(1).length();
         System.out.println(i);
     }
}


Compiles to the following bytecodes for main method:

   public static void main(java.lang.String[]);
     descriptor: ([Ljava/lang/String;)V
     flags: ACC_PUBLIC, ACC_STATIC
     Code:
       stack=2, locals=2, args_size=1
          0: new           #3                  // class CharRangeTest
          3: dup
          4: invokespecial #4                  // Method "":()V
          7: invokevirtual #5                  // Method 
str:()Ljava/lang/String;
         10: iconst_1
         11: invokevirtual #6                  // Method 
java/lang/String.substring:(I)Ljava/lang/String;
         14: invokevirtual #7                  // Method 
java/lang/String.length:()I
         17: istore_1
         18: getstatic     #8                  // Field 
java/lang/System.out:Ljava/io/PrintStream;
         21: iload_1
         22: invokevirtual #9                  // Method 
java/io/PrintStream.println:(I)V
         25: return
       LineNumberTable:
         line 11: 0
         line 12: 18
         line 13: 25
       CharacterRangeTable:
              0, 17,   2c09,   2c41,    1        //  0, 17, 11:09,   
11:65, statement
             18, 24,   3009,   301f,    1        // 18, 24, 12:09,   
12:31, statement
              0, 25,   282c,   3406,    2        //  0, 25, 10:44,   
13:06, block



CharacterRangeTable says that bytecodes at indexes 0 to 17 map to source 
code from (line 11, column 9) to (line 11, column 64) and that this 
range is a "statement". Unfortunately, the assignment to i of the result 
of the whole chain of invocations is a single Java "statement":

int i = new CharRangeTest().str().substring(1).length();

So if NPE happens anywhere in this statement, we could only pin-point 
the statement and not a particular null dereference. The only time this 
would give us some additional info is when there are more statements in 
the line, like:

     x.doSomething(); y.doSomeMore();

...but such code formatting is very rare if non-existent.


If adding column number to StackTraceElement (JDK-8020204) is currently 
not an easy thing to do, since it would require javac changes, adding 
byte code index (JDK-4185378 ) is trivial:


http://cr.openjdk.java.net/~plevart/jdk9-dev/StackTraceElement.byteCodeIndex/jdk.webrev.01/

http://cr.openjdk.java.net/~plevart/jdk9-dev/StackTraceElement.byteCodeIndex/hotspot.webrev.01/


With this patch, I get the following style of stack-traces:

Exception in thread "main" java.io.FileNotFoundException: /tmp/x (No 
such file or directory)
         at java.io.FileInputStream.open0(Native Method)
         at java.io.FileInputStream.open[2](FileInputStream.java:195)
         at java.io.FileInputStream.[97](FileInputStream.java:138)
         at java.io.FileInputStream.[17](FileInputStream.java:93)
         at Test.dump[9](Test.java:12)
         at Test.main[3](Test.java:19)


The numbers in square brackets are byte code indexes that pin-point the 
location in byte code. With the help of javap, one can use this info to 
find out the cause of exception even without having access to sources.


Regards, Peter



From paul.sandoz at oracle.com  Wed Jan 28 16:18:14 2015
From: paul.sandoz at oracle.com (Paul Sandoz)
Date: Wed, 28 Jan 2015 17:18:14 +0100
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: 
References: <54C7160D.4000802@oracle.com>
	<4E2E98F4-0B5A-41F4-9BF7-F598CBE6B66C@oracle.com>
	
Message-ID: 


On Jan 28, 2015, at 4:28 PM, Volker Simonis  wrote:

> I understand that the benefits of calling System.runFinalizersOnExit()
> may be questionable and I don?t have any objections to remove it in
> Java 9. But I'm strongly against changing it to throw an
> UnsupportedOperationException and downport that change to Java 8. In
> my eyes that?s really a no-go.
> 
> I've quickly scanned our SAP-internal Java repositories and found
> several occurrences of System.runFinalizersOnExit() altough most of
> them seem to be in testing/example code. It's perfectly fine if that
> code won't run out of the box in Java 9.
> 
> But it will be totally unacceptable if a product which has been
> certified and runs on Java 8 will suddenly fail with an
> UnsupportedOperationException just because of a new Java 8 update.
> 

Yes, good point, i have a hair-trigger for rarely used unsafe stuff :-)

Paul.


From Paul.Sandoz at oracle.com  Wed Jan 28 17:09:30 2015
From: Paul.Sandoz at oracle.com (Paul Sandoz)
Date: Wed, 28 Jan 2015 18:09:30 +0100
Subject: RFR 8050819: Please add java.util.Stream.ofNullable(T object)
In-Reply-To: <54C8160D.9030707@oracle.com>
References: <93EC212E-69DA-4874-8255-E73746404461@oracle.com>	<3F562D05-6BD1-4278-BB12-47CEEEFA5CF2@oracle.com>	<54C68AAD.4060202@oracle.com>
	<54C6BC6D.202@univ-mlv.fr> <54C8160D.9030707@oracle.com>
Message-ID: <3273ECDE-4819-4DBF-9F59-239C7D7F297E@oracle.com>


On Jan 27, 2015, at 11:49 PM, Stuart Marks  wrote:

> 
> On 1/26/15 2:15 PM, Remi Forax wrote:
>> 
>> On 01/26/2015 07:42 PM, Alan Bateman wrote:
>>> On 26/01/2015 18:32, Paul Sandoz wrote:
>>>> On Jan 20, 2015, at 7:05 PM, Paul Sandoz  wrote:
>>>>> Hi,
>>>>> 
>>>>> http://cr.openjdk.java.net/~psandoz/jdk9/JDK-8050819-Stream-ofNullable/webrev/
>>>>> 
>>>>> Stream.ofNullable can make it easier to work fluently when streams are
>>>>> produced, for example within flatMap.
>>>>> 
>>>> So far there have been "null" reviews :-)
> 
> I think I saw an empty review fly by. :-)
> 
> One quick note on the tests: the test data factory is
> 
>    TestData.Factory.ofSupplier("[0, 1)", ...)
> 
> which would seem to me to contain a single value zero, but they all use one as the value. Not that it really matters, but I was puzzling over it.

Thanks, it should be "{1}", i updated locally (in multiple places in the test source file).


> 
>>> This looks okay to me, and the naming consistent with Optional.
>>> 
>>> -Alan.
>> 
>> I'm not sure this method pull it's own weight,
>> it can be written like this:
>>   Optional.ofNullable(t).stream()
> 
> It's certainly possible, but it seems like an unnecessary level of indirection. The need for something like this has come up multiple times, and it seems logical for it to be next to Stream.of() and Stream.empty().

I was sitting on the fence with this feature for a while. There are many ways this can be surfaced but having a concise way that is easily found does seem useful.


> 
>> I'm pretty sure that the JIT will remove the creation of the intermediary Optional
>> and adding a method in the JDK that considers that null is an acceptable value
>> is in my opinion the wrong signal to send.
> 
> I guess having a method like this makes it easier for people to deal with their possibly-null-producing code in a new streams world. Such null-producing code already exists. I guess you could say that providing this method encourages people to continue producing nulls, but everybody is already doing this today, so....
> 

I had the same view as Remi (mostly still do) with regards to the wrong signal, but there is stuff out there that returns null and when one finds oneself on the wrong side of that null it's rather annoying.

Paul.


From miroslav.kos at oracle.com  Wed Jan 28 17:11:05 2015
From: miroslav.kos at oracle.com (Miroslav Kos)
Date: Wed, 28 Jan 2015 18:11:05 +0100
Subject: Review request 8057645: Deprivilege JAX-WS, JAXB, JAF to extension
	class loader
In-Reply-To: <54C69425.1040306@oracle.com>
References: <54C69425.1040306@oracle.com>
Message-ID: <54C91829.80601@oracle.com>


On 26/01/15 20:23, Mandy Chung wrote:
> http://cr.openjdk.java.net/~mchung/jdk9/webrevs/8057645/webrev.00/
>
> This patch proposes to move java.xml.ws, java.xml.bind, 
> java.activation out of the boot loader and be loaded by the extension 
> class loader.  We grant java.xml.ws and java.xml.bind the minimum set 
> of permissions.  java.activation hasAllPermission for now and that can 
> be revised in the future when JAF team identifies the permission set 
> required java.activation.
>
> Miroslav - can you confirm if the JAX-WS and JAXB standalone tests 
> pass with this patch?
Hi Mandy,

I just ran standalone JAX-WS tests and without SM results are ok, with 
SM enabled I will need some more time - our WS-harness (test framework) 
requires some permissions too so I'll need to investigate which ones are 
those. Up to now, we used single policy file which was union of 
permissions requred by WS-harness + JAX-WS runtime.

I'll ping you when have more results.
For JAX-B, we are currently unable to run standalone tests against JDK.

Thanks
Miran

> Existing code that assumes the defining class loader of JAX-WS, JAXB, 
> JAF classes may be impacted by this change (e.g. the class loader 
> delegation to the bootstrap class loader skipping the extension class 
> loader).  They are standalone technologies that used to be loaded by 
> non-null class loader before they were included in Java SE.  It should 
> be rare of such dependency. Callbacks may assume java.xml.ws and 
> java.xml.bind classes to have AllPermissions so that when running with 
> security manager, if the permission required for callback is not part 
> of the permission set granted to java.xml.ws and java.xml.bind, 
> SecurityException will be thrown. We need customer testings to 
> identify this callback permission case and revisit if they should be 
> granted with AllPermission for JDK 9.
>
> Mandy



From vladimir.x.ivanov at oracle.com  Wed Jan 28 17:12:23 2015
From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov)
Date: Wed, 28 Jan 2015 20:12:23 +0300
Subject: [9] RFR (XS): 8071787: Don't block inlining when
	DONT_INLINE_THRESHOLD=0
Message-ID: <54C91877.5040707@oracle.com>

http://cr.openjdk.java.net/~vlivanov/8071787/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8071787

For testing & performance measurements, sometimes it's useful to replace 
block inlining wrappers with trivial reinvokers.

This change extends DONT_INLINE_THRESHOLD in the following manner:
   DONT_INLINE_THRESHOLD = -1: no wrapper
   DONT_INLINE_THRESHOLD =  0: reinvoker
   DONT_INLINE_THRESHOLD >  0: counting wrapper

Before that DONT_INLINE_THRESHOLD=0 meant a counting wrapper which is 
removed on the first invocation. After the change, it's 
DONT_INLINE_THRESHOLD=1.

Testing: manual, java/lang/invoke

Best regards,
Vladimir Ivanov


From vladimir.x.ivanov at oracle.com  Wed Jan 28 17:22:57 2015
From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov)
Date: Wed, 28 Jan 2015 20:22:57 +0300
Subject: [9] RFR (XXS): 8071788: CountingWrapper.asType() is broken
Message-ID: <54C91AF1.3010602@oracle.com>

http://cr.openjdk.java.net/~vlivanov/8071788/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8071788

There's a type mismatch between MethodHandle and LambdaForm in 
CountingWrapper.asTypeUncached(). Sometimes, it leads to a VM crash.

The fix is to use adapted MethodHandle to construct LambdaForm.

There's no way to reproduce this problem with vanilla 8u40/9 binaries, 
because CountingWrapper is used only to block inlinining in GWT 
(MHI::profile() on target and fallback MethodHandles).

It means there's no way to call CountingWrapper.asType() on wrapped 
MethodHandles outside of java.lang.invoke code, and there are no such 
calls inside it.

Testing: manual, java/lang/invoke

Thanks!

Best regards,
Vladimir Ivanov


From martinrb at google.com  Wed Jan 28 17:35:25 2015
From: martinrb at google.com (Martin Buchholz)
Date: Wed, 28 Jan 2015 09:35:25 -0800
Subject: Q: 8071326: ThreadPoolExecutor in endless thread creation loop if
	workQueue.take() throws RuntimeException
In-Reply-To: <54C8D5B6.6070504@oracle.com>
References: <54C7B216.90404@oracle.com>
	
	<54C7FD28.8070708@oracle.com> <54C862ED.3040404@oracle.com>
	<54C8D5B6.6070504@oracle.com>
Message-ID: 

It's hard for me to think of something we could add to the javadoc that
would actually help future users enough to offset the confusion of adding
subtleties of rarely encountered behavior.  I also don't see an easy way to
improve the pool's reaction to exceptions coming from the queue.  Right now
the reporting mechanism is the uncaught exception handler, which is under
the user's control, although it is not obvious.

On Wed, Jan 28, 2015 at 4:27 AM, Lev Priima  wrote:

> Thanks Doug, David, Martin, especially Martin.
> Is it worth to update javadoc of  ThreadPoolExecutor#Queuing section with
> this caveat ?
>
> The original confusion in custom queue implementation raise up from
> javadoc, because BlockingQueue.take() interface specification does not
> prohibit explicitly to throw uncaught runtime exception/errors (as any
> other casual java code). But using this method in an exhaustive resource
> allocation loop obliges to deal with exceptional situations in
> work-producing methods more carefully.
>
> Best Regards,
> Lev
>
>
> On 28.01.2015 7:17, David Holmes wrote:
>
>> On 28/01/2015 7:03 AM, Lev Priima wrote:
>>
>>> Yes. And if we have BlockingQueue w/ some amount of tasks which fail
>>> with exceptions, same amount of threads(not limited by neither
>>> maximumPoolSize/corePoolSize) will hang under TPE which takes tasks from
>>> this queue.
>>>
>>> It may cause problems if queue has a big percentage of exception-fail
>>> tasks and we eventually get OOME while unable to create new native
>>> thread.
>>>
>>
>> If you use your pathological example then of course you can get into a
>> situation where the thread creation outpaces the thread termination - it
>> takes time and CPU cycles for a thread to actually complete.
>>
>> A BlockingQueue implementation should not have an expected failure mode
>> that results in regularly throwing Errors or RuntimeExceptions. Such a BQ
>> implementation would need to be fixed in my opinion.
>>
>> The TPE is working as designed - if errors/runtime-exceptions are
>> encountered the worker thread will terminate and be replaced by a fresh
>> worker. If you keep feeding the worker threads such exceptions then you
>> incur a high rate of thread churn. So don't do that. :)
>>
>> Cheers,
>> David
>>
>>  Lev
>>>
>>> On 01/27/2015 11:31 PM, Martin Buchholz wrote:
>>>
>>>>
>>>>
>>>> On Tue, Jan 27, 2015 at 7:43 AM, Lev Priima >>> > wrote:
>>>>
>>>>     And these thread will be cleaned only when whole TPE finished.
>>>>
>>>>
>>>> Is this really true?  Each thread should be replaced while running and
>>>> so the total number of threads retained by the TPE at any one time
>>>> should be no more than core pool size.
>>>>
>>>
>>>
>


From Roger.Riggs at Oracle.com  Wed Jan 28 18:30:05 2015
From: Roger.Riggs at Oracle.com (Roger Riggs)
Date: Wed, 28 Jan 2015 13:30:05 -0500
Subject: Java cmdline encoding challenges on Windows
In-Reply-To: 
References: 
Message-ID: <54C92AAD.1050006@Oracle.com>

Hi Martin,

Excellent, command line parsing of non-ascii has been a issuefor a while.

Is the patch small enough to include in email?
Sorry, but links to non-OpenJDK servers are not  covered by the OpenJDK 
agreements.
And the openjdk mailer strips off attachments as a defense mechanism.

Is there a Jira issue for this improvement?  Do you have a 
bugs.openjdk.java.net login?
Having an issue to attach the webrev to would clear up the technical issues

Thanks, Roger



On 1/27/2015 12:21 PM, Martin Sawicki (MS OPEN TECH) wrote:
> Hello,
> We're proposing an improvement to the OpenJDK intended to fix the currently existing problem with handling Unicode parameters on the command line in Windows (via cmd.exe), which prevents users for example in China from properly passing text strings in their own language via the java.exe command line.
>
> We have a code submission figured out and tested internally. I've uploaded our webrev package here:
>
> https://openjdkcontrib.blob.core.windows.net/unicodecmd/webrev-20150114.zip
>
> The crux of the change lies in using the "W" (wide character) version of the Windows APIs for fetching the command line parameters, rather than the "A" (ascii) version. But this code path is taken only when the following options are set:
>
> -Dwindows.UnicodeConsole=true - switches on Unicode support in the Windows console
> -Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; If not specified, UTF-8 is used by default. Ignored when windows.UnicodeConsole is not set to true.
>
> We'd appreciate a review and acceptance of this improvement.
>
> And, as this is our first contribution to this sub-project within the OpenJDK, I apologize for any steps in the submission process that I may have missed here and would appreciate guidance as needed.
>
> Best regards
>
> Martin Sawicki (and Kirk Shoop, and Valeriy Kopylov)
> Microsoft Open Technologies, Inc.
> A subsidiary of Microsoft Corp.
>



From scolebourne at joda.org  Wed Jan 28 19:38:59 2015
From: scolebourne at joda.org (Stephen Colebourne)
Date: Wed, 28 Jan 2015 19:38:59 +0000
Subject: RFR 8050819: Please add java.util.Stream.ofNullable(T object)
In-Reply-To: <3273ECDE-4819-4DBF-9F59-239C7D7F297E@oracle.com>
References: <93EC212E-69DA-4874-8255-E73746404461@oracle.com>
	<3F562D05-6BD1-4278-BB12-47CEEEFA5CF2@oracle.com>
	<54C68AAD.4060202@oracle.com>
	<54C6BC6D.202@univ-mlv.fr> <54C8160D.9030707@oracle.com>
	<3273ECDE-4819-4DBF-9F59-239C7D7F297E@oracle.com>
Message-ID: 

On 28 January 2015 at 17:09, Paul Sandoz  wrote:
>> I guess having a method like this makes it easier for people to deal with their possibly-null-producing code in a new streams world. Such null-producing code already exists. I guess you could say that providing this method encourages people to continue producing nulls, but everybody is already doing this today, so....
>>
>
> I had the same view as Remi (mostly still do) with regards to the wrong signal, but there is stuff out there that returns null and when one finds oneself on the wrong side of that null it's rather annoying.

Whether a team understands the benefits of avoiding null or not is
often orthogonal to whether the APIs a team interacts with avoid null.
A method like this is useful at those boundaries and painful when
missing. As such, I support the addition.

Stephen


From john.r.rose at oracle.com  Wed Jan 28 20:30:37 2015
From: john.r.rose at oracle.com (John Rose)
Date: Wed, 28 Jan 2015 12:30:37 -0800
Subject: [9] RFR (M): 8063137: Never-taken branches should be pruned when
	GWT LambdaForms are shared
In-Reply-To: <54C8A547.6050607@oracle.com>
References: <54B94766.2080102@oracle.com>
	<7B03B9FB-17B4-4AE0-92B8-F2DC5B231294@oracle.com>
	<54BEA7D7.6080008@oracle.com>
	<5BA1E369-ED87-4EBD-8408-B73B726D91BD@oracle.com>
	<54C66E4E.9050805@oracle.com>
	<915998BE-25E9-4196-BAC7-FE5527E10F83@oracle.com>
	<54C7B73F.50404@oracle.com>
	<8AD9A8CC-E570-4DE6-ABB1-10B00FACB8AB@oracle.com>
	<54C8A547.6050607@oracle.com>
Message-ID: 

On Jan 28, 2015, at 1:00 AM, Vladimir Ivanov  wrote:
> I polished the change a little according to your comments (diff against v03):
> http://cr.openjdk.java.net/~vlivanov/8063137/webrev.03-04/hotspot 

+1  Glad to see the AndI folds up easily; thanks for the cleanup.

From lance.andersen at oracle.com  Wed Jan 28 20:54:37 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Wed, 28 Jan 2015 15:54:37 -0500
Subject: Review request for JDK-8051709: Convert JAXP function tests:
	javax.xml.datatype to jtreg (testng) tests
In-Reply-To: <025b01d03ae9$15027d20$3f077760$@oracle.com>
References: <00b801d03933$4d0d3190$e72794b0$@oracle.com>
	<5A84A3CE-440F-49BC-87F6-50AF44B8D80E@oracle.com>
	<03d201d03a15$4bace290$e306a7b0$@oracle.com>
	<073FA30A-D79C-4F65-AB22-B0C23756C5AA@oracle.com>
	<025b01d03ae9$15027d20$3f077760$@oracle.com>
Message-ID: <8A015CB5-75A5-40B1-BB4D-631CB792815E@oracle.com>

Hi Frank

Overall, it is fine, a couple minor nits:

I thinking some of the comments in both class should be clearer as the comments can be confusing as to what the test should return
---------------
 332 
 333     /**
 334      * Test XMLGregorianCalendar#toString() Bug # 5049528:
 335      * XMLGregorianCalendar.toString throws IllegalStateException
 336      *
 337      */
 338     @Test(dataProvider = "calendar")
 339     public void checkToStringPos(final int year, final int month, final int day, final int hour, final int minute, final int second) {
 340         XMLGregorianCalendar calendar = datatypeFactory.newXMLGregorianCalendar(year, month, day, hour, minute, second, undef, undef);
 341         calendar.toString();
 342     }

-------------

This test is not expecting an IllegalStateException.  I would also probably validate that the value from toString() is valid

another example:

---------
 552     }
 553 
 554     /**
 555      * Test Duration#getXMLSchemaType() throws UnsupportedOperationException.
 556      *
 557      * 

 558      * Bug # 5049544: Duration.getXMLSchemaType throws
 559      * UnsupportedOperationException
 560      *
 561      */


--------

I would just review your comments to make them clearer.


The comment below has a typo:
----------
255     /**
 256      * Test XMLGregorianCalendar#toGregorianCalendar( TimeZone timezone, Locale
 257      * aLocale, XMLGregorianCalendar defaults)
 258      *
 259      * 

 260      * Bug # 5040542: toGregorianCalendar(...) does not use the defaults
 261      * XMLGregorianCalendar paramete
 262      *
 263      */

---------


no need for another review, I would just tweak the comments and push

Best
Lance

On Jan 28, 2015, at 5:56 AM, Frank Yuan  wrote:

> Hi Lance
>  
> I have updated the code as your suggestions, would you like to review it again?http://cr.openjdk.java.net/~fyuan/8051709/webrev.01/
>  
> Best Regards
> Frank
>  
>  
> From: Lance Andersen [mailto:lance.andersen at oracle.com] 
> Sent: Wednesday, January 28, 2015 3:57 AM
> To: Frank Yuan
> Cc: 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> Subject: Re: Review request for JDK-8051709: Convert JAXP function tests: javax.xml.datatype to jtreg (testng) tests
>  
> Hi Frank,
>  
> On Jan 27, 2015, at 4:40 AM, Frank Yuan  wrote:
> 
> 
> Thank you, Lance!
>  
> I applied some experience from your previous comments:)
>  
> I a glad they were useful :-)
> 
>  
> I have a question for your comment, could you check it below in line?
>  
> See below
> 
>  
> Best Regards
> Frank
>  
> From: Lance Andersen [mailto:lance.andersen at oracle.com] 
> Sent: Tuesday, January 27, 2015 3:24 AM
> To: Frank Yuan
> Cc: 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> Subject: Re: Review request for JDK-8051709: Convert JAXP function tests: javax.xml.datatype to jtreg (testng) tests
>  
> Hi Frank,
>  
> I think this looks good. 
>  
>  
> Not sure if you are going to add more tests in the future, but would be good to have tests such as
>  
> new Duration(x.toString()).equals(x)
>  
>  
> Perhaps a few more checks on expected toString() output?.
>  
> //Frank: I will do it
>  
>  
> For XMLGregorianCalendarTest.java,  I would consider at some point adding more permutations of some of the tests that are validating a bugs(now that you are adding this as a new test suite to openjdk)
>  
> //Frank: I am not sure what you mean, which bug do you want me to add test for?
>  
> For example checkIsValid()
>  
> I would add a DataProvider and add more permutations to test so that you can reduce other potential errors..
>  
> Again, a nice to have for a next update.
>  
> The problem I always have when we add one off tests, it becomes very hard to manage your test suite and really understand the quality of your coverage.  Better to enhance existing tests to fill in weaknesses as this helps keep your test suite from getting out of control?
>  
>  
> 
> 
>  
> Best
> Lance
>  
> On Jan 26, 2015, at 1:42 AM, Frank Yuan  wrote:
> 
> 
> 
> Hi, Joe and All
> 
> We are working on moving internal jaxp functional tests to open jdk repo.
> This is the datatype suite. Would you please review these test?  Any comment
> will be appreciated.
> 
> bug: https://bugs.openjdk.java.net/browse/JDK-8051709
> webrev: http://cr.openjdk.java.net/~fyuan/8051709/webrev.00/
> 
> 
> Thanks,
> 
> Frank
> 
>  
> 
> 
> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering 
> 1 Network Drive 
> Burlington, MA 01803
> Lance.Andersen at oracle.com
>  
> 
> 
> 
>  
> 
> 
> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering 
> 1 Network Drive 
> Burlington, MA 01803
> Lance.Andersen at oracle.com
>  
> 
> 



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From martinrb at google.com  Wed Jan 28 20:54:51 2015
From: martinrb at google.com (Martin Buchholz)
Date: Wed, 28 Jan 2015 12:54:51 -0800
Subject: Java cmdline encoding challenges on Windows
In-Reply-To: 
References: 
Message-ID: 

Hi Martin,

This was vaguely on my TODO list 10 years ago.
It makes sense for Microsoft to be funding this sort of improvement.
I'm your cheerleader (but not your reviewer).

See also incomplete work here:
http://bugs.java.com/view_bug.do?bug_id=4519026
http://mail.openjdk.java.net/pipermail/core-libs-dev/2009-March/001231.html

I once wrote,

I give Microsoft credit for doing the Right Thing, namely treating things
like filenames as TEXT with a fixed Unicode encoding instead of an
ambiguous bag of bytes.  But the cost was a 30-year migration for their own
code base and a completely incompatible API from the POSIX world.  Where
Win32 does offer a POSIX API (and this includes main(int argc, char**argv)
!), it is generally an imperfect emulation.  All code interacting with the
OS needs to have a windows-specific version, e.g. have to have a wmain just
for windows.




On Tue, Jan 27, 2015 at 9:21 AM, Martin Sawicki (MS OPEN TECH) <
marcins at microsoft.com> wrote:

> Hello,
> We're proposing an improvement to the OpenJDK intended to fix the
> currently existing problem with handling Unicode parameters on the command
> line in Windows (via cmd.exe), which prevents users for example in China
> from properly passing text strings in their own language via the java.exe
> command line.
>
> We have a code submission figured out and tested internally. I've uploaded
> our webrev package here:
>
> https://openjdkcontrib.blob.core.windows.net/unicodecmd/webrev-20150114.zip
>
> The crux of the change lies in using the "W" (wide character) version of
> the Windows APIs for fetching the command line parameters, rather than the
> "A" (ascii) version. But this code path is taken only when the following
> options are set:
>
> -Dwindows.UnicodeConsole=true - switches on Unicode support in the Windows
> console
> -Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; If
> not specified, UTF-8 is used by default. Ignored when
> windows.UnicodeConsole is not set to true.
>
> We'd appreciate a review and acceptance of this improvement.
>
> And, as this is our first contribution to this sub-project within the
> OpenJDK, I apologize for any steps in the submission process that I may
> have missed here and would appreciate guidance as needed.
>
> Best regards
>
> Martin Sawicki (and Kirk Shoop, and Valeriy Kopylov)
> Microsoft Open Technologies, Inc.
> A subsidiary of Microsoft Corp.
>
>


From lance.andersen at oracle.com  Wed Jan 28 21:46:07 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Wed, 28 Jan 2015 16:46:07 -0500
Subject: Review request for JDK-8051710: Convert JAXP function tests:
	javax.xml.jaxp14.* to jtreg (testng) tests
In-Reply-To: <027e01d03aec$353ff060$9fbfd120$@oracle.com>
References: <03a201d03a10$ef463d70$cdd2b850$@oracle.com>
	<54C848ED.7090108@oracle.com>
	<016401d03aa7$71428a40$53c79ec0$@oracle.com>
	<54C86908.7090207@oracle.com>
	<027e01d03aec$353ff060$9fbfd120$@oracle.com>
Message-ID: 

Hi Frank,

Overall they look fine.  Minor comments that you can address when you have time but should not stop you pushing your changes/additions


TransformTest -  just have setup throw Exception as you really don't need the granularity because your are not doing anything with it
XPathFactoryTest -  Please be consistent as to whether you are using javadoc comments or not to introduce methods (i.e.  clean up old or new tests)
DataProvider  invalid-parameterss - seems to be the same in multiple classes so I would move it to the base class  and override when needed
SchemaFactoryTest, ValidateHandlerTests, ValidatorTest -  When time permits, I would add a brief comment for each test

Best
Lance
On Jan 28, 2015, at 6:18 AM, Frank Yuan  wrote:

> Hi Joe
> 
> I have moved the jaxp14 tests to the corresponding packages, would you like
> to have a check? http://cr.openjdk.java.net/~fyuan/8051710/webrev.01/
> 
> There is an issue, SchemaFactoryTest is merged to
> validation.SchemaFactoryTest, but the validation package(Lance reviewed it)
> is waiting to push. I am not sure how to handle for your convenience, so
> this webrev also includes validation package. Maybe we can wait for pushing
> validation, or push this patch for both jaxp14 and validation if you think
> these code change is ok.
> 
> Best Regards
> Frank
> 
> -----Original Message-----
> From: huizhe wang [mailto:huizhe.wang at oracle.com] 
> Sent: Wednesday, January 28, 2015 12:44 PM
> To: Frank Yuan
> Cc: 'Lance Andersen'; 'Core-Libs-Dev'; 'jibing chen'; 'Gustavo Galimberti';
> sandeep.konchady at oracle.com; 'Alexandre (Shura) Iline'
> Subject: Re: Review request for JDK-8051710: Convert JAXP function tests:
> javax.xml.jaxp14.* to jtreg (testng) tests
> 
> 
> On 1/27/2015 7:06 PM, Frank Yuan wrote:
>> Thank you, Joe.
>> 
>> I will sort the tests as your suggestion, and have 3 questions to
> confirm
>> with you:
>> 1. Should I also sort the gap tests?
>> 2. Should I put astro suite at side of auctionportal?
> 
> Ah, in that case, you may put gap tests alongside the auctionportal as well.
>> 3. If a package has very few test, e.g. I may put XMLEventFactoryTest
> and
>> something else in javax.xml.stream.ptest, is it ok? (I would rename 
>> XMLEventFactoryTest as its small coverage)
> 
> Sounds good to me.
> 
> Thanks,
> Joe
> 
>> 
>> Best Regards
>> Frank
>> 
>> 
>> -----Original Message-----
>> From: huizhe wang [mailto:huizhe.wang at oracle.com]
>> Sent: Wednesday, January 28, 2015 10:27 AM
>> To: Frank Yuan
>> Cc: 'Lance Andersen'; 'Core-Libs-Dev'; 'jibing chen'; 'Gustavo
> Galimberti';
>> sandeep.konchady at oracle.com; 'Alexandre (Shura) Iline'
>> Subject: Re: Review request for JDK-8051710: Convert JAXP function
> tests:
>> javax.xml.jaxp14.* to jtreg (testng) tests
>> 
>> Hi Frank,
>> 
>> Nice refactoring the original tests, esp. the TransformerTest!
>> 
>> jaxp14 is legacy in the jaxp standalone world. While we are at this, 
>> you
> may
>> want to move these tests to their relevant packages since in the JDK
> world,
>> jaxp14 is no longer relevant (jaxp 1.4 was integrated into JDK 6).  As 
>> you've already split FactoryTest into various Factory tests, you may
> find
>> them a bit thin in terms of test coverage now that they are named 
>> *FactoryTest since they cover just one of the two newInstance methods. 
>> I would think it makes sense to move them into / combine with the 
>> Factory tests of their own packages.
>> 
>> Thanks,
>> Joe
>> 
>> On 1/27/2015 1:09 AM, Frank Yuan wrote:
>>> Hi, Joe, Lance and All
>>> 
>>> We are working on moving internal jaxp functional tests to open jdk
>> repo.
>>> This is the jaxp14 suite. Would you please review these test?  Any
>> comment
>>> will be appreciated.
>>> 
>>> bug: https://bugs.openjdk.java.net/browse/JDK-8051710
>>> webrev: http://cr.openjdk.java.net/~fyuan/8051710/webrev.00/
>>> 
>>> 
>>> Thanks,
>>> 
>>> Frank
>>> 
>> 
> 
> 



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From lance.andersen at oracle.com  Wed Jan 28 21:57:25 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Wed, 28 Jan 2015 16:57:25 -0500
Subject: Review request for JDK-8052401: JAXP function gap tests conversion
In-Reply-To: <025901d03ae8$312ef8d0$938cea70$@oracle.com>
References: <03af01d03a12$0ac947d0$205bd770$@oracle.com>
	<54C86729.20706@oracle.com>
	<025901d03ae8$312ef8d0$938cea70$@oracle.com>
Message-ID: <75768B9B-A5A9-4DEB-8186-CCE631701C47@oracle.com>

Hi Frank,

The tests look fine.  



> Thank you, Joe.
> 
> I have moved gatest to test.gatest, would you like to check again?
> http://cr.openjdk.java.net/~fyuan/8052401/webrev.01/
> 
> If the code has no problem, would you like to be the sponsor?
> 
> Best Regards
> Frank
> 
> -----Original Message-----
> From: huizhe wang [mailto:huizhe.wang at oracle.com] 
> Sent: Wednesday, January 28, 2015 12:36 PM
> To: Frank Yuan
> Cc: 'Lance Andersen'; 'Core-Libs-Dev'; 'jibing chen'; 'Gustavo Galimberti';
> sandeep.konchady at oracle.com; 'Alexandre (Shura) Iline'
> Subject: Re: Review request for JDK-8052401: JAXP function gap tests
> conversion
> 
> Looks good.
> 
> Joe
> 
> On 1/27/2015 1:17 AM, Frank Yuan wrote:
>> Hi, Joe, Lance and All
>> 
>> We are working on moving internal jaxp functional tests to open jdk
> repo.
>> This is the gaptest suite. Would you please review these test?  Any
> comment
>> will be appreciated.
>> 
>> bug: https://bugs.openjdk.java.net/browse/JDK-8052401
>> webrev: http://cr.openjdk.java.net/~fyuan/8052401/webrev.00/
>> 
>> 
>> Thanks,
>> 
>> Frank
>> 
> 
> 



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From kumar.x.srinivasan at oracle.com  Wed Jan 28 22:40:30 2015
From: kumar.x.srinivasan at oracle.com (Kumar Srinivasan)
Date: Wed, 28 Jan 2015 14:40:30 -0800
Subject: Java cmdline encoding challenges on Windows
In-Reply-To: <54C92AAD.1050006@Oracle.com>
References: 
	<54C92AAD.1050006@Oracle.com>
Message-ID: <54C9655E.2020000@oracle.com>

Roger,

There is an existing Enhancement request for this:
https://bugs.openjdk.java.net/browse/JDK-8029584

Kumar


On 1/28/2015 10:30 AM, Roger Riggs wrote:
> Hi Martin,
>
> Excellent, command line parsing of non-ascii has been a issuefor a while.
>
> Is the patch small enough to include in email?
> Sorry, but links to non-OpenJDK servers are not  covered by the 
> OpenJDK agreements.
> And the openjdk mailer strips off attachments as a defense mechanism.
>
> Is there a Jira issue for this improvement?  Do you have a 
> bugs.openjdk.java.net login?
> Having an issue to attach the webrev to would clear up the technical 
> issues
>
> Thanks, Roger
>
>
>
> On 1/27/2015 12:21 PM, Martin Sawicki (MS OPEN TECH) wrote:
>> Hello,
>> We're proposing an improvement to the OpenJDK intended to fix the 
>> currently existing problem with handling Unicode parameters on the 
>> command line in Windows (via cmd.exe), which prevents users for 
>> example in China from properly passing text strings in their own 
>> language via the java.exe command line.
>>
>> We have a code submission figured out and tested internally. I've 
>> uploaded our webrev package here:
>>
>> https://openjdkcontrib.blob.core.windows.net/unicodecmd/webrev-20150114.zip 
>>
>>
>> The crux of the change lies in using the "W" (wide character) version 
>> of the Windows APIs for fetching the command line parameters, rather 
>> than the "A" (ascii) version. But this code path is taken only when 
>> the following options are set:
>>
>> -Dwindows.UnicodeConsole=true - switches on Unicode support in the 
>> Windows console
>> -Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; 
>> If not specified, UTF-8 is used by default. Ignored when 
>> windows.UnicodeConsole is not set to true.
>>
>> We'd appreciate a review and acceptance of this improvement.
>>
>> And, as this is our first contribution to this sub-project within the 
>> OpenJDK, I apologize for any steps in the submission process that I 
>> may have missed here and would appreciate guidance as needed.
>>
>> Best regards
>>
>> Martin Sawicki (and Kirk Shoop, and Valeriy Kopylov)
>> Microsoft Open Technologies, Inc.
>> A subsidiary of Microsoft Corp.
>>
>



From huizhe.wang at oracle.com  Wed Jan 28 23:33:52 2015
From: huizhe.wang at oracle.com (huizhe wang)
Date: Wed, 28 Jan 2015 15:33:52 -0800
Subject: Review request for JDK-8051547: Convert JAXP function tests:
	javax.xml.validation.* to jtreg (testng) tests
In-Reply-To: <026c01d03ae9$3e1d4140$ba57c3c0$@oracle.com>
References: <007601d03917$ff381180$fda83480$@oracle.com>
	<1EA0C710-63B0-4299-AAC1-8C711FB9E059@oracle.com>
	<026c01d03ae9$3e1d4140$ba57c3c0$@oracle.com>
Message-ID: <54C971E0.7070407@oracle.com>

Hi Frank,

I noticed that the test "Schema01" is not in the webrev. Did you remove 
it? It's not a problem since what the two test cases did were covered by 
the Validator/ValidatorHandler tests. So, just wanted to make sure.

Thanks,
Joe

On 1/28/2015 2:57 AM, Frank Yuan wrote:
>
> Thank you!
>
> Then would you like to be the sponsor for this code change?
>
> Best Regards
>
> Frank
>
> *From:*Lance Andersen [mailto:lance.andersen at oracle.com]
> *Sent:* Tuesday, January 27, 2015 5:08 AM
> *To:* Frank Yuan
> *Cc:* 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> *Subject:* Re: Review request for JDK-8051547: Convert JAXP function 
> tests: javax.xml.validation.* to jtreg (testng) tests
>
> Hi Frank,
>
> This looks pretty good overall.
>
> Best
>
> Lance
>
> On Jan 25, 2015, at 10:27 PM, Frank Yuan  > wrote:
>
>
>
>     Hi, Joe and All
>
>     We are working on moving internal jaxp functional tests to open
>     jdk repo.
>     This is the validation suite. Would you please review these test?  Any
>     comment will be appreciated.
>
>     bug: https://bugs.openjdk.java.net/browse/JDK-8051547
>     webrev: http://cr.openjdk.java.net/~fyuan/8051547/webrev.00/
>     
>
>
>     Thanks,
>
>     Frank
>
> 
>
>
> Lance 
> Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering
> 1 Network Drive
> Burlington, MA 01803
> Lance.Andersen at oracle.com 
>
>
>



From stuart.marks at oracle.com  Thu Jan 29 01:36:57 2015
From: stuart.marks at oracle.com (Stuart Marks)
Date: Wed, 28 Jan 2015 17:36:57 -0800
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C8FB35.1010400@oracle.com>
References: <54C7160D.4000802@oracle.com> <54C8FB35.1010400@oracle.com>
Message-ID: <54C98EB9.8070907@oracle.com>

On 1/28/15 7:07 AM, Alan Bateman wrote:
> On 27/01/2015 04:37, Mandy Chung wrote:
>> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
>> and this method is inherently unsafe.  I am thinking to propose this method
>> in JDK 9 to throw UnsupportedOperationException.
>>
>> I believe it's rare for existing applications using System.runFinalizationOnExit.
>> My analysis on Maven Central ~315K artifacts that show about ~15 unique
>> artifacts calling System.runFinalizationOnExit while they all come from
>> only 5 classes.
>>
>> Any thought/feedback?
> It's broken in other ways beyond what is in the @deprecated note so I don't
> think it's much of a loss to finally disable it. I don't know if you have come
> up with candidate wording to replace the existing wording but having it
> reference the shutdown hooks and the ref API might be useful.

As I understand it, it's not System.runFinalizationOnExit() *itself* that's 
unsafe, it's the act of running finalizers on still-reachable objects at exit 
time that's unsafe.

A potential alternative would thus be to change the method call to be a no-op, 
and just disable the running of finalizers at exit time. This would let us 
remove the unsafe behavior, and it avoids forcing people to recompile code that 
calls rFOE.

Clearly, this would be a silent change in behavior, which is usually something 
we want to avoid. But the calling code can't tell the difference anyway. This is 
to be traded off against potential forced recompiles, which don't add much value 
beyond notification, but which may be inconvenient or even impractical in some 
cases. I'm not sure of the right tradeoff here.

s'marks


From amy.lu at oracle.com  Thu Jan 29 03:09:03 2015
From: amy.lu at oracle.com (Amy Lu)
Date: Thu, 29 Jan 2015 11:09:03 +0800
Subject: JDK 9 RFR of JDK-8069255: Suppress deprecation warnings in
	jdk.rmic module
In-Reply-To: <54BCA4A5.30202@oracle.com>
References: <54BCA4A5.30202@oracle.com>
Message-ID: <54C9A44F.6030603@oracle.com>

I updated the webrev, suppress deprecation warnings for 
jdk/src/jdk.rmic/share/classes/sun/tools/java/* now also covered.

Please review: http://cr.openjdk.java.net/~amlu/8069255/webrev.01/

Thanks,
Amy

On 1/19/15 2:31 PM, Amy Lu wrote:
> Please review these changes to address a few stray deprecation 
> warnings in the jdk.rmic module:
>
> JDK-8069255: Suppress deprecation warnings in jdk.rmic module
> http://cr.openjdk.java.net/~amlu/8069255/webrev.00/
>
> I also need a sponsor for this.
>
> Thanks,
> Amy
>



From john.r.rose at oracle.com  Thu Jan 29 03:10:34 2015
From: john.r.rose at oracle.com (John Rose)
Date: Wed, 28 Jan 2015 19:10:34 -0800
Subject: [9] RFR (XS): 8071787: Don't block inlining when
	DONT_INLINE_THRESHOLD=0
In-Reply-To: <54C91877.5040707@oracle.com>
References: <54C91877.5040707@oracle.com>
Message-ID: 

Good.  Consider fixing the typo in 'makeBlockInlningWrapper'.  ? John

On Jan 28, 2015, at 9:12 AM, Vladimir Ivanov  wrote:
> 
> http://cr.openjdk.java.net/~vlivanov/8071787/webrev.00/
> https://bugs.openjdk.java.net/browse/JDK-8071787
> 
> For testing & performance measurements, sometimes it's useful to replace block inlining wrappers with trivial reinvokers.
> 
> This change extends DONT_INLINE_THRESHOLD in the following manner:
>  DONT_INLINE_THRESHOLD = -1: no wrapper
>  DONT_INLINE_THRESHOLD =  0: reinvoker
>  DONT_INLINE_THRESHOLD >  0: counting wrapper
> 
> Before that DONT_INLINE_THRESHOLD=0 meant a counting wrapper which is removed on the first invocation. After the change, it's DONT_INLINE_THRESHOLD=1.
> 
> Testing: manual, java/lang/invoke
> 
> Best regards,
> Vladimir Ivanov



From john.r.rose at oracle.com  Thu Jan 29 03:11:49 2015
From: john.r.rose at oracle.com (John Rose)
Date: Wed, 28 Jan 2015 19:11:49 -0800
Subject: [9] RFR (XXS): 8071788: CountingWrapper.asType() is broken
In-Reply-To: <54C91AF1.3010602@oracle.com>
References: <54C91AF1.3010602@oracle.com>
Message-ID: <53D3F321-0259-4878-9767-EA909EF90810@oracle.com>

Good.

On Jan 28, 2015, at 9:22 AM, Vladimir Ivanov  wrote:
> 
> The fix is to use adapted MethodHandle to construct LambdaForm.



From mandy.chung at oracle.com  Thu Jan 29 03:43:18 2015
From: mandy.chung at oracle.com (Mandy Chung)
Date: Wed, 28 Jan 2015 19:43:18 -0800
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C98EB9.8070907@oracle.com>
References: <54C7160D.4000802@oracle.com> <54C8FB35.1010400@oracle.com>
	<54C98EB9.8070907@oracle.com>
Message-ID: <54C9AC56.1030704@oracle.com>


On 1/28/2015 5:36 PM, Stuart Marks wrote:
> On 1/28/15 7:07 AM, Alan Bateman wrote:
>> On 27/01/2015 04:37, Mandy Chung wrote:
>>> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
>>> and this method is inherently unsafe.  I am thinking to propose this 
>>> method
>>> in JDK 9 to throw UnsupportedOperationException.
>>>
>>> I believe it's rare for existing applications using 
>>> System.runFinalizationOnExit.
>>> My analysis on Maven Central ~315K artifacts that show about ~15 unique
>>> artifacts calling System.runFinalizationOnExit while they all come from
>>> only 5 classes.
>>>
>>> Any thought/feedback?
>> It's broken in other ways beyond what is in the @deprecated note so I 
>> don't
>> think it's much of a loss to finally disable it. I don't know if you 
>> have come
>> up with candidate wording to replace the existing wording but having it
>> reference the shutdown hooks and the ref API might be useful.
>
> As I understand it, it's not System.runFinalizationOnExit() *itself* 
> that's unsafe, it's the act of running finalizers on still-reachable 
> objects at exit time that's unsafe.
>
> A potential alternative would thus be to change the method call to be 
> a no-op, and just disable the running of finalizers at exit time. This 
> would let us remove the unsafe behavior, and it avoids forcing people 
> to recompile code that calls rFOE.
>
> Clearly, this would be a silent change in behavior, which is usually 
> something we want to avoid. But the calling code can't tell the 
> difference anyway. This is to be traded off against potential forced 
> recompiles, which don't add much value beyond notification, but which 
> may be inconvenient or even impractical in some cases. I'm not sure of 
> the right tradeoff here.

Throwing UOE is intentional so that applications depending on existing 
behavior will be modified not to use this deprecated method.   Making it 
a no-op makes it harder to diagnose for applications depending on the 
finalizers to be invoked on exit to free resources.  I opt to throw UOE 
(or remove the method that will leave it for another discussion some 
other day).

Mandy



From huizhe.wang at oracle.com  Thu Jan 29 05:42:02 2015
From: huizhe.wang at oracle.com (huizhe wang)
Date: Wed, 28 Jan 2015 21:42:02 -0800
Subject: Review request for JDK-8051547: Convert JAXP function tests:
	javax.xml.validation.* to jtreg (testng) tests
In-Reply-To: <006a01d03b69$d20963e0$761c2ba0$@oracle.com>
References: <007601d03917$ff381180$fda83480$@oracle.com>
	<1EA0C710-63B0-4299-AAC1-8C711FB9E059@oracle.com>
	<026c01d03ae9$3e1d4140$ba57c3c0$@oracle.com>
	<54C971E0.7070407@oracle.com>
	<006a01d03b69$d20963e0$761c2ba0$@oracle.com>
Message-ID: <54C9C82A.9080500@oracle.com>

I see, it's all good then.

Joe

On 1/28/2015 6:17 PM, Frank Yuan wrote:
>
> Actually, Schema01 is commented out from TestValidation.java although 
> Schema01.java exists in sqe repo.
>
> Best Regards
>
> Frank
>
> *From:*huizhe wang [mailto:huizhe.wang at oracle.com]
> *Sent:* Thursday, January 29, 2015 7:34 AM
> *To:* Frank Yuan
> *Cc:* 'Lance Andersen'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> *Subject:* Re: Review request for JDK-8051547: Convert JAXP function 
> tests: javax.xml.validation.* to jtreg (testng) tests
>
> Hi Frank,
>
> I noticed that the test "Schema01" is not in the webrev. Did you 
> remove it? It's not a problem since what the two test cases did were 
> covered by the Validator/ValidatorHandler tests. So, just wanted to 
> make sure.
>
> Thanks,
> Joe
>
> On 1/28/2015 2:57 AM, Frank Yuan wrote:
>
>     Thank you!
>
>     Then would you like to be the sponsor for this code change?
>
>     Best Regards
>
>     Frank
>
>     *From:*Lance Andersen [mailto:lance.andersen at oracle.com]
>     *Sent:* Tuesday, January 27, 2015 5:08 AM
>     *To:* Frank Yuan
>     *Cc:* 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
>     *Subject:* Re: Review request for JDK-8051547: Convert JAXP
>     function tests: javax.xml.validation.* to jtreg (testng) tests
>
>     Hi Frank,
>
>     This looks pretty good overall.
>
>     Best
>
>     Lance
>
>     On Jan 25, 2015, at 10:27 PM, Frank Yuan      > wrote:
>
>
>
>
>         Hi, Joe and All
>
>         We are working on moving internal jaxp functional tests to
>         open jdk repo.
>         This is the validation suite. Would you please review these
>         test?  Any
>         comment will be appreciated.
>
>         bug: https://bugs.openjdk.java.net/browse/JDK-8051547
>         webrev: http://cr.openjdk.java.net/~fyuan/8051547/webrev.00/
>         
>
>
>         Thanks,
>
>         Frank
>
>     
>
>
>     Lance
>     Andersen| Principal Member of Technical Staff | +1.781.442.2037
>     Oracle Java Engineering
>     1 Network Drive
>     Burlington, MA 01803
>     Lance.Andersen at oracle.com 
>
>
>
>



From peter.levart at gmail.com  Thu Jan 29 07:06:53 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Thu, 29 Jan 2015 08:06:53 +0100
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C9AC56.1030704@oracle.com>
References: <54C7160D.4000802@oracle.com>
	<54C8FB35.1010400@oracle.com>	<54C98EB9.8070907@oracle.com>
	<54C9AC56.1030704@oracle.com>
Message-ID: <54C9DC0D.3030206@gmail.com>

On 01/29/2015 04:43 AM, Mandy Chung wrote:
>
> On 1/28/2015 5:36 PM, Stuart Marks wrote:
>> On 1/28/15 7:07 AM, Alan Bateman wrote:
>>> On 27/01/2015 04:37, Mandy Chung wrote:
>>>> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
>>>> and this method is inherently unsafe.  I am thinking to propose 
>>>> this method
>>>> in JDK 9 to throw UnsupportedOperationException.
>>>>
>>>> I believe it's rare for existing applications using 
>>>> System.runFinalizationOnExit.
>>>> My analysis on Maven Central ~315K artifacts that show about ~15 
>>>> unique
>>>> artifacts calling System.runFinalizationOnExit while they all come 
>>>> from
>>>> only 5 classes.
>>>>
>>>> Any thought/feedback?
>>> It's broken in other ways beyond what is in the @deprecated note so 
>>> I don't
>>> think it's much of a loss to finally disable it. I don't know if you 
>>> have come
>>> up with candidate wording to replace the existing wording but having it
>>> reference the shutdown hooks and the ref API might be useful.
>>
>> As I understand it, it's not System.runFinalizationOnExit() *itself* 
>> that's unsafe, it's the act of running finalizers on still-reachable 
>> objects at exit time that's unsafe.
>>
>> A potential alternative would thus be to change the method call to be 
>> a no-op, and just disable the running of finalizers at exit time. 
>> This would let us remove the unsafe behavior, and it avoids forcing 
>> people to recompile code that calls rFOE.
>>
>> Clearly, this would be a silent change in behavior, which is usually 
>> something we want to avoid. But the calling code can't tell the 
>> difference anyway. This is to be traded off against potential forced 
>> recompiles, which don't add much value beyond notification, but which 
>> may be inconvenient or even impractical in some cases. I'm not sure 
>> of the right tradeoff here.
>
> Throwing UOE is intentional so that applications depending on existing 
> behavior will be modified not to use this deprecated method.   Making 
> it a no-op makes it harder to diagnose for applications depending on 
> the finalizers to be invoked on exit to free resources.  I opt to 
> throw UOE (or remove the method that will leave it for another 
> discussion some other day).
>
> Mandy
>

Wouldn't it be possible to make the method safe?

Finalizers are usually run against un-reachable objects. If the Shutdown 
sequence made sure all threads are stopped except the thread executing 
Shutdown sequence, then all objects would become effectively unreachable 
(except being reachable from shutdown thread that can assume the role of 
finalizer thread and run the finalizers...).

Peter



From david.holmes at oracle.com  Thu Jan 29 07:34:49 2015
From: david.holmes at oracle.com (David Holmes)
Date: Thu, 29 Jan 2015 17:34:49 +1000
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C9DC0D.3030206@gmail.com>
References: <54C7160D.4000802@oracle.com>	<54C8FB35.1010400@oracle.com>	<54C98EB9.8070907@oracle.com>	<54C9AC56.1030704@oracle.com>
	<54C9DC0D.3030206@gmail.com>
Message-ID: <54C9E299.9040000@oracle.com>

On 29/01/2015 5:06 PM, Peter Levart wrote:
> On 01/29/2015 04:43 AM, Mandy Chung wrote:
>>
>> On 1/28/2015 5:36 PM, Stuart Marks wrote:
>>> On 1/28/15 7:07 AM, Alan Bateman wrote:
>>>> On 27/01/2015 04:37, Mandy Chung wrote:
>>>>> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
>>>>> and this method is inherently unsafe.  I am thinking to propose
>>>>> this method
>>>>> in JDK 9 to throw UnsupportedOperationException.
>>>>>
>>>>> I believe it's rare for existing applications using
>>>>> System.runFinalizationOnExit.
>>>>> My analysis on Maven Central ~315K artifacts that show about ~15
>>>>> unique
>>>>> artifacts calling System.runFinalizationOnExit while they all come
>>>>> from
>>>>> only 5 classes.
>>>>>
>>>>> Any thought/feedback?
>>>> It's broken in other ways beyond what is in the @deprecated note so
>>>> I don't
>>>> think it's much of a loss to finally disable it. I don't know if you
>>>> have come
>>>> up with candidate wording to replace the existing wording but having it
>>>> reference the shutdown hooks and the ref API might be useful.
>>>
>>> As I understand it, it's not System.runFinalizationOnExit() *itself*
>>> that's unsafe, it's the act of running finalizers on still-reachable
>>> objects at exit time that's unsafe.
>>>
>>> A potential alternative would thus be to change the method call to be
>>> a no-op, and just disable the running of finalizers at exit time.
>>> This would let us remove the unsafe behavior, and it avoids forcing
>>> people to recompile code that calls rFOE.
>>>
>>> Clearly, this would be a silent change in behavior, which is usually
>>> something we want to avoid. But the calling code can't tell the
>>> difference anyway. This is to be traded off against potential forced
>>> recompiles, which don't add much value beyond notification, but which
>>> may be inconvenient or even impractical in some cases. I'm not sure
>>> of the right tradeoff here.
>>
>> Throwing UOE is intentional so that applications depending on existing
>> behavior will be modified not to use this deprecated method.   Making
>> it a no-op makes it harder to diagnose for applications depending on
>> the finalizers to be invoked on exit to free resources.  I opt to
>> throw UOE (or remove the method that will leave it for another
>> discussion some other day).
>>
>> Mandy
>>
>
> Wouldn't it be possible to make the method safe?
>
> Finalizers are usually run against un-reachable objects. If the Shutdown
> sequence made sure all threads are stopped except the thread executing

How do you propose to make sure all threads are "stopped"? That would 
potentially introduce more problems  and strange new behaviours.

Cheers,
David


> Shutdown sequence, then all objects would become effectively unreachable
> (except being reachable from shutdown thread that can assume the role of
> finalizer thread and run the finalizers...).
>
> Peter
>


From peter.firmstone at zeus.net.au  Thu Jan 29 09:08:24 2015
From: peter.firmstone at zeus.net.au (Peter Firmstone)
Date: Thu, 29 Jan 2015 19:08:24 +1000
Subject: Explicit Serialization API and Security
In-Reply-To: <54C25DB0.4080207@oracle.com>
References: <54B0CE06.2020408@zeus.net.au>
	<54B3EEC5.5040805@oracle.com>		<1421144693.7530.20.camel@Nokia-N900>
	<54B654F6.6060207@oracle.com>	<1421240334.8856.17.camel@Nokia-N900>	<54BEB91A.4020603@gmail.com>
	<54BFD374.5080803@oracle.com>	<54C00B66.5090302@zeus.net.au>
	<54C01D71.9010901@redhat.com> <54C25DB0.4080207@oracle.com>
Message-ID: <54C9F888.2040005@zeus.net.au>

Although not directly relevant to this discussion, here are some 
functional examples of deserialization constructors, I now have a fully 
functional validating ObjectInputStream.

Unfortunately in our project we have intra object dependencies as 
demonstrated by this example; a static validator, without a constructor 
falls short.

I also have an interface that provides direct access to the 
ObjectInputStream before the deserializing object has been constructed.

The following two classes are a good example as they show how to check 
invariants and provide full backward compatibility with existing Serial 
Form.  In this case validation is provided for unauthenticated network 
connections, so the classes also implement standard serialization for 
trusted connections, where we can relax validation.

Regards,

Peter.

/*
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
  * distributed with this work for additional information
  * regarding copyright ownership. The ASF licenses this file
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License. You may obtain a copy of the License at
  *
  *      http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
package com.sun.jini.reggie;

import com.sun.jini.proxy.MarshalledWrapper;
import java.io.IOException;
import java.io.InvalidObjectException;
import java.io.ObjectInput;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.ObjectStreamException;
import java.io.Serializable;
import java.rmi.MarshalledObject;
import java.rmi.RemoteException;
import java.rmi.UnmarshalException;
import java.util.logging.Level;
import java.util.logging.Logger;
import net.jini.admin.Administrable;
import net.jini.core.constraint.RemoteMethodControl;
import net.jini.core.discovery.LookupLocator;
import net.jini.core.event.EventRegistration;
import net.jini.core.event.RemoteEventListener;
import net.jini.core.lookup.ServiceID;
import net.jini.core.lookup.ServiceItem;
import net.jini.core.lookup.ServiceMatches;
import net.jini.core.lookup.ServiceRegistrar;
import net.jini.core.lookup.ServiceRegistration;
import net.jini.core.lookup.ServiceTemplate;
import net.jini.id.ReferentUuid;
import net.jini.id.ReferentUuids;
import net.jini.id.Uuid;
import net.jini.id.UuidFactory;
import org.apache.river.api.io.AtomicSerial;
import org.apache.river.api.io.AtomicSerial.GetArg;
import org.apache.river.api.io.AtomicSerial.ReadInput;
import org.apache.river.api.io.AtomicSerial.ReadObject;

/**
  * A RegistrarProxy is a proxy for a registrar.  Clients only see instances
  * via the ServiceRegistrar, Administrable and ReferentUuid interfaces.
  *
  * @author Sun Microsystems, Inc.
  *
  */
@AtomicSerial
class RegistrarProxy
     implements ServiceRegistrar, Administrable, ReferentUuid, Serializable
{
     private static final long serialVersionUID = 2L;

     private static final Logger logger =
     Logger.getLogger("com.sun.jini.reggie");

     /**
      * The registrar.
      *
      * @serial
      */
     final Registrar server;
     /**
      * The registrar's service ID.
      */
     transient ServiceID registrarID;

     /**
      * Returns RegistrarProxy or ConstrainableRegistrarProxy instance,
      * depending on whether given server implements RemoteMethodControl.
      */
     static RegistrarProxy getInstance(Registrar server,
                       ServiceID registrarID)
     {
     return (server instanceof RemoteMethodControl) ?
         new ConstrainableRegistrarProxy(server, registrarID, null) :
         new RegistrarProxy(server, registrarID);
     }

     @ReadInput
     private static RO getRO(){
     return new RO();
     }

     private static boolean check(GetArg arg) throws IOException{
     Registrar server = (Registrar) arg.get("server", null);
     if (server == null) throw new InvalidObjectException("null server");
     RO r = (RO) arg.getReader();
     if (r.registrarID == null) throw new InvalidObjectException("null 
ServiceID");
     return true;
     }

     RegistrarProxy(GetArg arg) throws IOException{
     this(arg, check(arg));
     }

     RegistrarProxy(GetArg arg, boolean check) throws IOException{
     server = (Registrar) arg.get("server", null);
     RO r = (RO) arg.getReader();
     registrarID = r.registrarID;
     }

     /** Constructor for use by getInstance(), 
ConstrainableRegistrarProxy. */
     RegistrarProxy(Registrar server, ServiceID registrarID) {
     this.server = server;
     this.registrarID = registrarID;
     }

     // Inherit javadoc
     @Override
     public Object getAdmin() throws RemoteException {
         return server.getAdmin();
     }

     // Inherit javadoc
     @Override
     public ServiceRegistration register(ServiceItem srvItem,
                     long leaseDuration)
     throws RemoteException
     {
     Item item = new Item(srvItem);
     if (item.serviceID != null) {
         Util.checkRegistrantServiceID(
         item.serviceID, logger, Level.WARNING);
     }
     return server.register(item, leaseDuration);
     }

     // Inherit javadoc
     @Override
     public Object lookup(ServiceTemplate tmpl) throws RemoteException {
     MarshalledWrapper wrapper = server.lookup(new Template(tmpl));
     if (wrapper == null)
         return null;
     try {
         return wrapper.get();
     } catch (IOException e) {
         throw new UnmarshalException("error unmarshalling return", e);
     } catch (ClassNotFoundException e) {
         throw new UnmarshalException("error unmarshalling return", e);
     }
     }

     // Inherit javadoc
     @Override
     public ServiceMatches lookup(ServiceTemplate tmpl, int maxMatches)
     throws RemoteException
     {
     return server.lookup(new Template(tmpl), maxMatches).get();
     }

     // Inherit javadoc
     @Override
     public EventRegistration notify(ServiceTemplate tmpl,
                     int transitions,
                     RemoteEventListener listener,
                     MarshalledObject handback,
                     long leaseDuration)
     throws RemoteException
     {
     return server.notify(new Template(tmpl), transitions, listener,
                  handback, leaseDuration);
     }

     // Inherit javadoc
     @Override
     public Class[] getEntryClasses(ServiceTemplate tmpl)
     throws RemoteException
     {
     return EntryClassBase.toClass(
                   server.getEntryClasses(new Template(tmpl)));
     }

     // Inherit javadoc
     @Override
     public Object[] getFieldValues(ServiceTemplate tmpl,
                    int setIndex, String field)
     throws NoSuchFieldException, RemoteException
     {
     /* check that setIndex and field are valid, convert field to index */
     ClassMapper.EntryField[] efields =
         ClassMapper.getFields(
                  tmpl.attributeSetTemplates[setIndex].getClass());
     int fidx;
     for (fidx = efields.length; --fidx >= 0; ) {
         if (field.equals(efields[fidx].field.getName()))
         break;
     }
     if (fidx < 0)
         throw new NoSuchFieldException(field);
     Object[] values = server.getFieldValues(new Template(tmpl),
                         setIndex, fidx);
     /* unmarshal each value, replacing with null on exception */
     if (values != null && efields[fidx].marshal) {
         for (int i = values.length; --i >= 0; ) {
         try {
             values[i] = ((MarshalledWrapper) values[i]).get();
             continue;
         } catch (Throwable e) {
             handleException(e);
         }
         values[i] = null;
         }
     }
     return values;
     }

     /**
      * Rethrow the exception if it is an Error, unless it is a 
LinkageError,
      * OutOfMemoryError, or StackOverflowError.  Otherwise print the
      * exception stack trace if debugging is enabled.
      */
     static void handleException(final Throwable e) {
     if (e instanceof Error &&
         !(e instanceof LinkageError ||
           e instanceof OutOfMemoryError ||
           e instanceof StackOverflowError))
     {
         throw (Error)e;
     }
     logger.log(Level.INFO, "unmarshalling failure", e);
     }

     // Inherit javadoc
     @Override
     public Class[] getServiceTypes(ServiceTemplate tmpl, String prefix)
     throws RemoteException
     {
     return ServiceTypeBase.toClass(
                    server.getServiceTypes(new Template(tmpl),
                               prefix));
     }

     @Override
     public ServiceID getServiceID() {
     return registrarID;
     }

     // Inherit javadoc
     @Override
     public LookupLocator getLocator() throws RemoteException {
     return server.getLocator();
     }

     // Inherit javadoc
     @Override
     public String[] getGroups() throws RemoteException {
     return server.getMemberGroups();
     }

     // Inherit javadoc
     @Override
     public Uuid getReferentUuid() {
     return UuidFactory.create(registrarID.getMostSignificantBits(),
                   registrarID.getLeastSignificantBits());
     }

     // Inherit javadoc
     @Override
     public int hashCode() {
     return registrarID.hashCode();
     }

     /** Proxies for servers with the same service ID are considered 
equal. */
     @Override
     public boolean equals(Object obj) {
     return ReferentUuids.compare(this, obj);
     }

     /**
      * Returns a string created from the proxy class name, the registrar's
      * service ID, and the result of the underlying proxy's toString 
method.
      *
      * @return String
      */
     @Override
     public String toString() {
     return this.getClass().getName() + "[registrar=" + registrarID
         + " " + server + "]";
     }

     /**
      * Writes the default serializable field value for this instance, 
followed
      * by the registrar's service ID encoded as specified by the
      * ServiceID.writeBytes method.
      */
     private void writeObject(ObjectOutputStream out) throws IOException {
     out.defaultWriteObject();
     registrarID.writeBytes(out);
     }

     /**
      * Reads the default serializable field value for this instance, 
followed
      * by the registrar's service ID encoded as specified by the
      * ServiceID.writeBytes method.  Verifies that the deserialized 
registrar
      * reference is non-null.
      */
     private void readObject(ObjectInputStream in)
     throws IOException, ClassNotFoundException
     {
     in.defaultReadObject();
     registrarID = new ServiceID(in);
     if (server == null) {
         throw new InvalidObjectException("null server");
     }
     }

     /**
      * Throws InvalidObjectException, since data for this class is 
required.
      */
     private void readObjectNoData() throws ObjectStreamException {
     throw new InvalidObjectException("no data");
     }

     private static class RO implements ReadObject{

     ServiceID registrarID;

     @Override
     public void read(ObjectInput in) throws IOException, 
ClassNotFoundException {
         registrarID = new ServiceID(in);
     }

     }
}

  /*
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
  * distributed with this work for additional information
  * regarding copyright ownership. The ASF licenses this file
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License. You may obtain a copy of the License at
  *
  *      http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
package com.sun.jini.reggie;

import com.sun.jini.proxy.ConstrainableProxyUtil;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.lang.reflect.Method;
import java.rmi.MarshalledObject;
import net.jini.admin.Administrable;
import net.jini.core.constraint.MethodConstraints;
import net.jini.core.constraint.RemoteMethodControl;
import net.jini.core.event.RemoteEventListener;
import net.jini.core.lookup.ServiceID;
import net.jini.core.lookup.ServiceItem;
import net.jini.core.lookup.ServiceRegistrar;
import net.jini.core.lookup.ServiceTemplate;
import net.jini.security.proxytrust.ProxyTrustIterator;
import net.jini.security.proxytrust.SingletonProxyTrustIterator;
import org.apache.river.api.io.AtomicSerial;
import org.apache.river.api.io.AtomicSerial.GetArg;

/**
  * RegistrarProxy subclass that supports constraints.
  *
  * @author Sun Microsystems, Inc.
  *
  */
@AtomicSerial
final class ConstrainableRegistrarProxy
     extends RegistrarProxy implements RemoteMethodControl
{
     private static final long serialVersionUID = 2L;

     /** Mappings between ServiceRegistrar and Registrar methods */
     static final Method[] methodMappings = {
     Util.getMethod(ServiceRegistrar.class, "getEntryClasses",
                new Class[]{ ServiceTemplate.class }),
     Util.getMethod(Registrar.class, "getEntryClasses",
                new Class[]{ Template.class }),

     Util.getMethod(ServiceRegistrar.class, "getFieldValues",
                new Class[]{ ServiceTemplate.class, int.class,
                     String.class }),
     Util.getMethod(Registrar.class, "getFieldValues",
                new Class[]{ Template.class, int.class, int.class }),

     Util.getMethod(ServiceRegistrar.class, "getGroups", new Class[0]),
     Util.getMethod(Registrar.class, "getMemberGroups", new Class[0]),

     Util.getMethod(ServiceRegistrar.class, "getLocator", new Class[0]),
     Util.getMethod(Registrar.class, "getLocator", new Class[0]),

     Util.getMethod(ServiceRegistrar.class, "getServiceTypes",
                new Class[]{ ServiceTemplate.class, String.class }),
     Util.getMethod(Registrar.class, "getServiceTypes",
                new Class[]{ Template.class, String.class }),

     Util.getMethod(ServiceRegistrar.class, "lookup",
                new Class[]{ ServiceTemplate.class }),
     Util.getMethod(Registrar.class, "lookup",
                new Class[]{ Template.class }),

     Util.getMethod(ServiceRegistrar.class, "lookup",
                new Class[]{ ServiceTemplate.class, int.class }),
     Util.getMethod(Registrar.class, "lookup",
                new Class[]{ Template.class, int.class }),

     Util.getMethod(ServiceRegistrar.class, "notify",
                new Class[]{ ServiceTemplate.class, int.class,
                     RemoteEventListener.class,
                     MarshalledObject.class, long.class }),
     Util.getMethod(Registrar.class, "notify",
                new Class[]{ Template.class, int.class,
                     RemoteEventListener.class,
                     MarshalledObject.class, long.class }),

     Util.getMethod(ServiceRegistrar.class, "register",
                new Class[]{ ServiceItem.class, long.class }),
     Util.getMethod(Registrar.class, "register",
                new Class[]{ Item.class, long.class }),

     Util.getMethod(Administrable.class, "getAdmin", new Class[0]),
     Util.getMethod(Administrable.class, "getAdmin", new Class[0])
     };

     /** Client constraints for this proxy, or null */
     private final MethodConstraints constraints;

     private static GetArg check(GetArg arg) throws IOException{
     RegistrarProxy sup = new RegistrarProxy(arg);
     MethodConstraints constraints = (MethodConstraints) 
arg.get("constraints", null);
     ConstrainableProxyUtil.verifyConsistentConstraints(
         constraints, sup.server, methodMappings);
     return arg;
     }

     ConstrainableRegistrarProxy(GetArg arg) throws IOException{
     super(check(arg));
     constraints = (MethodConstraints) arg.get("constraints", null);
     }

     /**
      * Creates new ConstrainableRegistrarProxy with given server reference,
      * service ID and client constraints.
      */
     ConstrainableRegistrarProxy(Registrar server,
                 ServiceID registrarID,
                 MethodConstraints constraints)
     {
     super((Registrar) ((RemoteMethodControl) server).setConstraints(
           ConstrainableProxyUtil.translateConstraints(
               constraints, methodMappings)),
           registrarID);
     this.constraints = constraints;
     }

     // javadoc inherited from RemoteMethodControl.setConstraints
     public RemoteMethodControl setConstraints(MethodConstraints 
constraints) {
     return new ConstrainableRegistrarProxy(
         server, registrarID, constraints);
     }

     // javadoc inherited from RemoteMethodControl.getConstraints
     public MethodConstraints getConstraints() {
     return constraints;
     }

     /**
      * Returns iterator used by ProxyTrustVerifier to retrieve a trust 
verifier
      * for this object.
      */
     private ProxyTrustIterator getProxyTrustIterator() {
     return new SingletonProxyTrustIterator(server);
     }

     /**
      * Verifies that the client constraints for this proxy are 
consistent with
      * those set on the underlying server ref.
      */
     private void readObject(ObjectInputStream in)
     throws IOException, ClassNotFoundException
     {
     in.defaultReadObject();
     ConstrainableProxyUtil.verifyConsistentConstraints(
         constraints, server, methodMappings);
     }
}


From peter.firmstone at zeus.net.au  Thu Jan 29 09:22:41 2015
From: peter.firmstone at zeus.net.au (Peter Firmstone)
Date: Thu, 29 Jan 2015 19:22:41 +1000
Subject: Explicit Serialization API and Security
In-Reply-To: <54C9F888.2040005@zeus.net.au>
References: <54B0CE06.2020408@zeus.net.au>
	<54B3EEC5.5040805@oracle.com>		<1421144693.7530.20.camel@Nokia-N900>
	<54B654F6.6060207@oracle.com>	<1421240334.8856.17.camel@Nokia-N900>	<54BEB91A.4020603@gmail.com>
	<54BFD374.5080803@oracle.com>	<54C00B66.5090302@zeus.net.au>
	<54C01D71.9010901@redhat.com> <54C25DB0.4080207@oracle.com>
	<54C9F888.2040005@zeus.net.au>
Message-ID: <54C9FBE1.1070702@zeus.net.au>

Just a quick note, to avoid stream corruption, object that aren't 
instantiated are replaced by an enum marker, Reference.DISCARDED, and 
null is returned in their place, fields are read and any trailling 
writeObject data is discarded.   Object that are not allowed to be 
deserialized are still read in, just not created.

This way, a stream may be able to avoid a MIM attack, by discarding 
unvalidated data inserted into the stream.  Altough it would be better 
to use a secure connection, there is a brief period where a proxy is 
used to establish trust with a third party service provider, after being 
obtained from a lookup service, this was premised on java's secure 
sandbox and serialization, when it was believed to be secure.

Regards,

Peter.

On 29/01/2015 7:08 PM, Peter Firmstone wrote:
> Although not directly relevant to this discussion, here are some 
> functional examples of deserialization constructors, I now have a 
> fully functional validating ObjectInputStream.
>
> Unfortunately in our project we have intra object dependencies as 
> demonstrated by this example; a static validator, without a 
> constructor falls short.
>
> I also have an interface that provides direct access to the 
> ObjectInputStream before the deserializing object has been constructed.
>
> The following two classes are a good example as they show how to check 
> invariants and provide full backward compatibility with existing 
> Serial Form.  In this case validation is provided for unauthenticated 
> network connections, so the classes also implement standard 
> serialization for trusted connections, where we can relax validation.
>
> Regards,
>
> Peter.
>
> /*
>  * Licensed to the Apache Software Foundation (ASF) under one
>  * or more contributor license agreements.  See the NOTICE file
>  * distributed with this work for additional information
>  * regarding copyright ownership. The ASF licenses this file
>  * to you under the Apache License, Version 2.0 (the
>  * "License"); you may not use this file except in compliance
>  * with the License. You may obtain a copy of the License at
>  *
>  *      http://www.apache.org/licenses/LICENSE-2.0
>  *
>  * Unless required by applicable law or agreed to in writing, software
>  * distributed under the License is distributed on an "AS IS" BASIS,
>  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
> implied.
>  * See the License for the specific language governing permissions and
>  * limitations under the License.
>  */
> package com.sun.jini.reggie;
>
> import com.sun.jini.proxy.MarshalledWrapper;
> import java.io.IOException;
> import java.io.InvalidObjectException;
> import java.io.ObjectInput;
> import java.io.ObjectInputStream;
> import java.io.ObjectOutputStream;
> import java.io.ObjectStreamException;
> import java.io.Serializable;
> import java.rmi.MarshalledObject;
> import java.rmi.RemoteException;
> import java.rmi.UnmarshalException;
> import java.util.logging.Level;
> import java.util.logging.Logger;
> import net.jini.admin.Administrable;
> import net.jini.core.constraint.RemoteMethodControl;
> import net.jini.core.discovery.LookupLocator;
> import net.jini.core.event.EventRegistration;
> import net.jini.core.event.RemoteEventListener;
> import net.jini.core.lookup.ServiceID;
> import net.jini.core.lookup.ServiceItem;
> import net.jini.core.lookup.ServiceMatches;
> import net.jini.core.lookup.ServiceRegistrar;
> import net.jini.core.lookup.ServiceRegistration;
> import net.jini.core.lookup.ServiceTemplate;
> import net.jini.id.ReferentUuid;
> import net.jini.id.ReferentUuids;
> import net.jini.id.Uuid;
> import net.jini.id.UuidFactory;
> import org.apache.river.api.io.AtomicSerial;
> import org.apache.river.api.io.AtomicSerial.GetArg;
> import org.apache.river.api.io.AtomicSerial.ReadInput;
> import org.apache.river.api.io.AtomicSerial.ReadObject;
>
> /**
>  * A RegistrarProxy is a proxy for a registrar.  Clients only see 
> instances
>  * via the ServiceRegistrar, Administrable and ReferentUuid interfaces.
>  *
>  * @author Sun Microsystems, Inc.
>  *
>  */
> @AtomicSerial
> class RegistrarProxy
>     implements ServiceRegistrar, Administrable, ReferentUuid, 
> Serializable
> {
>     private static final long serialVersionUID = 2L;
>
>     private static final Logger logger =
>     Logger.getLogger("com.sun.jini.reggie");
>
>     /**
>      * The registrar.
>      *
>      * @serial
>      */
>     final Registrar server;
>     /**
>      * The registrar's service ID.
>      */
>     transient ServiceID registrarID;
>
>     /**
>      * Returns RegistrarProxy or ConstrainableRegistrarProxy instance,
>      * depending on whether given server implements RemoteMethodControl.
>      */
>     static RegistrarProxy getInstance(Registrar server,
>                       ServiceID registrarID)
>     {
>     return (server instanceof RemoteMethodControl) ?
>         new ConstrainableRegistrarProxy(server, registrarID, null) :
>         new RegistrarProxy(server, registrarID);
>     }
>
>     @ReadInput
>     private static RO getRO(){
>     return new RO();
>     }
>
>     private static boolean check(GetArg arg) throws IOException{
>     Registrar server = (Registrar) arg.get("server", null);
>     if (server == null) throw new InvalidObjectException("null server");
>     RO r = (RO) arg.getReader();
>     if (r.registrarID == null) throw new InvalidObjectException("null 
> ServiceID");
>     return true;
>     }
>
>     RegistrarProxy(GetArg arg) throws IOException{
>     this(arg, check(arg));
>     }
>
>     RegistrarProxy(GetArg arg, boolean check) throws IOException{
>     server = (Registrar) arg.get("server", null);
>     RO r = (RO) arg.getReader();
>     registrarID = r.registrarID;
>     }
>
>     /** Constructor for use by getInstance(), 
> ConstrainableRegistrarProxy. */
>     RegistrarProxy(Registrar server, ServiceID registrarID) {
>     this.server = server;
>     this.registrarID = registrarID;
>     }
>
>     // Inherit javadoc
>     @Override
>     public Object getAdmin() throws RemoteException {
>         return server.getAdmin();
>     }
>
>     // Inherit javadoc
>     @Override
>     public ServiceRegistration register(ServiceItem srvItem,
>                     long leaseDuration)
>     throws RemoteException
>     {
>     Item item = new Item(srvItem);
>     if (item.serviceID != null) {
>         Util.checkRegistrantServiceID(
>         item.serviceID, logger, Level.WARNING);
>     }
>     return server.register(item, leaseDuration);
>     }
>
>     // Inherit javadoc
>     @Override
>     public Object lookup(ServiceTemplate tmpl) throws RemoteException {
>     MarshalledWrapper wrapper = server.lookup(new Template(tmpl));
>     if (wrapper == null)
>         return null;
>     try {
>         return wrapper.get();
>     } catch (IOException e) {
>         throw new UnmarshalException("error unmarshalling return", e);
>     } catch (ClassNotFoundException e) {
>         throw new UnmarshalException("error unmarshalling return", e);
>     }
>     }
>
>     // Inherit javadoc
>     @Override
>     public ServiceMatches lookup(ServiceTemplate tmpl, int maxMatches)
>     throws RemoteException
>     {
>     return server.lookup(new Template(tmpl), maxMatches).get();
>     }
>
>     // Inherit javadoc
>     @Override
>     public EventRegistration notify(ServiceTemplate tmpl,
>                     int transitions,
>                     RemoteEventListener listener,
>                     MarshalledObject handback,
>                     long leaseDuration)
>     throws RemoteException
>     {
>     return server.notify(new Template(tmpl), transitions, listener,
>                  handback, leaseDuration);
>     }
>
>     // Inherit javadoc
>     @Override
>     public Class[] getEntryClasses(ServiceTemplate tmpl)
>     throws RemoteException
>     {
>     return EntryClassBase.toClass(
>                   server.getEntryClasses(new Template(tmpl)));
>     }
>
>     // Inherit javadoc
>     @Override
>     public Object[] getFieldValues(ServiceTemplate tmpl,
>                    int setIndex, String field)
>     throws NoSuchFieldException, RemoteException
>     {
>     /* check that setIndex and field are valid, convert field to index */
>     ClassMapper.EntryField[] efields =
>         ClassMapper.getFields(
>                  tmpl.attributeSetTemplates[setIndex].getClass());
>     int fidx;
>     for (fidx = efields.length; --fidx >= 0; ) {
>         if (field.equals(efields[fidx].field.getName()))
>         break;
>     }
>     if (fidx < 0)
>         throw new NoSuchFieldException(field);
>     Object[] values = server.getFieldValues(new Template(tmpl),
>                         setIndex, fidx);
>     /* unmarshal each value, replacing with null on exception */
>     if (values != null && efields[fidx].marshal) {
>         for (int i = values.length; --i >= 0; ) {
>         try {
>             values[i] = ((MarshalledWrapper) values[i]).get();
>             continue;
>         } catch (Throwable e) {
>             handleException(e);
>         }
>         values[i] = null;
>         }
>     }
>     return values;
>     }
>
>     /**
>      * Rethrow the exception if it is an Error, unless it is a 
> LinkageError,
>      * OutOfMemoryError, or StackOverflowError.  Otherwise print the
>      * exception stack trace if debugging is enabled.
>      */
>     static void handleException(final Throwable e) {
>     if (e instanceof Error &&
>         !(e instanceof LinkageError ||
>           e instanceof OutOfMemoryError ||
>           e instanceof StackOverflowError))
>     {
>         throw (Error)e;
>     }
>     logger.log(Level.INFO, "unmarshalling failure", e);
>     }
>
>     // Inherit javadoc
>     @Override
>     public Class[] getServiceTypes(ServiceTemplate tmpl, String prefix)
>     throws RemoteException
>     {
>     return ServiceTypeBase.toClass(
>                    server.getServiceTypes(new Template(tmpl),
>                               prefix));
>     }
>
>     @Override
>     public ServiceID getServiceID() {
>     return registrarID;
>     }
>
>     // Inherit javadoc
>     @Override
>     public LookupLocator getLocator() throws RemoteException {
>     return server.getLocator();
>     }
>
>     // Inherit javadoc
>     @Override
>     public String[] getGroups() throws RemoteException {
>     return server.getMemberGroups();
>     }
>
>     // Inherit javadoc
>     @Override
>     public Uuid getReferentUuid() {
>     return UuidFactory.create(registrarID.getMostSignificantBits(),
>                   registrarID.getLeastSignificantBits());
>     }
>
>     // Inherit javadoc
>     @Override
>     public int hashCode() {
>     return registrarID.hashCode();
>     }
>
>     /** Proxies for servers with the same service ID are considered 
> equal. */
>     @Override
>     public boolean equals(Object obj) {
>     return ReferentUuids.compare(this, obj);
>     }
>
>     /**
>      * Returns a string created from the proxy class name, the 
> registrar's
>      * service ID, and the result of the underlying proxy's toString 
> method.
>      *
>      * @return String
>      */
>     @Override
>     public String toString() {
>     return this.getClass().getName() + "[registrar=" + registrarID
>         + " " + server + "]";
>     }
>
>     /**
>      * Writes the default serializable field value for this instance, 
> followed
>      * by the registrar's service ID encoded as specified by the
>      * ServiceID.writeBytes method.
>      */
>     private void writeObject(ObjectOutputStream out) throws IOException {
>     out.defaultWriteObject();
>     registrarID.writeBytes(out);
>     }
>
>     /**
>      * Reads the default serializable field value for this instance, 
> followed
>      * by the registrar's service ID encoded as specified by the
>      * ServiceID.writeBytes method.  Verifies that the deserialized 
> registrar
>      * reference is non-null.
>      */
>     private void readObject(ObjectInputStream in)
>     throws IOException, ClassNotFoundException
>     {
>     in.defaultReadObject();
>     registrarID = new ServiceID(in);
>     if (server == null) {
>         throw new InvalidObjectException("null server");
>     }
>     }
>
>     /**
>      * Throws InvalidObjectException, since data for this class is 
> required.
>      */
>     private void readObjectNoData() throws ObjectStreamException {
>     throw new InvalidObjectException("no data");
>     }
>
>     private static class RO implements ReadObject{
>
>     ServiceID registrarID;
>
>     @Override
>     public void read(ObjectInput in) throws IOException, 
> ClassNotFoundException {
>         registrarID = new ServiceID(in);
>     }
>
>     }
> }
>
>  /*
>  * Licensed to the Apache Software Foundation (ASF) under one
>  * or more contributor license agreements.  See the NOTICE file
>  * distributed with this work for additional information
>  * regarding copyright ownership. The ASF licenses this file
>  * to you under the Apache License, Version 2.0 (the
>  * "License"); you may not use this file except in compliance
>  * with the License. You may obtain a copy of the License at
>  *
>  *      http://www.apache.org/licenses/LICENSE-2.0
>  *
>  * Unless required by applicable law or agreed to in writing, software
>  * distributed under the License is distributed on an "AS IS" BASIS,
>  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
> implied.
>  * See the License for the specific language governing permissions and
>  * limitations under the License.
>  */
> package com.sun.jini.reggie;
>
> import com.sun.jini.proxy.ConstrainableProxyUtil;
> import java.io.IOException;
> import java.io.ObjectInputStream;
> import java.lang.reflect.Method;
> import java.rmi.MarshalledObject;
> import net.jini.admin.Administrable;
> import net.jini.core.constraint.MethodConstraints;
> import net.jini.core.constraint.RemoteMethodControl;
> import net.jini.core.event.RemoteEventListener;
> import net.jini.core.lookup.ServiceID;
> import net.jini.core.lookup.ServiceItem;
> import net.jini.core.lookup.ServiceRegistrar;
> import net.jini.core.lookup.ServiceTemplate;
> import net.jini.security.proxytrust.ProxyTrustIterator;
> import net.jini.security.proxytrust.SingletonProxyTrustIterator;
> import org.apache.river.api.io.AtomicSerial;
> import org.apache.river.api.io.AtomicSerial.GetArg;
>
> /**
>  * RegistrarProxy subclass that supports constraints.
>  *
>  * @author Sun Microsystems, Inc.
>  *
>  */
> @AtomicSerial
> final class ConstrainableRegistrarProxy
>     extends RegistrarProxy implements RemoteMethodControl
> {
>     private static final long serialVersionUID = 2L;
>
>     /** Mappings between ServiceRegistrar and Registrar methods */
>     static final Method[] methodMappings = {
>     Util.getMethod(ServiceRegistrar.class, "getEntryClasses",
>                new Class[]{ ServiceTemplate.class }),
>     Util.getMethod(Registrar.class, "getEntryClasses",
>                new Class[]{ Template.class }),
>
>     Util.getMethod(ServiceRegistrar.class, "getFieldValues",
>                new Class[]{ ServiceTemplate.class, int.class,
>                     String.class }),
>     Util.getMethod(Registrar.class, "getFieldValues",
>                new Class[]{ Template.class, int.class, int.class }),
>
>     Util.getMethod(ServiceRegistrar.class, "getGroups", new Class[0]),
>     Util.getMethod(Registrar.class, "getMemberGroups", new Class[0]),
>
>     Util.getMethod(ServiceRegistrar.class, "getLocator", new Class[0]),
>     Util.getMethod(Registrar.class, "getLocator", new Class[0]),
>
>     Util.getMethod(ServiceRegistrar.class, "getServiceTypes",
>                new Class[]{ ServiceTemplate.class, String.class }),
>     Util.getMethod(Registrar.class, "getServiceTypes",
>                new Class[]{ Template.class, String.class }),
>
>     Util.getMethod(ServiceRegistrar.class, "lookup",
>                new Class[]{ ServiceTemplate.class }),
>     Util.getMethod(Registrar.class, "lookup",
>                new Class[]{ Template.class }),
>
>     Util.getMethod(ServiceRegistrar.class, "lookup",
>                new Class[]{ ServiceTemplate.class, int.class }),
>     Util.getMethod(Registrar.class, "lookup",
>                new Class[]{ Template.class, int.class }),
>
>     Util.getMethod(ServiceRegistrar.class, "notify",
>                new Class[]{ ServiceTemplate.class, int.class,
>                     RemoteEventListener.class,
>                     MarshalledObject.class, long.class }),
>     Util.getMethod(Registrar.class, "notify",
>                new Class[]{ Template.class, int.class,
>                     RemoteEventListener.class,
>                     MarshalledObject.class, long.class }),
>
>     Util.getMethod(ServiceRegistrar.class, "register",
>                new Class[]{ ServiceItem.class, long.class }),
>     Util.getMethod(Registrar.class, "register",
>                new Class[]{ Item.class, long.class }),
>
>     Util.getMethod(Administrable.class, "getAdmin", new Class[0]),
>     Util.getMethod(Administrable.class, "getAdmin", new Class[0])
>     };
>
>     /** Client constraints for this proxy, or null */
>     private final MethodConstraints constraints;
>
>     private static GetArg check(GetArg arg) throws IOException{
>     RegistrarProxy sup = new RegistrarProxy(arg);
>     MethodConstraints constraints = (MethodConstraints) 
> arg.get("constraints", null);
>     ConstrainableProxyUtil.verifyConsistentConstraints(
>         constraints, sup.server, methodMappings);
>     return arg;
>     }
>
>     ConstrainableRegistrarProxy(GetArg arg) throws IOException{
>     super(check(arg));
>     constraints = (MethodConstraints) arg.get("constraints", null);
>     }
>
>     /**
>      * Creates new ConstrainableRegistrarProxy with given server 
> reference,
>      * service ID and client constraints.
>      */
>     ConstrainableRegistrarProxy(Registrar server,
>                 ServiceID registrarID,
>                 MethodConstraints constraints)
>     {
>     super((Registrar) ((RemoteMethodControl) server).setConstraints(
>           ConstrainableProxyUtil.translateConstraints(
>               constraints, methodMappings)),
>           registrarID);
>     this.constraints = constraints;
>     }
>
>     // javadoc inherited from RemoteMethodControl.setConstraints
>     public RemoteMethodControl setConstraints(MethodConstraints 
> constraints) {
>     return new ConstrainableRegistrarProxy(
>         server, registrarID, constraints);
>     }
>
>     // javadoc inherited from RemoteMethodControl.getConstraints
>     public MethodConstraints getConstraints() {
>     return constraints;
>     }
>
>     /**
>      * Returns iterator used by ProxyTrustVerifier to retrieve a trust 
> verifier
>      * for this object.
>      */
>     private ProxyTrustIterator getProxyTrustIterator() {
>     return new SingletonProxyTrustIterator(server);
>     }
>
>     /**
>      * Verifies that the client constraints for this proxy are 
> consistent with
>      * those set on the underlying server ref.
>      */
>     private void readObject(ObjectInputStream in)
>     throws IOException, ClassNotFoundException
>     {
>     in.defaultReadObject();
>     ConstrainableProxyUtil.verifyConsistentConstraints(
>         constraints, server, methodMappings);
>     }
> }



From peter.levart at gmail.com  Thu Jan 29 09:26:41 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Thu, 29 Jan 2015 10:26:41 +0100
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C9E299.9040000@oracle.com>
References: <54C7160D.4000802@oracle.com>	<54C8FB35.1010400@oracle.com>	<54C98EB9.8070907@oracle.com>	<54C9AC56.1030704@oracle.com>
	<54C9DC0D.3030206@gmail.com> <54C9E299.9040000@oracle.com>
Message-ID: <54C9FCD1.50204@gmail.com>

On 01/29/2015 08:34 AM, David Holmes wrote:
>> Wouldn't it be possible to make the method safe?
>>
>> Finalizers are usually run against un-reachable objects. If the Shutdown
>> sequence made sure all threads are stopped except the thread executing
>
> How do you propose to make sure all threads are "stopped"? That would 
> potentially introduce more problems  and strange new behaviours.

Yes, that would be a problem. Calling Thread.stop() for them would be 
dangerous. Even if all threads were captured in a special kind of 
safepoint except the shutdown thread, it would not be safe since 
finalizers could try to enter synchronized blocks that are held by the 
captured threads and a deadlock would occur.

  It seems there's no way out.

Regards, Peter

>
> Cheers,
> David
>
>
>> Shutdown sequence, then all objects would become effectively unreachable
>> (except being reachable from shutdown thread that can assume the role of
>> finalizer thread and run the finalizers...).
>>
>> Peter 



From martin.desruisseaux at geomatys.fr  Thu Jan 29 09:37:06 2015
From: martin.desruisseaux at geomatys.fr (Martin Desruisseaux)
Date: Thu, 29 Jan 2015 10:37:06 +0100
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C9AC56.1030704@oracle.com>
References: <54C7160D.4000802@oracle.com>
	<54C8FB35.1010400@oracle.com>	<54C98EB9.8070907@oracle.com>
	<54C9AC56.1030704@oracle.com>
Message-ID: <54C9FF42.9040300@geomatys.fr>

Le 29/01/15 04:43, Mandy Chung a ?crit :
> Throwing UOE is intentional so that applications depending on existing
> behavior will be modified not to use this deprecated method.   Making
> it a no-op makes it harder to diagnose for applications depending on
> the finalizers to be invoked on exit to free resources.  I opt to
> throw UOE (or remove the method that will leave it for another
> discussion some other day).

Maybe one or two years ago, I saw a similar discussion about "almost
removing" a method from the JDK. Some proposed to define a new
annotation in addition to @Deprecated, something like @Retired, which
would cause the javac and javadoc compilers to behave as if the method
has been removed while the method would still be available at runtime
for a transition period. Would it be something worth to consider? (or
alternatively an optional argument to @Deprecated?)

    Martin



From peter.levart at gmail.com  Thu Jan 29 09:38:01 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Thu, 29 Jan 2015 10:38:01 +0100
Subject: Q: 8071326: ThreadPoolExecutor in endless thread creation loop
	if workQueue.take() throws RuntimeException
In-Reply-To: 
References: <54C7B216.90404@oracle.com>		<54C7FD28.8070708@oracle.com>
	<54C862ED.3040404@oracle.com>	<54C8D5B6.6070504@oracle.com>
	
Message-ID: <54C9FF79.3020303@gmail.com>

On 01/28/2015 06:35 PM, Martin Buchholz wrote:
> It's hard for me to think of something we could add to the javadoc that
> would actually help future users enough to offset the confusion of adding
> subtleties of rarely encountered behavior.  I also don't see an easy way to
> improve the pool's reaction to exceptions coming from the queue.  Right now
> the reporting mechanism is the uncaught exception handler, which is under
> the user's control, although it is not obvious.

Well, there already exists these paragraphs in the class-level 
ThreadPoolExecutor javadoc:

  * 
Hook methods

  *
  * 
This class provides {@code protected} overridable
  * {@link #beforeExecute(Thread, Runnable)} and
  * {@link #afterExecute(Runnable, Throwable)} methods that are called
  * before and after execution of each task.  These can be used to
  * manipulate the execution environment; for example, reinitializing
  * ThreadLocals, gathering statistics, or adding log entries.
  * Additionally, method {@link #terminated} can be overridden to perform
  * any special processing that needs to be done once the Executor has
  * fully terminated.
  *
  * 
If hook or callback methods throw exceptions, internal worker
  * threads may in turn fail and abruptly terminate.



The last paragraph could explicitly spell-out what are the "callback" 
methods. That would be enough, I think.


Regards, peter

>
> On Wed, Jan 28, 2015 at 4:27 AM, Lev Priima  wrote:
>
>> Thanks Doug, David, Martin, especially Martin.
>> Is it worth to update javadoc of  ThreadPoolExecutor#Queuing section with
>> this caveat ?
>>
>> The original confusion in custom queue implementation raise up from
>> javadoc, because BlockingQueue.take() interface specification does not
>> prohibit explicitly to throw uncaught runtime exception/errors (as any
>> other casual java code). But using this method in an exhaustive resource
>> allocation loop obliges to deal with exceptional situations in
>> work-producing methods more carefully.
>>
>> Best Regards,
>> Lev
>>
>>
>> On 28.01.2015 7:17, David Holmes wrote:
>>
>>> On 28/01/2015 7:03 AM, Lev Priima wrote:
>>>
>>>> Yes. And if we have BlockingQueue w/ some amount of tasks which fail
>>>> with exceptions, same amount of threads(not limited by neither
>>>> maximumPoolSize/corePoolSize) will hang under TPE which takes tasks from
>>>> this queue.
>>>>
>>>> It may cause problems if queue has a big percentage of exception-fail
>>>> tasks and we eventually get OOME while unable to create new native
>>>> thread.
>>>>
>>> If you use your pathological example then of course you can get into a
>>> situation where the thread creation outpaces the thread termination - it
>>> takes time and CPU cycles for a thread to actually complete.
>>>
>>> A BlockingQueue implementation should not have an expected failure mode
>>> that results in regularly throwing Errors or RuntimeExceptions. Such a BQ
>>> implementation would need to be fixed in my opinion.
>>>
>>> The TPE is working as designed - if errors/runtime-exceptions are
>>> encountered the worker thread will terminate and be replaced by a fresh
>>> worker. If you keep feeding the worker threads such exceptions then you
>>> incur a high rate of thread churn. So don't do that. :)
>>>
>>> Cheers,
>>> David
>>>
>>>   Lev
>>>> On 01/27/2015 11:31 PM, Martin Buchholz wrote:
>>>>
>>>>>
>>>>> On Tue, Jan 27, 2015 at 7:43 AM, Lev Priima >>>> > wrote:
>>>>>
>>>>>      And these thread will be cleaned only when whole TPE finished.
>>>>>
>>>>>
>>>>> Is this really true?  Each thread should be replaced while running and
>>>>> so the total number of threads retained by the TPE at any one time
>>>>> should be no more than core pool size.
>>>>>
>>>>



From peter.firmstone at zeus.net.au  Thu Jan 29 09:40:55 2015
From: peter.firmstone at zeus.net.au (Peter Firmstone)
Date: Thu, 29 Jan 2015 19:40:55 +1000
Subject: Explicit Serialization API and Security
In-Reply-To: <54C9FBE1.1070702@zeus.net.au>
References: <54B0CE06.2020408@zeus.net.au>
	<54B3EEC5.5040805@oracle.com>		<1421144693.7530.20.camel@Nokia-N900>
	<54B654F6.6060207@oracle.com>	<1421240334.8856.17.camel@Nokia-N900>	<54BEB91A.4020603@gmail.com>
	<54BFD374.5080803@oracle.com>	<54C00B66.5090302@zeus.net.au>
	<54C01D71.9010901@redhat.com> <54C25DB0.4080207@oracle.com>
	<54C9F888.2040005@zeus.net.au> <54C9FBE1.1070702@zeus.net.au>
Message-ID: <54CA0027.2000104@zeus.net.au>

Logging output from failed validation:

NonActGrp-out: Jan 29, 2015 7:36:16 PM 
net.jini.jeri.BasicInvocationHandler invoke
NonActGrp-out: Jan 29, 2015 7:36:16 PM 
net.jini.jeri.BasicInvocationHandler invoke
NonActGrp-out: FAILED: outbound call 
net.jini.core.event.RemoteEventListener.notify remotely throws
NonActGrp-out: FAILED: outbound call 
net.jini.core.event.RemoteEventListener.notify remotely throws
NonActGrp-out: java.rmi.ServerException: RemoteException in server 
thread; nested exception is:
NonActGrp-out: java.rmi.ServerException: RemoteException in server 
thread; nested exception is:
NonActGrp-out:     java.rmi.UnmarshalException: unmarshalling 
method/arguments; nested exception is:
NonActGrp-out:     java.rmi.UnmarshalException: unmarshalling 
method/arguments; nested exception is:
NonActGrp-out:     java.io.InvalidObjectException: name must be equal to 
Object when superclass is null
NonActGrp-out:     java.io.InvalidObjectException: name must be equal to 
Object when superclass is null
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationDispatcher.dispatch(BasicInvocationDispatcher.java:646)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target$2.run(Target.java:493)
NonActGrp-out:     at 
net.jini.export.ServerContext.doWithServerContext(ServerContext.java:108)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:490)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.access$000(Target.java:57)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target$1.run(Target.java:466)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:463)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:428)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.DgcRequestDispatcher.dispatch(DgcRequestDispatcher.java:210)
NonActGrp-out:     at 
net.jini.jeri.connection.ServerConnectionManager$Dispatcher.dispatch(ServerConnectionManager.java:147)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.mux.MuxServer$1$1.run(MuxServer.java:247)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.mux.MuxServer$1.run(MuxServer.java:243)
NonActGrp-out:     at 
com.sun.jini.thread.ThreadPool$Task.run(ThreadPool.java:215)
NonActGrp-out:     at 
com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:277)
NonActGrp-out:     at java.lang.Thread.run(Thread.java:744)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Util.__________EXCEPTION_RECEIVED_FROM_SERVER__________(Util.java:110)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Util.exceptionReceivedFromServer(Util.java:103)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationHandler.unmarshalThrow(BasicInvocationHandler.java:1320)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:834)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationHandler.java:661)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:530)
NonActGrp-out:     at com.sun.proxy.$Proxy5.notify(Unknown Source)
NonActGrp-out:     at 
com.sun.jini.reggie.RegistrarImpl$EventTask.call(RegistrarImpl.java:2076)
NonActGrp-out:     at 
com.sun.jini.reggie.RegistrarImpl$EventTask.call(RegistrarImpl.java:2032)
NonActGrp-out:     at 
org.apache.river.impl.thread.SynchronousExecutors$Task.call(SynchronousExecutors.java:421)
NonActGrp-out:     at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
NonActGrp-out:     at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
NonActGrp-out:     at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
NonActGrp-out:     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
NonActGrp-out:     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
NonActGrp-out:     at java.lang.Thread.run(Thread.java:744)
NonActGrp-out: Caused by: java.rmi.UnmarshalException: unmarshalling 
method/arguments; nested exception is:
NonActGrp-out: Caused by: java.rmi.UnmarshalException: unmarshalling 
method/arguments; nested exception is:
NonActGrp-out:     java.io.InvalidObjectException: name must be equal to 
Object when superclass is null
NonActGrp-out:     java.io.InvalidObjectException: name must be equal to 
Object when superclass is null
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationDispatcher.dispatch(BasicInvocationDispatcher.java:621)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target$2.run(Target.java:493)
NonActGrp-out:     at 
net.jini.export.ServerContext.doWithServerContext(ServerContext.java:108)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:490)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.access$000(Target.java:57)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target$1.run(Target.java:466)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:463)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:428)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.DgcRequestDispatcher.dispatch(DgcRequestDispatcher.java:210)
NonActGrp-out:     at 
net.jini.jeri.connection.ServerConnectionManager$Dispatcher.dispatch(ServerConnectionManager.java:147)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.mux.MuxServer$1$1.run(MuxServer.java:247)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.mux.MuxServer$1.run(MuxServer.java:243)
NonActGrp-out:     at 
com.sun.jini.thread.ThreadPool$Task.run(ThreadPool.java:215)
NonActGrp-out:     at 
com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:277)
NonActGrp-out:     ... 1 more
NonActGrp-out:     ... 1 more
NonActGrp-out: Caused by: java.io.InvalidObjectException: name must be 
equal to Object when superclass is null
NonActGrp-out: Caused by: java.io.InvalidObjectException: name must be 
equal to Object when superclass is null
NonActGrp-out:     at 
com.sun.jini.reggie.EntryClass.check(EntryClass.java:96)
NonActGrp-out:     at 
com.sun.jini.reggie.EntryClass.(EntryClass.java:102)
NonActGrp-out:     at 
sun.reflect.GeneratedConstructorAccessor30.newInstance()
NonActGrp-out:     at 
sun.reflect.GeneratedConstructorAccessor30.newInstance()
NonActGrp-out:     at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
NonActGrp-out:     at 
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
NonActGrp-out:     at 
org.apache.river.api.io.AtomicSerial$Factory.instantiate(AtomicSerial.java:163)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.instantiateAtomicly(ValidatingObjectInputStream.java:228)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNewObject(ValidatingObjectInputStream.java:2071)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNonPrimitiveContent(ValidatingObjectInputStream.java:773)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObject(ValidatingObjectInputStream.java:2343)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObjectOverride(ValidatingObjectInputStream.java:2299)
NonActGrp-out:     at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFieldValues(ValidatingObjectInputStream.java:974)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFields(ValidatingObjectInputStream.java:924)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.instantiateAtomicly(ValidatingObjectInputStream.java:203)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNewObject(ValidatingObjectInputStream.java:2071)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNonPrimitiveContent(ValidatingObjectInputStream.java:773)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObject(ValidatingObjectInputStream.java:2343)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObjectOverride(ValidatingObjectInputStream.java:2299)
NonActGrp-out:     at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNewArray(ValidatingObjectInputStream.java:1638)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNonPrimitiveContent(ValidatingObjectInputStream.java:770)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObject(ValidatingObjectInputStream.java:2343)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObjectOverride(ValidatingObjectInputStream.java:2299)
NonActGrp-out:     at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFieldValues(ValidatingObjectInputStream.java:974)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFields(ValidatingObjectInputStream.java:924)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.instantiateAtomicly(ValidatingObjectInputStream.java:203)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNewObject(ValidatingObjectInputStream.java:2071)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNonPrimitiveContent(ValidatingObjectInputStream.java:773)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObject(ValidatingObjectInputStream.java:2343)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObjectOverride(ValidatingObjectInputStream.java:2299)
NonActGrp-out:     at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFieldValues(ValidatingObjectInputStream.java:974)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFields(ValidatingObjectInputStream.java:924)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.instantiateAtomicly(ValidatingObjectInputStream.java:203)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNewObject(ValidatingObjectInputStream.java:2100)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNonPrimitiveContent(ValidatingObjectInputStream.java:773)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObject(ValidatingObjectInputStream.java:2343)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObjectOverride(ValidatingObjectInputStream.java:2299)
NonActGrp-out:     at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Util.unmarshalValue(Util.java:223)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationDispatcher.unmarshalArguments(BasicInvocationDispatcher.java:1057)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationDispatcher.dispatch(BasicInvocationDispatcher.java:601)
NonActGrp-out:     ... 16 more
NonActGrp-out:     ... 16 more
NonActGrp-out:
NonActGrp-out:
NonActGrp-out: Jan 29, 2015 7:36:16 PM 
com.sun.jini.reggie.RegistrarImpl$EventTask call
NonActGrp-out: Jan 29, 2015 7:36:16 PM 
com.sun.jini.reggie.RegistrarImpl$EventTask call
NonActGrp-out: INFO: exception sending event
NonActGrp-out: INFO: exception sending event
NonActGrp-out: java.rmi.ServerException: RemoteException in server 
thread; nested exception is:
NonActGrp-out: java.rmi.ServerException: RemoteException in server 
thread; nested exception is:
NonActGrp-out:     java.rmi.UnmarshalException: unmarshalling 
method/arguments; nested exception is:
NonActGrp-out:     java.rmi.UnmarshalException: unmarshalling 
method/arguments; nested exception is:
NonActGrp-out:     java.io.InvalidObjectException: name must be equal to 
Object when superclass is null
NonActGrp-out:     java.io.InvalidObjectException: name must be equal to 
Object when superclass is null
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationDispatcher.dispatch(BasicInvocationDispatcher.java:646)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target$2.run(Target.java:493)
NonActGrp-out:     at 
net.jini.export.ServerContext.doWithServerContext(ServerContext.java:108)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:490)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.access$000(Target.java:57)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target$1.run(Target.java:466)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:463)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:428)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.DgcRequestDispatcher.dispatch(DgcRequestDispatcher.java:210)
NonActGrp-out:     at 
net.jini.jeri.connection.ServerConnectionManager$Dispatcher.dispatch(ServerConnectionManager.java:147)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.mux.MuxServer$1$1.run(MuxServer.java:247)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.mux.MuxServer$1.run(MuxServer.java:243)
NonActGrp-out:     at 
com.sun.jini.thread.ThreadPool$Task.run(ThreadPool.java:215)
NonActGrp-out:     at 
com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:277)
NonActGrp-out:     at java.lang.Thread.run(Thread.java:744)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Util.__________EXCEPTION_RECEIVED_FROM_SERVER__________(Util.java:110)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Util.exceptionReceivedFromServer(Util.java:103)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationHandler.unmarshalThrow(BasicInvocationHandler.java:1320)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:834)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationHandler.java:661)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:530)
NonActGrp-out:     at com.sun.proxy.$Proxy5.notify(Unknown Source)
NonActGrp-out:     at 
com.sun.jini.reggie.RegistrarImpl$EventTask.call(RegistrarImpl.java:2076)
NonActGrp-out:     at 
com.sun.jini.reggie.RegistrarImpl$EventTask.call(RegistrarImpl.java:2032)
NonActGrp-out:     at 
org.apache.river.impl.thread.SynchronousExecutors$Task.call(SynchronousExecutors.java:421)
NonActGrp-out:     at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
NonActGrp-out:     at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
NonActGrp-out:     at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
NonActGrp-out:     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
NonActGrp-out:     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
NonActGrp-out:     at java.lang.Thread.run(Thread.java:744)
NonActGrp-out: Caused by: java.rmi.UnmarshalException: unmarshalling 
method/arguments; nested exception is:
NonActGrp-out: Caused by: java.rmi.UnmarshalException: unmarshalling 
method/arguments; nested exception is:
NonActGrp-out:     java.io.InvalidObjectException: name must be equal to 
Object when superclass is null
NonActGrp-out:     java.io.InvalidObjectException: name must be equal to 
Object when superclass is null
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationDispatcher.dispatch(BasicInvocationDispatcher.java:621)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target$2.run(Target.java:493)
NonActGrp-out:     at 
net.jini.export.ServerContext.doWithServerContext(ServerContext.java:108)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:490)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.access$000(Target.java:57)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target$1.run(Target.java:466)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:463)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Target.dispatch(Target.java:428)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.DgcRequestDispatcher.dispatch(DgcRequestDispatcher.java:210)
NonActGrp-out:     at 
net.jini.jeri.connection.ServerConnectionManager$Dispatcher.dispatch(ServerConnectionManager.java:147)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.mux.MuxServer$1$1.run(MuxServer.java:247)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at java.security.AccessController.doPrivileged(Native 
Method)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.mux.MuxServer$1.run(MuxServer.java:243)
NonActGrp-out:     at 
com.sun.jini.thread.ThreadPool$Task.run(ThreadPool.java:215)
NonActGrp-out:     at 
com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:277)
NonActGrp-out:     ... 1 more
NonActGrp-out:     ... 1 more
NonActGrp-out: Caused by: java.io.InvalidObjectException: name must be 
equal to Object when superclass is null
NonActGrp-out: Caused by: java.io.InvalidObjectException: name must be 
equal to Object when superclass is null
NonActGrp-out:     at 
com.sun.jini.reggie.EntryClass.check(EntryClass.java:96)
NonActGrp-out:     at 
com.sun.jini.reggie.EntryClass.(EntryClass.java:102)
NonActGrp-out:     at 
sun.reflect.GeneratedConstructorAccessor30.newInstance()
NonActGrp-out:     at 
sun.reflect.GeneratedConstructorAccessor30.newInstance()
NonActGrp-out:     at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
NonActGrp-out:     at 
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
NonActGrp-out:     at 
org.apache.river.api.io.AtomicSerial$Factory.instantiate(AtomicSerial.java:163)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.instantiateAtomicly(ValidatingObjectInputStream.java:228)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNewObject(ValidatingObjectInputStream.java:2071)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNonPrimitiveContent(ValidatingObjectInputStream.java:773)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObject(ValidatingObjectInputStream.java:2343)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObjectOverride(ValidatingObjectInputStream.java:2299)
NonActGrp-out:     at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFieldValues(ValidatingObjectInputStream.java:974)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFields(ValidatingObjectInputStream.java:924)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.instantiateAtomicly(ValidatingObjectInputStream.java:203)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNewObject(ValidatingObjectInputStream.java:2071)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNonPrimitiveContent(ValidatingObjectInputStream.java:773)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObject(ValidatingObjectInputStream.java:2343)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObjectOverride(ValidatingObjectInputStream.java:2299)
NonActGrp-out:     at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNewArray(ValidatingObjectInputStream.java:1638)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNonPrimitiveContent(ValidatingObjectInputStream.java:770)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObject(ValidatingObjectInputStream.java:2343)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObjectOverride(ValidatingObjectInputStream.java:2299)
NonActGrp-out:     at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFieldValues(ValidatingObjectInputStream.java:974)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFields(ValidatingObjectInputStream.java:924)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.instantiateAtomicly(ValidatingObjectInputStream.java:203)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNewObject(ValidatingObjectInputStream.java:2071)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNonPrimitiveContent(ValidatingObjectInputStream.java:773)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObject(ValidatingObjectInputStream.java:2343)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObjectOverride(ValidatingObjectInputStream.java:2299)
NonActGrp-out:     at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFieldValues(ValidatingObjectInputStream.java:974)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readFields(ValidatingObjectInputStream.java:924)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.instantiateAtomicly(ValidatingObjectInputStream.java:203)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNewObject(ValidatingObjectInputStream.java:2100)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readNonPrimitiveContent(ValidatingObjectInputStream.java:773)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObject(ValidatingObjectInputStream.java:2343)
NonActGrp-out:     at 
org.apache.river.api.io.ValidatingObjectInputStream.readObjectOverride(ValidatingObjectInputStream.java:2299)
NonActGrp-out:     at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
NonActGrp-out:     at 
com.sun.jini.jeri.internal.runtime.Util.unmarshalValue(Util.java:223)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationDispatcher.unmarshalArguments(BasicInvocationDispatcher.java:1057)
NonActGrp-out:     at 
net.jini.jeri.BasicInvocationDispatcher.dispatch(BasicInvocationDispatcher.java:601)
NonActGrp-out:     ... 16 more
NonActGrp-out:     ... 16 more


From peter.firmstone at zeus.net.au  Thu Jan 29 10:10:15 2015
From: peter.firmstone at zeus.net.au (Peter Firmstone)
Date: Thu, 29 Jan 2015 20:10:15 +1000
Subject: Explicit Serialization API and Security
In-Reply-To: <54CA0027.2000104@zeus.net.au>
References: <54B0CE06.2020408@zeus.net.au>
	<54B3EEC5.5040805@oracle.com>		<1421144693.7530.20.camel@Nokia-N900>
	<54B654F6.6060207@oracle.com>	<1421240334.8856.17.camel@Nokia-N900>	<54BEB91A.4020603@gmail.com>
	<54BFD374.5080803@oracle.com>	<54C00B66.5090302@zeus.net.au>
	<54C01D71.9010901@redhat.com> <54C25DB0.4080207@oracle.com>
	<54C9F888.2040005@zeus.net.au> <54C9FBE1.1070702@zeus.net.au>
	<54CA0027.2000104@zeus.net.au>
Message-ID: <54CA0707.5010005@zeus.net.au>


Another example of intra dependencies, turns out we have a lot of these 
intra class dependencies in our project.

Cheers,

Peter.

/*
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
  * distributed with this work for additional information
  * regarding copyright ownership. The ASF licenses this file
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License. You may obtain a copy of the License at
  *
  *      http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
package net.jini.space;

import java.io.IOException;
import java.io.InvalidObjectException;
import java.io.ObjectInputStream;
import java.rmi.MarshalledObject;
import net.jini.core.entry.Entry;
import net.jini.core.entry.UnusableEntryException;
import net.jini.core.event.RemoteEvent;
import org.apache.river.api.io.AtomicSerial;
import org.apache.river.api.io.AtomicSerial.GetArg;

/**
  * A RemoteEvent marking the transition of an
  * Entry from {@link
  * JavaSpace05#registerForAvailabilityEvent unavailable to
  * available}.

  *
  * Note, by the time the event is delivered, the
  * Entry whose transition triggered this event may
  * have transitioned to a state where it is no longer visible
  * and/or available.

  *
  * @see JavaSpace05
  * @since 2.1
  */
@AtomicSerial
public abstract class AvailabilityEvent extends RemoteEvent {
     private static final long serialVersionUID = 1L;

     /**
      * true if this event signals a
      * transition from invisible to visible as well
      * as unavailable to available.
      *
      * @serial
      */
     private final boolean visibilityTransition;

     private static boolean check(GetArg arg) throws IOException{
     RemoteEvent sup = new RemoteEvent(arg);
     if (sup.getSource() == null)
         throw new InvalidObjectException("null source reference");
     if (!(sup.getSource() instanceof JavaSpace))
         throw new InvalidObjectException("source is not a JavaSpace");
     return arg.get("visibilityTransition", false);
     }

     private AvailabilityEvent(GetArg arg, boolean visibilityTransition) 
throws IOException{
     super(arg);
     this.visibilityTransition = visibilityTransition;
     }

     protected AvailabilityEvent(GetArg arg) throws IOException {
     this(arg, check(arg));
     }



     /**
      * Create a new AvailabilityEvent instance.
      * @param source    the event source
      * @param eventID   the event identifier
      * @param seqNum    the event sequence number
      * @param handback  the handback object
      * @param visibilityTransition true if this event
      *                  must also signal a transition from
      *                  invisible to visible
      * @throws NullPointerException if source is
      * null
      */
     protected AvailabilityEvent(JavaSpace        source,
                 long             eventID,
                 long             seqNum,
                 MarshalledObject handback,
                 boolean          visibilityTransition)
     {
     super(source, eventID, seqNum, handback);
     this.visibilityTransition = visibilityTransition;
     }

     /**
      * @throws InvalidObjectException if {@link #source} is 
null
      * or is not a {@link JavaSpace}
      */
     private void readObject(ObjectInputStream in)
     throws IOException, ClassNotFoundException
     {
     in.defaultReadObject();

     if (getSource() == null)
         throw new InvalidObjectException("null source reference");

     if (!(getSource() instanceof JavaSpace))
         throw new InvalidObjectException("source is not a JavaSpace");
     }

     /**
      * @throws InvalidObjectException if called
      */
     private void readObjectNoData() throws InvalidObjectException {
     throw new InvalidObjectException(
         "AvailabilityEvent should always have data");
     }

     /**
      * Returns a copy of the {@link Entry} whose transition
      * triggered this event. The returned Entry must
      * be unmarshalled in accordance with the Jini
      * Entry Specification.
      *
      * @return a copy of the {@link Entry} whose transition
      *         triggered this event
      * @throws UnusableEntryException if the Entry
      *         can't be unmarshalled in the client. The next call
      *         must re-attempt unmarshalling the
      * Entry
      */
     public abstract Entry getEntry() throws UnusableEntryException;

     /**
      * Returns a snapshot of the {@link Entry} whose
      * transition triggered this event. Snapshots are defined in
      * section JS.2.6 of the 
      * JavaSpaces Service Specification and are an
      * alternative representation of a given Entry
      * produced by a particular space for use with that same
      * space. Passing a snapshot to a space is generally more
      * efficient than passing the original Entry.

      *
      * Any snapshot returned by this method will meet the same
      * contract as the object returned by passing the result of
      * {@link #getEntry getEntry} to {@link JavaSpace#snapshot
      * JavaSpace.snapshot}.

      *
      * Generally there is a cost associated with calling the
      * JavaSpace.snapshot method and thus creating a
      * snapshot using that method is usually only worthwhile if
      * the resulting snapshot is used more than once. The cost of
      * invoking this method should be low and should be worthwhile
      * even if the resulting snapshot is used only once. 

      *
      * @return a snapshot of the {@link Entry} whose
      *         transition triggered this event
      */
     public abstract Entry getSnapshot();

     /**
      * Returns true if the transition that triggered
      * this event was a transition from {@linkplain
      * JavaSpace05#registerForAvailabilityEvent invisible to
      * visible} as well as a transition from unavailable to
      * available, and false otherwise. 

      *
      * @return true if the transition that triggered
      * this event was a transition from invisible to visible as
      * well as a transition from unavailable to available, and
      * false otherwise
      */
     public boolean isVisibilityTransition() {
     return visibilityTransition;
     }
}



From peter.firmstone at zeus.net.au  Thu Jan 29 10:29:01 2015
From: peter.firmstone at zeus.net.au (Peter Firmstone)
Date: Thu, 29 Jan 2015 20:29:01 +1000
Subject: Explicit Serialization API and Security
In-Reply-To: <54CA0707.5010005@zeus.net.au>
References: <54B0CE06.2020408@zeus.net.au>
	<54B3EEC5.5040805@oracle.com>		<1421144693.7530.20.camel@Nokia-N900>
	<54B654F6.6060207@oracle.com>	<1421240334.8856.17.camel@Nokia-N900>	<54BEB91A.4020603@gmail.com>
	<54BFD374.5080803@oracle.com>	<54C00B66.5090302@zeus.net.au>
	<54C01D71.9010901@redhat.com> <54C25DB0.4080207@oracle.com>
	<54C9F888.2040005@zeus.net.au> <54C9FBE1.1070702@zeus.net.au>
	<54CA0027.2000104@zeus.net.au> <54CA0707.5010005@zeus.net.au>
Message-ID: <54CA0B6D.2050105@zeus.net.au>

For the sharper eyed, you'll have noticed the readObjectNoData() method 
I forgot to check:

private static boolean check(GetArg arg) throws IOException{
     // If this class is not in the heirarchy of classes the stream
     // may have been tampered with, see the Serialization Specification
     // section 3.5, this is equivalent to readObjectNoData().
     Class [] serialClasses = arg.serialClasses();
     for (Class serialClass : serialClasses){
         if (serialClass.equals(AvailabilityEvent.class)){
         RemoteEvent sup = new RemoteEvent(arg);
         if (sup.getSource() == null)
             throw new InvalidObjectException("null source reference");
         if (!(sup.getSource() instanceof JavaSpace))
             throw new InvalidObjectException("source is not a JavaSpace");
         return arg.get("visibilityTransition", false);
         }
     }
     throw new InvalidObjectException(
         "AvailabilityEvent should always have data");
     }

On 29/01/2015 8:10 PM, Peter Firmstone wrote:
>
> Another example of intra dependencies, turns out we have a lot of 
> these intra class dependencies in our project.
>
> Cheers,
>
> Peter.
>
> /*
>  * Licensed to the Apache Software Foundation (ASF) under one
>  * or more contributor license agreements.  See the NOTICE file
>  * distributed with this work for additional information
>  * regarding copyright ownership. The ASF licenses this file
>  * to you under the Apache License, Version 2.0 (the
>  * "License"); you may not use this file except in compliance
>  * with the License. You may obtain a copy of the License at
>  *
>  *      http://www.apache.org/licenses/LICENSE-2.0
>  *
>  * Unless required by applicable law or agreed to in writing, software
>  * distributed under the License is distributed on an "AS IS" BASIS,
>  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
> implied.
>  * See the License for the specific language governing permissions and
>  * limitations under the License.
>  */
> package net.jini.space;
>
> import java.io.IOException;
> import java.io.InvalidObjectException;
> import java.io.ObjectInputStream;
> import java.rmi.MarshalledObject;
> import net.jini.core.entry.Entry;
> import net.jini.core.entry.UnusableEntryException;
> import net.jini.core.event.RemoteEvent;
> import org.apache.river.api.io.AtomicSerial;
> import org.apache.river.api.io.AtomicSerial.GetArg;
>
> /**
>  * A RemoteEvent marking the transition of an
>  * Entry from {@link
>  * JavaSpace05#registerForAvailabilityEvent unavailable to
>  * available}.

>  *
>  * Note, by the time the event is delivered, the
>  * Entry whose transition triggered this event may
>  * have transitioned to a state where it is no longer visible
>  * and/or available.

>  *
>  * @see JavaSpace05
>  * @since 2.1
>  */
> @AtomicSerial
> public abstract class AvailabilityEvent extends RemoteEvent {
>     private static final long serialVersionUID = 1L;
>
>     /**
>      * true if this event signals a
>      * transition from invisible to visible as well
>      * as unavailable to available.
>      *
>      * @serial
>      */
>     private final boolean visibilityTransition;
>
>     private static boolean check(GetArg arg) throws IOException{
>     RemoteEvent sup = new RemoteEvent(arg);
>     if (sup.getSource() == null)
>         throw new InvalidObjectException("null source reference");
>     if (!(sup.getSource() instanceof JavaSpace))
>         throw new InvalidObjectException("source is not a JavaSpace");
>     return arg.get("visibilityTransition", false);
>     }
>
>     private AvailabilityEvent(GetArg arg, boolean 
> visibilityTransition) throws IOException{
>     super(arg);
>     this.visibilityTransition = visibilityTransition;
>     }
>
>     protected AvailabilityEvent(GetArg arg) throws IOException {
>     this(arg, check(arg));
>     }
>
>
>
>     /**
>      * Create a new AvailabilityEvent instance.
>      * @param source    the event source
>      * @param eventID   the event identifier
>      * @param seqNum    the event sequence number
>      * @param handback  the handback object
>      * @param visibilityTransition true if this event
>      *                  must also signal a transition from
>      *                  invisible to visible
>      * @throws NullPointerException if source is
>      * null
>      */
>     protected AvailabilityEvent(JavaSpace        source,
>                 long             eventID,
>                 long             seqNum,
>                 MarshalledObject handback,
>                 boolean          visibilityTransition)
>     {
>     super(source, eventID, seqNum, handback);
>     this.visibilityTransition = visibilityTransition;
>     }
>
>     /**
>      * @throws InvalidObjectException if {@link #source} is 
> null
>      * or is not a {@link JavaSpace}
>      */
>     private void readObject(ObjectInputStream in)
>     throws IOException, ClassNotFoundException
>     {
>     in.defaultReadObject();
>
>     if (getSource() == null)
>         throw new InvalidObjectException("null source reference");
>
>     if (!(getSource() instanceof JavaSpace))
>         throw new InvalidObjectException("source is not a JavaSpace");
>     }
>
>     /**
>      * @throws InvalidObjectException if called
>      */
>     private void readObjectNoData() throws InvalidObjectException {
>     throw new InvalidObjectException(
>         "AvailabilityEvent should always have data");
>     }
>
>     /**
>      * Returns a copy of the {@link Entry} whose transition
>      * triggered this event. The returned Entry must
>      * be unmarshalled in accordance with the       * href=http://www.jini.org/standards/index.html>Jini
>      * Entry Specification.
>      *
>      * @return a copy of the {@link Entry} whose transition
>      *         triggered this event
>      * @throws UnusableEntryException if the Entry
>      *         can't be unmarshalled in the client. The next call
>      *         must re-attempt unmarshalling the
>      * Entry
>      */
>     public abstract Entry getEntry() throws UnusableEntryException;
>
>     /**
>      * Returns a snapshot of the {@link Entry} whose
>      * transition triggered this event. Snapshots are defined in
>      * section JS.2.6 of the       * href=http://www.jini.org/standards/index.html>
>      * JavaSpaces Service Specification and are an
>      * alternative representation of a given Entry
>      * produced by a particular space for use with that same
>      * space. Passing a snapshot to a space is generally more
>      * efficient than passing the original Entry.

>      *
>      * Any snapshot returned by this method will meet the same
>      * contract as the object returned by passing the result of
>      * {@link #getEntry getEntry} to {@link JavaSpace#snapshot
>      * JavaSpace.snapshot}.

>      *
>      * Generally there is a cost associated with calling the
>      * JavaSpace.snapshot method and thus creating a
>      * snapshot using that method is usually only worthwhile if
>      * the resulting snapshot is used more than once. The cost of
>      * invoking this method should be low and should be worthwhile
>      * even if the resulting snapshot is used only once. 

>      *
>      * @return a snapshot of the {@link Entry} whose
>      *         transition triggered this event
>      */
>     public abstract Entry getSnapshot();
>
>     /**
>      * Returns true if the transition that triggered
>      * this event was a transition from {@linkplain
>      * JavaSpace05#registerForAvailabilityEvent invisible to
>      * visible} as well as a transition from unavailable to
>      * available, and false otherwise. 

>      *
>      * @return true if the transition that triggered
>      * this event was a transition from invisible to visible as
>      * well as a transition from unavailable to available, and
>      * false otherwise
>      */
>     public boolean isVisibilityTransition() {
>     return visibilityTransition;
>     }
> }
>



From martijnverburg at gmail.com  Thu Jan 29 11:14:29 2015
From: martijnverburg at gmail.com (Martijn Verburg)
Date: Thu, 29 Jan 2015 11:14:29 +0000
Subject: JDK 9 RFR of JDK-8069255: Suppress deprecation warnings in
	jdk.rmic module
In-Reply-To: <54C9A44F.6030603@oracle.com>
References: <54BCA4A5.30202@oracle.com>
	<54C9A44F.6030603@oracle.com>
Message-ID: 

Hi Amy,

Idle curiosity here - are the warnings being suppressed because there is no
way of 'resolving' them?

Cheers,
Martijn

On 29 January 2015 at 03:09, Amy Lu  wrote:

> I updated the webrev, suppress deprecation warnings for
> jdk/src/jdk.rmic/share/classes/sun/tools/java/* now also covered.
>
> Please review: http://cr.openjdk.java.net/~amlu/8069255/webrev.01/
>
> Thanks,
> Amy
>
> On 1/19/15 2:31 PM, Amy Lu wrote:
>
>> Please review these changes to address a few stray deprecation warnings
>> in the jdk.rmic module:
>>
>> JDK-8069255: Suppress deprecation warnings in jdk.rmic module
>> http://cr.openjdk.java.net/~amlu/8069255/webrev.00/
>>
>> I also need a sponsor for this.
>>
>> Thanks,
>> Amy
>>
>>
>


From peter.firmstone at zeus.net.au  Thu Jan 29 11:27:38 2015
From: peter.firmstone at zeus.net.au (Peter Firmstone)
Date: Thu, 29 Jan 2015 21:27:38 +1000
Subject: Explicit Serialization API and Security
In-Reply-To: <54CA0B6D.2050105@zeus.net.au>
References: <54B0CE06.2020408@zeus.net.au>
	<54B3EEC5.5040805@oracle.com>		<1421144693.7530.20.camel@Nokia-N900>
	<54B654F6.6060207@oracle.com>	<1421240334.8856.17.camel@Nokia-N900>	<54BEB91A.4020603@gmail.com>
	<54BFD374.5080803@oracle.com>	<54C00B66.5090302@zeus.net.au>
	<54C01D71.9010901@redhat.com> <54C25DB0.4080207@oracle.com>
	<54C9F888.2040005@zeus.net.au> <54C9FBE1.1070702@zeus.net.au>
	<54CA0027.2000104@zeus.net.au> <54CA0707.5010005@zeus.net.au>
	<54CA0B6D.2050105@zeus.net.au>
Message-ID: <54CA192A.1060404@zeus.net.au>

Two examples:
  1. of a child class with super class invariants and
  2. Checking List and Map contents for type correctness.

/*
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
  * distributed with this work for additional information
  * regarding copyright ownership. The ASF licenses this file
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License. You may obtain a copy of the License at
  *
  *      http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
package net.jini.lease;

import java.io.IOException;
import java.io.InvalidObjectException;
import java.rmi.MarshalledObject;
import net.jini.core.event.RemoteEvent;
import net.jini.core.lease.Lease;
import org.apache.river.api.io.AtomicSerial;
import org.apache.river.api.io.AtomicSerial.GetArg;

/**
  * Event generated by a lease renewal set when its lease is about to
  * expire.
  *
  * @author Sun Microsystems, Inc.
  * @see LeaseRenewalSet
  */
@AtomicSerial
public class ExpirationWarningEvent extends RemoteEvent {
     private static final long serialVersionUID = -2020487536756927350L;

     private static GetArg check(GetArg arg) throws IOException {
     RemoteEvent sup = new RemoteEvent(arg);
     if (sup.getID() != LeaseRenewalSet.EXPIRATION_WARNING_EVENT_ID)
         throw new InvalidObjectException("Illegal object state");
     return arg;
     }

     public ExpirationWarningEvent(GetArg arg) throws IOException {
     super(check(arg));
     }

     /**
      * Simple constructor. Note event id is fixed to
      * LeaseRenewalSet.EXPIRATION_WARNING_EVENT_ID.
      *
      * @param source the LeaseRenewalSet that generated the
      *          event
      * @param seqNum the sequence number of this event
      * @param handback the MarshalledObject passed in as
      *          part of the event registration
      */
     public ExpirationWarningEvent(LeaseRenewalSet  source,
                                   long             seqNum,
                                   MarshalledObject handback)
     {
     super(source, LeaseRenewalSet.EXPIRATION_WARNING_EVENT_ID, seqNum,
           handback);
     }

     /**
      * Convenience method to retrieve the Lease associated
      * with the source of this event. This is the Lease
      * which is about to expire.
      * 

      * The Lease object returned will be equivalent (in the
      * sense of equals) to other Lease objects
      * associated with the set, but may not be the same object. One
      * notable consequence of having two different objects is that the
      * getExpiration method of the Lease
      * object returned by this method may return a different time than
      * the getExpiration methods of other
      * Lease objects granted on the same set.
      * 

      * The expiration time associated with the Lease object
      * returned by this method will reflect the expiration the lease had
      * when the event occurred. Renewal calls may have changed the
      * expiration time of the underlying lease between the time when the
      * event was generated and when it was delivered.
      *
      * @return the lease associated with the source of this event
      */
     public Lease getRenewalSetLease() {
     return ((LeaseRenewalSet) getSource()).getRenewalSetLease();
     }
}


/*
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
  * distributed with this work for additional information
  * regarding copyright ownership. The ASF licenses this file
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License. You may obtain a copy of the License at
  *
  *      http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
package net.jini.discovery;

import com.sun.jini.proxy.MarshalledWrapper;
import java.io.IOException;
import java.io.InvalidObjectException;
import java.io.ObjectInputStream;
import java.rmi.MarshalledObject;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import net.jini.core.event.RemoteEvent;
import net.jini.core.lookup.ServiceID;
import net.jini.core.lookup.ServiceRegistrar;
import net.jini.io.MarshalledInstance;
import org.apache.river.api.io.AtomicSerial.GetArg;

/**
  * Whenever the lookup discovery service discovers or discards a lookup
  * service matching the discovery/discard criteria of one or more of its
  * registrations, the lookup discovery service sends an instance of this
  * class to the listener corresponding to each such registration.
  * 

  * For each registration created by the lookup discovery service, an event
  * identifier is generated. That event identifier uniquely maps the
  * registration to the listener (submitted by the client to the lookup
  * discovery service during the registration process) and to the set of
  * groups and locators the client is interested in discovering. The event
  * identifier is unique across all other active registrations with the
  * lookup discovery service, and is sent to the listener as part of the
  * event.
  * 

  * Because clients of the lookup discovery service need to know not only
  * when a targeted lookup service has been discovered, but also when it
  * has been discarded, the lookup discovery service uses an instance of
  * this class to notify a client's registration(s) when either of these
  * events occurs.
  * 

  * This class extends RemoteEvent, adding the following additional items
  * of abstract state: a boolean indicating whether the lookup services
  * referenced by the event have been discovered or discarded; and a set
  * consisting of proxy objects where each proxy is a marshalled instance
  * of the ServiceRegistrar interface, and each is a proxy of one of the
  * recently discovered or discarded lookup service(s). Methods are defined
  * through which this additional state may be retrieved upon receipt of an
  * instance of this class.
  * 

  * The sequence numbers for a given event identifier are "strictly
  * increasing". This means that when any two such successive events
  * have sequence numbers differing by only a value of 1, then it is
  * guaranteed that no events have been missed. On the other hand, when
  * viewing the set of received events in order, if the difference
  * between the sequence numbers of two successive events is greater
  * than 1, then one or more events may or may not have been missed.
  * For example, a difference greater than 1 could occur if the lookup
  * discovery service crashes, even if no events are lost because of
  * the crash. When two successive events have sequence numbers whose
  * difference is greater than 1, there is said to be a "gap" between
  * the events.
  * 

  * When a gap occurs between events, the state of the locally managed
  * set of lookup services may or may not fall "out of sync" with the
  * corresponding remote state. For example, if the gap corresponds to
  * a missed event representing the (initial) discovery of a targeted
  * lookup service, the remote state will reflect this discovery whereas
  * the local state will not. When such a situation occurs, clients may
  * wish to employ the methods of the corresponding registration object
  * to query the current remote state in order to update the current
  * local state.
  * 

  * Thus, clients typically use this class to determine if conditions
  * are right for a loss of synchronization (by verifying the existence
  * of a gap in the event sequence). Clients then typically use the
  * methods provided by the registration object to both determine if a
  * loss of synchronization has actually occurred, and to correct
  * such a situation when it does occur.
  *
  * @author Sun Microsystems, Inc.
  *
  * @see net.jini.core.event.RemoteEvent
  * @see net.jini.core.lookup.ServiceRegistrar
  */
public class RemoteDiscoveryEvent extends RemoteEvent {

     private static final long serialVersionUID = -9171289945014585248L;

     /**
      * Flag indicating whether the event is a discovery event or a discard
      * event. If this variable is false, then the lookup 
services
      * referenced by this event were just discovered; if true,
      * then those lookup services were just discarded.
      *
      * @serial
      */
     protected final boolean discarded;

     /**
      * List consisting of marshalled proxy objects where each proxy 
implements
      * the ServiceRegistrar interface, and each is a proxy of
      * one of the recently discovered or discarded lookup service(s); the
      * lookup service(s) with which this event is associated.
      * 

      * Each proxy in this list is individually marshalled in order to add
      * an additional 'layer' of serialization. Placing this serialization
      * "wrapper" around each element prevents the deserialization mechanism
      * from attempting to deserialize the individual elements in the list.
      * That is, the deserialization mechanism will only deserialize the 
list
      * itself. After the list itself is successfully deserialized, the 
client
      * (or a third party, if the client requested that events be sent to a
      * third party such as a mailbox), can then attempt to unmarshal each
      * element separately. This allows each success to be captured, and 
each
      * failure to be noted.
      * 

      * If the elements of this list were not each marshalled separately,
      * then upon encountering failure while attempting to deserialize any
      * one element of the list, the deserialization mechanism's
      * readObject method will throw an 
IOException;
      * resulting in the loss of all of the elements of the list, even those
      * that could be successfully deserialized.
      *
      * @serial
      */
     private final List marshalledRegs;

     /**
      * Array containing a subset of the set of proxies to the lookup
      * service(s) with which this event is associated. The elements of this
      * array correspond to those elements of the marshalledRegs
      * array that were successfully unmarshalled (at least once) as a 
result
      * of one or more invocations of the getRegistrars method
      * of this event. Upon deserializing this event, this array is empty,
      * but of the same size as marshalledRegs; and will be
      * populated when the recipient of this event retrieves the registrars
      * corresponding to the elements of marshalledRegs.
      *
      * @serial
      */
     private final ServiceRegistrar[] regs;

     /**
      * Map from the service IDs of the registrars of this 
event
      * to the groups in which each registrar is a member.
      *
      * @serial
      */
     private final Map groups;

     private static GetArg check(GetArg arg) throws IOException {
     RemoteEvent sup = new RemoteEvent(arg);
     if(sup.getSource() == null)
             throw new 
InvalidObjectException("RemoteDiscoveryEvent.readObject "
                                             +"failure - source field is 
null");
     Collections.checkedList(
         (List) arg.get("marshalledRegs", null),
         MarshalledObject.class
     );
     // Also handles null case.
     if (!(arg.get("regs", null) instanceof ServiceRegistrar []))
         throw new ClassCastException();
     Collections.checkedMap(
         (Map) arg.get("groups", null),
         ServiceID.class,
         String.class
     );
     return arg;
     }

     public RemoteDiscoveryEvent(GetArg arg) throws IOException {
     super(check(arg));
     discarded = arg.get("discarded", false);
     marshalledRegs = new ArrayList(
         (List) arg.get("marshalledRegs", null));
     regs = ((ServiceRegistrar[]) arg.get("regs", null)).clone();
     // TreeMap to avoid hash collision attacks.
     groups = new TreeMap((Map) 
arg.get("groups", null));
     integrity = MarshalledWrapper.integrityEnforced(arg);
     }

     /**
      * Flag related to the verification of codebase integrity. A value of
      * true in this field indicates that the last time this
      * event was unmarshalled, the enforcement of codebase integrity was
      * in effect.
      */
     private transient boolean integrity;

     /**
      * Constructs a new instance of RemoteDiscoveryEvent.
      *
      * @param source     reference to the lookup discovery service that
      *                   generated the event
      * @param eventID    the event identifier that maps a particular
      *                   registration to its listener and targeted groups
      *                   and locators
      * @param seqNum     the sequence number of this event
      * @param handback   the client handback (null may be input)
      * @param discarded  flag indicating whether the event being 
constructed
      *                   is a discovery event or a discard event
      * @param groups     mapping from the registrars of this event to the
      *                   groups in which each registrar is a member
      *
      * @throws java.io.IOException when serialization failure occurs on
      *         every registrar of this event. That is, if at least one
      *         registrar is successfully serialized, then this exception
      *         will not be thrown.
      *
      * @throws java.lang.NullPointerException this exception occurs when
      *         either null is input for the map parameter, or
      *         at least one element of that map is null.
      *
      * @throws java.lang.IllegalArgumentException this exception occurs
      *         when an empty set of registrars is input.
      */
     public RemoteDiscoveryEvent(Object source,
                                 long eventID,
                                 long seqNum,
                                 MarshalledObject handback,
                                 boolean discarded,
                                 Map groups)    
throws IOException
     {
     super(source, eventID, seqNum, handback);
     this.discarded = discarded;
         if(groups != null) {
             /* If the set of registrars is empty, throw exception */
             if(groups.size() == 0) {
                 throw new IllegalArgumentException("empty input map");
             }
             ServiceRegistrar[] registrars =
                           (ServiceRegistrar[])(groups.keySet()).toArray
                                         (new 
ServiceRegistrar[groups.size()]);
             /* If any elements of the array are null, throw exception */
             for(int i=0;i(groups.size());
             this.marshalledRegs = new ArrayList(groups.size());
             int l = registrars.length;
             for(int i=0;itrue if this is a discard event, 
false
      *         if it is a discovery event.
      */
     public boolean isDiscarded() {
     return discarded;
     }//end isDiscarded

     /**
      * Returns an array consisting of instances of the ServiceRegistrar
      * interface. Each element in the returned set is a proxy to one of
      * the newly discovered or discarded lookup service(s) that caused
      * the current instance of this event class to be sent to the listener
      * of the client's registration. Note that a new array is returned
      * on every call.
      * 

      * When the lookup discovery service sends an instance of this event
      * class to the listener of a client's registration, the set of lookup
      * service proxies contained in the event is sent as a set of 
marshalled
      * instances of the ServiceRegistrar interface. Thus, in order to
      * construct the return set, this method attempts to unmarshal each
      * element of that set of proxies. Should a failure occur while
      * attempting to unmarshal any of the elements of the set of marshalled
      * proxy objects contained in the current instance of this class, this
      * method will throw an exception of type LookupUnmarshalException.
      * 

      * When a LookupUnmarshalException is thrown by this method, the
      * contents of the exception provides the client with the following
      * useful information: (1) the knowledge that a problem has occurred
      * while unmarshalling at least one of the as yet unmarshalled proxy
      * objects, (2) the set consisting of the proxy objects that were
      * successfully unmarshalled (either on the current invocation of
      * this method or on some previous invocation), (3) the set consisting
      * of the marshalled proxy objects that could not be unmarshalled
      * during the current or any previous invocation of this method, and
      * (4) the set of exceptions corresponding to each failed attempt at
      * unmarshalling during the current invocation of this method.
      * 

      * Typically, the type of exception that occurs when attempting to
      * unmarshal an element of the set of marshalled proxies is either an
      * IOException or a ClassNotFoundException. A ClassNotFoundException
      * occurs whenever a remote field of the marshalled proxy cannot be
      * retrieved (usually because the codebase of one of the field's 
classes
      * or interfaces is currently 'down'). To address this situation, the
      * client may wish to invoke this method at some later time when the
      * 'down' codebase(s) may be accessible. Thus, the client can invoke
      * this method multiple times until all of the elements of the set of
      * marshalled proxies can be successfully unmarshalled.
      * 

      * Note that once an element of the set of marshalled proxy objects has
      * been successfully unmarshalled on a particular invocation of this
      * method, the resulting unmarshalled proxy is stored for return on
      * all future invocations of this method. That is, once successfully
      * unmarshalled, no attempt will be made to unmarshal that element on
      * any future invocations of this method. Thus, if this method returns
      * successfully without throwing a LookupUnmarshalException, the client
      * is guaranteed that all marshalled proxies have been successfully
      * unmarshalled; and any future invocations of this method will return
      * successfully.
      *
      * @return an array consisting of references to the discovered or 
discarded
      *         lookup service(s) corresponding to this event.
      *
      * @throws net.jini.discovery.LookupUnmarshalException this exception
      *         occurs when at least one of the set of lookup service
      *         references cannot be deserialized (unmarshalled).
      *
      * @see net.jini.discovery.LookupUnmarshalException
      */
     public ServiceRegistrar[] getRegistrars() throws 
LookupUnmarshalException {
     synchronized (marshalledRegs) {
             if( marshalledRegs.size() > 0 ) {
                 List unmarshalledRegs = new ArrayList();
                 List exceptions = unmarshalRegistrars(marshalledRegs,
                                                            
unmarshalledRegs);
                 /* Add the un-marshalled elements to the end of regs */
                 insertRegistrars(regs,unmarshalledRegs);
                 if( exceptions.size() > 0 ) {
                     throw(new LookupUnmarshalException
                       ( clipNullsFromEnd(regs),
                         (MarshalledObject[])(marshalledRegs.toArray
                                (new 
MarshalledObject[marshalledRegs.size()])),
                         (Throwable[])(exceptions.toArray
                                (new Throwable[exceptions.size()])),
                         "failed to unmarshal at least one 
ServiceRegistrar") );
                 }//endif
             }//endif
             return clipNullsFromEnd(regs);
         }//end sync(marshalledRegs)
     }//end getRegistrars

     /**
      * Returns a set that maps to the service ID of each registrar 
referenced
      * by this event, the current set of groups in which each registrar 
is a
      * member.
      * 

      * To retrieve the set of member groups corresponding to any element
      * of the array returned by the getRegistrars method,
      * simply use the service ID of the desired element from that array as
      * the key to the get method of the Map 
object
      * returned by this method and cast to String[].
      * 

      * Note that the same Map object is returned on every
      * call to this method; that is, a copy is not made.
      *
      *  @return Map whose key set consists of the service IDs
      *          of each lookup service with which this event is associated,
      *          and whose values are String[] arrays 
containing
      *          the names of the groups in which the lookup service having
      *          the corresponding service ID is a member.
      */
     public Map getGroups() {
         return new HashMap(groups);
     }//end getGroups

     /**
      * Attempts to unmarshal each element of the first input argument. When
      * an element of that argument is successfully unmarshalled, that 
element
      * is removed from the first set and the resulting unmarshalled proxy
      * is placed in the set referenced by the second input argument.
      * Whenever failure occurs as a result of an attempt to unmarshal one
      * of the elements of the first set, the exception that is thrown as
      * as a result of that failure is placed in the returned set of
      * exceptions, and the element from the first set that caused the
      * failure is left in the first set to facilitate the creation of the
      * LookupUnmarshalException that will ultimately be thrown
      * from getRegistrars.
      * 

      * Note that there is a one-to-one correspondence between the 
exceptions
      * contained in the return set and the remaining elements in the first
      * set after all unmarshalling attempts have completed.
      *
      * @param marshalledRegs   an ArrayList object consisting of marshalled
      *                         instances of ServiceRegistrar, each
      *                         corresponding to a proxy to a lookup 
service.
      *
      * @param unmarshalledRegs an ArrayList object consisting of all
      *                         successfully unmarshalled proxies from
      *                         the first argument.
      *
      * @return an ArrayList consisting of the exceptions that occur as a
      *         result of attempts to unmarshal each element of the first
      *         argument to this method.
      */
     private List unmarshalRegistrars(List marshalledRegs,
                                           List unmarshalledRegs)
     {
         ArrayList exceptions = new ArrayList();
        /* Try to un-marshal each element in marshalledRegs; verify codebase
         * when appropriate.
         *
         * If current element is successfully un-marshalled:
         *    -- record the un-marshalled element
         *    -- delete the corresponding marshalled element from its set
         *
         * If current element cannot be un-marshalled:
         *    -- place the exception in the return object
         *    -- leave the corresponding marshalled element in its set
         *    -- increment the index to the next marshalled element
         *
         * Note that index 'n' used in the loop is only a counter. That is,
         * it is intentional that the element at index 'i' is the element
         * that is unmarshalled, not the element at index 'n'. This is 
because
         * whenever the element is successfully unmarshalled, the element is
         * removed from the set of marshalled registrars, decreasing 
that set
         * by 1 element. Thus, the 'next' element to unmarshal is 
actually at
         * the same index as the last element that was unmarshalled.
         */
         int i = 0;
         int nMarshalledRegs = marshalledRegs.size();
         for(int n=0;nnull element.
      *
      * @param regsArray array from which to copy elements
      *
      * @return array of ServiceRegistrar containing each 
element
      *         of the given array from its first element up to, but not
      *         including, the null element; and all subsequent
      *         elements. If the first element of the given array is
      * null, then this method will return an empty array.
      */
     private static ServiceRegistrar[] clipNullsFromEnd
                                                (ServiceRegistrar[] 
regsArray)
     {
         if( regsArray == null) return new ServiceRegistrar[0];
         int clippedLen = indexFirstNull(regsArray);
         ServiceRegistrar[] clippedArray = new ServiceRegistrar[clippedLen];
         for(int i=0; i
      * If the array is null (or has zero length), -1 will be returned. If
      * every element of the array is non-null, this method will return
      * the length of the array. Thus, after invoking this method, it is
      * important to test for these conditions to avoid the occurrence 
of an
      * IndexOutOfBoundsException when using the value returned by this
      * method.
      *
      * @param arr Object array to examine for the first occurrence of null
      *
      * @return the index of the first element in the input array that 
contains
      *         null. A value of -1 is returned if the input array is null;
      *         the length of the array is returned if no element in the
      *         array is null.
      */
     private static int indexFirstNull(Object[] arr) {
         int i = -1;
         if( (arr == null) || (arr.length == 0) ) return i;
         for(i=0;iInvalidObjectException if the state of the
      *         deserialized instance of this class is found to be invalid.
      */
     private void readObject(ObjectInputStream s)
                                throws IOException, ClassNotFoundException
     {
         s.defaultReadObject();
         /* Verify source */
         if(getSource() == null) {
             throw new 
InvalidObjectException("RemoteDiscoveryEvent.readObject "
                                             +"failure - source field is 
null");
         }//endif
         /* Retrieve the value of the integrity flag */
         integrity = MarshalledWrapper.integrityEnforced(s);
     }//end readObject

}//end class RemoteDiscoveryEvent


From amy.lu at oracle.com  Thu Jan 29 11:51:59 2015
From: amy.lu at oracle.com (Amy Lu)
Date: Thu, 29 Jan 2015 19:51:59 +0800
Subject: JDK 9 RFR of JDK-8069255: Suppress deprecation warnings in
	jdk.rmic module
In-Reply-To: 
References: <54BCA4A5.30202@oracle.com>	<54C9A44F.6030603@oracle.com>
	
Message-ID: <54CA1EDF.90403@oracle.com>

Thank you Martijn for looking into this.

The code will be reexamined and fixed later in
     JDK-8071911: Fix deprecation warnings in jdk.rmic module

See more background (which I should have given):
http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-December/030085.html

Thanks,
Amy


On 1/29/15 7:14 PM, Martijn Verburg wrote:
> Hi Amy,
>
> Idle curiosity here - are the warnings being suppressed because there 
> is no way of 'resolving' them?
>
> Cheers,
> Martijn
>
> On 29 January 2015 at 03:09, Amy Lu  > wrote:
>
>     I updated the webrev, suppress deprecation warnings for
>     jdk/src/jdk.rmic/share/classes/sun/tools/java/* now also covered.
>
>     Please review: http://cr.openjdk.java.net/~amlu/8069255/webrev.01/
>     
>
>     Thanks,
>     Amy
>
>     On 1/19/15 2:31 PM, Amy Lu wrote:
>
>         Please review these changes to address a few stray deprecation
>         warnings in the jdk.rmic module:
>
>         JDK-8069255: Suppress deprecation warnings in jdk.rmic module
>         http://cr.openjdk.java.net/~amlu/8069255/webrev.00/
>         
>
>         I also need a sponsor for this.
>
>         Thanks,
>         Amy
>
>
>



From martijnverburg at gmail.com  Thu Jan 29 11:57:16 2015
From: martijnverburg at gmail.com (Martijn Verburg)
Date: Thu, 29 Jan 2015 11:57:16 +0000
Subject: JDK 9 RFR of JDK-8069255: Suppress deprecation warnings in
	jdk.rmic module
In-Reply-To: <54CA1EDF.90403@oracle.com>
References: <54BCA4A5.30202@oracle.com> <54C9A44F.6030603@oracle.com>
	
	<54CA1EDF.90403@oracle.com>
Message-ID: 

Hi Amy,

Thanks - appreciate you digging into the history!

Cheers,
Martijn

On 29 January 2015 at 11:51, Amy Lu  wrote:

>  Thank you Martijn for looking into this.
>
>  The code will be reexamined and fixed later in
>      JDK-8071911: Fix deprecation warnings in jdk.rmic module
>
>  See more background (which I should have given):
>
> http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-December/030085.html
>
>  Thanks,
>  Amy
>
>
> On 1/29/15 7:14 PM, Martijn Verburg wrote:
>
> Hi Amy,
>
>  Idle curiosity here - are the warnings being suppressed because there is
> no way of 'resolving' them?
>
>  Cheers,
> Martijn
>
> On 29 January 2015 at 03:09, Amy Lu  wrote:
>
>> I updated the webrev, suppress deprecation warnings for
>> jdk/src/jdk.rmic/share/classes/sun/tools/java/* now also covered.
>>
>> Please review: http://cr.openjdk.java.net/~amlu/8069255/webrev.01/
>>
>> Thanks,
>> Amy
>>
>> On 1/19/15 2:31 PM, Amy Lu wrote:
>>
>>> Please review these changes to address a few stray deprecation warnings
>>> in the jdk.rmic module:
>>>
>>> JDK-8069255: Suppress deprecation warnings in jdk.rmic module
>>> http://cr.openjdk.java.net/~amlu/8069255/webrev.00/
>>>
>>> I also need a sponsor for this.
>>>
>>> Thanks,
>>> Amy
>>>
>>>
>>
>
>


From peter.firmstone at zeus.net.au  Thu Jan 29 12:20:17 2015
From: peter.firmstone at zeus.net.au (Peter Firmstone)
Date: Thu, 29 Jan 2015 22:20:17 +1000
Subject: Explicit Serialization API and Security
In-Reply-To: <54CA192A.1060404@zeus.net.au>
References: <54B0CE06.2020408@zeus.net.au>
	<54B3EEC5.5040805@oracle.com>		<1421144693.7530.20.camel@Nokia-N900>
	<54B654F6.6060207@oracle.com>	<1421240334.8856.17.camel@Nokia-N900>	<54BEB91A.4020603@gmail.com>
	<54BFD374.5080803@oracle.com>	<54C00B66.5090302@zeus.net.au>
	<54C01D71.9010901@redhat.com> <54C25DB0.4080207@oracle.com>
	<54C9F888.2040005@zeus.net.au> <54C9FBE1.1070702@zeus.net.au>
	<54CA0027.2000104@zeus.net.au> <54CA0707.5010005@zeus.net.au>
	<54CA0B6D.2050105@zeus.net.au> <54CA192A.1060404@zeus.net.au>
Message-ID: <54CA2581.9060906@zeus.net.au>

I decided to sample cpu load (see attached), with debugging enabled for 
the validating ObjectInputStream and JERI, so heaps of output to the 
console.

There are no performance optimisations with stream validation, I've just 
focused on correctness and security.

Thank you HotSpot developers, nice job :)

To give you some background in the tests, there's a heap of dynamic 
class loading going on with codebase downloads, Remote Invocations etc.

I'll profile it on Sparc T2+ in the near future with Oracle express.

Sure miss the sparc gear, OBP and when Solaris was open for a brief 
snapshot in time, can only use it for dev testing now, can't afford to 
use it for production.

Cheers,

Peter.







From lance.andersen at oracle.com  Thu Jan 29 14:22:10 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Thu, 29 Jan 2015 09:22:10 -0500
Subject: Review request for JDK-8051709: Convert JAXP function tests:
	javax.xml.datatype to jtreg (testng) tests
In-Reply-To: <01d501d03b9e$c52b1170$4f813450$@oracle.com>
References: <00b801d03933$4d0d3190$e72794b0$@oracle.com>
	<5A84A3CE-440F-49BC-87F6-50AF44B8D80E@oracle.com>
	<03d201d03a15$4bace290$e306a7b0$@oracle.com>
	<073FA30A-D79C-4F65-AB22-B0C23756C5AA@oracle.com>
	<025b01d03ae9$15027d20$3f077760$@oracle.com>
	<8A015CB5-75A5-40B1-BB4D-631CB792815E@oracle.com>
	<01d501d03b9e$c52b1170$4f813450$@oracle.com>
Message-ID: 

Hi Frank,

I see you decided to use documentation comments (/** vs /*) I would use '/*' otherwise  you need to at missing javadoc tags (param, exception)  to quiet IDEs and -Xdoclint

The update comments are OK

Best
Lance
On Jan 29, 2015, at 3:37 AM, Frank Yuan  wrote:

> Hi Lance
>  
> I have corrected these comments, would you like to have a look?http://cr.openjdk.java.net/~fyuan/8051709/webrev.02/
>  
> Best Regards
> Frank
>  
> From: Lance Andersen [mailto:lance.andersen at oracle.com] 
> Sent: Thursday, January 29, 2015 4:55 AM
> To: Frank Yuan
> Cc: 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> Subject: Re: Review request for JDK-8051709: Convert JAXP function tests: javax.xml.datatype to jtreg (testng) tests
>  
> Hi Frank
>  
> Overall, it is fine, a couple minor nits:
>  
> I thinking some of the comments in both class should be clearer as the comments can be confusing as to what the test should return
> ---------------
>  332 
>  333     /**
>  334      * Test XMLGregorianCalendar#toString() Bug # 5049528:
>  335      * XMLGregorianCalendar.toString throws IllegalStateException
>  336      *
>  337      */
>  338     @Test(dataProvider = "calendar")
>  339     public void checkToStringPos(final int year, final int month, final int day, final int hour, final int minute, final int second) {
>  340         XMLGregorianCalendar calendar = datatypeFactory.newXMLGregorianCalendar(year, month, day, hour, minute, second, undef, undef);
>  341         calendar.toString();
>  342     }
>  
> -------------
>  
> This test is not expecting an IllegalStateException.  I would also probably validate that the value from toString() is valid
>  
> another example:
>  
> ---------
>  552     }
>  553 
>  554     /**
>  555      * Test Duration#getXMLSchemaType() throws UnsupportedOperationException.
>  556      *
>  557      * 

>  558      * Bug # 5049544: Duration.getXMLSchemaType throws
>  559      * UnsupportedOperationException
>  560      *
>  561      */
>  
>  
> --------
>  
> I would just review your comments to make them clearer.
>  
>  
> The comment below has a typo:
> ----------
> 255     /**
>  256      * Test XMLGregorianCalendar#toGregorianCalendar( TimeZone timezone, Locale
>  257      * aLocale, XMLGregorianCalendar defaults)
>  258      *
>  259      * 

>  260      * Bug # 5040542: toGregorianCalendar(...) does not use the defaults
>  261      * XMLGregorianCalendar paramete
>  262      *
>  263      */
>  
> ---------
>  
>  
> no need for another review, I would just tweak the comments and push
>  
> Best
> Lance
>  
> On Jan 28, 2015, at 5:56 AM, Frank Yuan  wrote:
> 
> 
> Hi Lance
>  
> I have updated the code as your suggestions, would you like to review it again?http://cr.openjdk.java.net/~fyuan/8051709/webrev.01/
>  
> Best Regards
> Frank
>  
>  
> From: Lance Andersen [mailto:lance.andersen at oracle.com] 
> Sent: Wednesday, January 28, 2015 3:57 AM
> To: Frank Yuan
> Cc: 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> Subject: Re: Review request for JDK-8051709: Convert JAXP function tests: javax.xml.datatype to jtreg (testng) tests
>  
> Hi Frank,
>  
> On Jan 27, 2015, at 4:40 AM, Frank Yuan  wrote:
> 
> 
> 
> Thank you, Lance!
>  
> I applied some experience from your previous comments:)
>  
> I a glad they were useful :-)
> 
> 
>  
> I have a question for your comment, could you check it below in line?
>  
> See below
> 
> 
>  
> Best Regards
> Frank
>  
> From: Lance Andersen [mailto:lance.andersen at oracle.com] 
> Sent: Tuesday, January 27, 2015 3:24 AM
> To: Frank Yuan
> Cc: 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> Subject: Re: Review request for JDK-8051709: Convert JAXP function tests: javax.xml.datatype to jtreg (testng) tests
>  
> Hi Frank,
>  
> I think this looks good. 
>  
>  
> Not sure if you are going to add more tests in the future, but would be good to have tests such as
>  
> new Duration(x.toString()).equals(x)
>  
>  
> Perhaps a few more checks on expected toString() output?.
>  
> //Frank: I will do it
>  
>  
> For XMLGregorianCalendarTest.java,  I would consider at some point adding more permutations of some of the tests that are validating a bugs(now that you are adding this as a new test suite to openjdk)
>  
> //Frank: I am not sure what you mean, which bug do you want me to add test for?
>  
> For example checkIsValid()
>  
> I would add a DataProvider and add more permutations to test so that you can reduce other potential errors..
>  
> Again, a nice to have for a next update.
>  
> The problem I always have when we add one off tests, it becomes very hard to manage your test suite and really understand the quality of your coverage.  Better to enhance existing tests to fill in weaknesses as this helps keep your test suite from getting out of control?
>  
>  
> 
> 
> 
>  
> Best
> Lance
>  
> On Jan 26, 2015, at 1:42 AM, Frank Yuan  wrote:
> 
> 
> 
> 
> Hi, Joe and All
> 
> We are working on moving internal jaxp functional tests to open jdk repo.
> This is the datatype suite. Would you please review these test?  Any comment
> will be appreciated.
> 
> bug: https://bugs.openjdk.java.net/browse/JDK-8051709
> webrev: http://cr.openjdk.java.net/~fyuan/8051709/webrev.00/
> 
> 
> Thanks,
> 
> Frank
> 
>  
> 
> 
> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering 
> 1 Network Drive 
> Burlington, MA 01803
> Lance.Andersen at oracle.com
>  
> 
> 
> 
> 
>  
> 
> 
> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering 
> 1 Network Drive 
> Burlington, MA 01803
> Lance.Andersen at oracle.com
>  
> 
> 
> 
>  
> 
> 
> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering 
> 1 Network Drive 
> Burlington, MA 01803
> Lance.Andersen at oracle.com
>  
> 
> 



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From aleksej.efimov at oracle.com  Thu Jan 29 14:51:12 2015
From: aleksej.efimov at oracle.com (Aleksej Efimov)
Date: Thu, 29 Jan 2015 17:51:12 +0300
Subject: RFR: 8071585: Update JAX-WS RI integration to latest version
	(2.2.11-b150127.1410)
Message-ID: <54CA48E0.8030902@oracle.com>

Hi,
Can I have a review for a bulk update of JAX-B/WS from upstream projects -
webrev: http://cr.openjdk.java.net/~aefimov/8071585/webrev/
more details in JBS: https://bugs.openjdk.java.net/browse/JDK-8071585

There is a lot of changes (947 lines) but almost all of them are minor (comments changes/(c) years)

Thank you,
Aleksej



From Alan.Bateman at oracle.com  Thu Jan 29 15:06:41 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Thu, 29 Jan 2015 15:06:41 +0000
Subject: RFR: 8071585: Update JAX-WS RI integration to latest version
	(2.2.11-b150127.1410)
In-Reply-To: <54CA48E0.8030902@oracle.com>
References: <54CA48E0.8030902@oracle.com>
Message-ID: <54CA4C81.301@oracle.com>

On 29/01/2015 14:51, Aleksej Efimov wrote:
> Hi,
> Can I have a review for a bulk update of JAX-B/WS from upstream 
> projects -
> webrev: http://cr.openjdk.java.net/~aefimov/8071585/webrev/
> more details in JBS: https://bugs.openjdk.java.net/browse/JDK-8071585
>
> There is a lot of changes (947 lines) but almost all of them are minor 
> (comments changes/(c) years)
>
> Thank you,
> Aleksej
>
I see several changes where ">"is replaced by ">" in pre-formatted 
text (looks like < was changed to < at some point in the past).

This is probably a question for the upstream projects but I would think 

{@code ..}
 would make the source much easier to read.

-Alan.



From brent.christian at oracle.com  Thu Jan 29 17:44:53 2015
From: brent.christian at oracle.com (Brent Christian)
Date: Thu, 29 Jan 2015 09:44:53 -0800
Subject: No "read" FilePermission for JTREG "test.classes" - on Windows only
	(was Re: RFR 9: 8068578: ...)
In-Reply-To: <54C2E41A.9090609@oracle.com>
References: <54C2E41A.9090609@oracle.com>
Message-ID: <54CA7195.3000609@oracle.com>

Hi,

I ran my updated test through our automated testing system, and it 
failed *on Windows only*.  The toURI() call I added came back with an 
AccessControlException due to not being able to read the "test.classes" 
directory.  The test uses its own security policy.

I added

   permission java.io.FilePermission "<>", "read";

to the policy file and the test now passes on all platforms.  So I'll be 
pushing this additional change to the test's policy file.

Anybody know what might be going on - why Windows isn't able to read the 
directory that jtreg creates for test classes?

Thanks,
-Brent

java.security.AccessControlException: access denied 
("java.io.FilePermission" 
"C:\jprt\T\P1\235755.bachrist\s\jdk\testoutput\jdk_io\JTwork\classes\java\io\Serializable\subclassGC" 
"read")
	at 
java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at 
java.security.AccessController.checkPermission(AccessController.java:894)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
	at java.lang.SecurityManager.checkRead(SecurityManager.java:880)
	at java.io.File.isDirectory(File.java:844)
	at java.io.File.toURI(File.java:732)
	at SubclassGC.main(SubclassGC.java:53)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.javatest.regtest.MainWrapper$MainThread.run(MainWrapper.java:94)
	at java.lang.Thread.run(Thread.java:745)

On 1/23/15 4:15 PM, Brent Christian wrote:
> Hi,
>
> Please review this test case update.
>
> Bug:
> https://bugs.openjdk.java.net/browse/JDK-8068578
> Webrev:
> http://cr.openjdk.java.net/~bchristi/8068578/webrev.0/
>
> The test relies on creating a new URLClassLoader and using it to load a
> subclass of ObjectOutputStream.  The system classloader is cast to a
> URLClassLoader in order to call getURLs(), but the test can be made to
> work without this URL[].  We can instead create a URL based on the value
> of the "test.classes" system property.
>
> I have confirmed that the test still detects the original bug (6232010)
> in 5u6, and passes with later JDKs.
>
> Thanks,
> -Brent
>


From vladimir.x.ivanov at oracle.com  Thu Jan 29 18:18:13 2015
From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov)
Date: Thu, 29 Jan 2015 21:18:13 +0300
Subject: [9] RFR (XS): 8071787: Don't block inlining when
	DONT_INLINE_THRESHOLD=0
In-Reply-To: 
References: <54C91877.5040707@oracle.com>
	
Message-ID: <54CA7965.7030301@oracle.com>

Thanks, John!

Best regards,
Vladimir Ivanov

On 1/29/15 6:10 AM, John Rose wrote:
> Good.  Consider fixing the typo in 'makeBlockInlningWrapper'.  ? John
>
> On Jan 28, 2015, at 9:12 AM, Vladimir Ivanov  wrote:
>>
>> http://cr.openjdk.java.net/~vlivanov/8071787/webrev.00/
>> https://bugs.openjdk.java.net/browse/JDK-8071787
>>
>> For testing & performance measurements, sometimes it's useful to replace block inlining wrappers with trivial reinvokers.
>>
>> This change extends DONT_INLINE_THRESHOLD in the following manner:
>>   DONT_INLINE_THRESHOLD = -1: no wrapper
>>   DONT_INLINE_THRESHOLD =  0: reinvoker
>>   DONT_INLINE_THRESHOLD >  0: counting wrapper
>>
>> Before that DONT_INLINE_THRESHOLD=0 meant a counting wrapper which is removed on the first invocation. After the change, it's DONT_INLINE_THRESHOLD=1.
>>
>> Testing: manual, java/lang/invoke
>>
>> Best regards,
>> Vladimir Ivanov
>


From vladimir.x.ivanov at oracle.com  Thu Jan 29 18:18:22 2015
From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov)
Date: Thu, 29 Jan 2015 21:18:22 +0300
Subject: [9] RFR (XXS): 8071788: CountingWrapper.asType() is broken
In-Reply-To: <53D3F321-0259-4878-9767-EA909EF90810@oracle.com>
References: <54C91AF1.3010602@oracle.com>
	<53D3F321-0259-4878-9767-EA909EF90810@oracle.com>
Message-ID: <54CA796E.2090500@oracle.com>

Thanks, John!

Best regards,
Vladimir Ivanov

On 1/29/15 6:11 AM, John Rose wrote:
> Good.
>
> On Jan 28, 2015, at 9:22 AM, Vladimir Ivanov
> > wrote:
>>
>> The fix is to use adapted MethodHandle to construct LambdaForm.
>


From brian.burkhalter at oracle.com  Thu Jan 29 20:53:10 2015
From: brian.burkhalter at oracle.com (Brian Burkhalter)
Date: Thu, 29 Jan 2015 12:53:10 -0800
Subject: JDK 9 RFR of 8067669: Documentation for methods in Number
	incomplete regarding too large values. 
In-Reply-To: <20150119083203.GA8241@e6430>
References: <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com>
	<5494B41B.8060003@oracle.com>
	<0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com>
	<3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com>
	<54AB3DBE.5040806@oracle.com> <20150107080837.GB2073@e6430>
	
	<54ADC94E.4090607@oracle.com> <20150108151355.GA11334@e6430>
	<46FD3812-626A-40AE-8305-10D6B1CA7B4C@oracle.com>
	<20150119083203.GA8241@e6430>
Message-ID: 


On Jan 19, 2015, at 12:32 AM, Andreas Lundblad  wrote:

>> http://cr.openjdk.java.net/~bpb/8067669/webrev.01/
>> 
>> Note that the change at line 40 should be made even if the other diffs are rejected.
> 
> This patch is an improvement in my opinion since it does not indicate that any effort is made to round or truncate the number. The obvious place to look for further documentation would be in the implementing class. I am however still in favour of making this explicit by adding "The specific semantics of the conversion is defined by the subclass in question." which is also in line with the class level documentation.

So at this point I think there are three options:

1) webrev.01 above

2) webrev.01 plus the statement

	"The specific semantics of the conversion is defined by the subclass in question.?

added to the doc of each conversion method

3) resolve as ?Not an Issue.?

It would be good to decide on one of these and resolve this issue. I am inclined to either #1 or #3.

Thanks,

Brian

From joe.darcy at oracle.com  Thu Jan 29 20:53:58 2015
From: joe.darcy at oracle.com (joe darcy)
Date: Thu, 29 Jan 2015 12:53:58 -0800
Subject: JDK 9 RFR of JDK-8071434: doc updates for java.lang.Object
Message-ID: <54CA9DE6.1070804@oracle.com>

Hello,

Please review a few doc updates for java.lang.Object:

      JDK-8071434: doc updates for java.lang.Object
     http://cr.openjdk.java.net/~darcy/8071434.0/

Patch below.

Thanks,

-Joe

--- old/src/java.base/share/classes/java/lang/Object.java 2015-01-29 
12:50:41.099429597 -0800
+++ new/src/java.base/share/classes/java/lang/Object.java 2015-01-29 
12:50:40.771265589 -0800
@@ -1,5 +1,5 @@
  /*
- * Copyright (c) 1994, 2012, Oracle and/or its affiliates. All rights 
reserved.
+ * Copyright (c) 1994, 2015, Oracle and/or its affiliates. All rights 
reserved.
   * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
   *
   * This code is free software; you can redistribute it and/or modify it
@@ -86,12 +86,11 @@
       *     for unequal objects may improve the performance of hash tables.
       * 
       * 

-     * As much as is reasonably practical, the hashCode method defined by
-     * class {@code Object} does return distinct integers for distinct
-     * objects. (This is typically implemented by converting the internal
-     * address of the object into an integer, but this implementation
-     * technique is not required by the
-     * Java™ programming language.)
+     * As much as is reasonably practical, the hashCode method defined
+     * by class {@code Object} does return distinct integers for
+     * distinct objects. (The hashCode may or may not be implemented
+     * as some function of an object's memory address at some point
+     * in time.)
       *
       * @return  a hash code value for this object.
       * @see     java.lang.Object#equals(java.lang.Object)
@@ -344,10 +343,11 @@
       *         ... // Perform action appropriate to condition
       *     }
       * 
-     * (For more information on this topic, see Section 3.2.3 in Doug Lea's
-     * "Concurrent Programming in Java (Second Edition)" (Addison-Wesley,
-     * 2000), or Item 50 in Joshua Bloch's "Effective Java Programming
-     * Language Guide" (Addison-Wesley, 2001).
+     *
+     * (For more information on this topic, see section 14.2,
+     * Condition Queues, in Brian Goetz and others'"Java Concurrency in
+     * Practice" (Addison-Wesley, 2006) or Item 69 in Joshua Bloch's
+     * "Effective Java (Second Edition)" (Addison-Wesley, 2008).
       *
       * 
If the current thread is {@linkplain 
java.lang.Thread#interrupt()
       * interrupted} by any thread before or while it is waiting, then an



From brian.burkhalter at oracle.com  Thu Jan 29 21:01:21 2015
From: brian.burkhalter at oracle.com (Brian Burkhalter)
Date: Thu, 29 Jan 2015 13:01:21 -0800
Subject: JDK 9 RFR of JDK-8071434: doc updates for java.lang.Object
In-Reply-To: <54CA9DE6.1070804@oracle.com>
References: <54CA9DE6.1070804@oracle.com>
Message-ID: <18F81EEF-9700-4C39-B7BE-C846DC87A412@oracle.com>

Would it be appropriate to put the parenthetic information in the new lines 91-93 in an @implSpec or other such section?

Brian

On Jan 29, 2015, at 12:53 PM, joe darcy  wrote:

> Please review a few doc updates for java.lang.Object:
> 
>     JDK-8071434: doc updates for java.lang.Object
>    http://cr.openjdk.java.net/~darcy/8071434.0/



From Roger.Riggs at Oracle.com  Thu Jan 29 21:03:57 2015
From: Roger.Riggs at Oracle.com (Roger Riggs)
Date: Thu, 29 Jan 2015 16:03:57 -0500
Subject: JDK 9 RFR of JDK-8071434: doc updates for java.lang.Object
In-Reply-To: <54CA9DE6.1070804@oracle.com>
References: <54CA9DE6.1070804@oracle.com>
Message-ID: <54CAA03C.7070907@Oracle.com>

Hi Joe,

Looks fine (though I don't have the books to verify the references).

Roger

On 1/29/2015 3:53 PM, joe darcy wrote:
> Hello,
>
> Please review a few doc updates for java.lang.Object:
>
>      JDK-8071434: doc updates for java.lang.Object
>     http://cr.openjdk.java.net/~darcy/8071434.0/
>
> Patch below.
>
> Thanks,
>
> -Joe
>
> --- old/src/java.base/share/classes/java/lang/Object.java 2015-01-29 
> 12:50:41.099429597 -0800
> +++ new/src/java.base/share/classes/java/lang/Object.java 2015-01-29 
> 12:50:40.771265589 -0800
> @@ -1,5 +1,5 @@
>  /*
> - * Copyright (c) 1994, 2012, Oracle and/or its affiliates. All rights 
> reserved.
> + * Copyright (c) 1994, 2015, Oracle and/or its affiliates. All rights 
> reserved.
>   * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
>   *
>   * This code is free software; you can redistribute it and/or modify it
> @@ -86,12 +86,11 @@
>       *     for unequal objects may improve the performance of hash 
> tables.
>       * 
>       * 

> -     * As much as is reasonably practical, the hashCode method 
> defined by
> -     * class {@code Object} does return distinct integers for distinct
> -     * objects. (This is typically implemented by converting the 
> internal
> -     * address of the object into an integer, but this implementation
> -     * technique is not required by the
> -     * Java™ programming language.)
> +     * As much as is reasonably practical, the hashCode method defined
> +     * by class {@code Object} does return distinct integers for
> +     * distinct objects. (The hashCode may or may not be implemented
> +     * as some function of an object's memory address at some point
> +     * in time.)
>       *
>       * @return  a hash code value for this object.
>       * @see     java.lang.Object#equals(java.lang.Object)
> @@ -344,10 +343,11 @@
>       *         ... // Perform action appropriate to condition
>       *     }
>       * 
> -     * (For more information on this topic, see Section 3.2.3 in Doug 
> Lea's
> -     * "Concurrent Programming in Java (Second Edition)" 
> (Addison-Wesley,
> -     * 2000), or Item 50 in Joshua Bloch's "Effective Java Programming
> -     * Language Guide" (Addison-Wesley, 2001).
> +     *
> +     * (For more information on this topic, see section 14.2,
> +     * Condition Queues, in Brian Goetz and others'"Java Concurrency in
> +     * Practice" (Addison-Wesley, 2006) or Item 69 in Joshua Bloch's
> +     * "Effective Java (Second Edition)" (Addison-Wesley, 2008).
>       *
>       * 
If the current thread is {@linkplain 
> java.lang.Thread#interrupt()
>       * interrupted} by any thread before or while it is waiting, then an
>



From Roger.Riggs at Oracle.com  Thu Jan 29 21:07:23 2015
From: Roger.Riggs at Oracle.com (Roger Riggs)
Date: Thu, 29 Jan 2015 16:07:23 -0500
Subject: JDK 9 RFR of 8067669: Documentation for methods in Number
	incomplete regarding too large values.
In-Reply-To: 
References: <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com>	<5494B41B.8060003@oracle.com>	<0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com>	<3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com>	<54AB3DBE.5040806@oracle.com>
	<20150107080837.GB2073@e6430>		<54ADC94E.4090607@oracle.com>
	<20150108151355.GA11334@e6430>	<46FD3812-626A-40AE-8305-10D6B1CA7B4C@oracle.com>	<20150119083203.GA8241@e6430>
	
Message-ID: <54CAA10B.6010805@Oracle.com>

Hi Brian,

#1, The current webrev is fine as is.  (Perhaps with a 2015 copyright 
update).

Since the methods are abstract, the general description is sufficient and
the subclass would have more detail if specified.

Roger

On 1/29/2015 3:53 PM, Brian Burkhalter wrote:
> On Jan 19, 2015, at 12:32 AM, Andreas Lundblad  wrote:
>
>>> http://cr.openjdk.java.net/~bpb/8067669/webrev.01/
>>>
>>> Note that the change at line 40 should be made even if the other diffs are rejected.
>> This patch is an improvement in my opinion since it does not indicate that any effort is made to round or truncate the number. The obvious place to look for further documentation would be in the implementing class. I am however still in favour of making this explicit by adding "The specific semantics of the conversion is defined by the subclass in question." which is also in line with the class level documentation.
> So at this point I think there are three options:
>
> 1) webrev.01 above
>
> 2) webrev.01 plus the statement
>
> 	"The specific semantics of the conversion is defined by the subclass in question.?
>
> added to the doc of each conversion method
>
> 3) resolve as ?Not an Issue.?
>
> It would be good to decide on one of these and resolve this issue. I am inclined to either #1 or #3.
>
> Thanks,
>
> Brian



From joe.darcy at oracle.com  Fri Jan 30 00:02:00 2015
From: joe.darcy at oracle.com (joe darcy)
Date: Thu, 29 Jan 2015 16:02:00 -0800
Subject: JDK 9 RFR of JDK-8071959: java.lang.Object uses implicit default
	constructor
Message-ID: <54CAC9F8.5010105@oracle.com>

Hello,

Please review the patch below to fix

     JDK-8071959: java.lang.Object uses implicit default constructor

diff -r 458adf31ad5b src/java.base/share/classes/java/lang/Object.java
--- a/src/java.base/share/classes/java/lang/Object.java    Thu Jan 29 
15:14:44 2015 -0800
+++ b/src/java.base/share/classes/java/lang/Object.java    Thu Jan 29 
16:00:03 2015 -0800
@@ -42,6 +42,11 @@
      }

      /**
+     * Constructs a new object.
+     */
+    public Object() {}
+
+    /**
       * Returns the runtime class of this {@code Object}. The returned
       * {@code Class} object is the object that is locked by {@code
       * static synchronized} methods of the represented class.

At present, java.lang.Object relies on the default constructor generated 
by javac; how embarrassing!

Thanks,

-Joe


From lance.andersen at oracle.com  Fri Jan 30 00:04:21 2015
From: lance.andersen at oracle.com (Lance @ Oracle)
Date: Thu, 29 Jan 2015 19:04:21 -0500
Subject: JDK 9 RFR of JDK-8071959: java.lang.Object uses implicit default
	constructor
In-Reply-To: <54CAC9F8.5010105@oracle.com>
References: <54CAC9F8.5010105@oracle.com>
Message-ID: 

+1


Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive
Burlington, MA 01803
Lance.Andersen at oracle.com
Sent from my iPad

> On Jan 29, 2015, at 7:02 PM, joe darcy  wrote:
> 
> Hello,
> 
> Please review the patch below to fix
> 
>    JDK-8071959: java.lang.Object uses implicit default constructor
> 
> diff -r 458adf31ad5b src/java.base/share/classes/java/lang/Object.java
> --- a/src/java.base/share/classes/java/lang/Object.java    Thu Jan 29 15:14:44 2015 -0800
> +++ b/src/java.base/share/classes/java/lang/Object.java    Thu Jan 29 16:00:03 2015 -0800
> @@ -42,6 +42,11 @@
>     }
> 
>     /**
> +     * Constructs a new object.
> +     */
> +    public Object() {}
> +
> +    /**
>      * Returns the runtime class of this {@code Object}. The returned
>      * {@code Class} object is the object that is locked by {@code
>      * static synchronized} methods of the represented class.
> 
> At present, java.lang.Object relies on the default constructor generated by javac; how embarrassing!
> 
> Thanks,
> 
> -Joe


From joe.darcy at oracle.com  Fri Jan 30 00:25:22 2015
From: joe.darcy at oracle.com (Joseph D. Darcy)
Date: Thu, 29 Jan 2015 16:25:22 -0800
Subject: JDK 9 RFR of JDK-8071434: doc updates for java.lang.Object
In-Reply-To: <18F81EEF-9700-4C39-B7BE-C846DC87A412@oracle.com>
References: <54CA9DE6.1070804@oracle.com>
	<18F81EEF-9700-4C39-B7BE-C846DC87A412@oracle.com>
Message-ID: <54CACF72.9090900@oracle.com>

I don't think that usage would be inappropriate, but I don't think it is 
called for here either.

Thanks,

-Joe

On 1/29/2015 1:01 PM, Brian Burkhalter wrote:
> Would it be appropriate to put the parenthetic information in the new 
> lines 91-93 in an @implSpec or other such section?
>
> Brian
>
> On Jan 29, 2015, at 12:53 PM, joe darcy  > wrote:
>
>> Please review a few doc updates for java.lang.Object:
>>
>>     JDK-8071434: doc updates for java.lang.Object
>> http://cr.openjdk.java.net/~darcy/8071434.0/ 
>> 
>



From joe.darcy at oracle.com  Fri Jan 30 00:27:54 2015
From: joe.darcy at oracle.com (Joseph D. Darcy)
Date: Thu, 29 Jan 2015 16:27:54 -0800
Subject: JDK 9 RFR of 8067669: Documentation for methods in Number
	incomplete regarding too large values.
In-Reply-To: <54CAA10B.6010805@Oracle.com>
References: <14D2CBF9-D637-43C2-A703-DEE439D567C4@oracle.com>	<5494B41B.8060003@oracle.com>	<0130FA9E-0891-4627-BE4F-28A794D3D2C6@oracle.com>	<3978CBCB-2BBC-4CB3-8ECF-8A2D29A233E6@oracle.com>	<54AB3DBE.5040806@oracle.com>	<20150107080837.GB2073@e6430>		<54ADC94E.4090607@oracle.com>	<20150108151355.GA11334@e6430>	<46FD3812-626A-40AE-8305-10D6B1CA7B4C@oracle.com>	<20150119083203.GA8241@e6430>	
	<54CAA10B.6010805@Oracle.com>
Message-ID: <54CAD00A.7020904@oracle.com>

Hello,

I'm fine with version 1 as well.

Cheers,

-Joe

On 1/29/2015 1:07 PM, Roger Riggs wrote:
> Hi Brian,
>
> #1, The current webrev is fine as is.  (Perhaps with a 2015 copyright 
> update).
>
> Since the methods are abstract, the general description is sufficient and
> the subclass would have more detail if specified.
>
> Roger
>
> On 1/29/2015 3:53 PM, Brian Burkhalter wrote:
>> On Jan 19, 2015, at 12:32 AM, Andreas Lundblad 
>>  wrote:
>>
>>>> http://cr.openjdk.java.net/~bpb/8067669/webrev.01/
>>>>
>>>> Note that the change at line 40 should be made even if the other 
>>>> diffs are rejected.
>>> This patch is an improvement in my opinion since it does not 
>>> indicate that any effort is made to round or truncate the number. 
>>> The obvious place to look for further documentation would be in the 
>>> implementing class. I am however still in favour of making this 
>>> explicit by adding "The specific semantics of the conversion is 
>>> defined by the subclass in question." which is also in line with the 
>>> class level documentation.
>> So at this point I think there are three options:
>>
>> 1) webrev.01 above
>>
>> 2) webrev.01 plus the statement
>>
>>     "The specific semantics of the conversion is defined by the 
>> subclass in question.?
>>
>> added to the doc of each conversion method
>>
>> 3) resolve as ?Not an Issue.?
>>
>> It would be good to decide on one of these and resolve this issue. I 
>> am inclined to either #1 or #3.
>>
>> Thanks,
>>
>> Brian
>



From fweimer at redhat.com  Fri Jan 30 11:04:49 2015
From: fweimer at redhat.com (Florian Weimer)
Date: Fri, 30 Jan 2015 12:04:49 +0100
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54C7160D.4000802@oracle.com>
References: <54C7160D.4000802@oracle.com>
Message-ID: <54CB6551.8040803@redhat.com>

On 01/27/2015 05:37 AM, Mandy Chung wrote:
> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
> and this method is inherently unsafe.  I am thinking to propose this method
> in JDK 9 to throw UnsupportedOperationException.
> 
> I believe it's rare for existing applications using
> System.runFinalizationOnExit.
> My analysis on Maven Central ~315K artifacts that show about ~15 unique
> artifacts calling System.runFinalizationOnExit while they all come from
> only 5 classes.

Does this include test suites?

I see a few callers in Fedora in test suites, but only hsqldb as an
actual user in the installed JAR files.

-- 
Florian Weimer / Red Hat Product Security


From fweimer at redhat.com  Fri Jan 30 11:08:22 2015
From: fweimer at redhat.com (Florian Weimer)
Date: Fri, 30 Jan 2015 12:08:22 +0100
Subject: JDK 9 RFR of JDK-8071434: doc updates for java.lang.Object
In-Reply-To: <54CA9DE6.1070804@oracle.com>
References: <54CA9DE6.1070804@oracle.com>
Message-ID: <54CB6626.60701@redhat.com>

On 01/29/2015 09:53 PM, joe darcy wrote:
> +     * As much as is reasonably practical, the hashCode method defined
> +     * by class {@code Object} does return distinct integers for
> +     * distinct objects. (The hashCode may or may not be implemented
> +     * as some function of an object's memory address at some point
> +     * in time.)

Is it worth mentioning the performance cost of the default
implementation, for typical VMs?

-- 
Florian Weimer / Red Hat Product Security


From martijnverburg at gmail.com  Fri Jan 30 11:09:34 2015
From: martijnverburg at gmail.com (Martijn Verburg)
Date: Fri, 30 Jan 2015 11:09:34 +0000
Subject: JDK 9 RFR of JDK-8071434: doc updates for java.lang.Object
In-Reply-To: <54CAA03C.7070907@Oracle.com>
References: <54CA9DE6.1070804@oracle.com>
	<54CAA03C.7070907@Oracle.com>
Message-ID: 

Perhaps the authors in question would be happy to have a publicly hosted
snippet of that useful information?  I have both books but can appreciate
that there's a *large* number of Java devs who can't afford or get access
to those.

Cheers,
Martijn

On 29 January 2015 at 21:03, Roger Riggs  wrote:

> Hi Joe,
>
> Looks fine (though I don't have the books to verify the references).
>
> Roger
>
>
> On 1/29/2015 3:53 PM, joe darcy wrote:
>
>> Hello,
>>
>> Please review a few doc updates for java.lang.Object:
>>
>>      JDK-8071434: doc updates for java.lang.Object
>>     http://cr.openjdk.java.net/~darcy/8071434.0/
>>
>> Patch below.
>>
>> Thanks,
>>
>> -Joe
>>
>> --- old/src/java.base/share/classes/java/lang/Object.java 2015-01-29
>> 12:50:41.099429597 -0800
>> +++ new/src/java.base/share/classes/java/lang/Object.java 2015-01-29
>> 12:50:40.771265589 -0800
>> @@ -1,5 +1,5 @@
>>  /*
>> - * Copyright (c) 1994, 2012, Oracle and/or its affiliates. All rights
>> reserved.
>> + * Copyright (c) 1994, 2015, Oracle and/or its affiliates. All rights
>> reserved.
>>   * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
>>   *
>>   * This code is free software; you can redistribute it and/or modify it
>> @@ -86,12 +86,11 @@
>>       *     for unequal objects may improve the performance of hash
>> tables.
>>       * 
>>       * 

>> -     * As much as is reasonably practical, the hashCode method defined by
>> -     * class {@code Object} does return distinct integers for distinct
>> -     * objects. (This is typically implemented by converting the internal
>> -     * address of the object into an integer, but this implementation
>> -     * technique is not required by the
>> -     * Java™ programming language.)
>> +     * As much as is reasonably practical, the hashCode method defined
>> +     * by class {@code Object} does return distinct integers for
>> +     * distinct objects. (The hashCode may or may not be implemented
>> +     * as some function of an object's memory address at some point
>> +     * in time.)
>>       *
>>       * @return  a hash code value for this object.
>>       * @see     java.lang.Object#equals(java.lang.Object)
>> @@ -344,10 +343,11 @@
>>       *         ... // Perform action appropriate to condition
>>       *     }
>>       * 
>> -     * (For more information on this topic, see Section 3.2.3 in Doug
>> Lea's
>> -     * "Concurrent Programming in Java (Second Edition)" (Addison-Wesley,
>> -     * 2000), or Item 50 in Joshua Bloch's "Effective Java Programming
>> -     * Language Guide" (Addison-Wesley, 2001).
>> +     *
>> +     * (For more information on this topic, see section 14.2,
>> +     * Condition Queues, in Brian Goetz and others'"Java Concurrency in
>> +     * Practice" (Addison-Wesley, 2006) or Item 69 in Joshua Bloch's
>> +     * "Effective Java (Second Edition)" (Addison-Wesley, 2008).
>>       *
>>       * 
If the current thread is {@linkplain
>> java.lang.Thread#interrupt()
>>       * interrupted} by any thread before or while it is waiting, then an
>>
>>
>


From chris.hegarty at oracle.com  Fri Jan 30 13:36:14 2015
From: chris.hegarty at oracle.com (Chris Hegarty)
Date: Fri, 30 Jan 2015 13:36:14 +0000
Subject: RFR 8064924: Update java.net.URL to work with modules
Message-ID: 

This is phase 1, of getting java.net.URL work with modules. 

Being able to effectively specify URL protocol handler factories as fully qualified class names, through the 'java.protocol.handler.pkgs' system property is problematic. It requires the protocol handler factory implementation class to be public and accessible, as the current implementation tries to instantiate it through core reflection. Since the protocol handler factory must be an implementation of a URLStreamHandlerFactory, it lends itself to being retrofitted with a ServiceLoader lookup. Note, the 'java.protocol.handler.pkgs' system property mechanism predates ServiceLoader. URL protocol handlers would most likely have used service loader if it were available at the time. 

Some URL protocol handlers are fundamental to the platform itself, like 'file' and 'jar', it is not appropriate to attempt a service loader lookup for these, as they may lead to recursive initialization issues. However, Java Plugin, Webstart, and possibly other containers, do override the 'jar' handler. A new API will be provided for this purpose. Providing an API solution should not interfere with system initialization as it will only be called after the system comes up and user code is executing. 

The 'file' protocol handler factory will no longer be overridable, and the system property will no longer be consulted.

Webrev and spec diffs:
  http://cr.openjdk.java.net/~chegar/8064924/00/

I want to agree the approach and spec first, so that the spec changes can be submitted for approval. A comprehensive test will be added before the code changes are finalised.

-Chris.

From dl at cs.oswego.edu  Fri Jan 30 13:44:14 2015
From: dl at cs.oswego.edu (Doug Lea)
Date: Fri, 30 Jan 2015 08:44:14 -0500
Subject: [concurrency-interest] Q: 8071326: ThreadPoolExecutor in endless
	thread creation loop if workQueue.take() throws RuntimeException
In-Reply-To: <54C9FF79.3020303@gmail.com>
References: <54C7B216.90404@oracle.com>		<54C7FD28.8070708@oracle.com>	<54C862ED.3040404@oracle.com>	<54C8D5B6.6070504@oracle.com>	
	<54C9FF79.3020303@gmail.com>
Message-ID: <54CB8AAE.3040801@cs.oswego.edu>

On 01/29/2015 04:38 AM, Peter Levart wrote:

>   *
>   * 
If hook or callback methods throw exceptions, internal worker
>   * threads may in turn fail and abruptly terminate.
>
>
> The last paragraph could explicitly spell-out what are the "callback" methods.
> That would be enough, I think.

Good idea; thanks. Changed to:

!  * 
If hook, callback, or BlockingQueue methods throw exceptions,
!  * internal worker threads may in turn fail, abruptly terminate, and
!  * possibly be replaced.

-Doug



From fweimer at redhat.com  Fri Jan 30 13:50:38 2015
From: fweimer at redhat.com (Florian Weimer)
Date: Fri, 30 Jan 2015 14:50:38 +0100
Subject: ByteBuffer.wrap(array).getInt(offset)
In-Reply-To: <53F4D021.9050300@redhat.com>
References: <53F4BA80.8060306@redhat.com> <53F4D021.9050300@redhat.com>
Message-ID: <54CB8C2E.1070108@redhat.com>

On 08/20/2014 06:43 PM, Andrew Haley wrote:
> On 08/20/2014 04:10 PM, Florian Weimer wrote:
>> Is there already a way to compute the expression in the subject without 
>> the ByteBuffer allocation?  I saw quite a few equivalent formulations in 
>> the OpenJDK sources, and perhaps it's time to add a standardized 
>> solution.
> 
> Isn't this really calling for intrinsification of ByteBuffers?  With that,
> C2 would know that the ByteBuffer doesn't escape, and could collapse the
> lot into a load.

It's a lot of heavy machinery for something that should be quite simple.
 I also expect quite a few users during VM startup.

-- 
Florian Weimer / Red Hat Product Security


From Alan.Bateman at oracle.com  Fri Jan 30 14:32:37 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Fri, 30 Jan 2015 14:32:37 +0000
Subject: RFR 8064924: Update java.net.URL to work with modules
In-Reply-To: 
References: 
Message-ID: <54CB9605.9050805@oracle.com>

On 30/01/2015 13:36, Chris Hegarty wrote:
> This is phase 1, of getting java.net.URL work with modules.
>
> Being able to effectively specify URL protocol handler factories as fully qualified class names, through the 'java.protocol.handler.pkgs' system property is problematic. It requires the protocol handler factory implementation class to be public and accessible, as the current implementation tries to instantiate it through core reflection. Since the protocol handler factory must be an implementation of a URLStreamHandlerFactory, it lends itself to being retrofitted with a ServiceLoader lookup. Note, the 'java.protocol.handler.pkgs' system property mechanism predates ServiceLoader. URL protocol handlers would most likely have used service loader if it were available at the time.
>
> Some URL protocol handlers are fundamental to the platform itself, like 'file' and 'jar', it is not appropriate to attempt a service loader lookup for these, as they may lead to recursive initialization issues. However, Java Plugin, Webstart, and possibly other containers, do override the 'jar' handler. A new API will be provided for this purpose. Providing an API solution should not interfere with system initialization as it will only be called after the system comes up and user code is executing.
>
> The 'file' protocol handler factory will no longer be overridable, and the system property will no longer be consulted.
>
> Webrev and spec diffs:
>    http://cr.openjdk.java.net/~chegar/8064924/00/
>
> I want to agree the approach and spec first, so that the spec changes can be submitted for approval. A comprehensive test will be added before the code changes are finalised.
>
>
I've had a number of off-list discussions with Chris on this and I think 
the proposal and the ordering is good. It preserves behavior for those 
setting the system-wide URLStreamHandlerFactory with 
setURLStreamHandlerFactory and only really creates a migration issue for 
those setting the JDK 1.0/1.1 era pkgs property property.

One thing that I wasn't aware of is the hairy in SAAJ. That is some 
thing that will need to coordinated with upstream and hopefully Miroslav 
can help on that. As upstream might still have a constraint that the 
code compile with JDK 7 and JDK 8 then some adjustments might be required.

In any, the focus now is agreeing the API changes before digging into 
the code changes.

I've read through the javadoc and it looks good. A couple of suggestions:

- step 2: alternative wording: "The createURLStreamHandler method of 
each factory is invoked, in registration order, with the protocol 
string, until a factory returns non-null".

- step 3: "ServiceLoader mechanism is used" might be better than 
"ServiceLoader mechanism tries".

- step 3: "ServiceConfigurationErrors", maybe this should be singular as 
the first will cause the search to bail. Related to this is 
createURLStreamHandler terminating with an error or runtime exception, 
you may want to mention that too.

In the @apiNote in addURLStreamHandlerFactory then "containers" might 
not be understood by the reader. I don't have an alternative but maybe 
make it clear that this is for system-wide configuration and not 
intended to be used by transient applications.

-Alan




From peter.levart at gmail.com  Fri Jan 30 14:52:13 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Fri, 30 Jan 2015 15:52:13 +0100
Subject: JDK 9 RFR of JDK-8071959: java.lang.Object uses implicit default
	constructor
In-Reply-To: <54CAC9F8.5010105@oracle.com>
References: <54CAC9F8.5010105@oracle.com>
Message-ID: <54CB9A9D.4020007@gmail.com>

On 01/30/2015 01:02 AM, joe darcy wrote:
> Hello,
>
> Please review the patch below to fix
>
>     JDK-8071959: java.lang.Object uses implicit default constructor
>
> diff -r 458adf31ad5b src/java.base/share/classes/java/lang/Object.java
> --- a/src/java.base/share/classes/java/lang/Object.java    Thu Jan 29 
> 15:14:44 2015 -0800
> +++ b/src/java.base/share/classes/java/lang/Object.java    Thu Jan 29 
> 16:00:03 2015 -0800
> @@ -42,6 +42,11 @@
>      }
>
>      /**
> +     * Constructs a new object.
> +     */
> +    public Object() {}
> +
> +    /**
>       * Returns the runtime class of this {@code Object}. The returned
>       * {@code Class} object is the object that is locked by {@code
>       * static synchronized} methods of the represented class.
>
> At present, java.lang.Object relies on the default constructor 
> generated by javac; how embarrassing!
>
> Thanks,
>
> -Joe

Is javac able to compile it correctly? It never had to do that before 
(it inserted an implicit one instead). I guess the constructor in Object 
is special, since it does not call a super constructor.

Peter



From chris.hegarty at oracle.com  Fri Jan 30 15:35:20 2015
From: chris.hegarty at oracle.com (Chris Hegarty)
Date: Fri, 30 Jan 2015 15:35:20 +0000
Subject: RFR 8064924: Update java.net.URL to work with modules
In-Reply-To: <54CB9605.9050805@oracle.com>
References: 
	<54CB9605.9050805@oracle.com>
Message-ID: <17D44BD5-B456-40C9-A1DD-88B371052BB0@oracle.com>

Thanks for the comments Alan?

On 30 Jan 2015, at 14:32, Alan Bateman  wrote:

> On 30/01/2015 13:36, Chris Hegarty wrote:
>> This is phase 1, of getting java.net.URL work with modules.
>> 
>> Being able to effectively specify URL protocol handler factories as fully qualified class names, through the 'java.protocol.handler.pkgs' system property is problematic. It requires the protocol handler factory implementation class to be public and accessible, as the current implementation tries to instantiate it through core reflection. Since the protocol handler factory must be an implementation of a URLStreamHandlerFactory, it lends itself to being retrofitted with a ServiceLoader lookup. Note, the 'java.protocol.handler.pkgs' system property mechanism predates ServiceLoader. URL protocol handlers would most likely have used service loader if it were available at the time.
>> 
>> Some URL protocol handlers are fundamental to the platform itself, like 'file' and 'jar', it is not appropriate to attempt a service loader lookup for these, as they may lead to recursive initialization issues. However, Java Plugin, Webstart, and possibly other containers, do override the 'jar' handler. A new API will be provided for this purpose. Providing an API solution should not interfere with system initialization as it will only be called after the system comes up and user code is executing.
>> 
>> The 'file' protocol handler factory will no longer be overridable, and the system property will no longer be consulted.
>> 
>> Webrev and spec diffs:
>>   http://cr.openjdk.java.net/~chegar/8064924/00/
>> 
>> I want to agree the approach and spec first, so that the spec changes can be submitted for approval. A comprehensive test will be added before the code changes are finalised.
>> 
>> 
> I've had a number of off-list discussions with Chris on this and I think the proposal and the ordering is good. It preserves behavior for those setting the system-wide URLStreamHandlerFactory with setURLStreamHandlerFactory and only really creates a migration issue for those setting the JDK 1.0/1.1 era pkgs property property.
> 
> One thing that I wasn't aware of is the hairy in SAAJ. That is some thing that will need to coordinated with upstream and hopefully Miroslav can help on that. As upstream might still have a constraint that the code compile with JDK 7 and JDK 8 then some adjustments might be required.

Right. My changes are mainly a placeholder to remind me to revisit this. I suspect this code will continue to set the system property and call the new method reflectively, if available.   I?ll agree this approach with Miran before finalising the changes.

> In any, the focus now is agreeing the API changes before digging into the code changes.
> 
> I've read through the javadoc and it looks good. A couple of suggestions:
> 
> - step 2: alternative wording: "The createURLStreamHandler method of each factory is invoked, in registration order, with the protocol string, until a factory returns non-null?.

Thanks, updated.

> - step 3: "ServiceLoader mechanism is used" might be better than "ServiceLoader mechanism tries?.

Updated.

> - step 3: "ServiceConfigurationErrors", maybe this should be singular as the first will cause the search to bail. Related to this is createURLStreamHandler terminating with an error or runtime exception, you may want to mention that too.

I added a note about this.

> In the @apiNote in addURLStreamHandlerFactory then "containers" might not be understood by the reader. I don't have an alternative but maybe make it clear that this is for system-wide configuration and not intended to be used by transient applications.

I made a change to indicate that this is system-wide.

Update webrev and spec diffs:  
   http://cr.openjdk.java.net/~chegar/8064924/01/

-Chris.

> -Alan



From sean.mullan at oracle.com  Fri Jan 30 15:44:49 2015
From: sean.mullan at oracle.com (Sean Mullan)
Date: Fri, 30 Jan 2015 10:44:49 -0500
Subject: No "read" FilePermission for JTREG "test.classes" - on Windows
	only (was Re: RFR 9: 8068578: ...)
In-Reply-To: <54CA7195.3000609@oracle.com>
References: <54C2E41A.9090609@oracle.com> <54CA7195.3000609@oracle.com>
Message-ID: <54CBA6F1.3030604@oracle.com>

On 01/29/2015 12:44 PM, Brent Christian wrote:
> Hi,
>
> I ran my updated test through our automated testing system, and it
> failed *on Windows only*.  The toURI() call I added came back with an
> AccessControlException due to not being able to read the "test.classes"
> directory.  The test uses its own security policy.

Try adding -Djava.security.debug=access to the command line and seeing 
what ProtectionDomain is causing the failure and what permissions have 
been granted to it. There could be a file pathname canonicalization bug 
lurking in there.

--Sean

>
> I added
>
>    permission java.io.FilePermission "<>", "read";
>
> to the policy file and the test now passes on all platforms.  So I'll be
> pushing this additional change to the test's policy file.
>
> Anybody know what might be going on - why Windows isn't able to read the
> directory that jtreg creates for test classes?
>
> Thanks,
> -Brent
>
> java.security.AccessControlException: access denied
> ("java.io.FilePermission"
> "C:\jprt\T\P1\235755.bachrist\s\jdk\testoutput\jdk_io\JTwork\classes\java\io\Serializable\subclassGC"
> "read")
>      at
> java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
>
>      at
> java.security.AccessController.checkPermission(AccessController.java:894)
>      at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
>      at java.lang.SecurityManager.checkRead(SecurityManager.java:880)
>      at java.io.File.isDirectory(File.java:844)
>      at java.io.File.toURI(File.java:732)
>      at SubclassGC.main(SubclassGC.java:53)
>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>      at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>      at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>      at java.lang.reflect.Method.invoke(Method.java:498)
>      at
> com.sun.javatest.regtest.MainWrapper$MainThread.run(MainWrapper.java:94)
>      at java.lang.Thread.run(Thread.java:745)
>
> On 1/23/15 4:15 PM, Brent Christian wrote:
>> Hi,
>>
>> Please review this test case update.
>>
>> Bug:
>> https://bugs.openjdk.java.net/browse/JDK-8068578
>> Webrev:
>> http://cr.openjdk.java.net/~bchristi/8068578/webrev.0/
>>
>> The test relies on creating a new URLClassLoader and using it to load a
>> subclass of ObjectOutputStream.  The system classloader is cast to a
>> URLClassLoader in order to call getURLs(), but the test can be made to
>> work without this URL[].  We can instead create a URL based on the value
>> of the "test.classes" system property.
>>
>> I have confirmed that the test still detects the original bug (6232010)
>> in 5u6, and passes with later JDKs.
>>
>> Thanks,
>> -Brent
>>


From Roger.Riggs at Oracle.com  Fri Jan 30 15:58:49 2015
From: Roger.Riggs at Oracle.com (Roger Riggs)
Date: Fri, 30 Jan 2015 10:58:49 -0500
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and Runtime.exec
	should throw UnsupportedOperationException ...
Message-ID: <54CBAA39.3050004@Oracle.com>

Please review this clarification to the optional behavior of 
java.lang.Runtime and java.lang.ProcessBuilder
on platforms that don't support process creation.

Webrev:
   http://cr.openjdk.java.net/~rriggs/webrev-process-8055330/

Issue:
  8055330 https://bugs.openjdk.java.net/browse/JDK-8055330

ps: approved by CCC




From daniel.fuchs at oracle.com  Fri Jan 30 16:07:51 2015
From: daniel.fuchs at oracle.com (Daniel Fuchs)
Date: Fri, 30 Jan 2015 17:07:51 +0100
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and
	Runtime.exec should throw UnsupportedOperationException ...
In-Reply-To: <54CBAA39.3050004@Oracle.com>
References: <54CBAA39.3050004@Oracle.com>
Message-ID: <54CBAC57.2050100@oracle.com>

Looks good to me Roger.

-- daniel

On 30/01/15 16:58, Roger Riggs wrote:
> Please review this clarification to the optional behavior of
> java.lang.Runtime and java.lang.ProcessBuilder
> on platforms that don't support process creation.
>
> Webrev:
>    http://cr.openjdk.java.net/~rriggs/webrev-process-8055330/
>
> Issue:
>   8055330 https://bugs.openjdk.java.net/browse/JDK-8055330
>
> ps: approved by CCC
>
>



From mandy.chung at oracle.com  Fri Jan 30 17:03:01 2015
From: mandy.chung at oracle.com (Mandy Chung)
Date: Fri, 30 Jan 2015 09:03:01 -0800
Subject: Time to retire System.runFinalizersOnExit?
In-Reply-To: <54CB6551.8040803@redhat.com>
References: <54C7160D.4000802@oracle.com> <54CB6551.8040803@redhat.com>
Message-ID: <54CBB945.70908@oracle.com>

On 1/30/15 3:04 AM, Florian Weimer wrote:
> On 01/27/2015 05:37 AM, Mandy Chung wrote:
>> System.runFinalizationOnExit has been deprecated since 1998 (JDK 1.2)
>> and this method is inherently unsafe.  I am thinking to propose this method
>> in JDK 9 to throw UnsupportedOperationException.
>>
>> I believe it's rare for existing applications using
>> System.runFinalizationOnExit.
>> My analysis on Maven Central ~315K artifacts that show about ~15 unique
>> artifacts calling System.runFinalizationOnExit while they all come from
>> only 5 classes.
> Does this include test suites?

No.  I only analyzed the jar files.
>
> I see a few callers in Fedora in test suites, but only hsqldb as an
> actual user in the installed JAR files.

Do you see any issue converting them?

Mandy




From martinrb at google.com  Fri Jan 30 17:44:08 2015
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 30 Jan 2015 09:44:08 -0800
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and
	Runtime.exec should throw UnsupportedOperationException ...
In-Reply-To: <54CBAA39.3050004@Oracle.com>
References: <54CBAA39.3050004@Oracle.com>
Message-ID: 

Seems OK, but:

UOE should be specified in ProcessBuilder if and only if it is also
specified for the Runtime.exec family of methods, since they are wrappers
around ProcessBuilder.

Because there are so many methods, it may be better to have a blanket UOE
disclaimer added to the class descriptions instead - adding UOE to every
process-related method exceeds the normal bar for pedantry in the JDK
documentation.

On Fri, Jan 30, 2015 at 7:58 AM, Roger Riggs  wrote:

> Please review this clarification to the optional behavior of
> java.lang.Runtime and java.lang.ProcessBuilder
> on platforms that don't support process creation.
>
> Webrev:
>   http://cr.openjdk.java.net/~rriggs/webrev-process-8055330/
>
> Issue:
>  8055330 https://bugs.openjdk.java.net/browse/JDK-8055330
>
> ps: approved by CCC
>
>
>


From Roger.Riggs at Oracle.com  Fri Jan 30 18:57:29 2015
From: Roger.Riggs at Oracle.com (Roger Riggs)
Date: Fri, 30 Jan 2015 13:57:29 -0500
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and
	Runtime.exec should throw UnsupportedOperationException ...
In-Reply-To: 
References: <54CBAA39.3050004@Oracle.com>
	
Message-ID: <54CBD419.1060507@Oracle.com>

Hi Martin,

Since very few developers will care about this case, it doesn't seem 
necessary to
repeat the javadoc unless consistency is more important than readability.

In Runtime, the change is in the full exec method;  all of the other 
exec methods are
described as convenience methods and explicitly refer to the full exec 
method.

Roger

On 1/30/2015 12:44 PM, Martin Buchholz wrote:
> Seems OK, but:
>
> UOE should be specified in ProcessBuilder if and only if it is also 
> specified for the Runtime.exec family of methods, since they are 
> wrappers around ProcessBuilder.
>
> Because there are so many methods, it may be better to have a blanket 
> UOE disclaimer added to the class descriptions instead - adding UOE to 
> every process-related method exceeds the normal bar for pedantry in 
> the JDK documentation.
>
> On Fri, Jan 30, 2015 at 7:58 AM, Roger Riggs  > wrote:
>
>     Please review this clarification to the optional behavior of
>     java.lang.Runtime and java.lang.ProcessBuilder
>     on platforms that don't support process creation.
>
>     Webrev:
>     http://cr.openjdk.java.net/~rriggs/webrev-process-8055330/
>     
>
>     Issue:
>      8055330 https://bugs.openjdk.java.net/browse/JDK-8055330
>
>     ps: approved by CCC
>
>
>



From joel.franck at oracle.com  Fri Jan 30 19:58:27 2015
From: joel.franck at oracle.com (=?utf-8?Q?Joel_Borggr=C3=A9n-Franck?=)
Date: Fri, 30 Jan 2015 20:58:27 +0100
Subject: JDK 9 RFR of JDK-8071959: java.lang.Object uses implicit default
	constructor
In-Reply-To: <54CB9A9D.4020007@gmail.com>
References: <54CAC9F8.5010105@oracle.com> <54CB9A9D.4020007@gmail.com>
Message-ID: <36DEB760-3627-4DB3-97C1-A9E41A09C8FC@oracle.com>

Hi

Good question, but javac should be fine. I had to look it up, but there is logic to omit super() when generating the default ctor for Object (TypeEnter::DefaultConstructor), and also logic for omitting super() if we are compiling an explicit ctor for Object (Attr::visitMethodDef).

Looks good Joe, perhaps you could do a boot-cycle build just to be on the safe side?

cheers
/Joel
. 
> On 30 jan 2015, at 15:52, Peter Levart  wrote:
> 
> On 01/30/2015 01:02 AM, joe darcy wrote:
>> Hello,
>> 
>> Please review the patch below to fix
>> 
>>    JDK-8071959: java.lang.Object uses implicit default constructor
>> 
>> diff -r 458adf31ad5b src/java.base/share/classes/java/lang/Object.java
>> --- a/src/java.base/share/classes/java/lang/Object.java    Thu Jan 29 15:14:44 2015 -0800
>> +++ b/src/java.base/share/classes/java/lang/Object.java    Thu Jan 29 16:00:03 2015 -0800
>> @@ -42,6 +42,11 @@
>>     }
>> 
>>     /**
>> +     * Constructs a new object.
>> +     */
>> +    public Object() {}
>> +
>> +    /**
>>      * Returns the runtime class of this {@code Object}. The returned
>>      * {@code Class} object is the object that is locked by {@code
>>      * static synchronized} methods of the represented class.
>> 
>> At present, java.lang.Object relies on the default constructor generated by javac; how embarrassing!
>> 
>> Thanks,
>> 
>> -Joe
> 
> Is javac able to compile it correctly? It never had to do that before (it inserted an implicit one instead). I guess the constructor in Object is special, since it does not call a super constructor.
> 
> Peter
> 



From joe.darcy at oracle.com  Fri Jan 30 20:00:34 2015
From: joe.darcy at oracle.com (joe darcy)
Date: Fri, 30 Jan 2015 12:00:34 -0800
Subject: JDK 9 RFR of JDK-8071959: java.lang.Object uses implicit default
	constructor
In-Reply-To: <36DEB760-3627-4DB3-97C1-A9E41A09C8FC@oracle.com>
References: <54CAC9F8.5010105@oracle.com> <54CB9A9D.4020007@gmail.com>
	<36DEB760-3627-4DB3-97C1-A9E41A09C8FC@oracle.com>
Message-ID: <54CBE2E2.8060701@oracle.com>

Hello,

I was concerned about the special cases handling in javac might be 
problematic and made sure to do a build before I pushed the change. 
Since pushing the change yesterday, all the internal builds have been 
fine so I think we are in the clear :-)

Thanks,

-Joe

On 1/30/2015 11:58 AM, Joel Borggr?n-Franck wrote:
> Hi
>
> Good question, but javac should be fine. I had to look it up, but there is logic to omit super() when generating the default ctor for Object (TypeEnter::DefaultConstructor), and also logic for omitting super() if we are compiling an explicit ctor for Object (Attr::visitMethodDef).
>
> Looks good Joe, perhaps you could do a boot-cycle build just to be on the safe side?
>
> cheers
> /Joel
> .
>> On 30 jan 2015, at 15:52, Peter Levart  wrote:
>>
>> On 01/30/2015 01:02 AM, joe darcy wrote:
>>> Hello,
>>>
>>> Please review the patch below to fix
>>>
>>>     JDK-8071959: java.lang.Object uses implicit default constructor
>>>
>>> diff -r 458adf31ad5b src/java.base/share/classes/java/lang/Object.java
>>> --- a/src/java.base/share/classes/java/lang/Object.java    Thu Jan 29 15:14:44 2015 -0800
>>> +++ b/src/java.base/share/classes/java/lang/Object.java    Thu Jan 29 16:00:03 2015 -0800
>>> @@ -42,6 +42,11 @@
>>>      }
>>>
>>>      /**
>>> +     * Constructs a new object.
>>> +     */
>>> +    public Object() {}
>>> +
>>> +    /**
>>>       * Returns the runtime class of this {@code Object}. The returned
>>>       * {@code Class} object is the object that is locked by {@code
>>>       * static synchronized} methods of the represented class.
>>>
>>> At present, java.lang.Object relies on the default constructor generated by javac; how embarrassing!
>>>
>>> Thanks,
>>>
>>> -Joe
>> Is javac able to compile it correctly? It never had to do that before (it inserted an implicit one instead). I guess the constructor in Object is special, since it does not call a super constructor.
>>
>> Peter
>>



From Roger.Riggs at Oracle.com  Fri Jan 30 20:26:48 2015
From: Roger.Riggs at Oracle.com (Roger Riggs)
Date: Fri, 30 Jan 2015 15:26:48 -0500
Subject: RFR 8/9 8068284: Missing @throws in
	DateTimeFormatterBuilder.appendOffset, etc.
Message-ID: <54CBE908.2030005@Oracle.com>

Please review corrections for editorial issues in java.time javadoc.

Webrev:
    http://cr.openjdk.java.net/~rriggs/webrev-time-editorial-8068284/

Issues:
   8062803: 'principal' should be 'principle' in java.time package 
description

   8068285: Missing @throws in DateTimeFormatterBuilder.appendInstant
   8062796: java.time.format.DateTimeFormatter error in API doc example

Thanks, Roger






From lance.andersen at oracle.com  Fri Jan 30 20:33:00 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Fri, 30 Jan 2015 15:33:00 -0500
Subject: RFR 8/9 8068284: Missing @throws in
	DateTimeFormatterBuilder.appendOffset, etc.
In-Reply-To: <54CBE908.2030005@Oracle.com>
References: <54CBE908.2030005@Oracle.com>
Message-ID: 

Ship it :-)
On Jan 30, 2015, at 3:26 PM, Roger Riggs  wrote:

> Please review corrections for editorial issues in java.time javadoc.
> 
> Webrev:
>   http://cr.openjdk.java.net/~rriggs/webrev-time-editorial-8068284/
> 
> Issues:
>  8062803: 'principal' should be 'principle' in java.time package description
> 
>  8068285: Missing @throws in DateTimeFormatterBuilder.appendInstant
>  8062796: java.time.format.DateTimeFormatter error in API doc example
> 
> Thanks, Roger
> 
> 
> 
> 



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From Alan.Bateman at oracle.com  Fri Jan 30 21:10:07 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Fri, 30 Jan 2015 21:10:07 +0000
Subject: RFR 8064924: Update java.net.URL to work with modules
In-Reply-To: <17D44BD5-B456-40C9-A1DD-88B371052BB0@oracle.com>
References: 
	<54CB9605.9050805@oracle.com>
	<17D44BD5-B456-40C9-A1DD-88B371052BB0@oracle.com>
Message-ID: <54CBF32F.9070603@oracle.com>

On 30/01/2015 15:35, Chris Hegarty wrote:
> :
>
> Update webrev and spec diffs:
>     http://cr.openjdk.java.net/~chegar/8064924/01/
>
I think the javadoc looks much better now, thanks.

-Alan


From mandy.chung at oracle.com  Fri Jan 30 21:11:30 2015
From: mandy.chung at oracle.com (Mandy Chung)
Date: Fri, 30 Jan 2015 13:11:30 -0800
Subject: RFR 8/9 8068284: Missing @throws in
	DateTimeFormatterBuilder.appendOffset, etc.
In-Reply-To: <54CBE908.2030005@Oracle.com>
References: <54CBE908.2030005@Oracle.com>
Message-ID: <54CBF382.7050005@oracle.com>

On 1/30/15 12:26 PM, Roger Riggs wrote:
> Please review corrections for editorial issues in java.time javadoc.
>
> Webrev:
> http://cr.openjdk.java.net/~rriggs/webrev-time-editorial-8068284/
>

Looks fine.

Mandy

> Issues:
>   8062803: 'principal' should be 'principle' in java.time package 
> description
>
>   8068285: Missing @throws in DateTimeFormatterBuilder.appendInstant
>   8062796: java.time.format.DateTimeFormatter error in API doc example
>
> Thanks, Roger
>
>
>
>



From lance.andersen at oracle.com  Fri Jan 30 21:25:55 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Fri, 30 Jan 2015 16:25:55 -0500
Subject: Review request for JDK-8051709: Convert JAXP function tests:
	javax.xml.datatype to jtreg (testng) tests
In-Reply-To: <00c401d03c51$f891d9f0$e9b58dd0$@oracle.com>
References: <00b801d03933$4d0d3190$e72794b0$@oracle.com>
	<5A84A3CE-440F-49BC-87F6-50AF44B8D80E@oracle.com>
	<03d201d03a15$4bace290$e306a7b0$@oracle.com>
	<073FA30A-D79C-4F65-AB22-B0C23756C5AA@oracle.com>
	<025b01d03ae9$15027d20$3f077760$@oracle.com>
	<8A015CB5-75A5-40B1-BB4D-631CB792815E@oracle.com>
	<01d501d03b9e$c52b1170$4f813450$@oracle.com>
	
	<00c401d03c51$f891d9f0$e9b58dd0$@oracle.com>
Message-ID: 

Hi Frank

Looks fine

best
lance
On Jan 30, 2015, at 12:59 AM, Frank Yuan  wrote:

> Hi Lance
>  
> Changed the comment to '/*', could you have a check? http://cr.openjdk.java.net/~fyuan/8051709/webrev.03/
>  
> Best Regards
> Frank
>  
> From: Lance Andersen [mailto:lance.andersen at oracle.com] 
> Sent: Thursday, January 29, 2015 10:22 PM
> To: Frank Yuan
> Cc: 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> Subject: Re: Review request for JDK-8051709: Convert JAXP function tests: javax.xml.datatype to jtreg (testng) tests
>  
> Hi Frank,
>  
> I see you decided to use documentation comments (/** vs /*) I would use '/*' otherwise  you need to at missing javadoc tags (param, exception)  to quiet IDEs and -Xdoclint
>  
> The update comments are OK
>  
> Best
> Lance
> On Jan 29, 2015, at 3:37 AM, Frank Yuan  wrote:
> 
> 
> Hi Lance
>  
> I have corrected these comments, would you like to have a look?http://cr.openjdk.java.net/~fyuan/8051709/webrev.02/
>  
> Best Regards
> Frank
>  
> From: Lance Andersen [mailto:lance.andersen at oracle.com] 
> Sent: Thursday, January 29, 2015 4:55 AM
> To: Frank Yuan
> Cc: 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> Subject: Re: Review request for JDK-8051709: Convert JAXP function tests: javax.xml.datatype to jtreg (testng) tests
>  
> Hi Frank
>  
> Overall, it is fine, a couple minor nits:
>  
> I thinking some of the comments in both class should be clearer as the comments can be confusing as to what the test should return
> ---------------
>  332 
>  333     /**
>  334      * Test XMLGregorianCalendar#toString() Bug # 5049528:
>  335      * XMLGregorianCalendar.toString throws IllegalStateException
>  336      *
>  337      */
>  338     @Test(dataProvider = "calendar")
>  339     public void checkToStringPos(final int year, final int month, final int day, final int hour, final int minute, final int second) {
>  340         XMLGregorianCalendar calendar = datatypeFactory.newXMLGregorianCalendar(year, month, day, hour, minute, second, undef, undef);
>  341         calendar.toString();
>  342     }
>  
> -------------
>  
> This test is not expecting an IllegalStateException.  I would also probably validate that the value from toString() is valid
>  
> another example:
>  
> ---------
>  552     }
>  553 
>  554     /**
>  555      * Test Duration#getXMLSchemaType() throws UnsupportedOperationException.
>  556      *
>  557      * 

>  558      * Bug # 5049544: Duration.getXMLSchemaType throws
>  559      * UnsupportedOperationException
>  560      *
>  561      */
>  
>  
> --------
>  
> I would just review your comments to make them clearer.
>  
>  
> The comment below has a typo:
> ----------
> 255     /**
>  256      * Test XMLGregorianCalendar#toGregorianCalendar( TimeZone timezone, Locale
>  257      * aLocale, XMLGregorianCalendar defaults)
>  258      *
>  259      * 

>  260      * Bug # 5040542: toGregorianCalendar(...) does not use the defaults
>  261      * XMLGregorianCalendar paramete
>  262      *
>  263      */
>  
> ---------
>  
>  
> no need for another review, I would just tweak the comments and push
>  
> Best
> Lance
>  
> On Jan 28, 2015, at 5:56 AM, Frank Yuan  wrote:
> 
> 
> 
> Hi Lance
>  
> I have updated the code as your suggestions, would you like to review it again?http://cr.openjdk.java.net/~fyuan/8051709/webrev.01/
>  
> Best Regards
> Frank
>  
>  
> From: Lance Andersen [mailto:lance.andersen at oracle.com] 
> Sent: Wednesday, January 28, 2015 3:57 AM
> To: Frank Yuan
> Cc: 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> Subject: Re: Review request for JDK-8051709: Convert JAXP function tests: javax.xml.datatype to jtreg (testng) tests
>  
> Hi Frank,
>  
> On Jan 27, 2015, at 4:40 AM, Frank Yuan  wrote:
> 
> 
> 
> 
> Thank you, Lance!
>  
> I applied some experience from your previous comments:)
>  
> I a glad they were useful :-)
> 
> 
> 
>  
> I have a question for your comment, could you check it below in line?
>  
> See below
> 
> 
> 
>  
> Best Regards
> Frank
>  
> From: Lance Andersen [mailto:lance.andersen at oracle.com] 
> Sent: Tuesday, January 27, 2015 3:24 AM
> To: Frank Yuan
> Cc: 'huizhe wang'; 'Core-Libs-Dev'; 'Gustavo Galimberti'
> Subject: Re: Review request for JDK-8051709: Convert JAXP function tests: javax.xml.datatype to jtreg (testng) tests
>  
> Hi Frank,
>  
> I think this looks good. 
>  
>  
> Not sure if you are going to add more tests in the future, but would be good to have tests such as
>  
> new Duration(x.toString()).equals(x)
>  
>  
> Perhaps a few more checks on expected toString() output?.
>  
> //Frank: I will do it
>  
>  
> For XMLGregorianCalendarTest.java,  I would consider at some point adding more permutations of some of the tests that are validating a bugs(now that you are adding this as a new test suite to openjdk)
>  
> //Frank: I am not sure what you mean, which bug do you want me to add test for?
>  
> For example checkIsValid()
>  
> I would add a DataProvider and add more permutations to test so that you can reduce other potential errors..
>  
> Again, a nice to have for a next update.
>  
> The problem I always have when we add one off tests, it becomes very hard to manage your test suite and really understand the quality of your coverage.  Better to enhance existing tests to fill in weaknesses as this helps keep your test suite from getting out of control?
>  
>  
> 
> 
> 
> 
>  
> Best
> Lance
>  
> On Jan 26, 2015, at 1:42 AM, Frank Yuan  wrote:
> 
> 
> 
> 
> 
> Hi, Joe and All
> 
> We are working on moving internal jaxp functional tests to open jdk repo.
> This is the datatype suite. Would you please review these test?  Any comment
> will be appreciated.
> 
> bug: https://bugs.openjdk.java.net/browse/JDK-8051709
> webrev: http://cr.openjdk.java.net/~fyuan/8051709/webrev.00/
> 
> 
> Thanks,
> 
> Frank
> 
>  
> 
> 
> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering 
> 1 Network Drive 
> Burlington, MA 01803
> Lance.Andersen at oracle.com
>  
> 
> 
> 
> 
> 
>  
> 
> 
> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering 
> 1 Network Drive 
> Burlington, MA 01803
> Lance.Andersen at oracle.com
>  
> 
> 
> 
> 
>  
> 
> 
> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering 
> 1 Network Drive 
> Burlington, MA 01803
> Lance.Andersen at oracle.com
>  
> 
> 
> 
>  
> 
> 
> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering 
> 1 Network Drive 
> Burlington, MA 01803
> Lance.Andersen at oracle.com
>  
> 
> 



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From Alan.Bateman at oracle.com  Fri Jan 30 21:30:03 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Fri, 30 Jan 2015 21:30:03 +0000
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and
	Runtime.exec should throw UnsupportedOperationException ...
In-Reply-To: <54CBAA39.3050004@Oracle.com>
References: <54CBAA39.3050004@Oracle.com>
Message-ID: <54CBF7DB.9000909@oracle.com>

On 30/01/2015 15:58, Roger Riggs wrote:
> Please review this clarification to the optional behavior of 
> java.lang.Runtime and java.lang.ProcessBuilder
> on platforms that don't support process creation.
>
> Webrev:
>   http://cr.openjdk.java.net/~rriggs/webrev-process-8055330/
One suggest is to move this to after the "Starting an operating system 
.." paragraph. The reason is that it more logically following the list 
of highly platform specific issues. Otherwise looks okay to me.

-Alan




From Roger.Riggs at Oracle.com  Fri Jan 30 21:48:00 2015
From: Roger.Riggs at Oracle.com (Roger Riggs)
Date: Fri, 30 Jan 2015 16:48:00 -0500
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and
	Runtime.exec should throw UnsupportedOperationException ...
In-Reply-To: <54CBF7DB.9000909@oracle.com>
References: <54CBAA39.3050004@Oracle.com> <54CBF7DB.9000909@oracle.com>
Message-ID: <54CBFC10.9080502@Oracle.com>

Hi,

That does read better.

As in this webrev:
    http://cr.openjdk.java.net/~rriggs/webrev-process-8072034/

Roger


On 1/30/2015 4:30 PM, Alan Bateman wrote:
> On 30/01/2015 15:58, Roger Riggs wrote:
>> Please review this clarification to the optional behavior of 
>> java.lang.Runtime and java.lang.ProcessBuilder
>> on platforms that don't support process creation.
>>
>> Webrev:
>>   http://cr.openjdk.java.net/~rriggs/webrev-process-8055330/
> One suggest is to move this to after the "Starting an operating system 
> .." paragraph. The reason is that it more logically following the list 
> of highly platform specific issues. Otherwise looks okay to me.
>
> -Alan
>
>



From Roger.Riggs at Oracle.com  Fri Jan 30 22:25:12 2015
From: Roger.Riggs at Oracle.com (Roger Riggs)
Date: Fri, 30 Jan 2015 17:25:12 -0500
Subject: RFR 8/9: 8068278 ArrayIndexOutOfBoundsException instead of
	DateTimeException in j.t.chrono.JapaneseChronology.eraOf()
Message-ID: <54CC04C8.9070600@Oracle.com>

Please review this correction of a JapaneseEra range check in java.time.
The error was discovered during development of additional conformance 
tests (to be delivered separately).

Webrev:
    http://cr.openjdk.java.net/~rriggs//webrev-era-8068278

Issue:
 8068278 
ArrayIndexOutOfBoundsException instead of DateTimeException in 
j.t.chrono.JapaneseChronology.eraOf()

Thanks, Roger



From mandy.chung at oracle.com  Fri Jan 30 22:27:41 2015
From: mandy.chung at oracle.com (Mandy Chung)
Date: Fri, 30 Jan 2015 14:27:41 -0800
Subject: RFR 8/9: 8068278 ArrayIndexOutOfBoundsException instead of
	DateTimeException in j.t.chrono.JapaneseChronology.eraOf()
In-Reply-To: <54CC04C8.9070600@Oracle.com>
References: <54CC04C8.9070600@Oracle.com>
Message-ID: <54CC055D.9030602@oracle.com>

On 1/30/15 2:25 PM, Roger Riggs wrote:
> Please review this correction of a JapaneseEra range check in java.time.
> The error was discovered during development of additional conformance 
> tests (to be delivered separately).
>
> Webrev:
>    http://cr.openjdk.java.net/~rriggs//webrev-era-8068278
>

Looks fine to me.  Is it easy to write a regression test to go along 
with this fix?

Mandy

> Issue:
>  8068278 
> ArrayIndexOutOfBoundsException instead of DateTimeException in 
> j.t.chrono.JapaneseChronology.eraOf()
>
> Thanks, Roger
>



From lance.andersen at oracle.com  Fri Jan 30 22:31:36 2015
From: lance.andersen at oracle.com (Lance Andersen)
Date: Fri, 30 Jan 2015 17:31:36 -0500
Subject: RFR 8/9: 8068278 ArrayIndexOutOfBoundsException instead of
	DateTimeException in j.t.chrono.JapaneseChronology.eraOf()
In-Reply-To: <54CC04C8.9070600@Oracle.com>
References: <54CC04C8.9070600@Oracle.com>
Message-ID: 

Looks OK  and easier to read now

Best
Lance
On Jan 30, 2015, at 5:25 PM, Roger Riggs  wrote:

> Please review this correction of a JapaneseEra range check in java.time.
> The error was discovered during development of additional conformance tests (to be delivered separately).
> 
> Webrev:
>   http://cr.openjdk.java.net/~rriggs//webrev-era-8068278
> 
> Issue:
>  8068278 ArrayIndexOutOfBoundsException instead of DateTimeException in j.t.chrono.JapaneseChronology.eraOf()
> 
> Thanks, Roger
> 



Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com





From Roger.Riggs at Oracle.com  Fri Jan 30 22:33:45 2015
From: Roger.Riggs at Oracle.com (Roger Riggs)
Date: Fri, 30 Jan 2015 17:33:45 -0500
Subject: RFR 8/9: 8068278 ArrayIndexOutOfBoundsException instead of
	DateTimeException in j.t.chrono.JapaneseChronology.eraOf()
In-Reply-To: <54CC055D.9030602@oracle.com>
References: <54CC04C8.9070600@Oracle.com> <54CC055D.9030602@oracle.com>
Message-ID: <54CC06C9.2050101@Oracle.com>

Hi Mandy,

Thanks for the review.

I wrote the test (and it passed) but since the JCK folks are providing 
the tests it seemed
undesirable to have duplicate tests.

Roger

On 1/30/2015 5:27 PM, Mandy Chung wrote:
> On 1/30/15 2:25 PM, Roger Riggs wrote:
>> Please review this correction of a JapaneseEra range check in java.time.
>> The error was discovered during development of additional conformance 
>> tests (to be delivered separately).
>>
>> Webrev:
>>    http://cr.openjdk.java.net/~rriggs//webrev-era-8068278
>>
>
> Looks fine to me.  Is it easy to write a regression test to go along 
> with this fix?
>
> Mandy
>
>> Issue:
>>  8068278 
>> ArrayIndexOutOfBoundsException instead of DateTimeException in 
>> j.t.chrono.JapaneseChronology.eraOf()
>>
>> Thanks, Roger
>>
>



From martinrb at google.com  Fri Jan 30 23:41:36 2015
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 30 Jan 2015 15:41:36 -0800
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and
	Runtime.exec should throw UnsupportedOperationException ...
In-Reply-To: <54CBFC10.9080502@Oracle.com>
References: <54CBAA39.3050004@Oracle.com> <54CBF7DB.9000909@oracle.com>
	<54CBFC10.9080502@Oracle.com>
Message-ID: 

It often happens that some ancient thought was in my head 10 years ago, and
it takes a day or two to get paged back in.

Now I think that 10 years ago I definitely considered the possibility of an
OS that had no subprocess support at all, and I was comfortable with simply
throwing IOException, much as if whatever command you were looiking for
could not be found.  Maybe there are no files in the file system...  Right
now I'm thinking a sandboxed OS should throw IOException, in part because
UOE is not documented, while IOException is.  Nothing is stopping you from
throwing a new subclass of IOException, e.g.
YouAreTrappedInASandboxException extends IOException ...

On Fri, Jan 30, 2015 at 1:48 PM, Roger Riggs  wrote:

> Hi,
>
> That does read better.
>
> As in this webrev:
>    http://cr.openjdk.java.net/~rriggs/webrev-process-8072034/
>
> Roger
>
>
>
> On 1/30/2015 4:30 PM, Alan Bateman wrote:
>
>> On 30/01/2015 15:58, Roger Riggs wrote:
>>
>>> Please review this clarification to the optional behavior of
>>> java.lang.Runtime and java.lang.ProcessBuilder
>>> on platforms that don't support process creation.
>>>
>>> Webrev:
>>>   http://cr.openjdk.java.net/~rriggs/webrev-process-8055330/
>>>
>> One suggest is to move this to after the "Starting an operating system
>> .." paragraph. The reason is that it more logically following the list of
>> highly platform specific issues. Otherwise looks okay to me.
>>
>> -Alan
>>
>>
>>
>


From mandy.chung at oracle.com  Fri Jan 30 23:48:09 2015
From: mandy.chung at oracle.com (Mandy Chung)
Date: Fri, 30 Jan 2015 15:48:09 -0800
Subject: RFR 8/9: 8068278 ArrayIndexOutOfBoundsException instead of
	DateTimeException in j.t.chrono.JapaneseChronology.eraOf()
In-Reply-To: <54CC06C9.2050101@Oracle.com>
References: <54CC04C8.9070600@Oracle.com> <54CC055D.9030602@oracle.com>
	<54CC06C9.2050101@Oracle.com>
Message-ID: <54CC1839.30801@oracle.com>

On 1/30/15 2:33 PM, Roger Riggs wrote:
> Hi Mandy,
>
> Thanks for the review.
>
> I wrote the test (and it passed) but since the JCK folks are providing 
> the tests it seemed
> undesirable to have duplicate tests.

JDK developers don't run JCK tests and I think it'd be nice to have a 
regression test to go with a fix unless the bug is uncovered by an 
existing test.

Mandy
>
> Roger
>
> On 1/30/2015 5:27 PM, Mandy Chung wrote:
>> On 1/30/15 2:25 PM, Roger Riggs wrote:
>>> Please review this correction of a JapaneseEra range check in 
>>> java.time.
>>> The error was discovered during development of additional 
>>> conformance tests (to be delivered separately).
>>>
>>> Webrev:
>>>    http://cr.openjdk.java.net/~rriggs//webrev-era-8068278
>>>
>>
>> Looks fine to me.  Is it easy to write a regression test to go along 
>> with this fix?
>>
>> Mandy
>>
>>> Issue:
>>>  8068278 
>>> ArrayIndexOutOfBoundsException instead of DateTimeException in 
>>> j.t.chrono.JapaneseChronology.eraOf()
>>>
>>> Thanks, Roger
>>>
>>
>



From Alan.Bateman at oracle.com  Sat Jan 31 09:30:17 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Sat, 31 Jan 2015 09:30:17 +0000
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and
	Runtime.exec should throw UnsupportedOperationException ...
In-Reply-To: <54CBFC10.9080502@Oracle.com>
References: <54CBAA39.3050004@Oracle.com> <54CBF7DB.9000909@oracle.com>
	<54CBFC10.9080502@Oracle.com>
Message-ID: <54CCA0A9.3070000@oracle.com>

On 30/01/2015 21:48, Roger Riggs wrote:
> Hi,
>
> That does read better.
>
> As in this webrev:
>    http://cr.openjdk.java.net/~rriggs/webrev-process-8072034/
This looks better, thanks.

-Alan.


From martinrb at google.com  Sat Jan 31 16:15:42 2015
From: martinrb at google.com (Martin Buchholz)
Date: Sat, 31 Jan 2015 08:15:42 -0800
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and
	Runtime.exec should throw UnsupportedOperationException ...
In-Reply-To: 
References: <54CBAA39.3050004@Oracle.com> <54CBF7DB.9000909@oracle.com>
	<54CBFC10.9080502@Oracle.com>
	
Message-ID: 

It's not a big deal, but I am opposed to this change.  The old spec

     * 
In such cases an exception will be thrown. The exact nature
     * of the exception is system-dependent, but it will always be a
     * subclass of {@link IOException}.

is perfectly adequate for OSes with no subprocesses.
Users should be catching and handling IOException in any case.  Throwing a
RuntimeException seems wrong, and breaks the above promise!

I wish CCC deliberations were public and included API authors, where
available...


On Fri, Jan 30, 2015 at 3:41 PM, Martin Buchholz 
wrote:

> It often happens that some ancient thought was in my head 10 years ago,
> and it takes a day or two to get paged back in.
>
> Now I think that 10 years ago I definitely considered the possibility of
> an OS that had no subprocess support at all, and I was comfortable with
> simply throwing IOException, much as if whatever command you were looiking
> for could not be found.  Maybe there are no files in the file system...
> Right now I'm thinking a sandboxed OS should throw IOException, in part
> because UOE is not documented, while IOException is.  Nothing is stopping
> you from throwing a new subclass of IOException, e.g.
> YouAreTrappedInASandboxException extends IOException ...
>
> On Fri, Jan 30, 2015 at 1:48 PM, Roger Riggs 
> wrote:
>
>> Hi,
>>
>> That does read better.
>>
>> As in this webrev:
>>    http://cr.openjdk.java.net/~rriggs/webrev-process-8072034/
>>
>> Roger
>>
>>
>>
>> On 1/30/2015 4:30 PM, Alan Bateman wrote:
>>
>>> On 30/01/2015 15:58, Roger Riggs wrote:
>>>
>>>> Please review this clarification to the optional behavior of
>>>> java.lang.Runtime and java.lang.ProcessBuilder
>>>> on platforms that don't support process creation.
>>>>
>>>> Webrev:
>>>>   http://cr.openjdk.java.net/~rriggs/webrev-process-8055330/
>>>>
>>> One suggest is to move this to after the "Starting an operating system
>>> .." paragraph. The reason is that it more logically following the list of
>>> highly platform specific issues. Otherwise looks okay to me.
>>>
>>> -Alan
>>>
>>>
>>>
>>
>


From peter.levart at gmail.com  Sat Jan 31 19:42:59 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Sat, 31 Jan 2015 20:42:59 +0100
Subject: RFR 8064924: Update java.net.URL to work with modules
In-Reply-To: 
References: 
Message-ID: <54CD3043.2070709@gmail.com>

Hi Chris,

I looked at your solution to make URLStreamHandlerFactory interface a 
service provider using ServiceLoader API and in addition adding new URL 
static method to programmaticaly register URLStreamHandlerFactories. 
There are a couple of issues with your approach I'd like to discuss.

The programmatic approach using static URL method does give you some 
means of order in which registered URLStreamHandlerFactories are tried 
to create URLStreamHandler for particular protocol - the registration 
order. It means that this method should only be called by one "party" in 
the VM or else it is hard to control the order of registration.

ServiceLoader is a declarative approach, but does not give you order by 
default. Also, your ServiceLoader approach gives a way for 
URLStreamHandlerFactories to register in the system without security 
checks. If a particular 
META-INF/services/java.net.URLStreamHandlerFactory is found, it's 
content is used to load a class and instantiate a factory which is used 
in URL constructors then. Previously, one had to have a "setFactory" 
permission to set the URLStreamHandlerFactory or appropriate 
PropertyPermission for seting the package prefix property. This can be 
fixed.

The java.protocol.handler.pkgs system property manipulation in 
HttpSOAPConnection has a couple of issues as it is now (without your 
changes):

- first, the property can be set too late, when https protocol has 
already been used in URL constructions before any HttpSOAPConnection is 
used. The URLStreamHandler cached in URL will be used for ever 
regardless of HttpSOAPConnection setting the property later.
- second, the HttpSOAPConnection.initHttps is racy - it bases it's 
decision to instantiate and add a security Provider on the 
presence/absence of a particular package in the value of 
java.protocol.handler.pkgs system property, which it modifies 
afterwards. Concurrent calls to initHttps (which is called from 
HttpSOAPConnection.doGet and .post) can lead to multiple security 
Providers instantiations and adds...

Your question "why is jaxws using the old https client ??" is right on 
the spot. Is it just because nobody has updated it yet? Why does it have 
to choose the URLStreamHandlers based on JVM it is executing on (doesn't 
IBM JVM already arrange for it's own HTTPS support by default?). Is it 
maybe the security Provider that is dependent on these *old* HTTPS 
implementations because it is using their internal APIs? This needs to 
be cleared, I think.

Anyway, I think there is an alternative to programmatic registration 
approach of URLStreamHandlerFactories. Using just ServiceLoader and a 
new default method on URLStreamHandlerFactory interface to provide 
order. Here's what I'm thinking about:

http://cr.openjdk.java.net/~plevart/jdk9-dev/URLStreamHandlerFactory/webrev.01/

You see, even tests can be made to use this approach. I think 
declarative approach is better because it doesn't depend on 
initialization order.

Additional ideas you can take from this for your patch are the following:
- the way URLStreamHandlerFactories obtained from ServiceLoader are 
security-checked before used. The implementation class / code source has 
to be given "setFactory" permission if such factory is to be considered 
at all.
- the way URLStreamHandlers are cached using ConcurrentHashMap instead 
of Hashtable which simplifies caching "atomicity" checks and does not 
use lock(s) on fast-path.

So what do you think of ServiceLoader-only approach?


Regards, Peter


On 01/30/2015 02:36 PM, Chris Hegarty wrote:
> This is phase 1, of getting java.net.URL work with modules.
>
> Being able to effectively specify URL protocol handler factories as fully qualified class names, through the 'java.protocol.handler.pkgs' system property is problematic. It requires the protocol handler factory implementation class to be public and accessible, as the current implementation tries to instantiate it through core reflection. Since the protocol handler factory must be an implementation of a URLStreamHandlerFactory, it lends itself to being retrofitted with a ServiceLoader lookup. Note, the 'java.protocol.handler.pkgs' system property mechanism predates ServiceLoader. URL protocol handlers would most likely have used service loader if it were available at the time.
>
> Some URL protocol handlers are fundamental to the platform itself, like 'file' and 'jar', it is not appropriate to attempt a service loader lookup for these, as they may lead to recursive initialization issues. However, Java Plugin, Webstart, and possibly other containers, do override the 'jar' handler. A new API will be provided for this purpose. Providing an API solution should not interfere with system initialization as it will only be called after the system comes up and user code is executing.
>
> The 'file' protocol handler factory will no longer be overridable, and the system property will no longer be consulted.
>
> Webrev and spec diffs:
>    http://cr.openjdk.java.net/~chegar/8064924/00/
>
> I want to agree the approach and spec first, so that the spec changes can be submitted for approval. A comprehensive test will be added before the code changes are finalised.
>
> -Chris.



From Alan.Bateman at oracle.com  Sat Jan 31 21:03:01 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Sat, 31 Jan 2015 21:03:01 +0000
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and
	Runtime.exec should throw UnsupportedOperationException ...
In-Reply-To: 
References: <54CBAA39.3050004@Oracle.com>	<54CBF7DB.9000909@oracle.com>	<54CBFC10.9080502@Oracle.com>	
	
Message-ID: <54CD4305.4080607@oracle.com>

On 31/01/2015 16:15, Martin Buchholz wrote:
> It's not a big deal, but I am opposed to this change.
Just an FYI that Roger seems to have pushed the original patch, this new 
patch just moves the text down so that it flows a bit better.

> The old spec
>
>      * 
In such cases an exception will be thrown. The exact nature
>      * of the exception is system-dependent, but it will always be a
>      * subclass of {@link IOException}.
>
> is perfectly adequate for OSes with no subprocesses.
> Users should be catching and handling IOException in any case.  
> Throwing a RuntimeException seems wrong, and breaks the above promise!
>
It's lack of clarity in the above text that has been causing the 
testability issue for so long so I think it is good to make it clear how 
an implementation that does not support processes should behave. The 
options on the table seem to be to define a sub-type of IOE for this 
case, specify that an IOE be thrown with an UOE as the cause, or just 
throw UOE when this feature is not supported.

-Alan





From martinrb at google.com  Sat Jan 31 21:30:43 2015
From: martinrb at google.com (Martin Buchholz)
Date: Sat, 31 Jan 2015 13:30:43 -0800
Subject: RFR 9 8055330: (process spec) ProcessBuilder.start and
	Runtime.exec should throw UnsupportedOperationException ...
In-Reply-To: <54CD4305.4080607@oracle.com>
References: <54CBAA39.3050004@Oracle.com> <54CBF7DB.9000909@oracle.com>
	<54CBFC10.9080502@Oracle.com>
	
	
	<54CD4305.4080607@oracle.com>
Message-ID: 

>From a tck point of view, Process has always seems untestable, since any
process creation can fail at any time for any (external) reason.  Adding
special handling for OSes where a Process can never be created doesn't
help... please explain.

My feeling that we should consistently fail with IOException is hardening.
It's reasonable to throw a subclass that explains that you're on an OS
where no subprocesses are allowed, or you can only start subprocesses after
7pm, or the only command you can run is { "cthulu" }.

On Sat, Jan 31, 2015 at 1:03 PM, Alan Bateman 
wrote:

> On 31/01/2015 16:15, Martin Buchholz wrote:
>
>> It's not a big deal, but I am opposed to this change.
>>
> Just an FYI that Roger seems to have pushed the original patch, this new
> patch just moves the text down so that it flows a bit better.
>
>  The old spec
>>
>>      * 
In such cases an exception will be thrown. The exact nature
>>      * of the exception is system-dependent, but it will always be a
>>      * subclass of {@link IOException}.
>>
>> is perfectly adequate for OSes with no subprocesses.
>> Users should be catching and handling IOException in any case.  Throwing
>> a RuntimeException seems wrong, and breaks the above promise!
>>
>>  It's lack of clarity in the above text that has been causing the
> testability issue for so long so I think it is good to make it clear how an
> implementation that does not support processes should behave. The options
> on the table seem to be to define a sub-type of IOE for this case, specify
> that an IOE be thrown with an UOE as the cause, or just throw UOE when this
> feature is not supported.
>
> -Alan
>
>
>
>


From Alan.Bateman at oracle.com  Sat Jan 31 21:33:08 2015
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Sat, 31 Jan 2015 21:33:08 +0000
Subject: RFR 8064924: Update java.net.URL to work with modules
In-Reply-To: <54CD3043.2070709@gmail.com>
References: 
	<54CD3043.2070709@gmail.com>
Message-ID: <54CD4A14.2080205@oracle.com>

On 31/01/2015 19:42, Peter Levart wrote:
> Hi Chris,
>
> I looked at your solution to make URLStreamHandlerFactory interface a 
> service provider using ServiceLoader API and in addition adding new 
> URL static method to programmaticaly register 
> URLStreamHandlerFactories. There are a couple of issues with your 
> approach I'd like to discuss.
>
> The programmatic approach using static URL method does give you some 
> means of order in which registered URLStreamHandlerFactories are tried 
> to create URLStreamHandler for particular protocol - the registration 
> order. It means that this method should only be called by one "party" 
> in the VM or else it is hard to control the order of registration.
>
> ServiceLoader is a declarative approach, but does not give you order 
> by default. Also, your ServiceLoader approach gives a way for 
> URLStreamHandlerFactories to register in the system without security 
> checks. If a particular 
> META-INF/services/java.net.URLStreamHandlerFactory is found, it's 
> content is used to load a class and instantiate a factory which is 
> used in URL constructors then. Previously, one had to have a 
> "setFactory" permission to set the URLStreamHandlerFactory or 
> appropriate PropertyPermission for seting the package prefix property. 
> This can be fixed.
>
> :
>
> Anyway, I think there is an alternative to programmatic registration 
> approach of URLStreamHandlerFactories. Using just ServiceLoader and a 
> new default method on URLStreamHandlerFactory interface to provide 
> order. Here's what I'm thinking about:
>
> http://cr.openjdk.java.net/~plevart/jdk9-dev/URLStreamHandlerFactory/webrev.01/

I don't think we should we expose ordering values in 
URLStreamHandlerFactory, it looks a bit odd and not clear how an 
implementation can choose a useful value. There is a general issue that 
ServiceLoader doesn't currently support a means to order service 
providers but I think we can re-examine that when we move to modules and 
and linking. That is, have a consistent way to configure ordering that 
we can use everywhere that ServiceLoader is used rather than doing 
one-off solutions.

The other thing is that it's not clear how this would work for a factory 
for the jar protocol that is itself deployed in a JAR file on the class 
path. This is important for containers that want to do their own caching 
and they want their jar protocol handler configured system-wide before 
starting any applications. It's also part of the motive for the 
addURLStreamHandlerFactory in the original proposal.

I think you have good point with the setFactory permission, that does 
need to be looked at.

-Alan.


From peter.levart at gmail.com  Sat Jan 31 23:47:16 2015
From: peter.levart at gmail.com (Peter Levart)
Date: Sun, 01 Feb 2015 00:47:16 +0100
Subject: RFR 8064924: Update java.net.URL to work with modules
In-Reply-To: <54CD4A14.2080205@oracle.com>
References: 
	<54CD3043.2070709@gmail.com> <54CD4A14.2080205@oracle.com>
Message-ID: <54CD6984.2040004@gmail.com>

Hi Alan,

On 01/31/2015 10:33 PM, Alan Bateman wrote:
> On 31/01/2015 19:42, Peter Levart wrote:
>> Hi Chris,
>>
>> I looked at your solution to make URLStreamHandlerFactory interface a 
>> service provider using ServiceLoader API and in addition adding new 
>> URL static method to programmaticaly register  
>> URLStreamHandlerFactories. There are a couple of issues with your 
>> approach I'd like to discuss.
>>
>> The programmatic approach using static URL method does give you some 
>> means of order in which registered URLStreamHandlerFactories are 
>> tried to create URLStreamHandler for particular protocol - the 
>> registration order. It means that this method should only be called 
>> by one "party" in the VM or else it is hard to control the order of 
>> registration.
>>
>> ServiceLoader is a declarative approach, but does not give you order 
>> by default. Also, your ServiceLoader approach gives a way for 
>> URLStreamHandlerFactories to register in the system without security 
>> checks. If a particular 
>> META-INF/services/java.net.URLStreamHandlerFactory is found, it's 
>> content is used to load a class and instantiate a factory which is 
>> used in URL constructors then. Previously, one had to have a 
>> "setFactory" permission to set the URLStreamHandlerFactory or 
>> appropriate PropertyPermission for seting the package prefix 
>> property. This can be fixed.
>>
>> :
>>
>> Anyway, I think there is an alternative to programmatic registration 
>> approach of URLStreamHandlerFactories. Using just ServiceLoader and a 
>> new default method on URLStreamHandlerFactory interface to provide 
>> order. Here's what I'm thinking about:
>>
>> http://cr.openjdk.java.net/~plevart/jdk9-dev/URLStreamHandlerFactory/webrev.01/
>
> I don't think we should we expose ordering values in 
> URLStreamHandlerFactory, it looks a bit odd and not clear how an 
> implementation can choose a useful value. There is a general issue 
> that ServiceLoader doesn't currently support a means to order service 
> providers but I think we can re-examine that when we move to modules 
> and and linking. That is, have a consistent way to configure ordering 
> that we can use everywhere that ServiceLoader is used rather than 
> doing one-off solutions.

I agree. Putting the order on the SPI API is not the right solution. The 
order should be configured in one place. But there needs to be some kind 
of handle each service exposes for order configuration to reference. So 
one idea how to extend the ServiceLoader mechanism is this:

create a special class-scope runtime annotation...

public @interface ServiceProvider {
         String name();
}

...with which service implementation classes can optionally be 
annotated. This could enable ServiceLoader API extension like:

     public static  ServiceLoader load(Class service, String 
serviceProviderName)

that would return an Iterable over implementations that are annotated 
with a particular @ServiceProvider(name = ...) annotation (similar to 
security Providers but simpler).

In addition one could specify a system property with the key being 
prefixed by service interface FQ class name, like:

java.net.URLStreamHandlerFactory.serviceLoaderOrder=providerName1,providerName2,providerName3,...



>
> The other thing is that it's not clear how this would work for a 
> factory for the jar protocol that is itself deployed in a JAR file on 
> the class path. This is important for containers that want to do their 
> own caching and they want their jar protocol handler configured 
> system-wide before starting any applications. It's also part of the 
> motive for the addURLStreamHandlerFactory in the original proposal.

I see. But isn't URL.setURLStreamHandlerFactory() enough for that 
purpose? It can only be set once, but there can only be *one* container 
that wants it's jar protocol handler configured system-wide.

Regards, Peter

>
> I think you have good point with the setFactory permission, that does 
> need to be looked at.
>
> -Alan.