From peter.levart at gmail.com Tue Jul 1 18:35:57 2014 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 01 Jul 2014 20:35:57 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) Message-ID: <53B2FF8D.5040503@gmail.com> Hi, I propose a patch for this issue: https://bugs.openjdk.java.net/browse/JDK-7186258 The motivation to re-design caching of InetAddress-es was not this issue though, but a desire to attack synchronization bottlenecks in methods like URL.equals and URL.hashCode which use host name to IP address mapping. I plan to tackle the synchronization in URL in a follow-up proposal, but I wanted to 1st iron-out the "leaves" of the call-tree. Here's the proposed patch: http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ sun.net.InetAddressCachePolicy: - two static methods (get() and getNegative()) were synchronized. Removed synchronization and made underlying fields volatile. - also added a normalization of negative policy in setNegativeIfNotSet(). The logic in InetAddress doesn't cope with negative values distinct from InetAddressCachePolicy.FOREVER (-1), so this was a straight bug. The setIfNotSet() doesn't need this normalization, because checkValue() throws exception if passed-in value < InetAddressCachePolicy.FOREVER. java.net.InetAddress: - complete redesign of caching. Instead of distinct Positive/Negative caches, there's only one cache - a ConcurrentHashMap. The value in the map knows if it contains positive or negative answer. - the design of this cache is similar but much simpler than java.lang.reflect.WeakCache, since it doesn't have to deal with WeakReferences and keys are simpler (just strings - hostnames). Similarity is in how concurrent requests for the same key (hostname) are synchronized when the entry is not cached yet, but still avoid synchronization when entry is cached. This preserves the behaviour of original InetAddress caching code but simplifies it greatly (100+ lines removed). - I tried to preserve the interaction between InetAddress.getLocalHost() and InetAddress.getByName(). The getLocalHost() caches the local host address for 5 seconds privately. When it expires it performs new name service look-up and "refreshes" the entry in the InetAddress.getByName() cache although it has not expired yet. I think this is meant to prevent surprises when getLocalHost() returns newer address than getByName() which is called after that. - I also fixed the JDK-7186258 as a by-product (but don't know yet how to write a test for this issue - any ideas?) I created a JMH benchmark that tests the following methods: - InetAddress.getLocalHost() - InetAddress.getByName() (with positive and negative answer) Here're the results of running on my 4-core (8-threads) i7/Linux: http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf The getByNameNegative() test does not show much improvement in patched vs. original code. That's because by default the policy is to NOT cache negative answers. Requests for same hostname to the NameService(s) are synchronized. If "networkaddress.cache.negative.ttl" system property is set to some positive value, results are similar to those of getByNamePositive() test (the default policy for positive caching is 30 seconds). I ran the jtreg tests in test/java/net and have the same score as with original unpatched code. I have 3 failing tests from original and patched runs: JT Harness : Tests that failed java/net/MulticastSocket/Promiscuous.java: Test for interference when two sockets are bound to the same port but joined to different multicast groups java/net/MulticastSocket/SetLoopbackMode.java: Test MulticastSocket.setLoopbackMode java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken on Linux And 1 test that had error trying to be run: JT Harness : Tests that had errors java/net/URLPermission/nstest/lookup.sh: Because of: test result: Error. Can't find source file: jdk/testlibrary/*.java in directory-list: /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary All other 258 java/net tests pass. So what do you think? Regards, Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From martinrb at google.com Tue Jul 1 20:04:49 2014 From: martinrb at google.com (Martin Buchholz) Date: Tue, 1 Jul 2014 13:04:49 -0700 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53B2FF8D.5040503@gmail.com> References: <53B2FF8D.5040503@gmail.com> Message-ID: I haven't looked at the patch, but generally ... all uses of currentTimeMillis to measure elapsed time should be migrated to use difference of two nanoTime values, and such a change should be done independently of other changes. IIRC lookup.sh is a known flaky test being fixed elsewhere. On Tue, Jul 1, 2014 at 11:35 AM, Peter Levart wrote: > Hi, > > I propose a patch for this issue: > > https://bugs.openjdk.java.net/browse/JDK-7186258 > > The motivation to re-design caching of InetAddress-es was not this issue > though, but a desire to attack synchronization bottlenecks in methods like > URL.equals and URL.hashCode which use host name to IP address mapping. I > plan to tackle the synchronization in URL in a follow-up proposal, but I > wanted to 1st iron-out the "leaves" of the call-tree. Here's the proposed > patch: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ > > sun.net.InetAddressCachePolicy: > > - two static methods (get() and getNegative()) were synchronized. Removed > synchronization and made underlying fields volatile. > - also added a normalization of negative policy in setNegativeIfNotSet(). > The logic in InetAddress doesn't cope with negative values distinct from > InetAddressCachePolicy.FOREVER (-1), so this was a straight bug. The > setIfNotSet() doesn't need this normalization, because checkValue() throws > exception if passed-in value < InetAddressCachePolicy.FOREVER. > > java.net.InetAddress: > > - complete redesign of caching. Instead of distinct Positive/Negative > caches, there's only one cache - a ConcurrentHashMap. The value in the map > knows if it contains positive or negative answer. > - the design of this cache is similar but much simpler than > java.lang.reflect.WeakCache, since it doesn't have to deal with > WeakReferences and keys are simpler (just strings - hostnames). Similarity > is in how concurrent requests for the same key (hostname) are synchronized > when the entry is not cached yet, but still avoid synchronization when > entry is cached. This preserves the behaviour of original InetAddress > caching code but simplifies it greatly (100+ lines removed). > - I tried to preserve the interaction between InetAddress.getLocalHost() > and InetAddress.getByName(). The getLocalHost() caches the local host > address for 5 seconds privately. When it expires it performs new name > service look-up and "refreshes" the entry in the InetAddress.getByName() > cache although it has not expired yet. I think this is meant to prevent > surprises when getLocalHost() returns newer address than getByName() which > is called after that. > - I also fixed the JDK-7186258 as a by-product (but don't know yet how to > write a test for this issue - any ideas?) > > I created a JMH benchmark that tests the following methods: > > - InetAddress.getLocalHost() > - InetAddress.getByName() (with positive and negative answer) > > Here're the results of running on my 4-core (8-threads) i7/Linux: > > > http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf > > The getByNameNegative() test does not show much improvement in patched vs. > original code. That's because by default the policy is to NOT cache > negative answers. Requests for same hostname to the NameService(s) are > synchronized. If "networkaddress.cache.negative.ttl" system property is set > to some positive value, results are similar to those of getByNamePositive() > test (the default policy for positive caching is 30 seconds). > > I ran the jtreg tests in test/java/net and have the same score as with > original unpatched code. I have 3 failing tests from original and patched > runs: > > JT Harness : Tests that failed > java/net/MulticastSocket/Promiscuous.java: Test for interference when two > sockets are bound to the same port but joined to different multicast groups > java/net/MulticastSocket/SetLoopbackMode.java: Test > MulticastSocket.setLoopbackMode > java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken on > Linux > > And 1 test that had error trying to be run: > > JT Harness : Tests that had errors > java/net/URLPermission/nstest/lookup.sh: > > Because of: > > test result: Error. Can't find source file: jdk/testlibrary/*.java in > directory-list: > /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest > /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary > > All other 258 java/net tests pass. > > > > So what do you think? > > > Regards, Peter > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernd-2014 at eckenfels.net Tue Jul 1 20:06:21 2014 From: bernd-2014 at eckenfels.net (Bernd Eckenfels) Date: Tue, 1 Jul 2014 22:06:21 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53B2FF8D.5040503@gmail.com> References: <53B2FF8D.5040503@gmail.com> Message-ID: <20140701220621.00005e21.bernd-2014@eckenfels.net> Looks good, like it, Peter. some nits: instead of adding createTime and cacheNanos, only have a expireAfter? L782: is it better to use putIfAbsent unconditionally, instead of get/putIfAbsent in NameServicdeAddr? L732: I am unsure about the id field, isnt it enough to have the identity equality check for the replacement check and otherwise depend on equals()? L1223: What about moving the cache exiry inside the if (useCache) BTW1: might be the wrong RFR, but considering your good performance numbers for an active cache, would having 100ms or similiar default negative cache time make sense without impacting (visible) the semantic. Gruss Bernd Am Tue, 01 Jul 2014 20:35:57 +0200 schrieb Peter Levart : > Hi, > > I propose a patch for this issue: > > https://bugs.openjdk.java.net/browse/JDK-7186258 > > The motivation to re-design caching of InetAddress-es was not this > issue though, but a desire to attack synchronization bottlenecks in > methods like URL.equals and URL.hashCode which use host name to IP > address mapping. I plan to tackle the synchronization in URL in a > follow-up proposal, but I wanted to 1st iron-out the "leaves" of the > call-tree. Here's the proposed patch: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ > > sun.net.InetAddressCachePolicy: > > - two static methods (get() and getNegative()) were synchronized. > Removed synchronization and made underlying fields volatile. > - also added a normalization of negative policy in > setNegativeIfNotSet(). The logic in InetAddress doesn't cope with > negative values distinct from InetAddressCachePolicy.FOREVER (-1), so > this was a straight bug. The setIfNotSet() doesn't need this > normalization, because checkValue() throws exception if passed-in > value < InetAddressCachePolicy.FOREVER. > > java.net.InetAddress: > > - complete redesign of caching. Instead of distinct Positive/Negative > caches, there's only one cache - a ConcurrentHashMap. The value in > the map knows if it contains positive or negative answer. > - the design of this cache is similar but much simpler than > java.lang.reflect.WeakCache, since it doesn't have to deal with > WeakReferences and keys are simpler (just strings - hostnames). > Similarity is in how concurrent requests for the same key (hostname) > are synchronized when the entry is not cached yet, but still avoid > synchronization when entry is cached. This preserves the behaviour of > original InetAddress caching code but simplifies it greatly (100+ > lines removed). > - I tried to preserve the interaction between > InetAddress.getLocalHost() and InetAddress.getByName(). The > getLocalHost() caches the local host address for 5 seconds privately. > When it expires it performs new name service look-up and "refreshes" > the entry in the InetAddress.getByName() cache although it has not > expired yet. I think this is meant to prevent surprises when > getLocalHost() returns newer address than getByName() which is called > after that. > - I also fixed the JDK-7186258 as a by-product (but don't know yet > how to write a test for this issue - any ideas?) > > I created a JMH benchmark that tests the following methods: > > - InetAddress.getLocalHost() > - InetAddress.getByName() (with positive and negative answer) > > Here're the results of running on my 4-core (8-threads) i7/Linux: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf > > > The getByNameNegative() test does not show much improvement in > patched vs. original code. That's because by default the policy is to > NOT cache negative answers. Requests for same hostname to the > NameService(s) are synchronized. If > "networkaddress.cache.negative.ttl" system property is set to some > positive value, results are similar to those of getByNamePositive() > test (the default policy for positive caching is 30 seconds). > > I ran the jtreg tests in test/java/net and have the same score as > with original unpatched code. I have 3 failing tests from original > and patched runs: > > JT Harness : Tests that failed > java/net/MulticastSocket/Promiscuous.java: Test for interference when > two sockets are bound to the same port but joined to different > multicast groups > java/net/MulticastSocket/SetLoopbackMode.java: Test > MulticastSocket.setLoopbackMode > java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken > on Linux > > And 1 test that had error trying to be run: > > JT Harness : Tests that had errors > java/net/URLPermission/nstest/lookup.sh: > > Because of: > > test result: Error. Can't find source file: jdk/testlibrary/*.java in > directory-list: > /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest > /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary > > All other 258 java/net tests pass. > > > > So what do you think? > > > Regards, Peter > > From peter.levart at gmail.com Tue Jul 1 22:45:01 2014 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 02 Jul 2014 00:45:01 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <20140701220621.00005e21.bernd-2014@eckenfels.net> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> Message-ID: <53B339ED.2030007@gmail.com> Hi Bernd, Thanks for reviewing. On 07/01/2014 10:06 PM, Bernd Eckenfels wrote: > Looks good, like it, Peter. > > some nits: instead of adding createTime and cacheNanos, only have a > expireAfter? I could, yes. I just have to be careful not co compare two expireAfter values with '<' or '>'. The javadoc for System.nanoTime() states: " To compare two nanoTime values long t0 = System.nanoTime(); ... long t1 = System.nanoTime(); one should use t1 - t0 < 0, not t1 < t0, because of the possibility of numerical overflow. " The thing is that one never knows with System.nanoTime() when its start (value 0) is. It's not defined. So a later time instant can be negative while an earlier one can be positive. But I could compare two instants which are no more than 290 years apart with no problem using (t1 - t2) <=> 0 . I'll improve this. > L782: is it better to use putIfAbsent unconditionally, instead of > get/putIfAbsent in NameServicdeAddr? I want to keep the semantics of original code that guarantees that there will only be a single look-up to the name service per hostname in a given caching period. If several threads want to look-up the same hostname at the same time, only one will do it, others will wait and then use the result of that thread. That's why I re-check the cached value after entering the synchronized block and act upon what I find there... > L732: I am unsure about the id field, isnt it enough to have the > identity equality check for the replacement check and otherwise depend > on equals()? The ConcurrentSkipListSet (based on ConcurrentSkipListMap) that is used as an ordered concurrent (but not blocking or locking) queue of items to be expired is not using hashCode/equals, but compareTo instead. So two instances of CachedAddresses must never compareTo each other as being "equal" if one wants to keep them all in the ConcurrentSkipListSet... > L1223: What about moving the cache expiry inside the if (useCache) That's a possibility. InetAddress.getLocalHost() calls will not expire cache entries then. There are also other "optimizations" possible. For example if there is a "storm" of distinct host names to be looked-up, they enter the cache at about the same time, so they expire at about the same time too. It could happen that one thread must remove a lot of expired entries at one time. Perhaps an upper bound on the number of entries that get removed in one getByName() call could be specified. > BTW1: might be the wrong RFR, but considering your good performance > numbers for an active cache, would having 100ms or similiar default > negative cache time make sense without impacting (visible) the semantic. It depends on the "application". If some app is doing more than 10 look-ups of same non-existent hostname per second, it will have effect. Otherwise not. It seems that negative answers are not so frequent in practice that default policy would require caching them. One can always choose the policy manually. > > > Gruss > Bernd Regards, Peter > > Am Tue, 01 Jul 2014 20:35:57 +0200 > schrieb Peter Levart : > >> Hi, >> >> I propose a patch for this issue: >> >> https://bugs.openjdk.java.net/browse/JDK-7186258 >> >> The motivation to re-design caching of InetAddress-es was not this >> issue though, but a desire to attack synchronization bottlenecks in >> methods like URL.equals and URL.hashCode which use host name to IP >> address mapping. I plan to tackle the synchronization in URL in a >> follow-up proposal, but I wanted to 1st iron-out the "leaves" of the >> call-tree. Here's the proposed patch: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >> >> sun.net.InetAddressCachePolicy: >> >> - two static methods (get() and getNegative()) were synchronized. >> Removed synchronization and made underlying fields volatile. >> - also added a normalization of negative policy in >> setNegativeIfNotSet(). The logic in InetAddress doesn't cope with >> negative values distinct from InetAddressCachePolicy.FOREVER (-1), so >> this was a straight bug. The setIfNotSet() doesn't need this >> normalization, because checkValue() throws exception if passed-in >> value < InetAddressCachePolicy.FOREVER. >> >> java.net.InetAddress: >> >> - complete redesign of caching. Instead of distinct Positive/Negative >> caches, there's only one cache - a ConcurrentHashMap. The value in >> the map knows if it contains positive or negative answer. >> - the design of this cache is similar but much simpler than >> java.lang.reflect.WeakCache, since it doesn't have to deal with >> WeakReferences and keys are simpler (just strings - hostnames). >> Similarity is in how concurrent requests for the same key (hostname) >> are synchronized when the entry is not cached yet, but still avoid >> synchronization when entry is cached. This preserves the behaviour of >> original InetAddress caching code but simplifies it greatly (100+ >> lines removed). >> - I tried to preserve the interaction between >> InetAddress.getLocalHost() and InetAddress.getByName(). The >> getLocalHost() caches the local host address for 5 seconds privately. >> When it expires it performs new name service look-up and "refreshes" >> the entry in the InetAddress.getByName() cache although it has not >> expired yet. I think this is meant to prevent surprises when >> getLocalHost() returns newer address than getByName() which is called >> after that. >> - I also fixed the JDK-7186258 as a by-product (but don't know yet >> how to write a test for this issue - any ideas?) >> >> I created a JMH benchmark that tests the following methods: >> >> - InetAddress.getLocalHost() >> - InetAddress.getByName() (with positive and negative answer) >> >> Here're the results of running on my 4-core (8-threads) i7/Linux: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >> >> >> The getByNameNegative() test does not show much improvement in >> patched vs. original code. That's because by default the policy is to >> NOT cache negative answers. Requests for same hostname to the >> NameService(s) are synchronized. If >> "networkaddress.cache.negative.ttl" system property is set to some >> positive value, results are similar to those of getByNamePositive() >> test (the default policy for positive caching is 30 seconds). >> >> I ran the jtreg tests in test/java/net and have the same score as >> with original unpatched code. I have 3 failing tests from original >> and patched runs: >> >> JT Harness : Tests that failed >> java/net/MulticastSocket/Promiscuous.java: Test for interference when >> two sockets are bound to the same port but joined to different >> multicast groups >> java/net/MulticastSocket/SetLoopbackMode.java: Test >> MulticastSocket.setLoopbackMode >> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken >> on Linux >> >> And 1 test that had error trying to be run: >> >> JT Harness : Tests that had errors >> java/net/URLPermission/nstest/lookup.sh: >> >> Because of: >> >> test result: Error. Can't find source file: jdk/testlibrary/*.java in >> directory-list: >> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >> >> All other 258 java/net tests pass. >> >> >> >> So what do you think? >> >> >> Regards, Peter >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernd-2014 at eckenfels.net Tue Jul 1 22:58:53 2014 From: bernd-2014 at eckenfels.net (Bernd Eckenfels) Date: Wed, 2 Jul 2014 00:58:53 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53B339ED.2030007@gmail.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B339ED.2030007@gmail.com> Message-ID: <20140702005853.000054d5.bernd-2014@eckenfels.net> Hello, Am Wed, 02 Jul 2014 00:45:01 +0200 schrieb Peter Levart : > > L782: is it better to use putIfAbsent unconditionally, instead of > > get/putIfAbsent in NameServicdeAddr? > > I want to keep the semantics of original code that guarantees that > there will only be a single look-up to the name service per hostname > in a given caching period. If several threads want to look-up the > same hostname at the same time, only one will do it, others will wait > and then use the result of that thread. That's why I re-check the > cached value after entering the synchronized block and act upon what > I find there... Yes, I was talking about the get inside the synchronized with same semantic, but I havent completely tested my thinking through (idea was to re-use putIfAbsent for the first get): synchronized (this) { // re-check that we are still us addresses = cache.putIfAbsent(host, this); // address is null if we had added us // if address did not change it is this > > > L732: I am unsure about the id field, isnt it enough to have the > > identity equality check for the replacement check and otherwise > > depend on equals()? > > The ConcurrentSkipListSet (based on ConcurrentSkipListMap) that is > used as an ordered concurrent (but not blocking or locking) queue of > items to be expired is not using hashCode/equals, but compareTo > instead. So two instances of CachedAddresses must never compareTo > each other as being "equal" if one wants to keep them all in the > ConcurrentSkipListSet... Yes, I mean not using id in compareTo but idendity if (other==this) return 0; But maybe not worth it as you still need to order identical entries reliable (maybe by system hash). > > BTW1: might be the wrong RFR, but considering your good performance > > numbers for an active cache, would having 100ms or similiar default > > negative cache time make sense without impacting (visible) the > > semantic. > > It depends on the "application". If some app is doing more than 10 > look-ups of same non-existent hostname per second, it will have > effect. Otherwise not. It seems that negative answers are not so > frequent in practice that default policy would require caching them. > One can always choose the policy manually. Actually I think an application who retries lookups without rate limit might not be too uncommon as they typically fail only after timeouts. However if you have for example a dropped default route nameserver answer can return the error immediatelly and you get into a busy loop. But yes most likely most apps only try it a small time. Gruss Bernd From peter.levart at gmail.com Wed Jul 2 06:33:32 2014 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 02 Jul 2014 08:33:32 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <20140702005853.000054d5.bernd-2014@eckenfels.net> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B339ED.2030007@gmail.com> <20140702005853.000054d5.bernd-2014@eckenfels.net> Message-ID: <53B3A7BC.7070803@gmail.com> On 07/02/2014 12:58 AM, Bernd Eckenfels wrote: > Hello, > > Am Wed, 02 Jul 2014 00:45:01 +0200 > schrieb Peter Levart : >>> L782: is it better to use putIfAbsent unconditionally, instead of >>> get/putIfAbsent in NameServicdeAddr? >> I want to keep the semantics of original code that guarantees that >> there will only be a single look-up to the name service per hostname >> in a given caching period. If several threads want to look-up the >> same hostname at the same time, only one will do it, others will wait >> and then use the result of that thread. That's why I re-check the >> cached value after entering the synchronized block and act upon what >> I find there... > Yes, I was talking about the get inside the synchronized with same > semantic, but I havent completely tested my thinking through (idea was > to re-use putIfAbsent for the first get): > > synchronized (this) { > // re-check that we are still us > addresses = cache.putIfAbsent(host, this); > // address is null if we had added us > // if address did not change it is this You're right. A single putIfAbsent can do it. Thanks for pointing that out. Regards, Peter > > > >>> L732: I am unsure about the id field, isnt it enough to have the >>> identity equality check for the replacement check and otherwise >>> depend on equals()? >> The ConcurrentSkipListSet (based on ConcurrentSkipListMap) that is >> used as an ordered concurrent (but not blocking or locking) queue of >> items to be expired is not using hashCode/equals, but compareTo >> instead. So two instances of CachedAddresses must never compareTo >> each other as being "equal" if one wants to keep them all in the >> ConcurrentSkipListSet... > Yes, I mean not using id in compareTo but idendity if (other==this) > return 0; But maybe not worth it as you still need to order identical > entries reliable (maybe by system hash). > >>> BTW1: might be the wrong RFR, but considering your good performance >>> numbers for an active cache, would having 100ms or similiar default >>> negative cache time make sense without impacting (visible) the >>> semantic. >> It depends on the "application". If some app is doing more than 10 >> look-ups of same non-existent hostname per second, it will have >> effect. Otherwise not. It seems that negative answers are not so >> frequent in practice that default policy would require caching them. >> One can always choose the policy manually. > Actually I think an application who retries lookups without rate limit > might not be too uncommon as they typically fail only after timeouts. > However if you have for example a dropped default route nameserver > answer can return the error immediatelly and you get into a busy loop. > But yes most likely most apps only try it a small time. > > Gruss > Bernd From peter.levart at gmail.com Wed Jul 2 11:56:39 2014 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 02 Jul 2014 13:56:39 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <20140701220621.00005e21.bernd-2014@eckenfels.net> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> Message-ID: <53B3F377.3090208@gmail.com> Hi, I updated the webrev with first two suggestions from Bernd (expireTime instead of createTime and cacheNanos + only use putIfAbsent instead of get followed by putIfAbsent): http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.02/ Thanks, Bernd. The id field in CachedAddresses is necessary for compareTo to never return 0 for two different instances (used as element in ConcurrentSkipListSet). For last two suggestions I'm not sure whether they are desired, so I'm currently leaving them as is. Regards, Peter On 07/01/2014 10:06 PM, Bernd Eckenfels wrote: > Looks good, like it, Peter. > > some nits: instead of adding createTime and cacheNanos, only have a > expireAfter? > > L782: is it better to use putIfAbsent unconditionally, instead of > get/putIfAbsent in NameServicdeAddr? > > L732: I am unsure about the id field, isnt it enough to have the > identity equality check for the replacement check and otherwise depend > on equals()? > > L1223: What about moving the cache exiry inside the if (useCache) > > BTW1: might be the wrong RFR, but considering your good performance > numbers for an active cache, would having 100ms or similiar default > negative cache time make sense without impacting (visible) the semantic. > > > > Gruss > Bernd > > > Am Tue, 01 Jul 2014 20:35:57 +0200 > schrieb Peter Levart : > >> Hi, >> >> I propose a patch for this issue: >> >> https://bugs.openjdk.java.net/browse/JDK-7186258 >> >> The motivation to re-design caching of InetAddress-es was not this >> issue though, but a desire to attack synchronization bottlenecks in >> methods like URL.equals and URL.hashCode which use host name to IP >> address mapping. I plan to tackle the synchronization in URL in a >> follow-up proposal, but I wanted to 1st iron-out the "leaves" of the >> call-tree. Here's the proposed patch: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >> >> sun.net.InetAddressCachePolicy: >> >> - two static methods (get() and getNegative()) were synchronized. >> Removed synchronization and made underlying fields volatile. >> - also added a normalization of negative policy in >> setNegativeIfNotSet(). The logic in InetAddress doesn't cope with >> negative values distinct from InetAddressCachePolicy.FOREVER (-1), so >> this was a straight bug. The setIfNotSet() doesn't need this >> normalization, because checkValue() throws exception if passed-in >> value < InetAddressCachePolicy.FOREVER. >> >> java.net.InetAddress: >> >> - complete redesign of caching. Instead of distinct Positive/Negative >> caches, there's only one cache - a ConcurrentHashMap. The value in >> the map knows if it contains positive or negative answer. >> - the design of this cache is similar but much simpler than >> java.lang.reflect.WeakCache, since it doesn't have to deal with >> WeakReferences and keys are simpler (just strings - hostnames). >> Similarity is in how concurrent requests for the same key (hostname) >> are synchronized when the entry is not cached yet, but still avoid >> synchronization when entry is cached. This preserves the behaviour of >> original InetAddress caching code but simplifies it greatly (100+ >> lines removed). >> - I tried to preserve the interaction between >> InetAddress.getLocalHost() and InetAddress.getByName(). The >> getLocalHost() caches the local host address for 5 seconds privately. >> When it expires it performs new name service look-up and "refreshes" >> the entry in the InetAddress.getByName() cache although it has not >> expired yet. I think this is meant to prevent surprises when >> getLocalHost() returns newer address than getByName() which is called >> after that. >> - I also fixed the JDK-7186258 as a by-product (but don't know yet >> how to write a test for this issue - any ideas?) >> >> I created a JMH benchmark that tests the following methods: >> >> - InetAddress.getLocalHost() >> - InetAddress.getByName() (with positive and negative answer) >> >> Here're the results of running on my 4-core (8-threads) i7/Linux: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >> >> >> The getByNameNegative() test does not show much improvement in >> patched vs. original code. That's because by default the policy is to >> NOT cache negative answers. Requests for same hostname to the >> NameService(s) are synchronized. If >> "networkaddress.cache.negative.ttl" system property is set to some >> positive value, results are similar to those of getByNamePositive() >> test (the default policy for positive caching is 30 seconds). >> >> I ran the jtreg tests in test/java/net and have the same score as >> with original unpatched code. I have 3 failing tests from original >> and patched runs: >> >> JT Harness : Tests that failed >> java/net/MulticastSocket/Promiscuous.java: Test for interference when >> two sockets are bound to the same port but joined to different >> multicast groups >> java/net/MulticastSocket/SetLoopbackMode.java: Test >> MulticastSocket.setLoopbackMode >> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken >> on Linux >> >> And 1 test that had error trying to be run: >> >> JT Harness : Tests that had errors >> java/net/URLPermission/nstest/lookup.sh: >> >> Because of: >> >> test result: Error. Can't find source file: jdk/testlibrary/*.java in >> directory-list: >> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >> >> All other 258 java/net tests pass. >> >> >> >> So what do you think? >> >> >> Regards, Peter >> >> From claes.redestad at oracle.com Wed Jul 2 14:39:51 2014 From: claes.redestad at oracle.com (Claes Redestad) Date: Wed, 02 Jul 2014 16:39:51 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53B3F377.3090208@gmail.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> Message-ID: <53B419B7.1030600@oracle.com> Hi Peter, perhaps the synchronized(this)-block in NameServiceAddresses::get() can be replaced with a ReentrantLock? Applying this on top of your patch, I seem to improve scores on your benchmark for -t 4 by ~33% on my machine: --- a/src/share/classes/java/net/InetAddress.java Wed Jul 02 14:47:40 2014 +0200 +++ b/src/share/classes/java/net/InetAddress.java Wed Jul 02 14:57:24 2014 +0200 @@ -42,6 +42,7 @@ import java.util.concurrent.ConcurrentMap; import java.util.concurrent.ConcurrentSkipListSet; import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.locks.ReentrantLock; import sun.security.action.*; import sun.net.InetAddressCachePolicy; @@ -763,6 +764,7 @@ private static final class NameServiceAddresses implements Addresses { private final String host; private final InetAddress reqAddr; + private final ReentrantLock lock = new ReentrantLock(); NameServiceAddresses(String host, InetAddress reqAddr) { this.host = host; @@ -774,7 +776,8 @@ Addresses addresses; // only one thread is doing lookup to name service // for particular host at any time. - synchronized (this) { + lock.lock(); + try { // re-check that we are still us + re-install us if slot empty addresses = cache.putIfAbsent(host, this); if (addresses == null) { @@ -822,10 +825,13 @@ return inetAddresses; } // else addresses != this + + // delegate to different addresses when we are already replaced + // but outside of synchronized block to avoid any chance of dead-locking + return addresses.get(); + } finally { + lock.unlock(); } - // delegate to different addresses when we are already replaced - // but outside of synchronized block to avoid any chance of dead-locking - return addresses.get(); } } nit: line 1258: getAddressesFromNameService made package private but not used outside of InetAddress? Generally I think this looks good, but I'm not a reviewer. :) /Claes On 07/02/2014 01:56 PM, Peter Levart wrote: > Hi, > > I updated the webrev with first two suggestions from Bernd (expireTime > instead of createTime and cacheNanos + only use putIfAbsent instead of > get followed by putIfAbsent): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.02/ > > Thanks, Bernd. > > The id field in CachedAddresses is necessary for compareTo to never > return 0 for two different instances (used as element in > ConcurrentSkipListSet). > > For last two suggestions I'm not sure whether they are desired, so I'm > currently leaving them as is. > > > Regards, Peter > > On 07/01/2014 10:06 PM, Bernd Eckenfels wrote: >> Looks good, like it, Peter. >> >> some nits: instead of adding createTime and cacheNanos, only have a >> expireAfter? >> >> L782: is it better to use putIfAbsent unconditionally, instead of >> get/putIfAbsent in NameServicdeAddr? >> >> L732: I am unsure about the id field, isnt it enough to have the >> identity equality check for the replacement check and otherwise depend >> on equals()? >> >> L1223: What about moving the cache exiry inside the if (useCache) >> >> BTW1: might be the wrong RFR, but considering your good performance >> numbers for an active cache, would having 100ms or similiar default >> negative cache time make sense without impacting (visible) the semantic. >> >> >> >> Gruss >> Bernd >> >> >> Am Tue, 01 Jul 2014 20:35:57 +0200 >> schrieb Peter Levart : >> >>> Hi, >>> >>> I propose a patch for this issue: >>> >>> https://bugs.openjdk.java.net/browse/JDK-7186258 >>> >>> The motivation to re-design caching of InetAddress-es was not this >>> issue though, but a desire to attack synchronization bottlenecks in >>> methods like URL.equals and URL.hashCode which use host name to IP >>> address mapping. I plan to tackle the synchronization in URL in a >>> follow-up proposal, but I wanted to 1st iron-out the "leaves" of the >>> call-tree. Here's the proposed patch: >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >>> >>> >>> sun.net.InetAddressCachePolicy: >>> >>> - two static methods (get() and getNegative()) were synchronized. >>> Removed synchronization and made underlying fields volatile. >>> - also added a normalization of negative policy in >>> setNegativeIfNotSet(). The logic in InetAddress doesn't cope with >>> negative values distinct from InetAddressCachePolicy.FOREVER (-1), so >>> this was a straight bug. The setIfNotSet() doesn't need this >>> normalization, because checkValue() throws exception if passed-in >>> value < InetAddressCachePolicy.FOREVER. >>> >>> java.net.InetAddress: >>> >>> - complete redesign of caching. Instead of distinct Positive/Negative >>> caches, there's only one cache - a ConcurrentHashMap. The value in >>> the map knows if it contains positive or negative answer. >>> - the design of this cache is similar but much simpler than >>> java.lang.reflect.WeakCache, since it doesn't have to deal with >>> WeakReferences and keys are simpler (just strings - hostnames). >>> Similarity is in how concurrent requests for the same key (hostname) >>> are synchronized when the entry is not cached yet, but still avoid >>> synchronization when entry is cached. This preserves the behaviour of >>> original InetAddress caching code but simplifies it greatly (100+ >>> lines removed). >>> - I tried to preserve the interaction between >>> InetAddress.getLocalHost() and InetAddress.getByName(). The >>> getLocalHost() caches the local host address for 5 seconds privately. >>> When it expires it performs new name service look-up and "refreshes" >>> the entry in the InetAddress.getByName() cache although it has not >>> expired yet. I think this is meant to prevent surprises when >>> getLocalHost() returns newer address than getByName() which is called >>> after that. >>> - I also fixed the JDK-7186258 as a by-product (but don't know yet >>> how to write a test for this issue - any ideas?) >>> >>> I created a JMH benchmark that tests the following methods: >>> >>> - InetAddress.getLocalHost() >>> - InetAddress.getByName() (with positive and negative answer) >>> >>> Here're the results of running on my 4-core (8-threads) i7/Linux: >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >>> >>> >>> >>> The getByNameNegative() test does not show much improvement in >>> patched vs. original code. That's because by default the policy is to >>> NOT cache negative answers. Requests for same hostname to the >>> NameService(s) are synchronized. If >>> "networkaddress.cache.negative.ttl" system property is set to some >>> positive value, results are similar to those of getByNamePositive() >>> test (the default policy for positive caching is 30 seconds). >>> >>> I ran the jtreg tests in test/java/net and have the same score as >>> with original unpatched code. I have 3 failing tests from original >>> and patched runs: >>> >>> JT Harness : Tests that failed >>> java/net/MulticastSocket/Promiscuous.java: Test for interference when >>> two sockets are bound to the same port but joined to different >>> multicast groups >>> java/net/MulticastSocket/SetLoopbackMode.java: Test >>> MulticastSocket.setLoopbackMode >>> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken >>> on Linux >>> >>> And 1 test that had error trying to be run: >>> >>> JT Harness : Tests that had errors >>> java/net/URLPermission/nstest/lookup.sh: >>> >>> Because of: >>> >>> test result: Error. Can't find source file: jdk/testlibrary/*.java in >>> directory-list: >>> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >>> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >>> >>> All other 258 java/net tests pass. >>> >>> >>> >>> So what do you think? >>> >>> >>> Regards, Peter >>> >>> > From peter.levart at gmail.com Wed Jul 2 15:30:30 2014 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 02 Jul 2014 17:30:30 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53B419B7.1030600@oracle.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> <53B419B7.1030600@oracle.com> Message-ID: <53B42596.3080900@gmail.com> Hi Claes, Thank you for looking into this patch. On 07/02/2014 04:39 PM, Claes Redestad wrote: > Hi Peter, > > perhaps the synchronized(this)-block in NameServiceAddresses::get() > can be replaced with a ReentrantLock? Applying this on top of your > patch, I seem to improve scores on your benchmark for -t 4 by ~33% on > my machine: Which test? I would be surprised that this change has any impact but on "getByNameNegative" test which exhibits this lock at each iteration (since negative caching is disabled by default). I'll check this myself too. > > --- a/src/share/classes/java/net/InetAddress.java Wed Jul 02 > 14:47:40 2014 +0200 > +++ b/src/share/classes/java/net/InetAddress.java Wed Jul 02 > 14:57:24 2014 +0200 > @@ -42,6 +42,7 @@ > import java.util.concurrent.ConcurrentMap; > import java.util.concurrent.ConcurrentSkipListSet; > import java.util.concurrent.atomic.AtomicLong; > +import java.util.concurrent.locks.ReentrantLock; > > import sun.security.action.*; > import sun.net.InetAddressCachePolicy; > @@ -763,6 +764,7 @@ > private static final class NameServiceAddresses implements > Addresses { > private final String host; > private final InetAddress reqAddr; > + private final ReentrantLock lock = new ReentrantLock(); > > NameServiceAddresses(String host, InetAddress reqAddr) { > this.host = host; > @@ -774,7 +776,8 @@ > Addresses addresses; > // only one thread is doing lookup to name service > // for particular host at any time. > - synchronized (this) { > + lock.lock(); > + try { > // re-check that we are still us + re-install us if > slot empty > addresses = cache.putIfAbsent(host, this); > if (addresses == null) { > @@ -822,10 +825,13 @@ > return inetAddresses; > } > // else addresses != this > + > + // delegate to different addresses when we are > already replaced > + // but outside of synchronized block to avoid any > chance of dead-locking > + return addresses.get(); > + } finally { > + lock.unlock(); > } > - // delegate to different addresses when we are already > replaced > - // but outside of synchronized block to avoid any chance > of dead-locking > - return addresses.get(); > } > } > > nit: line 1258: getAddressesFromNameService made package private but > not used outside of InetAddress? Good catch. I seem to have inadvertently deleted the "private" keyword. > > Generally I think this looks good, but I'm not a reviewer. :) > > /Claes Regards, Peter > > On 07/02/2014 01:56 PM, Peter Levart wrote: >> Hi, >> >> I updated the webrev with first two suggestions from Bernd >> (expireTime instead of createTime and cacheNanos + only use >> putIfAbsent instead of get followed by putIfAbsent): >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.02/ >> >> >> Thanks, Bernd. >> >> The id field in CachedAddresses is necessary for compareTo to never >> return 0 for two different instances (used as element in >> ConcurrentSkipListSet). >> >> For last two suggestions I'm not sure whether they are desired, so >> I'm currently leaving them as is. >> >> >> Regards, Peter >> >> On 07/01/2014 10:06 PM, Bernd Eckenfels wrote: >>> Looks good, like it, Peter. >>> >>> some nits: instead of adding createTime and cacheNanos, only have a >>> expireAfter? >>> >>> L782: is it better to use putIfAbsent unconditionally, instead of >>> get/putIfAbsent in NameServicdeAddr? >>> >>> L732: I am unsure about the id field, isnt it enough to have the >>> identity equality check for the replacement check and otherwise depend >>> on equals()? >>> >>> L1223: What about moving the cache exiry inside the if (useCache) >>> >>> BTW1: might be the wrong RFR, but considering your good performance >>> numbers for an active cache, would having 100ms or similiar default >>> negative cache time make sense without impacting (visible) the >>> semantic. >>> >>> >>> >>> Gruss >>> Bernd >>> >>> >>> Am Tue, 01 Jul 2014 20:35:57 +0200 >>> schrieb Peter Levart : >>> >>>> Hi, >>>> >>>> I propose a patch for this issue: >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-7186258 >>>> >>>> The motivation to re-design caching of InetAddress-es was not this >>>> issue though, but a desire to attack synchronization bottlenecks in >>>> methods like URL.equals and URL.hashCode which use host name to IP >>>> address mapping. I plan to tackle the synchronization in URL in a >>>> follow-up proposal, but I wanted to 1st iron-out the "leaves" of the >>>> call-tree. Here's the proposed patch: >>>> >>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >>>> >>>> >>>> sun.net.InetAddressCachePolicy: >>>> >>>> - two static methods (get() and getNegative()) were synchronized. >>>> Removed synchronization and made underlying fields volatile. >>>> - also added a normalization of negative policy in >>>> setNegativeIfNotSet(). The logic in InetAddress doesn't cope with >>>> negative values distinct from InetAddressCachePolicy.FOREVER (-1), so >>>> this was a straight bug. The setIfNotSet() doesn't need this >>>> normalization, because checkValue() throws exception if passed-in >>>> value < InetAddressCachePolicy.FOREVER. >>>> >>>> java.net.InetAddress: >>>> >>>> - complete redesign of caching. Instead of distinct Positive/Negative >>>> caches, there's only one cache - a ConcurrentHashMap. The value in >>>> the map knows if it contains positive or negative answer. >>>> - the design of this cache is similar but much simpler than >>>> java.lang.reflect.WeakCache, since it doesn't have to deal with >>>> WeakReferences and keys are simpler (just strings - hostnames). >>>> Similarity is in how concurrent requests for the same key (hostname) >>>> are synchronized when the entry is not cached yet, but still avoid >>>> synchronization when entry is cached. This preserves the behaviour of >>>> original InetAddress caching code but simplifies it greatly (100+ >>>> lines removed). >>>> - I tried to preserve the interaction between >>>> InetAddress.getLocalHost() and InetAddress.getByName(). The >>>> getLocalHost() caches the local host address for 5 seconds privately. >>>> When it expires it performs new name service look-up and "refreshes" >>>> the entry in the InetAddress.getByName() cache although it has not >>>> expired yet. I think this is meant to prevent surprises when >>>> getLocalHost() returns newer address than getByName() which is called >>>> after that. >>>> - I also fixed the JDK-7186258 as a by-product (but don't know yet >>>> how to write a test for this issue - any ideas?) >>>> >>>> I created a JMH benchmark that tests the following methods: >>>> >>>> - InetAddress.getLocalHost() >>>> - InetAddress.getByName() (with positive and negative answer) >>>> >>>> Here're the results of running on my 4-core (8-threads) i7/Linux: >>>> >>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >>>> >>>> >>>> >>>> The getByNameNegative() test does not show much improvement in >>>> patched vs. original code. That's because by default the policy is to >>>> NOT cache negative answers. Requests for same hostname to the >>>> NameService(s) are synchronized. If >>>> "networkaddress.cache.negative.ttl" system property is set to some >>>> positive value, results are similar to those of getByNamePositive() >>>> test (the default policy for positive caching is 30 seconds). >>>> >>>> I ran the jtreg tests in test/java/net and have the same score as >>>> with original unpatched code. I have 3 failing tests from original >>>> and patched runs: >>>> >>>> JT Harness : Tests that failed >>>> java/net/MulticastSocket/Promiscuous.java: Test for interference when >>>> two sockets are bound to the same port but joined to different >>>> multicast groups >>>> java/net/MulticastSocket/SetLoopbackMode.java: Test >>>> MulticastSocket.setLoopbackMode >>>> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken >>>> on Linux >>>> >>>> And 1 test that had error trying to be run: >>>> >>>> JT Harness : Tests that had errors >>>> java/net/URLPermission/nstest/lookup.sh: >>>> >>>> Because of: >>>> >>>> test result: Error. Can't find source file: jdk/testlibrary/*.java in >>>> directory-list: >>>> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >>>> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >>>> >>>> All other 258 java/net tests pass. >>>> >>>> >>>> >>>> So what do you think? >>>> >>>> >>>> Regards, Peter >>>> >>>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From claes.redestad at oracle.com Wed Jul 2 15:57:54 2014 From: claes.redestad at oracle.com (Claes Redestad) Date: Wed, 02 Jul 2014 17:57:54 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53B42596.3080900@gmail.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> <53B419B7.1030600@oracle.com> <53B42596.3080900@gmail.com> Message-ID: <53B42C02.4000500@oracle.com> On 07/02/2014 05:30 PM, Peter Levart wrote: > Hi Claes, > > Thank you for looking into this patch. > > On 07/02/2014 04:39 PM, Claes Redestad wrote: >> Hi Peter, >> >> perhaps the synchronized(this)-block in NameServiceAddresses::get() >> can be replaced with a ReentrantLock? Applying this on top of your >> patch, I seem to improve scores on your benchmark for -t 4 by ~33% on >> my machine: > > Which test? I would be surprised that this change has any impact but > on "getByNameNegative" test which exhibits this lock at each iteration > (since negative caching is disabled by default). I'll check this > myself too. Feel free to ignore this. I don't seem to be able to benchmark properly today and was jumping to conclusions: rerunning with just a bit more rigor did not reproduce the improvement. In fact it doesn't even seem to benefit getByNameNegative. Perhaps you'll get different results. Thanks. /Claes > >> >> --- a/src/share/classes/java/net/InetAddress.java Wed Jul 02 >> 14:47:40 2014 +0200 >> +++ b/src/share/classes/java/net/InetAddress.java Wed Jul 02 >> 14:57:24 2014 +0200 >> @@ -42,6 +42,7 @@ >> import java.util.concurrent.ConcurrentMap; >> import java.util.concurrent.ConcurrentSkipListSet; >> import java.util.concurrent.atomic.AtomicLong; >> +import java.util.concurrent.locks.ReentrantLock; >> >> import sun.security.action.*; >> import sun.net.InetAddressCachePolicy; >> @@ -763,6 +764,7 @@ >> private static final class NameServiceAddresses implements >> Addresses { >> private final String host; >> private final InetAddress reqAddr; >> + private final ReentrantLock lock = new ReentrantLock(); >> >> NameServiceAddresses(String host, InetAddress reqAddr) { >> this.host = host; >> @@ -774,7 +776,8 @@ >> Addresses addresses; >> // only one thread is doing lookup to name service >> // for particular host at any time. >> - synchronized (this) { >> + lock.lock(); >> + try { >> // re-check that we are still us + re-install us if >> slot empty >> addresses = cache.putIfAbsent(host, this); >> if (addresses == null) { >> @@ -822,10 +825,13 @@ >> return inetAddresses; >> } >> // else addresses != this >> + >> + // delegate to different addresses when we are >> already replaced >> + // but outside of synchronized block to avoid any >> chance of dead-locking >> + return addresses.get(); >> + } finally { >> + lock.unlock(); >> } >> - // delegate to different addresses when we are already >> replaced >> - // but outside of synchronized block to avoid any chance >> of dead-locking >> - return addresses.get(); >> } >> } >> >> nit: line 1258: getAddressesFromNameService made package private but >> not used outside of InetAddress? > > Good catch. I seem to have inadvertently deleted the "private" keyword. > >> >> Generally I think this looks good, but I'm not a reviewer. :) >> >> /Claes > > Regards, Peter > >> >> On 07/02/2014 01:56 PM, Peter Levart wrote: >>> Hi, >>> >>> I updated the webrev with first two suggestions from Bernd >>> (expireTime instead of createTime and cacheNanos + only use >>> putIfAbsent instead of get followed by putIfAbsent): >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.02/ >>> >>> >>> Thanks, Bernd. >>> >>> The id field in CachedAddresses is necessary for compareTo to never >>> return 0 for two different instances (used as element in >>> ConcurrentSkipListSet). >>> >>> For last two suggestions I'm not sure whether they are desired, so >>> I'm currently leaving them as is. >>> >>> >>> Regards, Peter >>> >>> On 07/01/2014 10:06 PM, Bernd Eckenfels wrote: >>>> Looks good, like it, Peter. >>>> >>>> some nits: instead of adding createTime and cacheNanos, only have a >>>> expireAfter? >>>> >>>> L782: is it better to use putIfAbsent unconditionally, instead of >>>> get/putIfAbsent in NameServicdeAddr? >>>> >>>> L732: I am unsure about the id field, isnt it enough to have the >>>> identity equality check for the replacement check and otherwise depend >>>> on equals()? >>>> >>>> L1223: What about moving the cache exiry inside the if (useCache) >>>> >>>> BTW1: might be the wrong RFR, but considering your good performance >>>> numbers for an active cache, would having 100ms or similiar default >>>> negative cache time make sense without impacting (visible) the >>>> semantic. >>>> >>>> >>>> >>>> Gruss >>>> Bernd >>>> >>>> >>>> Am Tue, 01 Jul 2014 20:35:57 +0200 >>>> schrieb Peter Levart : >>>> >>>>> Hi, >>>>> >>>>> I propose a patch for this issue: >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-7186258 >>>>> >>>>> The motivation to re-design caching of InetAddress-es was not this >>>>> issue though, but a desire to attack synchronization bottlenecks in >>>>> methods like URL.equals and URL.hashCode which use host name to IP >>>>> address mapping. I plan to tackle the synchronization in URL in a >>>>> follow-up proposal, but I wanted to 1st iron-out the "leaves" of the >>>>> call-tree. Here's the proposed patch: >>>>> >>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >>>>> >>>>> >>>>> sun.net.InetAddressCachePolicy: >>>>> >>>>> - two static methods (get() and getNegative()) were synchronized. >>>>> Removed synchronization and made underlying fields volatile. >>>>> - also added a normalization of negative policy in >>>>> setNegativeIfNotSet(). The logic in InetAddress doesn't cope with >>>>> negative values distinct from InetAddressCachePolicy.FOREVER (-1), so >>>>> this was a straight bug. The setIfNotSet() doesn't need this >>>>> normalization, because checkValue() throws exception if passed-in >>>>> value < InetAddressCachePolicy.FOREVER. >>>>> >>>>> java.net.InetAddress: >>>>> >>>>> - complete redesign of caching. Instead of distinct Positive/Negative >>>>> caches, there's only one cache - a ConcurrentHashMap. The value in >>>>> the map knows if it contains positive or negative answer. >>>>> - the design of this cache is similar but much simpler than >>>>> java.lang.reflect.WeakCache, since it doesn't have to deal with >>>>> WeakReferences and keys are simpler (just strings - hostnames). >>>>> Similarity is in how concurrent requests for the same key (hostname) >>>>> are synchronized when the entry is not cached yet, but still avoid >>>>> synchronization when entry is cached. This preserves the behaviour of >>>>> original InetAddress caching code but simplifies it greatly (100+ >>>>> lines removed). >>>>> - I tried to preserve the interaction between >>>>> InetAddress.getLocalHost() and InetAddress.getByName(). The >>>>> getLocalHost() caches the local host address for 5 seconds privately. >>>>> When it expires it performs new name service look-up and "refreshes" >>>>> the entry in the InetAddress.getByName() cache although it has not >>>>> expired yet. I think this is meant to prevent surprises when >>>>> getLocalHost() returns newer address than getByName() which is called >>>>> after that. >>>>> - I also fixed the JDK-7186258 as a by-product (but don't know yet >>>>> how to write a test for this issue - any ideas?) >>>>> >>>>> I created a JMH benchmark that tests the following methods: >>>>> >>>>> - InetAddress.getLocalHost() >>>>> - InetAddress.getByName() (with positive and negative answer) >>>>> >>>>> Here're the results of running on my 4-core (8-threads) i7/Linux: >>>>> >>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >>>>> >>>>> >>>>> >>>>> The getByNameNegative() test does not show much improvement in >>>>> patched vs. original code. That's because by default the policy is to >>>>> NOT cache negative answers. Requests for same hostname to the >>>>> NameService(s) are synchronized. If >>>>> "networkaddress.cache.negative.ttl" system property is set to some >>>>> positive value, results are similar to those of getByNamePositive() >>>>> test (the default policy for positive caching is 30 seconds). >>>>> >>>>> I ran the jtreg tests in test/java/net and have the same score as >>>>> with original unpatched code. I have 3 failing tests from original >>>>> and patched runs: >>>>> >>>>> JT Harness : Tests that failed >>>>> java/net/MulticastSocket/Promiscuous.java: Test for interference when >>>>> two sockets are bound to the same port but joined to different >>>>> multicast groups >>>>> java/net/MulticastSocket/SetLoopbackMode.java: Test >>>>> MulticastSocket.setLoopbackMode >>>>> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken >>>>> on Linux >>>>> >>>>> And 1 test that had error trying to be run: >>>>> >>>>> JT Harness : Tests that had errors >>>>> java/net/URLPermission/nstest/lookup.sh: >>>>> >>>>> Because of: >>>>> >>>>> test result: Error. Can't find source file: jdk/testlibrary/*.java in >>>>> directory-list: >>>>> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >>>>> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >>>>> >>>>> All other 258 java/net tests pass. >>>>> >>>>> >>>>> >>>>> So what do you think? >>>>> >>>>> >>>>> Regards, Peter >>>>> >>>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.x.mcmahon at oracle.com Wed Jul 2 16:15:43 2014 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Wed, 02 Jul 2014 17:15:43 +0100 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53B2FF8D.5040503@gmail.com> References: <53B2FF8D.5040503@gmail.com> Message-ID: <53B4302F.5010603@oracle.com> Hi Peter, Thanks for contributing this. I've started to review it and will get back to you with comments later in the week. Regards, Michael. On 01/07/14 19:35, Peter Levart wrote: > Hi, > > I propose a patch for this issue: > > https://bugs.openjdk.java.net/browse/JDK-7186258 > > The motivation to re-design caching of InetAddress-es was not this > issue though, but a desire to attack synchronization bottlenecks in > methods like URL.equals and URL.hashCode which use host name to IP > address mapping. I plan to tackle the synchronization in URL in a > follow-up proposal, but I wanted to 1st iron-out the "leaves" of the > call-tree. Here's the proposed patch: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ > > sun.net.InetAddressCachePolicy: > > - two static methods (get() and getNegative()) were synchronized. > Removed synchronization and made underlying fields volatile. > - also added a normalization of negative policy in > setNegativeIfNotSet(). The logic in InetAddress doesn't cope with > negative values distinct from InetAddressCachePolicy.FOREVER (-1), so > this was a straight bug. The setIfNotSet() doesn't need this > normalization, because checkValue() throws exception if passed-in > value < InetAddressCachePolicy.FOREVER. > > java.net.InetAddress: > > - complete redesign of caching. Instead of distinct Positive/Negative > caches, there's only one cache - a ConcurrentHashMap. The value in the > map knows if it contains positive or negative answer. > - the design of this cache is similar but much simpler than > java.lang.reflect.WeakCache, since it doesn't have to deal with > WeakReferences and keys are simpler (just strings - hostnames). > Similarity is in how concurrent requests for the same key (hostname) > are synchronized when the entry is not cached yet, but still avoid > synchronization when entry is cached. This preserves the behaviour of > original InetAddress caching code but simplifies it greatly (100+ > lines removed). > - I tried to preserve the interaction between > InetAddress.getLocalHost() and InetAddress.getByName(). The > getLocalHost() caches the local host address for 5 seconds privately. > When it expires it performs new name service look-up and "refreshes" > the entry in the InetAddress.getByName() cache although it has not > expired yet. I think this is meant to prevent surprises when > getLocalHost() returns newer address than getByName() which is called > after that. > - I also fixed the JDK-7186258 as a by-product (but don't know yet how > to write a test for this issue - any ideas?) > > I created a JMH benchmark that tests the following methods: > > - InetAddress.getLocalHost() > - InetAddress.getByName() (with positive and negative answer) > > Here're the results of running on my 4-core (8-threads) i7/Linux: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf > > > The getByNameNegative() test does not show much improvement in patched > vs. original code. That's because by default the policy is to NOT > cache negative answers. Requests for same hostname to the > NameService(s) are synchronized. If > "networkaddress.cache.negative.ttl" system property is set to some > positive value, results are similar to those of getByNamePositive() > test (the default policy for positive caching is 30 seconds). > > I ran the jtreg tests in test/java/net and have the same score as with > original unpatched code. I have 3 failing tests from original and > patched runs: > > JT Harness : Tests that failed > java/net/MulticastSocket/Promiscuous.java: Test for interference when > two sockets are bound to the same port but joined to different > multicast groups > java/net/MulticastSocket/SetLoopbackMode.java: Test > MulticastSocket.setLoopbackMode > java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken > on Linux > > And 1 test that had error trying to be run: > > JT Harness : Tests that had errors > java/net/URLPermission/nstest/lookup.sh: > > Because of: > > test result: Error. Can't find source file: jdk/testlibrary/*.java in > directory-list: > /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest > /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary > > All other 258 java/net tests pass. > > > > So what do you think? > > > Regards, Peter > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernd-2014 at eckenfels.net Wed Jul 2 21:13:19 2014 From: bernd-2014 at eckenfels.net (Bernd Eckenfels) Date: Wed, 2 Jul 2014 23:13:19 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53B3F377.3090208@gmail.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> Message-ID: <20140702231319.000053e8.bernd-2014@eckenfels.net> Hello Peter, I think the comments in compareTo() are now superflucious ("with 0"). Greetings Bernd Am Wed, 02 Jul 2014 13:56:39 +0200 schrieb Peter Levart : > Hi, > > I updated the webrev with first two suggestions from Bernd > (expireTime instead of createTime and cacheNanos + only use > putIfAbsent instead of get followed by putIfAbsent): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.02/ From peter.levart at gmail.com Thu Jul 3 08:43:59 2014 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 03 Jul 2014 10:43:59 +0200 Subject: RFR 8049220: URL.factory data race Message-ID: <53B517CF.6060006@gmail.com> Hi, I noticed a data race in java.net.URL: https://bugs.openjdk.java.net/browse/JDK-8049220 I propose the following simple patch: http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.factory/webrev.01/ Regards, Peter From peter.levart at gmail.com Thu Jul 3 10:12:54 2014 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 03 Jul 2014 12:12:54 +0200 Subject: RFR JDK-8049228: Improve multithreaded scalability of InetAddress cache (was: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more)) In-Reply-To: References: <53B2FF8D.5040503@gmail.com> Message-ID: <53B52CA6.7080100@gmail.com> On 07/01/2014 10:04 PM, Martin Buchholz wrote: > I haven't looked at the patch, but generally ... all uses of > currentTimeMillis to measure elapsed time should be migrated to use > difference of two nanoTime values, and such a change should be done > independently of other changes. I have filed a RFE that is more suitable for this change: https://bugs.openjdk.java.net/browse/JDK-8049228 So, is it strictly necessary to fix JDK-7186258 alone before applying the change for JDK-8049228 although it would supersede it? Regards, Peter > > IIRC lookup.sh is a known flaky test being fixed elsewhere. > > > On Tue, Jul 1, 2014 at 11:35 AM, Peter Levart > wrote: > >> Hi, >> >> I propose a patch for this issue: >> >> https://bugs.openjdk.java.net/browse/JDK-7186258 >> >> The motivation to re-design caching of InetAddress-es was not this issue >> though, but a desire to attack synchronization bottlenecks in methods like >> URL.equals and URL.hashCode which use host name to IP address mapping. I >> plan to tackle the synchronization in URL in a follow-up proposal, but I >> wanted to 1st iron-out the "leaves" of the call-tree. Here's the proposed >> patch: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >> >> sun.net.InetAddressCachePolicy: >> >> - two static methods (get() and getNegative()) were synchronized. Removed >> synchronization and made underlying fields volatile. >> - also added a normalization of negative policy in setNegativeIfNotSet(). >> The logic in InetAddress doesn't cope with negative values distinct from >> InetAddressCachePolicy.FOREVER (-1), so this was a straight bug. The >> setIfNotSet() doesn't need this normalization, because checkValue() throws >> exception if passed-in value < InetAddressCachePolicy.FOREVER. >> >> java.net.InetAddress: >> >> - complete redesign of caching. Instead of distinct Positive/Negative >> caches, there's only one cache - a ConcurrentHashMap. The value in the map >> knows if it contains positive or negative answer. >> - the design of this cache is similar but much simpler than >> java.lang.reflect.WeakCache, since it doesn't have to deal with >> WeakReferences and keys are simpler (just strings - hostnames). Similarity >> is in how concurrent requests for the same key (hostname) are synchronized >> when the entry is not cached yet, but still avoid synchronization when >> entry is cached. This preserves the behaviour of original InetAddress >> caching code but simplifies it greatly (100+ lines removed). >> - I tried to preserve the interaction between InetAddress.getLocalHost() >> and InetAddress.getByName(). The getLocalHost() caches the local host >> address for 5 seconds privately. When it expires it performs new name >> service look-up and "refreshes" the entry in the InetAddress.getByName() >> cache although it has not expired yet. I think this is meant to prevent >> surprises when getLocalHost() returns newer address than getByName() which >> is called after that. >> - I also fixed the JDK-7186258 as a by-product (but don't know yet how to >> write a test for this issue - any ideas?) >> >> I created a JMH benchmark that tests the following methods: >> >> - InetAddress.getLocalHost() >> - InetAddress.getByName() (with positive and negative answer) >> >> Here're the results of running on my 4-core (8-threads) i7/Linux: >> >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >> >> The getByNameNegative() test does not show much improvement in patched vs. >> original code. That's because by default the policy is to NOT cache >> negative answers. Requests for same hostname to the NameService(s) are >> synchronized. If "networkaddress.cache.negative.ttl" system property is set >> to some positive value, results are similar to those of getByNamePositive() >> test (the default policy for positive caching is 30 seconds). >> >> I ran the jtreg tests in test/java/net and have the same score as with >> original unpatched code. I have 3 failing tests from original and patched >> runs: >> >> JT Harness : Tests that failed >> java/net/MulticastSocket/Promiscuous.java: Test for interference when two >> sockets are bound to the same port but joined to different multicast groups >> java/net/MulticastSocket/SetLoopbackMode.java: Test >> MulticastSocket.setLoopbackMode >> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken on >> Linux >> >> And 1 test that had error trying to be run: >> >> JT Harness : Tests that had errors >> java/net/URLPermission/nstest/lookup.sh: >> >> Because of: >> >> test result: Error. Can't find source file: jdk/testlibrary/*.java in >> directory-list: >> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >> >> All other 258 java/net tests pass. >> >> >> >> So what do you think? >> >> >> Regards, Peter >> >> From peter.levart at gmail.com Thu Jul 3 16:01:42 2014 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 03 Jul 2014 18:01:42 +0200 Subject: Gluing together URL.equals Message-ID: <53B57E66.3000908@gmail.com> Hi, We know that URL.equals and hashCode are fundamentally broken. But URL.equals is even more broken than hashCode. Nevertheless, URL.equals is used explicitly in the following places in JDK: java.security.CodeSource.matchLocation java.security.CodeSource.equals java.util.jar.JarVerifier.VerifierCodeSource.equals javax.sql.rowset.serial.SerialDatalink.equals java.lang.Package.isSealed javax.swing.JEditorPane.setPage javax.swing.text.html.FrameView.changedUpdate sun.applet.AppletViewer.getApplet sun.applet.AppletViewer.getApplets And I'm not counting places where it might be used because URLs are Objects (as keys in HashMaps, etc...) I'd like to discuss one of URL.equals pitfalls that might be able to get fixed and whether it is desirable to fix it. javadoc: "The equals method implements an equivalence relation on non-null object references: ... It is consistent: for any non-null reference values x and y, multiple invocations of x.equals(y) consistently return true or consistently return false, provided no information used in equals comparisons on the objects is modified." URL url1 = new URL("http://alias1/"); URL url2 = new URL("http://alias2/"); boolean answer1 = url1.equals(url2); ... boolean answer2 = url1.equals(url2); Can it happen that answer1 != answer2 ? Yes! Suppose that alias1 and alias2 are host names that resolve to the same IP address. Normally, answer1 and answer2 would be "true". But only if the name service that resolves the host names is up and running. If it's not, then the answer is "false". Suppose that while obtaining answer1 the DNS was restarting and while obtaining answer2 it was up and running... Then answer1 would be "false" while answer2 would be "true". The following URLStreamHandler method that is called for both URLs from equals method is responsible for such unstable behaviour: protected synchronized InetAddress getHostAddress(URL u) { if (u.hostAddress != null) return u.hostAddress; String host = u.getHost(); if (host == null || host.equals("")) { return null; } else { try { u.hostAddress = InetAddress.getByName(host); } catch (UnknownHostException ex) { return null; } catch (SecurityException se) { return null; } } return u.hostAddress; } As can be seen, the hostAddress is obtained by InetAddress.getByName() and then cached on the URL.hostAddress field. Leaving aside the fact that although this method is synchronized, caching of hostAddress is not synchronized properly (more on that later), the problem is that negative answer (UnknownHostException or SecurityException) is not cached. UnknownHostException is not cached by InetAddress.getByName() by default and SecurityException is dependent on the caller SecurityContext. Simple fix for this issue would be to cache negative answer on the URL field too. This would make URL.equals "consistent". So what's wrong with synchronization besides being a bottleneck? The problem is that getHostAddress() method is using the URLStreamHandler instance as a lock. Two URLs that are compared in the URL.equals method are passed to the URLStreamHandler.equals(URL u1, URL u2) method of the 1st URL's handler. This handler instance need not be the same as the 2nd URL's handler even though both URLs have same protocol. For example: URL url1 = new URL("http://alias1/"); URL.setURLStreamHandlerFactory(...a custom factory...); URL url2 = new URL("http://alias2/"); The "handler" instances of above two URLs are different, since the handler of 1st URL was created with default URLStreamHandlerFactory and the handler of 2nd URL was created with a custom URLStreamHandlerFactory. Now suppose one thread does: url1.equals(url2); and some other thread does: url2.equals(url1); This translates to, among other things, calling the following URLStreamHandler instance method: protected boolean hostsEqual(URL u1, URL u2) { InetAddress a1 = getHostAddress(u1); InetAddress a2 = getHostAddress(u2); // if we have internet address for both, compare them if (a1 != null && a2 != null) { return a1.equals(a2); // else, if both have host names, compare them } else if (u1.getHost() != null && u2.getHost() != null) return u1.getHost().equalsIgnoreCase(u2.getHost()); else return u1.getHost() == null && u2.getHost() == null; } So the two threads are reading and modifying URL.hostAddress field of both URLs, but each of them is holding a separate lock. You may say that creating URL instances, then changing the URLStreamHandlerFactory and creating some more URL instances and than comparing them among themselves is not happening a lot, but this could be fixed. Why not using the URL instance as a lock when reading/writing it's field? Would this be desirable? It would mean a lot less contention (and even less if caching of URL.hostAddress was implemented in a lock-free way). Because I know that URL.equals compatibility is important, I'm asking here if a fix for this issue is desirable at all. What about synchronization fix only (and keeping the "unstable" equals() behaviour)? Regards, Peter From Alan.Bateman at oracle.com Thu Jul 3 16:33:47 2014 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 03 Jul 2014 17:33:47 +0100 Subject: RFR 8049220: URL.factory data race In-Reply-To: <53B517CF.6060006@gmail.com> References: <53B517CF.6060006@gmail.com> Message-ID: <53B585EB.9060807@oracle.com> On 03/07/2014 09:43, Peter Levart wrote: > Hi, > > I noticed a data race in java.net.URL: > > https://bugs.openjdk.java.net/browse/JDK-8049220 > > I propose the following simple patch: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.factory/webrev.01/ > A good catch and the change looks good to me. I assume it just wasn't noticed because it can only be set once and probably rared used too. -Alan From martinrb at google.com Thu Jul 3 17:37:41 2014 From: martinrb at google.com (Martin Buchholz) Date: Thu, 3 Jul 2014 10:37:41 -0700 Subject: RFR JDK-8049228: Improve multithreaded scalability of InetAddress cache (was: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more)) In-Reply-To: <53B52CA6.7080100@gmail.com> References: <53B2FF8D.5040503@gmail.com> <53B52CA6.7080100@gmail.com> Message-ID: On Thu, Jul 3, 2014 at 3:12 AM, Peter Levart wrote: > On 07/01/2014 10:04 PM, Martin Buchholz wrote: > >> I haven't looked at the patch, but generally ... all uses of >> currentTimeMillis to measure elapsed time should be migrated to use >> difference of two nanoTime values, and such a change should be done >> independently of other changes. >> > > I have filed a RFE that is more suitable for this change: > > https://bugs.openjdk.java.net/browse/JDK-8049228 > > So, is it strictly necessary to fix JDK-7186258 alone before applying the > change for JDK-8049228 although it would supersede it? > I'm just an interested observer for this change, not a reviewer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Jul 3 17:57:01 2014 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 03 Jul 2014 18:57:01 +0100 Subject: RFR JDK-8049228: Improve multithreaded scalability of InetAddress cache In-Reply-To: <53B52CA6.7080100@gmail.com> References: <53B2FF8D.5040503@gmail.com> <53B52CA6.7080100@gmail.com> Message-ID: <53B5996D.9040200@oracle.com> On 03/07/2014 11:12, Peter Levart wrote: > > I have filed a RFE that is more suitable for this change: > > https://bugs.openjdk.java.net/browse/JDK-8049228 > > So, is it strictly necessary to fix JDK-7186258 alone before applying > the change for JDK-8049228 although it would supersede it? We can close JDK-7186258 as a dup once the JDK-8049228 has been reviewed + pushed. -Alan From martinrb at google.com Thu Jul 3 17:59:51 2014 From: martinrb at google.com (Martin Buchholz) Date: Thu, 3 Jul 2014 10:59:51 -0700 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53B419B7.1030600@oracle.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> <53B419B7.1030600@oracle.com> Message-ID: On Wed, Jul 2, 2014 at 7:39 AM, Claes Redestad wrote: > Hi Peter, > > perhaps the synchronized(this)-block in NameServiceAddresses::get() can > be replaced with a ReentrantLock? Applying this on top of your patch, I > seem to improve scores on your benchmark for -t 4 by ~33% on my machine: > Once upon a time ReentrantLock had some performance advantage over builtin synchronized; but today best practice is to prefer ReentrantLock only for the software engineering advantages like more flexibility or clarity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From claes.redestad at oracle.com Thu Jul 3 18:13:59 2014 From: claes.redestad at oracle.com (Claes Redestad) Date: Thu, 03 Jul 2014 20:13:59 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> <53B419B7.1030600@oracle.com> Message-ID: <53B59D67.5020706@oracle.com> On 07/03/2014 07:59 PM, Martin Buchholz wrote: > > > > On Wed, Jul 2, 2014 at 7:39 AM, Claes Redestad > > wrote: > > Hi Peter, > > perhaps the synchronized(this)-block in > NameServiceAddresses::get() can be replaced with a ReentrantLock? > Applying this on top of your patch, I seem to improve scores on > your benchmark for -t 4 by ~33% on my machine: > > > Once upon a time ReentrantLock had some performance advantage over > builtin synchronized; but today best practice is to prefer > ReentrantLock only for the software engineering advantages like more > flexibility or clarity. This was my feeling as well, but when I saw the remaining synchronized block in Peter's patch I got curious to see if that notion still held up. Sorry for being sloppy with the benchmarking and throwing a red herring into the mix, though. /Claes -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan at azulsystems.com Fri Jul 4 11:46:37 2014 From: ivan at azulsystems.com (Ivan Krylov) Date: Fri, 4 Jul 2014 11:46:37 +0000 Subject: Do not JDK-6967684 backport to JDK6? In-Reply-To: <53A798ED.3040905@lab.ntt.co.jp> References: <53A798ED.3040905@lab.ntt.co.jp> Message-ID: <4E485579-2E8C-44D5-A7C2-82C8223C111F@azulsystems.com> Don?t have an answer this but CC-ing openjdk6-dev Ivan On 23 Jun 2014, at 07:03, KUBOTA Yuji wrote: > Hi all, > > I got ArrayIndexOutOfBoundsException at java.text.SimpleDateFormat.subFormat > via sun.net.httpserver.ExchangeImpl when I used com.sun.net.httpserver > with OpenJDK6 as the tail of this mail. > > The reason why this exception occured is that httpserver (ExchangeImpl) uses > a non thread-safe SimpleDateFormat which has been modified at OpenJDK7 > by JDK-6967684. > ---- > Bug database: http://bugs.java.com/view_bug.do?bug_id=6967684 > Patch: http://hg.openjdk.java.net/jdk7/jdk7/jdk/rev/1371a2d5f3a8 > ML: http://mail.openjdk.java.net/pipermail/net-dev/2010-July/001946.html > ---- > > And the HEAD of jdk6 repos do not include this patch. > If anyone knows why this patch do not backport to OpenJDK6, please tell me. > > --Exception details--- > # java -version > java version "1.6.0_28" > OpenJDK Runtime Environment (IcedTea6 1.13.0pre) > (rhel-1.66.1.13.0.el6-x86_64) OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode) > > # stack trace > java.lang.ArrayIndexOutOfBoundsException: -2147483648 > java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1066) > java.text.SimpleDateFormat.format(SimpleDateFormat.java:899) > java.text.SimpleDateFormat.format(SimpleDateFormat.java:869) > java.text.DateFormat.format(DateFormat.java:333) > sun.net.httpserver.ExchangeImpl.sendResponseHeaders(ExchangeImpl.java:206) > sun.net.httpserver.HttpExchangeImpl.sendResponseHeaders(HttpExchangeImpl.java:86) > com.sun.xml.internal.ws.transport.http.server.ServerConnectionImpl.getOutput(ServerConnectionImpl.java:145) > com.sun.xml.internal.ws.transport.http.HttpAdapter.encodePacket(HttpAdapter.java:321) > com.sun.xml.internal.ws.transport.http.HttpAdapter.access$100(HttpAdapter.java:82) > com.sun.xml.internal.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:470) > com.sun.xml.internal.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:233) > com.sun.xml.internal.ws.transport.http.server.WSHttpHandler.handleExchange(WSHttpHandler.java:95) > com.sun.xml.internal.ws.transport.http.server.WSHttpHandler$HttpHandlerRunnable.run(WSHttpHandler.java:117) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:679) > ---------- > > Thanks in advance for your help! > > Best regards, > KUBOTA Yuji. > From paul.sandoz at oracle.com Fri Jul 4 14:54:30 2014 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 4 Jul 2014 16:54:30 +0200 Subject: RFR 8049220: URL.factory data race In-Reply-To: <53B585EB.9060807@oracle.com> References: <53B517CF.6060006@gmail.com> <53B585EB.9060807@oracle.com> Message-ID: On Jul 3, 2014, at 6:33 PM, Alan Bateman wrote: > On 03/07/2014 09:43, Peter Levart wrote: >> Hi, >> >> I noticed a data race in java.net.URL: >> >> https://bugs.openjdk.java.net/browse/JDK-8049220 >> >> I propose the following simple patch: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.factory/webrev.01/ >> > A good catch and the change looks good to me. Yes, well spotted. May i suggest that the following comment is updated: 1109 factory = fac; // volatile write to say something about ensuring safe publication of a constructed handle? since it is often quite tricky to reason about why a volatile write is needed (to stamp in, at least, a StoreStore barrier). For JMM v9 we may not need to mark such a ref as volatile. > I assume it just wasn't noticed because it can only be set once and probably rared used too. > Yeah, i wonder whether it would ever get optimized/inlined to the point at which re-ordering could practically happen. Paul. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From michael.x.mcmahon at oracle.com Fri Jul 4 18:33:26 2014 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Fri, 04 Jul 2014 19:33:26 +0100 Subject: RFR: 8048212 Two tests failed with "java.net.SocketException: Bad protocol option" on Windows after 8029607 In-Reply-To: <53B1AFD5.4020200@oracle.com> References: <53B19615.8080503@oracle.com> <53B1AFD5.4020200@oracle.com> Message-ID: <53B6F376.8040607@oracle.com> On 30/06/14 19:43, Alan Bateman wrote: > On 30/06/2014 17:53, Michael McMahon wrote: >> Could I get the following change reviewed please? >> >> http://cr.openjdk.java.net/~michaelm/8048212/webrev.1/ >> >> It fixes a problem caused by the fix for 8029607. The changes for >> that fix should not have been applied to Windows. So, this fix ensures >> that the IPv4 option is used always on Windows. >> > I don't know if Windows supports setting IP_TOS on an IPv6 socket but > in any case, it might be simplest to just go back to the original > behavior which was for this case to be a no-op. The spec allows for that. > > -Alan Sorry for the delay in getting back to this. The following is a simpler change that restores the original NIO behavior on Windows http://cr.openjdk.java.net/~michaelm/8048212/webrev.2/ Michael From Alan.Bateman at oracle.com Fri Jul 4 19:47:12 2014 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 04 Jul 2014 20:47:12 +0100 Subject: RFR: 8048212 Two tests failed with "java.net.SocketException: Bad protocol option" on Windows after 8029607 In-Reply-To: <53B6F376.8040607@oracle.com> References: <53B19615.8080503@oracle.com> <53B1AFD5.4020200@oracle.com> <53B6F376.8040607@oracle.com> Message-ID: <53B704C0.1000007@oracle.com> On 04/07/2014 19:33, Michael McMahon wrote: > > Sorry for the delay in getting back to this. > > The following is a simpler change that restores the original NIO > behavior on Windows > > http://cr.openjdk.java.net/~michaelm/8048212/webrev.2/ Yes, this much simpler. A minor nit is that L300 in Net.c should probably also check that the level is IPPROTO_IPV6. The comment could be updated too. -Alan From xuelei.fan at oracle.com Mon Jul 7 02:39:38 2014 From: xuelei.fan at oracle.com (Xuelei Fan) Date: Mon, 07 Jul 2014 10:39:38 +0800 Subject: RFR 8014870: Faster KDC availability check in Kerberos In-Reply-To: <17D2CB8C-001B-4929-8BD8-956ADF1E97DE@oracle.com> References: <17D2CB8C-001B-4929-8BD8-956ADF1E97DE@oracle.com> Message-ID: <53BA086A.5040004@oracle.com> I have not read the fix. I was just wondering that this fix save the wait time, but increase the networking traffics, and increase the workload of KDC servers. I think the KDC timeout should be corner cases for TCP, and it is tolerable for UDP connections. I'm not confident that this is a cost-effective update if we considering the overall system of Kerberos. Xuelei On 6/24/2014 4:17 PM, Wang Weijun wrote: > Hi All > > Please review the code change at > > http://cr.openjdk.java.net/~weijun/8014870/webrev.00/ > > In Kerberos, when trying to request for a ticket, we tried multiple KDC servers for multiple times. Before this fix, we connect to a server, wait for 30 seconds (the default kdc_timeout). If there is no answer, go to the next KDC, wait for another 30 seconds, and so on. If none of the KDCs replies. We do more rounds (max_retries, default 3) of connections, and fail at last. Altogether with 3 KDCs we will wait at most 3*30*3=270 seconds. > > After this fix, connections are non-blocking and made every second, so they can wait at the same time. The kdc_timeout default is also reduced to 10 seconds (the same as other vendors). At the worst case, we will wait 3*3+10=19 seconds. > > You might say that changing kdc_timeout to 10 seconds matters a lot here but actually the wait-together style is much more helpful. Suppose only the 3rd KDC is alive, the old code needs to wait for 60 seconds to be able to connect to it, while the new one only needs to wait for 2 seconds, and this has nothing to do with kdc_timeout. > > Because of this, I've thrown away the old krb5.kdc.bad.policy security property. It's now not worth remembering which KDC is alive and which is not. > > All changes are inside the KdcComm.java file. Others are test and removal of useless things. > > I've included net-dev@ because these are all NIO calls, which I was no familiar with. > > Thanks > Max > From michael.x.mcmahon at oracle.com Mon Jul 7 10:59:47 2014 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Mon, 07 Jul 2014 11:59:47 +0100 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53B3F377.3090208@gmail.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> Message-ID: <53BA7DA3.70205@oracle.com> Hi Peter, Is it necessary to remove the cache entry in the local host case (L1226) ? It seems redundant to cache it here, and also explicitly in the CachedLocalHost object Michael On 02/07/14 12:56, Peter Levart wrote: > Hi, > > I updated the webrev with first two suggestions from Bernd (expireTime > instead of createTime and cacheNanos + only use putIfAbsent instead of > get followed by putIfAbsent): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.02/ > > Thanks, Bernd. > > The id field in CachedAddresses is necessary for compareTo to never > return 0 for two different instances (used as element in > ConcurrentSkipListSet). > > For last two suggestions I'm not sure whether they are desired, so I'm > currently leaving them as is. > > > Regards, Peter > > On 07/01/2014 10:06 PM, Bernd Eckenfels wrote: >> Looks good, like it, Peter. >> >> some nits: instead of adding createTime and cacheNanos, only have a >> expireAfter? >> >> L782: is it better to use putIfAbsent unconditionally, instead of >> get/putIfAbsent in NameServicdeAddr? >> >> L732: I am unsure about the id field, isnt it enough to have the >> identity equality check for the replacement check and otherwise depend >> on equals()? >> >> L1223: What about moving the cache exiry inside the if (useCache) >> >> BTW1: might be the wrong RFR, but considering your good performance >> numbers for an active cache, would having 100ms or similiar default >> negative cache time make sense without impacting (visible) the semantic. >> >> >> >> Gruss >> Bernd >> >> >> Am Tue, 01 Jul 2014 20:35:57 +0200 >> schrieb Peter Levart : >> >>> Hi, >>> >>> I propose a patch for this issue: >>> >>> https://bugs.openjdk.java.net/browse/JDK-7186258 >>> >>> The motivation to re-design caching of InetAddress-es was not this >>> issue though, but a desire to attack synchronization bottlenecks in >>> methods like URL.equals and URL.hashCode which use host name to IP >>> address mapping. I plan to tackle the synchronization in URL in a >>> follow-up proposal, but I wanted to 1st iron-out the "leaves" of the >>> call-tree. Here's the proposed patch: >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >>> >>> >>> sun.net.InetAddressCachePolicy: >>> >>> - two static methods (get() and getNegative()) were synchronized. >>> Removed synchronization and made underlying fields volatile. >>> - also added a normalization of negative policy in >>> setNegativeIfNotSet(). The logic in InetAddress doesn't cope with >>> negative values distinct from InetAddressCachePolicy.FOREVER (-1), so >>> this was a straight bug. The setIfNotSet() doesn't need this >>> normalization, because checkValue() throws exception if passed-in >>> value < InetAddressCachePolicy.FOREVER. >>> >>> java.net.InetAddress: >>> >>> - complete redesign of caching. Instead of distinct Positive/Negative >>> caches, there's only one cache - a ConcurrentHashMap. The value in >>> the map knows if it contains positive or negative answer. >>> - the design of this cache is similar but much simpler than >>> java.lang.reflect.WeakCache, since it doesn't have to deal with >>> WeakReferences and keys are simpler (just strings - hostnames). >>> Similarity is in how concurrent requests for the same key (hostname) >>> are synchronized when the entry is not cached yet, but still avoid >>> synchronization when entry is cached. This preserves the behaviour of >>> original InetAddress caching code but simplifies it greatly (100+ >>> lines removed). >>> - I tried to preserve the interaction between >>> InetAddress.getLocalHost() and InetAddress.getByName(). The >>> getLocalHost() caches the local host address for 5 seconds privately. >>> When it expires it performs new name service look-up and "refreshes" >>> the entry in the InetAddress.getByName() cache although it has not >>> expired yet. I think this is meant to prevent surprises when >>> getLocalHost() returns newer address than getByName() which is called >>> after that. >>> - I also fixed the JDK-7186258 as a by-product (but don't know yet >>> how to write a test for this issue - any ideas?) >>> >>> I created a JMH benchmark that tests the following methods: >>> >>> - InetAddress.getLocalHost() >>> - InetAddress.getByName() (with positive and negative answer) >>> >>> Here're the results of running on my 4-core (8-threads) i7/Linux: >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >>> >>> >>> >>> The getByNameNegative() test does not show much improvement in >>> patched vs. original code. That's because by default the policy is to >>> NOT cache negative answers. Requests for same hostname to the >>> NameService(s) are synchronized. If >>> "networkaddress.cache.negative.ttl" system property is set to some >>> positive value, results are similar to those of getByNamePositive() >>> test (the default policy for positive caching is 30 seconds). >>> >>> I ran the jtreg tests in test/java/net and have the same score as >>> with original unpatched code. I have 3 failing tests from original >>> and patched runs: >>> >>> JT Harness : Tests that failed >>> java/net/MulticastSocket/Promiscuous.java: Test for interference when >>> two sockets are bound to the same port but joined to different >>> multicast groups >>> java/net/MulticastSocket/SetLoopbackMode.java: Test >>> MulticastSocket.setLoopbackMode >>> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken >>> on Linux >>> >>> And 1 test that had error trying to be run: >>> >>> JT Harness : Tests that had errors >>> java/net/URLPermission/nstest/lookup.sh: >>> >>> Because of: >>> >>> test result: Error. Can't find source file: jdk/testlibrary/*.java in >>> directory-list: >>> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >>> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >>> >>> All other 258 java/net tests pass. >>> >>> >>> >>> So what do you think? >>> >>> >>> Regards, Peter >>> >>> > From peter.levart at gmail.com Mon Jul 7 11:07:27 2014 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 07 Jul 2014 13:07:27 +0200 Subject: RFR 8049220: URL.factory data race In-Reply-To: References: <53B517CF.6060006@gmail.com> <53B585EB.9060807@oracle.com> Message-ID: <53BA7F6F.9040506@gmail.com> Hi Pavel, Alan and Paul, Thanks for reviewing. I accepted the suggestions from Pavel and Paul and created webrev.02: http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.factory/webrev.02/ Is this good to go into jdk9-dev? Regards, Peter On 07/04/2014 04:54 PM, Paul Sandoz wrote: > On Jul 3, 2014, at 6:33 PM, Alan Bateman wrote: >> On 03/07/2014 09:43, Peter Levart wrote: >>> Hi, >>> >>> I noticed a data race in java.net.URL: >>> >>> https://bugs.openjdk.java.net/browse/JDK-8049220 >>> >>> I propose the following simple patch: >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.factory/webrev.01/ >>> >> A good catch and the change looks good to me. > Yes, well spotted. May i suggest that the following comment is updated: > > 1109 factory = fac; // volatile write > > to say something about ensuring safe publication of a constructed handle? since it is often quite tricky to reason about why a volatile write is needed (to stamp in, at least, a StoreStore barrier). > > For JMM v9 we may not need to mark such a ref as volatile. > > >> I assume it just wasn't noticed because it can only be set once and probably rared used too. >> > Yeah, i wonder whether it would ever get optimized/inlined to the point at which re-ordering could practically happen. > > Paul. > From peter.levart at gmail.com Mon Jul 7 13:10:24 2014 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 07 Jul 2014 15:10:24 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53BA7DA3.70205@oracle.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> <53BA7DA3.70205@oracle.com> Message-ID: <53BA9C40.3080402@gmail.com> On 07/07/2014 12:59 PM, Michael McMahon wrote: > Hi Peter, > > Is it necessary to remove the cache entry in the local host case > (L1226) ? > It seems redundant to cache it here, and also explicitly in the > CachedLocalHost object > > Michael Hi Michael, Thanks for looking into this. getLocalHost() seems to have a special hard-coded policy of positive caching for 5 seconds that is independent of general getByName() caching policy (30 seconds by default). The behaviour of original code that I'm trying to replicate is such that when getLocalHost() notices a change of local host name -> address mapping, the mapping in global cache for this change is also updated. I think this is to avoid situations like: Let's say the local host name is "cube": InetAddress addr1 = InetAddress.getLocalHost(); InetAddress addr2 = InetAddress.getByName("cube"); // addr1.equals(addr2) here // 5 seconds later, cube -> IP mapping is updated in DNS or /etc/hosts ... addr1 = InetAddress.getLocalHost(); addr2 = InetAddress.getByName("cube"); // if getLocalHost() did not update global cache, // addr1 (new IP address) would be different from addr2 (old IP address) Another way to accomplish similar guarantee would be to special-case the caching policy in global cache which would check whether the entry is for local host name and set 'expiryTime' accordingly. This would be a little different behaviourally, because InetAddress.getByName() would get a 5 second expiry for local host name too, regardless of whether InetAddress.getLocalHost() has been called at all. But we could get rid of special CachedLocalHost class then. Is such behavioural change warranted? Regards, Peter > > On 02/07/14 12:56, Peter Levart wrote: >> Hi, >> >> I updated the webrev with first two suggestions from Bernd >> (expireTime instead of createTime and cacheNanos + only use >> putIfAbsent instead of get followed by putIfAbsent): >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.02/ >> >> >> Thanks, Bernd. >> >> The id field in CachedAddresses is necessary for compareTo to never >> return 0 for two different instances (used as element in >> ConcurrentSkipListSet). >> >> For last two suggestions I'm not sure whether they are desired, so >> I'm currently leaving them as is. >> >> >> Regards, Peter >> >> On 07/01/2014 10:06 PM, Bernd Eckenfels wrote: >>> Looks good, like it, Peter. >>> >>> some nits: instead of adding createTime and cacheNanos, only have a >>> expireAfter? >>> >>> L782: is it better to use putIfAbsent unconditionally, instead of >>> get/putIfAbsent in NameServicdeAddr? >>> >>> L732: I am unsure about the id field, isnt it enough to have the >>> identity equality check for the replacement check and otherwise depend >>> on equals()? >>> >>> L1223: What about moving the cache exiry inside the if (useCache) >>> >>> BTW1: might be the wrong RFR, but considering your good performance >>> numbers for an active cache, would having 100ms or similiar default >>> negative cache time make sense without impacting (visible) the >>> semantic. >>> >>> >>> >>> Gruss >>> Bernd >>> >>> >>> Am Tue, 01 Jul 2014 20:35:57 +0200 >>> schrieb Peter Levart : >>> >>>> Hi, >>>> >>>> I propose a patch for this issue: >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-7186258 >>>> >>>> The motivation to re-design caching of InetAddress-es was not this >>>> issue though, but a desire to attack synchronization bottlenecks in >>>> methods like URL.equals and URL.hashCode which use host name to IP >>>> address mapping. I plan to tackle the synchronization in URL in a >>>> follow-up proposal, but I wanted to 1st iron-out the "leaves" of the >>>> call-tree. Here's the proposed patch: >>>> >>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >>>> >>>> >>>> sun.net.InetAddressCachePolicy: >>>> >>>> - two static methods (get() and getNegative()) were synchronized. >>>> Removed synchronization and made underlying fields volatile. >>>> - also added a normalization of negative policy in >>>> setNegativeIfNotSet(). The logic in InetAddress doesn't cope with >>>> negative values distinct from InetAddressCachePolicy.FOREVER (-1), so >>>> this was a straight bug. The setIfNotSet() doesn't need this >>>> normalization, because checkValue() throws exception if passed-in >>>> value < InetAddressCachePolicy.FOREVER. >>>> >>>> java.net.InetAddress: >>>> >>>> - complete redesign of caching. Instead of distinct Positive/Negative >>>> caches, there's only one cache - a ConcurrentHashMap. The value in >>>> the map knows if it contains positive or negative answer. >>>> - the design of this cache is similar but much simpler than >>>> java.lang.reflect.WeakCache, since it doesn't have to deal with >>>> WeakReferences and keys are simpler (just strings - hostnames). >>>> Similarity is in how concurrent requests for the same key (hostname) >>>> are synchronized when the entry is not cached yet, but still avoid >>>> synchronization when entry is cached. This preserves the behaviour of >>>> original InetAddress caching code but simplifies it greatly (100+ >>>> lines removed). >>>> - I tried to preserve the interaction between >>>> InetAddress.getLocalHost() and InetAddress.getByName(). The >>>> getLocalHost() caches the local host address for 5 seconds privately. >>>> When it expires it performs new name service look-up and "refreshes" >>>> the entry in the InetAddress.getByName() cache although it has not >>>> expired yet. I think this is meant to prevent surprises when >>>> getLocalHost() returns newer address than getByName() which is called >>>> after that. >>>> - I also fixed the JDK-7186258 as a by-product (but don't know yet >>>> how to write a test for this issue - any ideas?) >>>> >>>> I created a JMH benchmark that tests the following methods: >>>> >>>> - InetAddress.getLocalHost() >>>> - InetAddress.getByName() (with positive and negative answer) >>>> >>>> Here're the results of running on my 4-core (8-threads) i7/Linux: >>>> >>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >>>> >>>> >>>> >>>> The getByNameNegative() test does not show much improvement in >>>> patched vs. original code. That's because by default the policy is to >>>> NOT cache negative answers. Requests for same hostname to the >>>> NameService(s) are synchronized. If >>>> "networkaddress.cache.negative.ttl" system property is set to some >>>> positive value, results are similar to those of getByNamePositive() >>>> test (the default policy for positive caching is 30 seconds). >>>> >>>> I ran the jtreg tests in test/java/net and have the same score as >>>> with original unpatched code. I have 3 failing tests from original >>>> and patched runs: >>>> >>>> JT Harness : Tests that failed >>>> java/net/MulticastSocket/Promiscuous.java: Test for interference when >>>> two sockets are bound to the same port but joined to different >>>> multicast groups >>>> java/net/MulticastSocket/SetLoopbackMode.java: Test >>>> MulticastSocket.setLoopbackMode >>>> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken >>>> on Linux >>>> >>>> And 1 test that had error trying to be run: >>>> >>>> JT Harness : Tests that had errors >>>> java/net/URLPermission/nstest/lookup.sh: >>>> >>>> Because of: >>>> >>>> test result: Error. Can't find source file: jdk/testlibrary/*.java in >>>> directory-list: >>>> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >>>> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >>>> >>>> All other 258 java/net tests pass. >>>> >>>> >>>> >>>> So what do you think? >>>> >>>> >>>> Regards, Peter >>>> >>>> >> > From michael.x.mcmahon at oracle.com Mon Jul 7 14:13:40 2014 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Mon, 07 Jul 2014 15:13:40 +0100 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <53BA9C40.3080402@gmail.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> <53BA7DA3.70205@oracle.com> <53BA9C40.3080402@gmail.com> Message-ID: <53BAAB14.1010907@oracle.com> Peter, Thanks for the explanation. No. I think your change is good. I've run tests here locally and I'm happy with it overall. Michael On 07/07/14 14:10, Peter Levart wrote: > On 07/07/2014 12:59 PM, Michael McMahon wrote: >> Hi Peter, >> >> Is it necessary to remove the cache entry in the local host case >> (L1226) ? >> It seems redundant to cache it here, and also explicitly in the >> CachedLocalHost object >> >> Michael > > Hi Michael, > > Thanks for looking into this. > > getLocalHost() seems to have a special hard-coded policy of positive > caching for 5 seconds that is independent of general getByName() > caching policy (30 seconds by default). The behaviour of original code > that I'm trying to replicate is such that when getLocalHost() notices > a change of local host name -> address mapping, the mapping in global > cache for this change is also updated. I think this is to avoid > situations like: > > Let's say the local host name is "cube": > > InetAddress addr1 = InetAddress.getLocalHost(); > InetAddress addr2 = InetAddress.getByName("cube"); > // addr1.equals(addr2) here > > // 5 seconds later, cube -> IP mapping is updated in DNS or /etc/hosts > ... > > addr1 = InetAddress.getLocalHost(); > addr2 = InetAddress.getByName("cube"); > // if getLocalHost() did not update global cache, > // addr1 (new IP address) would be different from addr2 (old IP address) > > > Another way to accomplish similar guarantee would be to special-case > the caching policy in global cache which would check whether the entry > is for local host name and set 'expiryTime' accordingly. This would be > a little different behaviourally, because InetAddress.getByName() > would get a 5 second expiry for local host name too, regardless of > whether InetAddress.getLocalHost() has been called at all. But we > could get rid of special CachedLocalHost class then. Is such > behavioural change warranted? > > Regards, Peter > >> >> On 02/07/14 12:56, Peter Levart wrote: >>> Hi, >>> >>> I updated the webrev with first two suggestions from Bernd >>> (expireTime instead of createTime and cacheNanos + only use >>> putIfAbsent instead of get followed by putIfAbsent): >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.02/ >>> >>> >>> Thanks, Bernd. >>> >>> The id field in CachedAddresses is necessary for compareTo to never >>> return 0 for two different instances (used as element in >>> ConcurrentSkipListSet). >>> >>> For last two suggestions I'm not sure whether they are desired, so >>> I'm currently leaving them as is. >>> >>> >>> Regards, Peter >>> >>> On 07/01/2014 10:06 PM, Bernd Eckenfels wrote: >>>> Looks good, like it, Peter. >>>> >>>> some nits: instead of adding createTime and cacheNanos, only have a >>>> expireAfter? >>>> >>>> L782: is it better to use putIfAbsent unconditionally, instead of >>>> get/putIfAbsent in NameServicdeAddr? >>>> >>>> L732: I am unsure about the id field, isnt it enough to have the >>>> identity equality check for the replacement check and otherwise depend >>>> on equals()? >>>> >>>> L1223: What about moving the cache exiry inside the if (useCache) >>>> >>>> BTW1: might be the wrong RFR, but considering your good performance >>>> numbers for an active cache, would having 100ms or similiar default >>>> negative cache time make sense without impacting (visible) the >>>> semantic. >>>> >>>> >>>> >>>> Gruss >>>> Bernd >>>> >>>> >>>> Am Tue, 01 Jul 2014 20:35:57 +0200 >>>> schrieb Peter Levart : >>>> >>>>> Hi, >>>>> >>>>> I propose a patch for this issue: >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-7186258 >>>>> >>>>> The motivation to re-design caching of InetAddress-es was not this >>>>> issue though, but a desire to attack synchronization bottlenecks in >>>>> methods like URL.equals and URL.hashCode which use host name to IP >>>>> address mapping. I plan to tackle the synchronization in URL in a >>>>> follow-up proposal, but I wanted to 1st iron-out the "leaves" of the >>>>> call-tree. Here's the proposed patch: >>>>> >>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >>>>> >>>>> >>>>> sun.net.InetAddressCachePolicy: >>>>> >>>>> - two static methods (get() and getNegative()) were synchronized. >>>>> Removed synchronization and made underlying fields volatile. >>>>> - also added a normalization of negative policy in >>>>> setNegativeIfNotSet(). The logic in InetAddress doesn't cope with >>>>> negative values distinct from InetAddressCachePolicy.FOREVER (-1), so >>>>> this was a straight bug. The setIfNotSet() doesn't need this >>>>> normalization, because checkValue() throws exception if passed-in >>>>> value < InetAddressCachePolicy.FOREVER. >>>>> >>>>> java.net.InetAddress: >>>>> >>>>> - complete redesign of caching. Instead of distinct Positive/Negative >>>>> caches, there's only one cache - a ConcurrentHashMap. The value in >>>>> the map knows if it contains positive or negative answer. >>>>> - the design of this cache is similar but much simpler than >>>>> java.lang.reflect.WeakCache, since it doesn't have to deal with >>>>> WeakReferences and keys are simpler (just strings - hostnames). >>>>> Similarity is in how concurrent requests for the same key (hostname) >>>>> are synchronized when the entry is not cached yet, but still avoid >>>>> synchronization when entry is cached. This preserves the behaviour of >>>>> original InetAddress caching code but simplifies it greatly (100+ >>>>> lines removed). >>>>> - I tried to preserve the interaction between >>>>> InetAddress.getLocalHost() and InetAddress.getByName(). The >>>>> getLocalHost() caches the local host address for 5 seconds privately. >>>>> When it expires it performs new name service look-up and "refreshes" >>>>> the entry in the InetAddress.getByName() cache although it has not >>>>> expired yet. I think this is meant to prevent surprises when >>>>> getLocalHost() returns newer address than getByName() which is called >>>>> after that. >>>>> - I also fixed the JDK-7186258 as a by-product (but don't know yet >>>>> how to write a test for this issue - any ideas?) >>>>> >>>>> I created a JMH benchmark that tests the following methods: >>>>> >>>>> - InetAddress.getLocalHost() >>>>> - InetAddress.getByName() (with positive and negative answer) >>>>> >>>>> Here're the results of running on my 4-core (8-threads) i7/Linux: >>>>> >>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >>>>> >>>>> >>>>> >>>>> The getByNameNegative() test does not show much improvement in >>>>> patched vs. original code. That's because by default the policy is to >>>>> NOT cache negative answers. Requests for same hostname to the >>>>> NameService(s) are synchronized. If >>>>> "networkaddress.cache.negative.ttl" system property is set to some >>>>> positive value, results are similar to those of getByNamePositive() >>>>> test (the default policy for positive caching is 30 seconds). >>>>> >>>>> I ran the jtreg tests in test/java/net and have the same score as >>>>> with original unpatched code. I have 3 failing tests from original >>>>> and patched runs: >>>>> >>>>> JT Harness : Tests that failed >>>>> java/net/MulticastSocket/Promiscuous.java: Test for interference when >>>>> two sockets are bound to the same port but joined to different >>>>> multicast groups >>>>> java/net/MulticastSocket/SetLoopbackMode.java: Test >>>>> MulticastSocket.setLoopbackMode >>>>> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken >>>>> on Linux >>>>> >>>>> And 1 test that had error trying to be run: >>>>> >>>>> JT Harness : Tests that had errors >>>>> java/net/URLPermission/nstest/lookup.sh: >>>>> >>>>> Because of: >>>>> >>>>> test result: Error. Can't find source file: jdk/testlibrary/*.java in >>>>> directory-list: >>>>> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >>>>> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >>>>> >>>>> All other 258 java/net tests pass. >>>>> >>>>> >>>>> >>>>> So what do you think? >>>>> >>>>> >>>>> Regards, Peter >>>>> >>>>> >>> >> > From Alan.Bateman at oracle.com Mon Jul 7 14:19:03 2014 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 07 Jul 2014 15:19:03 +0100 Subject: RFR 8049220: URL.factory data race In-Reply-To: <53BA7F6F.9040506@gmail.com> References: <53B517CF.6060006@gmail.com> <53B585EB.9060807@oracle.com> <53BA7F6F.9040506@gmail.com> Message-ID: <53BAAC57.5010904@oracle.com> On 07/07/2014 12:07, Peter Levart wrote: > Hi Pavel, Alan and Paul, > > Thanks for reviewing. I accepted the suggestions from Pavel and Paul > and created webrev.02: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.factory/webrev.02/ > > Is this good to go into jdk9-dev? The comments looks okay to me (and it clear). So I think this is okay to push to jdk9/dev. -Alan From paul.sandoz at oracle.com Mon Jul 7 15:04:48 2014 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 7 Jul 2014 17:04:48 +0200 Subject: RFR 8049220: URL.factory data race In-Reply-To: <53BA7F6F.9040506@gmail.com> References: <53B517CF.6060006@gmail.com> <53B585EB.9060807@oracle.com> <53BA7F6F.9040506@gmail.com> Message-ID: On Jul 7, 2014, at 1:07 PM, Peter Levart wrote: > Hi Pavel, Alan and Paul, > > Thanks for reviewing. I accepted the suggestions from Pavel and Paul and created webrev.02: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.factory/webrev.02/ > > Is this good to go into jdk9-dev? > Looks ok. Paul. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From xuelei.fan at oracle.com Tue Jul 8 12:40:39 2014 From: xuelei.fan at oracle.com (Xuelei Fan) Date: Tue, 08 Jul 2014 20:40:39 +0800 Subject: RFR 8014870: Faster KDC availability check in Kerberos In-Reply-To: <53BA086A.5040004@oracle.com> References: <17D2CB8C-001B-4929-8BD8-956ADF1E97DE@oracle.com> <53BA086A.5040004@oracle.com> Message-ID: <53BBE6C7.60507@oracle.com> Missed the security-dev list. On 7/7/2014 10:39 AM, Xuelei Fan wrote: > I have not read the fix. I was just wondering that this fix save the > wait time, but increase the networking traffics, and increase the > workload of KDC servers. I think the KDC timeout should be corner cases > for TCP, and it is tolerable for UDP connections. I'm not confident > that this is a cost-effective update if we considering the overall > system of Kerberos. > > Xuelei > > On 6/24/2014 4:17 PM, Wang Weijun wrote: >> Hi All >> >> Please review the code change at >> >> http://cr.openjdk.java.net/~weijun/8014870/webrev.00/ >> >> In Kerberos, when trying to request for a ticket, we tried multiple KDC servers for multiple times. Before this fix, we connect to a server, wait for 30 seconds (the default kdc_timeout). If there is no answer, go to the next KDC, wait for another 30 seconds, and so on. If none of the KDCs replies. We do more rounds (max_retries, default 3) of connections, and fail at last. Altogether with 3 KDCs we will wait at most 3*30*3=270 seconds. >> >> After this fix, connections are non-blocking and made every second, so they can wait at the same time. The kdc_timeout default is also reduced to 10 seconds (the same as other vendors). At the worst case, we will wait 3*3+10=19 seconds. >> >> You might say that changing kdc_timeout to 10 seconds matters a lot here but actually the wait-together style is much more helpful. Suppose only the 3rd KDC is alive, the old code needs to wait for 60 seconds to be able to connect to it, while the new one only needs to wait for 2 seconds, and this has nothing to do with kdc_timeout. >> >> Because of this, I've thrown away the old krb5.kdc.bad.policy security property. It's now not worth remembering which KDC is alive and which is not. >> >> All changes are inside the KdcComm.java file. Others are test and removal of useless things. >> >> I've included net-dev@ because these are all NIO calls, which I was no familiar with. >> >> Thanks >> Max >> > From weijun.wang at oracle.com Tue Jul 8 14:22:08 2014 From: weijun.wang at oracle.com (Wang Weijun) Date: Tue, 8 Jul 2014 22:22:08 +0800 Subject: RFR 8014870: Faster KDC availability check in Kerberos In-Reply-To: <53BBE6C7.60507@oracle.com> References: <17D2CB8C-001B-4929-8BD8-956ADF1E97DE@oracle.com> <53BA086A.5040004@oracle.com> <53BBE6C7.60507@oracle.com> Message-ID: If the first UDP response can be back in a second then there is no extra workload. This should be the most common case since Kerberos is usually used in an enterprise environment with a high network speed. In most cases, the re-sent of a request is due to failed KDCs or even false settings which would wait forever. You are right that it's not necessary to retry TCP. I will apply the max_retries parameter to UDP only. Thanks Max On Jul 8, 2014, at 20:40, Xuelei Fan wrote: > Missed the security-dev list. > > On 7/7/2014 10:39 AM, Xuelei Fan wrote: >> I have not read the fix. I was just wondering that this fix save the >> wait time, but increase the networking traffics, and increase the >> workload of KDC servers. I think the KDC timeout should be corner cases >> for TCP, and it is tolerable for UDP connections. I'm not confident >> that this is a cost-effective update if we considering the overall >> system of Kerberos. >> >> Xuelei >> >> On 6/24/2014 4:17 PM, Wang Weijun wrote: >>> Hi All >>> >>> Please review the code change at >>> >>> http://cr.openjdk.java.net/~weijun/8014870/webrev.00/ >>> From michael.x.mcmahon at oracle.com Tue Jul 8 14:58:23 2014 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Tue, 08 Jul 2014 15:58:23 +0100 Subject: RFR 7150092: NTLM authentication fail if user specified a different realm In-Reply-To: References: <4C46F4A8-4CCB-4075-B594-FA9B51C2F692@oracle.com> Message-ID: <53BC070F.6050909@oracle.com> Max, These changes look fine. Just a couple of minor comments: L130 in Client.java appears to be superfluous now. The comment at L186 in Server.java might probably should be removed or else expanded upon. Thanks Michael On 23/06/14 09:09, Wang Weijun wrote: > Ping again. > > On Jun 12, 2014, at 14:07, Wang Weijun wrote: > >> Hi All >> >> Please review the code change at >> >> http://cr.openjdk.java.net/~weijun/7150092/webrev.00/ >> >> The problem is that in NTLM, the server might prompt for a domain name (in Type 2 message), and the client can also provide one. Before this fix, if the two are different, the client chooses the one from the server. After this fix, the client will always uses its own even if it's empty. This is confirmed by looking at the behavior of IE/Firefox/Chrome. The server sent domain name was designed to be used to create the NTLMv2 response but it's now obsoleted by alist. Chrome/Firefox simply ignore it, so will Java. (IE does use it if there is no alist) >> >> There are some other small changes: >> >> Client.java >> ----------- >> >> 96-108: No one sends hostname and domain in the Type 1 message, so they are removed. Everyone adds a 0x4 flag which means Request Target. >> >> Removed old 137-139: That's the major change. >> >> 159: I used to detect whether there is an alist by looking at the length. This is not accurate if the domain is very long. The correct way is to look at the flag (0x800000 means alist is there) >> >> Server.java >> ----------- >> >> 98: Adds a flag 0x10000 which means the target name (line 99) written into the message is a domain >> >> 135: Always uses the client provided domain to search for password. This is also a part of the major change. >> >> NTLMClient.java >> --------------- >> >> If user has not responded to NameCallback and/or RealmCallback, it means they accept the default value. >> >> NTLMServer.java >> --------------- >> >> ntdomain could be empty or null, the 2-arg constructor of RealmCallback could fail in this case. Use 1-arg constructor. >> >> NTLMAuthentication.java >> ----------------------- >> >> According to my observation of IE/Firefox/Chrome, when user does not type in a domain name in the password prompting dialog, the domain sent to server is an empty string, and the host name is always full name. Update Java to be the same. >> >> NTLMTest.java >> ------------- >> >> Update the test to reflect code changes. >> >> Thanks >> Max >> From peter.levart at gmail.com Wed Jul 9 11:19:25 2014 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 09 Jul 2014 13:19:25 +0200 Subject: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more) In-Reply-To: <20140702005853.000054d5.bernd-2014@eckenfels.net> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B339ED.2030007@gmail.com> <20140702005853.000054d5.bernd-2014@eckenfels.net> Message-ID: <53BD253D.2070907@gmail.com> Hi Bernd, I noticed I haven't answered on two issues you exposed yet... On 07/02/2014 12:58 AM, Bernd Eckenfels wrote: >> > >>> > >L732: I am unsure about the id field, isnt it enough to have the >>> > >identity equality check for the replacement check and otherwise >>> > >depend on equals()? >> > >> >The ConcurrentSkipListSet (based on ConcurrentSkipListMap) that is >> >used as an ordered concurrent (but not blocking or locking) queue of >> >items to be expired is not using hashCode/equals, but compareTo >> >instead. So two instances of CachedAddresses must never compareTo >> >each other as being "equal" if one wants to keep them all in the >> >ConcurrentSkipListSet... > Yes, I mean not using id in compareTo but idendity if (other==this) > return 0; But maybe not worth it as you still need to order identical > entries reliable (maybe by system hash). Yes, if two CachedAddresses objects: ca1, ca2 have same expiryTime and c1 != c2, compareTo still has to be a total order for ConcurrentSkipListSet to function properly. System.identityHashCode() can theoretically return the same value for two different objects. > >>> > >BTW1: might be the wrong RFR, but considering your good performance >>> > >numbers for an active cache, would having 100ms or similiar default >>> > >negative cache time make sense without impacting (visible) the >>> > >semantic. >> > >> >It depends on the "application". If some app is doing more than 10 >> >look-ups of same non-existent hostname per second, it will have >> >effect. Otherwise not. It seems that negative answers are not so >> >frequent in practice that default policy would require caching them. >> >One can always choose the policy manually. > Actually I think an application who retries lookups without rate limit > might not be too uncommon as they typically fail only after timeouts. > However if you have for example a dropped default route nameserver > answer can return the error immediatelly and you get into a busy loop. With negative caching turned on, such loop will be even more busy. It won't make a load on name service, that's true, but it would consume even more CPU time locally. The catch is that default policy can be overridden on a per-app (JVM) basis with a system property. Changing default policy of negative caching is not something I'm trying to do with this patch. It could be done as a separate issue if desired. Regards, Peter > But yes most likely most apps only try it a small time. > > Gruss > Bernd From peter.levart at gmail.com Wed Jul 9 11:52:37 2014 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 09 Jul 2014 13:52:37 +0200 Subject: RFR JDK-8049228: Improve multithreaded scalability of InetAddress cache (was: RFR JDK-7186258: InetAddress$Cache should replace currentTimeMillis with nanoTime (+more)) In-Reply-To: <53BAAB14.1010907@oracle.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> <53BA7DA3.70205@oracle.com> <53BA9C40.3080402@gmail.com> <53BAAB14.1010907@oracle.com> Message-ID: <53BD2D05.8080100@gmail.com> Hi Michael, Thanks for testing. I have prepared another webrev: http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.03/ It only cleans up two comments suggested by Bernd (removed superfluous phrase "with 0" from statements about comparing time instants). So do you think this needs more testing / another review or can I consider this reviewed for jdk9-dev ? Regards, Peter On 07/07/2014 04:13 PM, Michael McMahon wrote: > Peter, > > Thanks for the explanation. No. I think your change is good. I've run > tests here locally > and I'm happy with it overall. > > Michael > > On 07/07/14 14:10, Peter Levart wrote: >> On 07/07/2014 12:59 PM, Michael McMahon wrote: >>> Hi Peter, >>> >>> Is it necessary to remove the cache entry in the local host case >>> (L1226) ? >>> It seems redundant to cache it here, and also explicitly in the >>> CachedLocalHost object >>> >>> Michael >> >> Hi Michael, >> >> Thanks for looking into this. >> >> getLocalHost() seems to have a special hard-coded policy of positive >> caching for 5 seconds that is independent of general getByName() >> caching policy (30 seconds by default). The behaviour of original >> code that I'm trying to replicate is such that when getLocalHost() >> notices a change of local host name -> address mapping, the mapping >> in global cache for this change is also updated. I think this is to >> avoid situations like: >> >> Let's say the local host name is "cube": >> >> InetAddress addr1 = InetAddress.getLocalHost(); >> InetAddress addr2 = InetAddress.getByName("cube"); >> // addr1.equals(addr2) here >> >> // 5 seconds later, cube -> IP mapping is updated in DNS or >> /etc/hosts ... >> >> addr1 = InetAddress.getLocalHost(); >> addr2 = InetAddress.getByName("cube"); >> // if getLocalHost() did not update global cache, >> // addr1 (new IP address) would be different from addr2 (old IP address) >> >> >> Another way to accomplish similar guarantee would be to special-case >> the caching policy in global cache which would check whether the >> entry is for local host name and set 'expiryTime' accordingly. This >> would be a little different behaviourally, because >> InetAddress.getByName() would get a 5 second expiry for local host >> name too, regardless of whether InetAddress.getLocalHost() has been >> called at all. But we could get rid of special CachedLocalHost class >> then. Is such behavioural change warranted? >> >> Regards, Peter >> >>> >>> On 02/07/14 12:56, Peter Levart wrote: >>>> Hi, >>>> >>>> I updated the webrev with first two suggestions from Bernd >>>> (expireTime instead of createTime and cacheNanos + only use >>>> putIfAbsent instead of get followed by putIfAbsent): >>>> >>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.02/ >>>> >>>> >>>> Thanks, Bernd. >>>> >>>> The id field in CachedAddresses is necessary for compareTo to never >>>> return 0 for two different instances (used as element in >>>> ConcurrentSkipListSet). >>>> >>>> For last two suggestions I'm not sure whether they are desired, so >>>> I'm currently leaving them as is. >>>> >>>> >>>> Regards, Peter >>>> >>>> On 07/01/2014 10:06 PM, Bernd Eckenfels wrote: >>>>> Looks good, like it, Peter. >>>>> >>>>> some nits: instead of adding createTime and cacheNanos, only have a >>>>> expireAfter? >>>>> >>>>> L782: is it better to use putIfAbsent unconditionally, instead of >>>>> get/putIfAbsent in NameServicdeAddr? >>>>> >>>>> L732: I am unsure about the id field, isnt it enough to have the >>>>> identity equality check for the replacement check and otherwise >>>>> depend >>>>> on equals()? >>>>> >>>>> L1223: What about moving the cache exiry inside the if (useCache) >>>>> >>>>> BTW1: might be the wrong RFR, but considering your good performance >>>>> numbers for an active cache, would having 100ms or similiar default >>>>> negative cache time make sense without impacting (visible) the >>>>> semantic. >>>>> >>>>> >>>>> >>>>> Gruss >>>>> Bernd >>>>> >>>>> >>>>> Am Tue, 01 Jul 2014 20:35:57 +0200 >>>>> schrieb Peter Levart : >>>>> >>>>>> Hi, >>>>>> >>>>>> I propose a patch for this issue: >>>>>> >>>>>> https://bugs.openjdk.java.net/browse/JDK-7186258 >>>>>> >>>>>> The motivation to re-design caching of InetAddress-es was not this >>>>>> issue though, but a desire to attack synchronization bottlenecks in >>>>>> methods like URL.equals and URL.hashCode which use host name to IP >>>>>> address mapping. I plan to tackle the synchronization in URL in a >>>>>> follow-up proposal, but I wanted to 1st iron-out the "leaves" of the >>>>>> call-tree. Here's the proposed patch: >>>>>> >>>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >>>>>> >>>>>> >>>>>> sun.net.InetAddressCachePolicy: >>>>>> >>>>>> - two static methods (get() and getNegative()) were synchronized. >>>>>> Removed synchronization and made underlying fields volatile. >>>>>> - also added a normalization of negative policy in >>>>>> setNegativeIfNotSet(). The logic in InetAddress doesn't cope with >>>>>> negative values distinct from InetAddressCachePolicy.FOREVER >>>>>> (-1), so >>>>>> this was a straight bug. The setIfNotSet() doesn't need this >>>>>> normalization, because checkValue() throws exception if passed-in >>>>>> value < InetAddressCachePolicy.FOREVER. >>>>>> >>>>>> java.net.InetAddress: >>>>>> >>>>>> - complete redesign of caching. Instead of distinct >>>>>> Positive/Negative >>>>>> caches, there's only one cache - a ConcurrentHashMap. The value in >>>>>> the map knows if it contains positive or negative answer. >>>>>> - the design of this cache is similar but much simpler than >>>>>> java.lang.reflect.WeakCache, since it doesn't have to deal with >>>>>> WeakReferences and keys are simpler (just strings - hostnames). >>>>>> Similarity is in how concurrent requests for the same key (hostname) >>>>>> are synchronized when the entry is not cached yet, but still avoid >>>>>> synchronization when entry is cached. This preserves the >>>>>> behaviour of >>>>>> original InetAddress caching code but simplifies it greatly (100+ >>>>>> lines removed). >>>>>> - I tried to preserve the interaction between >>>>>> InetAddress.getLocalHost() and InetAddress.getByName(). The >>>>>> getLocalHost() caches the local host address for 5 seconds >>>>>> privately. >>>>>> When it expires it performs new name service look-up and "refreshes" >>>>>> the entry in the InetAddress.getByName() cache although it has not >>>>>> expired yet. I think this is meant to prevent surprises when >>>>>> getLocalHost() returns newer address than getByName() which is >>>>>> called >>>>>> after that. >>>>>> - I also fixed the JDK-7186258 as a by-product (but don't know yet >>>>>> how to write a test for this issue - any ideas?) >>>>>> >>>>>> I created a JMH benchmark that tests the following methods: >>>>>> >>>>>> - InetAddress.getLocalHost() >>>>>> - InetAddress.getByName() (with positive and negative answer) >>>>>> >>>>>> Here're the results of running on my 4-core (8-threads) i7/Linux: >>>>>> >>>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >>>>>> >>>>>> >>>>>> >>>>>> The getByNameNegative() test does not show much improvement in >>>>>> patched vs. original code. That's because by default the policy >>>>>> is to >>>>>> NOT cache negative answers. Requests for same hostname to the >>>>>> NameService(s) are synchronized. If >>>>>> "networkaddress.cache.negative.ttl" system property is set to some >>>>>> positive value, results are similar to those of getByNamePositive() >>>>>> test (the default policy for positive caching is 30 seconds). >>>>>> >>>>>> I ran the jtreg tests in test/java/net and have the same score as >>>>>> with original unpatched code. I have 3 failing tests from original >>>>>> and patched runs: >>>>>> >>>>>> JT Harness : Tests that failed >>>>>> java/net/MulticastSocket/Promiscuous.java: Test for interference >>>>>> when >>>>>> two sockets are bound to the same port but joined to different >>>>>> multicast groups >>>>>> java/net/MulticastSocket/SetLoopbackMode.java: Test >>>>>> MulticastSocket.setLoopbackMode >>>>>> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting >>>>>> broken >>>>>> on Linux >>>>>> >>>>>> And 1 test that had error trying to be run: >>>>>> >>>>>> JT Harness : Tests that had errors >>>>>> java/net/URLPermission/nstest/lookup.sh: >>>>>> >>>>>> Because of: >>>>>> >>>>>> test result: Error. Can't find source file: >>>>>> jdk/testlibrary/*.java in >>>>>> directory-list: >>>>>> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >>>>>> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >>>>>> >>>>>> All other 258 java/net tests pass. >>>>>> >>>>>> >>>>>> >>>>>> So what do you think? >>>>>> >>>>>> >>>>>> Regards, Peter >>>>>> >>>>>> >>>> >>> >> > From peter.levart at gmail.com Wed Jul 9 12:12:20 2014 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 09 Jul 2014 14:12:20 +0200 Subject: RFR JDK-8049228: Improve multithreaded scalability of InetAddress cache In-Reply-To: <53BD2D62.5050504@oracle.com> References: <53B2FF8D.5040503@gmail.com> <20140701220621.00005e21.bernd-2014@eckenfels.net> <53B3F377.3090208@gmail.com> <53BA7DA3.70205@oracle.com> <53BA9C40.3080402@gmail.com> <53BAAB14.1010907@oracle.com> <53BD2D05.8080100@gmail.com> <53BD2D62.5050504@oracle.com> Message-ID: <53BD31A4.9020904@gmail.com> On 07/09/2014 01:54 PM, Claes Redestad wrote: > I see you're still expanding scope from private to package: > > - private static InetAddress[] getAddressesFromNameService(String > host, InetAddress reqAddr) > + static InetAddress[] getAddressesFromNameService(String host, > InetAddress reqAddr) > > /Claes Thanks for being alert, Claes. I think I own an explanation. After you have notified me about this the 1st time, I remembered why I dropped the "private" from method declaration in the first place. The method is now called from within NameServiceAddresses inner class. I wanted to suppress javac from generating synthetic access methods. Since java.net is a sealed package, there's no security issue, am I right? Regards, Peter > > On 07/09/2014 01:52 PM, Peter Levart wrote: >> Hi Michael, >> >> Thanks for testing. I have prepared another webrev: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.03/ >> >> >> It only cleans up two comments suggested by Bernd (removed >> superfluous phrase "with 0" from statements about comparing time >> instants). So do you think this needs more testing / another review >> or can I consider this reviewed for jdk9-dev ? >> >> Regards, Peter >> >> >> On 07/07/2014 04:13 PM, Michael McMahon wrote: >>> Peter, >>> >>> Thanks for the explanation. No. I think your change is good. I've >>> run tests here locally >>> and I'm happy with it overall. >>> >>> Michael >>> >>> On 07/07/14 14:10, Peter Levart wrote: >>>> On 07/07/2014 12:59 PM, Michael McMahon wrote: >>>>> Hi Peter, >>>>> >>>>> Is it necessary to remove the cache entry in the local host case >>>>> (L1226) ? >>>>> It seems redundant to cache it here, and also explicitly in the >>>>> CachedLocalHost object >>>>> >>>>> Michael >>>> >>>> Hi Michael, >>>> >>>> Thanks for looking into this. >>>> >>>> getLocalHost() seems to have a special hard-coded policy of >>>> positive caching for 5 seconds that is independent of general >>>> getByName() caching policy (30 seconds by default). The behaviour >>>> of original code that I'm trying to replicate is such that when >>>> getLocalHost() notices a change of local host name -> address >>>> mapping, the mapping in global cache for this change is also >>>> updated. I think this is to avoid situations like: >>>> >>>> Let's say the local host name is "cube": >>>> >>>> InetAddress addr1 = InetAddress.getLocalHost(); >>>> InetAddress addr2 = InetAddress.getByName("cube"); >>>> // addr1.equals(addr2) here >>>> >>>> // 5 seconds later, cube -> IP mapping is updated in DNS or >>>> /etc/hosts ... >>>> >>>> addr1 = InetAddress.getLocalHost(); >>>> addr2 = InetAddress.getByName("cube"); >>>> // if getLocalHost() did not update global cache, >>>> // addr1 (new IP address) would be different from addr2 (old IP >>>> address) >>>> >>>> >>>> Another way to accomplish similar guarantee would be to >>>> special-case the caching policy in global cache which would check >>>> whether the entry is for local host name and set 'expiryTime' >>>> accordingly. This would be a little different behaviourally, >>>> because InetAddress.getByName() would get a 5 second expiry for >>>> local host name too, regardless of whether >>>> InetAddress.getLocalHost() has been called at all. But we could get >>>> rid of special CachedLocalHost class then. Is such behavioural >>>> change warranted? >>>> >>>> Regards, Peter >>>> >>>>> >>>>> On 02/07/14 12:56, Peter Levart wrote: >>>>>> Hi, >>>>>> >>>>>> I updated the webrev with first two suggestions from Bernd >>>>>> (expireTime instead of createTime and cacheNanos + only use >>>>>> putIfAbsent instead of get followed by putIfAbsent): >>>>>> >>>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.02/ >>>>>> >>>>>> >>>>>> Thanks, Bernd. >>>>>> >>>>>> The id field in CachedAddresses is necessary for compareTo to >>>>>> never return 0 for two different instances (used as element in >>>>>> ConcurrentSkipListSet). >>>>>> >>>>>> For last two suggestions I'm not sure whether they are desired, >>>>>> so I'm currently leaving them as is. >>>>>> >>>>>> >>>>>> Regards, Peter >>>>>> >>>>>> On 07/01/2014 10:06 PM, Bernd Eckenfels wrote: >>>>>>> Looks good, like it, Peter. >>>>>>> >>>>>>> some nits: instead of adding createTime and cacheNanos, only have a >>>>>>> expireAfter? >>>>>>> >>>>>>> L782: is it better to use putIfAbsent unconditionally, instead of >>>>>>> get/putIfAbsent in NameServicdeAddr? >>>>>>> >>>>>>> L732: I am unsure about the id field, isnt it enough to have the >>>>>>> identity equality check for the replacement check and otherwise >>>>>>> depend >>>>>>> on equals()? >>>>>>> >>>>>>> L1223: What about moving the cache exiry inside the if (useCache) >>>>>>> >>>>>>> BTW1: might be the wrong RFR, but considering your good performance >>>>>>> numbers for an active cache, would having 100ms or similiar default >>>>>>> negative cache time make sense without impacting (visible) the >>>>>>> semantic. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Gruss >>>>>>> Bernd >>>>>>> >>>>>>> >>>>>>> Am Tue, 01 Jul 2014 20:35:57 +0200 >>>>>>> schrieb Peter Levart : >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I propose a patch for this issue: >>>>>>>> >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-7186258 >>>>>>>> >>>>>>>> The motivation to re-design caching of InetAddress-es was not this >>>>>>>> issue though, but a desire to attack synchronization >>>>>>>> bottlenecks in >>>>>>>> methods like URL.equals and URL.hashCode which use host name to IP >>>>>>>> address mapping. I plan to tackle the synchronization in URL in a >>>>>>>> follow-up proposal, but I wanted to 1st iron-out the "leaves" >>>>>>>> of the >>>>>>>> call-tree. Here's the proposed patch: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.01/ >>>>>>>> >>>>>>>> >>>>>>>> sun.net.InetAddressCachePolicy: >>>>>>>> >>>>>>>> - two static methods (get() and getNegative()) were synchronized. >>>>>>>> Removed synchronization and made underlying fields volatile. >>>>>>>> - also added a normalization of negative policy in >>>>>>>> setNegativeIfNotSet(). The logic in InetAddress doesn't cope with >>>>>>>> negative values distinct from InetAddressCachePolicy.FOREVER >>>>>>>> (-1), so >>>>>>>> this was a straight bug. The setIfNotSet() doesn't need this >>>>>>>> normalization, because checkValue() throws exception if passed-in >>>>>>>> value < InetAddressCachePolicy.FOREVER. >>>>>>>> >>>>>>>> java.net.InetAddress: >>>>>>>> >>>>>>>> - complete redesign of caching. Instead of distinct >>>>>>>> Positive/Negative >>>>>>>> caches, there's only one cache - a ConcurrentHashMap. The value in >>>>>>>> the map knows if it contains positive or negative answer. >>>>>>>> - the design of this cache is similar but much simpler than >>>>>>>> java.lang.reflect.WeakCache, since it doesn't have to deal with >>>>>>>> WeakReferences and keys are simpler (just strings - hostnames). >>>>>>>> Similarity is in how concurrent requests for the same key >>>>>>>> (hostname) >>>>>>>> are synchronized when the entry is not cached yet, but still avoid >>>>>>>> synchronization when entry is cached. This preserves the >>>>>>>> behaviour of >>>>>>>> original InetAddress caching code but simplifies it greatly (100+ >>>>>>>> lines removed). >>>>>>>> - I tried to preserve the interaction between >>>>>>>> InetAddress.getLocalHost() and InetAddress.getByName(). The >>>>>>>> getLocalHost() caches the local host address for 5 seconds >>>>>>>> privately. >>>>>>>> When it expires it performs new name service look-up and >>>>>>>> "refreshes" >>>>>>>> the entry in the InetAddress.getByName() cache although it has not >>>>>>>> expired yet. I think this is meant to prevent surprises when >>>>>>>> getLocalHost() returns newer address than getByName() which is >>>>>>>> called >>>>>>>> after that. >>>>>>>> - I also fixed the JDK-7186258 as a by-product (but don't know yet >>>>>>>> how to write a test for this issue - any ideas?) >>>>>>>> >>>>>>>> I created a JMH benchmark that tests the following methods: >>>>>>>> >>>>>>>> - InetAddress.getLocalHost() >>>>>>>> - InetAddress.getByName() (with positive and negative answer) >>>>>>>> >>>>>>>> Here're the results of running on my 4-core (8-threads) i7/Linux: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/InetAddress.Cache_bench_results.01.pdf >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> The getByNameNegative() test does not show much improvement in >>>>>>>> patched vs. original code. That's because by default the policy >>>>>>>> is to >>>>>>>> NOT cache negative answers. Requests for same hostname to the >>>>>>>> NameService(s) are synchronized. If >>>>>>>> "networkaddress.cache.negative.ttl" system property is set to some >>>>>>>> positive value, results are similar to those of >>>>>>>> getByNamePositive() >>>>>>>> test (the default policy for positive caching is 30 seconds). >>>>>>>> >>>>>>>> I ran the jtreg tests in test/java/net and have the same score as >>>>>>>> with original unpatched code. I have 3 failing tests from original >>>>>>>> and patched runs: >>>>>>>> >>>>>>>> JT Harness : Tests that failed >>>>>>>> java/net/MulticastSocket/Promiscuous.java: Test for >>>>>>>> interference when >>>>>>>> two sockets are bound to the same port but joined to different >>>>>>>> multicast groups >>>>>>>> java/net/MulticastSocket/SetLoopbackMode.java: Test >>>>>>>> MulticastSocket.setLoopbackMode >>>>>>>> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting >>>>>>>> broken >>>>>>>> on Linux >>>>>>>> >>>>>>>> And 1 test that had error trying to be run: >>>>>>>> >>>>>>>> JT Harness : Tests that had errors >>>>>>>> java/net/URLPermission/nstest/lookup.sh: >>>>>>>> >>>>>>>> Because of: >>>>>>>> >>>>>>>> test result: Error. Can't find source file: >>>>>>>> jdk/testlibrary/*.java in >>>>>>>> directory-list: >>>>>>>> /home/peter/work/hg/jdk9-dev/jdk/test/java/net/URLPermission/nstest >>>>>>>> >>>>>>>> /home/peter/work/hg/jdk9-dev/jdk/test/lib/testlibrary >>>>>>>> >>>>>>>> All other 258 java/net tests pass. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> So what do you think? >>>>>>>> >>>>>>>> >>>>>>>> Regards, Peter >>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>> >> > > From peter.firmstone at zeus.net.au Thu Jul 10 00:50:43 2014 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Thu, 10 Jul 2014 10:50:43 +1000 Subject: java.net.URI and RFC 3986 compliance Message-ID: <53BDE363.7050101@zeus.net.au> Are there parties on this list interested in updating java.net.URI to RFC3986? Is there anyone here who has previously attempted this? If so what issues did you find with regard to backward compatibility? Regards, Peter. From peter.levart at gmail.com Thu Jul 10 08:11:18 2014 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 10 Jul 2014 10:11:18 +0200 Subject: java.net.URI and RFC 3986 compliance In-Reply-To: <53BDE363.7050101@zeus.net.au> References: <53BDE363.7050101@zeus.net.au> Message-ID: <53BE4AA6.4000306@gmail.com> On 07/10/2014 02:50 AM, Peter Firmstone wrote: > Are there parties on this list interested in updating java.net.URI to > RFC3986? > > Is there anyone here who has previously attempted this? If so what > issues did you find with regard to backward compatibility? > > Regards, > > Peter. > Hi Peter, I think It's good that you started this discussion. Since you already have some experience with RFC3986 and possibly also know the differences to RFC 2396 (I have just started reading both RFCs, so I am in no way a relevant source of information in this area), please feel free to ignore my noob questions... Would it be possible to add a method to URI, say URI.normalize3986() and leave existing logic of URI unchanged? What could and what couldn't be achieved by that? Regards, Peter From Alan.Bateman at oracle.com Thu Jul 10 08:25:05 2014 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 10 Jul 2014 09:25:05 +0100 Subject: java.net.URI and RFC 3986 compliance In-Reply-To: <53BDE363.7050101@zeus.net.au> References: <53BDE363.7050101@zeus.net.au> Message-ID: <53BE4DE1.3080308@oracle.com> On 10/07/2014 01:50, Peter Firmstone wrote: > Are there parties on this list interested in updating java.net.URI to > RFC3986? > > Is there anyone here who has previously attempted this? If so what > issues did you find with regard to backward compatibility? Hopefully Michael McMahon will jump in and give some of the history. In summary, it was upgraded to RFC 3986 during JDK 6 (before OpenJDK). There were many incompatibilities understood and analyzed before the update was attempted but it was that the issues that came out of the wood work later that lead to a complete back-out of the update. I think some of the history should be in the bug database. -Alan From michael.x.mcmahon at oracle.com Thu Jul 10 08:43:07 2014 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Thu, 10 Jul 2014 09:43:07 +0100 Subject: java.net.URI and RFC 3986 compliance In-Reply-To: <53BE4AA6.4000306@gmail.com> References: <53BDE363.7050101@zeus.net.au> <53BE4AA6.4000306@gmail.com> Message-ID: <53BE521B.7030500@oracle.com> On 10/07/14 09:11, Peter Levart wrote: > On 07/10/2014 02:50 AM, Peter Firmstone wrote: >> Are there parties on this list interested in updating java.net.URI to >> RFC3986? >> >> Is there anyone here who has previously attempted this? If so what >> issues did you find with regard to backward compatibility? >> >> Regards, >> >> Peter. >> > > Hi Peter, > > I think It's good that you started this discussion. Since you already > have some experience with RFC3986 and possibly also know the > differences to RFC 2396 (I have just started reading both RFCs, so I > am in no way a relevant source of information in this area), please > feel free to ignore my noob questions... > > Would it be possible to add a method to URI, say URI.normalize3986() > and leave existing logic of URI unchanged? What could and what > couldn't be achieved by that? > > Regards, Peter > I think the starting point should be a list of the differences, and then we can see whether additional methods would suffice. Apart from anything else, the behavior of each method has to be fully specified and documented in the class. I'll try to locate any relevant history. But, support for RFC3986 is definitely desirable. I agree also, that it shouldn't be predicated on a global switch. Additional constructors/factory methods would be one approach, but it is dependent on what the differences are. Michael From peter.firmstone at zeus.net.au Thu Jul 10 09:00:56 2014 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Thu, 10 Jul 2014 19:00:56 +1000 Subject: java.net.URI and RFC 3986 compliance In-Reply-To: <53BE521B.7030500@oracle.com> References: <53BDE363.7050101@zeus.net.au> <53BE4AA6.4000306@gmail.com> <53BE521B.7030500@oracle.com> Message-ID: <53BE5648.2070900@zeus.net.au> We can certainly use different constructors and different parsers to achieve each form, where it will get interesting is equality. The existing URI implementation is more likely to give false negatives, while RFC 3986 is not likely to, due to its well defined normalisation. In fact, I'd be inclined to use RFC 3986 for equality and identity if possible. I started listing the differences, java.net.URI also deviates from the standard, so those deviations also need to be taken into account, when originally implementing RFC 3986 I started from Harmony's URI, however 3986 uses sufficiently different terminology, such that the strategy I used was to create different legal lists for each URI component and escape the rest. Some of the deviations in javal.net.URI were adopted by 3986. Regards, Peter. On 10/07/2014 6:43 PM, Michael McMahon wrote: > On 10/07/14 09:11, Peter Levart wrote: >> On 07/10/2014 02:50 AM, Peter Firmstone wrote: >>> Are there parties on this list interested in updating java.net.URI >>> to RFC3986? >>> >>> Is there anyone here who has previously attempted this? If so what >>> issues did you find with regard to backward compatibility? >>> >>> Regards, >>> >>> Peter. >>> >> >> Hi Peter, >> >> I think It's good that you started this discussion. Since you already >> have some experience with RFC3986 and possibly also know the >> differences to RFC 2396 (I have just started reading both RFCs, so I >> am in no way a relevant source of information in this area), please >> feel free to ignore my noob questions... >> >> Would it be possible to add a method to URI, say URI.normalize3986() >> and leave existing logic of URI unchanged? What could and what >> couldn't be achieved by that? >> >> Regards, Peter >> > > I think the starting point should be a list of the differences, and > then we can see whether additional methods > would suffice. Apart from anything else, the behavior of each method > has to be fully specified and documented in the class. > I'll try to locate any relevant history. > > But, support for RFC3986 is definitely desirable. I agree also, that > it shouldn't be predicated on a global switch. > Additional constructors/factory methods would be one approach, but it > is dependent on what the differences are. > > Michael > From kubota.yuji at lab.ntt.co.jp Thu Jul 10 09:33:26 2014 From: kubota.yuji at lab.ntt.co.jp (KUBOTA Yuji) Date: Thu, 10 Jul 2014 18:33:26 +0900 Subject: Do not JDK-6967684 backport to JDK6? In-Reply-To: <4E485579-2E8C-44D5-A7C2-82C8223C111F@azulsystems.com> References: <53A798ED.3040905@lab.ntt.co.jp> <4E485579-2E8C-44D5-A7C2-82C8223C111F@azulsystems.com> Message-ID: <53BE5DE6.3030100@lab.ntt.co.jp> Thank you for response and CC to proper ml :) I hope to backport this patch if there is no specific reason to block. Yuji (2014/07/04 20:46), Ivan Krylov wrote: > Don?t have an answer this but CC-ing openjdk6-dev > > Ivan > > > On 23 Jun 2014, at 07:03, KUBOTA Yuji wrote: > >> Hi all, >> >> I got ArrayIndexOutOfBoundsException at java.text.SimpleDateFormat.subFormat >> via sun.net.httpserver.ExchangeImpl when I used com.sun.net.httpserver >> with OpenJDK6 as the tail of this mail. >> >> The reason why this exception occured is that httpserver (ExchangeImpl) uses >> a non thread-safe SimpleDateFormat which has been modified at OpenJDK7 >> by JDK-6967684. >> ---- >> Bug database: http://bugs.java.com/view_bug.do?bug_id=6967684 >> Patch: http://hg.openjdk.java.net/jdk7/jdk7/jdk/rev/1371a2d5f3a8 >> ML: http://mail.openjdk.java.net/pipermail/net-dev/2010-July/001946.html >> ---- >> >> And the HEAD of jdk6 repos do not include this patch. >> If anyone knows why this patch do not backport to OpenJDK6, please tell me. >> >> --Exception details--- >> # java -version >> java version "1.6.0_28" >> OpenJDK Runtime Environment (IcedTea6 1.13.0pre) >> (rhel-1.66.1.13.0.el6-x86_64) OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode) >> >> # stack trace >> java.lang.ArrayIndexOutOfBoundsException: -2147483648 >> java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1066) >> java.text.SimpleDateFormat.format(SimpleDateFormat.java:899) >> java.text.SimpleDateFormat.format(SimpleDateFormat.java:869) >> java.text.DateFormat.format(DateFormat.java:333) >> sun.net.httpserver.ExchangeImpl.sendResponseHeaders(ExchangeImpl.java:206) >> sun.net.httpserver.HttpExchangeImpl.sendResponseHeaders(HttpExchangeImpl.java:86) >> com.sun.xml.internal.ws.transport.http.server.ServerConnectionImpl.getOutput(ServerConnectionImpl.java:145) >> com.sun.xml.internal.ws.transport.http.HttpAdapter.encodePacket(HttpAdapter.java:321) >> com.sun.xml.internal.ws.transport.http.HttpAdapter.access$100(HttpAdapter.java:82) >> com.sun.xml.internal.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:470) >> com.sun.xml.internal.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:233) >> com.sun.xml.internal.ws.transport.http.server.WSHttpHandler.handleExchange(WSHttpHandler.java:95) >> com.sun.xml.internal.ws.transport.http.server.WSHttpHandler$HttpHandlerRunnable.run(WSHttpHandler.java:117) >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> java.lang.Thread.run(Thread.java:679) >> ---------- >> >> Thanks in advance for your help! >> >> Best regards, >> KUBOTA Yuji. >> > From peter.levart at gmail.com Fri Jul 11 15:11:15 2014 From: peter.levart at gmail.com (Peter Levart) Date: Fri, 11 Jul 2014 17:11:15 +0200 Subject: URL constructor/equals/hashCode/sameFile/openConnection synchronization issues Message-ID: <53BFFE93.9010103@gmail.com> Hi, java.net.URL is supposed to behave as an immutable object, so URL instances can be shared among threads and among parts of code without fear that they will be modified. URL class has an unusual way to achieve this (or at least it tries to). Partly because of the design which uses: - URL constructor(s) that take a 'spec' String to be parsed into URL object - parsing is delegated to various URLStreamHandler(s) which are chosen in the URL constructor (depending on the protocol used in URL string to be parsed) An unitialized URL instance (this) is passed from constructor to the chosen URLStreamHandler which has the responsibility to parse the string and set back the fields that hold various parts of URL object being constructed. Consequently, these fields can not be declared as final, as definite assignment analysis doesn't cross method borders. It is therefore illegal to unsafely publish URL instances (via data races) to non-constructing threads because they can appear not fully initialized. Nevertheless URL, with the help of various URLStreamHandler implementations, tries hard to make URL appear stable at least where it is required to be stable. For example: URL.hashCode() is (almost) stable even if URL instance is unsafely published. This is achieved by making hashCode() synchronized and cache the result. At least one way of constructing URLs - constructors that take 'spec' String to be parsed - is also making sure that hashCode is computed from fully initialized fields, as parsing is delegated to URLStreamHandler which uses package-private URL.set() method to set back the parsed fields and the set() method is also synchronized. But making URL appear stable even though it is published unsafely doesn't seem to be the primary concern of URL synchronization. Other public URL constructors that take individual URL parts and don't delegate parsing to URLStreamHandler but set fields directly (not via set() method), are not synchronized. Primary concern of synchronization in URL appears to be driven from the fact that some URL operations like hasCode(), equals(), sameFile() and openConnection() read multiple URL fields and URL.set() which can be called from custom URLStreamHandler at any time (although this is not it's purpose - it should only call-back while parsing/constructing the URL) can set those fields. And those multi-field operations would like to see a "snapshot" of field values that is consistent. But synchronization that is performed to achieve that is questionable. Might be that in Java 1.0 times the JVM implementation assumptions were different and synchronization was correct, but nowadays Java memory model makes them invalid. URL.hasCode() apears to be the only method properly synchronized which makes it almost stable (doesn't return different results over time) but even hashCode() has a subtle bug or two. The initializer for hashCode field sets it to value -1 which represents "not yet computed" state. If URL is published unsafely, hashCode() method could see the "default" value of 0, which would be returned. A later call to hashCode() would see value -1 which would trigger computation and a different value would be returned. The other subtle bug is a relatively improbable event that hashCode computation results in value -1 which means "not yet computed". This can be seen as performance glitch (as hashCode will never be cached for such URL instance) or an issue which makes hashCode unstable for one of the reasons why equals() is unstable too (see below). If URL.hasCode() method is almost stable (doesn't return different results over time) and at least one way of URL construction makes sure that hashCode is also calculated from fully initialized parts, then URL.equals() and other methods that delegate to URLStreamHandler are a special story. URL.equals() can't be synchronized on URL instance, because it would have to be synchronized on both URL instances that are being compared and this is prone do dead-locks. Imagine: thread1: url1.equals(url2) thread2: url2.equals(url1) So equals() chooses to not be synchronized and therefore risks not being stable if URL instances are published unsafely. But it nevertheless uses synchronization. equals() delegates it's work to the 1st URL's URLStreamHandler which synchronizes on itself when calculating and caching the InetAddress of each individual URL's host name. InetAddress (if resolvable) is used in preference to host name for comparison (and also in hashCode()). URL.equals() risks not being stable for the following reasons: - URL instances published unsafely can appear not fully initialized to equals() even though they were constructed with constructors that delegate parsing to URLStreamHandler(s) which use synchronized URL.set() to set the fields, because URL.equals() is not synchronized. - URL.hostAddress that is calculated on demand and then cached on the URL instance should help make equals() stable in the presence of dynamic changes to host name -> IP address mapping, but caching is not performed for unsuccessful resolving. Temporary name service outage can make URL.equals() unstable. - URL.hostAddress caching is using the URLStreamHandler instance of the 1st URL as a lock for synchronizing read/write of hostAddress of both URLs being compared by equals(). But URLStreamHandler(s) of the two URL instances need not be the same instance even though they are for the same protocol. Imagine: URL url1 = new URL(null, "http://www.google.com/", handler1); URL url2 = new URL(null, "http://www.google.com/", handler2); ... thread1: url1.equals(url2); thread2: url2.equals(url1); Each thread could be using different instance of URLStreamHandler for synchronization and could overwrite each other the cached hostAddress on individual URLs. These hostAddress values could be different in the presence of dynamic changes to host name -> IP address mapping and could therefore make URL.equals() unstable. This is admittedly a very unlikely scenario, but is theoretically possible. URL.sameHost() has exactly the same issues as URL.equals() as it only makes one field comparison less. URL.openConnection() is a question in itself. It is delegated to URLStreamHandler. Some URLStreamHandlers make it synchronized and others not. Those that make it synchronized (on the URLStreamHandler instance) do this for no apparent reason. This synchronization can't help make URL fields stable for the time of openConnection() call since URL.set() is using a different lock (the URL instance itself). It only makes things worse since access to opening the connection to the resources is serialized and this presents a bottleneck. I tried to fix all these issues and came up with the following patch which I'm proposing: http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/webrev.01/ New JDK8 synchronization primitive: StampedLock is a perfect tool for solving these issues as synchronization in URL is only necessary to establish visibility of initialized state which doesn't change afterwards or changes at most once (when computing hashCode). StampedLock's tryOptimisticRead/validate is perfect for such situations as it only presents a negligible overhead of two volatile reads. The presented patch also contains an unrelated change which replaces usage of Hashtable with ConcurrentHashMap for holding the mapping from protocol to individual URLStreamhandler which makes for better scallability of URL constructors. Combined with synchronization scallability enhancements of InetAddress caching presented in an earlier proposal, the net result is much better scalability of URL constructor/equals/hashCode (executed by a JMH test on a 4-core i7/Linux box): http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/URL.synchronization_bench_results.pdf So with this patch java.net.URL could be treated as an unsafe-publication-tolerable class (like String). Well, not entirely, since getters for individual fields are still just unguarded normal reads. But those reads are usually performed under guard of a StampedLock when URL equals/hashCode/sameFile/openConnection/toExternalForm() operations are delegated to URLStreamHandler and therefore don't suffer from (in)visibility of published state. Fields could be made volatile to fix this if desired. I'm not entirely sure about why openConnection() in file: and mailto: URLStreamHandlers is synchronized. I don't see a reason, so I removed the synchronization. If there is a reason, please bring it forward. I ran the java/net jtreg tests with unpatched recent jdk9-dev and patched (combined changes of URL and InetAddress caching) and get the same score. Only the following 3 tests fail in both ocasions: JT Harness : Tests that failed java/net/MulticastSocket/Promiscuous.java: Test for interference when two sockets are bound to the same port but joined to different multicast groups java/net/MulticastSocket/SetLoopbackMode.java: Test MulticastSocket.setLoopbackMode java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken on Linux They seem to not like my multicast configuration or something. All other 258 tests pass. So what do you think? Regards, Peter From mark.sheppard at oracle.com Wed Jul 16 17:00:31 2014 From: mark.sheppard at oracle.com (Mark Sheppard) Date: Wed, 16 Jul 2014 18:00:31 +0100 Subject: RFR: JDK-8040810 - Uninitialised memory in jdk/src/windows/native/java/net: net_util_md.c, TwoStacksPlainSocketImpl.c, TwoStacksPlainDatagramSocketImpl.c, DualStackPlainSocketImpl.c, DualStackPlainDatagramSocketImpl.c Message-ID: <53C6AFAF.80108@oracle.com> Hi please oblige and review the following changes http://cr.openjdk.java.net/~msheppar/8040810/webrev/ which address the issue raised in https://bugs.openjdk.java.net/browse/JDK-8040810 resulting from static code analysis. these changes explicitly initialize local function variables, which are in the main out parameters to other function calls and hence are set within the called function. It can be reasonably argued that the initialization is unnecessary, but current coding guidance is to perform the initialization regards Mark From Alan.Bateman at oracle.com Wed Jul 16 17:28:23 2014 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 16 Jul 2014 18:28:23 +0100 Subject: RFR: JDK-8040810 - Uninitialised memory in jdk/src/windows/native/java/net: net_util_md.c, TwoStacksPlainSocketImpl.c, TwoStacksPlainDatagramSocketImpl.c, DualStackPlainSocketImpl.c, DualStackPlainDatagramSocketImpl.c In-Reply-To: <53C6AFAF.80108@oracle.com> References: <53C6AFAF.80108@oracle.com> Message-ID: <53C6B637.9010809@oracle.com> On 16/07/2014 18:00, Mark Sheppard wrote: > Hi > please oblige and review the following changes > > http://cr.openjdk.java.net/~msheppar/8040810/webrev/ > > which address the issue raised in > > https://bugs.openjdk.java.net/browse/JDK-8040810 > > resulting from static code analysis. > > these changes explicitly initialize local function variables, which > are in the main > out parameters to other function calls and hence are set within the > called function. > It can be reasonably argued that the initialization is unnecessary, > but current coding > guidance is to perform the initialization > I assume in NET_Bind that it isn't necessary to initialize rv, same thing in NET_SocketClose. Otherwise looks okay. -Alan From mark.sheppard at oracle.com Wed Jul 16 18:09:11 2014 From: mark.sheppard at oracle.com (Mark Sheppard) Date: Wed, 16 Jul 2014 19:09:11 +0100 Subject: RFR: JDK-8040810 - Uninitialised memory in jdk/src/windows/native/java/net: net_util_md.c, TwoStacksPlainSocketImpl.c, TwoStacksPlainDatagramSocketImpl.c, DualStackPlainSocketImpl.c, DualStackPlainDatagramSocketImpl.c In-Reply-To: <53C6B637.9010809@oracle.com> References: <53C6AFAF.80108@oracle.com> <53C6B637.9010809@oracle.com> Message-ID: <53C6BFC7.5010707@oracle.com> HI Alan, thanks for the response ... yes, rv = 0 is isn't strictly necessary (in for a penny, in for a pound approach) regards Mark On 16/07/2014 18:28, Alan Bateman wrote: > On 16/07/2014 18:00, Mark Sheppard wrote: >> Hi >> please oblige and review the following changes >> >> http://cr.openjdk.java.net/~msheppar/8040810/webrev/ >> >> which address the issue raised in >> >> https://bugs.openjdk.java.net/browse/JDK-8040810 >> >> resulting from static code analysis. >> >> these changes explicitly initialize local function variables, which >> are in the main >> out parameters to other function calls and hence are set within the >> called function. >> It can be reasonably argued that the initialization is unnecessary, >> but current coding >> guidance is to perform the initialization >> > I assume in NET_Bind that it isn't necessary to initialize rv, same > thing in NET_SocketClose. Otherwise looks okay. > > -Alan > From mark.sheppard at oracle.com Thu Jul 17 11:01:22 2014 From: mark.sheppard at oracle.com (Mark Sheppard) Date: Thu, 17 Jul 2014 12:01:22 +0100 Subject: RFR: JDK-8050922 - add additional diagnostic to java/net/MulticastSocket/TestInterfaces.java Message-ID: <53C7AD01.3090906@oracle.com> Hi, please oblige and review the following diagnostic output addition to the test java/net/MulticastSocket/TestInterfaces.java http://cr.openjdk.java.net/~msheppar/8050922/webrev/ for the task https://bugs.openjdk.java.net/browse/JDK-8050922 which will assist in diagnosing an intermittent failing test https://bugs.openjdk.java.net/browse/JDK-8041677 the addition displays the interface details, of the failing scenario, to std err regards Mark From chris.hegarty at oracle.com Thu Jul 17 12:57:43 2014 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 17 Jul 2014 13:57:43 +0100 Subject: RFR: JDK-8050922 - add additional diagnostic to java/net/MulticastSocket/TestInterfaces.java In-Reply-To: <53C7AD01.3090906@oracle.com> References: <53C7AD01.3090906@oracle.com> Message-ID: <05043DF2-1ED2-4B10-BA15-E1BBB60AC935@oracle.com> Looks ok to me Mark, and inline with what we have done to other similar tests. -Chris. On 17 Jul 2014, at 12:01, Mark Sheppard wrote: > Hi, > please oblige and review the following diagnostic output addition to > the test java/net/MulticastSocket/TestInterfaces.java > > http://cr.openjdk.java.net/~msheppar/8050922/webrev/ > > for the task > > https://bugs.openjdk.java.net/browse/JDK-8050922 > > which will assist in diagnosing an intermittent failing test > https://bugs.openjdk.java.net/browse/JDK-8041677 > > the addition displays the interface details, of the failing scenario, to std err > > regards > Mark From rieberandreas at gmail.com Thu Jul 17 18:03:49 2014 From: rieberandreas at gmail.com (Andreas Rieber) Date: Thu, 17 Jul 2014 20:03:49 +0200 Subject: JDK-6797318: Undeclared IAE thrown from HttpURLConnection.connect for some URLs Message-ID: <53C81005.4070400@gmail.com> Hi, i would like to contribute a patch for this old issue. The suggested fix in the issue looks right but instead of the IOE it would simply connect to localhost. The same wrong behaviour can be reproduced with ProxySelector.setDefault(null). I also checked other protocols and http, https and ftp are affected. The hostname is not checked currently and the used InetAddress.getAllByName(host) just returns the "loopbackAdress" if hostname is empty. Running the test under security manager got me a StringIndexOutOfBoundsException from the URLPermission/HostPortrange, where the hostname is not checked for empty string. With the test i try to cover all cases. Simple connect, with no ProxySelector, wrong redirect URL and all with security manager. Bug: https://bugs.openjdk.java.net/browse/JDK-6797318 Bug also fixed with no additional change (test included): https://bugs.openjdk.java.net/browse/JDK-6563286 Webrev: http://cr.openjdk.java.net/~arieber/6797318/webrev.00/ I did run jtreg */net */security on Ubuntu 14, Solaris 11 and Windows 8.1. Sponsor required as usually. cheers Andreas From peter.levart at gmail.com Wed Jul 23 08:29:54 2014 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 23 Jul 2014 10:29:54 +0200 Subject: RFR: JDK-8051713 - URL constructor/equals/hashCode/sameFile/openConnection synchronization issues In-Reply-To: <53BFFE93.9010103@gmail.com> References: <53BFFE93.9010103@gmail.com> Message-ID: <53CF7282.8010206@gmail.com> I created an issue for this: https://bugs.openjdk.java.net/browse/JDK-8051713 The proposed patch is still the following: http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/webrev.01/ Regards, Peter On 07/11/2014 05:11 PM, Peter Levart wrote: > Hi, > > java.net.URL is supposed to behave as an immutable object, so URL > instances can be shared among threads and among parts of code without > fear that they will be modified. URL class has an unusual way to > achieve this (or at least it tries to). Partly because of the design > which uses: > > - URL constructor(s) that take a 'spec' String to be parsed into URL > object > - parsing is delegated to various URLStreamHandler(s) which are chosen > in the URL constructor (depending on the protocol used in URL string > to be parsed) > > An unitialized URL instance (this) is passed from constructor to the > chosen URLStreamHandler which has the responsibility to parse the > string and set back the fields that hold various parts of URL object > being constructed. Consequently, these fields can not be declared as > final, as definite assignment analysis doesn't cross method borders. > It is therefore illegal to unsafely publish URL instances (via data > races) to non-constructing threads because they can appear not fully > initialized. Nevertheless URL, with the help of various > URLStreamHandler implementations, tries hard to make URL appear stable > at least where it is required to be stable. For example: > URL.hashCode() is (almost) stable even if URL instance is unsafely > published. This is achieved by making hashCode() synchronized and > cache the result. At least one way of constructing URLs - constructors > that take 'spec' String to be parsed - is also making sure that > hashCode is computed from fully initialized fields, as parsing is > delegated to URLStreamHandler which uses package-private URL.set() > method to set back the parsed fields and the set() method is also > synchronized. But making URL appear stable even though it is published > unsafely doesn't seem to be the primary concern of URL > synchronization. Other public URL constructors that take individual > URL parts and don't delegate parsing to URLStreamHandler but set > fields directly (not via set() method), are not synchronized. > > Primary concern of synchronization in URL appears to be driven from > the fact that some URL operations like hasCode(), equals(), sameFile() > and openConnection() read multiple URL fields and URL.set() which can > be called from custom URLStreamHandler at any time (although this is > not it's purpose - it should only call-back while parsing/constructing > the URL) can set those fields. And those multi-field operations would > like to see a "snapshot" of field values that is consistent. But > synchronization that is performed to achieve that is questionable. > Might be that in Java 1.0 times the JVM implementation assumptions > were different and synchronization was correct, but nowadays Java > memory model makes them invalid. > > URL.hasCode() apears to be the only method properly synchronized which > makes it almost stable (doesn't return different results over time) > but even hashCode() has a subtle bug or two. The initializer for > hashCode field sets it to value -1 which represents "not yet computed" > state. If URL is published unsafely, hashCode() method could see the > "default" value of 0, which would be returned. A later call to > hashCode() would see value -1 which would trigger computation and a > different value would be returned. The other subtle bug is a > relatively improbable event that hashCode computation results in value > -1 which means "not yet computed". This can be seen as performance > glitch (as hashCode will never be cached for such URL instance) or an > issue which makes hashCode unstable for one of the reasons why > equals() is unstable too (see below). > > If URL.hasCode() method is almost stable (doesn't return different > results over time) and at least one way of URL construction makes sure > that hashCode is also calculated from fully initialized parts, then > URL.equals() and other methods that delegate to URLStreamHandler are a > special story. URL.equals() can't be synchronized on URL instance, > because it would have to be synchronized on both URL instances that > are being compared and this is prone do dead-locks. Imagine: > > thread1: url1.equals(url2) > thread2: url2.equals(url1) > > So equals() chooses to not be synchronized and therefore risks not > being stable if URL instances are published unsafely. But it > nevertheless uses synchronization. equals() delegates it's work to the > 1st URL's URLStreamHandler which synchronizes on itself when > calculating and caching the InetAddress of each individual URL's host > name. InetAddress (if resolvable) is used in preference to host name > for comparison (and also in hashCode()). URL.equals() risks not being > stable for the following reasons: > > - URL instances published unsafely can appear not fully initialized to > equals() even though they were constructed with constructors that > delegate parsing to URLStreamHandler(s) which use synchronized > URL.set() to set the fields, because URL.equals() is not synchronized. > > - URL.hostAddress that is calculated on demand and then cached on the > URL instance should help make equals() stable in the presence of > dynamic changes to host name -> IP address mapping, but caching is not > performed for unsuccessful resolving. Temporary name service outage > can make URL.equals() unstable. > > - URL.hostAddress caching is using the URLStreamHandler instance of > the 1st URL as a lock for synchronizing read/write of hostAddress of > both URLs being compared by equals(). But URLStreamHandler(s) of the > two URL instances need not be the same instance even though they are > for the same protocol. Imagine: > > URL url1 = new URL(null, "http://www.google.com/", handler1); > URL url2 = new URL(null, "http://www.google.com/", handler2); > ... > thread1: url1.equals(url2); > thread2: url2.equals(url1); > > Each thread could be using different instance of URLStreamHandler for > synchronization and could overwrite each other the cached hostAddress > on individual URLs. These hostAddress values could be different in the > presence of dynamic changes to host name -> IP address mapping and > could therefore make URL.equals() unstable. This is admittedly a very > unlikely scenario, but is theoretically possible. > > URL.sameHost() has exactly the same issues as URL.equals() as it only > makes one field comparison less. > > URL.openConnection() is a question in itself. It is delegated to > URLStreamHandler. Some URLStreamHandlers make it synchronized and > others not. Those that make it synchronized (on the URLStreamHandler > instance) do this for no apparent reason. This synchronization can't > help make URL fields stable for the time of openConnection() call > since URL.set() is using a different lock (the URL instance itself). > It only makes things worse since access to opening the connection to > the resources is serialized and this presents a bottleneck. > > I tried to fix all these issues and came up with the following patch > which I'm proposing: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/webrev.01/ > > > New JDK8 synchronization primitive: StampedLock is a perfect tool for > solving these issues as synchronization in URL is only necessary to > establish visibility of initialized state which doesn't change > afterwards or changes at most once (when computing hashCode). > StampedLock's tryOptimisticRead/validate is perfect for such > situations as it only presents a negligible overhead of two volatile > reads. The presented patch also contains an unrelated change which > replaces usage of Hashtable with ConcurrentHashMap for holding the > mapping from protocol to individual URLStreamhandler which makes for > better scallability of URL constructors. Combined with synchronization > scallability enhancements of InetAddress caching presented in an > earlier proposal, the net result is much better scalability of URL > constructor/equals/hashCode (executed by a JMH test on a 4-core > i7/Linux box): > > http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/URL.synchronization_bench_results.pdf > > > So with this patch java.net.URL could be treated as an > unsafe-publication-tolerable class (like String). Well, not entirely, > since getters for individual fields are still just unguarded normal > reads. But those reads are usually performed under guard of a > StampedLock when URL > equals/hashCode/sameFile/openConnection/toExternalForm() operations > are delegated to URLStreamHandler and therefore don't suffer from > (in)visibility of published state. Fields could be made volatile to > fix this if desired. > > I'm not entirely sure about why openConnection() in file: and mailto: > URLStreamHandlers is synchronized. I don't see a reason, so I removed > the synchronization. If there is a reason, please bring it forward. > > I ran the java/net jtreg tests with unpatched recent jdk9-dev and > patched (combined changes of URL and InetAddress caching) and get the > same score. Only the following 3 tests fail in both ocasions: > > JT Harness : Tests that failed > java/net/MulticastSocket/Promiscuous.java: Test for interference when > two sockets are bound to the same port but joined to different > multicast groups > java/net/MulticastSocket/SetLoopbackMode.java: Test > MulticastSocket.setLoopbackMode > java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken > on Linux > > They seem to not like my multicast configuration or something. All > other 258 tests pass. > > So what do you think? > > > Regards, Peter > From kubota.yuji at lab.ntt.co.jp Thu Jul 24 06:28:30 2014 From: kubota.yuji at lab.ntt.co.jp (KUBOTA Yuji) Date: Thu, 24 Jul 2014 15:28:30 +0900 Subject: Do not JDK-6967684 backport to JDK6? In-Reply-To: <20140716162015.GF2392@redhat.com> References: <53A798ED.3040905@lab.ntt.co.jp> <4E485579-2E8C-44D5-A7C2-82C8223C111F@azulsystems.com> <53BE5DE6.3030100@lab.ntt.co.jp> <20140716162015.GF2392@redhat.com> Message-ID: <53D0A78E.7050700@lab.ntt.co.jp> (2014/07/17 1:20 JST), Omair Majid wrote: > Bug and patch look straight-forward and correct. No objections from me. Thank you for your help! and sorry for delayed reply. I do not have any role of OpenJDK projects. So, can someone please backport of JDK-6967684 to JDK6? Best regards, KUBOTA Yuji. From rob.mckenna at oracle.com Thu Jul 24 20:54:15 2014 From: rob.mckenna at oracle.com (Rob McKenna) Date: Thu, 24 Jul 2014 21:54:15 +0100 Subject: RFR: 8031435: Ftp download does not work properly for ftp user without password Message-ID: <53D17277.4040900@oracle.com> Hi folks, Very simple fix to allow FtpURLConnection connect without a password in the url. (i.e. ftp://user at server/ as opposed to the current ftp://user:@server) http://cr.openjdk.java.net/~robm/8031435/webrev.01/ -Rob From chris.hegarty at oracle.com Thu Jul 24 21:18:29 2014 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 24 Jul 2014 22:18:29 +0100 Subject: RFR: 8031435: Ftp download does not work properly for ftp user without password In-Reply-To: <53D17277.4040900@oracle.com> References: <53D17277.4040900@oracle.com> Message-ID: Looks good to me Rob. -Chris > On 24 Jul 2014, at 21:54, Rob McKenna wrote: > > Hi folks, > > Very simple fix to allow FtpURLConnection connect without a password in the url. (i.e. ftp://user at server/ as opposed to the current ftp://user:@server) > > http://cr.openjdk.java.net/~robm/8031435/webrev.01/ > > -Rob > From chris.hegarty at oracle.com Fri Jul 25 12:53:13 2014 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 25 Jul 2014 13:53:13 +0100 Subject: RFR: JDK-8051713 - URL constructor/equals/hashCode/sameFile/openConnection synchronization issues In-Reply-To: <53CF7282.8010206@gmail.com> References: <53BFFE93.9010103@gmail.com> <53CF7282.8010206@gmail.com> Message-ID: <53D25339.9070704@oracle.com> Hi Peter, This is certainly a thorny issue, and I agree with the approach of using StampedLock. Some small comments / questions: 1) Why abuse fieldsLock in hostAddress(), rather than grabbing the writeLock ? 2) Does the setAccessible in readObject need to be in a doPriv? Also should this throw an InternalError/AssertionError if it fails? I'll pop you patch into our internal build and test system. -Chris. On 23/07/14 09:29, Peter Levart wrote: > I created an issue for this: > > https://bugs.openjdk.java.net/browse/JDK-8051713 > > The proposed patch is still the following: > > http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/webrev.01/ > > Regards, Peter > > > On 07/11/2014 05:11 PM, Peter Levart wrote: >> Hi, >> >> java.net.URL is supposed to behave as an immutable object, so URL >> instances can be shared among threads and among parts of code without >> fear that they will be modified. URL class has an unusual way to >> achieve this (or at least it tries to). Partly because of the design >> which uses: >> >> - URL constructor(s) that take a 'spec' String to be parsed into URL >> object >> - parsing is delegated to various URLStreamHandler(s) which are chosen >> in the URL constructor (depending on the protocol used in URL string >> to be parsed) >> >> An unitialized URL instance (this) is passed from constructor to the >> chosen URLStreamHandler which has the responsibility to parse the >> string and set back the fields that hold various parts of URL object >> being constructed. Consequently, these fields can not be declared as >> final, as definite assignment analysis doesn't cross method borders. >> It is therefore illegal to unsafely publish URL instances (via data >> races) to non-constructing threads because they can appear not fully >> initialized. Nevertheless URL, with the help of various >> URLStreamHandler implementations, tries hard to make URL appear stable >> at least where it is required to be stable. For example: >> URL.hashCode() is (almost) stable even if URL instance is unsafely >> published. This is achieved by making hashCode() synchronized and >> cache the result. At least one way of constructing URLs - constructors >> that take 'spec' String to be parsed - is also making sure that >> hashCode is computed from fully initialized fields, as parsing is >> delegated to URLStreamHandler which uses package-private URL.set() >> method to set back the parsed fields and the set() method is also >> synchronized. But making URL appear stable even though it is published >> unsafely doesn't seem to be the primary concern of URL >> synchronization. Other public URL constructors that take individual >> URL parts and don't delegate parsing to URLStreamHandler but set >> fields directly (not via set() method), are not synchronized. >> >> Primary concern of synchronization in URL appears to be driven from >> the fact that some URL operations like hasCode(), equals(), sameFile() >> and openConnection() read multiple URL fields and URL.set() which can >> be called from custom URLStreamHandler at any time (although this is >> not it's purpose - it should only call-back while parsing/constructing >> the URL) can set those fields. And those multi-field operations would >> like to see a "snapshot" of field values that is consistent. But >> synchronization that is performed to achieve that is questionable. >> Might be that in Java 1.0 times the JVM implementation assumptions >> were different and synchronization was correct, but nowadays Java >> memory model makes them invalid. >> >> URL.hasCode() apears to be the only method properly synchronized which >> makes it almost stable (doesn't return different results over time) >> but even hashCode() has a subtle bug or two. The initializer for >> hashCode field sets it to value -1 which represents "not yet computed" >> state. If URL is published unsafely, hashCode() method could see the >> "default" value of 0, which would be returned. A later call to >> hashCode() would see value -1 which would trigger computation and a >> different value would be returned. The other subtle bug is a >> relatively improbable event that hashCode computation results in value >> -1 which means "not yet computed". This can be seen as performance >> glitch (as hashCode will never be cached for such URL instance) or an >> issue which makes hashCode unstable for one of the reasons why >> equals() is unstable too (see below). >> >> If URL.hasCode() method is almost stable (doesn't return different >> results over time) and at least one way of URL construction makes sure >> that hashCode is also calculated from fully initialized parts, then >> URL.equals() and other methods that delegate to URLStreamHandler are a >> special story. URL.equals() can't be synchronized on URL instance, >> because it would have to be synchronized on both URL instances that >> are being compared and this is prone do dead-locks. Imagine: >> >> thread1: url1.equals(url2) >> thread2: url2.equals(url1) >> >> So equals() chooses to not be synchronized and therefore risks not >> being stable if URL instances are published unsafely. But it >> nevertheless uses synchronization. equals() delegates it's work to the >> 1st URL's URLStreamHandler which synchronizes on itself when >> calculating and caching the InetAddress of each individual URL's host >> name. InetAddress (if resolvable) is used in preference to host name >> for comparison (and also in hashCode()). URL.equals() risks not being >> stable for the following reasons: >> >> - URL instances published unsafely can appear not fully initialized to >> equals() even though they were constructed with constructors that >> delegate parsing to URLStreamHandler(s) which use synchronized >> URL.set() to set the fields, because URL.equals() is not synchronized. >> >> - URL.hostAddress that is calculated on demand and then cached on the >> URL instance should help make equals() stable in the presence of >> dynamic changes to host name -> IP address mapping, but caching is not >> performed for unsuccessful resolving. Temporary name service outage >> can make URL.equals() unstable. >> >> - URL.hostAddress caching is using the URLStreamHandler instance of >> the 1st URL as a lock for synchronizing read/write of hostAddress of >> both URLs being compared by equals(). But URLStreamHandler(s) of the >> two URL instances need not be the same instance even though they are >> for the same protocol. Imagine: >> >> URL url1 = new URL(null, "http://www.google.com/", handler1); >> URL url2 = new URL(null, "http://www.google.com/", handler2); >> ... >> thread1: url1.equals(url2); >> thread2: url2.equals(url1); >> >> Each thread could be using different instance of URLStreamHandler for >> synchronization and could overwrite each other the cached hostAddress >> on individual URLs. These hostAddress values could be different in the >> presence of dynamic changes to host name -> IP address mapping and >> could therefore make URL.equals() unstable. This is admittedly a very >> unlikely scenario, but is theoretically possible. >> >> URL.sameHost() has exactly the same issues as URL.equals() as it only >> makes one field comparison less. >> >> URL.openConnection() is a question in itself. It is delegated to >> URLStreamHandler. Some URLStreamHandlers make it synchronized and >> others not. Those that make it synchronized (on the URLStreamHandler >> instance) do this for no apparent reason. This synchronization can't >> help make URL fields stable for the time of openConnection() call >> since URL.set() is using a different lock (the URL instance itself). >> It only makes things worse since access to opening the connection to >> the resources is serialized and this presents a bottleneck. >> >> I tried to fix all these issues and came up with the following patch >> which I'm proposing: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/webrev.01/ >> >> >> New JDK8 synchronization primitive: StampedLock is a perfect tool for >> solving these issues as synchronization in URL is only necessary to >> establish visibility of initialized state which doesn't change >> afterwards or changes at most once (when computing hashCode). >> StampedLock's tryOptimisticRead/validate is perfect for such >> situations as it only presents a negligible overhead of two volatile >> reads. The presented patch also contains an unrelated change which >> replaces usage of Hashtable with ConcurrentHashMap for holding the >> mapping from protocol to individual URLStreamhandler which makes for >> better scallability of URL constructors. Combined with synchronization >> scallability enhancements of InetAddress caching presented in an >> earlier proposal, the net result is much better scalability of URL >> constructor/equals/hashCode (executed by a JMH test on a 4-core >> i7/Linux box): >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/URL.synchronization_bench_results.pdf >> >> >> So with this patch java.net.URL could be treated as an >> unsafe-publication-tolerable class (like String). Well, not entirely, >> since getters for individual fields are still just unguarded normal >> reads. But those reads are usually performed under guard of a >> StampedLock when URL >> equals/hashCode/sameFile/openConnection/toExternalForm() operations >> are delegated to URLStreamHandler and therefore don't suffer from >> (in)visibility of published state. Fields could be made volatile to >> fix this if desired. >> >> I'm not entirely sure about why openConnection() in file: and mailto: >> URLStreamHandlers is synchronized. I don't see a reason, so I removed >> the synchronization. If there is a reason, please bring it forward. >> >> I ran the java/net jtreg tests with unpatched recent jdk9-dev and >> patched (combined changes of URL and InetAddress caching) and get the >> same score. Only the following 3 tests fail in both ocasions: >> >> JT Harness : Tests that failed >> java/net/MulticastSocket/Promiscuous.java: Test for interference when >> two sockets are bound to the same port but joined to different >> multicast groups >> java/net/MulticastSocket/SetLoopbackMode.java: Test >> MulticastSocket.setLoopbackMode >> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken >> on Linux >> >> They seem to not like my multicast configuration or something. All >> other 258 tests pass. >> >> So what do you think? >> >> >> Regards, Peter >> > From peter.levart at gmail.com Fri Jul 25 15:12:42 2014 From: peter.levart at gmail.com (Peter Levart) Date: Fri, 25 Jul 2014 17:12:42 +0200 Subject: RFR: JDK-8051713 - URL constructor/equals/hashCode/sameFile/openConnection synchronization issues In-Reply-To: <53D25339.9070704@oracle.com> References: <53BFFE93.9010103@gmail.com> <53CF7282.8010206@gmail.com> <53D25339.9070704@oracle.com> Message-ID: <53D273EA.5090203@gmail.com> Hi Chris, Thanks for looking into this... On 07/25/2014 02:53 PM, Chris Hegarty wrote: > Hi Peter, > > This is certainly a thorny issue, and I agree with the approach of > using StampedLock. > > Some small comments / questions: > > 1) Why abuse fieldsLock in hostAddress(), rather than grabbing the > writeLock ? URL.hostAddress() is called-back from URLStreamHandler when URL is delegating work for URL.equals, URL.hashCode and URL.sameFile methods to it. These methods hold readLock (URL.hashCode can also hold writeLock) while calling URLStreamHandler to do the work. StampedLock is not reentrant. Instead of abusing fieldsLock object as a monitor lock for caching hostAddress(), a CAS can be used to set hostAddress and still provide a stable value (in presence of dynamic name service mapping changes). > > 2) Does the setAccessible in readObject need to be in a doPriv? > Also should this throw an InternalError/AssertionError if it fails? You're absolutely right! It needs to be in doPriv. Here's new webrev that wraps setAccessible in doPriv and uses CAS instead of mutex for hostAddress caching. http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/webrev.02/ > > I'll pop you patch into our internal build and test system. Great. It would also be interesting to see if there are any performance differences. Do you have any performance tests at Oracle that include URL comparison? Differences might most obviously show-up when this patch is combined with InetAddress caching enhancements: http://cr.openjdk.java.net/~plevart/jdk9-dev/InetAddress.Cache/webrev.03/ Regards, Peter > > -Chris. > > On 23/07/14 09:29, Peter Levart wrote: >> I created an issue for this: >> >> https://bugs.openjdk.java.net/browse/JDK-8051713 >> >> The proposed patch is still the following: >> >> http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/webrev.01/ >> >> >> Regards, Peter >> >> >> On 07/11/2014 05:11 PM, Peter Levart wrote: >>> Hi, >>> >>> java.net.URL is supposed to behave as an immutable object, so URL >>> instances can be shared among threads and among parts of code without >>> fear that they will be modified. URL class has an unusual way to >>> achieve this (or at least it tries to). Partly because of the design >>> which uses: >>> >>> - URL constructor(s) that take a 'spec' String to be parsed into URL >>> object >>> - parsing is delegated to various URLStreamHandler(s) which are chosen >>> in the URL constructor (depending on the protocol used in URL string >>> to be parsed) >>> >>> An unitialized URL instance (this) is passed from constructor to the >>> chosen URLStreamHandler which has the responsibility to parse the >>> string and set back the fields that hold various parts of URL object >>> being constructed. Consequently, these fields can not be declared as >>> final, as definite assignment analysis doesn't cross method borders. >>> It is therefore illegal to unsafely publish URL instances (via data >>> races) to non-constructing threads because they can appear not fully >>> initialized. Nevertheless URL, with the help of various >>> URLStreamHandler implementations, tries hard to make URL appear stable >>> at least where it is required to be stable. For example: >>> URL.hashCode() is (almost) stable even if URL instance is unsafely >>> published. This is achieved by making hashCode() synchronized and >>> cache the result. At least one way of constructing URLs - constructors >>> that take 'spec' String to be parsed - is also making sure that >>> hashCode is computed from fully initialized fields, as parsing is >>> delegated to URLStreamHandler which uses package-private URL.set() >>> method to set back the parsed fields and the set() method is also >>> synchronized. But making URL appear stable even though it is published >>> unsafely doesn't seem to be the primary concern of URL >>> synchronization. Other public URL constructors that take individual >>> URL parts and don't delegate parsing to URLStreamHandler but set >>> fields directly (not via set() method), are not synchronized. >>> >>> Primary concern of synchronization in URL appears to be driven from >>> the fact that some URL operations like hasCode(), equals(), sameFile() >>> and openConnection() read multiple URL fields and URL.set() which can >>> be called from custom URLStreamHandler at any time (although this is >>> not it's purpose - it should only call-back while parsing/constructing >>> the URL) can set those fields. And those multi-field operations would >>> like to see a "snapshot" of field values that is consistent. But >>> synchronization that is performed to achieve that is questionable. >>> Might be that in Java 1.0 times the JVM implementation assumptions >>> were different and synchronization was correct, but nowadays Java >>> memory model makes them invalid. >>> >>> URL.hasCode() apears to be the only method properly synchronized which >>> makes it almost stable (doesn't return different results over time) >>> but even hashCode() has a subtle bug or two. The initializer for >>> hashCode field sets it to value -1 which represents "not yet computed" >>> state. If URL is published unsafely, hashCode() method could see the >>> "default" value of 0, which would be returned. A later call to >>> hashCode() would see value -1 which would trigger computation and a >>> different value would be returned. The other subtle bug is a >>> relatively improbable event that hashCode computation results in value >>> -1 which means "not yet computed". This can be seen as performance >>> glitch (as hashCode will never be cached for such URL instance) or an >>> issue which makes hashCode unstable for one of the reasons why >>> equals() is unstable too (see below). >>> >>> If URL.hasCode() method is almost stable (doesn't return different >>> results over time) and at least one way of URL construction makes sure >>> that hashCode is also calculated from fully initialized parts, then >>> URL.equals() and other methods that delegate to URLStreamHandler are a >>> special story. URL.equals() can't be synchronized on URL instance, >>> because it would have to be synchronized on both URL instances that >>> are being compared and this is prone do dead-locks. Imagine: >>> >>> thread1: url1.equals(url2) >>> thread2: url2.equals(url1) >>> >>> So equals() chooses to not be synchronized and therefore risks not >>> being stable if URL instances are published unsafely. But it >>> nevertheless uses synchronization. equals() delegates it's work to the >>> 1st URL's URLStreamHandler which synchronizes on itself when >>> calculating and caching the InetAddress of each individual URL's host >>> name. InetAddress (if resolvable) is used in preference to host name >>> for comparison (and also in hashCode()). URL.equals() risks not being >>> stable for the following reasons: >>> >>> - URL instances published unsafely can appear not fully initialized to >>> equals() even though they were constructed with constructors that >>> delegate parsing to URLStreamHandler(s) which use synchronized >>> URL.set() to set the fields, because URL.equals() is not synchronized. >>> >>> - URL.hostAddress that is calculated on demand and then cached on the >>> URL instance should help make equals() stable in the presence of >>> dynamic changes to host name -> IP address mapping, but caching is not >>> performed for unsuccessful resolving. Temporary name service outage >>> can make URL.equals() unstable. >>> >>> - URL.hostAddress caching is using the URLStreamHandler instance of >>> the 1st URL as a lock for synchronizing read/write of hostAddress of >>> both URLs being compared by equals(). But URLStreamHandler(s) of the >>> two URL instances need not be the same instance even though they are >>> for the same protocol. Imagine: >>> >>> URL url1 = new URL(null, "http://www.google.com/", handler1); >>> URL url2 = new URL(null, "http://www.google.com/", handler2); >>> ... >>> thread1: url1.equals(url2); >>> thread2: url2.equals(url1); >>> >>> Each thread could be using different instance of URLStreamHandler for >>> synchronization and could overwrite each other the cached hostAddress >>> on individual URLs. These hostAddress values could be different in the >>> presence of dynamic changes to host name -> IP address mapping and >>> could therefore make URL.equals() unstable. This is admittedly a very >>> unlikely scenario, but is theoretically possible. >>> >>> URL.sameHost() has exactly the same issues as URL.equals() as it only >>> makes one field comparison less. >>> >>> URL.openConnection() is a question in itself. It is delegated to >>> URLStreamHandler. Some URLStreamHandlers make it synchronized and >>> others not. Those that make it synchronized (on the URLStreamHandler >>> instance) do this for no apparent reason. This synchronization can't >>> help make URL fields stable for the time of openConnection() call >>> since URL.set() is using a different lock (the URL instance itself). >>> It only makes things worse since access to opening the connection to >>> the resources is serialized and this presents a bottleneck. >>> >>> I tried to fix all these issues and came up with the following patch >>> which I'm proposing: >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/webrev.01/ >>> >>> >>> >>> New JDK8 synchronization primitive: StampedLock is a perfect tool for >>> solving these issues as synchronization in URL is only necessary to >>> establish visibility of initialized state which doesn't change >>> afterwards or changes at most once (when computing hashCode). >>> StampedLock's tryOptimisticRead/validate is perfect for such >>> situations as it only presents a negligible overhead of two volatile >>> reads. The presented patch also contains an unrelated change which >>> replaces usage of Hashtable with ConcurrentHashMap for holding the >>> mapping from protocol to individual URLStreamhandler which makes for >>> better scallability of URL constructors. Combined with synchronization >>> scallability enhancements of InetAddress caching presented in an >>> earlier proposal, the net result is much better scalability of URL >>> constructor/equals/hashCode (executed by a JMH test on a 4-core >>> i7/Linux box): >>> >>> http://cr.openjdk.java.net/~plevart/jdk9-dev/URL.synchronization/URL.synchronization_bench_results.pdf >>> >>> >>> >>> So with this patch java.net.URL could be treated as an >>> unsafe-publication-tolerable class (like String). Well, not entirely, >>> since getters for individual fields are still just unguarded normal >>> reads. But those reads are usually performed under guard of a >>> StampedLock when URL >>> equals/hashCode/sameFile/openConnection/toExternalForm() operations >>> are delegated to URLStreamHandler and therefore don't suffer from >>> (in)visibility of published state. Fields could be made volatile to >>> fix this if desired. >>> >>> I'm not entirely sure about why openConnection() in file: and mailto: >>> URLStreamHandlers is synchronized. I don't see a reason, so I removed >>> the synchronization. If there is a reason, please bring it forward. >>> >>> I ran the java/net jtreg tests with unpatched recent jdk9-dev and >>> patched (combined changes of URL and InetAddress caching) and get the >>> same score. Only the following 3 tests fail in both ocasions: >>> >>> JT Harness : Tests that failed >>> java/net/MulticastSocket/Promiscuous.java: Test for interference when >>> two sockets are bound to the same port but joined to different >>> multicast groups >>> java/net/MulticastSocket/SetLoopbackMode.java: Test >>> MulticastSocket.setLoopbackMode >>> java/net/MulticastSocket/Test.java: IPv4 and IPv6 multicasting broken >>> on Linux >>> >>> They seem to not like my multicast configuration or something. All >>> other 258 tests pass. >>> >>> So what do you think? >>> >>> >>> Regards, Peter >>> >> From peter.firmstone at zeus.net.au Sat Jul 26 09:57:51 2014 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Sat, 26 Jul 2014 19:57:51 +1000 Subject: State of Serialization In-Reply-To: <53CD2D5B.9020006@oracle.com> References: <53CBA090.4070303@zeus.net.au> <53CD2D5B.9020006@oracle.com> Message-ID: <53D37B9F.4060008@zeus.net.au> I'm somewhat time poor at present, it's a draft. The intent is to open a discussion to: 1. Address security issues 2. Make Serializable2 api completely public and backward compatible with the existing serial stream protocol. 3. Allow Serializable to be switched off via a jvm switch On 22/07/2014 1:10 AM, Tom Hawtin wrote: > On 20/07/2014 11:57, Peter Firmstone wrote: > >> Since private methods are only be called by the ObjectOutputStream / >> ObjectInputStream, during de-serialisation, subclass are not responsible >> for calling these methods, hence subclass ProtectionDomain's are not >> present in the Thread's AccessControlContext and as such are missing >> from security checks, this is why it's currently essential for classes >> to ensure that de-serialisation isn't performed in a privileged context. > > It's more complicated than that. Even final serialisable classes may > have security checks. You've highlited an issue with security on the Java platform, it's possible for an object to escape after a security check has been performed by a constructor. It's bad practise for a Serializable object to have a security check from within it's constructor, however a Serializable object may extend an object with a zero argument constructor, such as ClassLoader. The trick for an attacker is to deserialize within a privileged context, even when a ClassCastException occurs, it does so after the object has been created, if an attacker can get a reference to it before it's garbage collected... Prior to calling a constructor, if a class hasn't been loaded, class static initializers are also called. Li Gong proposed the method guard pattern, page 176, inside Java 2 Platform Security, second edition, this fixes the issue of objects that potentially escape, however performance with the existing security infrastructure is a problem, that can be easily fixed, but that's another topic (on security-dev). What we really need to do is enable an administrator to limit classes allowed to be Serialized, via a configuration file, with Java 9, we'll need to know which module too. > >> To improve security, it would be preferable to use a deserialization >> constructor, required to be called by subclasses in the class >> hierarchies, placing their ProtectionDomains in the stack context, >> avoiding a number of security issues. Another benefit is the ability to >> use final fields, while checking invariants during construction. > > Certainly it would be better have a mechanism that better fitted in > with non-serialisation mechanisms. Addressing this without unraveling > too much when pulling on a thread, and without increasing complexity > of corner cases, is non-trivial. By providing a new interface, using only public api, but also the same stream format, an upgrade and backward compatible path can be provided. This allows both interfaces to co-exist and for corner cases to remain supported within Serializable, however it would also allow administrators to switch off Serializable and use only Serializable2. Complicated? How so? Regards, Peter. From kubota.yuji at lab.ntt.co.jp Mon Jul 28 00:30:49 2014 From: kubota.yuji at lab.ntt.co.jp (KUBOTA Yuji) Date: Mon, 28 Jul 2014 09:30:49 +0900 Subject: Do not JDK-6967684 backport to JDK6? In-Reply-To: <20140725144605.GA2422@redhat.com> References: <53A798ED.3040905@lab.ntt.co.jp> <4E485579-2E8C-44D5-A7C2-82C8223C111F@azulsystems.com> <53BE5DE6.3030100@lab.ntt.co.jp> <20140716162015.GF2392@redhat.com> <53D0A78E.7050700@lab.ntt.co.jp> <20140725144605.GA2422@redhat.com> Message-ID: <53D599B9.3060904@lab.ntt.co.jp> (2014/07/25 23:46, JST), Omair Majid wrote: > I have pushed it: > http://hg.openjdk.java.net/jdk6/jdk6/jdk/rev/2d334ea351f0 Many thanks for your time and help!! Cheers, Yuji KUBOTA. From peter.firmstone at zeus.net.au Mon Jul 28 09:28:29 2014 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Mon, 28 Jul 2014 19:28:29 +1000 Subject: State of Serialization In-Reply-To: <53D37B9F.4060008@zeus.net.au> References: <53CBA090.4070303@zeus.net.au> <53CD2D5B.9020006@oracle.com> <53D37B9F.4060008@zeus.net.au> Message-ID: <53D617BD.3030606@zeus.net.au> Updated files attached, including update to State of Serialization draft. Note that the attached source code can be provided under the Oracle agreement. I'm a little time poor, so please contribute suggestions etc. Regards, Peter. On 26/07/2014 7:57 PM, Peter Firmstone wrote: > I'm somewhat time poor at present, it's a draft. > > The intent is to open a discussion to: > > 1. Address security issues > 2. Make Serializable2 api completely public and backward compatible > with the existing serial stream protocol. > 3. Allow Serializable to be switched off via a jvm switch > > > On 22/07/2014 1:10 AM, Tom Hawtin wrote: >> On 20/07/2014 11:57, Peter Firmstone wrote: >> >>> Since private methods are only be called by the ObjectOutputStream / >>> ObjectInputStream, during de-serialisation, subclass are not >>> responsible >>> for calling these methods, hence subclass ProtectionDomain's are not >>> present in the Thread's AccessControlContext and as such are missing >>> from security checks, this is why it's currently essential for classes >>> to ensure that de-serialisation isn't performed in a privileged >>> context. >> >> It's more complicated than that. Even final serialisable classes may >> have security checks. > > You've highlited an issue with security on the Java platform, it's > possible for an object to escape after a security check has been > performed by a constructor. > > It's bad practise for a Serializable object to have a security check > from within it's constructor, however a Serializable object may extend > an object with a zero argument constructor, such as ClassLoader. > > The trick for an attacker is to deserialize within a privileged > context, even when a ClassCastException occurs, it does so after the > object has been created, if an attacker can get a reference to it > before it's garbage collected... > > Prior to calling a constructor, if a class hasn't been loaded, class > static initializers are also called. > > Li Gong proposed the method guard pattern, page 176, inside Java 2 > Platform Security, second edition, this fixes the issue of objects > that potentially escape, however performance with the existing > security infrastructure is a problem, that can be easily fixed, but > that's another topic (on security-dev). > > What we really need to do is enable an administrator to limit classes > allowed to be Serialized, via a configuration file, with Java 9, we'll > need to know which module too. > >> >>> To improve security, it would be preferable to use a deserialization >>> constructor, required to be called by subclasses in the class >>> hierarchies, placing their ProtectionDomains in the stack context, >>> avoiding a number of security issues. Another benefit is the ability to >>> use final fields, while checking invariants during construction. >> >> Certainly it would be better have a mechanism that better fitted in >> with non-serialisation mechanisms. Addressing this without unraveling >> too much when pulling on a thread, and without increasing complexity >> of corner cases, is non-trivial. > > By providing a new interface, using only public api, but also the same > stream format, an upgrade and backward compatible path can be provided. > > This allows both interfaces to co-exist and for corner cases to remain > supported within Serializable, however it would also allow > administrators to switch off Serializable and use only Serializable2. > > Complicated? How so? > > Regards, > > Peter. -------------- next part -------------- A non-text attachment was scrubbed... Name: State of Serialization.odt Type: application/vnd.oasis.opendocument.text Size: 17772 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Portable.java URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: PortableFactory.java URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: PortableObjectInputStream.java URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: PortableObjectOutputStream.java URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: PortablePermission.java URL: From peter.firmstone at zeus.net.au Mon Jul 28 09:51:17 2014 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Mon, 28 Jul 2014 19:51:17 +1000 Subject: State of Serialization In-Reply-To: <53D617BD.3030606@zeus.net.au> References: <53CBA090.4070303@zeus.net.au> <53CD2D5B.9020006@oracle.com> <53D37B9F.4060008@zeus.net.au> <53D617BD.3030606@zeus.net.au> Message-ID: <53D61D15.5070402@zeus.net.au> On 28/07/2014 7:28 PM, Peter Firmstone wrote: > Updated files attached, including update to State of Serialization draft. > For those who didn't get the attachment: *State of Java Serialization - Draft* Introduction The Java Serialization framework enables object state to be frozen, stored to disk or transferred over a network and unfrozen again into objects. While Java's Serialization capabilities are arguably more sophisticated than most at reconstructing complex object relationships, the ?magic? nothing to do, marker interface, Serializable is problematic. Due to the complexity of serialising state from an object with Serializable superclasses in inheritance hierarchies, private methods were chosen, allowing objects to write and read objects to and from streams. Each class in an Object's inheritance heirarchy that implements or inherits Serializable must be able to write out and read in serialized state, private methods cannot be overridden or called by subclasses, nor is their implementation enforced by the Java language syntax, hence Serializable is a marker interface only. Background Serialization was introduced in Java 1.1. The marker interface Serializable is problematic since implementation of its methods are optional. Developers can make objects serializable by simply declaring implements Serializable and providing a default zero argument constructor. Since private methods are only be called by the ObjectOutputStream / ObjectInputStream, during de-serialisation, subclass are not responsible for calling these methods, hence subclass ProtectionDomain's are not present in the Thread's AccessControlContext and as such are missing from security checks, this is why it's currently essential for classes to ensure that de-serialisation isn't performed in a privileged context. To improve security, it would be preferable to use a deserialization constructor, required to be called by subclasses in the class hierarchies, placing their ProtectionDomains in the stack context, avoiding a number of security issues. Another benefit is the ability to use final fields, while checking invariants during construction. Challenges Cyclic data structures Presently an ObjectInputStream creates a new Object using a zero arg constructor, then populates its fields from superclass to child class. It's possible to obtain a reference to an object before fields have been populated, allowing reconstruction of circular relationships between objects. Cyclic data exists because either during construction the ?this? reference has escaped to a trusted object, or after construction it has been passed via a mutating method. Options for repopulating cyclic data using deserialization constructors Option 1, deliberately allow ?this? to escape during construction In order for a deserialization constructor to populate a circular data structure with its reference, the constructor must accept a parameter object that reconstructs the circular data structure and the new object must be prepared to pass its ?this? reference to the parameter object during construction. Issues with allowing ?this? to escape during construction When ?this? escapes to reconstruct a circular data relationship during construction it causes the following issues: 1. It is not possible to us a static method to create field objects, validate invariants and throw an Exception, before the Object superclass constructor has been invoked to avoid finalizer attacks. 2. Defensive copying, if the ?this? reference is defensively copied during construction of a new field from the OIS, the state copied may only be partially constructed. 3. The ?this? reference may be copied by using a hand crafted object stream, prior to a constructor completing. Option 2 Create a new Serialization mechanism, preserve and limit existing Serializable classes. 1. Existing Serialization can be limited to classes known to be safe using a mechanism similar to look ahead serialization. http://www.ibm.com/developerworks/java/library/se-lookahead/index.html 2. Provide a configuration file allowing administrators to define trusted Serializable classes. 3. A working proposed alternative that provides ?most of the abilities of Java Serialization? without the security risks has been attached that can be used as a baseline. o Constructors are the only method of creating objects o Objects constructors are called with default minimal privileges (only grants in policy files made to all). o Objects are responsible for wiring up their own circular relationships. From amanda.jiang at oracle.com Mon Jul 28 22:25:31 2014 From: amanda.jiang at oracle.com (Amanda Jiang) Date: Mon, 28 Jul 2014 15:25:31 -0700 Subject: RFR 8047031: Need new tests to check socket permission Message-ID: <53D6CDDB.7020705@oracle.com> Hi All, Could you please review 1 new test to be added for SocketPermission. New test is added to check socket permissions, for instance - - java.net.SocketPermission with "connect", "resolve", "accept", "listen" with Socket, DatagramSocket, MulticastSocket etc. JDK Issue: https://bugs.openjdk.java.net/browse/JDK-8047031 WebRev: http://cr.openjdk.java.net/~rhalade/8047031/webrev.00/ Thanks, Amanda