Hi all, JDK7 is using zlib 1.2.3 (which was added to JDK7 back in 2009.) Zlib's latest version is 1.2.5 - is there any expectation to move to 1.2.5 in JDK7? It seems a real shame to ship JDK7 with a version of zlib that is so out of date. More than happy to help contribute towards making this happen. I would like to know if its already on someone's radar though or to understand why this is a mad idea :-) Cheers Steve
Steve Poole wrote:
Hi all,
JDK7 is using zlib 1.2.3 (which was added to JDK7 back in 2009.)
Zlib's latest version is 1.2.5 - is there any expectation to move to 1.2.5 in JDK7? It seems a real shame to ship JDK7 with a version of zlib that is so out of date.
More than happy to help contribute towards making this happen. I would like to know if its already on someone's radar though or to understand why this is a mad idea :-)
Cheers
Steve
I don't think it's on anyone's radar. Sherman did the last update, bringing us up from a patched 1.1.3 to 1.2.3. The main thing with zlib updates is testing and bake time. Have you done any testing with 1.2.5? While there are some good reasons listed in the ChangeLog, I'm curious if they translate into any measurable improvement with the zip APIs. On the testing it would be interesting to check if 1.2.5 is used by any of the distros as I think (but might be wrong) that OpenJDK/IcedTea6 project have changes to use /usr/lib/libzip.so rather than the snapshot in the jdk repository and so might already have experience using the latest version. -Alan
On 15 February 2011 13:57, Alan Bateman <Alan.Bateman@oracle.com> wrote:
Steve Poole wrote:
Hi all, JDK7 is using zlib 1.2.3 (which was added to JDK7 back in 2009.) Zlib's latest version is 1.2.5 - is there any expectation to move to 1.2.5 in JDK7? It seems a real shame to ship JDK7 with a version of zlib that is so out of date. More than happy to help contribute towards making this happen. I would like to know if its already on someone's radar though or to understand why this is a mad idea :-)
Cheers
Steve
I don't think it's on anyone's radar. Sherman did the last update, bringing us up from a patched 1.1.3 to 1.2.3. The main thing with zlib updates is testing and bake time. Have you done any testing with 1.2.5? While there are some good reasons listed in the ChangeLog, I'm curious if they translate into any measurable improvement with the zip APIs. On the testing it would be interesting to check if 1.2.5 is used by any of the distros as I think (but might be wrong) that OpenJDK/IcedTea6 project have changes to use /usr/lib/libzip.so rather than the snapshot in the jdk repository and so might already have experience using the latest version.
-Alan
Yes, IcedTea uses system libraries for everything bar LCMS, where local changes in OpenJDK mean we are still forced to use the in-tree version. There hasn't been any success upstreaming these changes, though I haven't looked at LCMS 2.x. So yes, that means we use the system zlib (currently 1.2.5 here), jpeg (8c) and png (1.4.5). 1.10 will also finally remove the static linking of libstdc++ and libgcc. This was done back in the early days of IcedTea as part of preparing OpenJDK for distro packaging, allowing these libraries to receive timely security updates and bug fixes. -- Andrew :-) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) Support Free Java! Contribute to GNU Classpath and the OpenJDK http://www.gnu.org/software/classpath http://openjdk.java.net PGP Key: F5862A37 (https://keys.indymedia.org/) Fingerprint = EA30 D855 D50F 90CD F54D 0698 0713 C3ED F586 2A37
* Andrew John Hughes:
So yes, that means we use the system zlib (currently 1.2.5 here), jpeg (8c) and png (1.4.5). 1.10 will also finally remove the static linking of libstdc++ and libgcc. This was done back in the early days of IcedTea as part of preparing OpenJDK for distro packaging, allowing these libraries to receive timely security updates and bug fixes.
It also increases compatibility with systems which use a different zlib version. The symbols of the version in OpenJDK haven't been mangled, and the VM crashes if the wrong routines are called due to a symbol clash. -- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
On Tue, Feb 15, 2011 at 5:40 PM, Florian Weimer <fweimer@bfk.de> wrote:
* Andrew John Hughes:
So yes, that means we use the system zlib (currently 1.2.5 here), jpeg (8c) and png (1.4.5). 1.10 will also finally remove the static linking of libstdc++ and libgcc. This was done back in the early days of IcedTea as part of preparing OpenJDK for distro packaging, allowing these libraries to receive timely security updates and bug fixes.
It also increases compatibility with systems which use a different zlib version. The symbols of the version in OpenJDK haven't been mangled, and the VM crashes if the wrong routines are called due to a symbol clash.
Such symbol clashes are ultimately caused by the fact that ELF is probably the only binary file format left today to use the moronic idea of process-scoped symbol lookup. The JVM could work around it by using the RTLD_DEEPBIND flag to dlopen() on Linux, and by linking libraries with "-Bdirect" on Solaris. Other ELF-using operating systems (BSD?) are probably out of luck. Damjan Jovanovic
On 2/15/2011 6:07 AM, Dr Andrew John Hughes wrote:
Yes, IcedTea uses system libraries for everything bar LCMS, where local changes in OpenJDK mean we are still forced to use the in-tree version. There hasn't been any success upstreaming these changes, though I haven't looked at LCMS 2.x.
LittleCMS 1.x didn't provide the support necessary to pass JCK. So we talked to the LittleCMS maintainer and he added the necessary APIs in 2.0 JDK 7 has had LittleCMS 2.0 for almost 6 months now and that is included without any code modifications, so I think it should now be possible to use a system library, although we didn't do the work to actually enable that, so its built into a JDK library which has the littlecms code and the glue code. We need to provide the ability to separate these. When we pushed LCMS 2.0, I asked for a bug to be filed to remember to do this work but I can't find it in the database. I'll ask for that to be filed if it wasn't already. NB It didn't seem super-urgent since we pulled in LCMS 2.0 relatively soon after its release we thought shipping distros weren't likely to have the library upgrade anyway, but that's probably changing. -phil.
On 15 February 2011 20:23, Phil Race <philip.race@oracle.com> wrote:
On 2/15/2011 6:07 AM, Dr Andrew John Hughes wrote:
Yes, IcedTea uses system libraries for everything bar LCMS, where local changes in OpenJDK mean we are still forced to use the in-tree version. There hasn't been any success upstreaming these changes, though I haven't looked at LCMS 2.x.
LittleCMS 1.x didn't provide the support necessary to pass JCK. So we talked to the LittleCMS maintainer and he added the necessary APIs in 2.0 JDK 7 has had LittleCMS 2.0 for almost 6 months now and that is included without any code modifications, so I think it should now be possible to use a system library, although we didn't do the work to actually enable that, so its built into a JDK library which has the littlecms code and the glue code. We need to provide the ability to separate these. When we pushed LCMS 2.0, I asked for a bug to be filed to remember to do this work but I can't find it in the database. I'll ask for that to be filed if it wasn't already. NB It didn't seem super-urgent since we pulled in LCMS 2.0 relatively soon after its release we thought shipping distros weren't likely to have the library upgrade anyway, but that's probably changing.
-phil.
Hi Phil, Thanks for making me aware of this. I briefly glanced at some for the release notes for LCMS 2 when it was released, and saw something that may support the missing functionality, but never had chance to look in detail. I've also not had chance to look at OpenJDK 7 recently, so it's great to hear that support has already gone in. Do you have any idea as to whether this would be safe to backport to OpenJDK 6? I presume it doesn't alter any public API. I've not seen any use of OpenJDK 7 by distros as yet. We've managed with the other libraries without in-tree support by using local patching. There's a big libraries patch in IcedTea that would be nice to integrate but it would need considerable work to do optional system linking rather than the current brute force of deleting the in-tree version and always linking. There's also no TCK for 7 yet, which is I believe what caught many of the issues with missing LCMS support before. I did a quick survey of distro support for LCMS 2. It's in Gentoo (which is what made me aware of it), but Ubuntu, Debian and Fedora all seem to still be on the 1.x series. So it doesn't seem to be changing yet. Maybe OpenJDK could be the kick they need to support it? ;-) -- Andrew :-) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) Support Free Java! Contribute to GNU Classpath and the OpenJDK http://www.gnu.org/software/classpath http://openjdk.java.net PGP Key: F5862A37 (https://keys.indymedia.org/) Fingerprint = EA30 D855 D50F 90CD F54D 0698 0713 C3ED F586 2A37
On 2/20/2011 9:39 AM, Dr Andrew John Hughes wrote:
On 15 February 2011 20:23, Phil Race<philip.race@oracle.com> wrote:
On 2/15/2011 6:07 AM, Dr Andrew John Hughes wrote:
Yes, IcedTea uses system libraries for everything bar LCMS, where local changes in OpenJDK mean we are still forced to use the in-tree version. There hasn't been any success upstreaming these changes, though I haven't looked at LCMS 2.x. LittleCMS 1.x didn't provide the support necessary to pass JCK. So we talked to the LittleCMS maintainer and he added the necessary APIs in 2.0 JDK 7 has had LittleCMS 2.0 for almost 6 months now and that is included without any code modifications, so I think it should now be possible to use a system library, although we didn't do the work to actually enable that, so its built into a JDK library which has the littlecms code and the glue code. We need to provide the ability to separate these. When we pushed LCMS 2.0, I asked for a bug to be filed to remember to do this work but I can't find it in the database. I'll ask for that to be filed if it wasn't already. NB It didn't seem super-urgent since we pulled in LCMS 2.0 relatively soon after its release we thought shipping distros weren't likely to have the library upgrade anyway, but that's probably changing.
-phil.
Hi Phil,
Thanks for making me aware of this. I briefly glanced at some for the release notes for LCMS 2 when it was released, and saw something that may support the missing functionality, but never had chance to look in detail. I've also not had chance to look at OpenJDK 7 recently, so it's great to hear that support has already gone in. Do you have any idea as to whether this would be safe to backport to OpenJDK 6? I presume it doesn't alter any public API.
It ought to be OK. If someone else wants to take on the work that is :-)
I've not seen any use of OpenJDK 7 by distros as yet. We've managed with the other libraries without in-tree support by using local patching. There's a big libraries patch in IcedTea that would be nice to integrate but it would need considerable work to do optional system linking rather than the current brute force of deleting the in-tree version and always linking. There's also no TCK for 7 yet, which is I believe what caught many of the issues with missing LCMS support before.
I don't know how distros would want to present/package the 7 EA builds so I'm not too surprised they aren't common. We believe LCMS 2.0 should pass JCK, but I don't know if a full JCK run has been performed against a fully open 7 build since it went in. A 6-open backport would find any problems there.
I did a quick survey of distro support for LCMS 2. It's in Gentoo (which is what made me aware of it), but Ubuntu, Debian and Fedora all seem to still be on the 1.x series. So it doesn't seem to be changing yet. Maybe OpenJDK could be the kick they need to support it? ;-)
yep. -phil.
On 22 February 2011 17:26, Phil Race <philip.race@oracle.com> wrote:
On 2/20/2011 9:39 AM, Dr Andrew John Hughes wrote:
On 15 February 2011 20:23, Phil Race<philip.race@oracle.com> wrote:
On 2/15/2011 6:07 AM, Dr Andrew John Hughes wrote:
Yes, IcedTea uses system libraries for everything bar LCMS, where local changes in OpenJDK mean we are still forced to use the in-tree version. There hasn't been any success upstreaming these changes, though I haven't looked at LCMS 2.x.
LittleCMS 1.x didn't provide the support necessary to pass JCK. So we talked to the LittleCMS maintainer and he added the necessary APIs in 2.0 JDK 7 has had LittleCMS 2.0 for almost 6 months now and that is included without any code modifications, so I think it should now be possible to use a system library, although we didn't do the work to actually enable that, so its built into a JDK library which has the littlecms code and the glue code. We need to provide the ability to separate these. When we pushed LCMS 2.0, I asked for a bug to be filed to remember to do this work but I can't find it in the database. I'll ask for that to be filed if it wasn't already. NB It didn't seem super-urgent since we pulled in LCMS 2.0 relatively soon after its release we thought shipping distros weren't likely to have the library upgrade anyway, but that's probably changing.
-phil.
Hi Phil,
Thanks for making me aware of this. I briefly glanced at some for the release notes for LCMS 2 when it was released, and saw something that may support the missing functionality, but never had chance to look in detail. I've also not had chance to look at OpenJDK 7 recently, so it's great to hear that support has already gone in. Do you have any idea as to whether this would be safe to backport to OpenJDK 6? I presume it doesn't alter any public API.
It ought to be OK. If someone else wants to take on the work that is :-)
Consider it on my TODO list ;-)
I've not seen any use of OpenJDK 7 by distros as yet. We've managed with the other libraries without in-tree support by using local patching. There's a big libraries patch in IcedTea that would be nice to integrate but it would need considerable work to do optional system linking rather than the current brute force of deleting the in-tree version and always linking. There's also no TCK for 7 yet, which is I believe what caught many of the issues with missing LCMS support before.
I don't know how distros would want to present/package the 7 EA builds so I'm not too surprised they aren't common.
We believe LCMS 2.0 should pass JCK, but I don't know if a full JCK run has been performed against a fully open 7 build since it went in. A 6-open backport would find any problems there.
I wasn't aware there was a JCK for 7 yet. At least, not one under the same terms as the one used for OpenJDK6.
I did a quick survey of distro support for LCMS 2. It's in Gentoo (which is what made me aware of it), but Ubuntu, Debian and Fedora all seem to still be on the 1.x series. So it doesn't seem to be changing yet. Maybe OpenJDK could be the kick they need to support it? ;-)
yep.
-phil.
-- Andrew :-) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) Support Free Java! Contribute to GNU Classpath and the OpenJDK http://www.gnu.org/software/classpath http://openjdk.java.net PGP Key: F5862A37 (https://keys.indymedia.org/) Fingerprint = EA30 D855 D50F 90CD F54D 0698 0713 C3ED F586 2A37
On 2/22/2011 3:51 PM, Dr Andrew John Hughes wrote:
On 22 February 2011 17:26, Phil Race<philip.race@oracle.com> wrote:
We believe LCMS 2.0 should pass JCK, but I don't know if a full JCK run has been performed against a fully open 7 build since it went in. A 6-open backport would find any problems there.
I wasn't aware there was a JCK for 7 yet. At least, not one under the same terms as the one used for OpenJDK6.
Strictly there cannot be one until the java se 7 spec is final because otherwise it doesn't know what to test, but it has to be worked on concurrently else you'd then have to wait a long time after the JDK work was done .. so that ongoing work is what I refer to. I don't think they run JDK 6 against it because it would fail straight off the bat on all the new APIs. However in this stable area I rather doubt its anything other than the existing JCK6 tests. -phil.
One of the benefits of using a 5-year old version is that it has been thoroughly tested by various software for 5 years, given we actually don't have lots of tests for zlib ourself, this is very important, at least for me, when considering upgrade, especially at this late stage of the release I would be very cautious to do it. While the changelog lists a "long" list of changes from 1.2.3. to 1.2.4, most of them are in code that are not included in JDK, the only thing that might bring in some enhancement is "Faster Z_HUFFMAN_ONLY compression for images and other specialized compression", but given my experience of measuring the improvement in the past, I doubt you actually benefit from it in most of the use scenario. -Sherman On 2/15/2011 5:57 AM, Alan Bateman wrote:
Steve Poole wrote:
Hi all, JDK7 is using zlib 1.2.3 (which was added to JDK7 back in 2009.) Zlib's latest version is 1.2.5 - is there any expectation to move to 1.2.5 in JDK7? It seems a real shame to ship JDK7 with a version of zlib that is so out of date. More than happy to help contribute towards making this happen. I would like to know if its already on someone's radar though or to understand why this is a mad idea :-)
Cheers
Steve I don't think it's on anyone's radar. Sherman did the last update, bringing us up from a patched 1.1.3 to 1.2.3. The main thing with zlib updates is testing and bake time. Have you done any testing with 1.2.5? While there are some good reasons listed in the ChangeLog, I'm curious if they translate into any measurable improvement with the zip APIs. On the testing it would be interesting to check if 1.2.5 is used by any of the distros as I think (but might be wrong) that OpenJDK/IcedTea6 project have changes to use /usr/lib/libzip.so rather than the snapshot in the jdk repository and so might already have experience using the latest version.
-Alan
On Tue, 2011-02-15 at 09:17 -0800, Xueming Shen wrote:
One of the benefits of using a 5-year old version is that it has been thoroughly tested by various software for 5 years, given we actually don't have lots of tests for zlib ourself, this is very important, at least for me, when considering upgrade, especially at this late stage
I can appreciate why you want to be cautious right now - but it is important that the codebase is regularly upgraded. If IcedTea have shown that 1.2.5 is "ok" then I'd say the risk is quite low. For JDK 8 it would make sense to discuss the pros and cons of relying on the system version of zlib. I can make the case either way though.
of the release I would be very cautious to do it. While the changelog lists a "long" list of changes from 1.2.3. to 1.2.4, most of them are in code that are not included in JDK,
So the risk is even lower then? :-)
the only thing that might bring in some enhancement is "Faster Z_HUFFMAN_ONLY compression for images and other specialized compression", but given my experience of measuring the improvement in the past, I doubt you actually benefit from it in most of the use scenario.
I'm not sure this is about any particular improvement - its really just about keeping current with someone else's codebase.
-Sherman
On 2/15/2011 5:57 AM, Alan Bateman wrote:
Steve Poole wrote:
Hi all, JDK7 is using zlib 1.2.3 (which was added to JDK7 back in 2009.) Zlib's latest version is 1.2.5 - is there any expectation to move to 1.2.5 in JDK7? It seems a real shame to ship JDK7 with a version of zlib that is so out of date. More than happy to help contribute towards making this happen. I would like to know if its already on someone's radar though or to understand why this is a mad idea :-)
Cheers
Steve I don't think it's on anyone's radar. Sherman did the last update, bringing us up from a patched 1.1.3 to 1.2.3. The main thing with zlib updates is testing and bake time. Have you done any testing with 1.2.5? While there are some good reasons listed in the ChangeLog, I'm curious if they translate into any measurable improvement with the zip APIs. On the testing it would be interesting to check if 1.2.5 is used by any of the distros as I think (but might be wrong) that OpenJDK/IcedTea6 project have changes to use /usr/lib/libzip.so rather than the snapshot in the jdk repository and so might already have experience using the latest version.
-Alan
Steve Poole wrote:
: I can appreciate why you want to be cautious right now - but it is important that the codebase is regularly upgraded. If IcedTea have shown that 1.2.5 is "ok" then I'd say the risk is quite low.
For JDK 8 it would make sense to discuss the pros and cons of relying on the system version of zlib. I can make the case either way though.
We should probably add a build option around this so there is the option of using the system zip library in preference to compiling the zlib copy in the repository. However, I don't think we can get away from keeping a copy in the repository as we need to be able to compile to platforms that don't have zlib. Do you have cycles to do testing with zlib 1.2.5, ideally on platforms other than Linux? (I suggest platforms other than Linux as it does appear that this version is used there and it would really help to get confirmation that all is well on other platforms too). -Alan.
On Wed, 2011-02-16 at 12:53 +0000, Alan Bateman wrote:
Steve Poole wrote:
: I can appreciate why you want to be cautious right now - but it is important that the codebase is regularly upgraded. If IcedTea have shown that 1.2.5 is "ok" then I'd say the risk is quite low.
For JDK 8 it would make sense to discuss the pros and cons of relying on the system version of zlib. I can make the case either way though.
We should probably add a build option around this so there is the option of using the system zip library in preference to compiling the zlib copy in the repository. However, I don't think we can get away from keeping a copy in the repository as we need to be able to compile to platforms that don't have zlib.
Do you have cycles to do testing with zlib 1.2.5, ideally on platforms other than Linux? (I suggest platforms other than Linux as it does appear that this version is used there and it would really help to get confirmation that all is well on other platforms too).
What sort of testing did you have in mind? You mean run the zlib tests and/or OpenJDK testcases?
-Alan.
Steve Poole wrote:
: What sort of testing did you have in mind? You mean run the zlib tests and/or OpenJDK testcases?
To be honest, I don't know. The zip tests should clearly be run but they are unlikely to stress the zip code in the same way that the IDEs, app servers, and other big applications do. If there is a compelling reason to do another zlib update in 7 then I would suggest running as many of these as you can to increase the confidence that there won't be an issues. If this is too much work then it might be better to just prepare the patch to go in once jdk8 opens for business. -Alan.
On Thu, 2011-02-17 at 10:20 +0000, Alan Bateman wrote:
Steve Poole wrote:
: What sort of testing did you have in mind? You mean run the zlib tests and/or OpenJDK testcases?
To be honest, I don't know. The zip tests should clearly be run but they are unlikely to stress the zip code in the same way that the IDEs, app servers, and other big applications do.
Running zlibs own tests is unlikely to be too helpful. I'd be quite happy to run any JDK tests you have and there are also testcases in Apache Harmony for the related Java APIs. I'm curious to understand just what level of testing is already done though?
If there is a compelling reason to do another zlib update in 7 then I would suggest running as many of these as you can to increase the confidence that there won't be an issues.
I think this is simply about currency. Getting fixes from other projects usually has a list of drag-alongs. So the closer you can keep to their latest production level the smaller that list is , and consequently the lower the risk is, when you need a specific fix later.
If this is too much work then it might be better to just prepare the patch to go in once jdk8 opens for business.
Agreed - I can imagine why zlib was embedded originally but I can't see why that should continue. It seem that the right direction is to remove the embedded code and rely on the platform providing the support.
-Alan.
Steve
Steve Poole wrote:
: Running zlibs own tests is unlikely to be too helpful. I'd be quite happy to run any JDK tests you have and there are also testcases in Apache Harmony for the related Java APIs. I'm curious to understand just what level of testing is already done though?
I wasn't clear. I meant the zip tests in the jdk repository rather than the zlib tests. The zip tests are in test/java/util/zip and test/java/util/jar. I don't know the Apache Harmony tests but if they can be be used with OpenJDK builds then great.
:
Agreed - I can imagine why zlib was embedded originally but I can't see why that should continue. It seem that the right direction is to remove the embedded code and rely on the platform providing the support.
That would work on platforms that have zlib installed (libzip on Linux, libz on Solaris, etc.) but my guess is that we would still need to build it on at least Windows. A possible starting point is a build option that determines whether it compiles the zlib copy in the repository or not. -Alan
participants (7)
-
Alan Bateman
-
Damjan Jovanovic
-
Dr Andrew John Hughes
-
Florian Weimer
-
Phil Race
-
Steve Poole
-
Xueming Shen