From brian.goetz at oracle.com Sun Jun 19 14:11:38 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Sun, 19 Jun 2022 10:11:38 -0400 Subject: New JEP: Classfile Processing API Message-ID: <641b055b-a384-525a-f71a-81ccebd53bc2@oracle.com> Generation, parsing, and transformation of classfiles is ubiquitous in the Java ecosystem.? Frameworks transform classfiles on the fly by hooking into classloaders or using `java.lang.instrument`; compilers and IDEs for all sorts of languages read and write classfiles; static analysis tools read classfiles; migration tools transform classfiles.? In the JDK, we have a static compiler (`javac`) and other tools (`javap`, `jlink`) that read or transform classfiles, and numerous platform features that operate by classfile generation (reflection, lambdas, dynamic proxies, method handles), as well as many tests that generate classfiles.? In fact, the JDK contains at least three libraries for processing classfiles: an internal one (used by JDK tools such as `javac`); a fork of ASM (used by lambdas and many other platform features), and a fork of BCEL (contained inside a fork of Xalan). And of course the ecosystem has many such libraries: ASM, javassist, cglib, BCEL, gnu.bytecode, ByteBuddy, and many others. Over the years, there have been many calls for Java to have an "official" classfile library, and there are numerous practical reasons to create one, and the shift to the six-month cadence has added two new reasons.? Firstly, tools that bundle classfile libraries are more likely to encounter classfile versions "from the future", because they use a classfile library that knows about Java N, but the customer may be running it on Java N+1, which was released only six months later.? Second, the pace of classfile format evolution has picked up significantly in recent years, with a significant fraction of releases having at least some change to the classfile format (new attributes, new constant pool forms, new bytecodes, new combinations of flags, etc).? Having an official classfile library would mean that applications and tools are guaranteed to have access to a library that is up-to-date to the latest classfile version that will run on the current JDK. It might seem an "obvious" choice to "just" standardize on ASM, but there are many reasons not to do so.? ASM is an old codebase with plenty of legacy baggage; the design priorities that informed its architecture are not ideal for the JDK in 2022; and the language has improved substantially since them (lambdas, records and sealed classes, pattern matching.)? What were the best possible API idioms in 2002 (visitors) may not be ideal two decades later. A draft JEP can be found here: https://openjdk.org/jeps/8280389 Such an API will have many consumers, both internal to the JDK and external: compilers, analysis tools, platform implementation, 3rd party libraries, tests, etc.? We will focus initially on the internal JDK consumers, incrementally migrating them off of the existing libraries and onto the new library.? Eventually, once all the internal consumers are migrated, we will be able to remove ASM from the JDK entirely.? Initially, the library will be internal-only (non-exported), to gain experience with and refine the API before committing to a public version. When the API is sufficiently stable, it will be exported for public use, first as a preview API and eventually as a permanent API. Comments on the JEP itself -- the concept, motivation, and design goals -- are invited for discussion.? The JEP includes some code examples, which are hopefully illustrative, but I will request that people hold off on API discussions for the time being, to make room for discussion on the JEP itself first. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Sun Jun 19 18:21:21 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Sun, 19 Jun 2022 14:21:21 -0400 Subject: New JEP: Classfile Processing API In-Reply-To: References: <641b055b-a384-525a-f71a-81ccebd53bc2@oracle.com> Message-ID: <82c5c1d3-f77c-38ba-e45e-00d104b3c2d1@oracle.com> Hello Per! Nice to hear from you again. > An aside (and not telling you anything new I'm sure) but a "fixup" > post-processing pass > is probably preferable to doing the code-generation twice: It's hard > for the latter to use > a mix of short and long jumps as needed. Yeah, it's unfortunate either way.? It turns out that 99+% of methods have no long jumps, either because they're shorter than 32K, or are longer but lucky.? So the retry approach is based on an optimistic assumption; we save having to do the fixup + compression pass almost all the time, at the cost of running the generation twice a tiny fraction of the time.? (Of course, if you know a method is going large, it might also be reasonable to indicate that when you start generating, to skip over the first pass -- this is something worth looking at.) Even so, there are still fixups needed for forward branches since you don't know the offset until later (and have to hope that someone actually emits the corresponding label before the method is done.) But this can be accumulated as you go and patched cheaply in place at the end if you commit to a specific branch offset width. Cheers, -Brian > When gnu.bytecodes generates code the instructions are appended to a byte > array in optimistic form. In addition "fixup" operations are appended > to the fixup > buffer (the arrays fixup_offset and fixup_labels).? Then when done > emitting instructions > we call processFixups, which iterates over the fixups (3 times!) to > adjust the bytecode > and assign final offsets to the labels. > > The processFixups method is complicated because it handles a number of > issues at > the same time: various optimizations (including moving some code > blocks around), > as well as assigning offsets to Labels. > > Feel free to get inspiration from: > > https://urldefense.com/v3/__https://gitlab.com/kashell/Kawa/-/blob/master/gnu/bytecode/CodeAttr.java__;!!ACWV5N9M2RV99hQ!NuFecLzqRMWfxSH4G-KYTp_9Jn6vyGPkm9yEuj7VQPxtb7CpXAzN5fAeyINsqhMRRkTrX_0Ua5gu7w$ > > The code is a bit convoluted, written more for performance then > readability. > However, it is at least somewhat commented - and I'm happy to answer > any questions. From per at bothner.com Sun Jun 19 19:08:28 2022 From: per at bothner.com (Per Bothner) Date: Sun, 19 Jun 2022 12:08:28 -0700 Subject: New JEP: Classfile Processing API In-Reply-To: <82c5c1d3-f77c-38ba-e45e-00d104b3c2d1@oracle.com> References: <641b055b-a384-525a-f71a-81ccebd53bc2@oracle.com> <82c5c1d3-f77c-38ba-e45e-00d104b3c2d1@oracle.com> Message-ID: <00f5d5ff-9c49-6169-31b6-172aad776189@bothner.com> On 6/19/22 11:21, Brian Goetz wrote: > Hello Per! > > Nice to hear from you again. > >> An aside (and not telling you anything new I'm sure) but a "fixup" post-processing pass >> is probably preferable to doing the code-generation twice: It's hard for the latter to use >> a mix of short and long jumps as needed. > > Yeah, it's unfortunate either way.? It turns out that 99+% of methods have no long jumps, either because they're shorter than 32K, or are longer but lucky.? So the retry approach is based on an optimistic assumption; we save having to do the fixup + compression pass almost all the time, at the cost of running the generation twice a tiny fraction of the time. Fair enough - though if yu have a method larger than 32K you really don't want to needlessly push the limit above 64K. Unless the library has a mechanism to deal with really large methods. Kawa has a testsuite that fails because it compiles to larger than 64K. (This is partly because each test is macro-expanded excessively. It is fixable in various ways; I've been putting off dealing with it, as I don't spend a lot of time on Kawa these days.) I've argued before that this JVM limit should be fixed - unless there is a library that can work around it in a reasonable way. One other optimization that processFixups handles is eliminating jumps to jumps. I found them painful to deal with otherwise. (In my recollection - this code appears to be from 2004.) I realize the JIT can deal with such infelicities in the bytecode, but I feel uncomfortable generating bloated bytecode. -- --Per Bothner per at bothner.com https://urldefense.com/v3/__http://per.bothner.com/__;!!ACWV5N9M2RV99hQ!KGLT5IGNQVgoDkEownu8qK7dMGRc_ONCa7W752Fpn3eLT06CbgEGJz6hLfDPY9Uj0xsDzTWM0L7408e2tPKT6g$ From brian.goetz at oracle.com Fri Jun 24 15:51:34 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 24 Jun 2022 11:51:34 -0400 Subject: New JEP: Classfile Processing API In-Reply-To: <641b055b-a384-525a-f71a-81ccebd53bc2@oracle.com> References: <641b055b-a384-525a-f71a-81ccebd53bc2@oracle.com> Message-ID: We've staged the current state of the in-development classfile API in the OpenJDK sandbox repo: https://github.com/openjdk/jdk-sandbox/tree/classfile-api-branch The README file there includes instructions for building, running the tests, and running the benchmarks.? The API is in the package `jdk.classfile` (a good starting point is the class `Classfile`, which contains the main entry points for parsing and generating classfiles).? It provides full coverage of the classfile format through Java 17.? The repo also includes experimental replacements for most uses of ASM in the JDK. There is Javadoc here: https://htmlpreview.github.io/?https://raw.githubusercontent.com/openjdk/jdk-sandbox/classfile-api-javadoc-branch/doc/classfile-api/javadoc/jdk/classfile/package-summary.html The package doc is a good place to start to get an overview of how it is used.? The docs so far are fairly thin, and need a lot of work, especially examples. How can people contribute? ?- Try it out!? Bring your favorite code examples from ASM, cglib, or your favorite bytecode library, and try them with this library.? Post experience reports (but please, let's keep subjective "you should do it like my library does" down to a dull roar.)? Bear in mind that your first idea of how to do it in the library might be suboptimal, so ask questions. ?- Tests.? Our test coverage is good, but there are still corners of the API that are not fully covered.? We'll be posting some coverage metrics soon; additional tests are always useful. ?- Examples.? Our docs need lots of examples, and well-crafted examples are worth a thousand words. Further discussion should take place on classfile-api-dev at openjdk.org (copied); you can sign up here: https://mail.openjdk.org/mailman/listinfo/classfile-api-dev On 6/19/2022 10:11 AM, Brian Goetz wrote: > > Generation, parsing, and transformation of classfiles is ubiquitous in > the Java ecosystem.? Frameworks transform classfiles on the fly by > hooking into classloaders or using `java.lang.instrument`; compilers > and IDEs for all sorts of languages read and write classfiles; static > analysis tools read classfiles; migration tools transform classfiles.? > In the JDK, we have a static compiler (`javac`) and other tools > (`javap`, `jlink`) that read or transform classfiles, and numerous > platform features that operate by classfile generation (reflection, > lambdas, dynamic proxies, method handles), as well as many tests that > generate classfiles.? In fact, the JDK contains at least three > libraries for processing classfiles: an internal one (used by JDK > tools such as `javac`); a fork of ASM (used by lambdas and many other > platform features), and a fork of BCEL (contained inside a fork of > Xalan).? And of course the ecosystem has many such libraries: ASM, > javassist, cglib, BCEL, gnu.bytecode, ByteBuddy, and many others. > > Over the years, there have been many calls for Java to have an > "official" classfile library, and there are numerous practical reasons > to create one, and the shift to the six-month cadence has added two > new reasons. Firstly, tools that bundle classfile libraries are more > likely to encounter classfile versions "from the future", because they > use a classfile library that knows about Java N, but the customer may > be running it on Java N+1, which was released only six months later.? > Second, the pace of classfile format evolution has picked up > significantly in recent years, with a significant fraction of releases > having at least some change to the classfile format (new attributes, > new constant pool forms, new bytecodes, new combinations of flags, > etc). Having an official classfile library would mean that > applications and tools are guaranteed to have access to a library that > is up-to-date to the latest classfile version that will run on the > current JDK. > > It might seem an "obvious" choice to "just" standardize on ASM, but > there are many reasons not to do so.? ASM is an old codebase with > plenty of legacy baggage; the design priorities that informed its > architecture are not ideal for the JDK in 2022; and the language has > improved substantially since them (lambdas, records and sealed > classes, pattern matching.)? What were the best possible API idioms in > 2002 (visitors) may not be ideal two decades later. > > A draft JEP can be found here: > > https://openjdk.org/jeps/8280389 > > Such an API will have many consumers, both internal to the JDK and > external: compilers, analysis tools, platform implementation, 3rd > party libraries, tests, etc.? We will focus initially on the internal > JDK consumers, incrementally migrating them off of the existing > libraries and onto the new library. Eventually, once all the internal > consumers are migrated, we will be able to remove ASM from the JDK > entirely.? Initially, the library will be internal-only > (non-exported), to gain experience with and refine the API before > committing to a public version.? When the API is sufficiently stable, > it will be exported for public use, first as a preview API and > eventually as a permanent API. > > Comments on the JEP itself -- the concept, motivation, and design > goals -- are invited for discussion.? The JEP includes some code > examples, which are hopefully illustrative, but I will request that > people hold off on API discussions for the time being, to make room > for discussion on the JEP itself first. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From per at bothner.com Fri Jun 24 17:45:59 2022 From: per at bothner.com (Per Bothner) Date: Fri, 24 Jun 2022 10:45:59 -0700 Subject: can API be implemented on top of reflection? Message-ID: <6b111c65-2020-108c-48bf-b9b9c7ec6ca1@bothner.com> One feature of the gnu.bytecode library (use by Kawa) is that it provides the same API whether the information is extracted from a .class file or from reflection. If this new library in some hypothetical future is to replace gnu.bytecode it would be nice to preserve that functionality. Based on a quick look at the API it seems like it should be possible to implement a ClassModel on top of reflection. That would be desirable, even if it's not a primary initial goal of this project. Still, it would be nice to provide that as either part of the library itself or as a testsuite to make sure it remains feasible. Perhaps ultimately the new API could a suitable replacement for the reflection API, but with the lessons of the "Mirrors" ideas in the 2004 paper (https://urldefense.com/v3/__https://bracha.org/mirrors.pdf__;!!ACWV5N9M2RV99hQ!IDvYjRq58nngngAfRsofjUK7vvaEaPCe7YoulgZOqXxMXjVzHtt9desKzRwOQSnSJCV3kYD_7gKAlshFvMXlaw$ ). Of course these ideas are not new to Brian and his co-workers, but I thought it worth mentioning them here. -- --Per Bothner per at bothner.com https://urldefense.com/v3/__http://per.bothner.com/__;!!ACWV5N9M2RV99hQ!IDvYjRq58nngngAfRsofjUK7vvaEaPCe7YoulgZOqXxMXjVzHtt9desKzRwOQSnSJCV3kYD_7gKAlsjvkYO5oQ$ From brian.goetz at oracle.com Fri Jun 24 18:45:54 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 24 Jun 2022 14:45:54 -0400 Subject: can API be implemented on top of reflection? In-Reply-To: <6b111c65-2020-108c-48bf-b9b9c7ec6ca1@bothner.com> References: <6b111c65-2020-108c-48bf-b9b9c7ec6ca1@bothner.com> Message-ID: <049ace17-1423-2a6c-3918-d20c37a57317@oracle.com> One of the choices we made is to stay as close to JVMS Ch4 as we can.? There were a number of reasons, one of the more subtle ones is that if you try to abstract over what's in the classfile, it means that even the simplest transforms have to make round-trips from the native format to the abstracted format and back.? Whereas if we give you something that maps roughly 1:1 with the classfile format (with exceptions; labels and constant pool indexes have to be abstracted), then simple transforms involve less needless data motion.? Which illustrates a key, deliberate design priority: that while most of the API surface is about reading and writing classfiles, most of the interesting use cases involve transforming classfiles, and we want this to be as free of impedence mismatch as possible. To make this concrete, symbolic information (e.g., classes) is exposed directly as constant pool entries -- but reflection doesn't give you those, so these would have to be faked.? (Obviously reflection will bottom out before it gives you method bodies, too.)?? A more subtle example is that it is hard to tell, reflectively, the difference between not having a RuntimeVisibleAnnotationsAttribute, and having one that contains no annotations (same for all "listy" attributes, like method exceptions).? And the fakery goes on; to create a synthetic Signature attribute, you'd have to reverse engineer all the information that reflection carves into little bits.? All of which is to say that the ClassModel that you'd get if you tried to abstract over reflection would involve a lot of fakery (and fidelity loss if you tried to then transform it into a new classfile.) So, its possible, but I'm not convinced that we'd like the result. > One feature of the gnu.bytecode library (use by Kawa) is that it provides > the same API whether the information is extracted from a .class file > or from reflection.? If this new library in some hypothetical future > is to replace gnu.bytecode it would be nice to preserve that > functionality. > > Based on a quick look at the API it seems like it should be possible > to implement > a ClassModel on top of reflection.? That would be desirable, > even if it's not a primary initial goal of this project.? Still, it > would be nice to provide that as either part of the library itself > or as a testsuite to make sure it remains feasible. > > Perhaps ultimately the new API could a suitable replacement for the > reflection API, but with the lessons of the "Mirrors" ideas in the > 2004 paper > (https://urldefense.com/v3/__https://bracha.org/mirrors.pdf__;!!ACWV5N9M2RV99hQ!IDvYjRq58nngngAfRsofjUK7vvaEaPCe7YoulgZOqXxMXjVzHtt9desKzRwOQSnSJCV3kYD_7gKAlshFvMXlaw$ > ). > > Of course these ideas are not new to Brian and his co-workers, > but I thought it worth mentioning them here. From per at bothner.com Sat Jun 25 03:12:14 2022 From: per at bothner.com (Per Bothner) Date: Fri, 24 Jun 2022 20:12:14 -0700 Subject: can API be implemented on top of reflection? In-Reply-To: <049ace17-1423-2a6c-3918-d20c37a57317@oracle.com> References: <6b111c65-2020-108c-48bf-b9b9c7ec6ca1@bothner.com> <049ace17-1423-2a6c-3918-d20c37a57317@oracle.com> Message-ID: <729b4c46-f16a-3ca0-c1f8-a3f44f9f8004@bothner.com> On 6/24/22 11:45, Brian Goetz wrote: > So, its possible, but I'm not convinced that we'd like the result. That sounds reasonable. Thanks for the explanation. It might work to have a low-level class-file-based API as well as a more higher-level one - like ClassModel extends ClassModeLow. But I understand that may be difficult and not a priority. -- --Per Bothner per at bothner.com https://urldefense.com/v3/__http://per.bothner.com/__;!!ACWV5N9M2RV99hQ!LduD-SZGY9rSYW1JRxTqKIQunIRkPUEj6j0bOreuwtEG0oBfPy2SSbXVoHm86aAFsaYmniZlDoNVI7yi-98XFA$ From adam.sotona at oracle.com Tue Jun 28 08:41:44 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Tue, 28 Jun 2022 08:41:44 +0000 Subject: Classfile Processing API - AccessFlag aligned with JDK Message-ID: Hi, Part of the Classfile Processing API is access flags modeling. We held our own copy of AccessFlag class while waiting for JDK-8266670 ?Better modeling of access flags in core reflection?. Good news is that java.lang.reflect.AccesFlag class has been recently integrated into JDK. And the two commits below refactored Classfile API (including tests, benchmarks and JDK use cases) to use java.lang.reflect.AccesFlag instead. Both branches (classfile-api-branch and classfile-api-dev-branch) and online API doc have been updated. Adam From: jdk-sandbox-changes on behalf of Adam Sotona Date: Monday, 27 June 2022 13:23 To: jdk-sandbox-changes at openjdk.org Subject: git: openjdk/jdk-sandbox: classfile-api-dev-branch: 2 new changesets Changeset: a1a316ea Author: Adam Sotona Date: 2022-06-27 12:54:51 +0000 URL: https://git.openjdk.org/jdk-sandbox/commit/a1a316ea4e4d1a2540852031f8bb5cb575bee8cf removal of jdk.classfile.jdktypes.AccessFlag and redirection to java.lang.reflect.AccesFlag in Classfile API and tests ! src/java.base/share/classes/jdk/classfile/AccessFlags.java ! src/java.base/share/classes/jdk/classfile/ClassBuilder.java ! src/java.base/share/classes/jdk/classfile/Classfile.java ! src/java.base/share/classes/jdk/classfile/FieldBuilder.java ! src/java.base/share/classes/jdk/classfile/MethodBuilder.java ! src/java.base/share/classes/jdk/classfile/attribute/InnerClassInfo.java ! src/java.base/share/classes/jdk/classfile/attribute/MethodParameterInfo.java ! src/java.base/share/classes/jdk/classfile/attribute/ModuleAttribute.java ! src/java.base/share/classes/jdk/classfile/attribute/ModuleExportInfo.java ! src/java.base/share/classes/jdk/classfile/attribute/ModuleOpenInfo.java ! src/java.base/share/classes/jdk/classfile/attribute/ModuleRequireInfo.java ! src/java.base/share/classes/jdk/classfile/impl/AccessFlagsImpl.java ! src/java.base/share/classes/jdk/classfile/impl/ClassImpl.java ! src/java.base/share/classes/jdk/classfile/impl/StackMapDecoder.java ! src/java.base/share/classes/jdk/classfile/impl/Util.java ! src/java.base/share/classes/jdk/classfile/impl/verifier/VerificationWrapper.java - src/java.base/share/classes/jdk/classfile/jdktypes/AccessFlag.java ! src/java.base/share/classes/jdk/classfile/snippets/PackageSnippets.java ! src/java.base/share/classes/jdk/classfile/transforms/CodeLocalsShifter.java ! test/jdk/jdk/classfile/AccessFlagsTest.java ! test/jdk/jdk/classfile/AdvancedTransformationsTest.java ! test/jdk/jdk/classfile/BuilderBlockTest.java ! test/jdk/jdk/classfile/LDCTest.java ! test/jdk/jdk/classfile/LowAdaptTest.java ! test/jdk/jdk/classfile/LvtTest.java ! test/jdk/jdk/classfile/MassAdaptCopyPrimitiveMatchCodeTest.java ! test/jdk/jdk/classfile/OneToOneTest.java ! test/jdk/jdk/classfile/OpcodesValidationTest.java ! test/jdk/jdk/classfile/StackMapsTest.java ! test/jdk/jdk/classfile/TempConstantPoolBuilderTest.java ! test/jdk/jdk/classfile/WriteTest.java ! test/jdk/jdk/classfile/examples/ExampleGallery.java ! test/micro/org/openjdk/bench/jdk/classfile/ReadMetadata.java ! test/micro/org/openjdk/bench/jdk/classfile/Write.java Changeset: ee472295 Author: Adam Sotona Date: 2022-06-27 13:22:35 +0000 URL: https://git.openjdk.org/jdk-sandbox/commit/ee472295ad4466ba07ddc6c22a23424b98bb9c89 redirection of AccessFlag use cases to java.lang.reflect.AccesFlag in JDK integrations ! src/java.base/share/classes/java/lang/Module.java ! src/java.base/share/classes/java/lang/invoke/ClassSpecializer.java ! src/java.base/share/classes/java/lang/invoke/GenerateJLIClassesHelper.java ! src/java.base/share/classes/java/lang/invoke/MethodHandleImpl.java ! src/jdk.jdeps/share/classes/com/sun/tools/javap/AttributeWriter.java ! src/jdk.jdeps/share/classes/com/sun/tools/javap/ClassWriter.java -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Tue Jun 28 20:19:01 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 28 Jun 2022 20:19:01 +0000 Subject: Review of Classfile API JEP Message-ID: <0B3D04A1-336B-46B1-821D-F1966C197E72@oracle.com> I made some minor edits, mostly updating the code examples, replacing the guarded pattern with a when clause on the pattern label. Paul. Non-goals - Not a goal to provide a view of a classfile at runtime by abstracting over reflective APIs and broadening reflection to give access to the byte codes of a method body. [In response to list discussion on this topic] - Not a goal to provide code analysis functionality [i.e. we are not gonna provide an equivalent to all that ASM supports, since in part the JDK does not use all of ASM]. - Not a goal to support the generation of incorrect classifies, for the purposes of testing consumers. [Although it may be possible in some cases e.g. unverifiable classifies] ? ? **JVM evolution.** The JVM, and class file format, are evolving much faster now than in the early years of Java. While some evolutions are simple (such as adding new attributes such as `NestMembers`), others are more complex; Project Valhalla will bring new bytecodes, new field descriptors, and new verification rules. At some point, it may be prohibitively expensive or complex to evolve existing libraries to support these new features. ? Plus it makes it harder for us to prototype new features. Plus plus writing bytecode tests using ASM is also hard, and we do have quite a few tests that do that. As a case in point we successfully wrote condy tests using a very early prototype of the API (that is committed in the test area). We can update those tests to use this API, and remove the prototype type. ? If we want to process a class file and keep everything unchanged except for removing methods whose names start with "debug", we would get a `ClassModel`, create a `ClassBuilder`, iterate the elements of the original `ClassModel`, and pass through all of them to the builder, except the methods we want to drop: ? Mention about information may be retained in the constant pool for the removed method?s code elements referring to constant pool entries, but the pool can be compacted if need be? From info at j-kuhn.de Thu Jun 30 13:31:57 2022 From: info at j-kuhn.de (Johannes Kuhn) Date: Thu, 30 Jun 2022 15:31:57 +0200 Subject: JDK-8179483: Return type inference in if Message-ID: <56e759b6-7ac2-10e7-3ab7-bb6021657b86@j-kuhn.de> So, I tried to extract the jdk.classfile API and compile it using Java 18 with preview features enabled. Using Eclipse. While doing that, I encountered some compilation problems - one is the use of something similar like this: T optionValue(Classfile.Option.Key key) {...} ... if (reader.optionValue(...)) {...} According to https://bugs.openjdk.org/browse/JDK-8179483 ('if' conditions are treated like assignments by inference) this abuses a bug in javac - and is not valid java. The places where I found a use like this are: https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java#L76 https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/BoundAttribute.java#L150 https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L153 https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L208 https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L111 https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L209 https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L277 As far as I am concerned, there are two possible fixes: * Make `if` an assignment context - but then why is `for (String s : EXPR) {}` not an assignment context for `Iterable`? * Change the code in jdk.classfile so it adds type hints in those cases. I am not qualified enough to judge which one is the better fix. - Johannes PS.: FYI: There seems also be a bug in ECJ w.r.t. handling of class hierarchies of sealed classes. I still try to create a small reproducer. From forax at univ-mlv.fr Thu Jun 30 14:32:59 2022 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 30 Jun 2022 16:32:59 +0200 (CEST) Subject: JDK-8179483: Return type inference in if In-Reply-To: <56e759b6-7ac2-10e7-3ab7-bb6021657b86@j-kuhn.de> References: <56e759b6-7ac2-10e7-3ab7-bb6021657b86@j-kuhn.de> Message-ID: <306137282.1472767.1656599579936.JavaMail.zimbra@u-pem.fr> ----- Original Message ----- > From: "Johannes Kuhn" > To: classfile-api-dev at openjdk.org > Sent: Thursday, June 30, 2022 3:31:57 PM > Subject: JDK-8179483: Return type inference in if > So, I tried to extract the jdk.classfile API and compile it using Java > 18 with preview features enabled. Using Eclipse. > > While doing that, I encountered some compilation problems - one is the > use of something similar like this: > > T optionValue(Classfile.Option.Key key) {...} > ... > if (reader.optionValue(...)) {...} > > According to https://bugs.openjdk.org/browse/JDK-8179483 ('if' > conditions are treated like assignments by inference) this abuses a bug > in javac - and is not valid java. > > The places where I found a use like this are: > > https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java#L76 > https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/BoundAttribute.java#L150 > https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L153 > https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L208 > https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L111 > https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L209 > https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L277 yes, this is an annoying bug of javac, i've also encounter it in student codes :) Here, the real question is why optionValue() has such weird signature, it means there is a @SuppressWarnings("unchecked") which is not safe in the implementation. This kind of signature is only valid if T is always null, like in Collections.emptyList(). > > As far as I am concerned, there are two possible fixes: > > * Make `if` an assignment context - but then why is `for (String s : > EXPR) {}` not an assignment context for `Iterable`? > * Change the code in jdk.classfile so it adds type hints in those cases. > > I am not qualified enough to judge which one is the better fix. > > - Johannes > > PS.: FYI: There seems also be a bug in ECJ w.r.t. handling of class > hierarchies of sealed classes. I still try to create a small reproducer. R?mi From info at j-kuhn.de Thu Jun 30 14:44:37 2022 From: info at j-kuhn.de (Johannes Kuhn) Date: Thu, 30 Jun 2022 16:44:37 +0200 Subject: JDK-8179483: Return type inference in if In-Reply-To: <306137282.1472767.1656599579936.JavaMail.zimbra@u-pem.fr> References: <56e759b6-7ac2-10e7-3ab7-bb6021657b86@j-kuhn.de> <306137282.1472767.1656599579936.JavaMail.zimbra@u-pem.fr> Message-ID: On 30-Jun-22 16:32, Remi Forax wrote: > ----- Original Message ----- >> From: "Johannes Kuhn" >> To: classfile-api-dev at openjdk.org >> Sent: Thursday, June 30, 2022 3:31:57 PM >> Subject: JDK-8179483: Return type inference in if > >> So, I tried to extract the jdk.classfile API and compile it using Java >> 18 with preview features enabled. Using Eclipse. >> >> While doing that, I encountered some compilation problems - one is the >> use of something similar like this: >> >> T optionValue(Classfile.Option.Key key) {...} >> ... >> if (reader.optionValue(...)) {...} >> >> According to https://bugs.openjdk.org/browse/JDK-8179483 ('if' >> conditions are treated like assignments by inference) this abuses a bug >> in javac - and is not valid java. >> >> The places where I found a use like this are: >> >> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java#L76 >> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/BoundAttribute.java#L150 >> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L153 >> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L208 >> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L111 >> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L209 >> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L277 > > yes, this is an annoying bug of javac, i've also encounter it in student codes :) > > Here, the real question is why optionValue() has such weird signature, it means there is a @SuppressWarnings("unchecked") which is not safe in the implementation. > This kind of signature is only valid if T is always null, like in Collections.emptyList(). > From some point of view, it provides a "nice" API - if it would return Object - as it should - then a cast is needed at the callsite. Which is IMHO not that bad, but preferences differ, and some people prefer to write clever code. I have to admit, the use of such a method is nice - but IMHO such a method can only ever fulfill its contract (return something the caller wants, but the caller doesn't tell what it wants) if it returns `null`. >> >> As far as I am concerned, there are two possible fixes: >> >> * Make `if` an assignment context - but then why is `for (String s : >> EXPR) {}` not an assignment context for `Iterable`? >> * Change the code in jdk.classfile so it adds type hints in those cases. >> >> I am not qualified enough to judge which one is the better fix. >> >> - Johannes >> >> PS.: FYI: There seems also be a bug in ECJ w.r.t. handling of class >> hierarchies of sealed classes. I still try to create a small reproducer. > > > R?mi - Johannes From forax at univ-mlv.fr Thu Jun 30 15:12:52 2022 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Thu, 30 Jun 2022 17:12:52 +0200 (CEST) Subject: JDK-8179483: Return type inference in if In-Reply-To: References: <56e759b6-7ac2-10e7-3ab7-bb6021657b86@j-kuhn.de> <306137282.1472767.1656599579936.JavaMail.zimbra@u-pem.fr> Message-ID: <1914332749.1524106.1656601972273.JavaMail.zimbra@u-pem.fr> ----- Original Message ----- > From: "Johannes Kuhn" > To: "Remi Forax" > Cc: "classfile-api-dev" > Sent: Thursday, June 30, 2022 4:44:37 PM > Subject: Re: JDK-8179483: Return type inference in if > On 30-Jun-22 16:32, Remi Forax wrote: >> ----- Original Message ----- >>> From: "Johannes Kuhn" >>> To: classfile-api-dev at openjdk.org >>> Sent: Thursday, June 30, 2022 3:31:57 PM >>> Subject: JDK-8179483: Return type inference in if >> >>> So, I tried to extract the jdk.classfile API and compile it using Java >>> 18 with preview features enabled. Using Eclipse. >>> >>> While doing that, I encountered some compilation problems - one is the >>> use of something similar like this: >>> >>> T optionValue(Classfile.Option.Key key) {...} >>> ... >>> if (reader.optionValue(...)) {...} >>> >>> According to https://bugs.openjdk.org/browse/JDK-8179483 ('if' >>> conditions are treated like assignments by inference) this abuses a bug >>> in javac - and is not valid java. >>> >>> The places where I found a use like this are: >>> >>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java#L76 >>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/BoundAttribute.java#L150 >>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L153 >>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L208 >>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L111 >>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L209 >>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L277 >> >> yes, this is an annoying bug of javac, i've also encounter it in student codes >> :) >> >> Here, the real question is why optionValue() has such weird signature, it means >> there is a @SuppressWarnings("unchecked") which is not safe in the >> implementation. >> This kind of signature is only valid if T is always null, like in >> Collections.emptyList(). >> > > From some point of view, it provides a "nice" API - if it would return > Object - as it should - then a cast is needed at the callsite. Which is > IMHO not that bad, but preferences differ, and some people prefer to > write clever code. It's clever until it's not, by example var list = new ArrayList(); list.add(optionValue(...)); the compiler will be happy to compile that snippet and *not* insert a cast because optionValue() says it returns a space giant and both E and T erases to Object. > > I have to admit, the use of such a method is nice - but IMHO such a > method can only ever fulfill its contract (return something the caller > wants, but the caller doesn't tell what it wants) if it returns `null`. This is a kind of okay if you are both the guy that write optionValue() and call it, otherwise it's a good way to waste people time and awake space giants :) R?mi From info at j-kuhn.de Thu Jun 30 15:32:17 2022 From: info at j-kuhn.de (Johannes Kuhn) Date: Thu, 30 Jun 2022 17:32:17 +0200 Subject: JDK-8179483: Return type inference in if In-Reply-To: <1914332749.1524106.1656601972273.JavaMail.zimbra@u-pem.fr> References: <56e759b6-7ac2-10e7-3ab7-bb6021657b86@j-kuhn.de> <306137282.1472767.1656599579936.JavaMail.zimbra@u-pem.fr> <1914332749.1524106.1656601972273.JavaMail.zimbra@u-pem.fr> Message-ID: <61fa9e8f-de6f-e27e-406d-ed201997a7d6@j-kuhn.de> On 30-Jun-22 17:12, forax at univ-mlv.fr wrote: > ----- Original Message ----- >> From: "Johannes Kuhn" >> To: "Remi Forax" >> Cc: "classfile-api-dev" >> Sent: Thursday, June 30, 2022 4:44:37 PM >> Subject: Re: JDK-8179483: Return type inference in if > >> On 30-Jun-22 16:32, Remi Forax wrote: >>> ----- Original Message ----- >>>> From: "Johannes Kuhn" >>>> To: classfile-api-dev at openjdk.org >>>> Sent: Thursday, June 30, 2022 3:31:57 PM >>>> Subject: JDK-8179483: Return type inference in if >>> >>>> So, I tried to extract the jdk.classfile API and compile it using Java >>>> 18 with preview features enabled. Using Eclipse. >>>> >>>> While doing that, I encountered some compilation problems - one is the >>>> use of something similar like this: >>>> >>>> T optionValue(Classfile.Option.Key key) {...} >>>> ... >>>> if (reader.optionValue(...)) {...} >>>> >>>> According to https://bugs.openjdk.org/browse/JDK-8179483 ('if' >>>> conditions are treated like assignments by inference) this abuses a bug >>>> in javac - and is not valid java. >>>> >>>> The places where I found a use like this are: >>>> >>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java#L76 >>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/BoundAttribute.java#L150 >>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L153 >>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L208 >>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L111 >>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L209 >>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L277 >>> >>> yes, this is an annoying bug of javac, i've also encounter it in student codes >>> :) >>> >>> Here, the real question is why optionValue() has such weird signature, it means >>> there is a @SuppressWarnings("unchecked") which is not safe in the >>> implementation. >>> This kind of signature is only valid if T is always null, like in >>> Collections.emptyList(). >>> >> >> From some point of view, it provides a "nice" API - if it would return >> Object - as it should - then a cast is needed at the callsite. Which is >> IMHO not that bad, but preferences differ, and some people prefer to >> write clever code. > > It's clever until it's not, by example > var list = new ArrayList(); > list.add(optionValue(...)); > > the compiler will be happy to compile that snippet and *not* insert a cast because optionValue() says it returns a space giant and both E and T erases to Object. That... sounds wrong. It should. Javac 17 does. https://gist.github.com/DasBrain/212f36f88978b7ddd4fb6ea3a97a392e (fails with CCE) Might be a different story if it's an ArrayList (T is a type variable). > >> >> I have to admit, the use of such a method is nice - but IMHO such a >> method can only ever fulfill its contract (return something the caller >> wants, but the caller doesn't tell what it wants) if it returns `null`. > > This is a kind of okay if you are both the guy that write optionValue() and call it, otherwise it's a good way to waste people time and awake space giants :) Except that optionValue() is a public method: https://htmlpreview.github.io/?https://raw.githubusercontent.com/openjdk/jdk-sandbox/classfile-api-javadoc-branch/doc/classfile-api/javadoc/jdk/classfile/ClassReader.html#optionValue(jdk.classfile.Classfile.Option.Key) (I didn't check _that_ before my initial report.) - Johannes From forax at univ-mlv.fr Thu Jun 30 17:04:33 2022 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Thu, 30 Jun 2022 19:04:33 +0200 (CEST) Subject: JDK-8179483: Return type inference in if In-Reply-To: <61fa9e8f-de6f-e27e-406d-ed201997a7d6@j-kuhn.de> References: <56e759b6-7ac2-10e7-3ab7-bb6021657b86@j-kuhn.de> <306137282.1472767.1656599579936.JavaMail.zimbra@u-pem.fr> <1914332749.1524106.1656601972273.JavaMail.zimbra@u-pem.fr> <61fa9e8f-de6f-e27e-406d-ed201997a7d6@j-kuhn.de> Message-ID: <994546543.1636315.1656608673223.JavaMail.zimbra@u-pem.fr> ----- Original Message ----- > From: "Johannes Kuhn" > To: "Remi Forax" > Cc: "classfile-api-dev" > Sent: Thursday, June 30, 2022 5:32:17 PM > Subject: Re: JDK-8179483: Return type inference in if > On 30-Jun-22 17:12, forax at univ-mlv.fr wrote: >> ----- Original Message ----- >>> From: "Johannes Kuhn" >>> To: "Remi Forax" >>> Cc: "classfile-api-dev" >>> Sent: Thursday, June 30, 2022 4:44:37 PM >>> Subject: Re: JDK-8179483: Return type inference in if >> >>> On 30-Jun-22 16:32, Remi Forax wrote: >>>> ----- Original Message ----- >>>>> From: "Johannes Kuhn" >>>>> To: classfile-api-dev at openjdk.org >>>>> Sent: Thursday, June 30, 2022 3:31:57 PM >>>>> Subject: JDK-8179483: Return type inference in if >>>> >>>>> So, I tried to extract the jdk.classfile API and compile it using Java >>>>> 18 with preview features enabled. Using Eclipse. >>>>> >>>>> While doing that, I encountered some compilation problems - one is the >>>>> use of something similar like this: >>>>> >>>>> T optionValue(Classfile.Option.Key key) {...} >>>>> ... >>>>> if (reader.optionValue(...)) {...} >>>>> >>>>> According to https://bugs.openjdk.org/browse/JDK-8179483 ('if' >>>>> conditions are treated like assignments by inference) this abuses a bug >>>>> in javac - and is not valid java. >>>>> >>>>> The places where I found a use like this are: >>>>> >>>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java#L76 >>>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/BoundAttribute.java#L150 >>>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L153 >>>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/CodeImpl.java#L208 >>>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L111 >>>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L209 >>>>> https://github.com/openjdk/jdk-sandbox/blob/classfile-api-branch/src/java.base/share/classes/jdk/classfile/impl/DirectCodeBuilder.java#L277 >>>> >>>> yes, this is an annoying bug of javac, i've also encounter it in student codes >>>> :) >>>> >>>> Here, the real question is why optionValue() has such weird signature, it means >>>> there is a @SuppressWarnings("unchecked") which is not safe in the >>>> implementation. >>>> This kind of signature is only valid if T is always null, like in >>>> Collections.emptyList(). >>>> >>> >>> From some point of view, it provides a "nice" API - if it would return >>> Object - as it should - then a cast is needed at the callsite. Which is >>> IMHO not that bad, but preferences differ, and some people prefer to >>> write clever code. >> >> It's clever until it's not, by example >> var list = new ArrayList(); >> list.add(optionValue(...)); >> >> the compiler will be happy to compile that snippet and *not* insert a cast >> because optionValue() says it returns a space giant and both E and T erases to >> Object. > > That... sounds wrong. It should. Javac 17 does. > https://gist.github.com/DasBrain/212f36f88978b7ddd4fb6ea3a97a392e (fails > with CCE) > Might be a different story if it's an ArrayList (T is a type variable). It even more fun, it depends on the compiler, javac 1.8 is happy to not insert the cast while javac 11 insert it. Here is the bytecode generated by javac 1.8. public static void main(java.lang.String[]); Code: 0: new #2 // class java/util/ArrayList 3: dup 4: invokespecial #3 // Method java/util/ArrayList."":()V 7: astore_1 8: aload_1 9: ldc #4 // String not an Integer 11: invokestatic #5 // Method uncheckedCast:(Ljava/lang/Object;)Ljava/lang/Object; 14: invokevirtual #6 // Method java/util/ArrayList.add:(Ljava/lang/Object;)Z 17: pop 18: return and the one generated by javac 11 public static void main(java.lang.String[]); Code: 0: new #2 // class java/util/ArrayList 3: dup 4: invokespecial #3 // Method java/util/ArrayList."":()V 7: astore_1 8: aload_1 9: ldc #4 // String not an Integer 11: invokestatic #5 // Method uncheckedCast:(Ljava/lang/Object;)Ljava/lang/Object; 14: checkcast #6 // class java/lang/Integer <----------- 17: invokevirtual #7 // Method java/util/ArrayList.add:(Ljava/lang/Object;)Z 20: pop 21: return R?mi