From adam.sotona at oracle.com Wed Aug 3 15:59:38 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Wed, 3 Aug 2022 15:59:38 +0000 Subject: StackMapGenerator debug output In-Reply-To: References: Message-ID: StackMapGenerator is now providing more detailed info about root cause of the failure and appends bytecode print directly in the exception. All the critical information should be now visible in the provided exception message. Alternatively, a system variable switch can enable debug print, or it can be printed to a logger (which may cause issues with JDK bootstrap), or attached as another suppressed exception. Please test it and send me your feedback. Thanks, Adam On 21.07.2022 13:21, "Michael van Acken" wrote: Related but geared towards the builder side of things: Is there a way to print out a trace of parts fed to CodeBuilder instances? Just this morning I had Classfile die on me because of a stack underflow, and it was quite hard to find out which parts were missing from the Code attribute. And that with a Code totalling just 5 instructions... If there would have been bytes output, then I could have inspected the situation with javap. But if I mess up and pass inconsistent data to CodeBuilder, causing it to throw instead of producing a byte array, then I have an observability gap. -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Wed Aug 3 16:51:49 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Wed, 3 Aug 2022 18:51:49 +0200 Subject: StackMapGenerator debug output In-Reply-To: References: Message-ID: Am Mi., 3. Aug. 2022 um 17:59 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > StackMapGenerator is now providing more detailed info about root cause of > the failure and appends bytecode print directly in the exception. > > All the critical information should be now visible in the provided > exception message. > > Alternatively, a system variable switch can enable debug print, or it can > be printed to a logger (which may cause issues with JDK bootstrap), or > attached as another suppressed exception. > Using the prior example of CodeBuilder.ifThenElse() inserting a GOTO to nowhere, I now get this extremely helpful output: ERROR at tcljc.switch-test/int-cmp-test:45 java.lang.VerifyError: Detected branch target out of bytecode range at bytecode offset 26 of method fnbody~1(int) - max stack: 65535 max locals: 65535 attributes: [] //stack map frame @0: {locals: [int], stack: []} 0: {opcode: ILOAD_0, slot: 0} 1: {opcode: ICONST_1, constant value: 1} 2: {opcode: IADD} 3: {opcode: ISTORE_1, slot: 1} 4: {opcode: ILOAD_0, slot: 0} 5: {opcode: ICONST_2, constant value: 2} 6: {opcode: IADD} 7: {opcode: ISTORE_2, slot: 2} 8: {opcode: ILOAD_0, slot: 0} 9: {opcode: ICONST_3, constant value: 3} 10: {opcode: IADD} 11: {opcode: ISTORE_3, slot: 3} 12: {opcode: ILOAD_1, slot: 1} 13: {opcode: ILOAD_2, slot: 2} 14: {opcode: IF_ICMPGE, target: 29} 17: {opcode: ILOAD_2, slot: 2} 18: {opcode: ILOAD_3, slot: 3} 19: {opcode: IF_ICMPGE, target: 24} 22: {opcode: ICONST_1, constant value: 1} 23: {opcode: IRETURN} 24: {opcode: ICONST_0, constant value: 0} 25: {opcode: IRETURN} 26: {opcode: GOTO, target: 31} 29: {opcode: ICONST_0, constant value: 0} 30: {opcode: IRETURN} at java.base/jdk.classfile.impl.StackMapGenerator.generatorError(StackMapGenerator.java:867) at java.base/jdk.classfile.impl.StackMapGenerator.detectFrameOffsets(StackMapGenerator.java:932) at java.base/jdk.classfile.impl.StackMapGenerator.generate(StackMapGenerator.java:284) [...] Thank you very much! -- mva > > > Please test it and send me your feedback. > > > > Thanks, > > Adam > > > > > > On 21.07.2022 13:21, "Michael van Acken" > wrote: > > > > Related but geared towards the builder side of things: > > > > Is there a way to print out a trace of parts fed to CodeBuilder instances? > > > > Just this morning I had Classfile die on me because of a stack underflow, > > and it was quite hard to find out which parts were missing from the Code > > attribute. And that with a Code totalling just 5 instructions... > > > > If there would have been bytes output, then I could have inspected the > > situation with javap. But if I mess up and pass inconsistent data to > > CodeBuilder, causing it to throw instead of producing a byte array, then > > I have an observability gap. > > > > -- mva > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Thu Aug 4 08:15:19 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Thu, 4 Aug 2022 10:15:19 +0200 Subject: StackMapGenerator debug output In-Reply-To: References: Message-ID: Am Mi., 3. Aug. 2022 um 17:59 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > StackMapGenerator is now providing more detailed info about root cause of > the failure and appends bytecode print directly in the exception. > > All the critical information should be now visible in the provided > exception message. > > Alternatively, a system variable switch can enable debug print, or it can > be printed to a logger (which may cause issues with JDK bootstrap), or > attached as another suppressed exception. > I just got a basic VerifyError without further information: java.lang.VerifyError: Operand stack underflow at java.base/jdk.classfile.impl.StackMapGenerator$Frame.decStack(StackMapGenerator.java:1014) at java.base/jdk.classfile.impl.StackMapGenerator.processInvokeInstructions(StackMapGenerator.java:787) at java.base/jdk.classfile.impl.StackMapGenerator.processBlock(StackMapGenerator.java:624) at java.base/jdk.classfile.impl.StackMapGenerator.processMethod(StackMapGenerator.java:420) at java.base/jdk.classfile.impl.StackMapGenerator.generate(StackMapGenerator.java:293) at java.base/jdk.classfile.impl.StackMapGenerator.(StackMapGenerator.java:232) [...] How can I activate the debug print you mention to get more information? -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Thu Aug 4 08:25:16 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Thu, 4 Aug 2022 08:25:16 +0000 Subject: [External] : Re: StackMapGenerator debug output In-Reply-To: References: Message-ID: I just got a basic VerifyError without further information: java.lang.VerifyError: Operand stack underflow at java.base/jdk.classfile.impl.StackMapGenerator$Frame.decStack(StackMapGenerator.java:1014) How can I activate the debug print you mention to get more information? -- mva A switch or other proposals are theoretical alternative implemenations in case the debug output directly in exception message cause issues. In this case the debug print failed and the reason of the failure is accessible from the VerifyError::getSuppressed. It would be greate to get the suppressed exception or the code causing the error with no dump. Thanks, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Thu Aug 4 08:54:14 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Thu, 4 Aug 2022 10:54:14 +0200 Subject: [External] : Re: StackMapGenerator debug output In-Reply-To: References: Message-ID: Am Do., 4. Aug. 2022 um 10:25 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > > > I just got a basic VerifyError without further information: > > > > java.lang.VerifyError: Operand stack underflow > > at > java.base/jdk.classfile.impl.StackMapGenerator$Frame.decStack(StackMapGenerator.java:1014) > > > > How can I activate the debug print you mention to get more information? > > > > -- mva > > > > A switch or other proposals are theoretical alternative implemenations in > case the debug output directly in exception message cause issues. > > > > In this case the debug print failed and the reason of the failure is > accessible from the VerifyError::getSuppressed. > > It would be greate to get the suppressed exception or the code causing the > error with no dump. > The VerifyError's getSuppressed() gives me an array of length 0, even after rebuilding from the most recent jdk-sandbox commit 3902fd2558. -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Thu Aug 4 08:57:03 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Thu, 4 Aug 2022 08:57:03 +0000 Subject: [External] : Re: StackMapGenerator debug output In-Reply-To: References: Message-ID: I?ve just added fallback to hex dump, so we can debug failing debug prints ;) The VerifyError's getSuppressed() gives me an array of length 0, even after rebuilding from the most recent jdk-sandbox commit 3902fd2558. -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Thu Aug 4 09:09:50 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Thu, 4 Aug 2022 11:09:50 +0200 Subject: [External] : Re: StackMapGenerator debug output In-Reply-To: References: Message-ID: Am Do., 4. Aug. 2022 um 10:57 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > I?ve just added fallback to hex dump, so we can debug failing debug prints > ;) > Doesn't seem to make a difference. Still no additional output or suppressed exception. -- mva > > > > > > The VerifyError's getSuppressed() gives me an array of length 0, > > even after rebuilding from the most recent jdk-sandbox commit 3902fd2558. > > > > -- mva > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Thu Aug 4 10:23:31 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Thu, 4 Aug 2022 10:23:31 +0000 Subject: [External] : Re: StackMapGenerator debug output In-Reply-To: References: Message-ID: May the code be empty? On 04.08.2022 11:10, "Michael van Acken" wrote: Am Do., 4. Aug. 2022 um 10:57 Uhr schrieb Adam Sotona >: I?ve just added fallback to hex dump, so we can debug failing debug prints ;) Doesn't seem to make a difference. Still no additional output or suppressed exception. -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Thu Aug 4 11:45:27 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Thu, 4 Aug 2022 13:45:27 +0200 Subject: [External] : Re: StackMapGenerator debug output In-Reply-To: References: Message-ID: Am Do., 4. Aug. 2022 um 12:23 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > May the code be empty? > The faulty code seems to be this here, at the very beginning of a method: :accept CodeBuilder start with Load[OP=ILOAD_0, slot=0] with Invoke[OP=INVOKESTATIC, m=pkg/ns0/is-odd$rec-even.__create(ILpkg/ns0/is-odd$rec-odd;)Lpkg/ns0/is-odd$rec-even;] The lookup type of the invoke wants a second argument that does not exist. -- mva > > > > On 04.08.2022 11:10, "Michael van Acken" > wrote: > > > > Am Do., 4. Aug. 2022 um 10:57 Uhr schrieb Adam Sotona < > adam.sotona at oracle.com>: > > I?ve just added fallback to hex dump, so we can debug failing debug prints > ;) > > > > Doesn't seem to make a difference. Still no additional output or > suppressed exception. > > > > -- mva > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Thu Aug 4 13:41:34 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Thu, 4 Aug 2022 13:41:34 +0000 Subject: [External] : Re: StackMapGenerator debug output In-Reply-To: References: Message-ID: The faulty code seems to be this here, at the very beginning of a method: :accept CodeBuilder start with Load[OP=ILOAD_0, slot=0] with Invoke[OP=INVOKESTATIC, m=pkg/ns0/is-odd$rec-even.__create(ILpkg/ns0/is-odd$rec-odd;)Lpkg/ns0/is-odd$rec-even;] The lookup type of the invoke wants a second argument that does not exist. -- mva Thanks for investigation, it should be fixed now. I missed two cases, which were throwing the VerifyError directly (without attached debug info). Thanks, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Thu Aug 4 13:54:55 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Thu, 4 Aug 2022 15:54:55 +0200 Subject: [External] : Re: StackMapGenerator debug output In-Reply-To: References: Message-ID: Am Do., 4. Aug. 2022 um 15:41 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > > > The faulty code seems to be this here, at the very beginning of a method: > > > > :accept CodeBuilder start > with Load[OP=ILOAD_0, slot=0] > with Invoke[OP=INVOKESTATIC, > m=pkg/ns0/is-odd$rec-even.__create(ILpkg/ns0/is-odd$rec-odd;)Lpkg/ns0/is-odd$rec-even;] > > > > The lookup type of the invoke wants a second argument that does not exist. > > > > -- mva > > > > Thanks for investigation, it should be fixed now. > > I missed two cases, which were throwing the VerifyError directly (without > attached debug info). > Yes, now this error scenario is pinpointed flawlessly as well. Thanks again! -- mva > > > Thanks, > > Adam > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Thu Aug 4 16:00:35 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Thu, 4 Aug 2022 16:00:35 +0000 Subject: Classfile API test coverage report Message-ID: Hi, I?m glad to announce recent significant improvements in the Classfile API tests coverage, mainly in CodeBuilder. Here is actual snapshot of JCov report: https://htmlpreview.github.io/?https://raw.githubusercontent.com/openjdk/jdk-sandbox/classfile-api-javadoc-branch/jcov-report/jdk/classfile/package-summary.html And while the numbers aren't that bad, they can always be better and any contribution to improve tests is always welcome. Thanks, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From heidinga at redhat.com Thu Aug 4 20:19:21 2022 From: heidinga at redhat.com (Dan Heidinga) Date: Thu, 4 Aug 2022 16:19:21 -0400 Subject: Experience report Message-ID: Congrats to all involved in building this api! It's been easy to pick up and get started with. My use case has been transforming classes to remove the use of invokedynamic and runtime class generation for lambda expressions [0]. And I found the examples in the package javadoc made it very easy to jump into the API and get something going right away. The connection between this api and the j.l.Constants package is also very convenient as it provides an easy path all the way from runtime types to symbolic info to classfile data. The examples in the package-summary give a good starting point, though I stumbled slightly when switching from purely reading to actually transforming classfiles. I hit the "Writing classes" examples first and rat-holed a bit there before realizing that I only needed the ClassModel::transform call. It might be worth re-ordering the flow to put transformation first as in my experience that's more common than writing new classfiles. One of the areas I was modifying was the NEST_HOST and NEST_MEMBERS attributes. Adapting an attribute, like adding new members to a NEST_MEMBERS attribute, feels unpolished as it requires removing the existing attribute and then adding an entirely new one. It would feel more straightforward if there was a way to generate a new attribute from the old with additions. Some kind of ::withSymbols method maybe? I ended up reading the existing members, creating a new attribute from the existing & new members, then replacing the entire attribute with the new one. It was easy to do with a composed transform (very nice by the way!), but my code feels ugly in this case - is there a better way to do an update of an attribute like this? There are several ::of methods on NestMembersAttribute that help but it might be worth adding a ::withSymbols(NestMembersAttribute, ClassDesc... additions) api. I can create a PR if there's interest in these kinds of additions. One of the areas I was slightly concerned about was object allocation. I haven't measured it but using the APIs - especially mixing ClassEntry & j.l.constant.ClassDesc - feels like it generates a lot of temporary objects to get a set of entries into a consistent format. I needed to convert from one format to the other to be able to create new attributes - ie: read the set of ClassEntries and then convert them to ClassDesc so they fit with the ClassDesc I already had in hand. The ::with* apis additions might help here as well? Or maybe there's a better pattern for what I'm trying to do. Guidance appreciated. Composable ClassTransforms are a very nice addition in the API. After writing the code that dropped an attribute, and replaced it with an updated attribute, I was able to lift that into a composed transform that could be reused in several places. I also appreciated how small updates to the classfile are so easy, straightforward, and clear. For example, this code sets the NestHost based on a passed in ClassModel and replaces an existing byte[]: ClassModel cm = /* passed in value */ ClassModel genModel = Classfile.parse(bytes); NestHostAttribute genHost = determineNestHost(cm); bytes = genModel.transform(ClassTransform.endHandler(b -> b.with(genHost))); That's been my experience so far - I'm eager to see this API shipping in the JDK. --Dan [0] https://mail.openjdk.org/pipermail/leyden-dev/2022-August/000055.html From brian.goetz at oracle.com Thu Aug 4 21:08:19 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 4 Aug 2022 17:08:19 -0400 Subject: Experience report In-Reply-To: References: Message-ID: <8ebb41d5-c598-e825-1062-14d1fe3b150d@oracle.com> Thanks for the experience report!? Some comments inline. On 8/4/2022 4:19 PM, Dan Heidinga wrote: > Congrats to all involved in building this api! It's been easy to pick > up and get started with. Just as streams was the first API we "designed for lambdas", this is the first API we "designed for pattern matching." > The examples in the package-summary give a good starting point, though > I stumbled slightly when switching from purely reading to actually > transforming classfiles. I hit the "Writing classes" examples first > and rat-holed a bit there before realizing that I only needed the > ClassModel::transform call. It might be worth re-ordering the flow to > put transformation first as in my experience that's more common than > writing new classfiles. I think we also need a lot more examples about transformation.? The transformation API may be simple, but there is some subtlety to it and it needs a lot more examples. > One of the areas I was modifying was the NEST_HOST and NEST_MEMBERS > attributes. Adapting an attribute, like adding new members to a > NEST_MEMBERS attribute, feels unpolished as it requires removing the > existing attribute and then adding an entirely new one. It would feel > more straightforward if there was a way to generate a new attribute > from the old with additions. Some kind of ::withSymbols method maybe? This use case (and similar: adding an annotation, adding an interface, etc) was something we struggled with for a long time.? I rewrote the transformation API several times, and each time this case was on my mind.? I am hoping that what is missing here is just the example in the documentation, rather than an entire new mechanism. The basic problem is that for any of the attributes that carry "list of stuff", and you want to add an item, you don't know until the end of the iteration whether you need to create a new attribute with one thing or transform the existing attribute.? There are a few ways to do it, some require more hoop jumping than others. Note that the Transform interfaces, while SAM types, have some extra methods -- atStart(b) and atEnd(b).? These are default methods whose default implementation does nothing, so that you can have a transform lambda, but if you want to use either of these methods in your transform, you need a class (anonymous or otherwise.) You also need to manage at least one bit of state: is there an element that carries the list of stuff in the stream, or not?? And there are two ways to manage this: either use the existing model to store your state, or track the state in your transform object. The first way is simpler in that you can get someone else (the model) to do your state management, but feels a little distasteful. What you do is: ?- have the accept method always ignore the list-carrier element (e.g., NestMembers) ?- have the atEnd method query builder().original() to see if the list-carrier element is present (e.g., findAttribute()); if it is present, get it and transform it and send it downstream, and if is not present, generate a new one. ??? new ClassTransform() { ??????? void accept(ClassBuilder b, ClassElement e) { ??????????? switch (e) { ??????????????? case NestMembersAttribute a -> break; ??????????????? ... other transform stuff ... ??????????? } ??????? } ??????? void atEnd(ClassBuilder b) { ??????????? Optional a = b.original().get().findAttribute(Attributes.NEST_MEMBERS); ??????????? if (a.isPresent()) { ??????????????? // transform a, send it to builder ??????????? } ??????????? else { ??????????????? // make new attribute, send it to builder ??????????? } ??????? } ??? } This is straightforward enough, but it feels a little disconnected. The other way involves a _stateful_ transform, where there is state in the transform class.? This feels a little more honest, but it means you have to use the XxxTransform::ofStateful factory.? Here, you transform elements as you go, and you record whether you saw the NestMembers attribute, and if you didn't, you emit a fresh one in the atEnd: ??? () -> new ClassTransform() { ??????? boolean foundNM = false; ??????? void accept(ClassBuilder b, ClassElement e) { ??????????? switch (e) { ??????????????? case NestMembersAttribute a -> { transform a; foundNM = true; } ??????????????? ... other transform stuff ... ??????????? } ??????? } ??????? void atEnd(ClassBuilder b) { ??????????? if (!foundNM) ??????????????? builder.accept(NestMembersAttribute.of(...)); ??????? } ??? } > I ended up reading the existing members, creating a new attribute from > the existing & new members, then replacing the entire attribute with > the new one. It was easy to do with a composed transform (very nice > by the way!), but my code feels ugly in this case - is there a better > way to do an update of an attribute like this? Does the second example above seem better? > There are several ::of methods on NestMembersAttribute that help but > it might be worth adding a ::withSymbols(NestMembersAttribute, > ClassDesc... additions) api. I can create a PR if there's interest in > these kinds of additions. Yes, we are interested in usability feedback like this.? A PR is a good place to start.? I am interested in ensuring that we apply some relatively uniform principles across the API about what factories we provide, though, to avoid whack-a-mole in the future.? So if we are missing some overloads for NestMembersAttribute::of, we're probably also missing the same in Interfaces and other places too. > One of the areas I was slightly concerned about was object allocation. > I haven't measured it but using the APIs - especially mixing > ClassEntry & j.l.constant.ClassDesc - feels like it generates a lot of > temporary objects to get a set of entries into a consistent format. > I needed to convert from one format to the other to be able to create > new attributes - ie: read the set of ClassEntries and then convert > them to ClassDesc so they fit with the ClassDesc I already had in > hand. The ::with* apis additions might help here as well? Or maybe > there's a better pattern for what I'm trying to do. Guidance > appreciated. Clearly we've opted into a higher degree of cost relative to ASM on this.? (On the other hand, we picked up a lot both from laziness and from constant pool sharing, so some of this cancels out.)? The basic principle is that there are both "close to the format" and "convenience" representations for both things, such as ClassEntry and ClassDesc.? (We struggled over whether to admit strings at all, and are currently biased against using strings for anything other than names, because anyone who has used ASM a lot has made the "com/foo/Bar" vs "com.foo.Bar" vs "Lcom/foo/Bar;" mistake at least a hundred bajillion times.)? But a ClassDesc is more expensive to make and manipulate than a string. So the advice is to get things into the most close-to-the-representation format as early as possible (e.g., ClassEntry).? And ClassEntry has both asSymbol() (a ClassDesc) and asInternalName() (a String), so you don't have to go through ClassDesc when transforming.? Adam is also adding ClassDesc::fromInternalName as an alternate factory to ClassDesc. My sense is that we have not quite finished rationalizing representations when it comes to transforming.? If you find yourself having to convert a lot of stuff to ClassDesc and back when transforming, either there's a better way to do it, or we should expose a better way to do it.? Hopefully one that doesn't leave users in the position of guessing "what kind of string is this." > Composable ClassTransforms are a very nice addition in the API. After > writing the code that dropped an attribute, and replaced it with an > updated attribute, I was able to lift that into a composed transform > that could be reused in several places. This was the holy grail I was seeking, and it took several iterations of the API to get there.? I'm glad we got there. From brian.goetz at oracle.com Thu Aug 4 21:19:30 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 4 Aug 2022 17:19:30 -0400 Subject: Experience report In-Reply-To: <8ebb41d5-c598-e825-1062-14d1fe3b150d@oracle.com> References: <8ebb41d5-c598-e825-1062-14d1fe3b150d@oracle.com> Message-ID: <3a9fe71e-8585-efe8-035b-1feb4bdd3e5e@oracle.com> > >> One of the areas I was slightly concerned about was object allocation. There are a few JMH micros that I found helpful for getting a sense of performance comparison.? The main one is "AdaptNull", which runs over a minor corpus of classfiles and runs various kinds of "do nothing" transforms that vary in terms of how deep they iterate or whether they share the CP.? The most representative ones are "SHARED_3" and "SHARED_3_NO_DEBUG", which do a three-level iteration (dive into methods, and again into code) with a shared CP.? These are both faster than the comparable ASM (ASM_3). -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Sun Aug 7 13:27:20 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Sun, 7 Aug 2022 15:27:20 +0200 Subject: Another experience report Message-ID: Here is another experience report, although in the more pedestrian setting of a compiler with Clojure as input. Over the past 4 weeks I moved an incomplete project of mine from a homegrown bytecode emitter to Classfile. I have now reached feature parity with the old codebase, and classfile generation is essentially complete. ### Reading Sole users of Classfile.parse() are the unit tests. They take a sequence of byte arrays and wrangle them into the data format expected by the existing test cases. In case of an expected/actual mismatch, the nested data is diffed and visualised side by side. Some minor work was involved in assigning nice names to labels, and to slide LocalVariable and ExceptionCatch to their prior places in the instruction sequences. The rest of the mapping essentially wrote itself by following the API's guidance. Because of the high density of instanceof checks on instruction types, this was the one place where I wished for shorter class names. ### Writing Manually writing parts of a class or a method's Code attribute is a pleasure. Concise, easy to write, and easy to read. Similarly, going from the intermediate representation to Classfile handlers is a breeze. I find the ability to represent bytecode instructions as immutable data especially useful. It allows me to stash away "simple" opcodes in a uniform way in the IR early while parsing the source code, and have them emit themselves when writing out the class. A single node class of the intermediate representation is sufficient to cover the bulk of opcodes (currently with a single exception; see below). Another huge boon is `block()`. I call it just once in my code base, but it reduces the work of managing local slots, their lifetimes, and their debug information by an enormous amount. With regard to performance, I can only say that it is fast. For example, turnaround time for the unit tests is usually below 150ms and it is the best I have managed so far. These tests encompass 2.3k build() and 1.5k parse() of class files on virtual threads, with file size ranging from tiny to small. ### Lower than I like The bulk of API usage stays on a single level of abstraction, for example working with *Desc entry points instead of *Entry. There are rare places where this slipped at little. The New*Array family of instructions gave me a hard time. I had forgotten that there are three of those, and there was some bumping around involved while re-learning this fact. (Btw, anewarray() accepts a primitive ClassDesc and converts it into a reference type, e.g. "I" to "LI;".) Two of the instructions have no ClassDesc factory, which meant I had to wrap the family into a intermediate representation node to carry essentially (ClassDesc,int) downstream to the CodeBuilder instance. I wonder if it is worthwhile to move the three under an umbrella interface, similar to what ConstantInstruction is doing. The one place where I use labelToBci() is try/catch/finally. There is the special case of exceptionCatch() failing for an empty region, a condition that in turn can lead to handler blocks becoming unreachable. For me, the only robust way to deal with this to a) guard against an empty region by inspecting the bcis and b) subsequently omitting the invalid/unreachable parts. Another single use only is constantPool(), to go from a DMHD instance to CodeBuilder's (field|invoke)Instruction. This was a consequence of DMHD only providing the lookupDescriptor() as String and not as an MTD as well. With hindsight, it may have been better for me to recover the MTD from the String regardless, and to stay on the level of *Desc throughout. Finally, is there a way to decide between tableswitch and lookupswitch? Lacking something better, I'm trying to emulate this code here: https://github.com/openjdk/jdk-sandbox/blob/master/src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/Gen.java#L1320 ### Lost in translation One feature I cannot duplicate with Classfile is try/catch/finally in expression position when the operand stack is not empty. The old bytecode generator dealt with this case by unwinding the operand stack into locals, evaluating the t/c/f, and then rebuilding the operand stack with the result on top. But to do this, one needs to know what the operand stack looks like at the point of the `try`. Echoing Dan's sentiment, I'm also looking forward to Classfile being part of the JDK. -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Sun Aug 7 15:46:23 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Sun, 7 Aug 2022 11:46:23 -0400 Subject: Another experience report In-Reply-To: References: Message-ID: Thanks again for the thoughtful feedback. > Because of the high density > of instanceof checks on instruction types, this was the one place > where I wished for shorter class names. We did consider a few alternatives to the XxxInstruction convention, such as XxxOp.? I'm open to rationalizing names at this point; the current conventions were chosen early on to be clear and consistent, so we could move on to actually implementing the right idioms, but it is reasonable to circle back and question naming (for some bounded time) now that the shape of the API is more settled. (Irritating confounding factor: if you name the builder methods load(), store(), etc, when you get to "return" and "instanceof", you have a problem...) > Another huge boon is `block()`.? I call it just once in my code base, > but it reduces the work of managing local slots, their lifetimes, and > their debug information by an enormous amount. These were added relatively late, especially the local-variable management, and we could especially use more feedback (and test coverage!) on that.? There are some missing bits here too, since `allocateLocal` doesn't automatically generate LVT/LVTT entries, and it should.? Feedback and contribution is welcome here. > With regard to performance, I can only say that it is fast. I'm sure authors of other frameworks will take that as "fighting words", but yes, it is likely to be somewhat surprising to many that such an allocation-heavy approach offers this degree of "fast enough" for most use cases. > ### Lower than I like > > The bulk of API usage stays on a single level of abstraction, for > example working with *Desc entry points instead of *Entry. There are > rare places where this slipped at little. I'm not following what you're getting at here.? Do you mean there are gaps where we are not consistent about choices of Desc vs Entry?? Or simply that you stuck to the Desc level as a user?? And if so, was there friction here? > The New*Array family of instructions gave me a hard time.? I had > forgotten that there are three of those, and there was some bumping > around involved while re-learning this fact. ?(Btw, anewarray() accepts > a primitive ClassDesc and converts it into a reference type, e.g. "I" > to "LI;".) ?Two of the instructions have no ClassDesc factory, which > meant I had to wrap the family into a intermediate representation node > to carry essentially (ClassDesc,int) downstream to the CodeBuilder > instance.? I wonder if it is worthwhile to move the three under an > umbrella interface, similar to what ConstantInstruction is doing. Quite possibly.? These instructions gave us a hard time too, because they are so irregular in how they work.? It is quite possible that there is a simplification here.? The constant instructions have a similar abstraction-resistence, and we did go around a few times with how much to lump and split there too.? The main constraint is that the lowest level modeling should map directly to the classfile spec, which means any creativity around merging representations has to happen one level up. When designing the groupings of instructions, there are a lot of forces pulling in different directions.? The main normalizing force is that transformation is an emergent property of reading+writing, and we want for the no-op transformation to do as little "lifting and lowering" work as possible. This is why no XxxInstruction uses XxxDesc in its data model; converting from CP entry -> descriptor string -> XxxDesc -> descriptor string -> CP entry for a no-op transform is expensive and pointless.? We have to worry about this because the default representation is a stream of element; if the elements are "too far" from what's in the classfile, then the lifting and lowering overhead will eat you. If you try to merge semi-related instructions like the constant or new-array instructions into a common representation, that representation will not be the natural representation for some or all of the instructions in the group.? When we hit walls like this, we fall back to splitting, and modeling the classfile spec directly, but there is probably more we can do atop them.? Suggestions welcome. > The one place where I use labelToBci() is try/catch/finally.? There is > the special case of exceptionCatch() failing for an empty region, a > condition that in turn can lead to handler blocks becoming > unreachable.? For me, the only robust way to deal with this to a) > guard against an empty region by inspecting the bcis and b) > subsequently omitting the invalid/unreachable parts. This raises a good question about how much the library wants to do to "fix" questionable bytecode.? We already NOP out unreachable bytecode (otherwise the verifier freaks out).? Should we just silently drop catch clauses associated with empty try blocks?? (We won't know that they are empty until after all the labels are resolved, so we can't usually detect this at the point of emitting the catch entry.)? What about when, by the time we get to the end of generation, a label used in a try-catch, or LVT[T], isn't bound? Should we throw, or just drop the entry?? I suspect one size does not fit all here and we have to design some more options-handling. > Another single use only is constantPool(), to go from a DMHD instance > to CodeBuilder's (field|invoke)Instruction.? This was a consequence of > DMHD only providing the lookupDescriptor() as String and not as an MTD > as well.? With hindsight, it may have been better for me to recover > the MTD from the String regardless, and to stay on the level of *Desc > throughout. Is there something missing that would bridge that for you? > Finally, is there a way to decide between tableswitch and > lookupswitch?? Lacking something better, I'm trying to emulate this > code here: > https://github.com/openjdk/jdk-sandbox/blob/master/src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/Gen.java#L1320 Most compilers use a heuristic based on the size of the two alternatives, comparing `(hi-lo) / count` to some density threshold.? You're mostly optimizing for bytecode size here, since when the JIT gets its hands on it, it has its own heuristics. > ### Lost in translation > > One feature I cannot duplicate with Classfile is try/catch/finally in > expression position when the operand stack is not empty.? The old > bytecode generator dealt with this case by unwinding the operand stack > into locals, evaluating the t/c/f, and then rebuilding the operand > stack with the result on top.? But to do this, one needs to know what > the operand stack looks like at the point of the `try`. Interesting point.? We do not build stack maps as we go, so we don't have our hands readily on the stack state.? However, I could imagine an overload of the try-catch builder that would let you feed it a TypeKind[], and that would use allocateLocal to automate the push/pop logic.? This is something you should be able to build from outside the library, too; this would be a good experiment try try. (You'd have to manually compute the stack state.) Cheers, -Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Sun Aug 7 17:45:47 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Sun, 7 Aug 2022 19:45:47 +0200 Subject: Another experience report In-Reply-To: References: Message-ID: Am So., 7. Aug. 2022 um 17:46 Uhr schrieb Brian Goetz < brian.goetz at oracle.com>: > ### Lower than I like > > The bulk of API usage stays on a single level of abstraction, for > example working with *Desc entry points instead of *Entry. There are > rare places where this slipped at little. > > > I'm not following what you're getting at here. Do you mean there are gaps > where we are not consistent about choices of Desc vs Entry? Or simply that > you stuck to the Desc level as a user? And if so, was there friction here? > This was intended as a comment on a common theme of the following four points, where a small gap in the API or in my understanding forced me slightly out of the lane in which I would have preferred to stay. As it is, I find Classfile highly regular and predictable, even in this early state. The one exception that comes to mind is two New*Array instructions not having a ClassDesc-based of() factory. > The one place where I use labelToBci() is try/catch/finally. There is > the special case of exceptionCatch() failing for an empty region, a > condition that in turn can lead to handler blocks becoming > unreachable. For me, the only robust way to deal with this to a) > guard against an empty region by inspecting the bcis and b) > subsequently omitting the invalid/unreachable parts. > > > This raises a good question about how much the library wants to do to > "fix" questionable bytecode. We already NOP out unreachable bytecode > (otherwise the verifier freaks out). Should we just silently drop catch > clauses associated with empty try blocks? (We won't know that they are > empty until after all the labels are resolved, so we can't usually detect > this at the point of emitting the catch entry.) What about when, by the > time we get to the end of generation, a label used in a try-catch, or > LVT[T], isn't bound? Should we throw, or just drop the entry? I suspect > one size does not fit all here and we have to design some more > options-handling. > I've currently disabled NOP-ing because I want to know about unreachable code early. If I find a situation where I cannot prevent such code with reasonable effort, I will have to revert to the default behaviour. exceptionCatch() on an empty region is an example for a situation that I cannot detect with reasonable effort upfront, and the default of throwing helped me to think through what is happening there and about the potential consequences. >From my limited experience with ifThenElse(), the higher level block-based entry points are in a better position to produce semantically equivalent code ("do what I mean") instead of just passing through an instruction stream verbatim ("do as I say"). This probably requires exclusive control over both sides of the involved labels' contract, position marking and position targeting/consumption. Another single use only is constantPool(), to go from a DMHD instance > to CodeBuilder's (field|invoke)Instruction. This was a consequence of > DMHD only providing the lookupDescriptor() as String and not as an MTD > as well. With hindsight, it may have been better for me to recover > the MTD from the String regardless, and to stay on the level of *Desc > throughout. > > > Is there something missing that would bridge that for you? > I have many uses of DMHD: handover points to the runtime are stored as DMHD, Java interop goes from Member to DMHD and then onward, every function arity gets eventually assigned one or two DMHD. The need to generate a field or invoke instruction from DMHD-like data arises quite often, and a CodeBuilder method to facilitate this from DMHD components feels kind of natural. This needs a mapping from DMHD$Kind to Opcode, and DMHD exposing its lookup as MTD would allow to keep this on the *Desc level. Right now I am using these two function variants: (defn invoke (^CodeBuilder [^CodeBuilder xb ^DirectMethodHandleDesc$Kind kind ^ClassDesc owner ^String method-name ^String lookup-descriptor ^boolean owner-interface?] (let [cp (.constantPool xb) owner (.classEntry cp owner) nm (.utf8Entry cp method-name) tp (.utf8Entry cp lookup-descriptor) nat (.natEntry cp nm tp) opc (case (.refKind kind) #_REF_getField 1 Opcode/GETFIELD #_REF_getStatic 2 Opcode/GETSTATIC #_REF_putField 3 Opcode/PUTFIELD #_REF_putStatic 4 Opcode/PUTSTATIC #_REF_invokeVirtual 5 Opcode/INVOKEVIRTUAL #_REF_invokeStatic 6 Opcode/INVOKESTATIC #_REF_invokeSpecial 7 Opcode/INVOKESPECIAL #_REF_newInvokeSpecial 8 Opcode/INVOKESPECIAL #_REF_invokeInterface 9 Opcode/INVOKEINTERFACE)] (if (< (.refKind kind) #_REF_invokeVirtual 5) (.fieldInstruction xb opc (.fieldRefEntry cp owner nat)) (.invokeInstruction xb opc (if owner-interface? (.interfaceMethodRefEntry cp owner nat) (.methodRefEntry cp owner nat)))))) (^CodeBuilder [^CodeBuilder xb ^DirectMethodHandleDesc mhd] (invoke xb (.kind mhd) (.owner mhd) (.methodName mhd) (.lookupDescriptor mhd) (.isOwnerInterface mhd)))) > Finally, is there a way to decide between tableswitch and > lookupswitch? Lacking something better, I'm trying to emulate this > code here: > > https://github.com/openjdk/jdk-sandbox/blob/master/src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/Gen.java#L1320 > > > Most compilers use a heuristic based on the size of the two alternatives, > comparing `(hi-lo) / count` to some density threshold. You're mostly > optimizing for bytecode size here, since when the JIT gets its hands on it, > it has its own heuristics. > Understood. It would be nice if Classfile would offer some "blessed" or just reasonable heuristic here. > > ### Lost in translation > > One feature I cannot duplicate with Classfile is try/catch/finally in > expression position when the operand stack is not empty. The old > bytecode generator dealt with this case by unwinding the operand stack > into locals, evaluating the t/c/f, and then rebuilding the operand > stack with the result on top. But to do this, one needs to know what > the operand stack looks like at the point of the `try`. > > > Interesting point. We do not build stack maps as we go, so we don't have > our hands readily on the stack state. However, I could imagine an overload > of the try-catch builder that would let you feed it a TypeKind[], and that > would use allocateLocal to automate the push/pop logic. This is something > you should be able to build from outside the library, too; this would be a > good experiment try try. (You'd have to manually compute the stack state.) > I'm currently experimenting with minimal stack tracking, basically an approximated flag "no stack operands" passed down during parsing. If a try is reached without this flag being present, it is wrapped in a no-argument closure and called. My hope is that this is much easier to get right than accurate stack tracking. --mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From heidinga at redhat.com Mon Aug 15 20:26:20 2022 From: heidinga at redhat.com (Dan Heidinga) Date: Mon, 15 Aug 2022 16:26:20 -0400 Subject: Experience report In-Reply-To: <8ebb41d5-c598-e825-1062-14d1fe3b150d@oracle.com> References: <8ebb41d5-c598-e825-1062-14d1fe3b150d@oracle.com> Message-ID: Thanks for the feedback, Brian. I've worked through the suggestions on the various ways to update an Attribute {use a stateful ClassTransform, use builder.original().get()} and found the "original().get()" approach felt the cleanest and clearest for what I was doing. While working on that, I also added the ::with & ::withSymbols methods I had proposed and opened a PR [0] to see if the fleshed out API fit with the general way you wanted to evolve this. If that approach looks good, then I can extend it to the other attributes that may benefit from similar methods. (See the PR for a list) The code using the new methods looks like: ClassTransform.endHandler(b -> { NestMembersAttribute newAttribute = b.original().get().findAttribute(Attributes.NEST_MEMBERS) .map(nma -> NestMembersAttribute.withSymbols(nma, nestMembers)) .orElse(NestMembersAttribute.ofSymbols(nestMembers)); b.with(newAttribute); } which ends up being a pretty straightforward way to split between creating a new attribute for just the new items creating a new attribute from the combo of old + additions. --Dan [0] https://github.com/openjdk/jdk-sandbox/pull/31 On Thu, Aug 4, 2022 at 5:08 PM Brian Goetz wrote: > Thanks for the experience report! Some comments inline. > > On 8/4/2022 4:19 PM, Dan Heidinga wrote: > > Congrats to all involved in building this api! It's been easy to pick > > up and get started with. > > Just as streams was the first API we "designed for lambdas", this is the > first API we "designed for pattern matching." > > > The examples in the package-summary give a good starting point, though > > I stumbled slightly when switching from purely reading to actually > > transforming classfiles. I hit the "Writing classes" examples first > > and rat-holed a bit there before realizing that I only needed the > > ClassModel::transform call. It might be worth re-ordering the flow to > > put transformation first as in my experience that's more common than > > writing new classfiles. > > I think we also need a lot more examples about transformation. The > transformation API may be simple, but there is some subtlety to it and > it needs a lot more examples. > > > One of the areas I was modifying was the NEST_HOST and NEST_MEMBERS > > attributes. Adapting an attribute, like adding new members to a > > NEST_MEMBERS attribute, feels unpolished as it requires removing the > > existing attribute and then adding an entirely new one. It would feel > > more straightforward if there was a way to generate a new attribute > > from the old with additions. Some kind of ::withSymbols method maybe? > > This use case (and similar: adding an annotation, adding an interface, > etc) was something we struggled with for a long time. I rewrote the > transformation API several times, and each time this case was on my > mind. I am hoping that what is missing here is just the example in the > documentation, rather than an entire new mechanism. > > The basic problem is that for any of the attributes that carry "list of > stuff", and you want to add an item, you don't know until the end of the > iteration whether you need to create a new attribute with one thing or > transform the existing attribute. There are a few ways to do it, some > require more hoop jumping than others. > > Note that the Transform interfaces, while SAM types, have some extra > methods -- atStart(b) and atEnd(b). These are default methods whose > default implementation does nothing, so that you can have a transform > lambda, but if you want to use either of these methods in your > transform, you need a class (anonymous or otherwise.) > > You also need to manage at least one bit of state: is there an element > that carries the list of stuff in the stream, or not? And there are two > ways to manage this: either use the existing model to store your state, > or track the state in your transform object. > > The first way is simpler in that you can get someone else (the model) to > do your state management, but feels a little distasteful. What you do is: > > - have the accept method always ignore the list-carrier element (e.g., > NestMembers) > - have the atEnd method query builder().original() to see if the > list-carrier element is present (e.g., findAttribute()); if it is > present, get it and transform it and send it downstream, and if is not > present, generate a new one. > > new ClassTransform() { > void accept(ClassBuilder b, ClassElement e) { > switch (e) { > case NestMembersAttribute a -> break; > ... other transform stuff ... > } > } > > void atEnd(ClassBuilder b) { > Optional a = > b.original().get().findAttribute(Attributes.NEST_MEMBERS); > if (a.isPresent()) { > // transform a, send it to builder > } > else { > // make new attribute, send it to builder > } > } > } > > This is straightforward enough, but it feels a little disconnected. > > The other way involves a _stateful_ transform, where there is state in > the transform class. This feels a little more honest, but it means you > have to use the XxxTransform::ofStateful factory. Here, you transform > elements as you go, and you record whether you saw the NestMembers > attribute, and if you didn't, you emit a fresh one in the atEnd: > > > () -> new ClassTransform() { > boolean foundNM = false; > > void accept(ClassBuilder b, ClassElement e) { > switch (e) { > case NestMembersAttribute a -> { transform a; foundNM = > true; } > ... other transform stuff ... > } > } > > void atEnd(ClassBuilder b) { > if (!foundNM) > builder.accept(NestMembersAttribute.of(...)); > } > } > > > I ended up reading the existing members, creating a new attribute from > > the existing & new members, then replacing the entire attribute with > > the new one. It was easy to do with a composed transform (very nice > > by the way!), but my code feels ugly in this case - is there a better > > way to do an update of an attribute like this? > > Does the second example above seem better? > > > There are several ::of methods on NestMembersAttribute that help but > > it might be worth adding a ::withSymbols(NestMembersAttribute, > > ClassDesc... additions) api. I can create a PR if there's interest in > > these kinds of additions. > > Yes, we are interested in usability feedback like this. A PR is a good > place to start. I am interested in ensuring that we apply some > relatively uniform principles across the API about what factories we > provide, though, to avoid whack-a-mole in the future. So if we are > missing some overloads for NestMembersAttribute::of, we're probably also > missing the same in Interfaces and other places too. > > > One of the areas I was slightly concerned about was object allocation. > > I haven't measured it but using the APIs - especially mixing > > ClassEntry & j.l.constant.ClassDesc - feels like it generates a lot of > > temporary objects to get a set of entries into a consistent format. > > I needed to convert from one format to the other to be able to create > > new attributes - ie: read the set of ClassEntries and then convert > > them to ClassDesc so they fit with the ClassDesc I already had in > > hand. The ::with* apis additions might help here as well? Or maybe > > there's a better pattern for what I'm trying to do. Guidance > > appreciated. > > Clearly we've opted into a higher degree of cost relative to ASM on > this. (On the other hand, we picked up a lot both from laziness and > from constant pool sharing, so some of this cancels out.) The basic > principle is that there are both "close to the format" and "convenience" > representations for both things, such as ClassEntry and ClassDesc. (We > struggled over whether to admit strings at all, and are currently biased > against using strings for anything other than names, because anyone who > has used ASM a lot has made the "com/foo/Bar" vs "com.foo.Bar" vs > "Lcom/foo/Bar;" mistake at least a hundred bajillion times.) But a > ClassDesc is more expensive to make and manipulate than a string. > > So the advice is to get things into the most close-to-the-representation > format as early as possible (e.g., ClassEntry). And ClassEntry has both > asSymbol() (a ClassDesc) and asInternalName() (a String), so you don't > have to go through ClassDesc when transforming. Adam is also adding > ClassDesc::fromInternalName as an alternate factory to ClassDesc. > > My sense is that we have not quite finished rationalizing > representations when it comes to transforming. If you find yourself > having to convert a lot of stuff to ClassDesc and back when > transforming, either there's a better way to do it, or we should expose > a better way to do it. Hopefully one that doesn't leave users in the > position of guessing "what kind of string is this." > > > Composable ClassTransforms are a very nice addition in the API. After > > writing the code that dropped an attribute, and replaced it with an > > updated attribute, I was able to lift that into a composed transform > > that could be reused in several places. > > This was the holy grail I was seeking, and it took several iterations of > the API to get there. I'm glad we got there. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Tue Aug 16 14:28:10 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 16 Aug 2022 10:28:10 -0400 Subject: Experience report In-Reply-To: References: <8ebb41d5-c598-e825-1062-14d1fe3b150d@oracle.com> Message-ID: <6da06023-3d91-d0bd-55c2-b6f868eddf9a@oracle.com> On 8/15/2022 4:26 PM, Dan Heidinga wrote: > Thanks for the feedback, Brian. > > I've worked through the suggestions on the various ways to update an > Attribute {use a stateful ClassTransform, use > builder.original().get()} and found the "original().get()" approach > felt the cleanest and clearest for what I was doing. By that, do you mean "cleanest of the terrible options", or "seemed a good enough option"? > While working on that, I also added the ::with & ::withSymbols methods > I had proposed and opened a PR [0] to see if the fleshed out API fit > with the general way you wanted to evolve this. I had a look at the API suggestions and I have a counter-suggestion. The operation that you are trying to enable is adding to (and I predict eventually, removing from) list-holding attributes.? You propose four new methods for each attribute: ??? NMA with(NMA base, List additions) ??? NMA with(NMA base, CE... additions) ??? NMA withSymbols(NMA base, List additions) ??? NMA withSymbols(NMA base, CD... additions) So that's 2 representations (CE/CD) x 2 API styles (list, varargs) x N attributes (4N), which will get worse if the attribute is not merely a holder for a list, but might have other stuff to be modified too.? If you want to support removals, that's 2x more. Really what you need is four methods, somewhere: ??? List adding(List, List) ??? List adding(List, CE...) ??? List addingSymbols(List, List) ??? List addingSymbols(List, CD...) and invoke it like: ??? case NMA a -> NMA.of(adding(a.nestMembers(), NEW_STUFF)); And this would cover all attributes, plus would support dealing with attributes that have more than just a List in them.? The question is where to put these to make them discoverable. From paul.sandoz at oracle.com Tue Aug 16 23:44:37 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 16 Aug 2022 23:44:37 +0000 Subject: Another experience report In-Reply-To: References: Message-ID: <67140AA9-AEDE-4FD5-9067-C0DA65DFBF0B@oracle.com> We started work on a high-level builder for constructing try/catch, and I think could help just like ifThenElse, as you mention. See CodeBuilder.trying. This is WIP. We want to experiment with supporting finally blocks, which means inlining the code of the finally block by replying the lambda before any exit points of the try/catch blocks. At the moment if one builds an empty try block with the high-level builder API then it throws an ISE. A more friendly approach is to drop the code in the catch blocks and inline code of the finally block, if any. Which is what the Java compiler will do for such source. A good yard-stick here is to generate similar code to that produced by the Java compiler for similar code structure. Label-wise exposing the ?break? from block can be very useful, see BlockCodeBuilder.breakLabel. If/when we add looping support exposing a ?continue? label will also be useful. Paul. > On Aug 7, 2022, at 10:45 AM, Michael van Acken wrote: > >> The one place where I use labelToBci() is try/catch/finally. There is >> the special case of exceptionCatch() failing for an empty region, a >> condition that in turn can lead to handler blocks becoming >> unreachable. For me, the only robust way to deal with this to a) >> guard against an empty region by inspecting the bcis and b) >> subsequently omitting the invalid/unreachable parts. > > This raises a good question about how much the library wants to do to "fix" questionable bytecode. We already NOP out unreachable bytecode (otherwise the verifier freaks out). Should we just silently drop catch clauses associated with empty try blocks? (We won't know that they are empty until after all the labels are resolved, so we can't usually detect this at the point of emitting the catch entry.) What about when, by the time we get to the end of generation, a label used in a try-catch, or LVT[T], isn't bound? Should we throw, or just drop the entry? I suspect one size does not fit all here and we have to design some more options-handling. > > I've currently disabled NOP-ing because I want to know about unreachable code early. > If I find a situation where I cannot prevent such code with reasonable effort, I will have > to revert to the default behaviour. exceptionCatch() on an empty region is an example > for a situation that I cannot detect with reasonable effort upfront, and the default of > throwing helped me to think through what is happening there and about the potential > consequences. > > From my limited experience with ifThenElse(), the higher level block-based entry > points are in a better position to produce semantically equivalent code ("do what I > mean") instead of just passing through an instruction stream verbatim ("do as I say"). > This probably requires exclusive control over both sides of the involved labels' > contract, position marking and position targeting/consumption. ? > > >> ### Lost in translation >> >> One feature I cannot duplicate with Classfile is try/catch/finally in >> expression position when the operand stack is not empty. The old >> bytecode generator dealt with this case by unwinding the operand stack >> into locals, evaluating the t/c/f, and then rebuilding the operand >> stack with the result on top. But to do this, one needs to know what >> the operand stack looks like at the point of the `try`. > > Interesting point. We do not build stack maps as we go, so we don't have our hands readily on the stack state. However, I could imagine an overload of the try-catch builder that would let you feed it a TypeKind[], and that would use allocateLocal to automate the push/pop logic. This is something you should be able to build from outside the library, too; this would be a good experiment try try. (You'd have to manually compute the stack state.) > > I'm currently experimenting with minimal stack tracking, basically an approximated > flag "no stack operands" passed down during parsing. If a try is reached without > this flag being present, it is wrapped in a no-argument closure and called. My > hope is that this is much easier to get right than accurate stack tracking. > > --mva > > From adam.sotona at oracle.com Wed Aug 17 15:45:22 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Wed, 17 Aug 2022 15:45:22 +0000 Subject: RFR: Classfile API stack maps processing Message-ID: Hi, I would like to ask for review of proposed change in the Classfile API stack maps processing. Primary motivation is to better align stack maps processing with the Classfile API and to allow manual constructions and transformations of stack maps. Pull request for review is here: https://github.com/openjdk/jdk-sandbox/pull/32 And it includes following changes: * New boolean Classfile.Option::processStackMaps with default to false has been added * StackMapTableAttribute become a CodeElement processed when the above option is explicitly enabled by user * StackMapTableAttribute, StackMapFrameInfo, VerificationTypeInfo, SimpleVerificationTypeInfo, ObjectVerificationTypeInfo and UninitializedVerificationTypeInfo have been simplified, refactored to use Labels instead of offsets and equipped with relevant static factory methods * Redundant content of the StackMapFrameInfo and its sub-classes reflecting compressed frame forms have been removed * UnboundStackMapTableAttribute with proper serialization has been implemented * CorpusTest with RebuildingTransformations has been extended to process stack maps using the new API New stack maps processing option affects inflation of stack maps and appearance of StackMapTableAttribute in the CodeElement stream. Enabled stack maps processing inflates and decompresses StackMapTableAttribute with translated offsets to Labels and pass it to CodeTransformation(s). User can individually filter StackMapTableAttribute out or transform it manually when explicitly enabled stack maps processing. User-provided or transformed StackMapTableAttribute passed to CodeBuilder overrides stack maps generation process for the actual method. Compressed source form of the parsed stack map frame can be identified from StackMapFrameInfo::tag All new created StackMapFrameInfos always represent full frames with complete list of locals and stack. Compression of StackMapTableAttribute frames is contextual operation and it is applied at the writing phase. Any NO-OP transformation of inflated and decompressed StackMapTableAttribute may result in a different compressed form. Performance boost tricks by-passing stack maps generation for unchanged methods are unaffected by this enhancement. Thanks for your comments, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Wed Aug 17 15:53:23 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 17 Aug 2022 11:53:23 -0400 Subject: RFR: Classfile API stack maps processing In-Reply-To: References: Message-ID: <16040899-0768-c4be-3c08-dac2edc96d8a@oracle.com> See the RFR, I think I found a place where we might end up putting TWO stack map attributes into the builder. With this patch, I believe the treatment of stack maps is intended to be: ?? if (generation not disabled) { ?????? try to reuse original stackmap, if transforming and code array/exception table is unchanged ?????? otherwise generate stack map ?? } ?? else { ?????? if a stack map has been sent to the builder, put that in the class file ?? } I am not sure this is the effect, though; since the user can send stack maps down the pipe and they are stored as attributes regardless of options, I think these may be written to the classfile even if we are in generation mode. On 8/17/2022 11:45 AM, Adam Sotona wrote: > > Hi, > > I would like to ask for review of proposed change in the Classfile API > stack maps processing. > > Primary motivation is to better align stack maps processing with the > Classfile API and to allow manual constructions and transformations of > stack maps. > > Pull request for review is here: > https://github.com/openjdk/jdk-sandbox/pull/32 > > And it includes following changes: > > * New boolean Classfile.Option::processStackMaps with default to > false has been added > * StackMapTableAttributebecome a CodeElement processed when the > above option is explicitly enabled by user > * StackMapTableAttribute, StackMapFrameInfo, VerificationTypeInfo, > SimpleVerificationTypeInfo, ObjectVerificationTypeInfo and > UninitializedVerificationTypeInfo have been simplified, refactored > to use Labels instead of offsets and equipped with relevant static > factory methods > * Redundant content of the StackMapFrameInfo and its sub-classes > reflecting compressed frame forms have been removed > * UnboundStackMapTableAttribute with proper serialization has been > implemented > * CorpusTest with RebuildingTransformations has been extended to > process stack maps using the new API > > New stack maps processing option affects inflation of stack maps and > appearance of StackMapTableAttribute in the CodeElement stream. > > Enabled stack maps processing inflates and decompresses > StackMapTableAttribute with translated offsets to Labels and pass it > to CodeTransformation(s). > > User can individually filter StackMapTableAttribute out or transform > it manually when explicitly enabled stack maps processing. > > User-provided or transformed StackMapTableAttribute passed to > CodeBuilder overrides stack maps generation process for the actual method. > > Compressed source form of the parsed stack map frame can be identified > from StackMapFrameInfo::tag > > All new created StackMapFrameInfos always represent full frames with > complete list of locals and stack. > > Compression of StackMapTableAttribute frames is contextual operation > and it is applied at the writing phase. > > Any NO-OP transformation of inflated and decompressed > StackMapTableAttribute may result in a different compressed form. > > Performance boost tricks by-passing stack maps generation for > unchanged methods are unaffected by this enhancement. > > Thanks for your comments, > > Adam > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Wed Aug 17 16:02:42 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Wed, 17 Aug 2022 16:02:42 +0000 Subject: RFR: Classfile API stack maps processing In-Reply-To: <16040899-0768-c4be-3c08-dac2edc96d8a@oracle.com> References: <16040899-0768-c4be-3c08-dac2edc96d8a@oracle.com> Message-ID: Yes, user can send the stack maps to the builder anytime and it will disable generation for the actual method. However user has to explicitly construct the attribute: cob.with(StackMapTableAttribute.of(List.of(StackMapFrameInfo.of(myLabel, locals, stack), ?)); Generation switch works independently, so user can keep the generation on for some methods and explicitly specify/transform StackMapTableAttribute for other methods of the same class. From: Brian Goetz Date: Wednesday, 17 August 2022 17:53 To: Adam Sotona , classfile-api-dev at openjdk.org Subject: Re: RFR: Classfile API stack maps processing See the RFR, I think I found a place where we might end up putting TWO stack map attributes into the builder. With this patch, I believe the treatment of stack maps is intended to be: if (generation not disabled) { try to reuse original stackmap, if transforming and code array/exception table is unchanged otherwise generate stack map } else { if a stack map has been sent to the builder, put that in the class file } I am not sure this is the effect, though; since the user can send stack maps down the pipe and they are stored as attributes regardless of options, I think these may be written to the classfile even if we are in generation mode. On 8/17/2022 11:45 AM, Adam Sotona wrote: Hi, I would like to ask for review of proposed change in the Classfile API stack maps processing. Primary motivation is to better align stack maps processing with the Classfile API and to allow manual constructions and transformations of stack maps. Pull request for review is here: https://github.com/openjdk/jdk-sandbox/pull/32 And it includes following changes: 1. New boolean Classfile.Option::processStackMaps with default to false has been added 2. StackMapTableAttribute become a CodeElement processed when the above option is explicitly enabled by user 3. StackMapTableAttribute, StackMapFrameInfo, VerificationTypeInfo, SimpleVerificationTypeInfo, ObjectVerificationTypeInfo and UninitializedVerificationTypeInfo have been simplified, refactored to use Labels instead of offsets and equipped with relevant static factory methods 4. Redundant content of the StackMapFrameInfo and its sub-classes reflecting compressed frame forms have been removed 5. UnboundStackMapTableAttribute with proper serialization has been implemented 6. CorpusTest with RebuildingTransformations has been extended to process stack maps using the new API New stack maps processing option affects inflation of stack maps and appearance of StackMapTableAttribute in the CodeElement stream. Enabled stack maps processing inflates and decompresses StackMapTableAttribute with translated offsets to Labels and pass it to CodeTransformation(s). User can individually filter StackMapTableAttribute out or transform it manually when explicitly enabled stack maps processing. User-provided or transformed StackMapTableAttribute passed to CodeBuilder overrides stack maps generation process for the actual method. Compressed source form of the parsed stack map frame can be identified from StackMapFrameInfo::tag All new created StackMapFrameInfos always represent full frames with complete list of locals and stack. Compression of StackMapTableAttribute frames is contextual operation and it is applied at the writing phase. Any NO-OP transformation of inflated and decompressed StackMapTableAttribute may result in a different compressed form. Performance boost tricks by-passing stack maps generation for unchanged methods are unaffected by this enhancement. Thanks for your comments, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Wed Aug 17 16:25:16 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 17 Aug 2022 12:25:16 -0400 Subject: RFR: Classfile API stack maps processing In-Reply-To: References: <16040899-0768-c4be-3c08-dac2edc96d8a@oracle.com> Message-ID: <17746a15-8885-fc40-c51e-574cef8a5e53@oracle.com> I'd like to be a little more aggressive on that: if generation is not disabled, then any stack map sent downstream is dropped on the floor, and either the original stackmap reused, or a new one generated. Generating stack maps is definitely an advanced, error-prone feature; users should have to turn it on. Given this, though, I think we can drop the PROCESS_STACK_MAP attribute, and always send it downstream, as this costs us almost nothing and the user can ignore it. We should definitely not generate Code attributes with more than one stack map attribute in them. On 8/17/2022 12:02 PM, Adam Sotona wrote: > > Yes, user can send the stack maps to the builder anytime and it will > disable generation for the actual method. > > However user has to explicitly construct the attribute: > > cob.with(StackMapTableAttribute.of(List.of(StackMapFrameInfo.of(myLabel, > locals, stack), ?)); > > Generation switch works independently, so user can keep the generation > on for some methods and explicitly specify/transform > StackMapTableAttribute for other methods of the same class. > > *From: *Brian Goetz > *Date: *Wednesday, 17 August 2022 17:53 > *To: *Adam Sotona , > classfile-api-dev at openjdk.org > *Subject: *Re: RFR: Classfile API stack maps processing > > See the RFR, I think I found a place where we might end up putting TWO > stack map attributes into the builder. > > With this patch, I believe the treatment of stack maps is intended to be: > > ?? if (generation not disabled) { > ?????? try to reuse original stackmap, if transforming and code > array/exception table is unchanged > ?????? otherwise generate stack map > ?? } > ?? else { > ?????? if a stack map has been sent to the builder, put that in the > class file > ?? } > > I am not sure this is the effect, though; since the user can send > stack maps down the pipe and they are stored as attributes regardless > of options, I think these may be written to the classfile even if we > are in generation mode. > > On 8/17/2022 11:45 AM, Adam Sotona wrote: > > Hi, > > I would like to ask for review of proposed change in the Classfile > API stack maps processing. > > Primary motivation is to better align stack maps processing with > the Classfile API and to allow manual constructions and > transformations of stack maps. > > Pull request for review is here: > https://github.com/openjdk/jdk-sandbox/pull/32 > > And it includes following changes: > > 1.New boolean Classfile.Option::processStackMaps with default to > false has been added > > 2.StackMapTableAttributebecome a CodeElement processed when the > above option is explicitly enabled by user > > 3.StackMapTableAttribute, StackMapFrameInfo, VerificationTypeInfo, > SimpleVerificationTypeInfo, ObjectVerificationTypeInfo and > UninitializedVerificationTypeInfo have been simplified, refactored > to use Labels instead of offsets and equipped with relevant static > factory methods > > 4.Redundant content of the StackMapFrameInfo and its sub-classes > reflecting compressed frame forms have been removed > > 5.UnboundStackMapTableAttribute with proper serialization has been > implemented > > 6.CorpusTest with RebuildingTransformations has been extended to > process stack maps using the new API > > New stack maps processing option affects inflation of stack maps > and appearance of StackMapTableAttribute in the CodeElement stream. > > Enabled stack maps processing inflates and decompresses > StackMapTableAttribute with translated offsets to Labels and pass > it to CodeTransformation(s). > > User can individually filter StackMapTableAttribute out or > transform it manually when explicitly enabled stack maps processing. > > User-provided or transformed StackMapTableAttribute passed to > CodeBuilder overrides stack maps generation process for the actual > method. > > Compressed source form of the parsed stack map frame can be > identified from StackMapFrameInfo::tag > > All new created StackMapFrameInfos always represent full frames > with complete list of locals and stack. > > Compression of StackMapTableAttribute frames is contextual > operation and it is applied at the writing phase. > > Any NO-OP transformation of inflated and decompressed > StackMapTableAttribute may result in a different compressed form. > > Performance boost tricks by-passing stack maps generation for > unchanged methods are unaffected by this enhancement. > > Thanks for your comments, > > Adam > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Wed Aug 17 16:48:15 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Wed, 17 Aug 2022 16:48:15 +0000 Subject: RFR: Classfile API stack maps processing In-Reply-To: <17746a15-8885-fc40-c51e-574cef8a5e53@oracle.com> References: <16040899-0768-c4be-3c08-dac2edc96d8a@oracle.com> <17746a15-8885-fc40-c51e-574cef8a5e53@oracle.com> Message-ID: I see it a bit opposite. 1. Processing stack maps is extremely advanced feature and 99.9% of users don?t want to touch it, so why PROCESS_STACK_MAP is off by default and GENERATE_STACK_MAPS is on by default 2. When the 0.1% user needs to process the stack maps, he needs full control over it. So when user enables PROCESS_STACK_MAP, full control of what drops to code builder is up-to user. 3. Most of the #2 cases are complex transformations (3-way merges, instrumentations, etc?) and per-method control is important. For example jdk.jfr class instrumentation involve all cases: * Some methods from the first (shared CP) source are unchanged and so original stack maps can be reused * Some methods from the second source (non-shared CP) are also unchanged, so inflated stack maps can be passed down the transformation * Some methods are combined from both sources, so user may need to manually combine stack maps or just little adjust existing maps * Some methods may be synthetized from scratch and generated stack maps are first choice By dropping PROCESS_STACK_MAP switch and keep it always on we would slow down a bit parsing for 99.9% cases. GENERATE_STACK_MAPS switch as the only one would not allow per-method individual handling. If we want to reduce number of switches I would rather drop GENERATE_STACK_MAPS as the StackMapTableAttribute is mandatory and when user does not provide/transform its own ? we should generate it. From: Brian Goetz Date: Wednesday, 17 August 2022 18:25 To: Adam Sotona , classfile-api-dev at openjdk.org Subject: Re: RFR: Classfile API stack maps processing I'd like to be a little more aggressive on that: if generation is not disabled, then any stack map sent downstream is dropped on the floor, and either the original stackmap reused, or a new one generated. Generating stack maps is definitely an advanced, error-prone feature; users should have to turn it on. Given this, though, I think we can drop the PROCESS_STACK_MAP attribute, and always send it downstream, as this costs us almost nothing and the user can ignore it. We should definitely not generate Code attributes with more than one stack map attribute in them. On 8/17/2022 12:02 PM, Adam Sotona wrote: Yes, user can send the stack maps to the builder anytime and it will disable generation for the actual method. However user has to explicitly construct the attribute: cob.with(StackMapTableAttribute.of(List.of(StackMapFrameInfo.of(myLabel, locals, stack), ?)); Generation switch works independently, so user can keep the generation on for some methods and explicitly specify/transform StackMapTableAttribute for other methods of the same class. From: Brian Goetz Date: Wednesday, 17 August 2022 17:53 To: Adam Sotona , classfile-api-dev at openjdk.org Subject: Re: RFR: Classfile API stack maps processing See the RFR, I think I found a place where we might end up putting TWO stack map attributes into the builder. With this patch, I believe the treatment of stack maps is intended to be: if (generation not disabled) { try to reuse original stackmap, if transforming and code array/exception table is unchanged otherwise generate stack map } else { if a stack map has been sent to the builder, put that in the class file } I am not sure this is the effect, though; since the user can send stack maps down the pipe and they are stored as attributes regardless of options, I think these may be written to the classfile even if we are in generation mode. On 8/17/2022 11:45 AM, Adam Sotona wrote: Hi, I would like to ask for review of proposed change in the Classfile API stack maps processing. Primary motivation is to better align stack maps processing with the Classfile API and to allow manual constructions and transformations of stack maps. Pull request for review is here: https://github.com/openjdk/jdk-sandbox/pull/32 And it includes following changes: 1. New boolean Classfile.Option::processStackMaps with default to false has been added 2. StackMapTableAttribute become a CodeElement processed when the above option is explicitly enabled by user 3. StackMapTableAttribute, StackMapFrameInfo, VerificationTypeInfo, SimpleVerificationTypeInfo, ObjectVerificationTypeInfo and UninitializedVerificationTypeInfo have been simplified, refactored to use Labels instead of offsets and equipped with relevant static factory methods 4. Redundant content of the StackMapFrameInfo and its sub-classes reflecting compressed frame forms have been removed 5. UnboundStackMapTableAttribute with proper serialization has been implemented 6. CorpusTest with RebuildingTransformations has been extended to process stack maps using the new API New stack maps processing option affects inflation of stack maps and appearance of StackMapTableAttribute in the CodeElement stream. Enabled stack maps processing inflates and decompresses StackMapTableAttribute with translated offsets to Labels and pass it to CodeTransformation(s). User can individually filter StackMapTableAttribute out or transform it manually when explicitly enabled stack maps processing. User-provided or transformed StackMapTableAttribute passed to CodeBuilder overrides stack maps generation process for the actual method. Compressed source form of the parsed stack map frame can be identified from StackMapFrameInfo::tag All new created StackMapFrameInfos always represent full frames with complete list of locals and stack. Compression of StackMapTableAttribute frames is contextual operation and it is applied at the writing phase. Any NO-OP transformation of inflated and decompressed StackMapTableAttribute may result in a different compressed form. Performance boost tricks by-passing stack maps generation for unchanged methods are unaffected by this enhancement. Thanks for your comments, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Wed Aug 17 16:57:57 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 17 Aug 2022 12:57:57 -0400 Subject: RFR: Classfile API stack maps processing In-Reply-To: References: <16040899-0768-c4be-3c08-dac2edc96d8a@oracle.com> <17746a15-8885-fc40-c51e-574cef8a5e53@oracle.com> Message-ID: <8886e077-819b-d2a1-b9f5-386d7c25c15f@oracle.com> > 1. Processing stack maps is extremely advanced feature and 99.9% of > users don?t want to touch it, so why PROCESS_STACK_MAPis off by > default and GENERATE_STACK_MAPS is on by default > Agree with this. > 1. When the 0.1% user needs to process the stack maps, he needs full > control over it. So when user enables PROCESS_STACK_MAP, full > control of what drops to code builder is up-to user. > OK, I don't mind two options, but I'd like to be clear on what they mean.? I don't think that merely turning on PROCESS_STACK_MAP will clue the user in to the fact that they have disengaged all the safety guards.? Some users may just want to _read_ the stack map. So I'm OK with having both PROCESS and GENERATE options, but we should be clear what they mean, and be conservative in their meanings.? Here's what makes sense to me: GENERATE: the stack map is *always* generated by the library.? Any stack maps sent downstream are still ignored. PROCESS: send the stack map to the user. PROCESS & !GENERATE: send the stack map to the user, write any stack map the user supplied to the classfile. So users who want full control select PROCESS and !GENERATE; users who want to see the stackmap select PROCESS only; users who don't care about stack maps (which is most of them) select neither. I think what this exposes is we need a way to override some options on a per-method basis, which is a separate problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Wed Aug 17 17:27:26 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Wed, 17 Aug 2022 17:27:26 +0000 Subject: RFR: Classfile API stack maps processing In-Reply-To: <8886e077-819b-d2a1-b9f5-386d7c25c15f@oracle.com> References: <16040899-0768-c4be-3c08-dac2edc96d8a@oracle.com> <17746a15-8885-fc40-c51e-574cef8a5e53@oracle.com> <8886e077-819b-d2a1-b9f5-386d7c25c15f@oracle.com> Message-ID: I thought PROCESS_STACK_MAP switch is clear the maps will pass to the CodeBuilder. It seems to me similar to PROCESS_DEBUG, PROCESS_LINE_NUMBERS and PROCESS_UNKNOWN_ATTRIBUTES. In all the above cases the relevant attributes appear or disappear from processing. I still ?slightly? think that if user drops something specific to CodeBuilder ? it should override even the Generator. However I?m ready to put Generator priority over the content dropped to CodeBuilder. What about another option to keep the status quo, drop PROCESS_STACK_MAP switch and do not stream StackMapAttribute at all? As users can still access it from CodeModel::findAttribute(Attributes.STACK_MAP_TABLE) and manually pass it to CodeBuilder, which will then override even Generator. From: Brian Goetz Date: Wednesday, 17 August 2022 18:58 To: Adam Sotona , classfile-api-dev at openjdk.org Subject: Re: RFR: Classfile API stack maps processing 1. Processing stack maps is extremely advanced feature and 99.9% of users don?t want to touch it, so why PROCESS_STACK_MAP is off by default and GENERATE_STACK_MAPS is on by default Agree with this. 1. When the 0.1% user needs to process the stack maps, he needs full control over it. So when user enables PROCESS_STACK_MAP, full control of what drops to code builder is up-to user. OK, I don't mind two options, but I'd like to be clear on what they mean. I don't think that merely turning on PROCESS_STACK_MAP will clue the user in to the fact that they have disengaged all the safety guards. Some users may just want to _read_ the stack map. So I'm OK with having both PROCESS and GENERATE options, but we should be clear what they mean, and be conservative in their meanings. Here's what makes sense to me: GENERATE: the stack map is *always* generated by the library. Any stack maps sent downstream are still ignored. PROCESS: send the stack map to the user. PROCESS & !GENERATE: send the stack map to the user, write any stack map the user supplied to the classfile. So users who want full control select PROCESS and !GENERATE; users who want to see the stackmap select PROCESS only; users who don't care about stack maps (which is most of them) select neither. I think what this exposes is we need a way to override some options on a per-method basis, which is a separate problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Wed Aug 17 17:39:44 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 17 Aug 2022 13:39:44 -0400 Subject: RFR: Classfile API stack maps processing In-Reply-To: References: <16040899-0768-c4be-3c08-dac2edc96d8a@oracle.com> <17746a15-8885-fc40-c51e-574cef8a5e53@oracle.com> <8886e077-819b-d2a1-b9f5-386d7c25c15f@oracle.com> Message-ID: > I thought PROCESS_STACK_MAP switch is clear the maps will pass to the > CodeBuilder. > > It seems to me similar to PROCESS_DEBUG, PROCESS_LINE_NUMBERS and > PROCESS_UNKNOWN_ATTRIBUTES. > > In all the above cases the relevant attributes appear or disappear > from processing. > It is structurally similar, but (a) slightly different and (b) the danger level is much higher. For PROCESS_{DEBUG,LINE_NUMBERS}, we explode the appropriate attribute into finer-grained elements, and reconstruct them into an attribute at the end.? (This reminds me that we need to make sure that we're not doing *both* reconstructed and pass-through processing in these cases.)? So this is opting into a more active mode of control over these elements.? And secondarily, messing up the line number tables or LVT will not result in unloadable classfiles. It seem just too easy for a user to do the following: ?- request PROCESS_STACK_MAP because there is something useful they want to see in it, and just passing it through to the builder because that's a reasonable default for an element you don't plan to manipulate; ?- adapt the instruction stream (say, to add logging), but without rewriting the stack map. Now, we will write a classfile with a wrong stack map.? This seems too easy and too dangerous.? Hence, asking users to _turn off_ generation as well as turning on "give me stack maps" seems a reasonable precaution. > I still ?slightly? think that if user drops something specific to > CodeBuilder ? it should override even the Generator. > > However I?m ready to put Generator priority over the content dropped > to CodeBuilder. > We faced a similar situation with BootstrapMethods attribute. THere, we concluded there is no case we want to let the user generate a BMA; if they want to generate BMA _entries_, they can do so with ConstantPoolBuilder.? There are various places where we filter out BMA in various places, such as attributes written to the ClassBuilder.? Stack maps feel about 90% along on the spectrum of [ line numbers ... bootstrap methods ] to me. > What about another option to keep the status quo, drop > PROCESS_STACK_MAPswitch and do not stream StackMapAttribute at all? > > As users can still access it from > CodeModel::findAttribute(Attributes.STACK_MAP_TABLE) and manually pass > it to CodeBuilder, which will then override even Generator. > This is more in line with how we handle BootstrapMethod attribute, and I think it is fine.? I think it is better, actually, because there is? no confusion about the interaction between "process" and "generate".? Here, "generate" indicates clear control over what to do with user-provided stack maps -- if we are in generate mode, ignore the user one.? No confusion.? So, +1 to this idea. > *From: *Brian Goetz > *Date: *Wednesday, 17 August 2022 18:58 > *To: *Adam Sotona , > classfile-api-dev at openjdk.org > *Subject: *Re: RFR: Classfile API stack maps processing > > > > 1.Processing stack maps is extremely advanced feature and 99.9% of > users don?t want to touch it, so why PROCESS_STACK_MAPis off by > default and GENERATE_STACK_MAPS is on by default > > > Agree with this. > > > 1.When the 0.1% user needs to process the stack maps, he needs > full control over it. So when user enables PROCESS_STACK_MAP, full > control of what drops to code builder is up-to user. > > > OK, I don't mind two options, but I'd like to be clear on what they > mean.? I don't think that merely turning on PROCESS_STACK_MAP will > clue the user in to the fact that they have disengaged all the safety > guards.? Some users may just want to _read_ the stack map.? So I'm OK > with having both PROCESS and GENERATE options, but we should be clear > what they mean, and be conservative in their meanings.? Here's what > makes sense to me: > > GENERATE: the stack map is *always* generated by the library. Any > stack maps sent downstream are still ignored. > PROCESS: send the stack map to the user. > PROCESS & !GENERATE: send the stack map to the user, write any stack > map the user supplied to the classfile. > > So users who want full control select PROCESS and !GENERATE; users who > want to see the stackmap select PROCESS only; users who don't care > about stack maps (which is most of them) select neither. > > I think what this exposes is we need a way to override some options on > a per-method basis, which is a separate problem. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Wed Aug 17 18:48:01 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Wed, 17 Aug 2022 18:48:01 +0000 Subject: RFR: Classfile API stack maps processing In-Reply-To: References: <16040899-0768-c4be-3c08-dac2edc96d8a@oracle.com> <17746a15-8885-fc40-c51e-574cef8a5e53@oracle.com> <8886e077-819b-d2a1-b9f5-386d7c25c15f@oracle.com> Message-ID: What about another option to keep the status quo, drop PROCESS_STACK_MAP switch and do not stream StackMapAttribute at all? As users can still access it from CodeModel::findAttribute(Attributes.STACK_MAP_TABLE) and manually pass it to CodeBuilder, which will then override even Generator. This is more in line with how we handle BootstrapMethod attribute, and I think it is fine. I think it is better, actually, because there is no confusion about the interaction between "process" and "generate". Here, "generate" indicates clear control over what to do with user-provided stack maps -- if we are in generate mode, ignore the user one. No confusion. So, +1 to this idea. OK, done, with the ?generator overrides all?. The access to manual processing of stack maps is a bit hidden, however available. We may later change decision to stream the SMTA (based on any existing switch or a new one) and it won?t affect processing. I figured out how to inflate remaining labels without loose of performance (without inflation of the whole SMTA). Per-method options to allow ?hybrid? transformations can be solved in another round. One thing remains (also for another round) ? we still use defaults for maxLocals and maxStack when the generator is off. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Wed Aug 17 19:32:37 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 17 Aug 2022 15:32:37 -0400 Subject: RFR: Classfile API stack maps processing In-Reply-To: References: <16040899-0768-c4be-3c08-dac2edc96d8a@oracle.com> <17746a15-8885-fc40-c51e-574cef8a5e53@oracle.com> <8886e077-819b-d2a1-b9f5-386d7c25c15f@oracle.com> Message-ID: <87fa3a3e-7d2d-d95c-dcfc-c6c3abd2a364@oracle.com> > The access to manual processing of stack maps is a bit hidden, however > available. > > We may later change decision to stream the SMTA (based on any existing > switch or a new one) and it won?t affect processing. I figured out how > to inflate remaining labels without loose of performance (without > inflation of the whole SMTA). > If we want to do that later, we can make SMTA a CompositeElement, where the frames are the elements, and a corresponding builder. Building would then produce an unbound SMTA which could be fed to CodeBuilder::withAttribute. > One thing remains (also for another round) ? we still use defaults for > maxLocals and maxStack when the generator is off. In the process of doing the local variable allocation stuff, I added some support for doing better, but it's not 100% yet.? It carries over maxLocal/maxStack during adaptation, and adjusts maxLocal based on allocated locals.? But it would need some more work to be reliable, since you can do your own manual local management. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Thu Aug 18 08:53:47 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Thu, 18 Aug 2022 10:53:47 +0200 Subject: methodHandleEntry() for Map$Entry::getKey Message-ID: Putting a method handle for getKey into the constant pool got me #9 = NameAndType #7:#8 // getKey:(Ljava/util/Map$Entry;)Ljava/lang/Object; #10 = Methodref #6.#9 // java/util/Map$Entry.getKey:(Ljava/util/Map$Entry;)Ljava/lang/Object; #11 = MethodHandle 9:#10 // REF_invokeInterface java/util/Map$Entry.getKey:(Ljava/util/Map$Entry;)Ljava/lang/Object; instead of the expected #33 = NameAndType #7:#32 // getKey:()Ljava/lang/Object; #34 = InterfaceMethodref #6.#33 // java/util/Map$Entry.getKey:()Ljava/lang/Object; That is, both tag and lookup type are off target. The attached patch allows me to continue for now. The MTD -> String -> MTD conversion is a bit heavy handed. -- mva diff --git a/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java b/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java index 1363f1e8c83..7c3395d0c55 100755 --- a/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java +++ b/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java @@ -346,7 +346,11 @@ public sealed interface ConstantPoolBuilder * @param descriptor the symbolic descriptor of the method handle */ default MethodHandleEntry methodHandleEntry(DirectMethodHandleDesc descriptor) { - return methodHandleEntry(descriptor.refKind(), methodRefEntry(descriptor.owner(), descriptor.methodName(), descriptor.invocationType())); + var lookupType = MethodTypeDesc.ofDescriptor(descriptor.lookupDescriptor()); + return methodHandleEntry(descriptor.refKind(), + descriptor.isOwnerInterface() ? + interfaceMethodRefEntry(descriptor.owner(), descriptor.methodName(), lookupType) : + methodRefEntry(descriptor.owner(), descriptor.methodName(), lookupType)); } /** -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Thu Aug 18 10:14:57 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Thu, 18 Aug 2022 10:14:57 +0000 Subject: methodHandleEntry() for Map$Entry::getKey In-Reply-To: References: Message-ID: Thanks for the report. It should be fixed now: https://github.com/openjdk/jdk-sandbox/commit/de17a3b0eb7c80935d701ba98014f2bd35c221b5 Thanks, Adam From: classfile-api-dev on behalf of Michael van Acken Date: Thursday, 18 August 2022 10:54 To: classfile-api-dev at openjdk.org Subject: methodHandleEntry() for Map$Entry::getKey Putting a method handle for getKey into the constant pool got me #9 = NameAndType #7:#8 // getKey:(Ljava/util/Map$Entry;)Ljava/lang/Object; #10 = Methodref #6.#9 // java/util/Map$Entry.getKey:(Ljava/util/Map$Entry;)Ljava/lang/Object; #11 = MethodHandle 9:#10 // REF_invokeInterface java/util/Map$Entry.getKey:(Ljava/util/Map$Entry;)Ljava/lang/Object; instead of the expected #33 = NameAndType #7:#32 // getKey:()Ljava/lang/Object; #34 = InterfaceMethodref #6.#33 // java/util/Map$Entry.getKey:()Ljava/lang/Object; That is, both tag and lookup type are off target. The attached patch allows me to continue for now. The MTD -> String -> MTD conversion is a bit heavy handed. -- mva diff --git a/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java b/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java index 1363f1e8c83..7c3395d0c55 100755 --- a/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java +++ b/src/java.base/share/classes/jdk/classfile/constantpool/ConstantPoolBuilder.java @@ -346,7 +346,11 @@ public sealed interface ConstantPoolBuilder * @param descriptor the symbolic descriptor of the method handle */ default MethodHandleEntry methodHandleEntry(DirectMethodHandleDesc descriptor) { - return methodHandleEntry(descriptor.refKind(), methodRefEntry(descriptor.owner(), descriptor.methodName(), descriptor.invocationType())); + var lookupType = MethodTypeDesc.ofDescriptor(descriptor.lookupDescriptor()); + return methodHandleEntry(descriptor.refKind(), + descriptor.isOwnerInterface() ? + interfaceMethodRefEntry(descriptor.owner(), descriptor.methodName(), lookupType) : + methodRefEntry(descriptor.owner(), descriptor.methodName(), lookupType)); } /** -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Thu Aug 18 10:31:00 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Thu, 18 Aug 2022 12:31:00 +0200 Subject: methodHandleEntry() for Map$Entry::getKey In-Reply-To: References: Message-ID: Am Do., 18. Aug. 2022 um 12:15 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > Thanks for the report. > > It should be fixed now: > > > https://github.com/openjdk/jdk-sandbox/commit/de17a3b0eb7c80935d701ba98014f2bd35c221b5 > Works fine. Thanks! -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Thu Aug 18 15:41:10 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Thu, 18 Aug 2022 15:41:10 +0000 Subject: Classfile API cleanup of labelToBci from CodeModel and CodeBuilder and removal of impl.LabelResolver Message-ID: Hi, I have another Classfile API change proposal to discuss. Method labelToBci from impl.LabelResolver is exposed through CodeModel. I propose to completely remove impl.LabelResolver and move the label resolution functionality to CodeAttribute::labelToBci. CodeAttribute is the only API subclass of CodeModel where the bytecode index can be successfully resolved and where it makes sense to use it together with CodeAttribute::codeArray. Also CodeBuilder exposes method labelToBci, even it does not subclass LabelResolver. I propose to remove CodeBuilder::labelToBci as it is just another exposure of internal functionality and should not serve to users. Pull request with the proposal is here: https://github.com/openjdk/jdk-sandbox/pull/33 Thanks, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Thu Aug 18 16:09:39 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 18 Aug 2022 12:09:39 -0400 Subject: Classfile API cleanup of labelToBci from CodeModel and CodeBuilder and removal of impl.LabelResolver In-Reply-To: References: Message-ID: > I propose to remove CodeBuilder::labelToBci as it is just another > exposure of internal functionality and should not serve to users. Worse, it is not reliable.? If you have a BufferedCB, if you call labelToBCI, you'll get an unhelpful "I don't know".? And you don't know when you have a buffered CB; you get one in certain forms of transform composition. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Thu Aug 18 17:24:53 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Thu, 18 Aug 2022 19:24:53 +0200 Subject: Classfile API cleanup of labelToBci from CodeModel and CodeBuilder and removal of impl.LabelResolver In-Reply-To: References: Message-ID: [Mail delivery is flaky today. Trying again...] Am Do., 18. Aug. 2022 um 17:41 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > Hi, > > I have another Classfile API change proposal to discuss. > > Method labelToBci from impl.LabelResolver is exposed through CodeModel. > > I propose to completely remove impl.LabelResolver and move the label > resolution functionality to CodeAttribute::labelToBci. > > CodeAttribute is the only API subclass of CodeModel where the bytecode > index can be successfully resolved and where it makes sense to use it > together with CodeAttribute::codeArray. > > > > Also CodeBuilder exposes method labelToBci, even it does not subclass > LabelResolver. > > I propose to remove CodeBuilder::labelToBci as it is just another exposure > of internal functionality and should not serve to users. > I have a single use of CodeBuilder.labelToBci(): decide whether the labels marking the start and end of a try block refer to the same position, as a proxy for "enclosed region is empty". If this is the case, then no exceptionCatch can be installed and all exception handlers become unreachable. How can this scenario still be supported? -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Thu Aug 18 20:51:42 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 18 Aug 2022 16:51:42 -0400 Subject: What to do about dead labels? Message-ID: <830b0f19-7d10-7ca9-d9ed-bd785c49a48a@oracle.com> There are a number of FIXME comments in the code that remind us that we haven't made a decision of what to do about dead labels, which are labels that are not assigned a point in the element stream. Sometimes a dead label is an outright error, such as: ??? .withCode(b -> { Label lab = Label.of(); ???????????????????? b.branch(GOTO, lab); ?????????????????? }); You can't jump to a location that is not defined.? But sometimes a label can get accidentally snipped out, and yet still show up in LVT, LVTT, or the exception table.? For example, suppose we have some code: ??? start(); ??? try {?????? // X ??????? int x; ??????? blah(); ??? }?????????? // Y ??? catch (FooException f) { ??????? // Z ??????? blahblah(); ??????? // W ??? } ??? // Q ??? end(); ??? moo(); When we traverse this, we will get something like: ?? exceptionCatch(X, Y, Z, W, FooException) ?? local("x", X, Y) ?? invoke start ?? label(X) ?? invoke blah ?? label(y) ?? goto Q ?? label(Z) ?? invoke blahblah ?? label(W) ?? label(Q) ?? invoke end ?? invoke moo If the user decides to transform this such that anything between "begin" and "end" are removed, we could get this stream: exceptionCatch(X, Y, Z, W, FooException) local("x", X, Y) ?? invoke start ?? invoke end ?? invoke moo and the labels in the exception table / local metadata are dead.? This really isn't the user's fault, especially as we send the metadata up front.? (One reason for this is that early rounds sending it in order had a measurable performance cost; we should re-measure that.? But also, its not always obvious that there is a "right" time, or that it would be immune to such transforms.) Separate from whether we should try to reorder the elements to reduce the error surface, we have two choices for how to deal with dead labels in metadata: ?- try to sanitize the metadata.? If we find an exception table / LVT / LVTT entry with dead labels, just drop it, and write the class out. ?- fail fast.? If we find an entry with data labels, throw. (I am hoping that there is a reasonable answer that is not "make it an option to do either.") -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Thu Aug 18 21:18:45 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 18 Aug 2022 21:18:45 +0000 Subject: What to do about dead labels? In-Reply-To: <830b0f19-7d10-7ca9-d9ed-bd785c49a48a@oracle.com> References: <830b0f19-7d10-7ca9-d9ed-bd785c49a48a@oracle.com> Message-ID: <8F1C2FA4-D993-410E-A04F-0DD73FA0629D@oracle.com> In some sense it is the users responsibility when removing code to remove all related artifacts, but in this example we are not making it easy. Failing-fast seems more justifiable if the pseudo-instruction exceptionCatch was reported immediately after one of the now non-existent labels. It seems useful to report it immediately before, or immediately after, the label for the start of the try region, then the user gets a heads up on the structure at the ?right? point. If we drop I worry it may hide bugs. Perhaps it comes down to an option to order such pseudo instructions or not? Paul. > On Aug 18, 2022, at 1:51 PM, Brian Goetz wrote: > > There are a number of FIXME comments in the code that remind us that we haven't made a decision of what to do about dead labels, which are labels that are not assigned a point in the element stream. > > Sometimes a dead label is an outright error, such as: > > .withCode(b -> { Label lab = Label.of(); > b.branch(GOTO, lab); > }); > > You can't jump to a location that is not defined. But sometimes a label can get accidentally snipped out, and yet still show up in LVT, LVTT, or the exception table. For example, suppose we have some code: > > start(); > try { // X > int x; > > blah(); > } // Y > catch (FooException f) { > // Z > blahblah(); > // W > } > // Q > end(); > moo(); > > When we traverse this, we will get something like: > > exceptionCatch(X, Y, Z, W, FooException) > local("x", X, Y) > invoke start > label(X) > invoke blah > label(y) > goto Q > label(Z) > invoke blahblah > label(W) > label(Q) > invoke end > invoke moo > > If the user decides to transform this such that anything between "begin" and "end" are removed, we could get this stream: > > exceptionCatch(X, Y, Z, W, FooException) > local("x", X, Y) > invoke start > invoke end > invoke moo > > and the labels in the exception table / local metadata are dead. This really isn't the user's fault, especially as we send the metadata up front. (One reason for this is that early rounds sending it in order had a measurable performance cost; we should re-measure that. But also, its not always obvious that there is a "right" time, or that it would be immune to such transforms.) > > Separate from whether we should try to reorder the elements to reduce the error surface, we have two choices for how to deal with dead labels in metadata: > > - try to sanitize the metadata. If we find an exception table / LVT / LVTT entry with dead labels, just drop it, and write the class out. > - fail fast. If we find an entry with data labels, throw. > > (I am hoping that there is a reasonable answer that is not "make it an option to do either.") > > > > From brian.goetz at oracle.com Thu Aug 18 21:41:54 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 18 Aug 2022 17:41:54 -0400 Subject: What to do about dead labels? In-Reply-To: <8F1C2FA4-D993-410E-A04F-0DD73FA0629D@oracle.com> References: <830b0f19-7d10-7ca9-d9ed-bd785c49a48a@oracle.com> <8F1C2FA4-D993-410E-A04F-0DD73FA0629D@oracle.com> Message-ID: <24c4cbb5-76e9-5079-66cc-a26a3452027a@oracle.com> FTR, ordering them means: ?- sorting the appropriate tables by bci ?- arranging to deliver them in order, which means another test-and-branch on each BCI The data structures we use have changed a lot, so the cost might not be as noticeable as when we last tried this. There is a sensible, canonical place to put LVT/LVTT information -- at the start of the range.? For try/catch, we could put it either at the start of the try range, or the start of the catch range; this seems a more or less arbitrary choice. If we pull on this string, we probably are committing to sending LineNo information in the same place -- and the LineNumberTable is much bigger (costs more to sort, if its not already sorted.) Agree dropping risks hiding bugs no matter what.? Probably should treat these as errors now, and come back for the reordering later. On 8/18/2022 5:18 PM, Paul Sandoz wrote: > In some sense it is the users responsibility when removing code to remove all related artifacts, but in this example we are not making it easy. > > Failing-fast seems more justifiable if the pseudo-instruction exceptionCatch was reported immediately after one of the now non-existent labels. It seems useful to report it immediately before, or immediately after, the label for the start of the try region, then the user gets a heads up on the structure at the ?right? point. > > If we drop I worry it may hide bugs. > > Perhaps it comes down to an option to order such pseudo instructions or not? > > Paul. > >> On Aug 18, 2022, at 1:51 PM, Brian Goetz wrote: >> >> There are a number of FIXME comments in the code that remind us that we haven't made a decision of what to do about dead labels, which are labels that are not assigned a point in the element stream. >> >> Sometimes a dead label is an outright error, such as: >> >> .withCode(b -> { Label lab = Label.of(); >> b.branch(GOTO, lab); >> }); >> >> You can't jump to a location that is not defined. But sometimes a label can get accidentally snipped out, and yet still show up in LVT, LVTT, or the exception table. For example, suppose we have some code: >> >> start(); >> try { // X >> int x; >> >> blah(); >> } // Y >> catch (FooException f) { >> // Z >> blahblah(); >> // W >> } >> // Q >> end(); >> moo(); >> >> When we traverse this, we will get something like: >> >> exceptionCatch(X, Y, Z, W, FooException) >> local("x", X, Y) >> invoke start >> label(X) >> invoke blah >> label(y) >> goto Q >> label(Z) >> invoke blahblah >> label(W) >> label(Q) >> invoke end >> invoke moo >> >> If the user decides to transform this such that anything between "begin" and "end" are removed, we could get this stream: >> >> exceptionCatch(X, Y, Z, W, FooException) >> local("x", X, Y) >> invoke start >> invoke end >> invoke moo >> >> and the labels in the exception table / local metadata are dead. This really isn't the user's fault, especially as we send the metadata up front. (One reason for this is that early rounds sending it in order had a measurable performance cost; we should re-measure that. But also, its not always obvious that there is a "right" time, or that it would be immune to such transforms.) >> >> Separate from whether we should try to reorder the elements to reduce the error surface, we have two choices for how to deal with dead labels in metadata: >> >> - try to sanitize the metadata. If we find an exception table / LVT / LVTT entry with dead labels, just drop it, and write the class out. >> - fail fast. If we find an entry with data labels, throw. >> >> (I am hoping that there is a reasonable answer that is not "make it an option to do either.") >> >> >> >> From adam.sotona at oracle.com Fri Aug 19 05:58:44 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Fri, 19 Aug 2022 05:58:44 +0000 Subject: [External] : Re: Classfile API cleanup of labelToBci from CodeModel and CodeBuilder and removal of impl.LabelResolver In-Reply-To: References: CAEYHehjaZpu_dkztGqxej5LYxa1X3swBv4rX6D1cVsbYAgayrQ@mail.gmail.com <202208181727.27IHR2bx031308@iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com> Message-ID: From: Michael van Acken > I have a single use of CodeBuilder.labelToBci(): decide whether the labels marking the start and end of a try block refer to the same position, as a proxy for "enclosed region is empty". If this is the case, then no exceptionCatch can be installed and all exception handlers become unreachable. How can this scenario still be supported? -- mva It is not guaranteed to get the right bytecode indexes from CodeBuilder::labelToBci, the method is really an exposure of internal implementation. Frequent common cases (like for example dropping exception handler when try block is empty) can be solved with CodeBuilder::trying support. CodeBuilder::trying uses CodeBuilder::block and before further actions asks CodeBuilder.BlockCodeBuilder::isEmpty. Now it throws an exception in case of an empty try block, however I can imagine it might skip all handlers instead: /** * Adds a "try-catch" block comprising one try block and zero or more catch blocks. * Exceptions thrown by instructions in the try block may be caught by catch blocks. * * @param tryHandler handler that receives a {@linkplain CodeBuilder} to * generate the body of the try block. * @param catchesHandler a handler that receives a {@linkplain CatchBuilder} * to generate bodies of catch blocks. * @return this builder * @see CatchBuilder */ default CodeBuilder trying(Consumer tryHandler, Consumer catchesHandler) { Label tryCatchEnd = newLabel(); // @@@ the tryHandler does not have access to tryCatchEnd BlockCodeBuilderImpl tryBlock = new BlockCodeBuilderImpl(this, tryCatchEnd); tryBlock.start(); tryHandler.accept(tryBlock); tryBlock.end(); // Check for empty try block if (!tryBlock.isEmpty()) { var catchBuilder = new CatchBuilderImpl(this, tryBlock, tryCatchEnd); catchesHandler.accept(catchBuilder); catchBuilder.finish(); } return this; } -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Fri Aug 19 06:34:25 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Fri, 19 Aug 2022 06:34:25 +0000 Subject: Classfile API is missing CodeBuilder hook for various instructions-consuming computation tools Message-ID: Hi, I?ve started working on various helper tools that can give user answers during code building. Answers to questions like: * What is actual instructions count * What is actual/max depts of the stack * Which locals have been used * What kinds or exact types are at locals * What kinds or exact types are on stack * Show me a log of all instructions * ? Each of this question has specific use case and calculation of each answer scarifies different amount of resources. Unfortunately implementation of tool answering any of the above questions requires: - either heavy use of BufferedCodeBuilder (and complicate existing building or transformation blocks a lot) - or hacking into CodeBuilder(s) implementation I would like to propose a similar approach as Linux ?tee? tool does, so user can attach another consumer of the CodeElements passed to the CodeBuilder. It might be just a single method in CodeBuilder: default void tee(Consumer secondListener, Consumer handler) { handler.accept(new TeeCodeBuilder(this, secondListener)); } Use case may then look for example like this: var stackCounter = ...//custom computation tool codeBuilder.tee(stackCounter, cb -> { ... //block of instructions to be counted }); stackCounter.actual(); stackCounter.max(); What do you think about this CodeBuilder hook for various instructions-consuming computation tools? Thanks, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Fri Aug 19 06:49:38 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Fri, 19 Aug 2022 08:49:38 +0200 Subject: [External] : Re: Classfile API cleanup of labelToBci from CodeModel and CodeBuilder and removal of impl.LabelResolver In-Reply-To: References: <202208181727.27IHR2bx031308@iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com> Message-ID: Am Fr., 19. Aug. 2022 um 07:58 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > From: Michael van Acken > > > > I have a single use of CodeBuilder.labelToBci(): decide whether the labels > marking > > the start and end of a try block refer to the same position, as a proxy > for "enclosed > region is empty". If this is the case, then no exceptionCatch can be > installed and > all exception handlers become unreachable. > > How can this scenario still be supported? > > -- mva > > > > It is not guaranteed to get the right bytecode indexes from > CodeBuilder::labelToBci, the method is really an exposure of internal > implementation. > > Frequent common cases (like for example dropping exception handler when > try block is empty) can be solved with CodeBuilder::trying support. > > CodeBuilder::trying uses CodeBuilder::block and before further actions > asks CodeBuilder.BlockCodeBuilder::isEmpty. > > Now it throws an exception in case of an empty try block, however I can > imagine it might skip all handlers instead: > This sounds interesting. Right now I am staying away from the higher level entry points, because of their WIP implementation -- as I found out when I tried to use ifThenElse(). `trying` doesn't seem to support finally right now. Having access to a BlockCodeBuilder instance and its isEmpty would solve my problem. Is this something that can be done from the public API? -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Fri Aug 19 07:01:24 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Fri, 19 Aug 2022 07:01:24 +0000 Subject: [External] : Re: Classfile API cleanup of labelToBci from CodeModel and CodeBuilder and removal of impl.LabelResolver In-Reply-To: References: <202208181727.27IHR2bx031308@iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com> Message-ID: From: Michael van Acken This sounds interesting. Right now I am staying away from the higher level entry points, because of their WIP implementation -- as I found out when I tried to use ifThenElse(). `trying` doesn't seem to support finally right now. CodeBuilder.CatchBuilder::catchingAll(?) represents finally block Having access to a BlockCodeBuilder instance and its isEmpty would solve my problem. Is this something that can be done from the public API? CodeBuilder::block(?) -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Fri Aug 19 07:13:16 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Fri, 19 Aug 2022 09:13:16 +0200 Subject: [External] : Re: Classfile API cleanup of labelToBci from CodeModel and CodeBuilder and removal of impl.LabelResolver In-Reply-To: References: <202208181727.27IHR2bx031308@iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com> Message-ID: Am Fr., 19. Aug. 2022 um 09:01 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > > > > > *From: *Michael van Acken > > This sounds interesting. Right now I am staying away from the higher > level entry points, > > because of their WIP implementation -- as I found out when I tried to use > ifThenElse(). > > `trying` doesn't seem to support finally right now. > > > > CodeBuilder.CatchBuilder::catchingAll(?) represents finally block > I was referring to the part where the code of the finally block is duplicated into all paths right after their respective handled regions. Looking at the CodeBuilder source code, I can't find any of this. > > Having access to a BlockCodeBuilder instance and its isEmpty would solve my > > problem. Is this something that can be done from the public API? > > > > CodeBuilder::block(?) > This method encapsulates a BlockCodeBuilder instance, but does not provide it to the caller (the `child` below): default CodeBuilder block(Consumer handler) { Label breakLabel = newLabel(); BlockCodeBuilderImpl child = new BlockCodeBuilderImpl(this, breakLabel); child.start(); handler.accept(child); child.end(); labelBinding(breakLabel); return this; } -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Fri Aug 19 07:58:54 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Fri, 19 Aug 2022 07:58:54 +0000 Subject: [External] : Re: Classfile API cleanup of labelToBci from CodeModel and CodeBuilder and removal of impl.LabelResolver In-Reply-To: References: <202208181727.27IHR2bx031308@iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com> Message-ID: From: Michael van Acken I was referring to the part where the code of the finally block is duplicated into all paths right after their respective handled regions. Looking at the CodeBuilder source code, I can't find any of this. This is an example use case of CodeBuilder::trying: codeBuilder.trying(tryBuilder -> { // try block code }, catchBuilder -> { catchBuilder.catching(CD_IOOBE, handlerBuilder -> { // catch handler code can }).catchingAll(finallyBuilder -> { // finally handler code }); }); This method encapsulates a BlockCodeBuilder instance, but does not provide it to the caller (the `child` below): This is an example use case of CodeBuilder::block: codeBuilder.block(blockBuilder -> { // block code if( blockBuilder.isEmpty()) ? }); -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.van.acken at gmail.com Fri Aug 19 10:07:40 2022 From: michael.van.acken at gmail.com (Michael van Acken) Date: Fri, 19 Aug 2022 12:07:40 +0200 Subject: [External] : Re: Classfile API cleanup of labelToBci from CodeModel and CodeBuilder and removal of impl.LabelResolver In-Reply-To: References: <202208181727.27IHR2bx031308@iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com> Message-ID: Am Fr., 19. Aug. 2022 um 09:58 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > > > > > *From: *Michael van Acken > > I was referring to the part where the code of the finally block is > duplicated > > into all paths right after their respective handled regions. Looking at > the > > CodeBuilder source code, I can't find any of this. > > > > This is an example use case of CodeBuilder::trying: > > codeBuilder.trying(tryBuilder -> { > > // try block code > > }, catchBuilder -> { > > catchBuilder.catching(CD_IOOBE, handlerBuilder -> { > > // catch handler code can > > }).catchingAll(finallyBuilder -> { > > // finally handler code > > }); > > }); > I tried to follow your suggestion, but was not able to complete the try/finally output. The Clojure code is probably not that helpful here, but if I put this method through javac I get output whose [8, 16[ (right after the region covered by exceptionCatch) I cannot emulate with the trying() construction you describe. public static void tryFinally() { try { System.out.println("Hello World!"); } finally { System.out.println("Finally..."); } } 0: getstatic #7 // Field java/lang/System.out:Ljava/io/PrintStream; 3: ldc #13 // String Hello World! 5: invokevirtual #15 // Method java/io/PrintStream.println:(Ljava/lang/String;)V 8: getstatic #7 // Field java/lang/System.out:Ljava/io/PrintStream; 11: ldc #21 // String Finally... 13: invokevirtual #15 // Method java/io/PrintStream.println:(Ljava/lang/String;)V 16: goto 30 19: astore_0 20: getstatic #7 // Field java/lang/System.out:Ljava/io/PrintStream; 23: ldc #21 // String Finally... 25: invokevirtual #15 // Method java/io/PrintStream.println:(Ljava/lang/String;)V 28: aload_0 29: athrow 30: return Exception table: from to target type 0 8 19 any > > This method encapsulates a BlockCodeBuilder instance, but does not > > provide it to the caller (the `child` below): > > > > This is an example use case of CodeBuilder::block: > > > > codeBuilder.block(blockBuilder -> { > > // block code > > > > if( blockBuilder.isEmpty()) ? > > }); > This is extremely helpful. I can either transport the isEmpty() status via side-effect out of the closure (ugly), or wrap the whole t/c/f in this block() (probably better). In either case I can get rid of my labelToBci calls. Right now I have to cast to BlockCodeBuilderImpl to get to isEmpty(). -- mva -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Fri Aug 19 13:39:38 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 19 Aug 2022 09:39:38 -0400 Subject: Classfile API is missing CodeBuilder hook for various instructions-consuming computation tools In-Reply-To: References: Message-ID: Before we start to do architectural surgery, let's back up a bit and try to distill some requirements and goals.? What problems led you here?? Are these tools we expect people to run full-time, or mostly for debugging when something goes wrong?? Can they be applied to ordinary classfiles, or do they have to work during building? Could this be framed as a kind of transform, instead of a new concept? On 8/19/2022 2:34 AM, Adam Sotona wrote: > > Hi, > > I?ve started working on various helper tools that can give user > answers during code building. > > Answers to questions like: > > * What is actual instructions count > * What is actual/max depts of the stack > * Which locals have been used > * What kinds or exact types are at locals > * What kinds or exact types are on stack > * Show me a log of all instructions > * ? > > Each of this question has specific use case and calculation of each > answer scarifies different amount of resources. > > Unfortunately implementation of tool answering any of the above > questions requires: > > - either heavy use of BufferedCodeBuilder (and complicate existing > building or transformation blocks a lot) > > - or hacking into CodeBuilder(s) implementation > > I would like to propose a similar approach as Linux ?tee? tool does, > so user can attach another consumer of the CodeElements passed to the > CodeBuilder. > > It might be just a single method in CodeBuilder: > > defaultvoid*tee*(Consumer secondListener, > Consumer handler) { > > handler.accept(newTeeCodeBuilder(this, secondListener)); > > ??? } > > Use case may then look for example like this: > > varstackCounter = ...//custom computation tool > /codeBuilder/./tee/(stackCounter, cb -> { > ??????????? ... //block of instructions to be counted > ??????? }); > ??????? stackCounter./actual/(); > ??????? stackCounter./max/(); > > What do you think about this CodeBuilder hook for various > instructions-consuming computation tools? > > Thanks, > > Adam > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Fri Aug 19 14:30:02 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Fri, 19 Aug 2022 14:30:02 +0000 Subject: Classfile API is missing CodeBuilder hook for various instructions-consuming computation tools In-Reply-To: References: Message-ID: These tools are helpers for specific use cases and they do not represent transformations, but rather they collect and provide information necessary for building code. For example I need to know actual stack when I plan to use stack-to-locals (and back) function in my generated code. Another example (which we may embed) is to count max stack when the generator is off. Optional custom logging of the exact instructions dropped to CodeBuilder is another requested use case. All of the above are expensive operations, so they should be applied on user request only. This API does not have much value for transformations, where user can easily listen to each transformed CodeElement and compute whatever is necessary. However when building code using various ?magic? (like blocks, constants auto-detection, and other conveniences?) it is important to see what instructions it really builds into. From: Brian Goetz Date: Friday, 19 August 2022 15:39 To: Adam Sotona , classfile-api-dev at openjdk.org Subject: Re: Classfile API is missing CodeBuilder hook for various instructions-consuming computation tools Before we start to do architectural surgery, let's back up a bit and try to distill some requirements and goals. What problems led you here? Are these tools we expect people to run full-time, or mostly for debugging when something goes wrong? Can they be applied to ordinary classfiles, or do they have to work during building? Could this be framed as a kind of transform, instead of a new concept? On 8/19/2022 2:34 AM, Adam Sotona wrote: Hi, I?ve started working on various helper tools that can give user answers during code building. Answers to questions like: 1. What is actual instructions count 2. What is actual/max depts of the stack 3. Which locals have been used 4. What kinds or exact types are at locals 5. What kinds or exact types are on stack 6. Show me a log of all instructions 7. ? Each of this question has specific use case and calculation of each answer scarifies different amount of resources. Unfortunately implementation of tool answering any of the above questions requires: - either heavy use of BufferedCodeBuilder (and complicate existing building or transformation blocks a lot) - or hacking into CodeBuilder(s) implementation I would like to propose a similar approach as Linux ?tee? tool does, so user can attach another consumer of the CodeElements passed to the CodeBuilder. It might be just a single method in CodeBuilder: default void tee(Consumer secondListener, Consumer handler) { handler.accept(new TeeCodeBuilder(this, secondListener)); } Use case may then look for example like this: var stackCounter = ...//custom computation tool codeBuilder.tee(stackCounter, cb -> { ... //block of instructions to be counted }); stackCounter.actual(); stackCounter.max(); What do you think about this CodeBuilder hook for various instructions-consuming computation tools? Thanks, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Fri Aug 19 15:03:33 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 19 Aug 2022 11:03:33 -0400 Subject: Classfile API is missing CodeBuilder hook for various instructions-consuming computation tools In-Reply-To: References: Message-ID: <01adc293-ad69-6784-0330-9cd07b69ffef@oracle.com> On 8/19/2022 10:30 AM, Adam Sotona wrote: > > These tools are helpers for specific use cases and they do not > represent transformations, but rather they collect and provide > information necessary for building code. > > For example I need to know actual stack when I plan to use > stack-to-locals (and back) function in my generated code. > > Another example (which we may embed) is to count max stack when the > generator is off. > > Optional custom logging of the exact instructions dropped to > CodeBuilder is another requested use case. > > All of the above are expensive operations, so they should be applied > on user request only. > That's what I thought.? So, since they are only applied when you really need them, can't these be treated as "generate the classfile, and then traverse it as a classfile with the API we already have?" This seems like it would work in all the cases when we *can* generate a classfile, which leaves the cases where we would fail to generate a classfile and therefore have nothing to analyze, which is probably a non-zero but not-large fraction of the cases, right? > This API does not have much value for transformations, where user can > easily listen to each transformed CodeElement and compute whatever is > necessary. However when building code using various ?magic? (like > blocks, constants auto-detection, and other conveniences?) it is > important to see what instructions it really builds into. > I think you misunderstood my question.? I didn't mean "is this useful in transformation".? I meant that, we *already* have composable transforms, where the output of one activity is piped into another; there's a lot of similarity here (and they'll use the same underlying plumbing.)? I was suggesting that you could think of teeing as being a (likely no-op) transformation on the building that is already in progress. -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Fri Aug 19 15:49:21 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Fri, 19 Aug 2022 15:49:21 +0000 Subject: Classfile API is missing CodeBuilder hook for various instructions-consuming computation tools In-Reply-To: <01adc293-ad69-6784-0330-9cd07b69ffef@oracle.com> References: <01adc293-ad69-6784-0330-9cd07b69ffef@oracle.com> Message-ID: From: Brian Goetz That's what I thought. So, since they are only applied when you really need them, can't these be treated as "generate the classfile, and then traverse it as a classfile with the API we already have?" This seems like it would work in all the cases when we *can* generate a classfile, which leaves the cases where we would fail to generate a classfile and therefore have nothing to analyze, which is probably a non-zero but not-large fraction of the cases, right? Right, all these cases are to support generation in progress. Can we now even start transformation of a bytecode fragment that has not been finished yet (with our current API) ? I think using transformations for these cases is an overkill. We just need to dynamically collect various information from the actual CodeBuilder. These should be much tighter co-processors connected to CodeBuilder. I think you misunderstood my question. I didn't mean "is this useful in transformation". I meant that, we *already* have composable transforms, where the output of one activity is piped into another; there's a lot of similarity here (and they'll use the same underlying plumbing.) I was suggesting that you could think of teeing as being a (likely no-op) transformation on the building that is already in progress. Using composable transformations for example to calculate actual stack content to emit multiple stack-to-locals and locals-to-stack sequences of instructions ? that is beyond my imagination yet. I?ve previously re-used javac stack tracker to implement this particular feature, where javac had full control of each individual instruction, so it was able to instruct stack-to-locals and back. However such approach prevents use of any advanced CodeBuilder features. When you for example optimize the code generator to use blocks, you lose track of the exact instructions passing down to the CodeBuilder::with. This API proposes an option to add listener of what falls down to the CodeBuilder when drop something into some of its convenience methods. I?m open to any other proposal how to hook these code helpers. We may for example implement them internally and enable them with specific switches. However even for the internal helpers we still need a hook to listen on CodeBuilder. Actually there is a hard-coded partial support for local variables allocation, which should also fall into this group of code-builder helpers (actually tracking instructions related to local variables). -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Tue Aug 23 22:39:44 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 23 Aug 2022 22:39:44 +0000 Subject: Try/catch/finally builder Message-ID: Hi, Here?s a prototype of a try/catch/finally builder (that should replace the current approach): https://github.com/openjdk/jdk-sandbox/compare/classfile-api-branch...PaulSandoz:jdk-sandbox:try-catch-finally-builder?expand=1 So far it all seems to work and produces correct code for nested building, generating similar code as the source compiler produces. I need to do more thorough testing and commit unit tests. ? One challenge is determining when a block exits. A branching instruction that exits the block should result in inlining of the finally code before the instruction, but it's hard to precisely determine if the branch target is within or outside of the block. At the moment I hard code to checking if the target is the break label of the block, otherwise it is assumed the branch does not exit the block. I don?t think this is generally decidable unless all block instructions are first buffered. Paul. From brian.goetz at oracle.com Wed Aug 24 01:12:13 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 24 Aug 2022 01:12:13 +0000 Subject: Try/catch/finally builder In-Reply-To: References: Message-ID: <6D1023A6-1B0D-4E88-88FD-60B913ACC134@oracle.com> Maybe the TCB should have an ?emit finally? method that causes the finally lambda to be called at that point? Sent from my MacBook Wheel > On Aug 23, 2022, at 6:39 PM, Paul Sandoz wrote: > > ?Hi, > > Here?s a prototype of a try/catch/finally builder (that should replace the current approach): > > https://github.com/openjdk/jdk-sandbox/compare/classfile-api-branch...PaulSandoz:jdk-sandbox:try-catch-finally-builder?expand=1 > > So far it all seems to work and produces correct code for nested building, generating similar code as the source compiler produces. > > I need to do more thorough testing and commit unit tests. > > ? > > One challenge is determining when a block exits. A branching instruction that exits the block should result in inlining of the finally code before the instruction, but it's hard to precisely determine if the branch target is within or outside of the block. At the moment I hard code to checking if the target is the break label of the block, otherwise it is assumed the branch does not exit the block. I don?t think this is generally decidable unless all block instructions are first buffered. > > Paul. From adam.sotona at oracle.com Wed Aug 24 08:50:58 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Wed, 24 Aug 2022 08:50:58 +0000 Subject: Try/catch/finally builder In-Reply-To: References: Message-ID: It looks good to me (I assume CodeBuilder::trying2 will replace CodeBuidler::trying) Just one implementation comment: For successful synchronous stack tracking it is critical to always declare exceptionCatch or exceptionCatchAll before the related handler block starts (before the handler Label mentioned in the exceptionCatch is resolved). Thanks, Adam From: classfile-api-dev on behalf of Paul Sandoz Date: Wednesday, 24 August 2022 0:39 To: classfile-api-dev at openjdk.org Subject: Try/catch/finally builder Hi, Here?s a prototype of a try/catch/finally builder (that should replace the current approach): https://github.com/openjdk/jdk-sandbox/compare/classfile-api-branch...PaulSandoz:jdk-sandbox:try-catch-finally-builder?expand=1 So far it all seems to work and produces correct code for nested building, generating similar code as the source compiler produces. I need to do more thorough testing and commit unit tests. ? One challenge is determining when a block exits. A branching instruction that exits the block should result in inlining of the finally code before the instruction, but it's hard to precisely determine if the branch target is within or outside of the block. At the moment I hard code to checking if the target is the break label of the block, otherwise it is assumed the branch does not exit the block. I don?t think this is generally decidable unless all block instructions are first buffered. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From heidinga at redhat.com Wed Aug 24 13:22:25 2022 From: heidinga at redhat.com (Dan Heidinga) Date: Wed, 24 Aug 2022 09:22:25 -0400 Subject: Experience report In-Reply-To: <6da06023-3d91-d0bd-55c2-b6f868eddf9a@oracle.com> References: <8ebb41d5-c598-e825-1062-14d1fe3b150d@oracle.com> <6da06023-3d91-d0bd-55c2-b6f868eddf9a@oracle.com> Message-ID: On Tue, Aug 16, 2022 at 10:28 AM Brian Goetz wrote: > > > On 8/15/2022 4:26 PM, Dan Heidinga wrote: > > Thanks for the feedback, Brian. > > > > I've worked through the suggestions on the various ways to update an > > Attribute {use a stateful ClassTransform, use > > builder.original().get()} and found the "original().get()" approach > > felt the cleanest and clearest for what I was doing. > > By that, do you mean "cleanest of the terrible options", or "seemed a > good enough option"? > After playing more with both options, I think both have their uses. original().get() works well for quick prototyping and is easy to get something up and running. In my case, it's guaranteed that the "original" ClassModel exists so I don't have to add any extra error handling for that case. ofStateful() seems like a more generally applicable solution to detecting something, holding it, and processing it later when you aren't guaranteed an original ClassModel exists (not entirely sure when that would be the case). This is really useful for the set of Attributes that have the "there can only be one" invariant. What made it feel like too much was trying to write the "new ClassTransform() { ...} " inline. Factoring that out into a separately named class makes the whole approach much clearer. My current approach is to use a separate class for SingleAttributeTransforms that accepts three arguments: * the Attribute class to detect * an ifPresent Function argument, and * an ifAbsent Supplier argument. I prototyped something [0] that works for my use case (though the generics aren't right yet as not all Attributes are ClassElements) but haven't thought about how well it would generalize to allow tracking attributes that are one-per method. Current use of it look like: ClassTransform singleAttribT(final ArrayList nestMembers) { return ClassTransform.ofStateful( () -> { return new SingleAttributeTransform( NestMembersAttribute.class, nma -> { return NestMembersAttribute.withSymbols((NestMembersAttribute)nma, nestMembers);}, () -> { return NestMembersAttribute.ofSymbols(nestMembers);}); } ); } All that to say - original().get() works for quickly hacking something up and ofStateful is likely the solution to reach for when writing something more maintainable. OfStateful benefits from defining a separate class out-of-line and then resulting code looks pretty good. [0] https://github.com/DanHeidinga/jdk-sandbox/pull/1/commits/cfa5acf8594075b8786aa1bf86dd8c8850175728 > > > While working on that, I also added the ::with & ::withSymbols methods > > I had proposed and opened a PR [0] to see if the fleshed out API fit > > with the general way you wanted to evolve this. > > I had a look at the API suggestions and I have a counter-suggestion. > > The operation that you are trying to enable is adding to (and I predict > eventually, removing from) list-holding attributes. You propose four > new methods for each attribute: > > NMA with(NMA base, List additions) > NMA with(NMA base, CE... additions) > NMA withSymbols(NMA base, List additions) > NMA withSymbols(NMA base, CD... additions) > > So that's 2 representations (CE/CD) x 2 API styles (list, varargs) x N > attributes (4N), which will get worse if the attribute is not merely a > holder for a list, but might have other stuff to be modified too. If > you want to support removals, that's 2x more. > > Really what you need is four methods, somewhere: > > List adding(List, List) > List adding(List, CE...) > List addingSymbols(List, List) > List addingSymbols(List, CD...) > > and invoke it like: > > case NMA a -> NMA.of(adding(a.nestMembers(), NEW_STUFF)); > > And this would cover all attributes, plus would support dealing with > attributes that have more than just a List in them. The question is > where to put these to make them discoverable. > I took another pass through the Attributes that deal with Lists of things and most cases are well handled by existing List helpers. It's only the CE/CD cases that need data conversion and would therefore benefit from additional methods to create their lists. If we expect to be content with these 4 methods, then adding them as static methods on ClassEntry seems reasonable. It makes them easy to find and avoids a garbage "Helpers/Utils" class. If we expect more of these methods (and I currently don't but that may be lack of imagination), then a garbage class might be better. --Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Wed Aug 24 13:54:07 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Wed, 24 Aug 2022 13:54:07 +0000 Subject: Classfile API Synchronous Stack Tracking in CodeBuilder Message-ID: Hi, I would like to hear your comments to proposed optional stack tracking functionality integrated in CodeBuilder. Here is the pull request proposing to add: * Classfile.Option::trackStack boolean option with default to false (the new option enabling stack tracking during code build) * Optional> CodeBuilder::stack ? a method returning view of the actual stack synchronously during the build process (if enabled) * Optional CodeBuilder::maxStackSize - a method calculating maxStackSize (if enabled) * impl.StackTracker implementation ? a lightweight synchronous stack tracker * StackTrackerTest tests Beside the stack tracking the pull request adds ConstantInstruction::typeKind and LoadableConstantEntry::typeKind methods self-describing ConstantInstruction and LoadableConstantEntry (not just) for the stack tracking purpose. Here is the pull request: https://github.com/openjdk/jdk-sandbox/pull/34 Primary purpose of the lightweight synchronous stack tracking is to support building of complex swap functions like stack-to-locals and locals-to-stack in CodeBuilder. Example use case can be found in the StackTrackerTest: Classfile.build(ClassDesc.of("Foo"), List.of(Classfile.Option.trackStack(true)), clb -> clb.withMethodBody("m", MethodTypeDesc.of(ConstantDescs.CD_Void), 0, cob -> { assertEquals(cob.stack().get(), List.of()); cob.aload(0); assertEquals(cob.stack().get(), List.of(ReferenceType)); cob.lconst_0(); assertEquals(cob.stack().get(), List.of(LongType, ReferenceType)); cob.trying(tryb -> { assertEquals(tryb.stack().get(), List.of(LongType, ReferenceType)); tryb.iconst_1(); assertEquals(tryb.stack().get(), List.of(IntType, LongType, ReferenceType)); tryb.ifThen(thb -> { assertEquals(thb.stack().get(), List.of(LongType, ReferenceType)); thb.constantInstruction(ClassDesc.of("Phee")); assertEquals(thb.stack().get(), List.of(ReferenceType, LongType, ReferenceType)); thb.athrow(); assertFalse(thb.stack().isPresent()); }); assertEquals(tryb.stack().get(), List.of(LongType, ReferenceType)); tryb.return_(); assertFalse(tryb.stack().isPresent()); }, catchb -> catchb.catching(ClassDesc.of("Phee"), cb -> { assertEquals(cb.stack().get(), List.of(ReferenceType)); cb.athrow(); assertFalse(cb.stack().isPresent()); })); assertTrue(cob.maxStackSize().isPresent()); assertEquals((int)cob.maxStackSize().get(), 4); })); Thanks, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Wed Aug 24 17:57:48 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 24 Aug 2022 17:57:48 +0000 Subject: Try/catch/finally builder In-Reply-To: <6D1023A6-1B0D-4E88-88FD-60B913ACC134@oracle.com> References: <6D1023A6-1B0D-4E88-88FD-60B913ACC134@oracle.com> Message-ID: <76FD98D4-EEF5-44E1-95E9-25441B4BEA19@oracle.com> Or, alternatively, the user can provide an optional BiPredicate, which is called for every branch instruction of a try or catch block, and returns true if the branches target is known to exit the block and therefore the finally blocks need to be inlined before it. The default implementation is specified to behave as it does in the prototype, and can be composed. That seems to give the developer the control they need without explicitly emitting finally blocks, which could get more complex when nesting trying builders. Paul. > On Aug 23, 2022, at 6:12 PM, Brian Goetz wrote: > > Maybe the TCB should have an ?emit finally? method that causes the finally lambda to be called at that point? > > Sent from my MacBook Wheel > >> On Aug 23, 2022, at 6:39 PM, Paul Sandoz wrote: >> >> ?Hi, >> >> Here?s a prototype of a try/catch/finally builder (that should replace the current approach): >> >> https://github.com/openjdk/jdk-sandbox/compare/classfile-api-branch...PaulSandoz:jdk-sandbox:try-catch-finally-builder?expand=1 >> >> So far it all seems to work and produces correct code for nested building, generating similar code as the source compiler produces. >> >> I need to do more thorough testing and commit unit tests. >> >> ? >> >> One challenge is determining when a block exits. A branching instruction that exits the block should result in inlining of the finally code before the instruction, but it's hard to precisely determine if the branch target is within or outside of the block. At the moment I hard code to checking if the target is the break label of the block, otherwise it is assumed the branch does not exit the block. I don?t think this is generally decidable unless all block instructions are first buffered. >> >> Paul. From paul.sandoz at oracle.com Wed Aug 24 18:01:54 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 24 Aug 2022 18:01:54 +0000 Subject: Try/catch/finally builder In-Reply-To: References: Message-ID: <5CD423E3-16E8-4F4E-95B5-E3EEEDEF4205@oracle.com> > On Aug 24, 2022, at 1:50 AM, Adam Sotona wrote: > > It looks good to me (I assume CodeBuilder::trying2 will replace CodeBuidler::trying) > Yes. > Just one implementation comment: > For successful synchronous stack tracking it is critical to always declare exceptionCatch or exceptionCatchAll before the related handler block starts (before the handler Label mentioned in the exceptionCatch is resolved). Ah! I will move the exception*() calls to be above the associated .start() and operate on the parent block. Paul. From brian.goetz at oracle.com Wed Aug 24 18:55:06 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 24 Aug 2022 14:55:06 -0400 Subject: Experience report In-Reply-To: References: <8ebb41d5-c598-e825-1062-14d1fe3b150d@oracle.com> <6da06023-3d91-d0bd-55c2-b6f868eddf9a@oracle.com> Message-ID: <71bfabc1-29fd-ad90-518c-45298d62c5ca@oracle.com> I'd say let's give this a try and see what we think of the resulting transformation code.? It feels like right-sizing the overhead at first look. On 8/24/2022 9:22 AM, Dan Heidinga wrote: > > > > I took another pass through the?Attributes that deal with Lists of > things and most cases are well handled by existing List helpers.? It's > only the CE/CD cases that need data conversion and would therefore > benefit from additional methods to create their lists. > > If we expect to be content with these 4 methods, then adding them as > static methods on ClassEntry seems reasonable. It makes them easy to > find and avoids a garbage "Helpers/Utils" class. If we expect more of > these methods (and I currently don't but that may be lack of > imagination), then a garbage class might be better. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Wed Aug 24 19:38:23 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 24 Aug 2022 19:38:23 +0000 Subject: Classfile API Synchronous Stack Tracking in CodeBuilder In-Reply-To: References: Message-ID: <31D2B21E-4EA2-4680-B882-5C5A866A78AE@oracle.com> The implementation looks good (I did not verify the correctness of all the instructions in StackTracker). Seems useful functionality that is otherwise laborious and error prone to implement. Requiring ExceptionCatch (and ExceptionCatchAll, not suppported) be reported before the target label is bound is awkward, but it's clear why. I think that would require documentation, and perhaps it should fail since it will otherwise report incorrect stack data? Paul. > On Aug 24, 2022, at 6:54 AM, Adam Sotona wrote: > > Hi, > I would like to hear your comments to proposed optional stack tracking functionality integrated in CodeBuilder. > > Here is the pull request proposing to add: > ? Classfile.Option::trackStack boolean option with default to false (the new option enabling stack tracking during code build) > ? Optional> CodeBuilder::stack ? a method returning view of the actual stack synchronously during the build process (if enabled) > ? Optional CodeBuilder::maxStackSize - a method calculating maxStackSize (if enabled) > ? impl.StackTracker implementation ? a lightweight synchronous stack tracker > ? StackTrackerTest tests > > Beside the stack tracking the pull request adds ConstantInstruction::typeKind and LoadableConstantEntry::typeKind methods self-describing ConstantInstruction and LoadableConstantEntry (not just) for the stack tracking purpose. > > Here is the pull request: > https://github.com/openjdk/jdk-sandbox/pull/34 > > Primary purpose of the lightweight synchronous stack tracking is to support building of complex swap functions like stack-to-locals and locals-to-stack in CodeBuilder. > > Example use case can be found in the StackTrackerTest: > Classfile.build(ClassDesc.of("Foo"), List.of(Classfile.Option.trackStack(true)), clb -> > clb.withMethodBody("m", MethodTypeDesc.of(ConstantDescs.CD_Void), 0, cob -> { > assertEquals(cob.stack().get(), List.of()); > cob.aload(0); > assertEquals(cob.stack().get(), List.of(ReferenceType)); > cob.lconst_0(); > assertEquals(cob.stack().get(), List.of(LongType, ReferenceType)); > cob.trying(tryb -> { > assertEquals(tryb.stack().get(), List.of(LongType, ReferenceType)); > tryb.iconst_1(); > assertEquals(tryb.stack().get(), List.of(IntType, LongType, ReferenceType)); > tryb.ifThen(thb -> { > assertEquals(thb.stack().get(), List.of(LongType, ReferenceType)); > thb.constantInstruction(ClassDesc.of("Phee")); > assertEquals(thb.stack().get(), List.of(ReferenceType, LongType, ReferenceType)); > thb.athrow(); > assertFalse(thb.stack().isPresent()); > }); > assertEquals(tryb.stack().get(), List.of(LongType, ReferenceType)); > tryb.return_(); > assertFalse(tryb.stack().isPresent()); > }, catchb -> catchb.catching(ClassDesc.of("Phee"), cb -> { > assertEquals(cb.stack().get(), List.of(ReferenceType)); > cb.athrow(); > assertFalse(cb.stack().isPresent()); > })); > assertTrue(cob.maxStackSize().isPresent()); > assertEquals((int)cob.maxStackSize().get(), 4); > })); > > Thanks, > Adam From brian.goetz at oracle.com Wed Aug 24 20:49:34 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 24 Aug 2022 16:49:34 -0400 Subject: What to do about dead labels? In-Reply-To: <8F1C2FA4-D993-410E-A04F-0DD73FA0629D@oracle.com> References: <830b0f19-7d10-7ca9-d9ed-bd785c49a48a@oracle.com> <8F1C2FA4-D993-410E-A04F-0DD73FA0629D@oracle.com> Message-ID: <1e1fddc9-a8c4-103b-fc92-0e18457eda77@oracle.com> > Perhaps it comes down to an option to order such pseudo instructions or not? I thought some more about this and I think we may not be able to do this. The order of the exception table is significant (JVMS 2.10); it indicates the order in which handlers are tried.? This may, or may not, correspond to the order in which the handlers appear in the code array.? (I recall we had some earlier bugs where we were not diligent about preserving this order.) The builder is going to write them to the exception_table in the order they are presented.? So if we perturb the presentation order, we'll be perturbing the order of the exception table, and changing the semantics of the classfile.? And if we can't do this for exceptions, there's little point in trying to do it for debug information or line numbers. So I think that brings us to: it's an error if if any of these reference dead labels.? And I think this is the behavior of the code now.? So I think all we need here is (a) to remove the FIXME comments in DirectCodeBuilder like "@@@ Filter out LVs whose boundary labels are not defined?" and (b) write some tests for this behavior. From paul.sandoz at oracle.com Wed Aug 24 20:51:19 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 24 Aug 2022 20:51:19 +0000 Subject: Try/catch/finally builder In-Reply-To: <76FD98D4-EEF5-44E1-95E9-25441B4BEA19@oracle.com> References: <6D1023A6-1B0D-4E88-88FD-60B913ACC134@oracle.com> <76FD98D4-EEF5-44E1-95E9-25441B4BEA19@oracle.com> Message-ID: Here?s an update: https://github.com/openjdk/jdk-sandbox/compare/classfile-api-branch...PaulSandoz:jdk-sandbox:try-catch-finally-builder?expand=1 sealed interface CatchFinallyBuilder permits CatchFinallyBuilderImpl { CatchFinallyBuilder catching(ClassDesc exceptionType, Consumer catchHandler); default void finally_(Consumer finallyHandler) { finally_(INLINE_FINALLY_ON_BREAK, finallyHandler); } void finally_(BiPredicate inlineFinallyTest, Consumer finallyHandler); BiPredicate INLINE_FINALLY_ON_BREAK = (b, i) -> b.breakLabel() == i.target(); } The terminal finally_ builder method accepts an optional predicate. Paul. On Aug 24, 2022, at 10:57 AM, Paul Sandoz > wrote: Or, alternatively, the user can provide an optional BiPredicate, which is called for every branch instruction of a try or catch block, and returns true if the branches target is known to exit the block and therefore the finally blocks need to be inlined before it. The default implementation is specified to behave as it does in the prototype, and can be composed. That seems to give the developer the control they need without explicitly emitting finally blocks, which could get more complex when nesting trying builders. Paul. On Aug 23, 2022, at 6:12 PM, Brian Goetz > wrote: Maybe the TCB should have an ?emit finally? method that causes the finally lambda to be called at that point? Sent from my MacBook Wheel On Aug 23, 2022, at 6:39 PM, Paul Sandoz > wrote: ?Hi, Here?s a prototype of a try/catch/finally builder (that should replace the current approach): https://github.com/openjdk/jdk-sandbox/compare/classfile-api-branch...PaulSandoz:jdk-sandbox:try-catch-finally-builder?expand=1 So far it all seems to work and produces correct code for nested building, generating similar code as the source compiler produces. I need to do more thorough testing and commit unit tests. ? One challenge is determining when a block exits. A branching instruction that exits the block should result in inlining of the finally code before the instruction, but it's hard to precisely determine if the branch target is within or outside of the block. At the moment I hard code to checking if the target is the break label of the block, otherwise it is assumed the branch does not exit the block. I don?t think this is generally decidable unless all block instructions are first buffered. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Wed Aug 24 21:02:57 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 24 Aug 2022 21:02:57 +0000 Subject: What to do about dead labels? In-Reply-To: <1e1fddc9-a8c4-103b-fc92-0e18457eda77@oracle.com> References: <830b0f19-7d10-7ca9-d9ed-bd785c49a48a@oracle.com> <8F1C2FA4-D993-410E-A04F-0DD73FA0629D@oracle.com> <1e1fddc9-a8c4-103b-fc92-0e18457eda77@oracle.com> Message-ID: <79478233-6EAD-4497-8822-207B41DBEA14@oracle.com> Good point, we cannot reorder. (Although FWIW I suspect we could get away with it for bytecode produced by javac at least for its current implementation, since the order would be preserved. That?s nice to know, but not helpful!) Paul. > On Aug 24, 2022, at 1:49 PM, Brian Goetz wrote: > > >> Perhaps it comes down to an option to order such pseudo instructions or not? > > I thought some more about this and I think we may not be able to do this. > > The order of the exception table is significant (JVMS 2.10); it indicates the order in which handlers are tried. This may, or may not, correspond to the order in which the handlers appear in the code array. (I recall we had some earlier bugs where we were not diligent about preserving this order.) > > The builder is going to write them to the exception_table in the order they are presented. So if we perturb the presentation order, we'll be perturbing the order of the exception table, and changing the semantics of the classfile. And if we can't do this for exceptions, there's little point in trying to do it for debug information or line numbers. > > So I think that brings us to: it's an error if if any of these reference dead labels. And I think this is the behavior of the code now. So I think all we need here is (a) to remove the FIXME comments in DirectCodeBuilder like "@@@ Filter out LVs whose boundary labels are not defined?" and (b) write some tests for this behavior. > From adam.sotona at oracle.com Thu Aug 25 06:18:24 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Thu, 25 Aug 2022 06:18:24 +0000 Subject: Classfile API Synchronous Stack Tracking in CodeBuilder In-Reply-To: <31D2B21E-4EA2-4680-B882-5C5A866A78AE@oracle.com> References: <31D2B21E-4EA2-4680-B882-5C5A866A78AE@oracle.com> Message-ID: Yes, it should be documented. Or maybe we can consider removal of handler label from ExceptionCatch and stream and collect it synchronously at the handler start position. Practically it defines handler block start, where try start and try end labels are just additional information. StackTracker is actually not verifying content nor merging stack at branch targets, however it indicates lost track by returning empty Optional. Handler block start usually follows "no-flow" instruction (goto, return, athrow...) - and it represents a (temporary) dead code (as the handler has not been yet declared), so the stack tracking is lost and stack method indicates failure by returning empty Optional. Thanks, Adam ________________________________ From: Paul Sandoz Requiring ExceptionCatch (and ExceptionCatchAll, not suppported) be reported before the target label is bound is awkward, but it's clear why. I think that would require documentation, and perhaps it should fail since it will otherwise report incorrect stack data? -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Thu Aug 25 10:04:28 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Thu, 25 Aug 2022 10:04:28 +0000 Subject: Classfile API Synchronous Stack Tracking in CodeBuilder In-Reply-To: References: <31D2B21E-4EA2-4680-B882-5C5A866A78AE@oracle.com> Message-ID: Oops, I missed the other thread about keeping the original order of handlers, so I?m taking the idea of synchronous streaming of ExceptionCatch back. Adam From: classfile-api-dev on behalf of Adam Sotona Date: Thursday, 25 August 2022 8:18 To: Paul Sandoz Cc: classfile-api-dev at openjdk.org Subject: Re: Classfile API Synchronous Stack Tracking in CodeBuilder Yes, it should be documented. Or maybe we can consider removal of handler label from ExceptionCatch and stream and collect it synchronously at the handler start position. Practically it defines handler block start, where try start and try end labels are just additional information. StackTracker is actually not verifying content nor merging stack at branch targets, however it indicates lost track by returning empty Optional. Handler block start usually follows "no-flow" instruction (goto, return, athrow...) - and it represents a (temporary) dead code (as the handler has not been yet declared), so the stack tracking is lost and stack method indicates failure by returning empty Optional. Thanks, Adam ________________________________ From: Paul Sandoz Requiring ExceptionCatch (and ExceptionCatchAll, not suppported) be reported before the target label is bound is awkward, but it's clear why. I think that would require documentation, and perhaps it should fail since it will otherwise report incorrect stack data? -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Thu Aug 25 12:49:23 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 25 Aug 2022 08:49:23 -0400 Subject: Try/catch/finally builder In-Reply-To: References: <6D1023A6-1B0D-4E88-88FD-60B913ACC134@oracle.com> <76FD98D4-EEF5-44E1-95E9-25441B4BEA19@oracle.com> Message-ID: <09f2222e-8b9e-527a-ac73-b8e1948f959a@oracle.com> I'm still a little uncomfortable at the BiPredicate approach, for two reasons: ?- It's more "magic" than having the user explicitly say "unroll the finally here" ?- I'm not convinced the usability is better You shared the following example: ``` ??????? try { ??????????? try { ??????????????? System.out.println("INNER TRY START"); ??????????????? String s = args[0]; ??????????????? if (s.length() < 10) { ??????????????????? [0] ??????????????????? return -1; ??????????????? } ??????????????? System.out.println("INNER TRY END"); ??????????????? // Exit ??????????????? // [1] ??????????? } catch (RuntimeException e) { ??????????????? System.out.println("INNER RUNTIME EXCEPTION"); ??????????????? // [2] ??????????????? throw e; ??????????? } finally { ??????????????? System.out.println("INNER FINALLY"); ??????????? } ??????????? System.out.println("OUTER TRY"); ??????????? // [3] ??????????? // Exit ??????? } catch (RuntimeException e) { ??????????? System.out.println("OUTER RUNTIME EXCEPTION"); ??????????? // [4] ??????????? throw e; ??????? } finally { ??????????? System.out.println("OUTER FINALLY"); ??????? } ``` I must admit that at first look, I don't know where the finally blocks have to be inserted, but looking at the javap output, it looks like: ?- [0] -- inner and outer finally ?- [1] -- inner finally (then falls out) ?- [2] -- nothing (handled as stack unwinds by any handler) ?- [3] -- outer ?- [4] -- nothing (handled by any handler) Plus a pair of "any" handlers, the first of which does the inner finally, the second does the outer, and the second is in scope for the first.? Whew! There's several groups of questions here. One is "what does the user know, and when do they know it."? A block like the above might not be generated with pure try-catch builder; it might be that the inner block comes from the class being adapted, and the outer block is being added by a transform.? So the user may not even know about the inner try-catch block!? In this case, we would have a problem at [0] only, where the code being adapted has already inlined the inner finally, but we would have to realize that prior to a return, we'd have to also unroll the inner finally. Which exposes problem #1 -- it's not just branch instructions that the BiPredicate would have to examine, its also returns (unless you are treating these unconditionally as needing finally unrolling.) Setting that aside, the real question is whether a branch target that is a branch target is internal to the try+catch blocks, or "outside".? Which we would handle by examining the label.? But the usability of this is not so great, because the label may not have been bound yet, and so all the user can do is compare it for identity with the N labels known to be in the block.? That sounds like kind of a pain for the user to implement. So here's another possible thought: what if BlockBuilder could dispense labels, as well as dispensing blocks, and could then keep track of containment?? So if you are branching to a label *outside this block builder* (including returning), we'd inline the finally at that point.? (With nested block builders, we'd have to keep track of which BBs belonged to other BBs, but we already do this.) On 8/24/2022 4:51 PM, Paul Sandoz wrote: > Here?s an update: > > https://github.com/openjdk/jdk-sandbox/compare/classfile-api-branch...PaulSandoz:jdk-sandbox:try-catch-finally-builder?expand=1 > sealed interface CatchFinallyBuilder permits CatchFinallyBuilderImpl { > CatchFinallyBuilder catching(ClassDesc exceptionType,Consumer catchHandler); > > default void finally_(Consumer finallyHandler) { > finally_(INLINE_FINALLY_ON_BREAK, finallyHandler); > } > > void finally_(BiPredicate inlineFinallyTest, > Consumer finallyHandler); > > BiPredicate INLINE_FINALLY_ON_BREAK = > (b, i) -> b.breakLabel() == i.target(); > } > The terminal finally_ builder method accepts an optional predicate. > > Paul. > >> On Aug 24, 2022, at 10:57 AM, Paul Sandoz wrote: >> >> Or, alternatively, the user can provide an optional >> BiPredicate, which is called for >> every branch instruction of a try or catch block, and returns true >> if?the branches target is known to exit the block and therefore the >> finally blocks need to be inlined before it. The default >> implementation is specified to behave as it does in the prototype, >> and?can be composed. >> >> That seems to give the developer the control they need without >> explicitly emitting finally blocks, which could get more complex when >> nesting trying builders. >> >> Paul. >> >>> On Aug 23, 2022, at 6:12 PM, Brian Goetz wrote: >>> >>> Maybe the TCB should have an ?emit finally? method that causes the >>> finally lambda to be called at that point? >>> >>> Sent from my MacBook Wheel >>> >>>> On Aug 23, 2022, at 6:39 PM, Paul Sandoz >>>> wrote: >>>> >>>> ?Hi, >>>> >>>> Here?s a prototype of a try/catch/finally builder (that should >>>> replace the current approach): >>>> >>>> https://github.com/openjdk/jdk-sandbox/compare/classfile-api-branch...PaulSandoz:jdk-sandbox:try-catch-finally-builder?expand=1 >>>> >>>> So far it all seems to work and produces correct code for nested >>>> building, generating similar code as the source compiler produces. >>>> >>>> I need to do more thorough testing and commit unit tests. >>>> >>>> ? >>>> >>>> One challenge is determining when a block exits. A branching >>>> instruction that exits the block should result in inlining of the >>>> finally code before the instruction, but it's hard to >>>> precisely?determine if the branch target is within or outside of >>>> the block. At the moment I hard code to checking if the target is >>>> the break label of the block, otherwise it is assumed the >>>> branch?does not exit the block. I don?t think this is generally >>>> decidable unless all block instructions are first buffered. >>>> >>>> Paul. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Fri Aug 26 13:58:44 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Fri, 26 Aug 2022 13:58:44 +0000 Subject: Classfile API Synchronous Stack Tracking in CodeBuilder In-Reply-To: References: Message-ID: Based on various optiona how to expose the StackTracker through the Classfile API I?ve decided to refactor the proposal in: https://github.com/openjdk/jdk-sandbox/pull/34 Instead of embedded implementation and Classfile.Option here is a proposal to expose StackTracker as CodeTransform and implement CodeBuilder::transforming method: /** * Builds code fragment generated by the handler and synchronously transformed. * @param transform the transform to apply to the code generated by the handler * @param handler handler that receives a {@linkplain BlockCodeBuilder} to * generate the code. * @return this builder */ default CodeBuilder transforming(CodeTransform transform, Consumer handler) { var resolved = transform.resolve(this); resolved.startHandler().run(); handler.accept(new TransformingCodeBuilder(this, resolved.consumer())); resolved.endHandler().run(); return this; } Modified usage of the StackTracker is reflected in the StackTrackerTest: Classfile.build(ClassDesc.of("Foo"), clb -> clb.withMethodBody("m", MethodTypeDesc.of(ConstantDescs.CD_Void), 0, cob -> { var stackTracker = new StackTracker(); cob.transforming(stackTracker, stcb -> { assertEquals(stackTracker.stack().get(), List.of()); stcb.aload(0); assertEquals(stackTracker.stack().get(), List.of(ReferenceType)); stcb.lconst_0(); assertEquals(stackTracker.stack().get(), List.of(LongType, ReferenceType)); stcb.trying(tryb -> { assertEquals(stackTracker.stack().get(), List.of(LongType, ReferenceType)); tryb.iconst_1(); assertEquals(stackTracker.stack().get(), List.of(IntType, LongType, ReferenceType)); tryb.ifThen(thb -> { assertEquals(stackTracker.stack().get(), List.of(LongType, ReferenceType)); thb.constantInstruction(ClassDesc.of("Phee")); assertEquals(stackTracker.stack().get(), List.of(ReferenceType, LongType, ReferenceType)); thb.athrow(); assertFalse(stackTracker.stack().isPresent()); }); assertEquals(stackTracker.stack().get(), List.of(LongType, ReferenceType)); tryb.return_(); assertFalse(stackTracker.stack().isPresent()); }, catchb -> catchb.catching(ClassDesc.of("Phee"), cb -> { assertEquals(stackTracker.stack().get(), List.of(ReferenceType)); cb.athrow(); assertFalse(stackTracker.stack().isPresent()); })); }); assertTrue(stackTracker.maxStackSize().isPresent()); assertEquals((int)stackTracker.maxStackSize().get(), 4); })); Thanks, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Fri Aug 26 14:12:50 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 26 Aug 2022 10:12:50 -0400 Subject: Classfile API Synchronous Stack Tracking in CodeBuilder In-Reply-To: References: Message-ID: I like how this leans on the existing Transform abstraction -- and it is useful beyond just tracking -- it means existing transforms can be combined with generation if that is convenient. If the user wants to chain some transforms together *and* wants to retain references to each element in the chain -- which they will probably want to do if there are multiple kinds of "trackers" here, such as tracking stack + tracking locals -- then there is a little trickiness to setting the pipeline up right.? The user will have to do something like: ??? withMethodBody(b -> { ??????? StackTracker st = StackTracker.of(); ??????? LocalTracker lt = LocalTracker.of(); ??????? transforming(st.andThen(lt), cb -> { ... build here, can use st/lt ... }); ??? }); What's happening here is that the transforms are stateful, but the user is creating the transform instances directly, rather than through a factory, so they can retain references to them. This is totally OK, as long as the user doesn't try to reuse the same StackTracker instance across multiple methods at the same time. The spec for StackTracker and friends will probably need some extensive examples to guide users to the right idiom. On 8/26/2022 9:58 AM, Adam Sotona wrote: > > Based on various optiona how to expose the StackTracker through the > Classfile API I?ve decided to refactor the proposal in: > > https://github.com/openjdk/jdk-sandbox/pull/34 > > Instead of embedded implementation and Classfile.Option here is a > proposal to expose StackTracker as CodeTransform and implement > CodeBuilder::transforming method: > > /** > > * > *Builds**code**fragment**generated**by**the**handler**and**synchronously**transformed.* > > * *@param*transform thetransformtoapplytothecodegeneratedbythehandler > > * *@param*handler handlerthatreceivesa {*@linkplain*BlockCodeBuilder} to > > * generatethecode. > > * *@return*thisbuilder > > */ > > defaultCodeBuilder *transforming*(CodeTransform transform, > Consumer handler) { > > varresolved = transform.resolve(this); > > resolved.startHandler().run(); > > ??? handler.accept(newTransformingCodeBuilder(this, resolved.consumer())); > > resolved.endHandler().run(); > > returnthis; > > } > > Modified usage of the StackTracker is reflected in the StackTrackerTest: > > Classfile./build/(ClassDesc./of/("Foo"), clb -> > ??? clb.withMethodBody("m", > MethodTypeDesc./of/(ConstantDescs./CD_Void/), 0, cob -> { > varstackTracker = newStackTracker(); > ??????? cob.*transforming*(stackTracker, stcb -> { > /assertEquals/(stackTracker.stack().get(), List./of/()); > ??????????? stcb.aload(0); > /assertEquals/(stackTracker.stack().get(), List./of/(/ReferenceType/)); > ??????????? stcb.lconst_0(); > /assertEquals/(stackTracker.stack().get(), List./of/(/LongType/, > /ReferenceType/)); > ??????????? stcb.trying(tryb -> { > /assertEquals/(stackTracker.stack().get(), List./of/(/LongType/, > /ReferenceType/)); > ? ??????????????tryb.iconst_1(); > /assertEquals/(stackTracker.stack().get(), List./of/(/IntType/, > /LongType/, /ReferenceType/)); > ??????????????? tryb.ifThen(thb -> { > /assertEquals/(stackTracker.stack().get(), List./of/(/LongType/, > /ReferenceType/)); > ??????????????????? thb.constantInstruction(ClassDesc./of/("Phee")); > /assertEquals/(stackTracker.stack().get(), List./of/(/ReferenceType/, > /LongType/, /ReferenceType/)); > ??????????????????? thb.athrow(); > /assertFalse/(stackTracker.stack().isPresent()); > ??????????????? }); > /assertEquals/(stackTracker.stack().get(), List./of/(/LongType/, > /ReferenceType/)); > ??????????????? tryb.return_(); > /assertFalse/(stackTracker.stack().isPresent()); > ??????????? }, catchb -> catchb.catching(ClassDesc./of/("Phee"), cb -> { > /assertEquals/(stackTracker.stack().get(), List./of/(/ReferenceType/)); > ??????????????? cb.athrow(); > /assertFalse/(stackTracker.stack().isPresent()); > ??????????? })); > ??????? }); > /assertTrue/(stackTracker.maxStackSize().isPresent()); > /assertEquals/((int)stackTracker.maxStackSize().get(), 4); > ??? })); > > Thanks, > > Adam > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Fri Aug 26 21:11:34 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 26 Aug 2022 21:11:34 +0000 Subject: Classfile API Synchronous Stack Tracking in CodeBuilder In-Reply-To: References: Message-ID: <88C8C064-DDBF-4AA3-985D-C86A40A9448E@oracle.com> Me too, I was wondering if this functionality could be separated out and composed. Paul. > On Aug 26, 2022, at 7:12 AM, Brian Goetz wrote: > > I like how this leans on the existing Transform abstraction -- and it is useful beyond just tracking -- it means existing transforms can be combined with generation if that is convenient. > > If the user wants to chain some transforms together *and* wants to retain references to each element in the chain -- which they will probably want to do if there are multiple kinds of "trackers" here, such as tracking stack + tracking locals -- then there is a little trickiness to setting the pipeline up right. The user will have to do something like: > > withMethodBody(b -> { > StackTracker st = StackTracker.of(); > LocalTracker lt = LocalTracker.of(); > transforming(st.andThen(lt), cb -> { ... build here, can use st/lt ... }); > }); > > What's happening here is that the transforms are stateful, but the user is creating the transform instances directly, rather than through a factory, so they can retain references to them. This is totally OK, as long as the user doesn't try to reuse the same StackTracker instance across multiple methods at the same time. > > The spec for StackTracker and friends will probably need some extensive examples to guide users to the right idiom. > > On 8/26/2022 9:58 AM, Adam Sotona wrote: >> Based on various optiona how to expose the StackTracker through the Classfile API I?ve decided to refactor the proposal in: >> https://github.com/openjdk/jdk-sandbox/pull/34 >> >> Instead of embedded implementation and Classfile.Option here is a proposal to expose StackTracker as CodeTransform and implement CodeBuilder::transforming method: >> /** >> * Builds code fragment generated by the handler and synchronously transformed. >> * @param transform the transform to apply to the code generated by the handler >> * @param handler handler that receives a {@linkplain BlockCodeBuilder} to >> * generate the code. >> * @return this builder >> */ >> default CodeBuilder transforming(CodeTransform transform, Consumer handler) { >> var resolved = transform.resolve(this); >> resolved.startHandler().run(); >> handler.accept(new TransformingCodeBuilder(this, resolved.consumer())); >> resolved.endHandler().run(); >> return this; >> } >> >> Modified usage of the StackTracker is reflected in the StackTrackerTest: >> >> Classfile.build(ClassDesc.of("Foo"), clb -> >> clb.withMethodBody("m", MethodTypeDesc.of(ConstantDescs.CD_Void), 0, cob -> { >> var stackTracker = new StackTracker(); >> cob.transforming(stackTracker, stcb -> { >> assertEquals(stackTracker.stack().get(), List.of()); >> stcb.aload(0); >> assertEquals(stackTracker.stack().get(), List.of(ReferenceType)); >> stcb.lconst_0(); >> assertEquals(stackTracker.stack().get(), List.of(LongType, ReferenceType)); >> stcb.trying(tryb -> { >> assertEquals(stackTracker.stack().get(), List.of(LongType, ReferenceType)); >> tryb.iconst_1(); >> assertEquals(stackTracker.stack().get(), List.of(IntType, LongType, ReferenceType)); >> tryb.ifThen(thb -> { >> assertEquals(stackTracker.stack().get(), List.of(LongType, ReferenceType)); >> thb.constantInstruction(ClassDesc.of("Phee")); >> assertEquals(stackTracker.stack().get(), List.of(ReferenceType, LongType, ReferenceType)); >> thb.athrow(); >> assertFalse(stackTracker.stack().isPresent()); >> }); >> assertEquals(stackTracker.stack().get(), List.of(LongType, ReferenceType)); >> tryb.return_(); >> assertFalse(stackTracker.stack().isPresent()); >> }, catchb -> catchb.catching(ClassDesc.of("Phee"), cb -> { >> assertEquals(stackTracker.stack().get(), List.of(ReferenceType)); >> cb.athrow(); >> assertFalse(stackTracker.stack().isPresent()); >> })); >> }); >> assertTrue(stackTracker.maxStackSize().isPresent()); >> assertEquals((int)stackTracker.maxStackSize().get(), 4); >> })); >> >> Thanks, >> Adam From paul.sandoz at oracle.com Fri Aug 26 22:33:13 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 26 Aug 2022 22:33:13 +0000 Subject: Try/catch/finally builder In-Reply-To: <09f2222e-8b9e-527a-ac73-b8e1948f959a@oracle.com> References: <6D1023A6-1B0D-4E88-88FD-60B913ACC134@oracle.com> <76FD98D4-EEF5-44E1-95E9-25441B4BEA19@oracle.com> <09f2222e-8b9e-527a-ac73-b8e1948f959a@oracle.com> Message-ID: <406E5A61-68A6-4F21-93D4-5313AC917B4C@oracle.com> ?Whew!? indeed. It gets even more complex if a finally block does silly stuff like exit with a return or break to an outer try statement (I forgot about that, so not currently supported). The testing matrix is already exploding in my head :-) Not easy to get right if the user has to explicitly inline, and not possible as you point out if code generation is layered via some other intermediaries. Yes, return instructions are implicitly supported since we know that exits the block: boolean isBlockExitingInstruction(CodeElement e) { return switch (e) { case BranchInstruction bi -> inlineFinally.test(this, bi); case ReturnInstruction ri -> true; default -> false; }; } Setting that aside, the real question is whether a branch target that is a branch target is internal to the try+catch blocks, or "outside?. Yes, that is what I am grappling with. Which we would handle by examining the label. But the usability of this is not so great, because the label may not have been bound yet, and so all the user can do is compare it for identity with the N labels known to be in the block. That sounds like kind of a pain for the user to implement. Indeed. So here's another possible thought: what if BlockBuilder could dispense labels, as well as dispensing blocks, and could then keep track of containment? So if you are branching to a label *outside this block builder* (including returning), we'd inline the finally at that point. (With nested block builders, we'd have to keep track of which BBs belonged to other BBs, but we already do this.) That?s a good idea. We can accumulate the set of labels produced by calls to newLabel, then compare branch targets against that set. Somehow we should ?disable? unbound labels when the block ends. We could just bind them all to the end, then they cannot be rebound in unwanted places. Seems ok for a block label to be bound in a nested block builder. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Fri Aug 26 22:47:17 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 26 Aug 2022 22:47:17 +0000 Subject: Try/catch/finally builder In-Reply-To: <406E5A61-68A6-4F21-93D4-5313AC917B4C@oracle.com> References: <6D1023A6-1B0D-4E88-88FD-60B913ACC134@oracle.com> <76FD98D4-EEF5-44E1-95E9-25441B4BEA19@oracle.com> <09f2222e-8b9e-527a-ac73-b8e1948f959a@oracle.com> <406E5A61-68A6-4F21-93D4-5313AC917B4C@oracle.com> Message-ID: <4F4860A8-2674-42D0-8C63-726D4460265F@oracle.com> > On Aug 26, 2022, at 3:33 PM, Paul Sandoz wrote: > > Seems ok for a block label to be bound in a nested block builder. > On second thoughts that?s also problematic, since it will not be part of the label set of the nested block, we can guard against this by overriding labelBinding. Thus we can branch out of a block, but not branch into a block. Paul. From adam.sotona at oracle.com Mon Aug 29 09:08:36 2022 From: adam.sotona at oracle.com (Adam Sotona) Date: Mon, 29 Aug 2022 09:08:36 +0000 Subject: Classfile API Synchronous Stack Tracking in CodeBuilder In-Reply-To: References: Message-ID: On 26.08.2022 16:12, "Brian Goetz" wrote: The spec for StackTracker and friends will probably need some extensive examples to guide users to the right idiom. Yes, requirement of package-info with relevant snippets for jdk.classfile.transforms package just appeared in my ToDo list. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Tue Aug 30 21:33:39 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 30 Aug 2022 21:33:39 +0000 Subject: Try/catch/finally builder In-Reply-To: <09f2222e-8b9e-527a-ac73-b8e1948f959a@oracle.com> References: <6D1023A6-1B0D-4E88-88FD-60B913ACC134@oracle.com> <76FD98D4-EEF5-44E1-95E9-25441B4BEA19@oracle.com> <09f2222e-8b9e-527a-ac73-b8e1948f959a@oracle.com> Message-ID: <3BC824FC-F922-4D9D-8B3B-EA368E0271B4@oracle.com> Here?s an initial try at that. https://github.com/openjdk/jdk-sandbox/compare/classfile-api-branch...PaulSandoz:jdk-sandbox:try-catch-finally-builder?expand=1 BlockCodeBuilder tracks labels created via newLabel, and returns a ?live? set of those labels. When a label is to be bound we reject labels that are not members of that set (we need to be careful to propagate the LabelTarget pseudoinstruction to the terminal builder). We inline a finally block if a the target label of a branch instruction is not a member of the block's labels. Ideally when block goes out of scope then all the block?s labels become ?dead? and therefore they can never be used as targets (or bound if not already). Paul. > On Aug 25, 2022, at 5:49 AM, Brian Goetz wrote: > > > So here's another possible thought: what if BlockBuilder could dispense labels, as well as dispensing blocks, and could then keep track of containment? So if you are branching to a label *outside this block builder* (including returning), we'd inline the finally at that point. (With nested block builders, we'd have to keep track of which BBs belonged to other BBs, but we already do this.) > From brian.goetz at oracle.com Tue Aug 30 21:59:23 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 30 Aug 2022 17:59:23 -0400 Subject: Try/catch/finally builder In-Reply-To: <3BC824FC-F922-4D9D-8B3B-EA368E0271B4@oracle.com> References: <6D1023A6-1B0D-4E88-88FD-60B913ACC134@oracle.com> <76FD98D4-EEF5-44E1-95E9-25441B4BEA19@oracle.com> <09f2222e-8b9e-527a-ac73-b8e1948f959a@oracle.com> <3BC824FC-F922-4D9D-8B3B-EA368E0271B4@oracle.com> Message-ID: Is there any case where we would want to nest BBs but allow the nested BB to bind a label defined by the parent BB?? I don't think so; what we're basically saying is that if you define a label in a block, it should be (a) defined before you finish building the block and (b) can only be defined in that block, not in sub-blocks.? Since LabelTarget is part of the base API, we probably have to document this constraint on valid labels targets. if we have nested trys, each with finally blocks, I think we have to walk the stack of BBs to get this right: ?? Y: ?? while (goop) { ? ?? ? try { ?????? X: ?? while (blah) { ?? try { ?? if (wah) continue X; ?????????????? // need A but not B here if (wooga) continue Y; ?????????????????? // need A and B here ?? } ?????????? finally { A } ?? } ?? finally { B } ??? } The first one uses a label defined in the outer try scope, so we only inline finally A here.? The second uses a label defined outside the outer try scope, so we have to inline A and B (in the right order) here. On 8/30/2022 5:33 PM, Paul Sandoz wrote: > Here?s an initial try at that. > > https://github.com/openjdk/jdk-sandbox/compare/classfile-api-branch...PaulSandoz:jdk-sandbox:try-catch-finally-builder?expand=1 > > BlockCodeBuilder tracks labels created via newLabel, and returns a ?live? set of those labels. When a label is to be bound we reject labels that are not members of that set (we need to be careful to propagate the LabelTarget pseudoinstruction to the terminal builder). > > We inline a finally block if a the target label of a branch instruction is not a member of the block's labels. > > Ideally when block goes out of scope then all the block?s labels become ?dead? and therefore they can never be used as targets (or bound if not already). > > Paul. > > >> On Aug 25, 2022, at 5:49 AM, Brian Goetz wrote: >> >> >> So here's another possible thought: what if BlockBuilder could dispense labels, as well as dispensing blocks, and could then keep track of containment? So if you are branching to a label *outside this block builder* (including returning), we'd inline the finally at that point. (With nested block builders, we'd have to keep track of which BBs belonged to other BBs, but we already do this.) >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Wed Aug 31 19:18:48 2022 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 31 Aug 2022 19:18:48 +0000 Subject: Try/catch/finally builder In-Reply-To: References: <6D1023A6-1B0D-4E88-88FD-60B913ACC134@oracle.com> <76FD98D4-EEF5-44E1-95E9-25441B4BEA19@oracle.com> <09f2222e-8b9e-527a-ac73-b8e1948f959a@oracle.com> <3BC824FC-F922-4D9D-8B3B-EA368E0271B4@oracle.com> Message-ID: <9B963A4E-A779-46C7-94F2-A237238C863A@oracle.com> > On Aug 30, 2022, at 2:59 PM, Brian Goetz wrote: > > Is there any case where we would want to nest BBs but allow the nested BB to bind a label defined by the parent BB? I don't think so; what we're basically saying is that if you define a label in a block, it should be (a) defined before you finish building the block and (b) can only be defined in that block, not in sub-blocks. Since LabelTarget is part of the base API, we probably have to document this constraint on valid labels targets. > Yes. > if we have nested trys, each with finally blocks, I think we have to walk the stack of BBs to get this right: > > Y: > while (goop) { > try { > X: > while (blah) { > try { > if (wah) continue X; > // need A but not B here > if (wooga) continue Y; > // need A and B here > } > finally { A } > } > finally { B } > } > > The first one uses a label defined in the outer try scope, so we only inline finally A here. The second uses a label defined outside the outer try scope, so we have to inline A and B (in the right order) here. > Indeed it's complex. I think the test membership of block labels can work, since X is a member of the outer try?s label set, where as Y is not. But, we have to be careful how we propagate the branch instruction up the chain of builders. At the moment it's a little messy (do we use parent or terminal etc?) and I need clear this up. We have three kinds of builder: - the root builder, not an instance of BlockCodeBuilder. - the terminal builder , an instance of TerminalCodeBuilder. - the parent block builder, an instance of BlockCodeBuilder. When a block builder is accepting code elements it can decide whether to propagate them directly to the root builder, skipping any intermediate parent block builders. Branch instructions can be propagated to the parent block builder if the target label is not a member of the block's label set. So in your example the branch instruction for ?continue X? gets propagated to the builder associated with the outer try, and then to the root builder, where as Y gets propagated all the way up. Paul.