From rafael.wth at gmail.com Tue Aug 13 09:51:38 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Tue, 13 Aug 2024 11:51:38 +0200 Subject: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions Message-ID: Hello! It's been a while since I last tried, but I now managed to implement both a JDK-based ClassReader and ClassWriter for ASM. This seems to work really well for a range of tests that I did as it allows me to avoid major rewrites of ASM-based code while still having forward-compatibility as long as there are no new language constructs as the JDK bundles parser and serializer. The code can be found here: https://github.com/raphw/asm-jdk-bridge A few notes and questions I have: 1. The OpenJDK API supports CharacterRangeTable which is a non-standard attribute that is not described in the specification. By introducing it to the official API, isn't the attribute in a way formalized? Is there a plan to standardize the attribute? 2. One thing that cannot be mapped directly from OpenJDK to ASM are line numbers. In ASM, they can be added later using a Label. In OpenJDK they have to be visited at "the right time". Using labels is otherwise common for other attributes, both in ASM and OpenJDK, but line numbers are the exception. Is there a reason for this? 3. It would be nice if there was a way to flag what attributes should be treated as UnknownAttributes. Right now, I retain UnknownAttributes as they are. But if a future release of the OpenJDK promotes an UnknownAttribute to a known one, I might miss it when copying them over as raw arrays. 4. I take it so that there are no plans to add support for manually defining StackMapFrame and method sizes? Sometimes, the OpenJDK erases information, for example if a local variable is never assigned a value, it will simply be treated as "N" value. This is not a big issue, but it can make slight transformations that can have very subtle implications when for example writing Java agents, so I would still hope for an option for this to be used by advanced users. Thanks! Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From chen.l.liang at oracle.com Tue Aug 13 21:29:32 2024 From: chen.l.liang at oracle.com (Chen Liang) Date: Tue, 13 Aug 2024 21:29:32 +0000 Subject: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions In-Reply-To: References: Message-ID: Hi Rafael, thanks for your adoption! 1. In fact, ClassFile API has quite a few non-standard (JVMS) attributes: CharacterRangeTable, CompilationID, ModuleHashes, ModuleResolution, ModuleTarget, SourceDebugExtension, SourceID, and seems there's little information available about these attributes. I think you can treat them as if they are unknown attributes in your translations to ASM. 2. Line numbers are naturally streamed just like labels. If an ASM user is streaming line numbers late, an alternative way is to buffer the other elements into an ASM tree first, then inject the line number tokens on the second round. (If LineNumberTableAttribute supports using Labels, then we can probably just write that attribute at last, too) 3. Unknown attributes are troubling the ClassFile API too. We have a stability() in AttributeMapper to indicate if we should retain or drop attributes. You can always copy any attribute over as-is by calling XxxBuilder::with(attribute). You should always handle the unknown attributes with a default branch in a switch if you wish to be forward compatible, or an exhaustive switch if you wish to fail fast on newer known attributes. 4. For stack map frames, you can use StackMapsOption.DROP_STACK_MAPS and pass a StackMapsTableAttribute in CodeBuilder.with; this is already how ProxyGenerator does it. However we don't allow users to specify max stacks and locals, which can be hurting performance sensitive use cases. Regards, Chen Liang ________________________________ From: classfile-api-dev on behalf of Rafael Winterhalter Sent: Tuesday, August 13, 2024 4:51 AM To: classfile-api-dev Subject: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions Hello! It's been a while since I last tried, but I now managed to implement both a JDK-based ClassReader and ClassWriter for ASM. This seems to work really well for a range of tests that I did as it allows me to avoid major rewrites of ASM-based code while still having forward-compatibility as long as there are no new language constructs as the JDK bundles parser and serializer. The code can be found here: https://github.com/raphw/asm-jdk-bridge A few notes and questions I have: 1. The OpenJDK API supports CharacterRangeTable which is a non-standard attribute that is not described in the specification. By introducing it to the official API, isn't the attribute in a way formalized? Is there a plan to standardize the attribute? 2. One thing that cannot be mapped directly from OpenJDK to ASM are line numbers. In ASM, they can be added later using a Label. In OpenJDK they have to be visited at "the right time". Using labels is otherwise common for other attributes, both in ASM and OpenJDK, but line numbers are the exception. Is there a reason for this? 3. It would be nice if there was a way to flag what attributes should be treated as UnknownAttributes. Right now, I retain UnknownAttributes as they are. But if a future release of the OpenJDK promotes an UnknownAttribute to a known one, I might miss it when copying them over as raw arrays. 4. I take it so that there are no plans to add support for manually defining StackMapFrame and method sizes? Sometimes, the OpenJDK erases information, for example if a local variable is never assigned a value, it will simply be treated as "N" value. This is not a big issue, but it can make slight transformations that can have very subtle implications when for example writing Java agents, so I would still hope for an option for this to be used by advanced users. Thanks! Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Tue Aug 13 22:08:12 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Wed, 14 Aug 2024 00:08:12 +0200 Subject: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions In-Reply-To: References: Message-ID: Thanks, I will try the manual stack map frame translation! As for line numbers: I also wanted to create the attribute manually. Unfortunately, LineNumberInfo only accepts a bci instead of a Label. If that can be adjusted (I think this is sensible as this is the only location where bci seems to be exposed), I could solve the problem that way. I also found a possible bug with the following stacktrace: java.lang.IllegalArgumentException: Expected type of (T*)V for constructor, found MethodTypeDesc[(MethodHandles$Lookup,String,Class)DynamicConstantBootstrap] at java.base/java.lang.constant.DirectMethodHandleDescImpl.validateConstructor(DirectMethodHandleDescImpl.java:107) at java.base/java.lang.constant.DirectMethodHandleDescImpl.(DirectMethodHandleDescImpl.java:75) at java.base/java.lang.constant.MethodHandleDesc.of(MethodHandleDesc.java:89) at java.base/jdk.internal.classfile.impl.AbstractPoolEntry$MethodHandleEntryImpl.asSymbol(AbstractPoolEntry.java:919) at java.base/java.lang.classfile.constantpool.ConstantDynamicEntry.asSymbol(ConstantDynamicEntry.java:65) at java.base/jdk.internal.classfile.impl.StackMapGenerator.processLdc(StackMapGenerator.java:705) This happens when using a constructor as the bootstrap for a dynamic constant. The check does not seem to consider that constructors are valid bootstraps similar to static methods. Other than that, I ran Byte Buddy's entire test suite with about 10.000 tests using that bridge now and only about 200 tests seem to fail. Most commonly this is due to "Unable to generate stack map frame for dead code at bytecode offset 36 of method foobar()", but this I might be able to solve with your suggestion. The reminder are mostly about edge cases that I will investigate further. Well done! If these two issues can be resolved, I plan to offer an adapter into Byte Buddy once this is public API. Rafael Am Di., 13. Aug. 2024 um 23:29 Uhr schrieb Chen Liang < chen.l.liang at oracle.com>: > Hi Rafael, thanks for your adoption! > > 1. In fact, ClassFile API has quite a few non-standard (JVMS) > attributes: CharacterRangeTable, CompilationID, ModuleHashes, > ModuleResolution, ModuleTarget, SourceDebugExtension, SourceID, and seems > there's little information available about these attributes. I think you > can treat them as if they are unknown attributes in your translations to > ASM. > 2. Line numbers are naturally streamed just like labels. If an ASM > user is streaming line numbers late, an alternative way is to buffer the > other elements into an ASM tree first, then inject the line number tokens > on the second round. (If LineNumberTableAttribute supports using Labels, > then we can probably just write that attribute at last, too) > 3. Unknown attributes are troubling the ClassFile API too. We have a > stability() in AttributeMapper to indicate if we should retain or drop > attributes. You can always copy any attribute over as-is by calling > XxxBuilder::with(attribute). You should always handle the unknown > attributes with a default branch in a switch if you wish to be forward > compatible, or an exhaustive switch if you wish to fail fast on newer known > attributes. > 4. For stack map frames, you can use StackMapsOption.DROP_STACK_MAPS > and pass a StackMapsTableAttribute in CodeBuilder.with; this is already how > ProxyGenerator does it. However we don't allow users to specify max stacks > and locals, which can be hurting performance sensitive use cases. > > > Regards, > Chen Liang > ------------------------------ > *From:* classfile-api-dev on behalf > of Rafael Winterhalter > *Sent:* Tuesday, August 13, 2024 4:51 AM > *To:* classfile-api-dev > *Subject:* ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and > questions > > Hello! > > It's been a while since I last tried, but I now managed to implement both > a JDK-based ClassReader and ClassWriter for ASM. This seems to work really > well for a range of tests that I did as it allows me to avoid major > rewrites of ASM-based code while still having forward-compatibility as long > as there are no new language constructs as the JDK bundles parser and > serializer. > > The code can be found here: https://github.com/raphw/asm-jdk-bridge > > A few notes and questions I have: > > 1. The OpenJDK API supports CharacterRangeTable which is a non-standard > attribute that is not described in the specification. By introducing it to > the official API, isn't the attribute in a way formalized? Is there a plan > to standardize the attribute? > > 2. One thing that cannot be mapped directly from OpenJDK to ASM are line > numbers. In ASM, they can be added later using a Label. In OpenJDK they > have to be visited at "the right time". Using labels is otherwise common > for other attributes, both in ASM and OpenJDK, but line numbers are the > exception. Is there a reason for this? > > 3. It would be nice if there was a way to flag what attributes should be > treated as UnknownAttributes. Right now, I retain UnknownAttributes as they > are. But if a future release of the OpenJDK promotes an UnknownAttribute to > a known one, I might miss it when copying them over as raw arrays. > > 4. I take it so that there are no plans to add support for manually > defining StackMapFrame and method sizes? Sometimes, the OpenJDK erases > information, for example if a local variable is never assigned a value, it > will simply be treated as "N" value. This is not a big issue, but it can > make slight transformations that can have very subtle implications when for > example writing Java agents, so I would still hope for an option for this > to be used by advanced users. > > Thanks! Rafael > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chen.l.liang at oracle.com Tue Aug 13 22:43:04 2024 From: chen.l.liang at oracle.com (Chen Liang) Date: Tue, 13 Aug 2024 22:43:04 +0000 Subject: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions In-Reply-To: References: Message-ID: Hi Rafael, thanks for the quick followup! Indeed, the line number label handling enhancement would be helpful. (and that actually applies to all user attribute encoding of Labels, as currently user attributes cannot convert labels to bcis too) For the following stacktrace, I don't think JDK's constants API is wrong: you should use (Lookup, String, Class) -> void type for the bootstrap method's method handle type. I checked bytebuddy's JavaConstant.Dynamic and JavaConstantDynamicTest and still have no clue how that happens, or how it passes elsewhere. It would be great if you can provide a simple generated class file or its javap output, especially if this class can run on java command line so we can check the VM behavior. And congratulations on the progress! The "dead code" warning shouldn't happen usually, as ClassFile API has PATCH_DEAD_CODE by default for DeadCodeOption. These errors should go away if you restore the option. Thanks again for trying out! Chen Liang ________________________________ From: Rafael Winterhalter Sent: Tuesday, August 13, 2024 5:08 PM To: Chen Liang Cc: classfile-api-dev Subject: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions Thanks, I will try the manual stack map frame translation! As for line numbers: I also wanted to create the attribute manually. Unfortunately, LineNumberInfo only accepts a bci instead of a Label. If that can be adjusted (I think this is sensible as this is the only location where bci seems to be exposed), I could solve the problem that way. I also found a possible bug with the following stacktrace: java.lang.IllegalArgumentException: Expected type of (T*)V for constructor, found MethodTypeDesc[(MethodHandles$Lookup,String,Class)DynamicConstantBootstrap] at java.base/java.lang.constant.DirectMethodHandleDescImpl.validateConstructor(DirectMethodHandleDescImpl.java:107) at java.base/java.lang.constant.DirectMethodHandleDescImpl.(DirectMethodHandleDescImpl.java:75) at java.base/java.lang.constant.MethodHandleDesc.of(MethodHandleDesc.java:89) at java.base/jdk.internal.classfile.impl.AbstractPoolEntry$MethodHandleEntryImpl.asSymbol(AbstractPoolEntry.java:919) at java.base/java.lang.classfile.constantpool.ConstantDynamicEntry.asSymbol(ConstantDynamicEntry.java:65) at java.base/jdk.internal.classfile.impl.StackMapGenerator.processLdc(StackMapGenerator.java:705) This happens when using a constructor as the bootstrap for a dynamic constant. The check does not seem to consider that constructors are valid bootstraps similar to static methods. Other than that, I ran Byte Buddy's entire test suite with about 10.000 tests using that bridge now and only about 200 tests seem to fail. Most commonly this is due to "Unable to generate stack map frame for dead code at bytecode offset 36 of method foobar()", but this I might be able to solve with your suggestion. The reminder are mostly about edge cases that I will investigate further. Well done! If these two issues can be resolved, I plan to offer an adapter into Byte Buddy once this is public API. Rafael Am Di., 13. Aug. 2024 um 23:29 Uhr schrieb Chen Liang >: Hi Rafael, thanks for your adoption! 1. In fact, ClassFile API has quite a few non-standard (JVMS) attributes: CharacterRangeTable, CompilationID, ModuleHashes, ModuleResolution, ModuleTarget, SourceDebugExtension, SourceID, and seems there's little information available about these attributes. I think you can treat them as if they are unknown attributes in your translations to ASM. 2. Line numbers are naturally streamed just like labels. If an ASM user is streaming line numbers late, an alternative way is to buffer the other elements into an ASM tree first, then inject the line number tokens on the second round. (If LineNumberTableAttribute supports using Labels, then we can probably just write that attribute at last, too) 3. Unknown attributes are troubling the ClassFile API too. We have a stability() in AttributeMapper to indicate if we should retain or drop attributes. You can always copy any attribute over as-is by calling XxxBuilder::with(attribute). You should always handle the unknown attributes with a default branch in a switch if you wish to be forward compatible, or an exhaustive switch if you wish to fail fast on newer known attributes. 4. For stack map frames, you can use StackMapsOption.DROP_STACK_MAPS and pass a StackMapsTableAttribute in CodeBuilder.with; this is already how ProxyGenerator does it. However we don't allow users to specify max stacks and locals, which can be hurting performance sensitive use cases. Regards, Chen Liang ________________________________ From: classfile-api-dev > on behalf of Rafael Winterhalter > Sent: Tuesday, August 13, 2024 4:51 AM To: classfile-api-dev > Subject: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions Hello! It's been a while since I last tried, but I now managed to implement both a JDK-based ClassReader and ClassWriter for ASM. This seems to work really well for a range of tests that I did as it allows me to avoid major rewrites of ASM-based code while still having forward-compatibility as long as there are no new language constructs as the JDK bundles parser and serializer. The code can be found here: https://github.com/raphw/asm-jdk-bridge A few notes and questions I have: 1. The OpenJDK API supports CharacterRangeTable which is a non-standard attribute that is not described in the specification. By introducing it to the official API, isn't the attribute in a way formalized? Is there a plan to standardize the attribute? 2. One thing that cannot be mapped directly from OpenJDK to ASM are line numbers. In ASM, they can be added later using a Label. In OpenJDK they have to be visited at "the right time". Using labels is otherwise common for other attributes, both in ASM and OpenJDK, but line numbers are the exception. Is there a reason for this? 3. It would be nice if there was a way to flag what attributes should be treated as UnknownAttributes. Right now, I retain UnknownAttributes as they are. But if a future release of the OpenJDK promotes an UnknownAttribute to a known one, I might miss it when copying them over as raw arrays. 4. I take it so that there are no plans to add support for manually defining StackMapFrame and method sizes? Sometimes, the OpenJDK erases information, for example if a local variable is never assigned a value, it will simply be treated as "N" value. This is not a big issue, but it can make slight transformations that can have very subtle implications when for example writing Java agents, so I would still hope for an option for this to be used by advanced users. Thanks! Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Wed Aug 14 08:12:51 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Wed, 14 Aug 2024 10:12:51 +0200 Subject: Re-application of code builder and explicit reset Message-ID: Hello, I find that the re-appliction of a code builder in case of overflowing labels is one less intuitive thing in the code builder API. In any case, I would add this very explicitly to the javadoc. Finally, I wanted to suggest to add an overload: MethodBuilder withCode(Consumer code, Runnable onReset); In case that the code builder is stateful, this allows to reset some cached properties that become invalid after the first run. I understand that statefulness of the consumer might not be intended, but since ASM is stateful today, I think I will not be the only one in need. Currently, I prepend the reset to the builder, but it requires an extra check for the first run making it rather clunky. Personally: ASM patches the byte code without reapplication of the visitor. Was this considered for the Class File builder API? Best regards, Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Wed Aug 14 10:36:23 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Wed, 14 Aug 2024 12:36:23 +0200 Subject: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions In-Reply-To: References: Message-ID: Hello, after some further fixes, I am down to 30 failed tests of 10436 tests in total now, only using the ClassFile API for reading and writing classes with ASM as an intermediate. I have not got the stack map frame generation to work yet, but will tackle this next. The remaining 1. I will try to create a reproducer, but in short: DirectMethodHandleDesc.Kind kind = DirectMethodHandleDesc.Kind.CONSTRUCTOR; ClassDesc owner = ClassDesc.of("sample.DynamicConstantBootstrap"); String name = ""; String lookupDescriptor = "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/Class;)V"; DirectMethodHandleDesc methodHandle = MethodHandleDesc.of(kind, owner, name, lookupDescriptor); System.out.println(methodHandle.invocationType().returnType()); The printed return type will be the owner type. If I change the kind to STATIC, it will be void as expected. When adding this to a LCD instruction, it will be converted to a constant pool entry where the return type is validated against it being a constructor returning void. The conversion happens in BytecodeHelpers.handleConstantDescToHandleInfo where the patched return type is used for creating the symbol what fails validation. 2. The line number thing causes another row of issues. If, as suggested previously, the line number attribute could accept a label rather than a BCI, this could be fixed easily. 3. Some tests fail due to the impossibility of handling JSR and RET instructions. I fully understand that those are old, but especially JDBC drivers are often compiled to rather old Java versions. Java agents often instrument those, and it would be a pity if those would not be supported therefore. Is there a plan to add support for these instructions? Fortunately, stack map frame generation does not need to consider them as these old class files do not require stack map frames. Adding them is not the worst for this reason. With these three things in place, I think I can offer full support for the new API starting the day this becomes non-experimental. Best regards, Rafael Am Mi., 14. Aug. 2024 um 00:43 Uhr schrieb Chen Liang < chen.l.liang at oracle.com>: > Hi Rafael, thanks for the quick followup! > Indeed, the line number label handling enhancement would be helpful. (and > that actually applies to all user attribute encoding of Labels, as > currently user attributes cannot convert labels to bcis too) > > For the following stacktrace, I don't think JDK's constants API is wrong: > you should use (Lookup, String, Class) -> void type for the bootstrap > method's method handle type. I checked bytebuddy's JavaConstant.Dynamic and > JavaConstantDynamicTest and still have no clue how that happens, or how it > passes elsewhere. It would be great if you can provide a simple generated > class file or its javap output, especially if this class can run on java > command line so we can check the VM behavior. > > And congratulations on the progress! The "dead code" warning shouldn't > happen usually, as ClassFile API has PATCH_DEAD_CODE by default for > DeadCodeOption. These errors should go away if you restore the option. > > Thanks again for trying out! > Chen Liang > ------------------------------ > *From:* Rafael Winterhalter > *Sent:* Tuesday, August 13, 2024 5:08 PM > *To:* Chen Liang > *Cc:* classfile-api-dev > *Subject:* [External] : Re: ASM to OpenJDK ClassReader/ClassWriter > bridge: experiences and questions > > Thanks, I will try the manual stack map frame translation! As for line > numbers: I also wanted to create the attribute manually. Unfortunately, > LineNumberInfo only accepts a bci instead of a Label. If that can be > adjusted (I think this is sensible as this is the only location where bci > seems to be exposed), I could solve the problem that way. > > I also found a possible bug with the following stacktrace: > > java.lang.IllegalArgumentException: Expected type of (T*)V for > constructor, found > MethodTypeDesc[(MethodHandles$Lookup,String,Class)DynamicConstantBootstrap] > at > java.base/java.lang.constant.DirectMethodHandleDescImpl.validateConstructor(DirectMethodHandleDescImpl.java:107) > at > java.base/java.lang.constant.DirectMethodHandleDescImpl.(DirectMethodHandleDescImpl.java:75) > at > java.base/java.lang.constant.MethodHandleDesc.of(MethodHandleDesc.java:89) > at > java.base/jdk.internal.classfile.impl.AbstractPoolEntry$MethodHandleEntryImpl.asSymbol(AbstractPoolEntry.java:919) > at > java.base/java.lang.classfile.constantpool.ConstantDynamicEntry.asSymbol(ConstantDynamicEntry.java:65) > at > java.base/jdk.internal.classfile.impl.StackMapGenerator.processLdc(StackMapGenerator.java:705) > > This happens when using a constructor as the bootstrap for a dynamic > constant. The check does not seem to consider that constructors are valid > bootstraps similar to static methods. > > Other than that, I ran Byte Buddy's entire test suite with about 10.000 > tests using that bridge now and only about 200 tests seem to fail. Most > commonly this is due to "Unable to generate stack map frame for dead code > at bytecode offset 36 of method foobar()", but this I might be able to > solve with your suggestion. The reminder are mostly about edge cases that I > will investigate further. > > Well done! If these two issues can be resolved, I plan to offer an adapter > into Byte Buddy once this is public API. > Rafael > > Am Di., 13. Aug. 2024 um 23:29 Uhr schrieb Chen Liang < > chen.l.liang at oracle.com>: > > Hi Rafael, thanks for your adoption! > > 1. In fact, ClassFile API has quite a few non-standard (JVMS) > attributes: CharacterRangeTable, CompilationID, ModuleHashes, > ModuleResolution, ModuleTarget, SourceDebugExtension, SourceID, and seems > there's little information available about these attributes. I think you > can treat them as if they are unknown attributes in your translations to > ASM. > 2. Line numbers are naturally streamed just like labels. If an ASM > user is streaming line numbers late, an alternative way is to buffer the > other elements into an ASM tree first, then inject the line number tokens > on the second round. (If LineNumberTableAttribute supports using Labels, > then we can probably just write that attribute at last, too) > 3. Unknown attributes are troubling the ClassFile API too. We have a > stability() in AttributeMapper to indicate if we should retain or drop > attributes. You can always copy any attribute over as-is by calling > XxxBuilder::with(attribute). You should always handle the unknown > attributes with a default branch in a switch if you wish to be forward > compatible, or an exhaustive switch if you wish to fail fast on newer known > attributes. > 4. For stack map frames, you can use StackMapsOption.DROP_STACK_MAPS > and pass a StackMapsTableAttribute in CodeBuilder.with; this is already how > ProxyGenerator does it. However we don't allow users to specify max stacks > and locals, which can be hurting performance sensitive use cases. > > > Regards, > Chen Liang > ------------------------------ > *From:* classfile-api-dev on behalf > of Rafael Winterhalter > *Sent:* Tuesday, August 13, 2024 4:51 AM > *To:* classfile-api-dev > *Subject:* ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and > questions > > Hello! > > It's been a while since I last tried, but I now managed to implement both > a JDK-based ClassReader and ClassWriter for ASM. This seems to work really > well for a range of tests that I did as it allows me to avoid major > rewrites of ASM-based code while still having forward-compatibility as long > as there are no new language constructs as the JDK bundles parser and > serializer. > > The code can be found here: https://github.com/raphw/asm-jdk-bridge > > > A few notes and questions I have: > > 1. The OpenJDK API supports CharacterRangeTable which is a non-standard > attribute that is not described in the specification. By introducing it to > the official API, isn't the attribute in a way formalized? Is there a plan > to standardize the attribute? > > 2. One thing that cannot be mapped directly from OpenJDK to ASM are line > numbers. In ASM, they can be added later using a Label. In OpenJDK they > have to be visited at "the right time". Using labels is otherwise common > for other attributes, both in ASM and OpenJDK, but line numbers are the > exception. Is there a reason for this? > > 3. It would be nice if there was a way to flag what attributes should be > treated as UnknownAttributes. Right now, I retain UnknownAttributes as they > are. But if a future release of the OpenJDK promotes an UnknownAttribute to > a known one, I might miss it when copying them over as raw arrays. > > 4. I take it so that there are no plans to add support for manually > defining StackMapFrame and method sizes? Sometimes, the OpenJDK erases > information, for example if a local variable is never assigned a value, it > will simply be treated as "N" value. This is not a big issue, but it can > make slight transformations that can have very subtle implications when for > example writing Java agents, so I would still hope for an option for this > to be used by advanced users. > > Thanks! Rafael > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Wed Aug 14 11:05:55 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Wed, 14 Aug 2024 13:05:55 +0200 Subject: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions In-Reply-To: References: Message-ID: Quick finding: my tests that check for incompatible change in annotation values compared to their class file. There was a change in Java 21 (if I recall correctly) to display the actual value. This seems to have regressed. Since Byte Buddy emulates the JDKs behavior, these tests are failing now compared to the mainline. For example, if an annotation changes its property from int to array int[], previously compiled annotations are toString-ed on mainline JDK as: incompatibleValueArray=/* Warning type mismatch! "java.lang.Integer[42]" */ The ClassFile API branch toString-s the annotation as it was done prior to Java 21: incompatibleValueArray=/* Warning type mismatch! "Array with component tag: I" */ I suggest to use the mainline expression as this makes debugging things much easier. Thanks, Rafael Am Mi., 14. Aug. 2024 um 12:36 Uhr schrieb Rafael Winterhalter < rafael.wth at gmail.com>: > Hello, > > after some further fixes, I am down to 30 failed tests of 10436 tests in > total now, only using the ClassFile API for reading and writing classes > with ASM as an intermediate. I have not got the stack map frame generation > to work yet, but will tackle this next. The remaining > > 1. I will try to create a reproducer, but in short: > > DirectMethodHandleDesc.Kind kind = > DirectMethodHandleDesc.Kind.CONSTRUCTOR; > ClassDesc owner = ClassDesc.of("sample.DynamicConstantBootstrap"); > String name = ""; > String lookupDescriptor = > "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/Class;)V"; > DirectMethodHandleDesc methodHandle = MethodHandleDesc.of(kind, > owner, name, lookupDescriptor); > System.out.println(methodHandle.invocationType().returnType()); > > The printed return type will be the owner type. If I change the kind to > STATIC, it will be void as expected. When adding this to a LCD instruction, > it will be converted to a constant pool entry where the return type is > validated against it being a constructor returning void. The conversion > happens in BytecodeHelpers.handleConstantDescToHandleInfo where the patched > return type is used for creating the symbol what fails validation. > > 2. The line number thing causes another row of issues. If, as suggested > previously, the line number attribute could accept a label rather than a > BCI, this could be fixed easily. > > 3. Some tests fail due to the impossibility of handling JSR and RET > instructions. I fully understand that those are old, but especially JDBC > drivers are often compiled to rather old Java versions. Java agents often > instrument those, and it would be a pity if those would not be supported > therefore. Is there a plan to add support for these instructions? > Fortunately, stack map frame generation does not need to consider them as > these old class files do not require stack map frames. Adding them is not > the worst for this reason. > > With these three things in place, I think I can offer full support for the > new API starting the day this becomes non-experimental. > > Best regards, Rafael > > Am Mi., 14. Aug. 2024 um 00:43 Uhr schrieb Chen Liang < > chen.l.liang at oracle.com>: > >> Hi Rafael, thanks for the quick followup! >> Indeed, the line number label handling enhancement would be helpful. (and >> that actually applies to all user attribute encoding of Labels, as >> currently user attributes cannot convert labels to bcis too) >> >> For the following stacktrace, I don't think JDK's constants API is wrong: >> you should use (Lookup, String, Class) -> void type for the bootstrap >> method's method handle type. I checked bytebuddy's JavaConstant.Dynamic and >> JavaConstantDynamicTest and still have no clue how that happens, or how it >> passes elsewhere. It would be great if you can provide a simple generated >> class file or its javap output, especially if this class can run on java >> command line so we can check the VM behavior. >> >> And congratulations on the progress! The "dead code" warning shouldn't >> happen usually, as ClassFile API has PATCH_DEAD_CODE by default for >> DeadCodeOption. These errors should go away if you restore the option. >> >> Thanks again for trying out! >> Chen Liang >> ------------------------------ >> *From:* Rafael Winterhalter >> *Sent:* Tuesday, August 13, 2024 5:08 PM >> *To:* Chen Liang >> *Cc:* classfile-api-dev >> *Subject:* [External] : Re: ASM to OpenJDK ClassReader/ClassWriter >> bridge: experiences and questions >> >> Thanks, I will try the manual stack map frame translation! As for line >> numbers: I also wanted to create the attribute manually. Unfortunately, >> LineNumberInfo only accepts a bci instead of a Label. If that can be >> adjusted (I think this is sensible as this is the only location where bci >> seems to be exposed), I could solve the problem that way. >> >> I also found a possible bug with the following stacktrace: >> >> java.lang.IllegalArgumentException: Expected type of (T*)V for >> constructor, found >> MethodTypeDesc[(MethodHandles$Lookup,String,Class)DynamicConstantBootstrap] >> at >> java.base/java.lang.constant.DirectMethodHandleDescImpl.validateConstructor(DirectMethodHandleDescImpl.java:107) >> at >> java.base/java.lang.constant.DirectMethodHandleDescImpl.(DirectMethodHandleDescImpl.java:75) >> at >> java.base/java.lang.constant.MethodHandleDesc.of(MethodHandleDesc.java:89) >> at >> java.base/jdk.internal.classfile.impl.AbstractPoolEntry$MethodHandleEntryImpl.asSymbol(AbstractPoolEntry.java:919) >> at >> java.base/java.lang.classfile.constantpool.ConstantDynamicEntry.asSymbol(ConstantDynamicEntry.java:65) >> at >> java.base/jdk.internal.classfile.impl.StackMapGenerator.processLdc(StackMapGenerator.java:705) >> >> This happens when using a constructor as the bootstrap for a dynamic >> constant. The check does not seem to consider that constructors are valid >> bootstraps similar to static methods. >> >> Other than that, I ran Byte Buddy's entire test suite with about 10.000 >> tests using that bridge now and only about 200 tests seem to fail. Most >> commonly this is due to "Unable to generate stack map frame for dead code >> at bytecode offset 36 of method foobar()", but this I might be able to >> solve with your suggestion. The reminder are mostly about edge cases that I >> will investigate further. >> >> Well done! If these two issues can be resolved, I plan to offer an >> adapter into Byte Buddy once this is public API. >> Rafael >> >> Am Di., 13. Aug. 2024 um 23:29 Uhr schrieb Chen Liang < >> chen.l.liang at oracle.com>: >> >> Hi Rafael, thanks for your adoption! >> >> 1. In fact, ClassFile API has quite a few non-standard (JVMS) >> attributes: CharacterRangeTable, CompilationID, ModuleHashes, >> ModuleResolution, ModuleTarget, SourceDebugExtension, SourceID, and seems >> there's little information available about these attributes. I think you >> can treat them as if they are unknown attributes in your translations to >> ASM. >> 2. Line numbers are naturally streamed just like labels. If an ASM >> user is streaming line numbers late, an alternative way is to buffer the >> other elements into an ASM tree first, then inject the line number tokens >> on the second round. (If LineNumberTableAttribute supports using Labels, >> then we can probably just write that attribute at last, too) >> 3. Unknown attributes are troubling the ClassFile API too. We have a >> stability() in AttributeMapper to indicate if we should retain or drop >> attributes. You can always copy any attribute over as-is by calling >> XxxBuilder::with(attribute). You should always handle the unknown >> attributes with a default branch in a switch if you wish to be forward >> compatible, or an exhaustive switch if you wish to fail fast on newer known >> attributes. >> 4. For stack map frames, you can use StackMapsOption.DROP_STACK_MAPS >> and pass a StackMapsTableAttribute in CodeBuilder.with; this is already how >> ProxyGenerator does it. However we don't allow users to specify max stacks >> and locals, which can be hurting performance sensitive use cases. >> >> >> Regards, >> Chen Liang >> ------------------------------ >> *From:* classfile-api-dev on behalf >> of Rafael Winterhalter >> *Sent:* Tuesday, August 13, 2024 4:51 AM >> *To:* classfile-api-dev >> *Subject:* ASM to OpenJDK ClassReader/ClassWriter bridge: experiences >> and questions >> >> Hello! >> >> It's been a while since I last tried, but I now managed to implement both >> a JDK-based ClassReader and ClassWriter for ASM. This seems to work really >> well for a range of tests that I did as it allows me to avoid major >> rewrites of ASM-based code while still having forward-compatibility as long >> as there are no new language constructs as the JDK bundles parser and >> serializer. >> >> The code can be found here: https://github.com/raphw/asm-jdk-bridge >> >> >> A few notes and questions I have: >> >> 1. The OpenJDK API supports CharacterRangeTable which is a non-standard >> attribute that is not described in the specification. By introducing it to >> the official API, isn't the attribute in a way formalized? Is there a plan >> to standardize the attribute? >> >> 2. One thing that cannot be mapped directly from OpenJDK to ASM are line >> numbers. In ASM, they can be added later using a Label. In OpenJDK they >> have to be visited at "the right time". Using labels is otherwise common >> for other attributes, both in ASM and OpenJDK, but line numbers are the >> exception. Is there a reason for this? >> >> 3. It would be nice if there was a way to flag what attributes should be >> treated as UnknownAttributes. Right now, I retain UnknownAttributes as they >> are. But if a future release of the OpenJDK promotes an UnknownAttribute to >> a known one, I might miss it when copying them over as raw arrays. >> >> 4. I take it so that there are no plans to add support for manually >> defining StackMapFrame and method sizes? Sometimes, the OpenJDK erases >> information, for example if a local variable is never assigned a value, it >> will simply be treated as "N" value. This is not a big issue, but it can >> make slight transformations that can have very subtle implications when for >> example writing Java agents, so I would still hope for an option for this >> to be used by advanced users. >> >> Thanks! Rafael >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From chen.l.liang at oracle.com Wed Aug 14 15:29:29 2024 From: chen.l.liang at oracle.com (Chen Liang) Date: Wed, 14 Aug 2024 15:29:29 +0000 Subject: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions In-Reply-To: References: Message-ID: Rafael, first for your annotation output, how did you produce those outputs? I don't think ClassFile API performs any check to ensure an annotation's structure matches that of its annotation interface. 1. Thanks for the reproducer. It is indeed a bug with BytecodeHelpers, which should use lookupDescriptor instead of invocationType. Filed https://bugs.openjdk.org/browse/JDK-8338406 which I will fix. 2. I am thinking of exposing the label to bci conversion to attribute writers for Code: that way, we can create a custom LineNumberTable attribute that only performs writing, and in writing it converts its labels to bci, which should fit our need here. What do you think? 3. These instructions are supported: You need to call `codeBuilder.with(DiscontinuedInstruction.JsrInstruction.of(...))` to create them instead of through convenience factories. And our stack maps generation automatically fails and falls back to stack counter in default configuration if it encounters a Java 6 class file with jsr/ret. For the annotation thing, if you can provide the source code, I can help you diagnose. I am currently working on both core reflection and classfile API areas. Chen ________________________________ From: Rafael Winterhalter Sent: Wednesday, August 14, 2024 6:05 AM To: Chen Liang Cc: classfile-api-dev Subject: Re: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions Quick finding: my tests that check for incompatible change in annotation values compared to their class file. There was a change in Java 21 (if I recall correctly) to display the actual value. This seems to have regressed. Since Byte Buddy emulates the JDKs behavior, these tests are failing now compared to the mainline. For example, if an annotation changes its property from int to array int[], previously compiled annotations are toString-ed on mainline JDK as: incompatibleValueArray=/* Warning type mismatch! "java.lang.Integer[42]" */ The ClassFile API branch toString-s the annotation as it was done prior to Java 21: incompatibleValueArray=/* Warning type mismatch! "Array with component tag: I" */ I suggest to use the mainline expression as this makes debugging things much easier. Thanks, Rafael Am Mi., 14. Aug. 2024 um 12:36 Uhr schrieb Rafael Winterhalter >: Hello, after some further fixes, I am down to 30 failed tests of 10436 tests in total now, only using the ClassFile API for reading and writing classes with ASM as an intermediate. I have not got the stack map frame generation to work yet, but will tackle this next. The remaining 1. I will try to create a reproducer, but in short: DirectMethodHandleDesc.Kind kind = DirectMethodHandleDesc.Kind.CONSTRUCTOR; ClassDesc owner = ClassDesc.of("sample.DynamicConstantBootstrap"); String name = ""; String lookupDescriptor = "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/Class;)V"; DirectMethodHandleDesc methodHandle = MethodHandleDesc.of(kind, owner, name, lookupDescriptor); System.out.println(methodHandle.invocationType().returnType()); The printed return type will be the owner type. If I change the kind to STATIC, it will be void as expected. When adding this to a LCD instruction, it will be converted to a constant pool entry where the return type is validated against it being a constructor returning void. The conversion happens in BytecodeHelpers.handleConstantDescToHandleInfo where the patched return type is used for creating the symbol what fails validation. 2. The line number thing causes another row of issues. If, as suggested previously, the line number attribute could accept a label rather than a BCI, this could be fixed easily. 3. Some tests fail due to the impossibility of handling JSR and RET instructions. I fully understand that those are old, but especially JDBC drivers are often compiled to rather old Java versions. Java agents often instrument those, and it would be a pity if those would not be supported therefore. Is there a plan to add support for these instructions? Fortunately, stack map frame generation does not need to consider them as these old class files do not require stack map frames. Adding them is not the worst for this reason. With these three things in place, I think I can offer full support for the new API starting the day this becomes non-experimental. Best regards, Rafael Am Mi., 14. Aug. 2024 um 00:43 Uhr schrieb Chen Liang >: Hi Rafael, thanks for the quick followup! Indeed, the line number label handling enhancement would be helpful. (and that actually applies to all user attribute encoding of Labels, as currently user attributes cannot convert labels to bcis too) For the following stacktrace, I don't think JDK's constants API is wrong: you should use (Lookup, String, Class) -> void type for the bootstrap method's method handle type. I checked bytebuddy's JavaConstant.Dynamic and JavaConstantDynamicTest and still have no clue how that happens, or how it passes elsewhere. It would be great if you can provide a simple generated class file or its javap output, especially if this class can run on java command line so we can check the VM behavior. And congratulations on the progress! The "dead code" warning shouldn't happen usually, as ClassFile API has PATCH_DEAD_CODE by default for DeadCodeOption. These errors should go away if you restore the option. Thanks again for trying out! Chen Liang ________________________________ From: Rafael Winterhalter > Sent: Tuesday, August 13, 2024 5:08 PM To: Chen Liang > Cc: classfile-api-dev > Subject: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions Thanks, I will try the manual stack map frame translation! As for line numbers: I also wanted to create the attribute manually. Unfortunately, LineNumberInfo only accepts a bci instead of a Label. If that can be adjusted (I think this is sensible as this is the only location where bci seems to be exposed), I could solve the problem that way. I also found a possible bug with the following stacktrace: java.lang.IllegalArgumentException: Expected type of (T*)V for constructor, found MethodTypeDesc[(MethodHandles$Lookup,String,Class)DynamicConstantBootstrap] at java.base/java.lang.constant.DirectMethodHandleDescImpl.validateConstructor(DirectMethodHandleDescImpl.java:107) at java.base/java.lang.constant.DirectMethodHandleDescImpl.(DirectMethodHandleDescImpl.java:75) at java.base/java.lang.constant.MethodHandleDesc.of(MethodHandleDesc.java:89) at java.base/jdk.internal.classfile.impl.AbstractPoolEntry$MethodHandleEntryImpl.asSymbol(AbstractPoolEntry.java:919) at java.base/java.lang.classfile.constantpool.ConstantDynamicEntry.asSymbol(ConstantDynamicEntry.java:65) at java.base/jdk.internal.classfile.impl.StackMapGenerator.processLdc(StackMapGenerator.java:705) This happens when using a constructor as the bootstrap for a dynamic constant. The check does not seem to consider that constructors are valid bootstraps similar to static methods. Other than that, I ran Byte Buddy's entire test suite with about 10.000 tests using that bridge now and only about 200 tests seem to fail. Most commonly this is due to "Unable to generate stack map frame for dead code at bytecode offset 36 of method foobar()", but this I might be able to solve with your suggestion. The reminder are mostly about edge cases that I will investigate further. Well done! If these two issues can be resolved, I plan to offer an adapter into Byte Buddy once this is public API. Rafael Am Di., 13. Aug. 2024 um 23:29 Uhr schrieb Chen Liang >: Hi Rafael, thanks for your adoption! 1. In fact, ClassFile API has quite a few non-standard (JVMS) attributes: CharacterRangeTable, CompilationID, ModuleHashes, ModuleResolution, ModuleTarget, SourceDebugExtension, SourceID, and seems there's little information available about these attributes. I think you can treat them as if they are unknown attributes in your translations to ASM. 2. Line numbers are naturally streamed just like labels. If an ASM user is streaming line numbers late, an alternative way is to buffer the other elements into an ASM tree first, then inject the line number tokens on the second round. (If LineNumberTableAttribute supports using Labels, then we can probably just write that attribute at last, too) 3. Unknown attributes are troubling the ClassFile API too. We have a stability() in AttributeMapper to indicate if we should retain or drop attributes. You can always copy any attribute over as-is by calling XxxBuilder::with(attribute). You should always handle the unknown attributes with a default branch in a switch if you wish to be forward compatible, or an exhaustive switch if you wish to fail fast on newer known attributes. 4. For stack map frames, you can use StackMapsOption.DROP_STACK_MAPS and pass a StackMapsTableAttribute in CodeBuilder.with; this is already how ProxyGenerator does it. However we don't allow users to specify max stacks and locals, which can be hurting performance sensitive use cases. Regards, Chen Liang ________________________________ From: classfile-api-dev > on behalf of Rafael Winterhalter > Sent: Tuesday, August 13, 2024 4:51 AM To: classfile-api-dev > Subject: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions Hello! It's been a while since I last tried, but I now managed to implement both a JDK-based ClassReader and ClassWriter for ASM. This seems to work really well for a range of tests that I did as it allows me to avoid major rewrites of ASM-based code while still having forward-compatibility as long as there are no new language constructs as the JDK bundles parser and serializer. The code can be found here: https://github.com/raphw/asm-jdk-bridge A few notes and questions I have: 1. The OpenJDK API supports CharacterRangeTable which is a non-standard attribute that is not described in the specification. By introducing it to the official API, isn't the attribute in a way formalized? Is there a plan to standardize the attribute? 2. One thing that cannot be mapped directly from OpenJDK to ASM are line numbers. In ASM, they can be added later using a Label. In OpenJDK they have to be visited at "the right time". Using labels is otherwise common for other attributes, both in ASM and OpenJDK, but line numbers are the exception. Is there a reason for this? 3. It would be nice if there was a way to flag what attributes should be treated as UnknownAttributes. Right now, I retain UnknownAttributes as they are. But if a future release of the OpenJDK promotes an UnknownAttribute to a known one, I might miss it when copying them over as raw arrays. 4. I take it so that there are no plans to add support for manually defining StackMapFrame and method sizes? Sometimes, the OpenJDK erases information, for example if a local variable is never assigned a value, it will simply be treated as "N" value. This is not a big issue, but it can make slight transformations that can have very subtle implications when for example writing Java agents, so I would still hope for an option for this to be used by advanced users. Thanks! Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Wed Aug 14 16:34:51 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Wed, 14 Aug 2024 18:34:51 +0200 Subject: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions In-Reply-To: References: Message-ID: Brilliant, I am now down to: (1) Waiting for a fix for the bug relating to BytecodeHelpers. (2) issues with inability to define line numbers by labels, I think your idea to add a translation method from labels to BCI (and having an equivalent way of reading a BCI in the reader) is the right thing to do, this will generally be useful for custom attributes. (3) thanks, this resolves some more failing tests. I now also got the manual computation of stack map frames to work. I would however love if I could control the exact frames that are issued. Ideally agents change as little as possible to stay compatible with other byte code manipulation tools. This could be offered as a "write only" API similar to what you suggest for line numbers. What do you think? As for the annotations, I am still trying to find out what has changed. If I understand it better, I can file a separate bug report, in case that it is a regression. Thanks, Rafael Am Mi., 14. Aug. 2024 um 17:29 Uhr schrieb Chen Liang < chen.l.liang at oracle.com>: > Rafael, first for your annotation output, how did you produce those > outputs? I don't think ClassFile API performs any check to ensure an > annotation's structure matches that of its annotation interface. > > > 1. Thanks for the reproducer. It is indeed a bug with BytecodeHelpers, > which should use lookupDescriptor instead of invocationType. Filed > https://bugs.openjdk.org/browse/JDK-8338406 which I will fix. > 2. I am thinking of exposing the label to bci conversion to attribute > writers for Code: that way, we can create a custom LineNumberTable > attribute that only performs writing, and in writing it converts its labels > to bci, which should fit our need here. What do you think? > 3. These instructions are supported: You need to call > `codeBuilder.with(DiscontinuedInstruction.JsrInstruction.of(...))` to > create them instead of through convenience factories. And our stack maps > generation automatically fails and falls back to stack counter in default > configuration if it encounters a Java 6 class file with jsr/ret. > > For the annotation thing, if you can provide the source code, I can help > you diagnose. I am currently working on both core reflection and classfile > API areas. > > Chen > ------------------------------ > *From:* Rafael Winterhalter > *Sent:* Wednesday, August 14, 2024 6:05 AM > *To:* Chen Liang > *Cc:* classfile-api-dev > *Subject:* Re: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter > bridge: experiences and questions > > Quick finding: my tests that check for incompatible change in annotation > values compared to their class file. There was a change in Java 21 (if I > recall correctly) to display the actual value. This seems to have > regressed. Since Byte Buddy emulates the JDKs behavior, these tests are > failing now compared to the mainline. For example, if an annotation changes > its property from int to array int[], previously compiled annotations are > toString-ed on mainline JDK as: > > incompatibleValueArray=/* Warning type mismatch! "java.lang.Integer[42]" */ > > The ClassFile API branch toString-s the annotation as it was done prior to > Java 21: > > incompatibleValueArray=/* Warning type mismatch! "Array with component > tag: I" */ > > I suggest to use the mainline expression as this makes debugging things > much easier. > Thanks, Rafael > > Am Mi., 14. Aug. 2024 um 12:36 Uhr schrieb Rafael Winterhalter < > rafael.wth at gmail.com>: > > Hello, > > after some further fixes, I am down to 30 failed tests of 10436 tests in > total now, only using the ClassFile API for reading and writing classes > with ASM as an intermediate. I have not got the stack map frame generation > to work yet, but will tackle this next. The remaining > > 1. I will try to create a reproducer, but in short: > > DirectMethodHandleDesc.Kind kind = > DirectMethodHandleDesc.Kind.CONSTRUCTOR; > ClassDesc owner = ClassDesc.of("sample.DynamicConstantBootstrap"); > String name = ""; > String lookupDescriptor = > "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/Class;)V"; > DirectMethodHandleDesc methodHandle = MethodHandleDesc.of(kind, > owner, name, lookupDescriptor); > System.out.println(methodHandle.invocationType().returnType()); > > The printed return type will be the owner type. If I change the kind to > STATIC, it will be void as expected. When adding this to a LCD instruction, > it will be converted to a constant pool entry where the return type is > validated against it being a constructor returning void. The conversion > happens in BytecodeHelpers.handleConstantDescToHandleInfo where the patched > return type is used for creating the symbol what fails validation. > > 2. The line number thing causes another row of issues. If, as suggested > previously, the line number attribute could accept a label rather than a > BCI, this could be fixed easily. > > 3. Some tests fail due to the impossibility of handling JSR and RET > instructions. I fully understand that those are old, but especially JDBC > drivers are often compiled to rather old Java versions. Java agents often > instrument those, and it would be a pity if those would not be supported > therefore. Is there a plan to add support for these instructions? > Fortunately, stack map frame generation does not need to consider them as > these old class files do not require stack map frames. Adding them is not > the worst for this reason. > > With these three things in place, I think I can offer full support for the > new API starting the day this becomes non-experimental. > > Best regards, Rafael > > Am Mi., 14. Aug. 2024 um 00:43 Uhr schrieb Chen Liang < > chen.l.liang at oracle.com>: > > Hi Rafael, thanks for the quick followup! > Indeed, the line number label handling enhancement would be helpful. (and > that actually applies to all user attribute encoding of Labels, as > currently user attributes cannot convert labels to bcis too) > > For the following stacktrace, I don't think JDK's constants API is wrong: > you should use (Lookup, String, Class) -> void type for the bootstrap > method's method handle type. I checked bytebuddy's JavaConstant.Dynamic and > JavaConstantDynamicTest and still have no clue how that happens, or how it > passes elsewhere. It would be great if you can provide a simple generated > class file or its javap output, especially if this class can run on java > command line so we can check the VM behavior. > > And congratulations on the progress! The "dead code" warning shouldn't > happen usually, as ClassFile API has PATCH_DEAD_CODE by default for > DeadCodeOption. These errors should go away if you restore the option. > > Thanks again for trying out! > Chen Liang > ------------------------------ > *From:* Rafael Winterhalter > *Sent:* Tuesday, August 13, 2024 5:08 PM > *To:* Chen Liang > *Cc:* classfile-api-dev > *Subject:* [External] : Re: ASM to OpenJDK ClassReader/ClassWriter > bridge: experiences and questions > > Thanks, I will try the manual stack map frame translation! As for line > numbers: I also wanted to create the attribute manually. Unfortunately, > LineNumberInfo only accepts a bci instead of a Label. If that can be > adjusted (I think this is sensible as this is the only location where bci > seems to be exposed), I could solve the problem that way. > > I also found a possible bug with the following stacktrace: > > java.lang.IllegalArgumentException: Expected type of (T*)V for > constructor, found > MethodTypeDesc[(MethodHandles$Lookup,String,Class)DynamicConstantBootstrap] > at > java.base/java.lang.constant.DirectMethodHandleDescImpl.validateConstructor(DirectMethodHandleDescImpl.java:107) > at > java.base/java.lang.constant.DirectMethodHandleDescImpl.(DirectMethodHandleDescImpl.java:75) > at > java.base/java.lang.constant.MethodHandleDesc.of(MethodHandleDesc.java:89) > at > java.base/jdk.internal.classfile.impl.AbstractPoolEntry$MethodHandleEntryImpl.asSymbol(AbstractPoolEntry.java:919) > at > java.base/java.lang.classfile.constantpool.ConstantDynamicEntry.asSymbol(ConstantDynamicEntry.java:65) > at > java.base/jdk.internal.classfile.impl.StackMapGenerator.processLdc(StackMapGenerator.java:705) > > This happens when using a constructor as the bootstrap for a dynamic > constant. The check does not seem to consider that constructors are valid > bootstraps similar to static methods. > > Other than that, I ran Byte Buddy's entire test suite with about 10.000 > tests using that bridge now and only about 200 tests seem to fail. Most > commonly this is due to "Unable to generate stack map frame for dead code > at bytecode offset 36 of method foobar()", but this I might be able to > solve with your suggestion. The reminder are mostly about edge cases that I > will investigate further. > > Well done! If these two issues can be resolved, I plan to offer an adapter > into Byte Buddy once this is public API. > Rafael > > Am Di., 13. Aug. 2024 um 23:29 Uhr schrieb Chen Liang < > chen.l.liang at oracle.com>: > > Hi Rafael, thanks for your adoption! > > 1. In fact, ClassFile API has quite a few non-standard (JVMS) > attributes: CharacterRangeTable, CompilationID, ModuleHashes, > ModuleResolution, ModuleTarget, SourceDebugExtension, SourceID, and seems > there's little information available about these attributes. I think you > can treat them as if they are unknown attributes in your translations to > ASM. > 2. Line numbers are naturally streamed just like labels. If an ASM > user is streaming line numbers late, an alternative way is to buffer the > other elements into an ASM tree first, then inject the line number tokens > on the second round. (If LineNumberTableAttribute supports using Labels, > then we can probably just write that attribute at last, too) > 3. Unknown attributes are troubling the ClassFile API too. We have a > stability() in AttributeMapper to indicate if we should retain or drop > attributes. You can always copy any attribute over as-is by calling > XxxBuilder::with(attribute). You should always handle the unknown > attributes with a default branch in a switch if you wish to be forward > compatible, or an exhaustive switch if you wish to fail fast on newer known > attributes. > 4. For stack map frames, you can use StackMapsOption.DROP_STACK_MAPS > and pass a StackMapsTableAttribute in CodeBuilder.with; this is already how > ProxyGenerator does it. However we don't allow users to specify max stacks > and locals, which can be hurting performance sensitive use cases. > > > Regards, > Chen Liang > ------------------------------ > *From:* classfile-api-dev on behalf > of Rafael Winterhalter > *Sent:* Tuesday, August 13, 2024 4:51 AM > *To:* classfile-api-dev > *Subject:* ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and > questions > > Hello! > > It's been a while since I last tried, but I now managed to implement both > a JDK-based ClassReader and ClassWriter for ASM. This seems to work really > well for a range of tests that I did as it allows me to avoid major > rewrites of ASM-based code while still having forward-compatibility as long > as there are no new language constructs as the JDK bundles parser and > serializer. > > The code can be found here: https://github.com/raphw/asm-jdk-bridge > > > A few notes and questions I have: > > 1. The OpenJDK API supports CharacterRangeTable which is a non-standard > attribute that is not described in the specification. By introducing it to > the official API, isn't the attribute in a way formalized? Is there a plan > to standardize the attribute? > > 2. One thing that cannot be mapped directly from OpenJDK to ASM are line > numbers. In ASM, they can be added later using a Label. In OpenJDK they > have to be visited at "the right time". Using labels is otherwise common > for other attributes, both in ASM and OpenJDK, but line numbers are the > exception. Is there a reason for this? > > 3. It would be nice if there was a way to flag what attributes should be > treated as UnknownAttributes. Right now, I retain UnknownAttributes as they > are. But if a future release of the OpenJDK promotes an UnknownAttribute to > a known one, I might miss it when copying them over as raw arrays. > > 4. I take it so that there are no plans to add support for manually > defining StackMapFrame and method sizes? Sometimes, the OpenJDK erases > information, for example if a local variable is never assigned a value, it > will simply be treated as "N" value. This is not a big issue, but it can > make slight transformations that can have very subtle implications when for > example writing Java agents, so I would still hope for an option for this > to be used by advanced users. > > Thanks! Rafael > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Wed Aug 14 20:00:02 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Wed, 14 Aug 2024 22:00:02 +0200 Subject: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions In-Reply-To: References: Message-ID: Hello, I figured out the difference when it comes to annotation. ASM represents arrays of primitive values as constants, not as actual arrays, and passes this down differently. This made a difference when it came to handling incompatible values, but I found the cause of this error in Byte Buddy. ASM simply never triggered this behaviour. I am down to about 10 errors which are all verifier errors for class files that are generated by the class file API. All those tests target rather obscure code, for example from obfuscators, so they are hard to reproduce. As a next step, I will however create a version of Byte Buddy that allows to plug the new visitors, so possibly you can investigate from there. From the dumped class files, it seems as if the local variable or stack sizes are too small for these classes. Finally, I cannot really follow the argument why stack and local variable sizes for methods cannot be supplied manually. You mentioned it was for safety reasons? When it comes to byte code, there is a million ways to create a failed verification already, so why not allow for a manual specification? Most tools already provide these values anyways. Best regards, Rafael Am Mi., 14. Aug. 2024 um 18:34 Uhr schrieb Rafael Winterhalter < rafael.wth at gmail.com>: > Brilliant, I am now down to: > > (1) Waiting for a fix for the bug relating to BytecodeHelpers. > (2) issues with inability to define line numbers by labels, I think your > idea to add a translation method from labels to BCI (and having an > equivalent way of reading a BCI in the reader) is the right thing to do, > this will generally be useful for custom attributes. > (3) thanks, this resolves some more failing tests. > > I now also got the manual computation of stack map frames to work. I would > however love if I could control the exact frames that are issued. Ideally > agents change as little as possible to stay compatible with other byte code > manipulation tools. This could be offered as a "write only" API similar to > what you suggest for line numbers. What do you think? > > As for the annotations, I am still trying to find out what has changed. If > I understand it better, I can file a separate bug report, in case that it > is a regression. > > Thanks, Rafael > > Am Mi., 14. Aug. 2024 um 17:29 Uhr schrieb Chen Liang < > chen.l.liang at oracle.com>: > >> Rafael, first for your annotation output, how did you produce those >> outputs? I don't think ClassFile API performs any check to ensure an >> annotation's structure matches that of its annotation interface. >> >> >> 1. Thanks for the reproducer. It is indeed a bug with >> BytecodeHelpers, which should use lookupDescriptor instead of >> invocationType. Filed https://bugs.openjdk.org/browse/JDK-8338406 which >> I will fix. >> 2. I am thinking of exposing the label to bci conversion to attribute >> writers for Code: that way, we can create a custom LineNumberTable >> attribute that only performs writing, and in writing it converts its labels >> to bci, which should fit our need here. What do you think? >> 3. These instructions are supported: You need to call >> `codeBuilder.with(DiscontinuedInstruction.JsrInstruction.of(...))` to >> create them instead of through convenience factories. And our stack maps >> generation automatically fails and falls back to stack counter in default >> configuration if it encounters a Java 6 class file with jsr/ret. >> >> For the annotation thing, if you can provide the source code, I can help >> you diagnose. I am currently working on both core reflection and classfile >> API areas. >> >> Chen >> ------------------------------ >> *From:* Rafael Winterhalter >> *Sent:* Wednesday, August 14, 2024 6:05 AM >> *To:* Chen Liang >> *Cc:* classfile-api-dev >> *Subject:* Re: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter >> bridge: experiences and questions >> >> Quick finding: my tests that check for incompatible change in annotation >> values compared to their class file. There was a change in Java 21 (if I >> recall correctly) to display the actual value. This seems to have >> regressed. Since Byte Buddy emulates the JDKs behavior, these tests are >> failing now compared to the mainline. For example, if an annotation changes >> its property from int to array int[], previously compiled annotations are >> toString-ed on mainline JDK as: >> >> incompatibleValueArray=/* Warning type mismatch! "java.lang.Integer[42]" >> */ >> >> The ClassFile API branch toString-s the annotation as it was done prior >> to Java 21: >> >> incompatibleValueArray=/* Warning type mismatch! "Array with component >> tag: I" */ >> >> I suggest to use the mainline expression as this makes debugging things >> much easier. >> Thanks, Rafael >> >> Am Mi., 14. Aug. 2024 um 12:36 Uhr schrieb Rafael Winterhalter < >> rafael.wth at gmail.com>: >> >> Hello, >> >> after some further fixes, I am down to 30 failed tests of 10436 tests in >> total now, only using the ClassFile API for reading and writing classes >> with ASM as an intermediate. I have not got the stack map frame generation >> to work yet, but will tackle this next. The remaining >> >> 1. I will try to create a reproducer, but in short: >> >> DirectMethodHandleDesc.Kind kind = >> DirectMethodHandleDesc.Kind.CONSTRUCTOR; >> ClassDesc owner = ClassDesc.of("sample.DynamicConstantBootstrap"); >> String name = ""; >> String lookupDescriptor = >> "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/Class;)V"; >> DirectMethodHandleDesc methodHandle = MethodHandleDesc.of(kind, >> owner, name, lookupDescriptor); >> System.out.println(methodHandle.invocationType().returnType()); >> >> The printed return type will be the owner type. If I change the kind to >> STATIC, it will be void as expected. When adding this to a LCD instruction, >> it will be converted to a constant pool entry where the return type is >> validated against it being a constructor returning void. The conversion >> happens in BytecodeHelpers.handleConstantDescToHandleInfo where the patched >> return type is used for creating the symbol what fails validation. >> >> 2. The line number thing causes another row of issues. If, as suggested >> previously, the line number attribute could accept a label rather than a >> BCI, this could be fixed easily. >> >> 3. Some tests fail due to the impossibility of handling JSR and RET >> instructions. I fully understand that those are old, but especially JDBC >> drivers are often compiled to rather old Java versions. Java agents often >> instrument those, and it would be a pity if those would not be supported >> therefore. Is there a plan to add support for these instructions? >> Fortunately, stack map frame generation does not need to consider them as >> these old class files do not require stack map frames. Adding them is not >> the worst for this reason. >> >> With these three things in place, I think I can offer full support for >> the new API starting the day this becomes non-experimental. >> >> Best regards, Rafael >> >> Am Mi., 14. Aug. 2024 um 00:43 Uhr schrieb Chen Liang < >> chen.l.liang at oracle.com>: >> >> Hi Rafael, thanks for the quick followup! >> Indeed, the line number label handling enhancement would be helpful. (and >> that actually applies to all user attribute encoding of Labels, as >> currently user attributes cannot convert labels to bcis too) >> >> For the following stacktrace, I don't think JDK's constants API is wrong: >> you should use (Lookup, String, Class) -> void type for the bootstrap >> method's method handle type. I checked bytebuddy's JavaConstant.Dynamic and >> JavaConstantDynamicTest and still have no clue how that happens, or how it >> passes elsewhere. It would be great if you can provide a simple generated >> class file or its javap output, especially if this class can run on java >> command line so we can check the VM behavior. >> >> And congratulations on the progress! The "dead code" warning shouldn't >> happen usually, as ClassFile API has PATCH_DEAD_CODE by default for >> DeadCodeOption. These errors should go away if you restore the option. >> >> Thanks again for trying out! >> Chen Liang >> ------------------------------ >> *From:* Rafael Winterhalter >> *Sent:* Tuesday, August 13, 2024 5:08 PM >> *To:* Chen Liang >> *Cc:* classfile-api-dev >> *Subject:* [External] : Re: ASM to OpenJDK ClassReader/ClassWriter >> bridge: experiences and questions >> >> Thanks, I will try the manual stack map frame translation! As for line >> numbers: I also wanted to create the attribute manually. Unfortunately, >> LineNumberInfo only accepts a bci instead of a Label. If that can be >> adjusted (I think this is sensible as this is the only location where bci >> seems to be exposed), I could solve the problem that way. >> >> I also found a possible bug with the following stacktrace: >> >> java.lang.IllegalArgumentException: Expected type of (T*)V for >> constructor, found >> MethodTypeDesc[(MethodHandles$Lookup,String,Class)DynamicConstantBootstrap] >> at >> java.base/java.lang.constant.DirectMethodHandleDescImpl.validateConstructor(DirectMethodHandleDescImpl.java:107) >> at >> java.base/java.lang.constant.DirectMethodHandleDescImpl.(DirectMethodHandleDescImpl.java:75) >> at >> java.base/java.lang.constant.MethodHandleDesc.of(MethodHandleDesc.java:89) >> at >> java.base/jdk.internal.classfile.impl.AbstractPoolEntry$MethodHandleEntryImpl.asSymbol(AbstractPoolEntry.java:919) >> at >> java.base/java.lang.classfile.constantpool.ConstantDynamicEntry.asSymbol(ConstantDynamicEntry.java:65) >> at >> java.base/jdk.internal.classfile.impl.StackMapGenerator.processLdc(StackMapGenerator.java:705) >> >> This happens when using a constructor as the bootstrap for a dynamic >> constant. The check does not seem to consider that constructors are valid >> bootstraps similar to static methods. >> >> Other than that, I ran Byte Buddy's entire test suite with about 10.000 >> tests using that bridge now and only about 200 tests seem to fail. Most >> commonly this is due to "Unable to generate stack map frame for dead code >> at bytecode offset 36 of method foobar()", but this I might be able to >> solve with your suggestion. The reminder are mostly about edge cases that I >> will investigate further. >> >> Well done! If these two issues can be resolved, I plan to offer an >> adapter into Byte Buddy once this is public API. >> Rafael >> >> Am Di., 13. Aug. 2024 um 23:29 Uhr schrieb Chen Liang < >> chen.l.liang at oracle.com>: >> >> Hi Rafael, thanks for your adoption! >> >> 1. In fact, ClassFile API has quite a few non-standard (JVMS) >> attributes: CharacterRangeTable, CompilationID, ModuleHashes, >> ModuleResolution, ModuleTarget, SourceDebugExtension, SourceID, and seems >> there's little information available about these attributes. I think you >> can treat them as if they are unknown attributes in your translations to >> ASM. >> 2. Line numbers are naturally streamed just like labels. If an ASM >> user is streaming line numbers late, an alternative way is to buffer the >> other elements into an ASM tree first, then inject the line number tokens >> on the second round. (If LineNumberTableAttribute supports using Labels, >> then we can probably just write that attribute at last, too) >> 3. Unknown attributes are troubling the ClassFile API too. We have a >> stability() in AttributeMapper to indicate if we should retain or drop >> attributes. You can always copy any attribute over as-is by calling >> XxxBuilder::with(attribute). You should always handle the unknown >> attributes with a default branch in a switch if you wish to be forward >> compatible, or an exhaustive switch if you wish to fail fast on newer known >> attributes. >> 4. For stack map frames, you can use StackMapsOption.DROP_STACK_MAPS >> and pass a StackMapsTableAttribute in CodeBuilder.with; this is already how >> ProxyGenerator does it. However we don't allow users to specify max stacks >> and locals, which can be hurting performance sensitive use cases. >> >> >> Regards, >> Chen Liang >> ------------------------------ >> *From:* classfile-api-dev on behalf >> of Rafael Winterhalter >> *Sent:* Tuesday, August 13, 2024 4:51 AM >> *To:* classfile-api-dev >> *Subject:* ASM to OpenJDK ClassReader/ClassWriter bridge: experiences >> and questions >> >> Hello! >> >> It's been a while since I last tried, but I now managed to implement both >> a JDK-based ClassReader and ClassWriter for ASM. This seems to work really >> well for a range of tests that I did as it allows me to avoid major >> rewrites of ASM-based code while still having forward-compatibility as long >> as there are no new language constructs as the JDK bundles parser and >> serializer. >> >> The code can be found here: https://github.com/raphw/asm-jdk-bridge >> >> >> A few notes and questions I have: >> >> 1. The OpenJDK API supports CharacterRangeTable which is a non-standard >> attribute that is not described in the specification. By introducing it to >> the official API, isn't the attribute in a way formalized? Is there a plan >> to standardize the attribute? >> >> 2. One thing that cannot be mapped directly from OpenJDK to ASM are line >> numbers. In ASM, they can be added later using a Label. In OpenJDK they >> have to be visited at "the right time". Using labels is otherwise common >> for other attributes, both in ASM and OpenJDK, but line numbers are the >> exception. Is there a reason for this? >> >> 3. It would be nice if there was a way to flag what attributes should be >> treated as UnknownAttributes. Right now, I retain UnknownAttributes as they >> are. But if a future release of the OpenJDK promotes an UnknownAttribute to >> a known one, I might miss it when copying them over as raw arrays. >> >> 4. I take it so that there are no plans to add support for manually >> defining StackMapFrame and method sizes? Sometimes, the OpenJDK erases >> information, for example if a local variable is never assigned a value, it >> will simply be treated as "N" value. This is not a big issue, but it can >> make slight transformations that can have very subtle implications when for >> example writing Java agents, so I would still hope for an option for this >> to be used by advanced users. >> >> Thanks! Rafael >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From chen.l.liang at oracle.com Wed Aug 14 21:09:17 2024 From: chen.l.liang at oracle.com (Chen Liang) Date: Wed, 14 Aug 2024 21:09:17 +0000 Subject: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions In-Reply-To: References: Message-ID: For 3 I am talking about the of(Object) instead of the ofXyz ones, indeed. We seek to remove AnnotationValue.of(Object). Also thanks for the suggestion of this method on CodeBuilder. I personally lean more on dropping automatic computation once this method is called, but feel free to start another thread for more input. Chen ________________________________ From: Rafael Winterhalter Sent: Wednesday, August 14, 2024 3:55 PM To: Chen Liang Cc: classfile-api-dev Subject: Re: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions As for (3): Do you mean the AnnotationValue.of(Object) method, or the entire hierarchy of AnnotationValue.ofYYY(...) methods? I am only using the latter methods, and I would not know of an alternative tree model. Can you give me a pointer? As for (4): I would implement this similar to stack map frames. Compute them by default but offer an option to keep them at zero by default: ClassFile.MethodSizeOptions.COMPUTE ClassFile.MethodSizeOptions.ZERO Then there could be a method on CodeBuilder such as "maxs(short, short)" where the relevant values are simply written to the relevant class file location. Best regards, Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Wed Aug 14 21:13:42 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Wed, 14 Aug 2024 23:13:42 +0200 Subject: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions In-Reply-To: References: Message-ID: I see, I do not use the Object-version of the method. The method requires loaded enums anyways what ASM does not provide, and it leaves annotations out, so it could not be used successfully anyways. I think it can be safely removed. Am Mi., 14. Aug. 2024 um 23:09 Uhr schrieb Chen Liang < chen.l.liang at oracle.com>: > For 3 I am talking about the of(Object) instead of the ofXyz ones, indeed. > We seek to remove AnnotationValue.of(Object). > Also thanks for the suggestion of this method on CodeBuilder. I personally > lean more on dropping automatic computation once this method is called, but > feel free to start another thread for more input. > > Chen > > ------------------------------ > *From:* Rafael Winterhalter > *Sent:* Wednesday, August 14, 2024 3:55 PM > *To:* Chen Liang > *Cc:* classfile-api-dev > *Subject:* Re: [External] : Re: ASM to OpenJDK ClassReader/ClassWriter > bridge: experiences and questions > > As for (3): Do you mean the AnnotationValue.of(Object) method, or the > entire hierarchy of AnnotationValue.ofYYY(...) methods? I am only using the > latter methods, and I would not know of an alternative tree model. Can you > give me a pointer? > As for (4): I would implement this similar to stack map frames. Compute > them by default but offer an option to keep them at zero by default: > > ClassFile.MethodSizeOptions.COMPUTE > ClassFile.MethodSizeOptions.ZERO > > Then there could be a method on CodeBuilder such as "maxs(short, short)" > where the relevant values are simply written to the relevant class file > location. > > Best regards, Rafael > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Thu Aug 15 21:33:08 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Thu, 15 Aug 2024 23:33:08 +0200 Subject: Display verification error on retransformations Message-ID: Hello, if a Java agent fails to retransform due to a verification error (using Instrumentation.retransformClasses), this error does not contain a message. This is like it used to be for all verification errors, when a class was loaded for the first time, but this feature of displaying a message and byte code was added later. I understand that retransformation might trigger multiple classes where many of them could be faulty, but even if only the first error is displayed (and possibly the name of the class in question), this would be of great help when debugging. This might not even be the right mailing list for this, but as you are touching class files and since this API likely will be used to implement agents, possibly this mail just meets the right people to ask for a background? Is this a feature that was considered for the JVM? Thanks, Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From chen.l.liang at oracle.com Thu Aug 15 21:52:25 2024 From: chen.l.liang at oracle.com (Chen Liang) Date: Thu, 15 Aug 2024 21:52:25 +0000 Subject: Display verification error on retransformations In-Reply-To: References: Message-ID: Unfortunately, ClassFile API is not used to produce verification error messages by hotspot. You can use ClassFile.verify to verify after generating a class if you encounter an error, or toggle the verification or bytecode dumping in your application with a system property for easier reproduction and debugging. For more serviceability questions, you can ask about instrumentation and such at serviceability-dev list. Chen ________________________________ From: classfile-api-dev on behalf of Rafael Winterhalter Sent: Thursday, August 15, 2024 4:33 PM To: classfile-api-dev Subject: Display verification error on retransformations Hello, if a Java agent fails to retransform due to a verification error (using Instrumentation.retransformClasses), this error does not contain a message. This is like it used to be for all verification errors, when a class was loaded for the first time, but this feature of displaying a message and byte code was added later. I understand that retransformation might trigger multiple classes where many of them could be faulty, but even if only the first error is displayed (and possibly the name of the class in question), this would be of great help when debugging. This might not even be the right mailing list for this, but as you are touching class files and since this API likely will be used to implement agents, possibly this mail just meets the right people to ask for a background? Is this a feature that was considered for the JVM? Thanks, Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Mon Aug 19 18:05:45 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Mon, 19 Aug 2024 20:05:45 +0200 Subject: Wrong stack computation with (retained) dead code Message-ID: Hello, this is tested with a recent 24 EA. I discovered that some generated classes in Byte Buddy fail to verify with: Operand stack overflow - Exceeded stack size. This seems to happen when adding dead code, even if frames are created manually. For example, if I use the class file API to create the following method, I'd create the above verify error: String m() { LDC "x" ARETURN F_SAME LDC "x" LDC "x" ARETURN } This code might appear meaningless, but there are a bunch of code generators that create such weird code, that is why Byte Buddy supports it. It seems like the class file API computes the required stack size at 1, not as 2. Currently, I cannot override this. If this class is picked up by a Java agent and simply passed and returned, the max size will therefore decrease and the verification error will surface. Best regards, Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Mon Aug 19 20:22:46 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Mon, 19 Aug 2024 22:22:46 +0200 Subject: StackMapTable format error: bad type array size in method Message-ID: Hello, when testing some of the corner cases within the unit tests of Byte Buddy, I found some further errors when it comes to dead code and using ClassWriter. All those cases work when using ASM. As a reproducer, consider the following: ClassFile classFile = ClassFile.of(ClassFile.StackMapsOption.DROP_STACK_MAPS); byte[] bytes = classFile.build(ClassDesc.of("foo.Bar"), classBuilder -> classBuilder.withMethod( "foo", MethodTypeDesc.ofDescriptor("()J"), 0, methodBuilder -> { methodBuilder.withCode(codeBuilder -> { codeBuilder.new_(ClassDesc.of(RuntimeException.class.getName())); codeBuilder.dup(); codeBuilder.invokespecial(ClassDesc.of(RuntimeException.class.getName()), "", MethodTypeDesc.ofDescriptor("()V")); codeBuilder.athrow(); Label f2 = codeBuilder.newBoundLabel(); codeBuilder.lstore(1); Label f3 = codeBuilder.newBoundLabel(); codeBuilder.lload(1); codeBuilder.lreturn(); codeBuilder.with(StackMapTableAttribute.of(List.of( StackMapFrameInfo.of(f2, List.of(StackMapFrameInfo.ObjectVerificationTypeInfo.of(ClassDesc.of("foo.Bar"))), List.of(StackMapFrameInfo.SimpleVerificationTypeInfo.ITEM_LONG)), StackMapFrameInfo.of(f3, List.of(StackMapFrameInfo.ObjectVerificationTypeInfo.of(ClassDesc.of("foo.Bar")), StackMapFrameInfo.SimpleVerificationTypeInfo.ITEM_LONG), List.of()) ))); }); })); new ClassLoader() { @Override protected Class findClass(String name) throws ClassNotFoundException { if (name.equals("foo.Bar")) { return defineClass(name, bytes, 0, bytes.length); } else { return super.findClass(name); } } }.findClass("foo.Bar").getMethods(); It gives a class format error. Java agents sometimes have to process all kinds of strange byte code, so ideally these cases should be supported. Is this a bug in the stack map frame attribute writer? Thanks! Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Mon Aug 19 20:26:02 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Mon, 19 Aug 2024 22:26:02 +0200 Subject: Wrong stack computation with (retained) dead code In-Reply-To: References: Message-ID: Here is a simple reproducer of this: ClassFile classFile = ClassFile.of(ClassFile.StackMapsOption.DROP_STACK_MAPS); byte[] bytes = classFile.build(ClassDesc.of("foo.Bar"), classBuilder -> classBuilder.withMethod( "foo", MethodTypeDesc.ofDescriptor("()Ljava/lang/Object;"), 0, methodBuilder -> { methodBuilder.withCode(codeBuilder -> { codeBuilder.ldc("x"); codeBuilder.areturn(); Label label = codeBuilder.newBoundLabel(); codeBuilder.ldc("x"); codeBuilder.ldc("x"); codeBuilder.areturn(); codeBuilder.with(StackMapTableAttribute.of(List.of( StackMapFrameInfo.of(label, List.of(StackMapFrameInfo.ObjectVerificationTypeInfo.of(ClassDesc.of("foo.Bar"))), List.of())))); }); })); new ClassLoader() { @Override protected Class findClass(String name) throws ClassNotFoundException { if (name.equals("foo.Bar")) { return defineClass(name, bytes, 0, bytes.length); } else { return super.findClass(name); } } }.findClass("foo.Bar").getMethods(); Am Mo., 19. Aug. 2024 um 20:05 Uhr schrieb Rafael Winterhalter < rafael.wth at gmail.com>: > Hello, > > this is tested with a recent 24 EA. > > I discovered that some generated classes in Byte Buddy fail to verify > with: Operand stack overflow - Exceeded stack size. This seems to happen > when adding dead code, even if frames are created manually. For example, if > I use the class file API to create the following method, I'd create the > above verify error: > > String m() { > LDC "x" > ARETURN > F_SAME > LDC "x" > LDC "x" > ARETURN > } > > This code might appear meaningless, but there are a bunch of code > generators that create such weird code, that is why Byte Buddy supports it. > It seems like the class file API computes the required stack size at 1, not > as 2. Currently, I cannot override this. > > If this class is picked up by a Java agent and simply passed and returned, > the max size will therefore decrease and the verification error will > surface. > > Best regards, Rafael > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.sotona at oracle.com Tue Aug 20 09:32:25 2024 From: adam.sotona at oracle.com (Adam Sotona) Date: Tue, 20 Aug 2024 09:32:25 +0000 Subject: StackMapTable format error: bad type array size in method In-Reply-To: References: Message-ID: Hello, There is a problem in StackCounter calculation of maxLocals in combination with custom StackMapTableAttribute covering ?dead code?. Feel free to report it as a bug or let me know and I?ll fill it. Thank you, Adam From: classfile-api-dev on behalf of Rafael Winterhalter Date: Monday, 19 August 2024 at 22:23 To: classfile-api-dev Subject: StackMapTable format error: bad type array size in method Hello, when testing some of the corner cases within the unit tests of Byte Buddy, I found some further errors when it comes to dead code and using ClassWriter. All those cases work when using ASM. As a reproducer, consider the following: ClassFile classFile = ClassFile.of(ClassFile.StackMapsOption.DROP_STACK_MAPS); byte[] bytes = classFile.build(ClassDesc.of("foo.Bar"), classBuilder -> classBuilder.withMethod( "foo", MethodTypeDesc.ofDescriptor("()J"), 0, methodBuilder -> { methodBuilder.withCode(codeBuilder -> { codeBuilder.new_(ClassDesc.of(RuntimeException.class.getName())); codeBuilder.dup(); codeBuilder.invokespecial(ClassDesc.of(RuntimeException.class.getName()), "", MethodTypeDesc.ofDescriptor("()V")); codeBuilder.athrow(); Label f2 = codeBuilder.newBoundLabel(); codeBuilder.lstore(1); Label f3 = codeBuilder.newBoundLabel(); codeBuilder.lload(1); codeBuilder.lreturn(); codeBuilder.with(StackMapTableAttribute.of(List.of( StackMapFrameInfo.of(f2, List.of(StackMapFrameInfo.ObjectVerificationTypeInfo.of(ClassDesc.of("foo.Bar"))), List.of(StackMapFrameInfo.SimpleVerificationTypeInfo.ITEM_LONG)), StackMapFrameInfo.of(f3, List.of(StackMapFrameInfo.ObjectVerificationTypeInfo.of(ClassDesc.of("foo.Bar")), StackMapFrameInfo.SimpleVerificationTypeInfo.ITEM_LONG), List.of()) ))); }); })); new ClassLoader() { @Override protected Class findClass(String name) throws ClassNotFoundException { if (name.equals("foo.Bar")) { return defineClass(name, bytes, 0, bytes.length); } else { return super.findClass(name); } } }.findClass("foo.Bar").getMethods(); It gives a class format error. Java agents sometimes have to process all kinds of strange byte code, so ideally these cases should be supported. Is this a bug in the stack map frame attribute writer? Thanks! Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Tue Aug 20 10:00:22 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Tue, 20 Aug 2024 12:00:22 +0200 Subject: StackMapTable format error: bad type array size in method In-Reply-To: References: Message-ID: I filed it here: https://bugs.openjdk.org/browse/JDK-8338661 Am Di., 20. Aug. 2024 um 11:32 Uhr schrieb Adam Sotona < adam.sotona at oracle.com>: > Hello, > > There is a problem in StackCounter calculation of maxLocals in combination > with custom StackMapTableAttribute covering ?dead code?. > > > > Feel free to report it as a bug or let me know and I?ll fill it. > > > > Thank you, > > Adam > > > > *From: *classfile-api-dev on behalf > of Rafael Winterhalter > *Date: *Monday, 19 August 2024 at 22:23 > *To: *classfile-api-dev > *Subject: *StackMapTable format error: bad type array size in method > > Hello, > > when testing some of the corner cases within the unit tests of Byte Buddy, > I found some further errors when it comes to dead code and using > ClassWriter. All those cases work when using ASM. As a reproducer, consider > the following: > > > ClassFile classFile = ClassFile.*of*(ClassFile.StackMapsOption.*DROP_STACK_MAPS*); > byte[] bytes = classFile.build(ClassDesc.*of*("foo.Bar"), classBuilder -> classBuilder.withMethod( > "foo", > MethodTypeDesc.*ofDescriptor*("()J"), > 0, > methodBuilder -> { > methodBuilder.withCode(codeBuilder -> { > codeBuilder.new_(ClassDesc.*of*(RuntimeException.class.getName())); > codeBuilder.dup(); > codeBuilder.invokespecial(ClassDesc.*of*(RuntimeException.class.getName()), > "", > MethodTypeDesc.*ofDescriptor*("()V")); > codeBuilder.athrow(); > Label f2 = codeBuilder.newBoundLabel(); > codeBuilder.lstore(1); > Label f3 = codeBuilder.newBoundLabel(); > codeBuilder.lload(1); > codeBuilder.lreturn(); > codeBuilder.with(StackMapTableAttribute.*of*(List.*of*( > StackMapFrameInfo.*of*(f2, > List.*of*(StackMapFrameInfo.ObjectVerificationTypeInfo.*of*(ClassDesc.*of*("foo.Bar"))), > List.*of*(StackMapFrameInfo.SimpleVerificationTypeInfo.*ITEM_LONG*)), > StackMapFrameInfo.*of*(f3, > List.*of*(StackMapFrameInfo.ObjectVerificationTypeInfo.*of*(ClassDesc.*of*("foo.Bar")), > StackMapFrameInfo.SimpleVerificationTypeInfo.*ITEM_LONG*), > List.*of*()) > ))); > }); > })); > new ClassLoader() { > @Override > protected Class findClass(String name) throws ClassNotFoundException { > if (name.equals("foo.Bar")) { > return defineClass(name, bytes, 0, bytes.length); > } else { > return super.findClass(name); > } > } > }.findClass("foo.Bar").getMethods(); > > It gives a class format error. Java agents sometimes have to process all > kinds of strange byte code, so ideally these cases should be supported. Is > this a bug in the stack map frame attribute writer? > > > > Thanks! Rafael > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Thu Aug 22 07:46:34 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Thu, 22 Aug 2024 09:46:34 +0200 Subject: Byte Buddy: down to 2 test errors only, and more feedback Message-ID: I can happily report that I am down to two test errors (out of over 10.000 tests!) when only using the adapters for the ASM API that I wrote. Those two are related to the inability of assigning line numbers late since the attribute requires BCI instead of labels, as I discussed prior. If it becomes possible to translate labels from an attribute writer, or if the standard attribute for line numbers becomes based on labels, I would be able to fully build Byte Buddy without ASM class readers and class writers. With that being said, I still encounter some problematic bits: 1. It is rather hard to translate between attributes in ASM and the Class File API. Ideally, I would want to bridge the Class File APIs AttributeMapper interface to write and read from ASM and the other way round. In ASM, this requires some API extensions as the current API for reading and writing is not accessible for outside users. It therefore requires some hacks and the use of reflection. The other way is even more difficult since I cannot implement interfaces as BufWriter myself to direct to ASM. Could it be considered to create a superinterface of the parameters in the AttributeMapper that only contain the relevant methods and keep those unsealed? This way, they are not bound to the JDK's implementation. 2. I find it a bit confusing that the AttributeMapper reads the attribute name automatically (this is needed anyways to find the right mapper), but does not write it automatically. Writing the name is always required to create a valid class file, and right now reading and writing is asymmetric as a result. I suggest to write the attribute name before invoking AttributeMapper::writeAttribute as it can be read from AttributeMapper::name. Or was this intentional to allow multifaceted attribute translations? 3. I would still like to control what stack map frames are generated when placing the attribute manually. In theory, I can write my own AttributeMapper for this once the mapper allows to translate BCI to Labels, but I wonder if the JDK should offer this? As a general rule I experienced byte code manipulation to be the most reliable when as much as possible is retained in its original state since some tools make assumptions. 4. I would still love to control the values for max size and local variable array for the same reason as just stated. Also, if the computation in the JDK would show a bug, the bug can be adressed by the using tool rather than requiring a JDK upgrade. Byte Buddy, for instance, does a lot of book keeping and is battle tested for many corner cases. I would assume that its computation can normally be considered reliable. Thanks for the great job so far and for addressing my suggestions! Best regards, Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Thu Aug 22 15:35:13 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Thu, 22 Aug 2024 17:35:13 +0200 Subject: Stack Map frames dropped when copying full method Message-ID: Hello, I stumbled over something I did not expect and I am wondering if this behaviour is intended. I create a class where I create a method. I add the stack map frames manually to avoid resolving super classes, since I already have the meta data. For this, I set ClassFile.StackMapsOption.DROP_STACK_MAPS. For the same class, I read a method from another class, take the MethodModel and add it. In the resulting class, the stack map frames for the copied method are dropped, too. This I find surprising, because I only intended to avoid computation for stack map frames that were not there from before. I did not expect the frames to be dropped if I copied a full method model including all of its attributes. What do you think here? Is there a way to retain the stack map table attribute for method models that are copied over while being able to resolve frames manually for new methods? Thanks! Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafael.wth at gmail.com Thu Aug 29 11:05:00 2024 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Thu, 29 Aug 2024 13:05:00 +0200 Subject: Working with attributes: experiences and suggestions Message-ID: Hello, I recently implemented a bridge between attributes in the Class File API and ASM, which mostly succeeded, but I wanted to point out that this area is one with the least compatibility. Making this production ready would also require minor adjustments in the ASM API, but it seems like the team there is open to those. I also understand that it's not a project goal to replicate the ASM, but I assume that the ASM API is built upon actual use cases and that a good compatibility will make migration easier for those who aim for it. To begin with, there are two minor incompatibilities that I do not think will have a big impact, but which would be nice to fix: - ASM exposes className, superClassName and interfaceNames from its ClassReader. The Class File API only exposes className and superClassName. One can still navigate through the attributed element to get hold of the interface names unless attributing a record component info where the parent is not exposed. This is mainly an inconvenience. - ASM allows for translating constant pool indices to class file offsets. In theory, one can read the entire class file using the Class File API's ClassReader and offer the same translation, but I would want to avoid manual parsing. The ASM API is offset based and only allows access to most constants using an offset. The Class File API offers access for either offset or index, but includes no possibility to translate. Maybe ASM could however offer a way to access by either offset or index, too, and deprecate the translation method. The Class File API is more convenient here and there is no reason to ask a user to make conversions. The more complex operation where I cannot currently succeed are attributes on the code attribute. Unfortunately, those are the most common custom attributes, and they often carry references to the code. Here I struggle to use the Class File API altogether. Normally, a custom attribute on code stores offsets to the byte code. In some way, I want to add some information about a section of code. When reading such an attribute I would read those offsets, which are typically presented as labels. I think that from AttributeMapper::read, it should be possible to tell the Class File API that labels should be inserted at given positions. Those labels could then be discovered when walking through the instructions. For example one could add a method "bciToLabel" method somewhere. The resolved custom attribute could then only contain these labels such that a user of the Class File API can match the labeled offset to a segment in the code. At the same time, AttributeMapper::write should contain a form of "labelToBci" method such that (possibly altered) byte code offsets can be stored back in the class file. Possibly getBci could be exposed on the Label, too, maybe returning an OptionalInt to accommodate labels where the offset is not yet bound. The issue with this is that form of logic is only relevant to attributes on the code attribute. ASM has itself a special handling of such attributes where one overrides the isCodeAttribute method of org.ow2.asm.Attribute class. When reading such an attribute, it is also passed additional information. On reading, it is passed the beginning offset of the code attribute itself and an array of labels that are included in the code. There is however no way of inserting new labels as of today, something I consider a lacking feature of ASM. When writing such an attribute, ASM is including most data from the CodeAttribute to which the attribute is written to. Personlly, I also think that the Class File API should add the context to which an attribute is written to as an argument, similar to when reading an attribute. It is fully possible to write different data for example when adding an attribute to a field, or to a method. Ideally, it should be possible to (lazily) resolve previously written, different attributes from the target when writing, to add contextual information. Thanks! If you are curious how the back and forth mapping of ASM and JDK attributes currently looks like, you can check this class: https://github.com/raphw/asm-jdk-bridge/blob/main/src/main/java/codes/rafael/asmjdkbridge/AsmAttribute.java#L59 Best regards, Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Thu Aug 29 16:04:35 2024 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 29 Aug 2024 11:04:35 -0500 Subject: Convenience methods for `loadConstant` Message-ID: Today I noticed 8339217 (`loadConstant` overloads for `int`, `long`, etc.) come across, which is great (I was just thinking of writing an email to complain about a lack of this and possibly propose a PR for it). However I have one other frequent headache around loading constants, and that is the fact that I am very frequently finding myself loading constants for `Constable` objects which do not also implement `ConstantDesc`. Thus I find myself frequently typing the sequence `.describeConstable().orElseThrow()`. The most common offenders are `Class`, `MethodType`, and `MethodHandle`. I find this problematic not only because of the repetitive typing, but also because if I want to actually provide some consistent exception message I have to write a utility method, which I would then likely need to replicate since I use this pattern quite often. I would like to propose a `loadConstant(Constable)` variant, which throws a reasonably descriptive exception when the `Constable` cannot be described. One problem with such a proposal is that if there are overloads for both `ClassDesc` and `Constable`, then anything implementing both would cause a compilation issue. So maybe this could be done as `loadConstable(Constable)` or `loadConstantOrElseThrow(Constable)` to avoid the overload ambiguity? If this proposal makes sense in any form, then I could open a bug and propose a PR, if that is OK. -- - DML ? he/him -------------- next part -------------- An HTML attachment was scrubbed... URL: From david32768 at btinternet.com Sat Aug 31 07:18:10 2024 From: david32768 at btinternet.com (david32768 at btinternet.com) Date: Sat, 31 Aug 2024 08:18:10 +0100 (BST) Subject: Switch targets are not inflated in CodeModel if no StackMap. Message-ID: <3b879ddf.278d8.191a74c728c.Webtop.250@btinternet.com> Method jdk/internal/CodeImpl.inflateJumpTargets should also inflate switch instructions. ? Perhaps BranchInstruction, JsrInstruction, TableSwitchInstruction, LookupSwitchInstruction should implement an interface with method List