[External] : Re: ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and questions

Wed Aug 14 10:36:23 UTC 2024

Hello,

after some further fixes, I am down to 30 failed tests of 10436 tests in
total now, only using the ClassFile API for reading and writing classes
with ASM as an intermediate. I have not got the stack map frame generation
to work yet, but will tackle this next. The remaining

1. I will try to create a reproducer, but in short:

        DirectMethodHandleDesc.Kind kind =
DirectMethodHandleDesc.Kind.CONSTRUCTOR;
        ClassDesc owner = ClassDesc.of("sample.DynamicConstantBootstrap");
        String name = "<init>";
        String lookupDescriptor =
"(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/Class;)V";
        DirectMethodHandleDesc methodHandle = MethodHandleDesc.of(kind,
owner, name, lookupDescriptor);
        System.out.println(methodHandle.invocationType().returnType());

The printed return type will be the owner type. If I change the kind to
STATIC, it will be void as expected. When adding this to a LCD instruction,
it will be converted to a constant pool entry where the return type is
validated against it being a constructor returning void. The conversion
happens in BytecodeHelpers.handleConstantDescToHandleInfo where the patched
return type is used for creating the symbol what fails validation.

2. The line number thing causes another row of issues. If, as suggested
previously, the line number attribute could accept a label rather than a
BCI, this could be fixed easily.

3. Some tests fail due to the impossibility of handling JSR and RET
instructions. I fully understand that those are old, but especially JDBC
drivers are often compiled to rather old Java versions. Java agents often
instrument those, and it would be a pity if those would not be supported
therefore. Is there a plan to add support for these instructions?
Fortunately, stack map frame generation does not need to consider them as
these old class files do not require stack map frames. Adding them is not
the worst for this reason.

With these three things in place, I think I can offer full support for the
new API starting the day this becomes non-experimental.

Best regards, Rafael

Am Mi., 14. Aug. 2024 um 00:43 Uhr schrieb Chen Liang <
chen.l.liang at oracle.com>:

> Hi Rafael, thanks for the quick followup!
> Indeed, the line number label handling enhancement would be helpful. (and
> that actually applies to all user attribute encoding of Labels, as
> currently user attributes cannot convert labels to bcis too)
>
> For the following stacktrace, I don't think JDK's constants API is wrong:
> you should use (Lookup, String, Class) -> void type for the bootstrap
> method's method handle type. I checked bytebuddy's JavaConstant.Dynamic and
> JavaConstantDynamicTest and still have no clue how that happens, or how it
> passes elsewhere. It would be great if you can provide a simple generated
> class file or its javap output, especially if this class can run on java
> command line so we can check the VM behavior.
>
> And congratulations on the progress! The "dead code" warning shouldn't
> happen usually, as ClassFile API has PATCH_DEAD_CODE by default for
> DeadCodeOption. These errors should go away if you restore the option.
>
> Thanks again for trying out!
> Chen Liang
> ------------------------------
> *From:* Rafael Winterhalter <rafael.wth at gmail.com>
> *Sent:* Tuesday, August 13, 2024 5:08 PM
> *To:* Chen Liang <chen.l.liang at oracle.com>
> *Cc:* classfile-api-dev <classfile-api-dev at openjdk.org>
> *Subject:* [External] : Re: ASM to OpenJDK ClassReader/ClassWriter
> bridge: experiences and questions
>
> Thanks, I will try the manual stack map frame translation! As for line
> numbers: I also wanted to create the attribute manually. Unfortunately,
> LineNumberInfo only accepts a bci instead of a Label. If that can be
> adjusted (I think this is sensible as this is the only location where bci
> seems to be exposed), I could solve the problem that way.
>
> I also found a possible bug with the following stacktrace:
>
> java.lang.IllegalArgumentException: Expected type of (T*)V for
> constructor, found
> MethodTypeDesc[(MethodHandles$Lookup,String,Class)DynamicConstantBootstrap]
> at
> java.base/java.lang.constant.DirectMethodHandleDescImpl.validateConstructor(DirectMethodHandleDescImpl.java:107)
> at
> java.base/java.lang.constant.DirectMethodHandleDescImpl.<init>(DirectMethodHandleDescImpl.java:75)
> at
> java.base/java.lang.constant.MethodHandleDesc.of(MethodHandleDesc.java:89)
> at
> java.base/jdk.internal.classfile.impl.AbstractPoolEntry$MethodHandleEntryImpl.asSymbol(AbstractPoolEntry.java:919)
> at
> java.base/java.lang.classfile.constantpool.ConstantDynamicEntry.asSymbol(ConstantDynamicEntry.java:65)
> at
> java.base/jdk.internal.classfile.impl.StackMapGenerator.processLdc(StackMapGenerator.java:705)
>
> This happens when using a constructor as the bootstrap for a dynamic
> constant. The check does not seem to consider that constructors are valid
> bootstraps similar to static methods.
>
> Other than that, I ran Byte Buddy's entire test suite with about 10.000
> tests using that bridge now and only about 200 tests seem to fail. Most
> commonly this is due to "Unable to generate stack map frame for dead code
> at bytecode offset 36 of method foobar()", but this I might be able to
> solve with your suggestion. The reminder are mostly about edge cases that I
> will investigate further.
>
> Well done! If these two issues can be resolved, I plan to offer an adapter
> into Byte Buddy once this is public API.
> Rafael
>
> Am Di., 13. Aug. 2024 um 23:29 Uhr schrieb Chen Liang <
> chen.l.liang at oracle.com>:
>
> Hi Rafael, thanks for your adoption!
>
>    1. In fact, ClassFile API has quite a few non-standard (JVMS)
>    attributes: CharacterRangeTable, CompilationID, ModuleHashes,
>    ModuleResolution, ModuleTarget, SourceDebugExtension, SourceID, and seems
>    there's little information available about these attributes. I think you
>    can treat them as if they are unknown attributes in your translations to
>    ASM.
>    2. Line numbers are naturally streamed just like labels. If an ASM
>    user is streaming line numbers late, an alternative way is to buffer the
>    other elements into an ASM tree first, then inject the line number tokens
>    on the second round. (If LineNumberTableAttribute supports using Labels,
>    then we can probably just write that attribute at last, too)
>    3. Unknown attributes are troubling the ClassFile API too. We have a
>    stability() in AttributeMapper to indicate if we should retain or drop
>    attributes. You can always copy any attribute over as-is by calling
>    XxxBuilder::with(attribute). You should always handle the unknown
>    attributes with a default branch in a switch if you wish to be forward
>    compatible, or an exhaustive switch if you wish to fail fast on newer known
>    attributes.
>    4. For stack map frames, you can use StackMapsOption.DROP_STACK_MAPS
>    and pass a StackMapsTableAttribute in CodeBuilder.with; this is already how
>    ProxyGenerator does it. However we don't allow users to specify max stacks
>    and locals, which can be hurting performance sensitive use cases.
>
>
> Regards,
> Chen Liang
> ------------------------------
> *From:* classfile-api-dev <classfile-api-dev-retn at openjdk.org> on behalf
> of Rafael Winterhalter <rafael.wth at gmail.com>
> *Sent:* Tuesday, August 13, 2024 4:51 AM
> *To:* classfile-api-dev <classfile-api-dev at openjdk.org>
> *Subject:* ASM to OpenJDK ClassReader/ClassWriter bridge: experiences and
> questions
>
> Hello!
>
> It's been a while since I last tried, but I now managed to implement both
> a JDK-based ClassReader and ClassWriter for ASM. This seems to work really
> well for a range of tests that I did as it allows me to avoid major
> rewrites of ASM-based code while still having forward-compatibility as long
> as there are no new language constructs as the JDK bundles parser and
> serializer.
>
> The code can be found here: https://github.com/raphw/asm-jdk-bridge
> <https://urldefense.com/v3/__https://github.com/raphw/asm-jdk-bridge__;!!ACWV5N9M2RV99hQ!KAQFTclWgq_IwySyN468Wdy3QUCwqLln_sFAi0SSvWoBHHyKMExsxZbmfhPm0axz2oGf-EbHxWkP391GDU6uYQ$>
>
> A few notes and questions I have:
>
> 1. The OpenJDK API supports CharacterRangeTable which is a non-standard
> attribute that is not described in the specification. By introducing it to
> the official API, isn't the attribute in a way formalized? Is there a plan
> to standardize the attribute?
>
> 2. One thing that cannot be mapped directly from OpenJDK to ASM are line
> numbers. In ASM, they can be added later using a Label. In OpenJDK they
> have to be visited at "the right time". Using labels is otherwise common
> for other attributes, both in ASM and OpenJDK, but line numbers are the
> exception. Is there a reason for this?
>
> 3. It would be nice if there was a way to flag what attributes should be
> treated as UnknownAttributes. Right now, I retain UnknownAttributes as they
> are. But if a future release of the OpenJDK promotes an UnknownAttribute to
> a known one, I might miss it when copying them over as raw arrays.
>
> 4. I take it so that there are no plans to add support for manually
> defining StackMapFrame and method sizes? Sometimes, the OpenJDK erases
> information, for example if a local variable is never assigned a value, it
> will simply be treated as "N" value. This is not a big issue, but it can
> make slight transformations that can have very subtle implications when for
> example writing Java agents, so I would still hope for an option for this
> to be used by advanced users.
>
> Thanks! Rafael
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/classfile-api-dev/attachments/20240814/edc38a05/attachment-0001.htm>