<div dir="ltr">Hi Konstantin,<div><div><div>What you propose has a large overlap with Project Babylon (<a href="https://openjdk.org/projects/babylon/">https://openjdk.org/projects/babylon/</a>), which accomplishes "code reflection" from the Java compiler. The project itself also ships a code model that's suitable for representing ASTs from different programming languages, including ones from Java bytecode. Then your proposal would be much simpler - to generate an expression tree from class files.</div></div></div><div><br></div><div>Since the discuss list isn't really for development questions, I am replying to babylon-dev mailing list instead and we will sail from there.</div><div><br></div><div>Regards</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 21, 2024 at 8:32 PM Konstantin Triger <<a href="mailto:kostat@gmail.com">kostat@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Author: Konstantin Triger</div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Type: Feature</div><div>Scope: SE</div><div>Relates to: <a href="https://openjdk.org/jeps/466" style="text-decoration-line:none;color:rgb(67,114,145);font-family:"DejaVu Sans","Bitstream Vera Sans","Luxi Sans",Verdana,Arial,Helvetica;font-size:13.3333px" target="_blank">JEP 466: Class-File API (Second Preview)</a></div><div>Template: 2.0</div><div><br></div><div>Summary</div><div>-------</div><div><br clear="all"><div>Provide a standard API for parsing and semantic analysis of Java methods.</div><div><br></div><div>Goals</div><div>-----</div><div><br></div><div>* Provide an API to convert the bytecode of a Java method into an expression tree (AST) suitable for semantic analysis in runtime.</div><div><br></div><div>Non-Goals</div><div>---------</div><div><br></div><div>* It is not a goal to obsolete existing libraries or Class-File API. Rather, the intention is to provide a higher-level abstraction.</div><div>* It is not a goal to support all the possible language constructs. Non logical instructions, such as try/catch, etc are out of scope.</div><div><br></div><div>Motivation</div><div>----------</div><div><br></div><div>Programs usually go beyond the boundaries of a single execution environment and delegate part of their logic to external servers such as databases.</div><div>Even if executed within the same environment (process), there might be engines that do not understand Java bytecode and require input in another language, DSL or a configuration file.</div><div>In a common Java program, a lot of configuration is supplied via external files or spread in annotations, making them hard to find and maintain.</div><div><br></div><div><div>While Java and other ecosystem languages excel in expressing logic or instructions in general, this <i>_information_</i> is extremely hard to get and process in runtime. This makes it practically impossible to use Java to _<i>express the logic_</i>, but _<i>delegate_</i> its execution to an external engine, i.e. convert Java bytecode to SQL, another DSL or provide a "fluent configuration".</div></div><div><br></div><div>Consider the following use cases:</div><div>1. SQL execution</div><div>```java</div><div><div>JPA.SQL((Person p) -> {</div><div> // intended for execution inside an SQL database</div><div> SELECT(p);</div><div> FROM(p);</div><div> WHERE(p.getName() == name);</div><div>});</div></div><div>```</div><div><br></div><div>2. DSL creation</div><div><div>```java</div><div>QueryBuilder<Restaurant> builder = Mongo.queryBuilder(Restaurant.class);</div><div><br></div><div>// the following lambda is converted to Bson</div><div>Bson filter = builder.filter(r -> r.getStars() >= 2 && r.getStars() < 5</div><div> && r.getCategories().contains("Bakery"));</div></div><div>```</div><div><br></div><div>2.1 What are the options today?</div><div>Let's take an example of the new Jakarta API for NoSQL databases: <a href="https://www.jnosql.org/javadoc/jakarta.nosql.core/jakarta/nosql/Template.html" target="_blank">https://www.jnosql.org/javadoc/jakarta.nosql.core/jakarta/nosql/Template.html</a></div><div><br></div><div>An example of a select method looks like this:</div><div>```java</div><div><div>List<Book> books = template.select(Book.class)</div><div> .where("author")</div><div> .eq("Joshua Bloch")</div><div> .and("edition")</div><div> .gt(3)</div><div> .results();</div></div><div>```</div><div><br></div><div>But could be this:</div><div>```java</div><div>var author = "Joshua Bloch";</div><div>var edition = 3;</div><div><br></div><div>// Under the hood use the Expression Trees API to parse the lambda</div><div><div>List<Book> books = template.select((Book b) -> b.getAuthor() == author && b.getEdition() == edition).results();</div></div><div>```</div><div><br></div><div>3. Fluent Configuration</div><div>```java</div><div><div>// Configure DateOfBirth column</div><div>modelBuilder.entity<Student>()</div><div> .property(p::getDateOfBirth) // use Expression Trees API to get the property name</div><div> .hasColumnName("DOB")</div><div> .hasColumnOrder(2)</div><div> .hasColumnType("datetime2");</div></div><div>```</div><div><br></div><div>Description</div><div>------------</div><div><br></div><div>### The following design goals and principles used for the Expression Trees API:</div><div>* The expression tree can be created in runtime by passing a lambda instance or java.lang.reflection.Method instance with the corresponding class instance, if the passed method is an instance method.</div><div>* The expression tree is a logical/semantic representation of a method body, including the method parameters. Possible node types are not tied to Java language and able to express a construct in any imperative language compiled to Java bytecode.</div><div>* To be successfully parsed, the method must not contain any non-supported constructs; otherwise a runtime exception is thrown.</div><div>* Methods accepting Lambdas intended for parsing can be marked with an annotation, thus tools and IDEs can have a fair ability to identify lambdas with non-supported constructs.</div><div>* The expression tree is immutable. This facilitates reliable sharing when a tree is being analyzed or transformed.</div><div><br></div><div>### Nodes and Expression Types</div><div><br></div><div>Nodes are Java classes. Each node may have a different Expression Type. For example, `BinaryExpression` node might have type `Add` or `Divide`.</div><div><br></div><div>#### Nodes</div><div>```</div><div> - abstract Expression - provides the base class from which the classes that describe expression tree</div><div> nodes are derived. It also contains static factory methods to create the</div><div> various node types.</div><div> - ConstantExpression extends Expression - Describes an expression that has a constant value.</div><div> - UnaryExpression extends Expression - Describes an expression that has a unary operator.</div><div> - BinaryExpression extends UnaryExpression - Describes an expression that has a binary operator.</div><div> - ParameterExpression extends Expression - Describes an indexed argument or parameter expression.</div><div> - abstract InvocableExpression extends Expression - Describes an expression that can be invoked by applying</div><div> 0 or more arguments and might return a result.</div><div> - MemberExpression extends InvocableExpression - Describes accessing a field or method.</div><div> - LambdaExpression<F> extends InvocableExpression - Describes a lambda expression.</div><div> Captures a block of code that is equivalent to a method body.</div><div><div> - DelegateExpression extends InvocableExpression - Describes a higher order construct, where InvocableExpression</div><div> is chained.</div></div><div> - InvocationExpression extends Expression - Describes an expression that applies a list of argument expressions</div><div> to an InvocableExpression</div><div> - BlockExpression extends Expression - Describes a sequence of expressions</div><div> - NewArrayInitExpression extends Expression - Describes a one-dimensional array and initialising it from a list of elements.</div><div>```</div><div><br></div><div>#### Expression Types</div><div>```</div><div><div> - Add // A node that represents arithmetic addition.</div><div> - BitwiseAnd // A node that represents a bitwise AND operation.</div><div> - LogicalAnd // A node that represents a logical AND operation.</div><div> - ArrayIndex // A node that represents indexing into an array.</div><div> - ArrayLength // A node that represents getting the length of an array.</div><div> - Coalesce // A node that represents a null coalescing operation.</div><div> - Conditional // A node that represents a conditional operation.</div><div> - Constant // A node that represents an expression that has a constant value.</div><div> - Convert // A node that represents a cast or conversion operation.</div><div> - Divide // A node that represents arithmetic division.</div><div> - Equal // A node that represents an equality comparison.</div><div> - ExclusiveOr // A node that represents a bitwise XOR operation.</div><div> - GreaterThan // A node that represents a "greater than" numeric comparison.</div><div> - GreaterThanOrEqual // A node that represents a "greater than or equal" numeric comparison.</div><div> - Invoke // A node that represents application of a list of argument expressions to an InvocableExpression.</div><div> - IsNull // A node that represents a null test.</div><div> - IsNonNull // A node that represents a non null test.</div><div> - Lambda // A node that represents a lambda expression.</div><div> - Delegate // A node that represents a lambda chain expression.</div><div> - LeftShift // A node that represents a bitwise left-shift operation.</div><div> - LessThan // A node that represents a "less than" numeric comparison.</div><div> - LessThanOrEqual // A node that represents a "less than or equal" numeric comparison.</div><div> - FieldAccess // A node that represents reading from a field.</div><div> - MethodAccess // A node that represents a method call.</div><div> - Modulo // A node that represents an arithmetic remainder operation.</div><div> - Multiply // A node that represents arithmetic multiplication.</div><div> - Negate // A node that represents an arithmetic negation operation.</div><div> - New // A node that represents calling a constructor to create a new object.</div><div> - NewArrayInit // An operation that creates a new one-dimensional array and initializes it from a list of elements.</div><div> - BitwiseNot // A node that represents a bitwise complement operation.</div><div> - LogicalNot // A node that represents a logical NOT operation.</div><div> - NotEqual // A node that represents an inequality comparison.</div><div> - BitwiseOr // A node that represents a bitwise OR operation.</div><div> - LogicalOr // A node that represents a logical OR operation.</div><div> - Parameter // A node that represents a parameter index defined in the context of the expression.</div><div> - RightShift // A node that represents a bitwise right-shift operation.</div><div> - Subtract // A node that represents arithmetic subtraction.</div><div> - InstanceOf // A node that represents a type test.</div><div> - Block // A node that contains a sequence of other nodes.</div></div><div>```</div><div><br></div><div>### Entry Point<br></div><div><br></div><div>`LambdaExpression` will introduce a static method that returns a parsed Expression Tree:</div><div><br></div><div>```java</div><div>public static <T> LambdaExpression<T> parse(T lambda) {...}<br></div><div>```</div><div><br></div><div>### Analysis and transformation</div><div><br></div><div>There is a builtin support for a visitor pattern via `ExpressionVisitor` interface that defines `visit` methods for all the non-abstract node classes.</div><div>In addition, there is an `abstract SimpleExpressionVisitor implements ExpressionVisitor`. By default, it performs a recursive traversal of expressions and their sub-expressions.</div><div>If none is modified an original expression is returned; if any sub-expression is modified, then a new expression is created, recursively.</div><div><br></div><div>With this a member expression transformer might look like this:</div><div><br></div><div>```java</div><div><div>val transformer = new SimpleExpressionVisitor() {</div><div> @Override</div><div> public Expression visit(MemberExpression e) {</div><div> ...</div><div> return <transformed expression></div><div> }</div><div>};</div></div><div>```</div><div><br></div><div>Let's consider a hypothetical fluent builder that needs to extract property name:</div><div>```java</div><div><div>Fluent<Customer> f = new Fluent<Customer>();</div><div>f.property(Customer::getData)...;</div></div><div>```</div><div><br></div><div>To extract the Member, the user will need to override a single `visit()` method. Like the following:</div><div><br></div><div>```java</div><div><div>class MemberExtractor extends SimpleExpressionVisitor {</div><div> private MemberExpression memberExpression;</div><div><br></div><div> @Override</div><div> public Expression visit(MemberExpression e) {</div><div> memberExpression = e;</div><div> return e;</div><div> }</div><div>}</div></div><div>```</div><div><br></div><div>A more complete implementation for the Fluent class above can be <a href="https://github.com/streamx-co/ExTree/blob/master/src/test/java/co/streamx/fluent/extree/Fluent.java" target="_blank">found here</a>.</div><div><br></div><div>### Other Java ecosystem languages and "special" methods</div><div><br></div><div>Some constructs do not have a direct translation into the Java byte code and are implemented using a runtime library method call. The runtime library developer might be interested in translating a method call expression into a different expression.</div><div>Consider the following code:</div><div>```java</div><div><div>var b1 = new BigDecimal(67891);</div><div>var b2 = new BigDecimal(12346);</div><div>if (b1.compareTo(b2) == 0) {</div></div><div>...</div><div>```</div><div><br></div><div>There should be a possibility for a library developer to transform an expression calling to `compareTo()` method and then comparing to zero into a logical comparizon expression between the two decimals.</div><div>Currently the mechanism to achieve this is not specified in this draft, but eventually must be introduced in this or a follow-up JEP.</div><div><br></div><div>### Practical usability / Existing art</div><div><br></div><div>To be successful in achieving the JEP goals, there is a need for comprehensive POC projects.</div><div><br></div><div>I have created 3 prototype projects:</div><div>1. <a href="https://github.com/streamx-co/ExTree" target="_blank">ExTree</a>. The project prototypes this JEP on top of ASM.</div><div>2. <a href="https://github.com/streamx-co/FluentJPA" target="_blank">FluentJPA</a>. The project uses Java to write SQL and integrates with JPA.</div><div>3. <a href="https://github.com/streamx-co/FluentMongo" target="_blank">FluentMongo</a>. The project uses Java to write Mongo queries i.e. Mongo BSON documents.</div><div><br></div><div>These projects helped to shape this API.</div><div><br></div><div>Testing</div><div>----</div><div><br></div><div>TBD</div><div><br></div><div>Alternatives</div><div>-----</div><div><br></div><div>An obvious alternative is to not integrate this into JRE and keep it as a separate project.</div><div>Being considered low level and possibly "fragile" technology, it will be hardly accepted into the mainstream projects where it might serve the large Java community.</div><div><br></div><div><div>Dependencies</div><div>----</div><div>JEP 466</div></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr">Regards,<br>Konstantin Triger</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div>