<!DOCTYPE html><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body><div style="font-family: sans-serif;"><div class="markdown" style="white-space: normal;">
<p dir="auto">On 28 Nov 2022, at 3:11, Maurizio Cimadamore wrote:</p>
</div><div class="plaintext" style="white-space: normal;"><blockquote style="margin: 0 0 5px; padding-left: 5px; border-left: 2px solid #777777; color: #777777;"><p dir="auto">Hi,
<br>
as others have pointed out in this thread, while supporting C++ from jextract/Panama is not impossible, there are several obstacles to get there. The biggest one is that the C++ platform compiler plays a very important role in how things get compiled (e.g. which mangled name an overloaded member function will get, among other things). Often, high-level parsing libraries such as libclang do not offer enough information for jextract to be able to reconstruct the full picture. Even ignoring low-level details such as vtables and inline functions, when we have tried to create a small wrapper around jextract to "lower" C++ code into C we ran into several problems esp. around templates, as the clang API was not exposing information about template instantiation correctly.</p>
</blockquote></div>
<div class="markdown" style="white-space: normal;">
<p dir="auto">Thanks, Maurizio. This is a good discussion to revisit from time to time. I’d like to give some more details here about our thinking about addressing API points beyond the system ABI. Please view it as a very speculative roadmap of possible futures, not as a description of the present state of Panama technology currently released in the OpenJDK.</p>
<p dir="auto">A basic tactic that would serve many (not all) use cases would be to have jextract emit bindings for C++ API points, with arbitrarily generated names and plain-C ABI linkage. They would be coded in C++ source code as tiny functions with C linkage, one function per C++ API point. They would have to be separately compiled by the C++ compiler on each platform. The jextract tool and Panama loader would have to manage synthetic DLL assets containing the code for these.</p>
<p dir="auto">(High performance users will balk at having everything go through tiny stub functions. For many users, though, it would be helpful to remove the barriers, even if speedbumps are still present. And progress in this direction could be followed by work to remove the speedbumps, pleasing more users.)</p>
<p dir="auto">I think eventually we will try such a thing, again. (Again, since we have tried it already, which is why we know what the problems are.) As Maurizio says, there are at least two or three independent issues here:</p>
<ol>
<li>
<p dir="auto">Getting clang to cough up the details of the C++ API points to jextract. Apparently this is difficult, because while the C metadata is stable and complete, the C++ metadata is not. (Last I heard.) This seems to require some love from the core developers of clang.</p>
</li>
<li>
<p dir="auto">Getting the user to give enough guidance at jextract time to tell the system which template instances to build. (Hard problem, perhaps not possible to solve in complete generality.)</p>
</li>
<li>
<p dir="auto">Having jextract and the Panama library (and JVM) agree on how to package the synthetic DLL assets created for C++ API points, especially if they don’t have stable names in some expanded ABI.</p>
</li>
</ol>
<p dir="auto">By “expanded ABI” I mean something akin to the system C ABI, but with extra features, such as extra register usage. Some work on Panama has prototyped mechanisms for building method handles to any “roll your own” ABI; this hasn’t been productized because it is a niche functionality. For example, such mechanisms would be useful for API points that make heavy use of SIMD registers, heavier use than is contemplated by the standard ABI.</p>
<p dir="auto">An alternative to 3 and its synthetic DLL asserts would be to deliver some other form of asset, such as C++ source code, assembly code, bitcode, etc., which the JVM could load and transform on the fly to something it can hook a method handle into. I pick on synthetic DLLs because that seems the most straightforward way to go, and other projects have done similar things. But recent very cool work makes it clear there is may be role, in Panama, for load-time assembly of callable resources, perhaps as an alternative to loading synthetic DLLs created by jextract.</p>
<p dir="auto">Notice that, even if we had a stable C++ ABI to work with, all three challenges would still be plenty hard.</p>
<p dir="auto">Here is a sketch of one way to boil down C++ API points (given sufficient metadata) into plain-C API points, which I made in 2016, shortly after Panama started. It doesn’t require anybody but the C++ compiler to know about object layouts or v-tables or calling sequences:</p>
<p dir="auto"><a href="http://cr.openjdk.java.net/~jrose/panama/cppapi.cpp.txt" style="color: #3983C4;">http://cr.openjdk.java.net/~jrose/panama/cppapi.cpp.txt</a></p>
<p dir="auto">Regarding Problem 2 (user advice on templates), notice that a solution to that problem would also quickly give rise to a solution for delivering API points for C inline functions, C function-like macros, and C object-like macros. The point is that if the user is giving advice to jextract about template instances, the user can also give advice about those other constructs (perhaps with a bit more type information), and the whole mess can be funneled down into a synthetic DLL with C API points. Just imagine something like a template-instance declaration, but which names a macro or C inline function, with any required non-obvious type information. It could be spelled something like <code style="margin: 0; padding: 0 0.4em; border-radius: 3px; background-color: #F7F7F7;">__JXBIND FILE* __JXCMACRO_stdin;</code> or <code style="margin: 0; padding: 0 0.4em; border-radius: 3px; background-color: #F7F7F7;">__JXBIND int __JXCMACRO_max(int,int);</code>, and the latter could be instantiated as several overloaded bindings. A small header file, stirred into the cauldron of jextract input, would be a natural place to position such ad hoc advice from the user.</p>
<p dir="auto">I do hope the three hard challenges mentioned above will eventually be solved at some stage in Panama. Probably one at a time, but probably in the order given, although Problem 2 could be started on without delving into clang internals very deeply. Problem 1 is the big blocker, and requires clang expertise we don’t have in Panama. Perhaps someone in the community will take a run at one or more?</p>
<p dir="auto">— John</p>
<p dir="auto">P.S. I know all three problems are soluble because in the ‘90s I wrote a C++ binding generator that solved them, for a single platform at a time, for an embedded Scheme interpreter I was working on. (Instead of clang, I wrote my own C++ grammar and analyzer. That was a nightmare, but it opened up some other nice daydreams.) Like Panama, this system doubled down on dynamic generation of call sites to ABI points, as directed by metadata either extracted from header files or hand-coded PINVOKE-style by the user, and it’s not a coincidence that Panama resembles it in those ways. Making C++ subclasses dynamically from Scheme code, with lambdas for overrides, and then feeding them to unsuspecting C++ libraries, was fun and useful. It was only an internal technology at Sun; the tiny amount of public information about it can be found at <a href="http://rosehome.org/esh/" style="color: #3983C4;">http://rosehome.org/esh/</a>. I moved away from it to Java quickly.</p>
<p dir="auto">P.P.S. In view of the above facts, Samuel Audet’s simple statement that “anything outside platform ABIs for C is out of scope for Panama” appears misleadingly absolute. The C system ABIs are a sweet spot, not a limitation, and Panama can today, or may tomorrow, do lots of non-obvious tricks with ABIs that let us reach many families of API points beyond C.</p>
</div></div></body>
</html>