Reading binary data

[Maurizio Cimadamore]

On 04/12/2019 16:14, Samuel Audet wrote:
> Hi, Maurizio,
>
> On 11/23/19 8:25 PM, Maurizio Cimadamore wrote:
>> Hi Samuel
>>
>> On 23/11/2019 05:11, Samuel Audet wrote:
>>> Maurizio,
>>>
>>> To be clear, I'm not advocating for the status quo, far from it. 
>>> Since you brought up Jigsaw, let's take as an example. In its 
>>> current incarnation, JPMS has been quite literally rammed through 
>>> over considerable opposition from the Java community:
>>> https://www.theregister.co.uk/2017/05/12/oracle_loses_jpms_vote/
>>
>> This is yesterday news. And this pretty summarizes your approach 
>> towards this project as a whole: pick up a piece of negative 
>> feedback, infer some general consequences from it, rinse and repeat. 
>> How much were you involved in the design of the module system? Did 
>> you contribute _anything_ on jigsaw-dev (which is where this 
>> discussion should happen, BTW) ? I'm guessing not. So, please, leave 
>> comments on whether JPMS is good or bad to people who actually have a 
>> stake in it.
>>
>> What I was actually trying to do was to position some of the things 
>> we're exploring for native access in the wider context of what has 
>> happened in the Java platform over the last few years, and the 
>> general desire of making the Java platform safer, and its 
>> configuration more predictable. Whether _you_ like it or not, that's 
>> where the platform is going.
>
> It's not a question of whether I like it or not, it's about having the 
> tools that we need. Recently, both Ioannis and Ty have been trying to 
> make you understand the same thing that I've been trying to make you 
> understand about safety, and you're not taking them seriously either. 
> How many people is it going to take for you to understand the 
> importance of this?
I think you are just mixing (as usual) pears with apples. But I will let 
Ty and Ioannis to speak for themselves, rather than try to put words 
into their mouths.
>
> We absolutely need unsafe access to memory regardless of the opinions 
> or desires of anyone. Preventing us from having more usable APIs in 
> that sense will continue to hamper us in our work, and will only 
> weaken the Java platform vs other platforms when it comes to software 
> that depends on, for example, accelerators like GPUs. If that's not 
> where OpenJDK intends to take the Java platform **in priority**, 
> that's perfectly fine, but please make it clear so that we can stop 
> wasting time arguing like this.

Who is *We*, *us* ? As for wasting time arguing - I'd very much like to 
stop (as you do). So - maybe stop arguing?

You seem to be determined in having me admitting that some of the 
important use cases *you* (again, don't know who *you* is) have in mind 
is not on our radar - and you infer that from the fact that we'd like to 
have a clear separation between safe parts of the API and less safe 
parts of the API. That is simply an incorrect inference, which is more 
similar to FUD than to a real argument.

>
>
>>> Instead of establishing in priority a weak module system like that 
>>> or even developing something like jextract, have you considered that 
>>> the Java community at large might have prioritized a "safer JNI"? I 
>>> don't see that kind of discussion of priorities happening anywhere 
>>> at OpenJDK! You have a fully top-down approach. From what I 
>>> understand of our discussions, you do not seem to believe this is a 
>>> problem, but in fact it very much /is/ a problem.
>>>
>> And here we are again, mixing things that have very little to do with 
>> each other. What has JPMS have to do with jextract? But, more 
>> importantly, why do you keep assuming that nobody cares about 
>> jextract? There might be other solutions available for doing what 
>> jextract does, but how are you so sure that no Java developer cares 
>> about that? The feedback on this very mailing list (from people who 
>> actually tried things), as well as some of the comments/questions on 
>> social media (most of which were repeatedly shut down by _you_) seem 
>> to suggest quite the contrary. How do we establish our priorities? By 
>> going to conferences, talking to real developers and customers, 
>> taking in feedback, be it positive, or negative. How do _you_ 
>> establish what's best for the Java ecosystem as a whole? This is a 
>> question that, after a year or so, I'm still struggling to answer.
>
> Well, think about it this way. If OpenJDK prioritized unsafe access to 
> memory and low-level APIs similar to JNA, JNR, or Substrate VM's, or 
> anything else that is actually usable without a parser like jextract, 
> then the **whole** community would be able to work on tools on top of 
> those low-level APIs that you provide.

The fact that we have a JEP currently in candidate state for the memory 
access API and we don't have one for jextract rings any bell? Does the 
fact that we have foreign-memaccess, and foreign-abi, but do not have 
(yet) a foreign-jextract ring some other bell?

But, even putting all that aside (and that's a lot!), do we have to ask 
your permissions before looking into solutions which will simplify the 
work required for extracting native APIs (either by using an 
out-of-the-box tool, or by building something more complex based on the 
API, which is based on the years of experience we matured working with 
libclang) ? I hope not!

>
> Right now, all those low-level APIs are restricted, internal, not even 
> part of Java SE, 

false, false, false, again

The fact that they are not part of Java SE is because we have to 
distribute using the incubating mechanism - so they are _for now_ in a 
separate module. From other discussions, it is clear that the amount of 
restricted operations is actually pretty small, compared to the whole 
surface of the API. I don't know where you get your facts from.

> let alone Android, and they're not even yet faster than JNI or 
> sun.misc.Unsafe,

Well, as for downcall, there's not much to optimize (beside dropping 
native transitions, which we will look into). Upcalls are a different 
beast and what we have in linkToNative seems to offer serious boosts 
over JNI.

As for Unsafe, unsafe access (when it works well) boils down to a single 
'mov' instruction - how can we possibly go faster than that?

The challenge, for the memory access API is to provide a _safe_ API 
which gives same performances as Unsafe - which means you don't have to 
choose between safety and performances. You don't care about the former 
- a lot of people (including Ioannis :-) ) do.


> and actually slower for AOT compilers. Moreover, they are always 
> changing and because of this lack of stability, most of the community 
> is not able to use them extensively enough to provide (more) feedback. 
> It's not possible to do anything substantial with an API that is so 
> unstable! Although you're not willing to talk about what you might 
> have to do to support C++, I can imagine that it's going to end up 
> making even more aggressive changes, but maybe not? Not knowing what 
> your intentions are isn't helping.

I think these is a big ball of excuses as to why _you_ are not doing 
anything with the API. Luckily, in this very mailing list, I've seen 
plenty of people who, unlike you, were willing to get their hands dirty.

As for stability, the memory access API did not have any significant API 
change in months - and now it's a JEP. I'm also not aware of any 
breaking changes in the foreign API which is being distributed through 
EA. There's not a single piece of concrete evidence in anything you say. 
This fact alone is a big red herring.

>
> If those low-level bits could be stabilized first, then I'm sure that 
> the rest of the community (millions of them) would start experimenting 
> on them, including myself. That's how you work with the community: You 
> give them what they need to help you. On its own by itself, OpenJDK is 
> never ever going to be able to provide everything that customers that 
> you meet at those conferences require. Compromises need to be made.
Aren't we just doing that? Again, memory access API is now a JEP; we're 
mostly working on that and foreign-abi. How isn't this the low-level bit 
stabilization you advocate so much?
>
>
>>> I admit my attitude isn't ideal, but I'm still trying to make you 
>>> understand my point in this. Let's try to work on features that are 
>>> most useful to everyone, first. We all have limited resources, so 
>>> let's try to make the most of them, and that means working with the 
>>> community! In any case, I'll keep trying to do my best and be as 
>>> least annoying as possible. :)
>>
>> The real elephant in the room here is that we've been arguing for 
>> over an year _without you having written a single line of code_ using 
>> whatever we're doing. In reality, I don't see why there couldn't be 
>> an experimental version of JavaCPP targeting memory access VH and MH 
>> instead of JNI - then it would be good to have a discussion about 
>> what worked and what didn't. I think until you do something along 
>> those lines, not many people will really take you seriously around 
>> here, sorry.
>
> We've talked about this before. Once you're able to provide a viable 
> alternative to JNI, that runs faster for both JIT and AOT compilers, 
> whose API is somewhat stable, that allows full unsafe access somehow, 
> and that is slated to become part of Java SE, even just as an initial 
> draft of a JEP, then I will start to look at it.

Well, (I start to sound like a broken record) the memory access API is a 
JEP now; any plans to replace JavaCPP JNI-based struct access with that? 
Surely that should be much faster?

But in general your attitude of "I will touch it only when it's fully 
done" makes me think that you are essentially looking at the wrong 
mailing list. This is for people who likes to makes their hands dirty 
and play a bit with what's available - if nobody did that (and followed 
your approach), the feedback we'd get would be exactly zero, nada, nil - 
would you care to formulate how such an attitude is going to contribute 
to make whatever we are doing _better_ ?

>
> For what it's worth, I think the problem is in part that jextract has 
> been prioritized,
false - as pointed out several times above, you seem to have got it 
completely backwards. If you are speaking about 2-3 years ago - that's a 
different story, but I'd say that w/o the _experience_ (and the 
feedback) we have accumulated with those experiments, we would have 
never been able to figure out what the low-level bits would look like.
> but this tool has no chance whatsoever of making it into Java SE. 
> Doing so would require writing specs for essentially the whole of 
> C/C++ and LLVM. That's never going to happen... unless OpenJDK decides 
> to do without the JCP and JSRs entirely, at which point we're going to 
> have everyone creating forks of the JDK. Either way, something will 
> happen, and if we have to wait until all hell breaks loose like this, 
> I've made my peace with that. I did what I could to prevent it. Issues 
> will get resolved one way or another. Before we get to that point 
> though, if Panama decides to cooperate more with the community and 
> accepts that compromises need to be made, I will be here to help!

I think these arguments are beyond desperate. Time to move on.

Maurizio


> Samuel