[foreign-abi] minimizing the surface of restricted native access

[Ioannis Tsakpinis]

Hey Maurizio,

Some thoughts based on my experience as the maintainer of LWJGL:

> But, thanks to the Nothing region, we now have a way out: the
> dereference operation can be made safe, if the MemoryAddress generated
> by the VarHandle (upon read) is backed by the special Nothing region.
> This means that the user won't be able to dereference such an address,
> but this might be ok in cases where the address just needs to be written
> somewhere else, or passed to some native library.

Sounds good for opaque addresses/handles.

> MemoryAddress rebase(MemorySegment)
>
> The semantics of this method is simple: if the address is contained in
> the supplied segment, this method returns a _new_ address that is
> re-interpreted as an offset into the supplied segment.

Sounds good for managed segments.

> But what if a client really wants to dereference a _random_ pointer
> generated by a library? In this case, the client has no well-known
> segment which can be used as a rebase target, so, what can be done? This
> is where we need to veer into restricted native access territory, and
> provide some escape hatch by which an untrusted address can be made
> trusted again. There are many ways to do that; perhaps the simplest is
> to add a new method on MemoryAddress:
>
> MemoryAddress asUnchecked()
>
> which returns a new memory address which point to same location as the
> original address, but whose access is neither spatially, nor temporally
> checked.

This doesn't sound so good. :)

There are many functions/structs in LWJGL that fall into this category.
Almost always, there's additional information available that defines the
returned pointer's spatial bounds. The most common cases are:

a) The user specifies the buffer size as a parameter to the function.
b) The buffer size is returned by the function (as the return value or as a
separate output parameter).
c) The struct contains another member, which specifies the buffer's size.
d) It's a null-terminated string.

In idiomatic LWJGL code, these pointers are wrapped in NIO buffers (of the
appropriate type) with the correct capacity according to that additional
information. Accessing memory via those buffers is then perfectly safe and
subject to the standard bound checking within NIO. As an example, a very
common OpenGL function:

void *glMapBufferRange(
    GLenum target,
   GLintptr offset,
   GLsizeiptr length,
   GLbitfield access
);

It maps a buffer object (which lives in GPU memory) to host-addressable
memory. If the offset/length parameters are invalid, NULL is returned. If
not, we get a pointer that is wrapped in a ByteBuffer with capacity equal
to length.

I have not used the memory access API yet, but I guess the equivalent here
would be to create a MemorySegment that starts at the returned pointer and
ends after length bytes. That way the mapped memory would be safely
accessible, just like any other managed segment.

Having to go through the Everything segment and dropping spatial bound
checking would feel like a regression as an LWJGL user. It would also mean
that the majority of LWJGL bindings would fall under "untrusted/unsafe"
territory. Off the top of my head, functions like glMapBufferRange exist in
OpenGL, OpenGL ES, OpenCL, Vulkan, CUDA, Assimp, bgfx, every custom memory
allocator and possibly many more.

Two more notes:

1. Temporarily, the pointer returned by glMapBufferRange lives until a call
to glUnmapBuffer. I don't have a good answer for how to automatically close
the corresponding MemorySegment. It should be like: mark the MemorySegment
as closed, memory barrier (to flush any writes to the buffer and avoid
reordering), call the glUnmapBuffer function.

2. Bounds checking is so important to LWJGL, that it's literally impossible
to generate a binding, without specifying the relationships between function
parameters or struct members. We also use libclang to parse headers and
automate most of the process, but that's not enough. Every pointer
parameter or return value must then be annotated with metadata that
precisely define its bounds. In the worst case, the bounds cannot be derived
and the pointer is marked as unsafe (in that case it is mapped to an opaque
pointer, not a buffer). Thankfully, this occurs extremely rarely, in badly
designed C APIs.

> There is, of course a limitation: this model assumes that a client will
> never need/want to 'free' a random pointer obtained by calling a native
> library: since the Nothing segment is always alive (by definition), it
> cannot be closed. While this restriction will be ok in most cases, there
> will be some APIs requiring this - examples are strdup, vasprintf, which
> require the client to specifically 'free' the given address. That said,
> this problem doesn't seem too difficult to solve: for instance a client
> can request a native MethodHandle which points to the 'free' stdlib
> function, and then can call it directly on the required address! I think
> such an approach would actually be preferable to an approach where
> MemorySegment::close() secretly implies calling free() on some address -
> which might be the case now, but might not be later on (e.g. if we
> replace the allocator used internally by the memory access API).

Related to the above, there needs to be an alternative to the stdlib-backed
MemorySegment::allocateNative. We've seen this before:

ByteBuffer::allocateDirect is too expensive for short allocations. Users
allocate a big buffer, then do sub-allocations out of that buffer. They
soon end up writing a "memory allocator" in Java. That memory allocator is
bad (in at least one important dimension).

In LWJGL there are two alternatives:

1. Use a specialized memory allocator (jemalloc, rpmalloc, etc). Either
change the default, or use it where it makes sense. Again, this means
"random" pointers returned, which need to be bounded and trusted.
2. There is a "stack allocation" API. I won't expand on this before I see
your proposal (mentioned in another thread recently), but it's critically
important (both in terms of performance and of writing clean code).

> Thoughts?

I highly appreciate your efforts to make the memory access API as safe as
possible. We strive for it in LWJGL too. However, lets also address the
elephant in the room. There's going to be a way to create NULL pointers in
Panama. Users are going to pass that NULL pointer to a function parameter
that does not accept NULLs [1]. Users are going to pass all kinds of illegal
parameter values. The JVM is going to crash, a lot, and you're probably
going to be blamed for it. There are also a million different ways to crash
the JVM process once native API access becomes easy, that have nothing to
do with the bindings themselves. Writing a compute shader that does
nasty things, or even a shader that's legitimate but the GPU driver doesn't
like this week. Calling OpenGL functions in the wrong order (you can't
really validate a massive state machine on every function call to avoid the
crash).

Anyway, I'm just saying there needs to be a balance between safety and API
usability. I'm not saying giving up on security, but perfect crash-freedom
is a lost cause.

- Ioannis

[1] This is another aspect that LWJGL deals with manually after libclang
automatic-generation. Pointer parameters are explicitly annotated as
nullable, otherwise we check for NULL and throw at runtime.