Java 17 bug (?) on Windows with MemorySegment.asSlice and offset =

[Maurizio Cimadamore]

I suppose you mean using the linker API to call mmap, and then create a 
native segment out of the obtained address?


I put together a gist with the code for this here:


https://gist.github.com/mcimadamore/128ee904157bb6c729a10596e69edffd


I'm not sure what you refer to when you say "prepare a PR" - in the 
sense that for the average Java developers, using mapped memory segments 
obtained through FileChannel is gonna be good enough - whereas experts 
might want to use mmap (or Windows equivalent) directly, using the 
linker API - but in that case I think experts will want to have full 
access to underlying functionalities, rather than having to be bound to 
a solution that is the MCD between what can be supported across the 
various platforms (which is what FileChannel::map does).


Maurizio


On 12/05/2022 16:00, Radosław Smogura wrote:
> Hi all,
>
> Very nice findings!
>
> I got bit crazy idea to use mmap directly, with NICL, instead of using 
> existing solution ��
>
> What if I would prepare such PR, or it's too early for such move?
>
> Kind regards,
> Radoslaw Smogura
> ------------------------------------------------------------------------
> *From:* panama-dev <panama-dev-retn at openjdk.java.net> on behalf of 
> Maurizio Cimadamore <maurizio.cimadamore at oracle.com>
> *Sent:* Thursday, May 12, 2022 16:05
> *To:* Uwe Schindler <uschindler at apache.org>; erel at eth.gl 
> <erel at eth.gl>; panama-dev at openjdk.java.net <panama-dev at openjdk.java.net>
> *Subject:* Re: Java 17 bug (?) on Windows with MemorySegment.asSlice 
> and offset =
> Bug is here:
> https://bugs.openjdk.java.net/browse/JDK-8286637
>
> Maurizio
>
> On 12/05/2022 15:01, Uwe Schindler wrote:
> > Hi Maurizio,
> >
> >> mapAddress = MapViewOfFile(
> >>           mapping,             /* Handle of file mapping object */
> >>           mapAccess,           /* Read and write access */
> >>           highOffset,          /* High word of offset */
> >>           lowOffset,           /* Low word of offset */
> >>           (DWORD)len);         /* Number of bytes to map */ <------
> >>
> >> ```
> >>
> >> That cast seems questionable, as DWORD is 32-bit, so casting an 8-byte
> >> value bigger than 2^32 into that will surely fail.
> >>
> >> Of course, this was never an issue with ByteBuffer, as you could never
> >> create such a big memory mapped region. But with memory segments this
> >> becomes problematic.
> >>
> >> Fixing the cast to use SIZE_T instead solves the issue you are seeing.
> > Do you have a PR / bug report for this already? That's actually a 
> good thing that Erel found that problem! This reminds me about the 
> other issue I found with ignoring the offset on windows (exactly the 
> code you also suspected to be buggy; the code where the rounding of 
> base address was applied at two different places). Luckily those bugs 
> were quickly found by unit tests on Lucene's side (in JDK 16 already).
> > Maurizio
> > Lucene is testing with MMapDirectory, too - but not on Windows with 
> huge indexes like 8 Gigabytes. Actually for fragmentation reasons we 
> still split larger files into chunks, but those chunk sizes are always 
> multiplies of a power of two - good luck! And during testing we create 
> much smaller chunks to test the chunking, so we have no huge mappings 
> during testing. One note about your workaround: The power of 2 
> workaround does not work well, if you have an unaligned start offset 
> (like mapping at offset 7 of file for 2 gigabytes), because then the 
> total mapped size is larger (a few bytes added to get the file offset 
> aligned).
> >
> > I am glad that we did not yet have release Lucene with support for 
> mapped memory segments! �� My problem with those bugs is that you 
> cannot test it easily with our unit/integration tests in our testing 
> infrastructure, because building indexes of that size is not an 
> everyday task.
> >
> > Uwe
> >