compose MemorySegments

[Maurizio Cimadamore]

On 11/06/2021 15:47, Chris Vest wrote:
>
> On Fri, 11 Jun 2021 at 15:53, Pedro Lamarão 
> <pedro.lamarao at prodist.com.br <mailto:pedro.lamarao at prodist.com.br>> 
> wrote:
>
>     Em sex., 11 de jun. de 2021 às 00:17, leerho <leerho at gmail.com
>     <mailto:leerho at gmail.com>> escreveu:
>
>     > This is an interesting problem and a difficult one to do efficiently
>     > without copies or additional allocations. I think the OS does
>     similar
>     > tricks when it maps virtual memory addresses into CPU caches or main
>     > memory.  To the user's program it appears as a contiguous memory
>     space but
>     > in reality could be broken into many separate physical chunks. 
>     I'm sure
>     > there are some ways one could do this in C++, but probably not
>     as efficient
>     > as the OS could do it.
>     >
>
>     Virtual memory is "accelerated" by the hardware's MMU;
>     that would be a very hard performance target to replicate in
>     userspace.
>
> The tricks we'd need to play (if we can assume aligned, whole-page 
> chunks, anyway) are available to us in user-space via the mmap(2) 
> system call.
> The RUMA paper shows how you can build dynamic data structures on top 
> of this:
> http://www.vldb.org/pvldb/vol9/p768-schuhknecht.pdf 
> <https://urldefense.com/v3/__http://www.vldb.org/pvldb/vol9/p768-schuhknecht.pdf__;!!GqivPVa7Brio!JJL0aXTaH7gXH__nL02lBeAZEZ43ONrEV4SM6cUU_zqPi4gGL6Ba2WYhCcH-e35TksWN5t8$>
>
Thanks for sharing - very interesting application of mmap indeed.

I presume it should be possible to build something on top of the memory 
access API and linker API which takes two - or more segments (assuming 
each segment is a whole-page chunks), and return a _new_ segment with 
new size, after having rewired the pages of the various segments so that 
they end up in contiguous virtual addresses.

This should effectively create a segment which can probably support the 
full segment API (including bulk copy, vectorized mismatch and what's not).

Maurizio

>
>     -- 
>     Pedro Lamarão
>     https://www.prodist.com.br
>     <https://urldefense.com/v3/__https://www.prodist.com.br__;!!GqivPVa7Brio!JJL0aXTaH7gXH__nL02lBeAZEZ43ONrEV4SM6cUU_zqPi4gGL6Ba2WYhCcH-e35TMy2Xz04$>
>     Securing Critical Systems
>     Tel: +55 11 4380-6585
>
>     Antes de imprimir esta mensagem e seus anexos, certifique-se que seja
>     realmente necessário.
>     Proteger o meio ambiente é nosso dever.
>     Before printing this e-mail or attachments, be sure it is necessary.
>     It is in our hands to protect the environment.
>