more panama notes - ffmpeg

[Michael Zucchi]

Hi guys,

As a bit of a weekend project i started prototyping work on porting 
jjmpeg to panama.  jjmpeg is a 'lean' binding that only binds what it 
needs to, hides most of the messy details and tries to present a 'sane' 
interface.  As FFmpeg wasn't really designed as a 'public' api there is 
a lot that isn't worth exposing or just shouldn't be.  This vastly 
reduces the api size by ignoring all the stuff that isn't needed as a 
user of the library although it also means important parts that I don't 
use may not be available.

I think i've mentioned before that there are now other alternatives for 
reading/writing video for java but when I started it there was really 
just 'xuggler' as a cross-platform solution: which was gigantic, hard to 
compile, miserable to use.  Things like media players aren't any use for 
it's use-case which is accessing the pixel bits and not for playing movies.

In this iteration for panama i'm using an application-specific generator 
which directly creates the public classes so there is no extra glue 
required to call some low-level interface.  It is controlled by a 
meta-data file.  Based on that it is mostly automated with a sprinkling 
of hand-rolled cases where the generator isn't sufficient or not worth 
extending (it's messy enough).  I've managed to avoid the need for a 
Pointer<> or Array<> type so far by designing around those cases.  Most 
of the complexity is with the setup/config stage which isn't performance 
critical - at the end of the day you just get a buffer of pixel or sound 
samples and that will just be a bytebuffer or memorysegment and that's 
the only time performance matters.

I don't use jextract mostly because I want far more control in what gets 
created, it also doesn't work yet[1].

It's still work in progress, maybe 60% coverage of prior versions.  It's 
mostly been straightforward but here's some notes about the story so 
far.  The main panama specific issue is with ByteBuffer vs <T extends 
Buffer>, and maybe varargs upcalls.

unsafe

    I just have to use foriegnunsafe /everywhere/. Some values created
    need to be freeable by libav*, so you need to use av_malloc() for
    all allocations, and that generally requires unsafe to be able to
    initialise them.

    When you access the pixel or sample data it is via FFmpeg managed
    memory buffers, so unsafe is required there.

<T extends Buffer> vs ByteBuffer

    In JNI any type of direct buffer (ByteBuffer, IntBuffer, etc) can be
    accessed using GetDirectBufferAddress and GetDirectBufferCapacity. 
    But the only possible way to pass a buffer address to C with panama
    is with a ByteBuffer via MemorySegment.of*().

    jjmpeg has an implementation of the JavaFX PixelReader[2] (and
    Writer) interface which allows one to directly write libswscale
    processed data into a WritableImage.  The problem is the read pixels
    call takes a <T extends Buffer> and not a ByteBuffer.

    In practice - until openjfx 13 - it always uses the Buffer method
    and that is always passed a ByteBuffer.  But openjfx adds changes
    that allow for an IntBuffer to be used instead.  A workaround for
    JavaFX 13+ is just to ensure the WritableImage is backed by the
    correct type, amongst other requirements.

    I don't know of any other similar cases in the jdk but i haven't looked.

vector access

    libav* exposes various arrays as a length + pointer to pointers. 
    Since this is external memory it needs to be wrapped in an unsafe
    segment before dereferencing the pointers elements, every time.  At
    least with the current design.

    typedef struct AVFormatContext { ... unsigned int nb_streams;
    AVStream **streams; ... }

    Becomes:

    public final static VarHandle field$nb_streams = MemoryHandles.withOffset(i32Handle, 44);
    public final static VarHandle field$streams = MemoryHandles.withOffset(addrHandle, 48);
    public AVStream getStream(int index) {
    	int length = (int)field$nb_streams.get(addr());
    	if (Integer.compareUnsigned(index, length) >= 0) { throw new ArrayIndexOutOfBoundsException(); }
    	MemoryAddress base = Memory.ofNative((MemoryAddress)field$streams.get(addr()), length * 8).baseAddress();
    	return resolve((MemoryAddress)addrVHandle.get(base, (long)index), AVStream::new);
    }

    This is absolutely not a performance critical path so who cares to
    be honest.  Memory.ofNative() just calls
    ForiegnUnsafe.ofNativeUnchecked().

sentinel-terminated lists

    In a couple of places the api exposes arrays which aren't defined by
    a length but by a terminating sentinel value.  Since this is
    external memory the process to determine the length requires an
    extra step:

    MemoryAddress list = Memory.ofNative(data,
    Integer.MAX_VALUE).baseAddress(); int len = 0; while
    (read_value(list, len) != sentinel) len++; allocate array copy len
    elements

    I guess the api could expose an iterator or stream but this adds
    temporal issues for non-const data.

    For strings specifically I now just call libc:strlen() which makes
    the code quite simple:

    String str = new String(Memory.ofNative(cstr,
    strlen(cstr)).toByteArray())

structure fields

    I generate sparse MemoryLayouts that only cover the items i'm using
    by just padding over the private parts.  Actually for now I
    generally don't use the layout and just use the offsets from the c
    compiler to directly instantiate the variable handles, but they're
    there incase I change my mind or find another use for them.  They
    are needed for FunctionDescriptions for down calls which take
    pass-by-value structs but that isn't common.

    Many of these fields may be able to be removed entirely by using
    some generic "get/set" calls from libavutil/opts.h, but I need to
    delve further.

These rest are just informational, these aren't things any sane library 
does internally let alone exposes it.

AVDictionary

    This is just a Map<String,String> but implemented so poorly it
    beggars belief (I think it's legacy from the libav split). 
    Internally it's an array of pointer pairs with a length.  But /every
    time you change it's key set/, the pointer array is reallocated and
    copied.  This requires the calls to take an AVDictionary **.  It
    also has a bizarrely complicated insert and query interfaces that
    let you control all sorts of things like treating the arguments as
    constants to be copied, or allocated memory to steal.  The array and
    all strings must be allocated using av_malloc() as it may freely
    call av_free() whenever it wants.

    Solution here is what i did with jjmpeg/jni - the Java class is just
    a plain TreeMap<> and I have code which converts between that and
    the (private) C structure.

av_log_set_callback

    void av_log_set_callback(void (*callback)(void*, int, const char*,
    va_list));

    Rather than a formatted string the callback gets the parameters for
    vprintf etc.  The va_list, which gets turned into (I presume) some
    platform specific struct.

    (jextract output on just the function above)

    private static final FunctionDescriptor
    av_log_set_callback$callback$DESC = FunctionDescriptor.ofVoid(false,
    MemoryLayouts.SysV.C_POINTER, MemoryLayouts.SysV.C_INT,
    MemoryLayouts.SysV.C_POINTER, MemoryLayout.ofSequence(1,
    MemoryLayout.ofStruct(
    MemoryLayouts.SysV.C_INT.withName("gp_offset"),
    MemoryLayouts.SysV.C_INT.withName("fp_offset"),
    MemoryLayouts.SysV.C_POINTER.withName("overflow_arg_area"),
    MemoryLayouts.SysV.C_POINTER.withName("reg_save_area")
    ).withName("__va_list_tag")) );

    In an earlier jjmpeg i just formatted the string before passing it
    back to Java.  I don't see any sane way to handle it here.

opt.h

    This is another way to access members and provides generic
    getter/setters.  But it's quite complex and the api isn't great and
    one has to delve into the source of each 'object' to determine what
    the symbolic name is for every field as they might not match.  But
    it might be able to be used to remove most of the structure peeking
    and poking, at some size+perf cost.

Cheers,
  Z



[2] 
https://openjfx.io/javadoc/13/javafx.graphics/javafx/scene/image/PixelReader.html

[1]

notzed at shitzone:~/src/panamaz/test-ffmpeg$ cat ffmpeg-api.h #include 
<libavformat/avformat.h> #include <libavcodec/avcodec.h> #include 
<libswscale/swscale.h> #include <libavutil/imgutils.h> 
notzed at shitzone:~/src/panamaz/test-ffmpeg$ 
/opt/panama-jextract/bin/jextract -I /usr/lib64/clang/9.0.1/include 
ffmpeg-api.h WARNING: Using incubator modules: jdk.incubator.foreign, 
jdk.incubator.jextract Exception in thread "main" 
java.util.ConcurrentModificationException at 
java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1226) at 
jdk.incubator.jextract/jdk.internal.jextract.impl.TreeMaker.checkCache(TreeMaker.java:57) 
at 
jdk.incubator.jextract/jdk.internal.jextract.impl.TreeMaker.createVar(TreeMaker.java:222) 
at 
jdk.incubator.jextract/jdk.internal.jextract.impl.TreeMaker.createTree(TreeMaker.java:82) 
at 
jdk.incubator.jextract/jdk.internal.jextract.impl.TreeMaker.createFunction(TreeMaker.java:142) 
at 
jdk.incubator.jextract/jdk.internal.jextract.impl.TreeMaker.createTree(TreeMaker.java:84) 
at 
jdk.incubator.jextract/jdk.internal.jextract.impl.Parser.lambda$parse$2(Parser.java:91) 
at 
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) 
at 
java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) 
at 
jdk.incubator.jextract/jdk.internal.jextract.impl.Parser.parse(Parser.java:72) 
at 
jdk.incubator.jextract/jdk.internal.jextract.impl.JextractTaskImpl.parse(JextractTaskImpl.java:59) 
at 
jdk.incubator.jextract/jdk.internal.jextract.impl.JextractTaskImpl.parse(JextractTaskImpl.java:53) 
at 
jdk.incubator.jextract/jdk.incubator.jextract.tool.Main.run(Main.java:187) 
at 
jdk.incubator.jextract/jdk.incubator.jextract.tool.Main.main(Main.java:97)