Hermetic Java project meeting

[Magnus Ihse Bursie]

I will not be able to participate in the meeting today.

Let me report a bit on my work this week.

I have made a proof of concept branch which can properly compile static 
java on macos with clang and linux with gcc. (With "properly" I mean 
hiding non-exported symbols). I still have problems replicating the 
build with clang on linux. I suspect my linux workstation has a broken 
installation of clang. That machine has grown more erratic over time, 
and I need to reinstall it, but I'm procrastinating spending the time 
doing that... So for now I'm just skipping the clang on linux part.

I have also made great progress on windows. Julian's hint about objcopy 
working fine on COFF, and being present in cygwin, got me to realize 
that this was in effect a possible way forward. My PoC currently manages 
to extract a list of exported symbols from the static libs using 
dumpbin. This is necessary since --localize-hidden does not work on COFF 
(the concept of "hidden" symbols is an ELF-only concept), and instead we 
need to use the --keep-global-symbols option, which (despite the name) 
converts all symbols in the given list to global and all other to local. 
I am currently working actively with getting the next steps done in this 
PoC.

I have initiated talks with SAP, and they are interested in helping out 
getting static linking working on AIX (given that it is possible to do 
with not too much effort.)

I have also tried to extract all the changes (and only the changes) 
related to static build from the hermetic-java-runtime branch (ignoring 
the JavaHome/resource loading changes), to see if I could get something 
like StaticLink.gmk in mainline. I thought I was doing quite fine, but 
after a while I realized my testing was botched since the launcher had 
actually loaded the libraries dynamically instead, even though they were 
statically linked. :-( I am currently trying to bisect my way thought my 
repo to understand where things went wrong.

This problem was exaggerated by the fact that we cannot build *only* 
static libs, so the dynamic ones where there alongside to be 
(improperly) picked up. I might want to spend some time on fixing this 
first. That will help both with speeding up the build/test cycle for 
static builds, and help avoid that kind of issue repeating. That will 
require some more refactoring in the core build/link code though.

Experimenting with the static launcher on linux/gcc and macos made me 
realize that we will need to know the set of external libraries needed 
by each individual JDK library. When building a dynamic library, this 
knowledge (e.g. -liconv -framework Application) is encoded into the 
lib*.so file by the linker. But not so for a static library. Instead, we 
need to store this information for each JDK library, and then in the 
end, when we want to pick up all static libraries and link them together 
to form the "javastatic" executable, we need to pass this set of 
external libraries to the linker.

This was done haphazardly in StaticLink.gmk in the hermetic-java-runtime 
branch, where an arbitrary subset of external libraries were hard-coded. 
Before integration in mainline can be possible, this information needs 
to be collected correctly and automatically for all included JDK 
libraries. Fortunately, it is not likely to be too hard. I basically 
just need to store the information from the LIBS provided to the 
NativeCompilation, and pick that up for all static libraries we include 
in the static launcher. (A complication is that we need to de-duplicate 
the list, and that some libraries are specified using two words, like 
"-framework Application" on macos, so it will take some care getting it 
right.)

I have also been thinking about possible ways that we can share compiled 
obj files between static and dynamic libraries, even if we cannot do it 
fully. Most files do not need the STATIC_BUILD define and will thus be 
identical for both static and dynamic builds. It might be possible to 
just hard-code the exact files that needs to be different. It's ugly, 
and I still would like to make sure we press forward with the spec 
changes to JNI/JVMTI, but it would work as a stop-gap measure.

/Magnus