RFR: JDK-8200178 Remove mapfiles for JDK native libraries

Fri Mar 23 22:03:59 UTC 2018

On 2018-03-23 17:05, mandy chung wrote:
> This is a very good change and no more mapfile to maintain!!
Thank you!

>
> Please do file JBS issues for the component teams to clean up their 
> exports.
I have now filed:

https://bugs.openjdk.java.net/browse/JDK-8200191 -- for java.base
https://bugs.openjdk.java.net/browse/JDK-8200192 -- for java.desktop
https://bugs.openjdk.java.net/browse/JDK-8200193 -- for jdk.security.auth

/Magnus

>
> Mandy
>
> On 3/23/18 7:30 AM, Erik Joelsson wrote:
>> I have looked at the build changes and they look good.
>>
>> Will you file followups for each component team to look over their 
>> exported symbols, at least for the libraries with 
>> $(EXPORT_ALL_SYMBOLS)? It sure looks like there is some technical 
>> debt laying around here.
>>
>> /Erik
>>
>>
>> On 2018-03-23 06:56, Magnus Ihse Bursie wrote:
>>> With modern compilers, we can use compiler directives (such as 
>>> _attribute__((visibility("default"))), or __declspec(dllexport)) to 
>>> control symbol visibility, directly in the source code. This has 
>>> historically not been present on all compilers, so we had to resort 
>>> to using mapfiles (also known as linker scripts).
>>>
>>> This is no longer the case. Now all compilers we use support symbol 
>>> visibility directives, in one form or another. We should start using 
>>> this. Since this has been the only way to control symbol visibility 
>>> on Windows, for most of the shared code, we already have proper 
>>> JNIEXPORT decorations in place.
>>>
>>> If we fix the remaining platform-specific files to have proper 
>>> JNIEXPORT tagging, then we can finally get rid of mapfiles.
>>>
>>> This fix removed mapfiles for all JDK libraries. It does not touch 
>>> hotspot libraries nor JDK executables; they will have to wait for a 
>>> future fix -- this was complex enough. This change will not have any 
>>> impact on macosx, since we do not use mapfiles there, but instead 
>>> export all symbols. (This is not a good idea, but I'll address that 
>>> separately.) This change will also have a minimal impact on Windows. 
>>> The only reason Windows is impacted at all, is that some changes 
>>> needed by Solaris and Linux were simpler to fix for all platforms.
>>>
>>> I have strived for this change to have no impact on the actual 
>>> generated code. Unfortunately, this was not possible to fully 
>>> achieve. I do not believe that these changes will have any actual 
>>> impact on the product, though. I will present the differences more 
>>> in detail further down. Those who are not interested can probably 
>>> skip that.
>>>
>>> The patch has passed tier1 testing and is currently running tier2 
>>> and tier3. Since the running code is more or less (see caveat below) 
>>> unmodified, I don't expect any testing issues.
>>>
>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8200178
>>> WebRev: 
>>> http://cr.openjdk.java.net/~ihse/JDK-8200178-remove-mapfiles/webrev.01
>>>
>>> Details on changes:
>>> Most of the source code changes are (unsurprisingly) in java.base 
>>> and java.desktop. Remaining changes are in jdk.crypto.ucrypto, 
>>> jdk.hotspot.agent, jdk.jdi and jdk.jdwp.agent.
>>>
>>> Source code changes does almost to 100% consists in decorating an 
>>> exported function with JNIEXPORT. I have also followed the 
>>> long-standing convention of adding JNICALL. This is a no-op on 
>>> non-Windows platforms, so for most of the changes this is purely 
>>> cosmetic (and possibly adding in robustness, should the function 
>>> ever be used on Windows in the future). I have also followed the 
>>> stylistic convention of putting "JNIEXPORT <return type> JNICALL" on 
>>> a separate line. For some functions, however, this might cause a 
>>> change in calling convention on Windows. Since this can not apply to 
>>> exported functions on Windows (otherwise they would already have had 
>>> JNIEXPORT), I do not think this matters anything.
>>>
>>> A few libraries did not have a mapfile, on Linux and/or Solaris. 
>>> This actually meant that all symbols were exported. It is highly 
>>> unclear if this was known and intended by the original make rule 
>>> writer. I have emulated this by adding the flag 
>>> $(EXPORT_ALL_SYMBOLS) to these libraries. Hopefully, we can remove 
>>> this flag and fix proper exported symbols in the future.
>>>
>>> I have run the complete build using COMPARE_BUILD, and made a 
>>> thourough analysis of the differences for Linux and Solaris. All 
>>> native libraries have symbol differences, but most of them are 
>>> trivial and/or harmless. As a result, most libraries have disasm 
>>> differences as well, but these too seem trivial and harmless. The 
>>> differences in symbols that are common to all libraries include:
>>>  * Internal symbols such as __bss_start, _edata, _end and _fini are 
>>> now global. (They are imported as such from the compiler 
>>> libraries/archives, and we have no linker script to override this 
>>> behavior).
>>>  * The versioning tag SUNWprivate_1.1 is not included, and thus 
>>> neither the .gnu.version_d symbol.
>>>  * There are a few differences in the symbol and/or mangling of some 
>>> local functions. I'm not sure what's causing this,
>>> but it's unlikely to have any effect on the product.
>>>
>>> Another common source for change in symbols is due to previous 
>>> platform differences. For instance, if we had "JNIEXPORT int JNICALL 
>>> do_foo() { ... }", but do_foo was not in the mapfile, the symbol was 
>>> exported on Windows but not on Linux and Solaris. (Presumable since 
>>> it was not needed there, even though it was compiled for those 
>>> platforms as well.) Now, with the mapfiles gone, do_foo() will be 
>>> exported on all platforms. And contrary, functions that are compiled 
>>> on all platforms, and were exported in mapfiles, but now have gotten 
>>> an JNIEXPORT decoration, will now be visible even on Windows. (This 
>>> accounts for half of the noticed symbol differences on Windows.) I 
>>> could have made the JNIEXPORT conditional on OS, but I didn't think 
>>> the mess in source code were worth the keeping of binary confidence 
>>> with the old build.
>>>
>>> A third common source for change in symbols is due to exported 
>>> functions "leaking" across library borders. For instance, some 
>>> functions in java.desktop is compiled in both libawt_xawt and 
>>> libawt_headless, but they were previously only included in the 
>>> mapfile for one of these libraries. Now, since the visibility is 
>>> determined by the source code itself, it gets exported in both 
>>> libraries. A variant of this is when a library depends on another 
>>> JDK library, and includes the header file from that other library, 
>>> which in turn declares a function as JNIEXPORT. This will cause the 
>>> including library to also export the function. This accounts for the 
>>> other half of the changes on Windows. A typical example of this is 
>>> that multiple libraries now re-export hotspot symbols from 
>>> libjvm.so, like jio_fprintf. (I have not listed the libjvm 
>>> re-exports below.)
>>>
>>> Note that  Java_java_io_FileOutputStream_close0 in 
>>> java.base/unix/native/libjava/FileOutputStream_md.c is no longer 
>>> exported,
>>> and can probably be removed.
>>>
>>> Here is a detailed table showing and accounting for all the 
>>> remaining differences found on Linux and Solaris:
>>> java.base/unix/native/libjava: Java_java_io_FileOutputStream_close0 
>>> is now also exported on unix platforms due to JNIEXPORT.
>>>
>>> java.base/jspawnlauncher: On solaris, we also include 
>>> libjava/childproc.o, which
>>> now exports less functions than it used to (it used to export all 
>>> functions, now it is compiled with visibility=hidden).
>>>
>>> java.base/java(w).exe: Is now also exporting the following symbols 
>>> due to added JNIEXPORT in libjli on Windows:
>>> (Yes, executables can export symbols on Windows. Confusing, I know.)
>>>  JLI_AddArgsFromEnvVar
>>>  JLI_CmdToArgs
>>>  JLI_GetAppArgIndex
>>>  JLI_GetStdArgc
>>>  JLI_GetStdArgs
>>>  JLI_InitArgProcessing
>>>  JLI_Launch
>>>  JLI_List_add
>>>  JLI_List_new
>>>  JLI_ManifestIterate
>>>  JLI_MemAlloc
>>>  JLI_MemFree
>>>  JLI_PreprocessArg
>>>  JLI_ReportErrorMessage
>>>  JLI_ReportErrorMessageSys
>>>  JLI_ReportExceptionDescription
>>>  JLI_ReportMessage
>>>  JLI_SetTraceLauncher
>>>  JLI_StringDup
>>>
>>> java.desktop:/libawt_xawt: The following symbols are now also 
>>> exported on linux and solaris due to JNIEXPORT:
>>>  awt_DrawingSurface_FreeDrawingSurfaceInfo
>>>  awt_DrawingSurface_GetDrawingSurfaceInfo
>>>  awt_DrawingSurface_Lock
>>>  awt_DrawingSurface_Unlock
>>>  awt_GetColor
>>>
>>> The following symbols are now also exported on linux and solaris due 
>>> to JNIEXPORT (they were previously
>>>  exported only in libawt):
>>>  Java_sun_awt_DebugSettings_setCTracingOn__Z
>>>  Java_sun_awt_DebugSettings_setCTracingOn__ZLjava_lang_String_2
>>>  Java_sun_awt_DebugSettings_setCTracingOn__ZLjava_lang_String_2I
>>>  Java_sun_awt_X11GraphicsConfig_getNumColors
>>>
>>> java.desktop:/libawt_headless: The following symbols are now also 
>>> exported due to JNIEXPORT (they were previously
>>>  exported only in libawt_xawt and/or libawt):
>>>  Java_sun_java2d_opengl_GLXGraphicsConfig_getGLXConfigInfo
>>>  Java_sun_java2d_opengl_GLXGraphicsConfig_getOGLCapabilities
>>>  Java_sun_java2d_x11_X11PMBlitLoops_updateBitmask
>>>  Java_sun_java2d_x11_X11SurfaceData_isShmPMAvailable
>>>  X11SurfaceData_GetOps
>>>
>>> java.desktop/libawt: The following symbols are now also exported on 
>>> Windows, due to added
>>> JNIEXPORT:
>>>  SurfaceData_InitOps
>>>  mul8table
>>>  div8table
>>>  doDrawPath
>>>  doFillPath
>>>  g_CMpDataID
>>>  initInverseGrayLut
>>>  make_dither_arrays
>>>  make_uns_ordered_dither_array
>>>  path2DFloatCoordsID
>>>  path2DNumTypesID
>>>  path2DTypesID
>>>  path2DWindingRuleID
>>>  sg2dStrokeHintID
>>>  std_img_oda_blue
>>>  std_img_oda_green
>>>  std_img_oda_red
>>>  std_odas_computed
>>>  sunHints_INTVAL_STROKE_PURE
>>>
>>> java.desktop/libawt on solaris:
>>> A number of "#pragma weak" directives was previously overridden by 
>>> the mapfile.
>>> Now these directives are respected, so these symbols are now weak 
>>> instead of local:
>>>  ByteGrayToIntArgbPreConvert_F
>>>  ByteGrayToIntArgbPreScaleConvert_F
>>>  IntArgbBmToFourByteAbgrPreScaleXparOver_F
>>>  IntArgbToIntRgbXorBlit_F
>>>  IntBgrToIntBgrAlphaMaskBlit_F
>>>
>>> java.desktop/libawt on solaris: These are now also exported due to 
>>> JNIEXPORT in libmlib_image.
>>>  j2d_mlib_ImageCreate
>>>  j2d_mlib_ImageCreateStruct
>>>  j2d_mlib_ImageDelete
>>>
>>> java.desktop/libawt on solaris: This is now also exported due to 
>>> JNIEXPORT:
>>>  GrPrim_CompGetXorColor
>>>  SurfaceData_GetOpsNoSetup
>>>  SurfaceData_IntersectBoundsXYWH
>>>  SurfaceData_SetOps
>>>  Transform_GetInfo
>>>  Transform_transform
>>>
>>> java.desktop/libsplashscreen: JNI_OnLoad is now exported on linux 
>>> and solaris due to JNIEXPORT.
>>> libspashscreen also had JNIEXPORT (actually a pure 
>>> _declspec(dllexport)) but no JNICALL, which I added as
>>> a part of converting to JNIEXPORT. The same goes for libmlib_image .
>>>
>>> jdk.sctp/libsctp: handleSocketError is now exported on linux and 
>>> solaris due to JNIEXPORT in libnio.
>>>
>>> java.instrument:/libinstrument: Agent_OnUnload is now also exported 
>>> on linux and solaris platforms due to JNIEXPORT.
>>> JLI_ManifestIterate is now also exported on Windows, due to added 
>>> JNIEXPORT in libjli.
>>>
>>> jdk.management/libmanagement_ext: 
>>> Java_com_sun_management_internal_Flag_setDoubleValue is now also 
>>> exported on linux and solaris platforms due to JNIEXPORT.
>>>
>>> /Magnus
>>>
>>>
>>
>