[8u] RFR: 8037825: Fix warnings and enable "warnings as errors" in serviceability native libraries

[Hohensee, Paul]

Please review this backport to help 8u build using Xcode 12.

Original issue: https://bugs.openjdk.java.net/browse/JDK-8037825
Original commit: http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f5a18100873c
8u webrev: https://cr.openjdk.java.net/~phh/8037825/webrev.8u.jdk.00/

The patch applied cleanly, except that the change to InvocationAdapter.c was already in place, so was left out of the backport. There are no semantic changes.

Thanks,
Paul