[8u] RFR 8264752: SIGFPE crash with option FlightRecorderOptions:threadbuffersize=30M

[kalinshi(施慧)]

Please help review JDK-8264752 backport to jdk8u-dev. This PR is already backported to jdk11u.

The original patch does not apply cleanly, conflicts are:
- ClassFileInstaller doesn't need full name in TestBadOptionValues.java (same conflict in jdk11u)
- Change "log_error(arguments)" to "tty->print_cr" in JfrOptionSet::adjust_memory_options

Original bug: https://bugs.openjdk.java.net/browse/JDK-8264752
Original patch: https://git.openjdk.java.net/jdk/commit/377b3461

8u webrev: crosses hotspot and jdk repository
http://cr.openjdk.java.net/~hshi/8264752/webrev_total/

Test: jdk_jfr with fastdebug/release build pass

Regards
Hui