[9] RFR(S): 8160651: StubRoutines::_dtan does not restore callee save register rbx

[Tobias Hartmann]

Hi,

please review the following patch:

https://bugs.openjdk.java.net/browse/JDK-8160651
http://cr.openjdk.java.net/~thartmann/8160651/webrev.00/

StubRoutines::_dtan overwrites the callee save register $rbx but does not restore the original value before returning. The problem is in MacroAssembler::fast_tan() introduced by JDK-8152907 [1]: we set $rbx to 32768 (0x8000) but do not pop the saved value from the stack before returning. This causes several test failures (see linked bugs) because $rbx contains incorrect values. The implementations of MacroAssembler::fast_cos() are MacroAssembler::fast_sin() are similar, but correctly restore $rbx [2].

I also removed the unused LibraryCallKit::inline_trig().

Tested with failing tests, JPRT and RBT (running).

Thanks,
Tobias

[1] http://hg.openjdk.java.net/jdk9/hs-comp/hotspot/rev/33f10a35ce20#l16.1052
[2] http://hg.openjdk.java.net/jdk9/hs-comp/hotspot/rev/33f10a35ce20#l8.3947