sigsegv on porter stemmer (Lucene, but also otherwise)

[Uwe Schindler]

Hi thanks for taking care!

Thanks for the workaround. We already found another workaround to get this
running in our 2-hourly Lucene builds at:

https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/
https://builds.apache.org/job/Lucene-Solr-tests-only-3.x-java7/

It would be nice if you could look into the console logs of the failed
builds on Saturday - you can see the bug in the earlier builds only (with
always different stack traces). We drilled it down to one method (not sure
if this information clipped out of the bug report. We then disabled
compilation for only this affected method, its PorterStemmer.ends(...):

-XX:CompileCommand=exclude,org/apache/lucene/analysis/en/PorterStemmer,ends
-XX:CompileCommand=exclude,org/apache/lucene/analysis/PorterStemmer,ends

(we have different class names in stable 3.x branch and trunk).

We also see other random test failures not happening with Java 5 and Java 6,
it would be nice, if you could review, too.

One big bug in loops affected also Java 1.6.0_18 (still not fixed): our
DataInput.readVInt method was incorrectly compiled in the case that
MappedByteBuffer.get()/lucene.DataInput.readByte() was inlined, leading to
simply wrong results (the method returned a decoded integer that was
different than expected results). See the unwinded loops in
https://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/src/java/org/apache
/lucene/store/DataInput.java

I hope this all helps you in finding more bad loop optimization bugs, all of
those issues seem to be related to this special optimization in loops. The
latest lucene builds also contains a failure in a test case only happening
with Java 7 (not on every test run, so unreproducible). So it might be good
for you to watch our Lucene builds also for other bugs.

Some of the other developers already say, we should not trust any loops in
java anymore and recommend not to use Java 7 with Apache Lucene/Solr, and
that's bad news :(

Thanks for the help,
Uwe

On Mon, Jul 25, 2011 at 7:49 PM, Vladimir Kozlov
<vladimir.kozlov at oracle.com> wrote:
> Thank you very much, Dawid, for providing the test case. Bug was filed in
> wrong category so we did not know about it. I will work on it since it
could
> be my changes in loop optimizations. Use next flag as workaround:
>
> -XX:-UseLoopPredicate
>
> Thanks,
> Vladimir
>
> Dawid Weiss wrote:
>>
>> Hello everyone,
>>
>> I am an Apache Lucene developer, we've been running tests with Java
>> 1.7 and this came up:
>>
>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7070134
>>
>> Porter stemmer is pretty widely used for shallow NLP, not only in
>> Lucene. It'd be interesting to hear from jit gurus what's causing this
>> (the problem does not occur in 1.6). Thanks in advance,
>>
>> Dawid

-----
Uwe Schindler
uschindler at apache.org 
Apache Lucene PMC Member / Committer
Bremen, Germany
http://lucene.apache.org/