sigsegv on porter stemmer (Lucene, but also otherwise)

[Vladimir Kozlov]

Thank you, Uwe

I looked on logs and most of them, it seems, related to the problem described in 
your bug 7070134.

And I looked on LUCENE-2975 which describes DataInput.readVInt() problem. It 
references Hotspot bug 5091921. The loop optimization changes I mentioned before 
were done to fix that bug so it should be fixed in jdk7. But, as we know now, 
those changes introduced several new problems, like yours 7070134. I am working 
on them.

Regards,
Vladimir

Uwe Schindler wrote:
> Hi thanks for taking care!
> 
> Thanks for the workaround. We already found another workaround to get this
> running in our 2-hourly Lucene builds at:
> 
> https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/
> https://builds.apache.org/job/Lucene-Solr-tests-only-3.x-java7/
> 
> It would be nice if you could look into the console logs of the failed
> builds on Saturday - you can see the bug in the earlier builds only (with
> always different stack traces). We drilled it down to one method (not sure
> if this information clipped out of the bug report. We then disabled
> compilation for only this affected method, its PorterStemmer.ends(...):
> 
> -XX:CompileCommand=exclude,org/apache/lucene/analysis/en/PorterStemmer,ends
> -XX:CompileCommand=exclude,org/apache/lucene/analysis/PorterStemmer,ends
> 
> (we have different class names in stable 3.x branch and trunk).
> 
> We also see other random test failures not happening with Java 5 and Java 6,
> it would be nice, if you could review, too.
> 
> One big bug in loops affected also Java 1.6.0_18 (still not fixed): our
> DataInput.readVInt method was incorrectly compiled in the case that
> MappedByteBuffer.get()/lucene.DataInput.readByte() was inlined, leading to
> simply wrong results (the method returned a decoded integer that was
> different than expected results). See the unwinded loops in
> https://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/src/java/org/apache
> /lucene/store/DataInput.java
> 
> I hope this all helps you in finding more bad loop optimization bugs, all of
> those issues seem to be related to this special optimization in loops. The
> latest lucene builds also contains a failure in a test case only happening
> with Java 7 (not on every test run, so unreproducible). So it might be good
> for you to watch our Lucene builds also for other bugs.
> 
> Some of the other developers already say, we should not trust any loops in
> java anymore and recommend not to use Java 7 with Apache Lucene/Solr, and
> that's bad news :(
> 
> Thanks for the help,
> Uwe
> 
> On Mon, Jul 25, 2011 at 7:49 PM, Vladimir Kozlov
> <vladimir.kozlov at oracle.com> wrote:
>> Thank you very much, Dawid, for providing the test case. Bug was filed in
>> wrong category so we did not know about it. I will work on it since it
> could
>> be my changes in loop optimizations. Use next flag as workaround:
>>
>> -XX:-UseLoopPredicate
>>
>> Thanks,
>> Vladimir
>>
>> Dawid Weiss wrote:
>>> Hello everyone,
>>>
>>> I am an Apache Lucene developer, we've been running tests with Java
>>> 1.7 and this came up:
>>>
>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7070134
>>>
>>> Porter stemmer is pretty widely used for shallow NLP, not only in
>>> Lucene. It'd be interesting to hear from jit gurus what's causing this
>>> (the problem does not occur in 1.6). Thanks in advance,
>>>
>>> Dawid
> 
> -----
> Uwe Schindler
> uschindler at apache.org 
> Apache Lucene PMC Member / Committer
> Bremen, Germany
> http://lucene.apache.org/
> 
>