free(): corrupted unsorted chunks

Oliver Kohll oliver at agilechilli.com
Thu Jun 21 15:06:51 UTC 2018 

Hi Thomas,


Unfortunately not, that was /var/log/tomcat8/catalina.out, a snippet from
just before to afterwards when I restarted is below. Everything after
'free(): corrupted unsorted chunks', from 'Note: Picked up
JDK_JAVA_OPTIONS' on was only logged on restart. The warnings are normal.


The lines before were from my application. However they're different for
each crash.


Thanks

Oliver


---


Thu 2018/06/21 09:29:19.913|Warn|Scheduler|View CTG - (Fitting Sync) has
already been scheduled again, shouldn't be reset

Thu 2018/06/21 09:29:21.639|Info|ReportDownloader|User [username] exporting
report GMP from table n0) ff - GMP

Thu 2018/06/21 09:29:23.506|Info|Scheduler|Starting action for googlesyncer
view b4) Fitting Dates (Remedial Sync)

Thu 2018/06/21 09:29:23.911|Info|Scheduler|Starting action for googlesyncer
view b5) Survey Dates (Survey Sync)

Thu 2018/06/21 09:29:23.914|Warn|GoogleSyncer|1 other calendar syncs taking
place, delaying for a minute

Thu 2018/06/21 09:29:23.941|Warn|Scheduler|View (Survey Sync) has already
been scheduled again, shouldn't be reset

Thu 2018/06/21 09:29:24.106|Info|Scheduler|Starting action for
workflowrunner view m1.0) Customer Order.orders to lock

Thu 2018/06/21 09:29:24.338|Info|Scheduler|Starting action for googlesyncer
view a) leads.AGS (Lead Sync)

*free(): corrupted unsorted chunks*

NOTE: Picked up JDK_JAVA_OPTIONS:
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

21-Jun-2018 09:35:56.096 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/common/classes], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.102 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/common], exists: [false], isDirectory: [false],
canRead: [false]

21-Jun-2018 09:35:56.102 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/common/classes], exists: [false],
isDirectory: [false], canRead: [false]

21-Jun-2018 09:35:56.102 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/common], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.104 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/server/classes], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.104 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/server], exists: [false], isDirectory: [false],
canRead: [false]

21-Jun-2018 09:35:56.105 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/server/classes], exists: [false],
isDirectory: [false], canRead: [false]

21-Jun-2018 09:35:56.105 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/server], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.105 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/shared/classes], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.105 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/shared], exists: [false], isDirectory: [false],
canRead: [false]

21-Jun-2018 09:35:56.106 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/shared/classes], exists: [false],
isDirectory: [false], canRead: [false]

21-Jun-2018 09:35:56.106 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/shared], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.795 WARNING [main]
org.apache.tomcat.util.digester.SetPropertiesRule.begin
[SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property
'debug' to '0' did not find a matching property.

21-Jun-2018 09:35:56.806 WARNING [main]
org.apache.tomcat.util.digester.SetPropertiesRule.begin
[SetPropertiesRule]{Server/Service/Engine/Host/Context/Realm/Realm} Setting
property 'debug' to '99' did not find a matching property.

21-Jun-2018 09:35:56.807 WARNING [main]
org.apache.tomcat.util.digester.SetPropertiesRule.begin
[SetPropertiesRule]{Server/Service/Engine/Host/Context/Realm/Realm} Setting
property 'digest' to 'MD5' did not find a matching property.

21-Jun-2018 09:35:56.809 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server version:
  Apache Tomcat/8.5.30 (Ubuntu)

21-Jun-2018 09:35:56.810 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server built:
  May 30 2018 13:37:13 UTC

21-Jun-2018 09:35:56.810 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server number:
  8.5.30.0

21-Jun-2018 09:35:56.810 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log OS Name:
  Linux

21-Jun-2018 09:35:56.810 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log OS Version:
  4.15.13-x86_64-linode106

21-Jun-2018 09:35:56.811 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Architecture:
  amd64

21-Jun-2018 09:35:56.811 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Java Home:
  /usr/lib/jvm/java-11-openjdk-amd64

21-Jun-2018 09:35:56.811 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log JVM Version:
  10.0.1+10-Ubuntu-3ubuntu1

21-Jun-2018 09:35:56.811 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
  Oracle Corporation

21-Jun-2018 09:35:56.811 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:
  /var/lib/tomcat8

21-Jun-2018 09:35:56.812 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:
  /usr/share/tomcat8

21-Jun-2018 09:35:56.812 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.base/java.lang=ALL-UNNAMED

21-Jun-2018 09:35:56.812 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.base/java.io=ALL-UNNAMED

21-Jun-2018 09:35:56.812 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

21-Jun-2018 09:35:56.812 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument:
-Djava.util.logging.config.file=/var/lib/tomcat8/conf/logging.properties

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.awt.headless=true

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xmx8g

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djdk.tls.ephemeralDHKeySize=2048

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dignore.endorsed.dirs=

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dcatalina.base=/var/lib/tomcat8

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dcatalina.home=/usr/share/tomcat8

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.io.tmpdir=/tmp/tomcat8-tomcat8-tmp

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR
based Apache Tomcat Native library [1.2.16] using APR version [1.6.3].

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
capabilities: IPv6 [true], sendfile [true], accept filters [false], random
[true].

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL
configuration: useAprConnector [false], useOpenSSL [true]

21-Jun-2018 09:35:56.818 INFO [main]
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
successfully initialized [OpenSSL 1.1.0g  2 Nov 2017]

21-Jun-2018 09:35:56.858 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["http-nio-8080"]

21-Jun-2018 09:35:56.876 INFO [main]
org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared
selector for servlet write/read

21-Jun-2018 09:35:56.884 INFO [main]
org.apache.coyote.http11.AbstractHttp11Protocol.configureUpgradeProtocol
The ["https-openssl-apr-8443"] connector has been configured to support
negotiation to [h2] via ALPN

21-Jun-2018 09:35:56.884 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["https-openssl-apr-8443"]

21-Jun-2018 09:35:56.926 INFO [main]
org.apache.catalina.startup.Catalina.load Initialization processed in 773 ms

21-Jun-2018 09:35:56.997 INFO [main]
org.apache.catalina.core.StandardService.startInternal Starting service
[Catalina]

21-Jun-2018 09:35:56.998 INFO [main]
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
Engine: Apache Tomcat/8.5.30 (Ubuntu)

On 21 June 2018 at 15:31:12, Thomas Stüfe (thomas.stuefe at gmail.com) wrote:

Whereever you got the "free(): corrupted unsorted chunks" from should
contain actually more information. For an example, see:

https://www.unix.com/red-hat/233292-free-corrupted-unsorted-chunks.html

Thanks, Thomas

On Thu, Jun 21, 2018 at 4:27 PM, Oliver Kohll <oliver at agilechilli.com>
wrote:
> Not that I can see I'm afraid - there's nothing else in either
> /var/log/tomcat8/catalina.out (or any of the other files in that
directory)
> or /var/log/syslog. I also can't see anything likely in /tmp
>
> I'm Googling how to get tomcat to create a stack trace to a file on a
crash,
> but the consensus seems to be they should be in catalina.out by default.
>
> On 21 June 2018 at 13:28:46, Thomas Stüfe (thomas.stuefe at gmail.com)
wrote:
>
> Hi Oliver,
>
> Is there a stack trace on stderr?
>
> Thanks, Thomas
>
>
>
>
>
> On Thu, Jun 21, 2018 at 1:59 PM, Oliver Kohll <oliver at agilechilli.com>
> wrote:
>> Hi,
>>
>> I'm a Java developer getting a crash and error message 'free():
corrupted
>> unsorted chunks' in the log. this sounds like it could be a low level
>> issue
>> in the core Java libraries so I'm posting here, but if it would be
better
>> in another list (or even a different community) please let me know.
>>
>> So I'm requesting help because I don't know how to debug this further.
In
>> summary, the issue is
>>
>> We're getting a crash nearly once a day on our production system, at a
>> time
>> of day when it's most busy (mid morning). This crash hasn't been seen on
>> our development environment, which is the same apart from the server has
>> less memory and CPU resources.
>>
>> The environment is:
>> * A VPS host
>> * Ubuntu LTS 18 (Bionic)
>> * The default Java 10 packaged as 11 I believe (
>>
>>
https://askubuntu.com/questions/1037646/why-is-openjdk-10-packaged-as-openjdk-11
>> )
>> * However, javac seems to be from JDK8:
>> root at server:~# file /etc/alternatives/java /etc/alternatives/javac
>> /etc/alternatives/java: symbolic link to
>> /usr/lib/jvm/java-11-openjdk-amd64/bin/java
>> /etc/alternatives/javac: symbolic link to
>> /usr/lib/jvm/java-8-openjdk-amd64/bin/javac
>>
>> We're running our webapp on Tomcat 8 (also the Ubuntu default). When
this
>> happens, the server stops responding to HTTP requests (in fact the java
>> process has quit) and needs to be restarted.
>>
>> The crash doesn't seem to correspond to a particular activity on the
>> server
>> as far as I can see, though it's hard to tell as many people are logged
in
>> at the same time.
>>
>> Does anyone have any pointers of
>> a) how to track the bug further so I can report it, ideally without a
big
>> impact on our live server
>> b) more immediately, any workarounds to try
>>
>> In the meantime I will see if I can downgrade to JDK8 on the test
server,
>> but I'm not sure if that's possible or whether it would make a
difference.
>>
>> Regards
>> Oliver

More information about the core-libs-dev mailing list