Odd OS-level memory reporting for Java

[Shawn Heisey]

On 1/7/2016 9:29 AM, Yu Zhang wrote:
> What is the parameters for java? Did you use AlwaysPreTouch, give -Xms
> -Xmx?
>
> I am not sure about the Linux output. But the Windows one
> 15,861,867(KB) is reasonable for 14g heap, I think.

This is the full commandline for the Linux machine.  It is starting
jetty, which then runs Solr4.9.1.  These parameters are included by an
init script that I wrote:

/usr/bin/java -Xms4096M -Xmx8192M
-Dlog4j.configuration=file:etc/log4j.properties -XX:+UseG1GC
-XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8m
-XX:MaxGCPauseMillis=250 -XX:InitiatingHeapOccupancyPercent=75
-XX:+UseLargePages -XX:+AggressiveOpts -verbose:gc -Xloggc:logs/gc.log
-XX:+PrintGCDateStamps -XX:+PrintGCDetails
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8686
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dsolr.solr.home=/index/solr4 -Djetty.port=8981 -DSTOP.PORT=8078
-DSTOP.KEY=redacted -jar /opt/solr4/start.jar

If "Commit" is the proper number to look at, then Windows may not have
the same issue Linux does. I was under the impression that "Working Set"
was what I should be looking at, especially since the other Java
processes and the SQL Server process are showing a smaller number there
than under Commit.  I do not know what those other java processes are.
The Windows machine *is* showing the same shared memory inflation that I
see on the Linux machine.

I have complete control over the Linux machine, so I can definitely
answer questions about that one.  I understand Linux a lot more, and to
be honest I don't really care about Windows.  I'm trying to help this
user with what they perceive as memory problems, although I suspect that
everything is working just like it should.

The following info is not completely confirmed, but based on the user's
posting history on the solr-user mailing list, I believe that the
Windows machine is using the arguments set by the included 5.x start
script, which will mean a list like this that I obtained from running
Solr 5.4.0 on my Windows 7 desktop.  Xms and Xmx are only 512m on my
machine, this will definitely not be the case for the machine that
produced the screenshot:

-DSTOP.KEY=solrrocks
-DSTOP.PORT=7983
-Djava.io.tmpdir=C:\Users\sheisey\Downloads\solr-5.4.0\server\tmp
-Djetty.home=C:\Users\sheisey\Downloads\solr-5.4.0\server
-Djetty.port=8983
-Dlog4j.configuration=file:C:\Users\sheisey\Downloads\solr-5.4.0\server\resources\log4j.properties
-Dsolr.install.dir=C:\Users\sheisey\Downloads\solr-5.4.0
-Dsolr.solr.home=C:\Users\sheisey\Downloads\solr-5.4.0\server\solr
-Duser.timezone=UTC
-XX:+CMSParallelRemarkEnabled
-XX:+CMSScavengeBeforeRemark
-XX:+ParallelRefProcEnabled
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:ConcGCThreads=4
-XX:MaxTenuringThreshold=8
-XX:NewRatio=3
-XX:ParallelGCThreads=4
-XX:PretenureSizeThreshold=64m
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-Xloggc:C:\Users\sheisey\Downloads\solr-5.4.0\server\logs/solr_gc.log
-Xms512m
-Xmx512m
-Xss256k
-verbose:gc

It is probably worth mentioning that I have another Linux machine
running Java 8u66 and Solr 5.3 with CMS GC (the same settings that I
just listed for my Win7 desktop machine) is *not* showing the same SHR
and RES inflation:

https://www.dropbox.com/s/i49s2uyfetwo3xq/solr-mem-prod-8g-heap.png?dl=0

Thanks,
Shawn