RFC: regarding metaspace(metadata?) dump
Ioi Lam
ioi.lam at oracle.com
Thu Jan 12 19:36:36 UTC 2023
On 1/12/2023 12:29 AM, Yi Yang wrote:
> Thanks Ioi and Thomas for your valuable thoughts! I list some pros
> about metaspace dump.
>
> 1. Standardization. The format of metadata dump is standard and
> well-formed. It could be seamlessly integrated with
> DevOps/Diagnose/APM platforms, while SA is interactively and the
> output of jcmd is not well-formed and parser-unfriendly, and its
> content is subject to change. You can't expect DevOps platform to use
> SA/coredump/gdb conveniently and automatically in production environment.
>
The reason I mentioned SA is for implementing the dumper that you're
proposing on top of SA, because SA already has APIs for reflective
access into the metadata. This would be much less intrusive than
modifying HotSpot itself.
I once implemented something similar (but much more simple: jcmd
GC.class_stats). It was removed in
https://bugs.openjdk.org/browse/JDK-8232759 due to lack of usage and
cost of maintenance.
Thanks
- Ioi
> 2. Functionality. MetaspaceDumpBeforeFullGC could generate a small
> dump for further debugging, it works as well as heap dump.
>
> 3. JVM Metadata. Codecache dump, method counter, method data, they are
> unexplored scopes, reconstruct human-readable representation of
> compiled method.
>
> 4. Complexity. In my humble opinion, all of this stuff such as
> by-chunktype/show-loaders/VM.classloader_stats/VM.classloader_hierarchy/etc
> could be done in other place. VM is eligible to provides a standard
> and rich raw metadata output, third-party parser and UI render display
> them in their way instead of continuously adding new
> filter/grouping/hierarcy/VM.method_counter features when we do want to
> know them. Heap dump is a good example, it dumps all
> objects/symbols/etc to a binary file, and third-party tools
> orchestrate them to histogram/thread/classloader/domtree.
>
> Basically, I know the content of metaspace dump file has overlap with
> some existing tools such as SA/jcmd/coredump/gdb/HeapDump, as Thomas
> commented inlinely, I don’t think metaspace dump can troubleshoot many
> problems that it is the only solution. I think metadata dump is more
> about providing a standardized and parser-friendly framework, so that
> users at all levels can inspect JVM metadata information they care
> about. In addition, something like M(etaspace)AT could orchestrate
> dump content with filter/grouping/hierarchy options.
>
> > Also you mentioned that "Internally we implemented a metaspace dump
> that generates human-readable text". Can you share how this tool was
> implemented?
>
> That's not surprising, it iterates CLD/Classes/etc and dumps basic
> information about metaspace, a demo metaspace dump can be found at
> https://gist.github.com/y1yang0/683d8a58dd946b3e9180682863df55ea
> <https://urldefense.com/v3/__https://gist.github.com/y1yang0/683d8a58dd946b3e9180682863df55ea__;!!ACWV5N9M2RV99hQ!OOiFdPBm2uADyfD3T_L4JCOp6GC489SlckPnHoP6Q3GSqkmE9oBiBtQzm42_AWGASDogbJFRSVxHhIM6r4-7l_U$>
>
> ------------------------------------------------------------------
> From:Thomas Stüfe <thomas.stuefe at gmail.com>
> Send Time:2023 Jan. 12 (Thu.) 15:06
> To:"YANG, Yi" <qingfeng.yy at alibaba-inc.com>
> Cc:HotSpot Open Source Developers <hotspot-dev at openjdk.java.net>;
> hotspot-runtime-dev <hotspot-runtime-dev at openjdk.java.net>;
> hotspot-dev <hotspot-dev at openjdk.org>
> Subject:Re: RFC: regarding metaspace(metadata?) dump
>
> Hi Yi,
>
> A lot of what you try to do already exists. For example, we also
> have `VM.metaspace`. This is a quite powerful command to analyze
> Metaspace-related issues, especially for fragmentation and other
> wastages. Generally speaking, it is the tool you use to look at
> the underpinnings of metaspace, the allocator, while tools like
> `VM.classloaders`, `VM.classloader_stats` and `VM.classes` look at
> things "from above", e.g. walk the CLDG. All these tools have
> already a bit of overlap.
>
> For analyzing OOMEs, you need several tools, since it can be
> caused by multiple issues. E.g. tools that walk the CLDG don't see
> fragmentation, or unclaimed metaspace for dead loaders.
>
> Please find more remarks inline.
>
> On Wed, Jan 11, 2023 at 1:56 PM Yi Yang
> <qingfeng.yy at alibaba-inc.com> wrote:
> Hi,
>
> Internally, we often receive feedback from users and ask for help
> on metaspace-related issues, for example
> 1. Users are eager to know which GroovyClassLoader loads which
> classes, why they are not unloaded,
> and why they are leading to Metaspace OOME.
>
> There are several tools to do this, for example:
>
> `VM.metaspace show-loaders show-classes`
> `VM.classloaders show-classes`
>
> both show you loaded classes by loader, and the former also shows
> you metaspace stats needed to understand OOMEs. None of these
> tools shows you why loaders are kept alive, but for that you need
> heap- and GC-root-analysis. This quickly enters the territory of
> Eclipse MAT and similar tools, where having a text-based tool
> alone gets cumbersome.
> 2. They want to know the class structure of dynamically generated
> classes in some scenarios such as
> deserialization
>
> Interesting. This seems to be a very specific query; not sure how
> general the need for this is. `VM.classes -verbose` shows a part
> of the story.
> 3. Finding memory leaking about duplicated classes
>
> Again,
>
> `VM.metaspace show-loaders show-classes`
> `VM.classloaders show-classes`
>
> but also `VM.classes` and `VM.classloader_stats` are your friends
> here.
>
> ...
> Internally we implemented a metaspace dump that generates
> human-readable text, it looks something like this:
>
> [Basic Information]
> Dump Reason : JCMD
> MaxMetaspaceSize : 18446744073709547520 B
> CompressedClassSpaceSize : 1073741824 B
> Class Space Used : 309992 B
> Class Space Capacity : 395264 B
> ...
> [Class Loader Data]
> ClassLoaderData : loader = 0x000000008024f928, loader_klass =
> 0x0000000800010098, loader_klass_name =
> sun/misc/Launcher$AppClassLoader, label = N/A
> Class Used Chunks :
> * Chunk : [0x0000000800060000, 0x0000000800060230,
> 0x0000000800060800)
> NonClass Used Chunks :
> * Chunk : [0x00007fd8379c1000, 0x00007fd8379c1350,
> 0x00007fd8379c2000)
> Klasses :
> Klass : 0x0000000800060028, name = Test, size = 520 B
> ConstantPool : 0x00007fd8379c1050, size = 296 B
> ...
>
>
> `VM.metaspace` shows you the chunk composition of arenas if needed.
>
> E.g. : `VM.metaspace by-chunktype show-loaders`
>
> ```
> Usage per loader:
>
> 1: CLD 0x00007f72fc29b820: "app" instance of
> jdk.internal.loader.ClassLoaders$AppClassLoader
> Loaded classes:
> 1: de.stuefe.repros.MiscUtils$$Lambda$1/0x0000000801001448
> 2: de.stuefe.repros.MiscUtils
> 3: de.stuefe.repros.Simple2
> 4: de.stuefe.repros.Simple
> 5: de.stuefe.repros.SimpleBase
> 6: de.stuefe.repros.I2
> 7: de.stuefe.repros.I1
> -total-: 7 classes
> Non-Class:
> Usage by chunk level:
> 4m chunks: (none)
> 2m chunks: (none)
> 1m chunks: (none)
> 512k chunks: (none)
> 256k chunks: (none)
> 128k chunks: (none)
> 64k chunks: (none)
> 32k chunks: (none)
> 16k chunks: (none)
> 8k chunks: 1 chunk, 8,00 KB capacity, 8,00 KB
> (100%) committed, 8,00 KB (100%) used, 0 bytes ( 0%)
> free, 0 bytes ( 0%) waste
> 4k chunks: 1 chunk, 4,00 KB capacity, 4,00 KB
> (100%) committed, 256 bytes ( 6%) used, 3,75 KB ( 94%)
> free, 0 bytes ( 0%) waste
> 2k chunks: (none)
> 1k chunks: (none)
> -total-: 2 chunks, 12,00 KB capacity, 12,00
> KB (100%) committed, 8,25 KB ( 69%) used, 3,75 KB ( 31%)
> free, 0 bytes ( 0%) waste
> deallocated: 1 blocks with 24 bytes
>
> Class:
> Usage by chunk level:
> .... and so forth
> ```
>
> but for analyzing potential fragmentation issues (which have been
> rare since JEP 387) the "Waste" section at the end of the printout
> is much more helpful, e.g.:
>
> ```
> Waste (unused committed space):(percentages refer to total
> committed size 384,00 KB):
> Waste in chunks in use: 0 bytes ( 0%)
> Free in chunks in use: 85,01 KB ( 22%)
> In free chunks: 0 bytes ( 0%)
> Deallocated from chunks in use: 928 bytes ( <1%) (3 blocks)
> -total-: 85,91 KB ( 22%)
> ```
>
> It has been working effectively for several years and has helped
> many users solve metaspace-related problems.
> But a more user-friendly way is that JDK can inherently support
> this capability. We hope that format of the metaspace
> dump file can take both flexibility and compatibility into
> account, and the content of dump file should be detailed
> enough to meet the needs of both application developers and
> lower-level developers.
>
> Based on above considerations, I think using JSON as its file
> format is an appropriate solution(But XML or binary
> format are still not excluded as candidates). Specifically, in
> earlier thoughts, I thought the format of the metaspace
> file could be as follows(pretty printed)
>
> https://gist.github.com/y1yang0/ab3034b6381b8a9d215602c89af4e9c3
> <https://urldefense.com/v3/__https://gist.github.com/y1yang0/ab3034b6381b8a9d215602c89af4e9c3__;!!ACWV5N9M2RV99hQ!OOiFdPBm2uADyfD3T_L4JCOp6GC489SlckPnHoP6Q3GSqkmE9oBiBtQzm42_AWGASDogbJFRSVxHhIM6rMmRvrg$>
>
> Using the JSON format, we can flexibly add new fields without
> breaking compatibility. It is debatable as to which data
> to write. We can reach a consensus that third-party
> parsers(Metaspace Analyzer Tool) can at least reconstruct Java
> source code from the dump file. Based on this, we can write more
> useful information for low-level troubleshooting
> or debugging. (e.g. the init_state of InstanceKlass).
> In addition, we can even output the native code and associated
> information with regard to Method, third-party parser
> can reconstruct the human-readable assembly representation of the
> compiled method based on dump file. To some extent,
> we have implemented code cache dump by the way. For this reason,
> I'm not sure if the title of the RFC proposal should
> be called metaspace dump, maybe metadata dump? It looks more like
> a metadata-dump framework.
>
> Do you have any thoughts about metaspace/metadata dump? Looking
> forward to hearing your feedback, any comments are invaluable!
>
> Best regards,
> Yi Yang
>
>
> Analyzing the structure of generated classes sounds interesting,
> and could help with analyzing issues with bytecode instrumentation
> tools.
>
> For analyzing general metaspace OOMEs we are already covered quite
> well. Not perfect, but your proposal does intersect with existing
> tools a lot. To keep code complexity down, I'd rather avoid adding
> duplicate features.
>
> Cheers, Thomas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-dev/attachments/20230112/7564792e/attachment-0003.htm>
More information about the hotspot-dev
mailing list