Visual heap: card table, heap cards, roots, reference chains
Hi - this question comes out of personal and professional interest in the JVM's garbage collectors and how they work at the most detailed level possible. I'm wondering if anyone has developed a visualizer or, short of that, even a debug/trace log that could be visualized, which would show a deeper, more granular view of the JVM heap's micro-structures at runtime. It may sound a little nuts to people who are steeped in it, but, for educational and troubleshooting purposes, it would be very interesting to be able to visualize the runtime heap in Java at the macro level (generations) and at the micro-level (heap words, card table, GC roots and chain of references). I'm not expecting something like that would be usable in production or for very large heaps (unless it could be saved off in files and viewed offline). But I can imagine a view in a tool like VisualVM that shows the next level of detail of how the heap is structured at the micro-level and how it changes as programs execute and garbage collection occurs. Watching it in slow motion and replaying captured traces in offline mode would be interesting. I've been lurking on the list for a few. I have the OpenJDK jdk8 project compiling on Windows and a doxygen/GraphViz config for generating documentation and dependency graphs of the C++ code. I've been reading up on CMS, G1 and the GC papers and patents behind them (as well as the Mono/Windows CLR). My interest and experience with testing and tuning JVM heaps and GC for large e-comm sites hosted at AT&T has led to the questions and interest expressed above. Sorry to interject from out of left field. But I highly appreciate any pointers or thoughts on this. And I thank you for making and keeping this project open source. Kind regards, Scott Stirling Principal Technical Architect AT&T, Boston
Hi Scott, Tony Printezis did a lot of work on GC visualization (GCSpy) with his students while he was at Glasgow. There's probably a senior thesis or two from Glasgow, supervised by Tony, that may be of interest and probably a paper from Tony. As I recall, one of Richard Jones' students picked up that work and took it a bit further. (All this was more than 10-11 years ago or even earlier.) So talking with Tony or Richard Jones or their websites might provide some bkgrd... As you say, a good pedagogical tool... And worthwhile for that reason. --Ramki ysr1729 On Aug 14, 2013, at 9:27, "STIRLING, SCOTT" <ss4766@att.com> wrote:
Hi – this question comes out of personal and professional interest in the JVM’s garbage collectors and how they work at the most detailed level possible. I’m wondering if anyone has developed a visualizer or, short of that, even a debug/trace log that could be visualized, which would show a deeper, more granular view of the JVM heap’s micro-structures at runtime.
It may sound a little nuts to people who are steeped in it, but, for educational and troubleshooting purposes, it would be very interesting to be able to visualize the runtime heap in Java at the macro level (generations) and at the micro-level (heap words, card table, GC roots and chain of references). I’m not expecting something like that would be usable in production or for very large heaps (unless it could be saved off in files and viewed offline). But I can imagine a view in a tool like VisualVM that shows the next level of detail of how the heap is structured at the micro-level and how it changes as programs execute and garbage collection occurs. Watching it in slow motion and replaying captured traces in offline mode would be interesting.
I’ve been lurking on the list for a few. I have the OpenJDK jdk8 project compiling on Windows and a doxygen/GraphViz config for generating documentation and dependency graphs of the C++ code. I’ve been reading up on CMS, G1 and the GC papers and patents behind them (as well as the Mono/Windows CLR). My interest and experience with testing and tuning JVM heaps and GC for large e-comm sites hosted at AT&T has led to the questions and interest expressed above. Sorry to interject from out of left field. But I highly appreciate any pointers or thoughts on this. And I thank you for making and keeping this project open source.
Kind regards, Scott Stirling Principal Technical Architect AT&T, Boston
Hi Scott and Ramki, On Thu, Aug 15, 2013 at 2:01 PM, Srinivas Ramakrishna <ysr1729@gmail.com>wrote:
Hi Scott,
Tony Printezis did a lot of work on GC visualization (GCSpy) with his students while he was at Glasgow. There's probably a senior thesis or two from Glasgow, supervised by Tony, that may be of interest and probably a paper from Tony. As I recall, one of Richard Jones' students picked up that work and took it a bit further. (All this was more than 10-11 years ago or even earlier.)
So talking with Tony or Richard Jones or their websites might provide some bkgrd... As you say, a good pedagogical tool... And worthwhile for that reason.
--Ramki
Which reminds me of two discussion threads from last year [1][2], a GCSpy rewrite for HotSpot. I replied to both threads off-list, but I reckon that attempt to rewrite didn't work.
cc'ing Martin Skurla ("Crazy Java") to see if he actually made any progress on it. - Kris [1]: http://mail.openjdk.java.net/pipermail/serviceability-dev/2012-March/005465.... [2]: http://mail.openjdk.java.net/pipermail/serviceability-dev/2012-March/005516....
ysr1729
On Aug 14, 2013, at 9:27, "STIRLING, SCOTT" <ss4766@att.com> wrote:
Hi – this question comes out of personal and professional interest in the JVM’s garbage collectors and how they work at the most detailed level possible. I’m wondering if anyone has developed a visualizer or, short of that, even a debug/trace log that could be visualized, which would show a deeper, more granular view of the JVM heap’s micro-structures at runtime.* ***
** **
It may sound a little nuts to people who are steeped in it, but, for educational and troubleshooting purposes, it would be very interesting to be able to visualize the runtime heap in Java at the macro level (generations) and at the micro-level (heap words, card table, GC roots and chain of references). I’m not expecting something like that would be usable in production or for very large heaps (unless it could be saved off in files and viewed offline). But I can imagine a view in a tool like VisualVM that shows the next level of detail of how the heap is structured at the micro-level and how it changes as programs execute and garbage collection occurs. Watching it in slow motion and replaying captured traces in offline mode would be interesting.****
** **
I’ve been lurking on the list for a few. I have the OpenJDK jdk8 project compiling on Windows and a doxygen/GraphViz config for generating documentation and dependency graphs of the C++ code. I’ve been reading up on CMS, G1 and the GC papers and patents behind them (as well as the Mono/Windows CLR). My interest and experience with testing and tuning JVM heaps and GC for large e-comm sites hosted at AT&T has led to the questions and interest expressed above. Sorry to interject from out of left field. But I highly appreciate any pointers or thoughts on this. And I thank you for making and keeping this project open source.****
** **
Kind regards,****
Scott Stirling****
Principal Technical Architect****
AT&T, Boston ****
This is great, Ramki and Krystal. Thank you very much. Found some screenshots of GCSpy and that's pretty much exactly what I had in mind! Here's the GCSpy page at http://www.cs.kent.ac.uk/projects/gc/gcspy/, though it's several years old. GCSpy, it seems, was built to run on 32 bit Linux/BSD systems. The source is available. There's some C, C++ and Java source in the codebase. Sure seems like a good place to start, though. I read Martin Skurla's emails too. Looks like he was working with the serviceability-dev list, and was making good progress, but got stuck (last message there) on tracking references when heap size grows dynamically and memory addresses change. I pretty much always set -Xms and -Xmx to the same value, which would be an acceptable limitation to work around that issue, in my opinion. Thanks again. I will reach out to Mark offline too. Kind regards, Scott Stirling AT&T, Boston From: Krystal Mok [mailto:rednaxelafx@gmail.com] Sent: Thursday, August 15, 2013 11:36 AM To: Srinivas Ramakrishna Cc: STIRLING, SCOTT; hotspot-gc-dev@openjdk.java.net; Crazy Java Subject: Re: Visual heap: card table, heap cards, roots, reference chains Hi Scott and Ramki, On Thu, Aug 15, 2013 at 2:01 PM, Srinivas Ramakrishna <ysr1729@gmail.com<mailto:ysr1729@gmail.com>> wrote: Hi Scott, Tony Printezis did a lot of work on GC visualization (GCSpy) with his students while he was at Glasgow. There's probably a senior thesis or two from Glasgow, supervised by Tony, that may be of interest and probably a paper from Tony. As I recall, one of Richard Jones' students picked up that work and took it a bit further. (All this was more than 10-11 years ago or even earlier.) So talking with Tony or Richard Jones or their websites might provide some bkgrd... As you say, a good pedagogical tool... And worthwhile for that reason. --Ramki Which reminds me of two discussion threads from last year [1][2], a GCSpy rewrite for HotSpot. I replied to both threads off-list, but I reckon that attempt to rewrite didn't work. cc'ing Martin Skurla ("Crazy Java") to see if he actually made any progress on it. - Kris [1]: http://mail.openjdk.java.net/pipermail/serviceability-dev/2012-March/005465.... [2]: http://mail.openjdk.java.net/pipermail/serviceability-dev/2012-March/005516.... ysr1729 On Aug 14, 2013, at 9:27, "STIRLING, SCOTT" <ss4766@att.com<mailto:ss4766@att.com>> wrote: Hi - this question comes out of personal and professional interest in the JVM's garbage collectors and how they work at the most detailed level possible. I'm wondering if anyone has developed a visualizer or, short of that, even a debug/trace log that could be visualized, which would show a deeper, more granular view of the JVM heap's micro-structures at runtime. It may sound a little nuts to people who are steeped in it, but, for educational and troubleshooting purposes, it would be very interesting to be able to visualize the runtime heap in Java at the macro level (generations) and at the micro-level (heap words, card table, GC roots and chain of references). I'm not expecting something like that would be usable in production or for very large heaps (unless it could be saved off in files and viewed offline). But I can imagine a view in a tool like VisualVM that shows the next level of detail of how the heap is structured at the micro-level and how it changes as programs execute and garbage collection occurs. Watching it in slow motion and replaying captured traces in offline mode would be interesting. I've been lurking on the list for a few. I have the OpenJDK jdk8 project compiling on Windows and a doxygen/GraphViz config for generating documentation and dependency graphs of the C++ code. I've been reading up on CMS, G1 and the GC papers and patents behind them (as well as the Mono/Windows CLR). My interest and experience with testing and tuning JVM heaps and GC for large e-comm sites hosted at AT&T has led to the questions and interest expressed above. Sorry to interject from out of left field. But I highly appreciate any pointers or thoughts on this. And I thank you for making and keeping this project open source. Kind regards, Scott Stirling Principal Technical Architect AT&T, Boston
Hi, On Mon, 2013-08-19 at 13:20 +0000, STIRLING, SCOTT wrote:
This is great, Ramki and Krystal. Thank you very much. Found some screenshots of GCSpy and that’s pretty much exactly what I had in mind!
Here’s the GCSpy page at http://www.cs.kent.ac.uk/projects/gc/gcspy/, though it’s several years old.
GCSpy, it seems, was built to run on 32 bit Linux/BSD systems. The source is available. There’s some C, C++ and Java source in the codebase. Sure seems like a good place to start, though.
Note that GCSpy does not ship with any of the integrations the papers use as far as I remember. They probably would not compile anyway due to their age. E.g. using gcspy with jikes crashes it the last time I tried (already > 1 year ago). You may want to take a look at http://ssw.uni-linz.ac.at/General/Staff/TS/ that contains a source ball for a more up-to-date version with some UI bugs fixed and some enhancements which may or may not be useful. Hth, Thomas
participants (4)
-
Krystal Mok
-
Srinivas Ramakrishna
-
STIRLING, SCOTT
-
Thomas Schatzl