From spammails at web.de  Tue Nov  6 10:49:40 2018
From: spammails at web.de (Christian)
Date: Tue, 6 Nov 2018 11:49:40 +0100
Subject: List of latest OpenJDK versions
Message-ID: <631FEFB8-C114-492C-A588-61B26A2CA6D0@web.de>

Hi,
is there a website or an API call where I can get the LATEST version number of each major OpenJDK release (12, 11, 10, ..) programmatically?

If not, is there a stable URL to always download the latest officially released OpenJDK version?

I want to use this to make sure that my server installations are up to date.

Best Regards,

Christian

Sent from my iPhone

From forax at univ-mlv.fr  Tue Nov  6 10:57:12 2018
From: forax at univ-mlv.fr (Remi Forax)
Date: Tue, 6 Nov 2018 11:57:12 +0100 (CET)
Subject: List of latest OpenJDK versions
In-Reply-To: <631FEFB8-C114-492C-A588-61B26A2CA6D0@web.de>
References: <631FEFB8-C114-492C-A588-61B26A2CA6D0@web.de>
Message-ID: <191328267.397009.1541501832185.JavaMail.zimbra@u-pem.fr>

Hi Christian,
i use https://github.com/sormuras/bach/blob/master/install-jdk.sh which is maintained by another Chistian
(and used by TravisCI)

cheers,
R?mi

----- Mail original -----
> De: "Christian" <spammails at web.de>
> ?: "discuss" <discuss at openjdk.java.net>
> Envoy?: Mardi 6 Novembre 2018 11:49:40
> Objet: List of latest OpenJDK versions

> Hi,
> is there a website or an API call where I can get the LATEST version number of
> each major OpenJDK release (12, 11, 10, ..) programmatically?
> 
> If not, is there a stable URL to always download the latest officially released
> OpenJDK version?
> 
> I want to use this to make sure that my server installations are up to date.
> 
> Best Regards,
> 
> Christian
> 
> Sent from my iPhone

From lars.francke at gmail.com  Fri Nov  9 10:03:38 2018
From: lars.francke at gmail.com (Lars Francke)
Date: Fri, 9 Nov 2018 11:03:38 +0100
Subject: Introduction
Message-ID: <CAD-Ua_j3X+JJYLHg9Jcgu7Ga9VU51EMLufQnuoa1TP4fyBBV8w@mail.gmail.com>

Hi everyone,

I just filed my OCA and Dalibor suggested that I introduce myself here so
I'll do just that.

I'm based in Hamburg, Germany and have spent almost all my professional
life self-employed[1] working in the "Big Data" (it's a terrible buzzword)
/ Hadoop field (I started ~2008). I'm a committer on Apache HBase, Hive and
ORC as well as an Apache Foundation member and a contributor to a bunch of
other tools (Knox, NiFi, Hadoop etc.).

As I'm helping clients mostly with troubleshooting and fire fighting I have
to dive into (Open)JDK implementation details every once in a while.
Unfortunately this is often Kerberos related.

Hence my first attempt at a contribution will be a simple additional log
message in the JDK that would have saved me an hour of searching around.
I've already asked for comments on security-dev.

OpenJDK as an ecosystem is pretty large so it'll take me a while to get
started. Please excuse any beginner mistakes I might make. On the other
hand I'm happy to help with any of the tools (list above) so just shoot me
an email if you've got a question.

Cheers,
Lars

[1] first alone and now with a colleague in our own small company <
http://www.opencore.com>

From martijnverburg at gmail.com  Fri Nov  9 10:08:40 2018
From: martijnverburg at gmail.com (Martijn Verburg)
Date: Fri, 9 Nov 2018 10:08:40 +0000
Subject: Introduction
In-Reply-To: <CAD-Ua_j3X+JJYLHg9Jcgu7Ga9VU51EMLufQnuoa1TP4fyBBV8w@mail.gmail.com>
References: <CAD-Ua_j3X+JJYLHg9Jcgu7Ga9VU51EMLufQnuoa1TP4fyBBV8w@mail.gmail.com>
Message-ID: <CAP7YuATwM8xMOKoAYRDu9hcwgXXCOAi7jF9qGV2wxQ9mXVEwfA@mail.gmail.com>

Hi Lars and welcome!

There'as an adoption-discuss mailing list that you might find help for
queries like "I'm looking at the code in area X, what mailing list do I
post to for clarification?"

Cheers,
Martijn


On Fri, 9 Nov 2018 at 10:05, Lars Francke <lars.francke at gmail.com> wrote:

> Hi everyone,
>
> I just filed my OCA and Dalibor suggested that I introduce myself here so
> I'll do just that.
>
> I'm based in Hamburg, Germany and have spent almost all my professional
> life self-employed[1] working in the "Big Data" (it's a terrible buzzword)
> / Hadoop field (I started ~2008). I'm a committer on Apache HBase, Hive and
> ORC as well as an Apache Foundation member and a contributor to a bunch of
> other tools (Knox, NiFi, Hadoop etc.).
>
> As I'm helping clients mostly with troubleshooting and fire fighting I have
> to dive into (Open)JDK implementation details every once in a while.
> Unfortunately this is often Kerberos related.
>
> Hence my first attempt at a contribution will be a simple additional log
> message in the JDK that would have saved me an hour of searching around.
> I've already asked for comments on security-dev.
>
> OpenJDK as an ecosystem is pretty large so it'll take me a while to get
> started. Please excuse any beginner mistakes I might make. On the other
> hand I'm happy to help with any of the tools (list above) so just shoot me
> an email if you've got a question.
>
> Cheers,
> Lars
>
> [1] first alone and now with a colleague in our own small company <
> http://www.opencore.com>
>

From uschindler at apache.org  Fri Nov  9 19:07:11 2018
From: uschindler at apache.org (Uwe Schindler)
Date: Fri, 9 Nov 2018 20:07:11 +0100
Subject: Introduction: Uwe Schindler
Message-ID: <000101d4785f$6c390750$44ab15f0$@apache.org>

Hi everyone,

I just filed my OCA and Dalibor suggested that I introduce myself here so
I'll do just that. Thanks to Rory O'Donnell for inviting me to help in the
OpenJDK project as a contributor, mostly with testing, bug reporting, but for
sure also some patches. 

I'm based in Bremen, Germany. I'm a committer and member of the project
management committee of Apache Lucene and Solr, Apache POI; I manage the
Policeman ForbiddenAPIs Checker; I am also a contributor to Elasticsearch and
Apache TIKA.

Many of you may already know me from bug reports about issues in preview
builds of OpenJDK that caused issues with Hotspot in Apache Lucene
infrastructure (including Solr and Elasticsearch). I am one of the designers
of Lucene's randomized testing framework that made it possible to detect
those bugs. On my consulting company's servers the 24/7 testing of Apache
Lucene is running.

I work at two places: The University of Bremen, where I am maintaining the
PANGAEA Data Library [1]. I also have a company that provides consulting and
support for Apache Lucene, Apache Solr, Elasticsearch [2]. I help users with
setting up the deployments, give them support when configuring their JVM and
allow them to manage huge indexes solely with mmapped directories.

Cheers,
Uwe

[1] https://www.pangaea.de
[2] https://www.sd-datasolutions.de 

-----
Uwe Schindler
uschindler at apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/



From martijnverburg at gmail.com  Fri Nov  9 20:12:00 2018
From: martijnverburg at gmail.com (Martijn Verburg)
Date: Fri, 9 Nov 2018 20:12:00 +0000
Subject: Introduction: Uwe Schindler
In-Reply-To: <000101d4785f$6c390750$44ab15f0$@apache.org>
References: <000101d4785f$6c390750$44ab15f0$@apache.org>
Message-ID: <CAP7YuASX6sHFdt=Q89s-opwvVVFH=DVbx8UqU1cjeKgSVuSijw@mail.gmail.com>

Hi Uwe and welcome to OpenJDK!

You may find adoption-discuss a useful place to ask questions like "How do
I get the class library sources into IntelliJ?" etc :-)

Cheers,
Martijn


On Fri, 9 Nov 2018 at 19:08, Uwe Schindler <uschindler at apache.org> wrote:

> Hi everyone,
>
> I just filed my OCA and Dalibor suggested that I introduce myself here so
> I'll do just that. Thanks to Rory O'Donnell for inviting me to help in the
> OpenJDK project as a contributor, mostly with testing, bug reporting, but
> for
> sure also some patches.
>
> I'm based in Bremen, Germany. I'm a committer and member of the project
> management committee of Apache Lucene and Solr, Apache POI; I manage the
> Policeman ForbiddenAPIs Checker; I am also a contributor to Elasticsearch
> and
> Apache TIKA.
>
> Many of you may already know me from bug reports about issues in preview
> builds of OpenJDK that caused issues with Hotspot in Apache Lucene
> infrastructure (including Solr and Elasticsearch). I am one of the
> designers
> of Lucene's randomized testing framework that made it possible to detect
> those bugs. On my consulting company's servers the 24/7 testing of Apache
> Lucene is running.
>
> I work at two places: The University of Bremen, where I am maintaining the
> PANGAEA Data Library [1]. I also have a company that provides consulting
> and
> support for Apache Lucene, Apache Solr, Elasticsearch [2]. I help users
> with
> setting up the deployments, give them support when configuring their JVM
> and
> allow them to manage huge indexes solely with mmapped directories.
>
> Cheers,
> Uwe
>
> [1] https://www.pangaea.de
> [2] https://www.sd-datasolutions.de
>
> -----
> Uwe Schindler
> uschindler at apache.org
> ASF Member, Apache Lucene PMC / Committer
> Bremen, Germany
> http://lucene.apache.org/
>
>
>

From jcbeyler at google.com  Mon Nov 19 18:47:10 2018
From: jcbeyler at google.com (JC Beyler)
Date: Mon, 19 Nov 2018 10:47:10 -0800
Subject: Call For Discussion: New Project: Atlantis
Message-ID: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>

Hi all,

I would like to start a discussion about the possible creation of the
Atlantis Project with myself, Jean Christophe Beyler, as the Lead and the
Serviceability group as the sponsoring group.

In accordance with the OpenJDK guidelines [1], this project would provide a
venue to explore and incubate monitoring and performance analysis features
that could be integrated into the Hotspot JVM and the JVM Tool Interface.
This includes working, evaluating, and incubating a Thread-Sanitizer
implementation [2] for Java.

The Google platform team has historically worked on providing new features,
including monitoring hooks due to certain monitoring features not being
available in a low-overhead, turned on-by-default manner. Therefore, for
Google Java users, the platform team augmented or added mechanisms to
assess various metrics such as but not limited to:

- Thread Sanitizer support for Java (see JDK-8208520 [2]) (though this one
specifically is not a low-overhead one, sorry ;-))
- Lock contention profiles to better understand where threads are spending
time in locks
- Heap dumping mechanisms to improve heap dump times for large heaps
- Efficient thread tagging to help filter profiling
- etc.

All of the above have added changes to the JDK in various forms and the
Atlantis project proposes to investigate, in the open, how these changes
could be made general enough to be pushed into the mainline, or should be
dropped due to being too specific or could be done using other existing
mechanisms. The project would not be a Google-only project and will have a
mailing list to discuss and potentially accept any contribution trying to
extend and/or enhance current monitoring systems.

The evaluation criteria to consider a feature/improvement includes but
would not be limited to:

     * Performance: metrics and new hooks should not be hurting performance
when disabled
     * General usefulness for the community at large
     * No working alternatives: sometimes it may be that there are new
technologies available (ie JFR events for example) that might be leveraged
to deprecate existing solutions

If one or more prototypes indicate a new metric, a new monitoring system,
or a feature offers substantial improvements, the Atlantis project would
shepherd a JEP to integrate the change into the mainline of OpenJDK.

The current suggested initial Reviewers and Committers would be but is most
likely not going to be limited to:
* Jean Christophe Beyler
* Serguei Spitsyn
* Chuck Rasbold
* Liam Cushon

What do you all think? :-)

Thanks,
Jc

[1] http://openjdk.java.net/projects/#new-project
[2] https://bugs.openjdk.java.net/browse/JDK-8208520

From thomas.stuefe at gmail.com  Tue Nov 20 14:38:30 2018
From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=)
Date: Tue, 20 Nov 2018 15:38:30 +0100
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
Message-ID: <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>

In general I like this, it is certainly worth a try.

In an ideal world, there would be no need for this: serviceability-dev
would be the place to be, full of both knowledgeable outsiders and
friendly reviewers with lots of spare time on their hand :) however,
in reality we do not have that many outsiders (those who are here are
valuable, but they are few), and we lack reviewers.

By concentrating discussions on larger features into an own project
with an own mailing list maybe we could speed up incubation of larger
features, especially when contributed from outside Oracle. Similar
important is speedy rejection for features that just do not fit, since
it can save time and hurt on the contributors side.

I'm worried about fragmentation though. Once we have this project,
what is then the role of serviceability-dev? Just smallish everyday
fixes? Also, the fact that VM developer feedback is hard to come by is
mostly due to a lack of time on their part, and a new mailing list
will not fix that.

But as I wrote, I think it is worth a try.

Thanks, Thomas

On Mon, Nov 19, 2018 at 7:48 PM JC Beyler <jcbeyler at google.com> wrote:
>
> Hi all,
>
> I would like to start a discussion about the possible creation of the
> Atlantis Project with myself, Jean Christophe Beyler, as the Lead and the
> Serviceability group as the sponsoring group.
>
> In accordance with the OpenJDK guidelines [1], this project would provide a
> venue to explore and incubate monitoring and performance analysis features
> that could be integrated into the Hotspot JVM and the JVM Tool Interface.
> This includes working, evaluating, and incubating a Thread-Sanitizer
> implementation [2] for Java.
>
> The Google platform team has historically worked on providing new features,
> including monitoring hooks due to certain monitoring features not being
> available in a low-overhead, turned on-by-default manner. Therefore, for
> Google Java users, the platform team augmented or added mechanisms to
> assess various metrics such as but not limited to:
>
> - Thread Sanitizer support for Java (see JDK-8208520 [2]) (though this one
> specifically is not a low-overhead one, sorry ;-))
> - Lock contention profiles to better understand where threads are spending
> time in locks
> - Heap dumping mechanisms to improve heap dump times for large heaps
> - Efficient thread tagging to help filter profiling
> - etc.
>
> All of the above have added changes to the JDK in various forms and the
> Atlantis project proposes to investigate, in the open, how these changes
> could be made general enough to be pushed into the mainline, or should be
> dropped due to being too specific or could be done using other existing
> mechanisms. The project would not be a Google-only project and will have a
> mailing list to discuss and potentially accept any contribution trying to
> extend and/or enhance current monitoring systems.
>
> The evaluation criteria to consider a feature/improvement includes but
> would not be limited to:
>
>      * Performance: metrics and new hooks should not be hurting performance
> when disabled
>      * General usefulness for the community at large
>      * No working alternatives: sometimes it may be that there are new
> technologies available (ie JFR events for example) that might be leveraged
> to deprecate existing solutions
>
> If one or more prototypes indicate a new metric, a new monitoring system,
> or a feature offers substantial improvements, the Atlantis project would
> shepherd a JEP to integrate the change into the mainline of OpenJDK.
>
> The current suggested initial Reviewers and Committers would be but is most
> likely not going to be limited to:
> * Jean Christophe Beyler
> * Serguei Spitsyn
> * Chuck Rasbold
> * Liam Cushon
>
> What do you all think? :-)
>
> Thanks,
> Jc
>
> [1] http://openjdk.java.net/projects/#new-project
> [2] https://bugs.openjdk.java.net/browse/JDK-8208520

From neugens at redhat.com  Tue Nov 20 15:14:11 2018
From: neugens at redhat.com (Mario Torre)
Date: Tue, 20 Nov 2018 16:14:11 +0100
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>
Message-ID: <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>

On Tue, 2018-11-20 at 15:38 +0100, Thomas St?fe wrote:
> I'm worried about fragmentation though. Once we have this project,
> what is then the role of serviceability-dev? Just smallish everyday
> fixes? Also, the fact that VM developer feedback is hard to come by
> is
> mostly due to a lack of time on their part, and a new mailing list
> will not fix that.

Generally projects have their own mailing lists for development until
they are "finished", I would expect that for this project it would work
the same, once it's integrated into mainline it becomes part of the
serviceability code and the project mailing list is folded into
serviceability-dev, until then it's somewhat an independent unit.

Cheers,
Mario
-- 
Mario Torre
Associate Manager, Software Engineering
Red Hat GmbH <https://www.redhat.com>
9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898


From adinn at redhat.com  Tue Nov 20 15:38:35 2018
From: adinn at redhat.com (Andrew Dinn)
Date: Tue, 20 Nov 2018 15:38:35 +0000
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>
 <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>
Message-ID: <91e0480a-a593-13d4-5b0c-b45bc7d4563c@redhat.com>

On 20/11/2018 15:14, Mario Torre wrote:
> On Tue, 2018-11-20 at 15:38 +0100, Thomas St?fe wrote:
>> I'm worried about fragmentation though. Once we have this project,
>> what is then the role of serviceability-dev? Just smallish everyday
>> fixes? Also, the fact that VM developer feedback is hard to come by
>> is
>> mostly due to a lack of time on their part, and a new mailing list
>> will not fix that.
> 
> Generally projects have their own mailing lists for development until
> they are "finished", I would expect that for this project it would work
> the same, once it's integrated into mainline it becomes part of the
> serviceability code and the project mailing list is folded into
> serviceability-dev, until then it's somewhat an independent unit.
Well, there is a requirement that anything new of a significant size orc
complexity that gets developed under a JEP and then integrated into the
mainline repos will have someone available to maintain it.

JC, can you provide assurance that Google staff involved in pushing any
code upstream will also be available to help perform that maintenance
effort, in this case presumably as members of the serviceability team?

n.b. Modulo the above detail I'm all for this proposal. I'd be very
pleased if the project brings opportunities for Google either to
upstream their existing work to everyone's benefit or, where for
whatever reason that is not feasible, to help tailor existing upstream
capabilities (such as JFR events) so they might encompass Google's
downstream needs.

It's very welcome to see Google so willing to contribute to OpenJDK.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

From hohensee at amazon.com  Tue Nov 20 15:52:50 2018
From: hohensee at amazon.com (Hohensee, Paul)
Date: Tue, 20 Nov 2018 15:52:50 +0000
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <91e0480a-a593-13d4-5b0c-b45bc7d4563c@redhat.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>
 <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>
 <91e0480a-a593-13d4-5b0c-b45bc7d4563c@redhat.com>
Message-ID: <35D18349-1BDC-4BDA-ACAA-A31511724ED0@amazon.com>

+1. I'd be happy to participate.

Thanks,

Paul

?On 11/20/18, 7:40 AM, "discuss on behalf of Andrew Dinn" <discuss-bounces at openjdk.java.net on behalf of adinn at redhat.com> wrote:

    On 20/11/2018 15:14, Mario Torre wrote:
    > On Tue, 2018-11-20 at 15:38 +0100, Thomas St?fe wrote:
    >> I'm worried about fragmentation though. Once we have this project,
    >> what is then the role of serviceability-dev? Just smallish everyday
    >> fixes? Also, the fact that VM developer feedback is hard to come by
    >> is
    >> mostly due to a lack of time on their part, and a new mailing list
    >> will not fix that.
    > 
    > Generally projects have their own mailing lists for development until
    > they are "finished", I would expect that for this project it would work
    > the same, once it's integrated into mainline it becomes part of the
    > serviceability code and the project mailing list is folded into
    > serviceability-dev, until then it's somewhat an independent unit.
    Well, there is a requirement that anything new of a significant size orc
    complexity that gets developed under a JEP and then integrated into the
    mainline repos will have someone available to maintain it.
    
    JC, can you provide assurance that Google staff involved in pushing any
    code upstream will also be available to help perform that maintenance
    effort, in this case presumably as members of the serviceability team?
    
    n.b. Modulo the above detail I'm all for this proposal. I'd be very
    pleased if the project brings opportunities for Google either to
    upstream their existing work to everyone's benefit or, where for
    whatever reason that is not feasible, to help tailor existing upstream
    capabilities (such as JFR events) so they might encompass Google's
    downstream needs.
    
    It's very welcome to see Google so willing to contribute to OpenJDK.
    
    regards,
    
    
    Andrew Dinn
    -----------
    Senior Principal Software Engineer
    Red Hat UK Ltd
    Registered in England and Wales under Company Registration No. 03798903
    Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
    


From neugens at redhat.com  Tue Nov 20 16:21:55 2018
From: neugens at redhat.com (Mario Torre)
Date: Tue, 20 Nov 2018 17:21:55 +0100
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <91e0480a-a593-13d4-5b0c-b45bc7d4563c@redhat.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>
 <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>
 <91e0480a-a593-13d4-5b0c-b45bc7d4563c@redhat.com>
Message-ID: <56eaa0de0d7d095413be0c1a89c0efe071669620.camel@redhat.com>

On Tue, 2018-11-20 at 15:38 +0000, Andrew Dinn wrote:
> On 20/11/2018 15:14, Mario Torre wrote:
> > On Tue, 2018-11-20 at 15:38 +0100, Thomas St?fe wrote:
> > > I'm worried about fragmentation though. Once we have this
> > > project,
> > > what is then the role of serviceability-dev? Just smallish
> > > everyday
> > > fixes? Also, the fact that VM developer feedback is hard to come
> > > by
> > > is
> > > mostly due to a lack of time on their part, and a new mailing
> > > list
> > > will not fix that.
> > 
> > Generally projects have their own mailing lists for development
> > until
> > they are "finished", I would expect that for this project it would
> > work
> > the same, once it's integrated into mainline it becomes part of the
> > serviceability code and the project mailing list is folded into
> > serviceability-dev, until then it's somewhat an independent unit.
> 
> Well, there is a requirement that anything new of a significant size
> orc
> complexity that gets developed under a JEP and then integrated into
> the
> mainline repos will have someone available to maintain it.

Yes, indeed!

> JC, can you provide assurance that Google staff involved in pushing
> any
> code upstream will also be available to help perform that maintenance
> effort, in this case presumably as members of the serviceability
> team?
> 
> n.b. Modulo the above detail I'm all for this proposal. I'd be very
> pleased if the project brings opportunities for Google either to
> upstream their existing work to everyone's benefit or, where for
> whatever reason that is not feasible, to help tailor existing
> upstream
> capabilities (such as JFR events) so they might encompass Google's
> downstream needs.
> 
> It's very welcome to see Google so willing to contribute to OpenJDK.

I agree!

I actually wanted to ask that one written requirement in the JEP would
be to extend Flight Recorder whenever it makes sense, even if this
means going an extra step (in other words, the default option), so that
we avoid fragmentation and the proliferation of a multitude of tools.

I understand the project is meant to be some sort of umbrealla for
experiementation? This will probably already cure most of the risk of
such fragmentation, however, if it's a written statement in the JEP
goals it makes more explicit that we are interested in enhancing the
current frameworks rather than creating alternative versions (unless it
*really* makes sense to do so).

Cheers,
Mario
-- 
Mario Torre
Associate Manager, Software Engineering
Red Hat GmbH <https://www.redhat.com>
9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898


From thomas.stuefe at gmail.com  Tue Nov 20 16:41:04 2018
From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=)
Date: Tue, 20 Nov 2018 17:41:04 +0100
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>
 <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>
Message-ID: <CAA-vtUz3+03jeONsFczscdjfgGyLO+fmJvOtVoP6wMVz--VCYg@mail.gmail.com>

On Tue, Nov 20, 2018 at 4:14 PM Mario Torre <neugens at redhat.com> wrote:
>
> On Tue, 2018-11-20 at 15:38 +0100, Thomas St?fe wrote:
> > I'm worried about fragmentation though. Once we have this project,
> > what is then the role of serviceability-dev? Just smallish everyday
> > fixes? Also, the fact that VM developer feedback is hard to come by
> > is
> > mostly due to a lack of time on their part, and a new mailing list
> > will not fix that.
>
> Generally projects have their own mailing lists for development until
> they are "finished", I would expect that for this project it would work
> the same, once it's integrated into mainline it becomes part of the
> serviceability code and the project mailing list is folded into
> serviceability-dev, until then it's somewhat an independent unit.
>

Thanks for clarifying, Mario.

..Thomas

> Cheers,
> Mario
> --
> Mario Torre
> Associate Manager, Software Engineering
> Red Hat GmbH <https://www.redhat.com>
> 9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898
>

From jesper.wilhelmsson at oracle.com  Tue Nov 20 18:41:30 2018
From: jesper.wilhelmsson at oracle.com (jesper.wilhelmsson at oracle.com)
Date: Tue, 20 Nov 2018 19:41:30 +0100
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <56eaa0de0d7d095413be0c1a89c0efe071669620.camel@redhat.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>
 <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>
 <91e0480a-a593-13d4-5b0c-b45bc7d4563c@redhat.com>
 <56eaa0de0d7d095413be0c1a89c0efe071669620.camel@redhat.com>
Message-ID: <2806730D-AEBF-4491-8442-F81AA2C1966D@oracle.com>

> On 20 Nov 2018, at 17:21, Mario Torre <neugens at redhat.com> wrote:
> 
> On Tue, 2018-11-20 at 15:38 +0000, Andrew Dinn wrote:
>> On 20/11/2018 15:14, Mario Torre wrote:
>>> On Tue, 2018-11-20 at 15:38 +0100, Thomas St?fe wrote:
>>>> I'm worried about fragmentation though. Once we have this
>>>> project,
>>>> what is then the role of serviceability-dev? Just smallish
>>>> everyday
>>>> fixes? Also, the fact that VM developer feedback is hard to come
>>>> by
>>>> is
>>>> mostly due to a lack of time on their part, and a new mailing
>>>> list
>>>> will not fix that.
>>> 
>>> Generally projects have their own mailing lists for development
>>> until
>>> they are "finished", I would expect that for this project it would
>>> work
>>> the same, once it's integrated into mainline it becomes part of the
>>> serviceability code and the project mailing list is folded into
>>> serviceability-dev, until then it's somewhat an independent unit.
>> 
>> Well, there is a requirement that anything new of a significant size
>> orc
>> complexity that gets developed under a JEP and then integrated into
>> the
>> mainline repos will have someone available to maintain it.
> 
> Yes, indeed!
> 
>> JC, can you provide assurance that Google staff involved in pushing
>> any
>> code upstream will also be available to help perform that maintenance
>> effort, in this case presumably as members of the serviceability
>> team?
>> 
>> n.b. Modulo the above detail I'm all for this proposal. I'd be very
>> pleased if the project brings opportunities for Google either to
>> upstream their existing work to everyone's benefit or, where for
>> whatever reason that is not feasible, to help tailor existing
>> upstream
>> capabilities (such as JFR events) so they might encompass Google's
>> downstream needs.
>> 
>> It's very welcome to see Google so willing to contribute to OpenJDK.
> 
> I agree!
> 
> I actually wanted to ask that one written requirement in the JEP would
> be to extend Flight Recorder whenever it makes sense, even if this
> means going an extra step (in other words, the default option), so that
> we avoid fragmentation and the proliferation of a multitude of tools.

+1

Given that it is an explicit goal to extend and utilize existing frameworks rather than adding another one, I welcome this project.
/Jesper

> 
> I understand the project is meant to be some sort of umbrealla for
> experiementation? This will probably already cure most of the risk of
> such fragmentation, however, if it's a written statement in the JEP
> goals it makes more explicit that we are interested in enhancing the
> current frameworks rather than creating alternative versions (unless it
> *really* makes sense to do so).
> 
> Cheers,
> Mario
> --
> Mario Torre
> Associate Manager, Software Engineering
> Red Hat GmbH <https://www.redhat.com <https://www.redhat.com/>>
> 9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898


From jcbeyler at google.com  Tue Nov 20 20:36:16 2018
From: jcbeyler at google.com (JC Beyler)
Date: Tue, 20 Nov 2018 12:36:16 -0800
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <2806730D-AEBF-4491-8442-F81AA2C1966D@oracle.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>
 <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>
 <91e0480a-a593-13d4-5b0c-b45bc7d4563c@redhat.com>
 <56eaa0de0d7d095413be0c1a89c0efe071669620.camel@redhat.com>
 <2806730D-AEBF-4491-8442-F81AA2C1966D@oracle.com>
Message-ID: <CAF9BGByAv=m3L-LKNjU0whMeL58JL8naW2Y0++OkSU6CArTTtQ@mail.gmail.com>

Thanks all for the questions and feedback. I've tried to summarize the
questions/concerns here:

- I want to be really clear that I don't have all the answers, I believe
very strongly in trial and error
    - Part of this project's motivation is to have these conversations :)
    - Another part is that I think there is something to be gained by this
project and it is exactly what *Mario* has said above: that there be an
"umbrella for experimentation; This will probably already cure most of the
risk of such fragmentation, however, if it's a written statement in the JEP
goals it makes more explicit that we are interested in enhancing the
current frameworks rather than creating alternative versions (unless it
*really* makes sense to do so)."
        -> Exactly, I'm not interested in creating the next "shiny thing"
that actually does the same thing as existing items; I am interested in
looking at an issue/monitoring feature; considering all existing
frameworks; seeing where it would fit best; figure out if it is feasible
acceptable; then move forward".

- (*From Thomas *mainly but also *Mario*) Is there a risk of fragmentation
of knowledge or fragmenting serviceability?
    - I don't think so nor do I want to; I'm actually against fragmentation
in almost all cases to be very honest and clear; that is why I've tried to
work closely with the serviceability team and engineers to figure out what
could/should be the scope of this project
    - Let us not forget that in the end, the work done in this project
still needs a JEP to get into the mainline and that is our last
barrier/hurdle that ensures consistency in OpenJDK and not fragmentation
    - To finish this subject (and re-iterate), if an item is to do
"yet-another-monitoring/performance-analysis tool" that could be done by
the various ones we have already available, I believe it has no place in
this project; on the other hand, if it can be proved that X cannot be done
at all in a product environment with current systems then we can/should
start asking the questions:
         - Is there a solution close, how could it be extended, what do the
current developers of that mechanism think, should we try to prototype
something to see the extent of changes, should we push it to being a JEP
from this project? It is for those cases that I see a path forward via this
project.
    - I think that the balance between what could go directly to mainline
or via this project has to be found, I don't have the right answer, I have
the items I know I'd like to work on openly via this project and then see
what comes from them; but I'm open to suggestions :-)
       - Which means I don't have the right answer of what should go
directly to serviceability-dev or here when it's a "Hey I have this" or
"Hey could we that" type of question; I think we will see what happens,
right now, I'm personally going to concentrate on the internal features we
have at Google to start the conversation and the project work and see where
that takes me/us if that helps. I'm open to any explicit guideline that is
deemed necessary/warranted or we can do what I would hope is possible too:
by having myself, Serguei, and core members of serviceability, we will work
out the balance together as we move forward.

*From Andrew *mainly*:* I'm going to quote Andrew's question/comment here:
"JC, can you provide assurance that Google staff involved in pushing any
code upstream will also be available to help perform that maintenance
effort, in this case presumably as members of the serviceability team?"

(Pre-amble: I believe it is a valid question for any relatively new-comer
to be asked this question and hopefully my answer does help have confidence
in my actual resolve)

I actually had a longer answer but thought I was diluting the actual answer
that you are looking/hoping for:
   - I believe that this is true for any JEP being submitted: ie if you
push code upstream, you should be available for maintaining that code
onwards
   - I sincerely only can talk about myself and I think we are in
agreement, any work coming in the mainline is additional burden and that is
what the JEP process is about: defining the work, the testing, the risks
and the maintenance; I believe that as we move forward, we would only
shepherd JEPs to the mainline in which we are confident that they will be
maintained right and not be added burden to the rest of the team
   - Finally, I think that this is the biggest reason we need to have the
right people as reviewers, experts, and committed people to ensure this
very fact; we do not want contribute-and-run scenarios; this not only hurts
the code-base, it hurts the community at large.

So I don't answer your question directly because I want this not to be a
Google-specific project, nor do I want it to be Google is pushing code via
this project. You have my assurance that any JEP that I contribute to and
push forward has my commitment to be helping the maintenance of AND that
I'll make very clear to whoever is trying to do work in Atlantis and trying
to then go to the mainline via a JEP that the JEP must have a engineer
maintenance assurance that we, the community, feel comfortable with. So I
believe I do however answer that I think your sentence is the right one
across the board and we should all abide by:
"There is a requirement that anything new of a significant size or
complexity that gets developed under a JEP and then integrated into the
mainline repos will have someone available to maintain it."; I don't think
this needs to be stated in the project description but I can definitely add
it as I believe in this probably more strongly than most ;-)

To finish this first long answer, and as someone recommended to me as well,
I am hoping to get serviceability core members and members from the various
monitoring tools (JVMTI, JMX, JMC and JFR) to be interested and be able to
follow up with opinions about key items:
    - "I think this could be done via JVMTI, JMX, JMC and JFR; we should
either go do it via their projects/branches"
    - I have no problem a monitoring/perf. analysis team saying let's
prototype/experiment this idea here in Atlantis; then we can see if will
merge it with the mainline
    - This participation can be ad-hoc or via being an initial
reviewer/committer

And that should solve the questions that Mario had about that side.

Hopefully my answer helps understand that this project is an attempt to
have a venue to create conversations about current internal systems or
non-existant ones and that we can see what "sticks" and what doesn't; where
it would "stick" potentially, what it would like and then how could be best
push it forward with the support of the whole community.

I'll finish the way I started, I don't believe in fragmentation, I don't
believe in trying to build "yet-another-X" when the existing systems
suffice,
Jc

On Tue, Nov 20, 2018 at 10:41 AM <jesper.wilhelmsson at oracle.com> wrote:

> On 20 Nov 2018, at 17:21, Mario Torre <neugens at redhat.com> wrote:
>
> On Tue, 2018-11-20 at 15:38 +0000, Andrew Dinn wrote:
>
> On 20/11/2018 15:14, Mario Torre wrote:
>
> On Tue, 2018-11-20 at 15:38 +0100, Thomas St?fe wrote:
>
> I'm worried about fragmentation though. Once we have this
> project,
> what is then the role of serviceability-dev? Just smallish
> everyday
> fixes? Also, the fact that VM developer feedback is hard to come
> by
> is
> mostly due to a lack of time on their part, and a new mailing
> list
> will not fix that.
>
>
> Generally projects have their own mailing lists for development
> until
> they are "finished", I would expect that for this project it would
> work
> the same, once it's integrated into mainline it becomes part of the
> serviceability code and the project mailing list is folded into
> serviceability-dev, until then it's somewhat an independent unit.
>
>
> Well, there is a requirement that anything new of a significant size
> orc
> complexity that gets developed under a JEP and then integrated into
> the
> mainline repos will have someone available to maintain it.
>
>
> Yes, indeed!
>
> JC, can you provide assurance that Google staff involved in pushing
> any
> code upstream will also be available to help perform that maintenance
> effort, in this case presumably as members of the serviceability
> team?
>
> n.b. Modulo the above detail I'm all for this proposal. I'd be very
> pleased if the project brings opportunities for Google either to
> upstream their existing work to everyone's benefit or, where for
> whatever reason that is not feasible, to help tailor existing
> upstream
> capabilities (such as JFR events) so they might encompass Google's
> downstream needs.
>
> It's very welcome to see Google so willing to contribute to OpenJDK.
>
>
> I agree!
>
> I actually wanted to ask that one written requirement in the JEP would
> be to extend Flight Recorder whenever it makes sense, even if this
> means going an extra step (in other words, the default option), so that
> we avoid fragmentation and the proliferation of a multitude of tools.
>
>
> +1
>
> Given that it is an explicit goal to extend and utilize existing
> frameworks rather than adding another one, I welcome this project.
> /Jesper
>
>
> I understand the project is meant to be some sort of umbrealla for
> experiementation? This will probably already cure most of the risk of
> such fragmentation, however, if it's a written statement in the JEP
> goals it makes more explicit that we are interested in enhancing the
> current frameworks rather than creating alternative versions (unless it
> *really* makes sense to do so).
>
> Cheers,
> Mario
> --
> Mario Torre
> Associate Manager, Software Engineering
> Red Hat GmbH <https://www.redhat.com>
> 9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898
>
>
>

-- 

Thanks,
Jc

From chuansheng.lcs at alibaba-inc.com  Wed Nov 21 03:46:36 2018
From: chuansheng.lcs at alibaba-inc.com (Jonathan Lu)
Date: Wed, 21 Nov 2018 11:46:36 +0800
Subject: =?UTF-8?B?UmU6IENhbGwgRm9yIERpc2N1c3Npb246IE5ldyBQcm9qZWN0OiBBdGxhbnRpcw==?=
In-Reply-To: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
Message-ID: <f92eb2fc-27fa-49d8-a111-8798f65029be.chuansheng.lcs@alibaba-inc.com>

Hi JC,

Personally, I like this idea.

And I'm especially interested in the heap dumping part.
It costs a lot to create, transfer, and analyze huge heap dump files for my applications.
Maybe some fundamental changes can be explored in this project, I guess,
 such as making an enhanced format of heap dump file :)

Thank you
Jonathan


------------------------------------------------------------------
From:JC Beyler <jcbeyler at google.com>
Send Time:2018?11?20?(???) 02:48
To:discuss <discuss at openjdk.java.net>
Cc:serguei.spitsyn <serguei.spitsyn at oracle.com>
Subject:Call For Discussion: New Project: Atlantis

Hi all,

I would like to start a discussion about the possible creation of the
Atlantis Project with myself, Jean Christophe Beyler, as the Lead and the
Serviceability group as the sponsoring group.

In accordance with the OpenJDK guidelines [1], this project would provide a
venue to explore and incubate monitoring and performance analysis features
that could be integrated into the Hotspot JVM and the JVM Tool Interface.
This includes working, evaluating, and incubating a Thread-Sanitizer
implementation [2] for Java.

The Google platform team has historically worked on providing new features,
including monitoring hooks due to certain monitoring features not being
available in a low-overhead, turned on-by-default manner. Therefore, for
Google Java users, the platform team augmented or added mechanisms to
assess various metrics such as but not limited to:

- Thread Sanitizer support for Java (see JDK-8208520 [2]) (though this one
specifically is not a low-overhead one, sorry ;-))
- Lock contention profiles to better understand where threads are spending
time in locks
- Heap dumping mechanisms to improve heap dump times for large heaps
- Efficient thread tagging to help filter profiling
- etc.

All of the above have added changes to the JDK in various forms and the
Atlantis project proposes to investigate, in the open, how these changes
could be made general enough to be pushed into the mainline, or should be
dropped due to being too specific or could be done using other existing
mechanisms. The project would not be a Google-only project and will have a
mailing list to discuss and potentially accept any contribution trying to
extend and/or enhance current monitoring systems.

The evaluation criteria to consider a feature/improvement includes but
would not be limited to:

     * Performance: metrics and new hooks should not be hurting performance
when disabled
     * General usefulness for the community at large
     * No working alternatives: sometimes it may be that there are new
technologies available (ie JFR events for example) that might be leveraged
to deprecate existing solutions

If one or more prototypes indicate a new metric, a new monitoring system,
or a feature offers substantial improvements, the Atlantis project would
shepherd a JEP to integrate the change into the mainline of OpenJDK.

The current suggested initial Reviewers and Committers would be but is most
likely not going to be limited to:
* Jean Christophe Beyler
* Serguei Spitsyn
* Chuck Rasbold
* Liam Cushon

What do you all think? :-)

Thanks,
Jc

[1] http://openjdk.java.net/projects/#new-project
[2] https://bugs.openjdk.java.net/browse/JDK-8208520


From shiva20grk at gmail.com  Sat Nov 24 08:01:22 2018
From: shiva20grk at gmail.com (Shiva Kumar K R)
Date: Sat, 24 Nov 2018 13:31:22 +0530
Subject: How can I download specific version of openjdk binaries
Message-ID: <CA+n21sQC_hzC=Vwp3sE4=XuEJJgAVvCv6rCp=2W-2GAe=NTr+Q@mail.gmail.com>

Hi All,
I need to download older version of openjdk binaries for Linux without
using apt.

Thanks,
Shiva

From aph at redhat.com  Sat Nov 24 10:38:42 2018
From: aph at redhat.com (Andrew Haley)
Date: Sat, 24 Nov 2018 10:38:42 +0000
Subject: How can I download specific version of openjdk binaries
In-Reply-To: <CA+n21sQC_hzC=Vwp3sE4=XuEJJgAVvCv6rCp=2W-2GAe=NTr+Q@mail.gmail.com>
References: <CA+n21sQC_hzC=Vwp3sE4=XuEJJgAVvCv6rCp=2W-2GAe=NTr+Q@mail.gmail.com>
Message-ID: <f25e227b-aa51-38b6-95f7-9b9d8878139f@redhat.com>

On 11/24/18 8:01 AM, Shiva Kumar K R wrote:
> I need to download older version of openjdk binaries for Linux without
> using apt.

Please be more specific. We need lots of detail: exactly which OS, which
OpenJDK release.

-- 
Andrew Haley
Java Platform Lead Engineer
Red Hat UK Ltd. <https://www.redhat.com>
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

From peterhull90 at gmail.com  Sat Nov 24 14:50:01 2018
From: peterhull90 at gmail.com (Peter Hull)
Date: Sat, 24 Nov 2018 14:50:01 +0000
Subject: JNI - question about jmethodid values.
Message-ID: <CAK9Gx1ePUfKSNe6g7C7bfWZTD9R58v9Pm+OM4Ehs17JqR6-R3w@mail.gmail.com>

I have recently been looking at some native code that's part of the
NetBeans profiler. This code was crashing for me because it was based
on incorrect assumptions about the sizes of various primitive types.
There is some code which is intended to map jmethodid values to jint
values (see below). This is trivial on 32 bit platforms because the
are the same size. However on 64-bit platforms this is not the case
because jmethodid is (according to the docs) to be treated as a
pointer to an opaque structure and is 64-bit.
So, the code splits the 64-bit jmethodid into a 30-bit 'offset' part
and a 34-bit 'base' part. It then stores the high part in one of four
slots and returns a 32 bit int which is 2 bits for the slot id and 30
bits from the offset part. For the reverse process, it looks up the
slot id, gets the base address and ORs it with the offset. This seems
to work in practice but it is based on the assumption that all
jmethodids will have one of only four base parts - any more than four
can't be specified by a 2 bit id.
So my question is: is this a sound assumption? The answer needs some
detailed knowledge of the internals of the JRE I suppose, so if this
mailing list is not the right place to ask, what is the right place?
I want to fix my crash in the netbeans profiler but I want to do the
right thing rather than just patching it up (which I have done by
doing casts more carefully)
Thanks for your help.
Peter

For reference, the code is on github:
https://github.com/apache/incubator-netbeans/blob/5405b8c93eeafdd213cffcc59df002464af8d838/profiler/lib.profiler/native/src-jdk15/Stacks.c#L66-L92
and: https://github.com/apache/incubator-netbeans/blob/5405b8c93eeafdd213cffcc59df002464af8d838/profiler/lib.profiler/native/src-jdk15/Stacks.c#L66-L92

From thomas.stuefe at gmail.com  Sat Nov 24 15:41:18 2018
From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=)
Date: Sat, 24 Nov 2018 16:41:18 +0100
Subject: JNI - question about jmethodid values.
In-Reply-To: <CAK9Gx1ePUfKSNe6g7C7bfWZTD9R58v9Pm+OM4Ehs17JqR6-R3w@mail.gmail.com>
References: <CAK9Gx1ePUfKSNe6g7C7bfWZTD9R58v9Pm+OM4Ehs17JqR6-R3w@mail.gmail.com>
Message-ID: <CAA-vtUztzLqMLssNWXmf78v3TGQmqMa8vGn0exEeV2f689=Mbw@mail.gmail.com>

Hi Peter,

(adding serviceability)

A jmethodid is a pointer to malloc'ed memory.

In more detail, jmethodid points to a table slot containing pointers
to Method* objects, and that table itself lives in C heap. See class
JNIMethodBlock, JNIMethodBlockNode:
http://hg.openjdk.java.net/jdk/jdk/file/30a02b4e6c06/src/hotspot/share/oops/method.cpp#l1928

If I understand the NetBeans profiler coding right, they assume the
jmethodIDs to cluster in up to four clusters of relative vicinity,
with each cluster spanning 30 bits? That is quite an assumption to
make.

This may work more or less accidentally, since malloc'ed pointers are
also clustered naturally. A Clib implementation may allocate only in
the traditional data segment (below sbrk) - I believe AIX and Solaris
do this - which means all malloced pointers are as close to each other
as the data segment is large. But C-Libraries often allocate C-Heap
memory in separate mmape'd blocks (e.g. glibc arenas), and there is no
guarantee where those blocks are. You may still often get lucky - if
all jmethod id block allocations happen in a short timeframe, or if
you just have not many users of C-Heap in the process.

But it is not guaranteed to work. I would probably rather use a
hashmap or similar.

Side note: I believe (and someone please correct me if I am wrong)
that jmethodid used to live in the PermGen, in the Java Heap, and that
only since the PermGen removal in JDK8 they live in C-Heap. If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.

Best Regards, Thomas


On Sat, Nov 24, 2018 at 3:51 PM Peter Hull <peterhull90 at gmail.com> wrote:
>
> I have recently been looking at some native code that's part of the
> NetBeans profiler. This code was crashing for me because it was based
> on incorrect assumptions about the sizes of various primitive types.
> There is some code which is intended to map jmethodid values to jint
> values (see below). This is trivial on 32 bit platforms because the
> are the same size. However on 64-bit platforms this is not the case
> because jmethodid is (according to the docs) to be treated as a
> pointer to an opaque structure and is 64-bit.
> So, the code splits the 64-bit jmethodid into a 30-bit 'offset' part
> and a 34-bit 'base' part. It then stores the high part in one of four
> slots and returns a 32 bit int which is 2 bits for the slot id and 30
> bits from the offset part. For the reverse process, it looks up the
> slot id, gets the base address and ORs it with the offset. This seems
> to work in practice but it is based on the assumption that all
> jmethodids will have one of only four base parts - any more than four
> can't be specified by a 2 bit id.
> So my question is: is this a sound assumption? The answer needs some
> detailed knowledge of the internals of the JRE I suppose, so if this
> mailing list is not the right place to ask, what is the right place?
> I want to fix my crash in the netbeans profiler but I want to do the
> right thing rather than just patching it up (which I have done by
> doing casts more carefully)
> Thanks for your help.
> Peter
>
> For reference, the code is on github:
> https://github.com/apache/incubator-netbeans/blob/5405b8c93eeafdd213cffcc59df002464af8d838/profiler/lib.profiler/native/src-jdk15/Stacks.c#L66-L92
> and: https://github.com/apache/incubator-netbeans/blob/5405b8c93eeafdd213cffcc59df002464af8d838/profiler/lib.profiler/native/src-jdk15/Stacks.c#L66-L92

From shiva20grk at gmail.com  Mon Nov 26 02:40:30 2018
From: shiva20grk at gmail.com (Shiva Kumar K R)
Date: Mon, 26 Nov 2018 08:10:30 +0530
Subject: How can I download specific version of openjdk binaries
In-Reply-To: <f25e227b-aa51-38b6-95f7-9b9d8878139f@redhat.com>
References: <CA+n21sQC_hzC=Vwp3sE4=XuEJJgAVvCv6rCp=2W-2GAe=NTr+Q@mail.gmail.com>
 <f25e227b-aa51-38b6-95f7-9b9d8878139f@redhat.com>
Message-ID: <CA+n21sTfhnkXsQh3AEgu=BBXqRBFGGLiYUiLjTPKHfiaoJt7qw@mail.gmail.com>

My requirement is for Ubuntu 16.04, openjdk 8u 191.

On Sat 24 Nov, 2018, 4:08 PM Andrew Haley <aph at redhat.com wrote:

> On 11/24/18 8:01 AM, Shiva Kumar K R wrote:
> > I need to download older version of openjdk binaries for Linux without
> > using apt.
>
> Please be more specific. We need lots of detail: exactly which OS, which
> OpenJDK release.
>
> --
> Andrew Haley
> Java Platform Lead Engineer
> Red Hat UK Ltd. <https://www.redhat.com>
> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671
>

From peterhull90 at gmail.com  Mon Nov 26 12:01:44 2018
From: peterhull90 at gmail.com (Peter Hull)
Date: Mon, 26 Nov 2018 12:01:44 +0000
Subject: JNI - question about jmethodid values.
In-Reply-To: <CAA-vtUztzLqMLssNWXmf78v3TGQmqMa8vGn0exEeV2f689=Mbw@mail.gmail.com>
References: <CAK9Gx1ePUfKSNe6g7C7bfWZTD9R58v9Pm+OM4Ehs17JqR6-R3w@mail.gmail.com>
 <CAA-vtUztzLqMLssNWXmf78v3TGQmqMa8vGn0exEeV2f689=Mbw@mail.gmail.com>
Message-ID: <CAK9Gx1ev6ov7iWt84s8whrAqP6Enb8kbY83Evz15d_KHmAh6nw@mail.gmail.com>

Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428

On Sat, Nov 24, 2018 at 3:41 PM Thomas St?fe <thomas.stuefe at gmail.com> wrote:
> A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
>
> But it is not guaranteed to work. I would probably rather use a
> hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
>
> If this is
> true, this 4x30bit assumption may actually have worked before jdk8,
> since the java heap is allocated as one continuous space, with the
> PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)

Yours sincerely,
Peter

From thomas.stuefe at gmail.com  Mon Nov 26 13:00:13 2018
From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=)
Date: Mon, 26 Nov 2018 14:00:13 +0100
Subject: JNI - question about jmethodid values.
In-Reply-To: <CAK9Gx1ev6ov7iWt84s8whrAqP6Enb8kbY83Evz15d_KHmAh6nw@mail.gmail.com>
References: <CAK9Gx1ePUfKSNe6g7C7bfWZTD9R58v9Pm+OM4Ehs17JqR6-R3w@mail.gmail.com>
 <CAA-vtUztzLqMLssNWXmf78v3TGQmqMa8vGn0exEeV2f689=Mbw@mail.gmail.com>
 <CAK9Gx1ev6ov7iWt84s8whrAqP6Enb8kbY83Evz15d_KHmAh6nw@mail.gmail.com>
Message-ID: <CAA-vtUyc-36Hzf0-KVAY_tN342zaqSnwkfhEvL517zsehjeuwQ@mail.gmail.com>

Hi Peter,

On Mon, Nov 26, 2018 at 1:02 PM Peter Hull <peterhull90 at gmail.com> wrote:
>
> Hi Thomas,
> Thank you very much for the detailed explanation. For your
> information, the relevant NetBeans bug is
> https://issues.apache.org/jira/browse/NETBEANS-1428
>
> On Sat, Nov 24, 2018 at 3:41 PM Thomas St?fe <thomas.stuefe at gmail.com> wrote:
> > A jmethodid is a pointer to malloc'ed memory.
> OK. Just in case I haven't understood it - does this mean a jmethodid,
> once 'created', won't change (after garbage collection or whatever)?

yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.

> >
> > But it is not guaranteed to work. I would probably rather use a
> > hashmap or similar.
> I need to look at the implications on more detail but think it would
> make sense to use long/jlong instead of int/jint on all platforms; the
> extra memory use shouldn't be a problem. I think the IDs are just
> stored on the Java side and used to get the method name and signature
> later. That should be a loss-free cast, shouldn't it?

Sure.

> >
> > If this is
> > true, this 4x30bit assumption may actually have worked before jdk8,
> > since the java heap is allocated as one continuous space, with the
> > PermGen clustered in one part of it.
> Indeed we did only start to get crashes on JDK9 and later (only
> observed on Windows, macOS seems OK and other platforms have not been
> tested)
>
> Yours sincerely,
> Peter

Cheers, Thomas

From thomas.stuefe at gmail.com  Mon Nov 26 15:42:47 2018
From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=)
Date: Mon, 26 Nov 2018 16:42:47 +0100
Subject: How can I download specific version of openjdk binaries
In-Reply-To: <CA+n21sTfhnkXsQh3AEgu=BBXqRBFGGLiYUiLjTPKHfiaoJt7qw@mail.gmail.com>
References: <CA+n21sQC_hzC=Vwp3sE4=XuEJJgAVvCv6rCp=2W-2GAe=NTr+Q@mail.gmail.com>
 <f25e227b-aa51-38b6-95f7-9b9d8878139f@redhat.com>
 <CA+n21sTfhnkXsQh3AEgu=BBXqRBFGGLiYUiLjTPKHfiaoJt7qw@mail.gmail.com>
Message-ID: <CAA-vtUzhKNWvFGK1U6V-Lkq94YrdQe0jx832ZQcQrRmRUH25Ug@mail.gmail.com>

You could have to ask Canonical for it. But from what I can see they
are currently at 8u181:
https://packages.ubuntu.com/hu/xenial/java/openjdk-8-jdk .

Alternatively, adoptopenjdk currently build 8u192:
https://adoptopenjdk.net/installation.html#x64_linux-jdk

Or, you could build it yourself.

Thomas


On Mon, Nov 26, 2018 at 3:41 AM Shiva Kumar K R <shiva20grk at gmail.com> wrote:
>
> My requirement is for Ubuntu 16.04, openjdk 8u 191.
>
> On Sat 24 Nov, 2018, 4:08 PM Andrew Haley <aph at redhat.com wrote:
>
> > On 11/24/18 8:01 AM, Shiva Kumar K R wrote:
> > > I need to download older version of openjdk binaries for Linux without
> > > using apt.
> >
> > Please be more specific. We need lots of detail: exactly which OS, which
> > OpenJDK release.
> >
> > --
> > Andrew Haley
> > Java Platform Lead Engineer
> > Red Hat UK Ltd. <https://www.redhat.com>
> > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671
> >

From jcbeyler at google.com  Mon Nov 26 19:15:03 2018
From: jcbeyler at google.com (JC Beyler)
Date: Mon, 26 Nov 2018 11:15:03 -0800
Subject: JNI - question about jmethodid values.
In-Reply-To: <CAA-vtUyc-36Hzf0-KVAY_tN342zaqSnwkfhEvL517zsehjeuwQ@mail.gmail.com>
References: <CAK9Gx1ePUfKSNe6g7C7bfWZTD9R58v9Pm+OM4Ehs17JqR6-R3w@mail.gmail.com>
 <CAA-vtUztzLqMLssNWXmf78v3TGQmqMa8vGn0exEeV2f689=Mbw@mail.gmail.com>
 <CAK9Gx1ev6ov7iWt84s8whrAqP6Enb8kbY83Evz15d_KHmAh6nw@mail.gmail.com>
 <CAA-vtUyc-36Hzf0-KVAY_tN342zaqSnwkfhEvL517zsehjeuwQ@mail.gmail.com>
Message-ID: <CAF9BGBxmN2TQuYvG=CVqXFSoDNCYYMe0TB9P7EBjASUfaw8PDg@mail.gmail.com>

Hi all,

Just added my two cents on one comment:

On Mon, Nov 26, 2018 at 5:00 AM Thomas St?fe <thomas.stuefe at gmail.com>
wrote:

> Hi Peter,
>
> On Mon, Nov 26, 2018 at 1:02 PM Peter Hull <peterhull90 at gmail.com> wrote:
> >
> > Hi Thomas,
> > Thank you very much for the detailed explanation. For your
> > information, the relevant NetBeans bug is
> > https://issues.apache.org/jira/browse/NETBEANS-1428
> >
> > On Sat, Nov 24, 2018 at 3:41 PM Thomas St?fe <thomas.stuefe at gmail.com>
> wrote:
> > > A jmethodid is a pointer to malloc'ed memory.
> > OK. Just in case I haven't understood it - does this mean a jmethodid,
> > once 'created', won't change (after garbage collection or whatever)?
>
> yes. It lives on, unchanged, forever. Even if the associated class is
> unloaded. That must be since outside JNI/JVMTI code may call in with
> outdated jmethodids which we must be able to handle gracefully.
>
>
This is the current design and most likely will remain for quite a bit but
it is not defined by the SPEC to be like this so, in essence, it is an
implementation detail that could be not true in future releases ;-)

Thanks,
Jc


> > >
> > > But it is not guaranteed to work. I would probably rather use a
> > > hashmap or similar.
> > I need to look at the implications on more detail but think it would
> > make sense to use long/jlong instead of int/jint on all platforms; the
> > extra memory use shouldn't be a problem. I think the IDs are just
> > stored on the Java side and used to get the method name and signature
> > later. That should be a loss-free cast, shouldn't it?
>
> Sure.
>
> > >
> > > If this is
> > > true, this 4x30bit assumption may actually have worked before jdk8,
> > > since the java heap is allocated as one continuous space, with the
> > > PermGen clustered in one part of it.
> > Indeed we did only start to get crashes on JDK9 and later (only
> > observed on Windows, macOS seems OK and other platforms have not been
> > tested)
> >
> > Yours sincerely,
> > Peter
>
> Cheers, Thomas
>


-- 

Thanks,
Jc

From jcbeyler at google.com  Mon Nov 26 19:22:30 2018
From: jcbeyler at google.com (JC Beyler)
Date: Mon, 26 Nov 2018 11:22:30 -0800
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <f92eb2fc-27fa-49d8-a111-8798f65029be.chuansheng.lcs@alibaba-inc.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <f92eb2fc-27fa-49d8-a111-8798f65029be.chuansheng.lcs@alibaba-inc.com>
Message-ID: <CAF9BGBxLSeB4aVT5a8RuRmRtp47wwt_-OZ0pGEMXCFeOCF9a0w@mail.gmail.com>

Hi Jonathan,

Thanks for the interest and sounds good to me! We could start conversations
on this topic once the project opens and see what we could do in that
problem-space :)
Jc

On Tue, Nov 20, 2018 at 7:46 PM Jonathan Lu <chuansheng.lcs at alibaba-inc.com>
wrote:

> Hi JC,
>
> Personally, I like this idea.
>
> And I'm especially interested in the heap dumping part.
> It costs a lot to create, transfer, and analyze huge heap dump files for
> my applications.
> Maybe some fundamental changes can be explored in this project, I guess,
> such as making an enhanced format of heap dump file :)
>
> Thank you
> Jonathan
>
> ------------------------------------------------------------------
> From:JC Beyler <jcbeyler at google.com>
> Send Time:2018?11?20?(???) 02:48
> To:discuss <discuss at openjdk.java.net>
> Cc:serguei.spitsyn <serguei.spitsyn at oracle.com>
> Subject:Call For Discussion: New Project: Atlantis
>
> Hi all,
>
> I would like to start a discussion about the possible creation of the
> Atlantis Project with myself, Jean Christophe Beyler, as the Lead and the
> Serviceability group as the sponsoring group.
>
> In accordance with the OpenJDK guidelines [1], this project would provide a
> venue to explore and incubate monitoring and performance analysis features
> that could be integrated into the Hotspot JVM and the JVM Tool Interface.
> This includes working, evaluating, and incubating a Thread-Sanitizer
> implementation [2] for Java.
>
> The Google platform team has historically worked on providing new features,
> including monitoring hooks due to certain monitoring features not being
> available in a low-overhead, turned on-by-default manner. Therefore, for
> Google Java users, the platform team augmented or added mechanisms to
> assess various metrics such as but not limited to:
>
> - Thread Sanitizer support for Java (see JDK-8208520 [2]) (though this one
> specifically is not a low-overhead one, sorry ;-))
> - Lock contention profiles to better understand where threads are spending
> time in locks
> - Heap dumping mechanisms to improve heap dump times for large heaps
> - Efficient thread tagging to help filter profiling
> - etc.
>
> All of the above have added changes to the JDK in various forms and the
> Atlantis project proposes to investigate, in the open, how these changes
> could be made general enough to be pushed into the mainline, or should be
> dropped due to being too specific or could be done using other existing
> mechanisms. The project would not be a Google-only project and will have a
> mailing list to discuss and potentially accept any contribution trying to
> extend and/or enhance current monitoring systems.
>
> The evaluation criteria to consider a feature/improvement includes but
> would not be limited to:
>
>      * Performance: metrics and new hooks should not be hurting performance
> when disabled
>      * General usefulness for the community at large
>      * No working alternatives: sometimes it may be that there are new
> technologies available (ie JFR events for example) that might be leveraged
> to deprecate existing solutions
>
> If one or more prototypes indicate a new metric, a new monitoring system,
> or a feature offers substantial improvements, the Atlantis project would
> shepherd a JEP to integrate the change into the mainline of OpenJDK.
>
> The current suggested initial Reviewers and Committers would be but is most
> likely not going to be limited to:
> * Jean Christophe Beyler
> * Serguei Spitsyn
> * Chuck Rasbold
> * Liam Cushon
>
> What do you all think? :-)
>
> Thanks,
> Jc
>
> [1] http://openjdk.java.net/projects/#new-project
> [2] https://bugs.openjdk.java.net/browse/JDK-8208520
>
>
>

-- 

Thanks,
Jc

From thomas.stuefe at gmail.com  Mon Nov 26 19:33:03 2018
From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=)
Date: Mon, 26 Nov 2018 20:33:03 +0100
Subject: JNI - question about jmethodid values.
In-Reply-To: <CAF9BGBxmN2TQuYvG=CVqXFSoDNCYYMe0TB9P7EBjASUfaw8PDg@mail.gmail.com>
References: <CAK9Gx1ePUfKSNe6g7C7bfWZTD9R58v9Pm+OM4Ehs17JqR6-R3w@mail.gmail.com>
 <CAA-vtUztzLqMLssNWXmf78v3TGQmqMa8vGn0exEeV2f689=Mbw@mail.gmail.com>
 <CAK9Gx1ev6ov7iWt84s8whrAqP6Enb8kbY83Evz15d_KHmAh6nw@mail.gmail.com>
 <CAA-vtUyc-36Hzf0-KVAY_tN342zaqSnwkfhEvL517zsehjeuwQ@mail.gmail.com>
 <CAF9BGBxmN2TQuYvG=CVqXFSoDNCYYMe0TB9P7EBjASUfaw8PDg@mail.gmail.com>
Message-ID: <CAA-vtUwH_H1HcwHmEL8XDs3tQ0gJWgu84O=CqZbdXG5Jsb-3PQ@mail.gmail.com>

Hey JC,

On Mon, Nov 26, 2018 at 8:15 PM JC Beyler <jcbeyler at google.com> wrote:
>
> Hi all,
>
> Just added my two cents on one comment:
>
> On Mon, Nov 26, 2018 at 5:00 AM Thomas St?fe <thomas.stuefe at gmail.com> wrote:
>>
>> Hi Peter,
>>
>> On Mon, Nov 26, 2018 at 1:02 PM Peter Hull <peterhull90 at gmail.com> wrote:
>> >
>> > Hi Thomas,
>> > Thank you very much for the detailed explanation. For your
>> > information, the relevant NetBeans bug is
>> > https://issues.apache.org/jira/browse/NETBEANS-1428
>> >
>> > On Sat, Nov 24, 2018 at 3:41 PM Thomas St?fe <thomas.stuefe at gmail.com> wrote:
>> > > A jmethodid is a pointer to malloc'ed memory.
>> > OK. Just in case I haven't understood it - does this mean a jmethodid,
>> > once 'created', won't change (after garbage collection or whatever)?
>>
>> yes. It lives on, unchanged, forever. Even if the associated class is
>> unloaded. That must be since outside JNI/JVMTI code may call in with
>> outdated jmethodids which we must be able to handle gracefully.
>>
>
> This is the current design and most likely will remain for quite a bit but it is not defined by the SPEC to be like this so, in essence, it is an implementation detail that could be not true in future releases ;-)

I politely disagree :)

JNI spec says that you have a function like this:

jmethodID GetMethodID(..)

returning a jmethodid which you can then use in subsequent JNI
functions. These functions are required to behave in a predictable way
if the jmethodid has gotten invalid. The spec has no way to inform the
caller if a jmethodid has gotten invalid by class unloading. Since the
jmethodid is handed up to the caller and stored by him, there is no
way to change its value on the fly either.

So even though the spec does not specifically say that jmethodid lives
forever, it is laid out in a way that prevents jmethodids from ever
becoming invalid, and from changing their numerical value. This can of
course change, but not without changing the JNI spec.

I think my point is that Peter can rely on this behavior without fear
that it will change in the future without him noticing. Before this
behavior changes, the JNI spec would have to change.

Cheers, Thomas







>
> Thanks,
> Jc
>
>>
>> > >
>> > > But it is not guaranteed to work. I would probably rather use a
>> > > hashmap or similar.
>> > I need to look at the implications on more detail but think it would
>> > make sense to use long/jlong instead of int/jint on all platforms; the
>> > extra memory use shouldn't be a problem. I think the IDs are just
>> > stored on the Java side and used to get the method name and signature
>> > later. That should be a loss-free cast, shouldn't it?
>>
>> Sure.
>>
>> > >
>> > > If this is
>> > > true, this 4x30bit assumption may actually have worked before jdk8,
>> > > since the java heap is allocated as one continuous space, with the
>> > > PermGen clustered in one part of it.
>> > Indeed we did only start to get crashes on JDK9 and later (only
>> > observed on Windows, macOS seems OK and other platforms have not been
>> > tested)
>> >
>> > Yours sincerely,
>> > Peter
>>
>> Cheers, Thomas
>
>
>
> --
>
> Thanks,
> Jc

From thomas.stuefe at gmail.com  Mon Nov 26 21:51:21 2018
From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=)
Date: Mon, 26 Nov 2018 22:51:21 +0100
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <CAF9BGByAv=m3L-LKNjU0whMeL58JL8naW2Y0++OkSU6CArTTtQ@mail.gmail.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>
 <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>
 <91e0480a-a593-13d4-5b0c-b45bc7d4563c@redhat.com>
 <56eaa0de0d7d095413be0c1a89c0efe071669620.camel@redhat.com>
 <2806730D-AEBF-4491-8442-F81AA2C1966D@oracle.com>
 <CAF9BGByAv=m3L-LKNjU0whMeL58JL8naW2Y0++OkSU6CArTTtQ@mail.gmail.com>
Message-ID: <CAA-vtUw5uvHKB+An6FiFjix_SwTb17P7a5Gwcwx=eeK7cpQiwA@mail.gmail.com>

Hi JC, guys,

(JC: I know we had a similar discussion offlist some days ago, sorry
for raising the same points, still being unsure here).

Here is what I think I understand:

We want a "council" of sorts of experts in the general
monitoring/serviceability area. A one-stop point to discuss larger
planned contributions. They should advice on new developments,
redirect them, shepherd them, curtail them when necessary. Impose a
common strategy we all consent to (e.g. "JFR first").

No one should be able to add unnecessary features or reinvent the
wheel. We want to avoid fragmentation and keep the codebase simple and
consistent.

But also, no valuable contributions should find themselves barred. If
anything, we want to make the process of contributing smoother, and
that also includes the process of rejecting proposals.

And if a contribution is considered worthwhile, it would be incubated
separately from mainline, until it is ready to be upmerged.

..

Here is where I start getting confused: are we not doing most of this
already? I would have thought most of this is common sense and at
least in theory no-one would argue these points? (e.g. noone would
argue that reinventing the wheel is good :)

Currently, we have the JEP process, we have area-specific mailing
lists (serviceability-dev, jfr-dev etc). We have the sandbox repo for
test branches. And larger contributions are already spun off own into
own feature-specific projects with own mailing lists and repos.

Based on our earlier discussions, I thought the main thing missing was
a way to reach the experts via a cross-area channel. Since posting
contribution proposals to serviceability-dev has been considered
"spamming" (which surprised me btw), svc-dev is out of the question.
But would not just a new mailing list suffice for this? Why a whole
new umbrella project, new reviewer roles etc?

Especially since this umbrella project now conflicts with the idea of
feature-specific projects. Now we have three options for a larger
contribution: We can either discuss it on the area-specific mailing
lists and add it directly to mainline, optionally with a JEP; we can
also create an own feature-specific project, incubate it an own
repository with an own mailing list, then add it to mainline. Now,
with Atlantis, we have the third option of adding it to the
Atlantis-specific repository (?), as a new branch (?) and discuss it
on the Atlantis-specific mailing list.

Sorry,  I really just try to understand. I think you have valid pain
points you try to solve, I am just not sure if a whole new umbrella
project is the best solution. Or, I could just not understanding the
proposal.

Thanks & Kind Regards, Thomas

p.s. if I as an author go for the third option, via atlantis-dev: I
would like to be reasonably sure that once my contribution meets
approval and gets incubated, chances are reasonably high that it will
be merged into mainline later. Because, if atlantis-dev greenlights a
feature and helps incubating it but later it gets barred from
mainline, that would make Atlantis moot. This means that atlantis-dev
must be monitored by the resident area experts.


On Tue, Nov 20, 2018 at 9:36 PM JC Beyler <jcbeyler at google.com> wrote:
>
> Thanks all for the questions and feedback. I've tried to summarize the questions/concerns here:
>
> - I want to be really clear that I don't have all the answers, I believe very strongly in trial and error
>     - Part of this project's motivation is to have these conversations :)
>     - Another part is that I think there is something to be gained by this project and it is exactly what Mario has said above: that there be an "umbrella for experimentation; This will probably already cure most of the risk of such fragmentation, however, if it's a written statement in the JEP goals it makes more explicit that we are interested in enhancing the current frameworks rather than creating alternative versions (unless it *really* makes sense to do so)."
>         -> Exactly, I'm not interested in creating the next "shiny thing" that actually does the same thing as existing items; I am interested in looking at an issue/monitoring feature; considering all existing frameworks; seeing where it would fit best; figure out if it is feasible acceptable; then move forward".
>
> - (From Thomas mainly but also Mario) Is there a risk of fragmentation of knowledge or fragmenting serviceability?
>     - I don't think so nor do I want to; I'm actually against fragmentation in almost all cases to be very honest and clear; that is why I've tried to work closely with the serviceability team and engineers to figure out what could/should be the scope of this project
>     - Let us not forget that in the end, the work done in this project still needs a JEP to get into the mainline and that is our last barrier/hurdle that ensures consistency in OpenJDK and not fragmentation
>     - To finish this subject (and re-iterate), if an item is to do "yet-another-monitoring/performance-analysis tool" that could be done by the various ones we have already available, I believe it has no place in this project; on the other hand, if it can be proved that X cannot be done at all in a product environment with current systems then we can/should start asking the questions:
>          - Is there a solution close, how could it be extended, what do the current developers of that mechanism think, should we try to prototype something to see the extent of changes, should we push it to being a JEP from this project? It is for those cases that I see a path forward via this project.
>     - I think that the balance between what could go directly to mainline or via this project has to be found, I don't have the right answer, I have the items I know I'd like to work on openly via this project and then see what comes from them; but I'm open to suggestions :-)
>        - Which means I don't have the right answer of what should go directly to serviceability-dev or here when it's a "Hey I have this" or "Hey could we that" type of question; I think we will see what happens, right now, I'm personally going to concentrate on the internal features we have at Google to start the conversation and the project work and see where that takes me/us if that helps. I'm open to any explicit guideline that is deemed necessary/warranted or we can do what I would hope is possible too: by having myself, Serguei, and core members of serviceability, we will work out the balance together as we move forward.
>
> From Andrew mainly: I'm going to quote Andrew's question/comment here:
> "JC, can you provide assurance that Google staff involved in pushing any code upstream will also be available to help perform that maintenance effort, in this case presumably as members of the serviceability team?"
>
> (Pre-amble: I believe it is a valid question for any relatively new-comer to be asked this question and hopefully my answer does help have confidence in my actual resolve)
>
> I actually had a longer answer but thought I was diluting the actual answer that you are looking/hoping for:
>    - I believe that this is true for any JEP being submitted: ie if you push code upstream, you should be available for maintaining that code onwards
>    - I sincerely only can talk about myself and I think we are in agreement, any work coming in the mainline is additional burden and that is what the JEP process is about: defining the work, the testing, the risks and the maintenance; I believe that as we move forward, we would only shepherd JEPs to the mainline in which we are confident that they will be maintained right and not be added burden to the rest of the team
>    - Finally, I think that this is the biggest reason we need to have the right people as reviewers, experts, and committed people to ensure this very fact; we do not want contribute-and-run scenarios; this not only hurts the code-base, it hurts the community at large.
>
> So I don't answer your question directly because I want this not to be a Google-specific project, nor do I want it to be Google is pushing code via this project. You have my assurance that any JEP that I contribute to and push forward has my commitment to be helping the maintenance of AND that I'll make very clear to whoever is trying to do work in Atlantis and trying to then go to the mainline via a JEP that the JEP must have a engineer maintenance assurance that we, the community, feel comfortable with. So I believe I do however answer that I think your sentence is the right one across the board and we should all abide by:
> "There is a requirement that anything new of a significant size or complexity that gets developed under a JEP and then integrated into the mainline repos will have someone available to maintain it."; I don't think this needs to be stated in the project description but I can definitely add it as I believe in this probably more strongly than most ;-)
>
> To finish this first long answer, and as someone recommended to me as well, I am hoping to get serviceability core members and members from the various monitoring tools (JVMTI, JMX, JMC and JFR) to be interested and be able to follow up with opinions about key items:
>     - "I think this could be done via JVMTI, JMX, JMC and JFR; we should either go do it via their projects/branches"
>     - I have no problem a monitoring/perf. analysis team saying let's prototype/experiment this idea here in Atlantis; then we can see if will merge it with the mainline
>     - This participation can be ad-hoc or via being an initial reviewer/committer
>
> And that should solve the questions that Mario had about that side.
>
> Hopefully my answer helps understand that this project is an attempt to have a venue to create conversations about current internal systems or non-existant ones and that we can see what "sticks" and what doesn't; where it would "stick" potentially, what it would like and then how could be best push it forward with the support of the whole community.
>
> I'll finish the way I started, I don't believe in fragmentation, I don't believe in trying to build "yet-another-X" when the existing systems suffice,
> Jc
>
> On Tue, Nov 20, 2018 at 10:41 AM <jesper.wilhelmsson at oracle.com> wrote:
>>
>> On 20 Nov 2018, at 17:21, Mario Torre <neugens at redhat.com> wrote:
>>
>> On Tue, 2018-11-20 at 15:38 +0000, Andrew Dinn wrote:
>>
>> On 20/11/2018 15:14, Mario Torre wrote:
>>
>> On Tue, 2018-11-20 at 15:38 +0100, Thomas St?fe wrote:
>>
>> I'm worried about fragmentation though. Once we have this
>> project,
>> what is then the role of serviceability-dev? Just smallish
>> everyday
>> fixes? Also, the fact that VM developer feedback is hard to come
>> by
>> is
>> mostly due to a lack of time on their part, and a new mailing
>> list
>> will not fix that.
>>
>>
>> Generally projects have their own mailing lists for development
>> until
>> they are "finished", I would expect that for this project it would
>> work
>> the same, once it's integrated into mainline it becomes part of the
>> serviceability code and the project mailing list is folded into
>> serviceability-dev, until then it's somewhat an independent unit.
>>
>>
>> Well, there is a requirement that anything new of a significant size
>> orc
>> complexity that gets developed under a JEP and then integrated into
>> the
>> mainline repos will have someone available to maintain it.
>>
>>
>> Yes, indeed!
>>
>> JC, can you provide assurance that Google staff involved in pushing
>> any
>> code upstream will also be available to help perform that maintenance
>> effort, in this case presumably as members of the serviceability
>> team?
>>
>> n.b. Modulo the above detail I'm all for this proposal. I'd be very
>> pleased if the project brings opportunities for Google either to
>> upstream their existing work to everyone's benefit or, where for
>> whatever reason that is not feasible, to help tailor existing
>> upstream
>> capabilities (such as JFR events) so they might encompass Google's
>> downstream needs.
>>
>> It's very welcome to see Google so willing to contribute to OpenJDK.
>>
>>
>> I agree!
>>
>> I actually wanted to ask that one written requirement in the JEP would
>> be to extend Flight Recorder whenever it makes sense, even if this
>> means going an extra step (in other words, the default option), so that
>> we avoid fragmentation and the proliferation of a multitude of tools.
>>
>>
>> +1
>>
>> Given that it is an explicit goal to extend and utilize existing frameworks rather than adding another one, I welcome this project.
>> /Jesper
>>
>>
>> I understand the project is meant to be some sort of umbrealla for
>> experiementation? This will probably already cure most of the risk of
>> such fragmentation, however, if it's a written statement in the JEP
>> goals it makes more explicit that we are interested in enhancing the
>> current frameworks rather than creating alternative versions (unless it
>> *really* makes sense to do so).
>>
>> Cheers,
>> Mario
>> --
>> Mario Torre
>> Associate Manager, Software Engineering
>> Red Hat GmbH <https://www.redhat.com>
>> 9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898
>>
>>
>
>
> --
>
> Thanks,
> Jc

From jcbeyler at google.com  Tue Nov 27 00:52:55 2018
From: jcbeyler at google.com (JC Beyler)
Date: Mon, 26 Nov 2018 16:52:55 -0800
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <CAA-vtUw5uvHKB+An6FiFjix_SwTb17P7a5Gwcwx=eeK7cpQiwA@mail.gmail.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <CAA-vtUwVL23ODfJXkfW+AH7ZH4nVfxZYzgmuch3N4Tkwk5uCOg@mail.gmail.com>
 <9826bcd570f56e1078908f60907ad4838394b6bb.camel@redhat.com>
 <91e0480a-a593-13d4-5b0c-b45bc7d4563c@redhat.com>
 <56eaa0de0d7d095413be0c1a89c0efe071669620.camel@redhat.com>
 <2806730D-AEBF-4491-8442-F81AA2C1966D@oracle.com>
 <CAF9BGByAv=m3L-LKNjU0whMeL58JL8naW2Y0++OkSU6CArTTtQ@mail.gmail.com>
 <CAA-vtUw5uvHKB+An6FiFjix_SwTb17P7a5Gwcwx=eeK7cpQiwA@mail.gmail.com>
Message-ID: <CAF9BGBz4uTcUt7rAyMGzpOxadHik6Euz=ukmeOAMbYrdZdj-=A@mail.gmail.com>

Hi Thomas,

Let me inline my answers and hopefully it will help understand what I'm
trying to accomplish.

On Mon, Nov 26, 2018 at 1:51 PM Thomas St?fe <thomas.stuefe at gmail.com>
wrote:

> Hi JC, guys,
>
> (JC: I know we had a similar discussion offlist some days ago, sorry
> for raising the same points, still being unsure here).
>

No worries :). Please note that your comments are well received,
appreciated and I believe that they are useful to have in order to have all
involved agreeing on key parts, feeling good about it and not worried about
anything involved with this potential project.


>
> Here is what I think I understand:
>
> We want a "council" of sorts of experts in the general
> monitoring/serviceability area. A one-stop point to discuss larger
> planned contributions. They should advice on new developments,
> redirect them, shepherd them, curtail them when necessary. Impose a
> common strategy we all consent to (e.g. "JFR first").
>

Not at all to be honest. In a nutshell, I just want a place where I can
work on a few feature implementations that are either tricky/have multiple
implementation possibilities and that will take time to get right (seems to
me to be the reason to have a project :-)). And because I'd like to get it
right as soon as possible, I'd rather have conversations about the
implementations in the open with whoever is interested and able to follow.
And because I'd like to do it with people inter-company, it seems also the
way to go. For example, having conversations about Thread Sanitizing (TSAN)
details and where we want to go seems really difficult to me right now for
example.


>
> No one should be able to add unnecessary features or reinvent the
> wheel. We want to avoid fragmentation and keep the codebase simple and
> consistent.
>

Yes I believe that sums up my opinion on that as well.


>
> But also, no valuable contributions should find themselves barred. If
> anything, we want to make the process of contributing smoother, and
> that also includes the process of rejecting proposals.
>

No, it really has never been my plan for Atlantis to now be a new
"official" route/habit or system for serviceability/monitoring systems. It
actually has never been in my plans for Atlantis to be anything
automatically massive/long lasting. I have TSAN and a few other monitoring
items that might interest the community but where there has to be a
conversation about the implementation details and an easy way for others in
the community to use/test and play with these before we can ascertain
usefulness across the board and then feel comfortable trying to push it
forward.

To be honest, I have debated on having the same bar of entry as Amber that
works on "features that have been accepted as candidate JEPs under the
OpenJDK JEP process". I chose not because I thought that I would prefer to
keep things a bit more open to debate in the initial stages of this project
(and because I'd like to be able to work on items in the open that might
not make it to a JEP because we figure out they are not useful for the
community).


>
> And if a contribution is considered worthwhile, it would be incubated
> separately from mainline, until it is ready to be upmerged.
>

Yes, were it to be put in Atlantis then this would be a reasonable way.


>
> ..
>
> Here is where I start getting confused: are we not doing most of this
> already? I would have thought most of this is common sense and at
> least in theory no-one would argue these points? (e.g. noone would
> argue that reinventing the wheel is good :)
>

Not exactly. Basically the problem is that right now the flow is the
following:
  - For a feature you're insterested, you can send an email about it and
see if it "sticks" (sound familiar Thomas? ;-))
  - If it doesn't or is "determined" not useful for the community, then
it's probably dead in the water (maybe not if you can come back with a
counter?)
  - But if it is deemed interesting, the implementation you had internally
might be very different than what could/should be done for the mainline
      - So now what do you do? How do you work in the open with other
people interested in helping? Say you and I wanted to work on a similar
feature, how could we do so?
            (Except a sandbox without any open mailing list)

 It is that last question that I'm trying to do better with the TSAN
feature because JEP-331 was a bit difficult in that respect :)



> Currently, we have the JEP process, we have area-specific mailing
> lists (serviceability-dev, jfr-dev etc). We have the sandbox repo for
> test branches. And larger contributions are already spun off own into
> own feature-specific projects with own mailing lists and repos.
>

  - This is exactly right:
     - I want exactly the "larger contributions are already spun off own
into own feature-specific projects with own mailing lists and repos."
because the TSAN project is big enough and complex enough to require it in
my opinion
     - I also have the other features that I know I'd like to work under
this same umbrella before presenting it to the main serviceability



>
> Based on our earlier discussions, I thought the main thing missing was
> a way to reach the experts via a cross-area channel. Since posting
> contribution proposals to serviceability-dev has been considered
> "spamming" (which surprised me btw), svc-dev is out of the question.
> But would not just a new mailing list suffice for this? Why a whole
> new umbrella project, new reviewer roles etc?
>


Actually the main thing I'm trying to do right now is to have the right
avenue to work on the TSAN feature and others, in the open, where others
could come and contribute so that we can figure out if the feature(s) are
mainline-able, of interest, and what would be the implementation and
use-cases that would work for most of the users in an inter-company world.
To do that and to do that in the open, I find no real way than to have a
side project where we can have conversations about implementation details
without adding noise to the serviceability-dev mailing list.

Btw, I'm not sure that proposing a feature is considered "spamming"; I
could see you and I having regular technical conversations about how to
implement a given "big" feature and talking about various bugs we are
seeing with our current feature version would then become disruptive.

If we have only a mailing list, it's already a good start but the project
allows us to have our own repos and branches to better serve what I'm
trying to accomplish, which is to have a better avenue for implementing the
serviceability features that I currently think we should have in the
mainline but do it in the open. I am kind of using the Amber project (
https://openjdk.java.net/projects/amber/) as a template for this.


>
> Especially since this umbrella project now conflicts with the idea of
> feature-specific projects. Now we have three options for a larger
> contribution: We can either discuss it on the area-specific mailing
> lists and add it directly to mainline, optionally with a JEP; we can
> also create an own feature-specific project, incubate it an own
> repository with an own mailing list, then add it to mainline. Now,
> with Atlantis, we have the third option of adding it to the
> Atlantis-specific repository (?), as a new branch (?) and discuss it
> on the Atlantis-specific mailing list.
>

I think that is the kind of thing that is open to conversation of what
would/could be done. I believe in doing tiny baby steps and seeing what
happens. Currently, my only goal was to open a project, have a repo where I
could start working on the TSAN feature and have people follow it and be
able to test it :-). I'd like it to be a bit open so that if someone comes
with a feature that could be useful but might need a major rewrite, we have
a means to do that easier than what I saw with my JEP-331 work but in no
way am I trying to make a "third" route to getting something in.

It's the major reason why I've wanted to have some "expert" serviceability
engineers at the start of this project to help maintain cohesiveness,
coherent systems, and not make things more complicated. I'm actually trying
to just make it easier for me, JC, to work on the features that I know are
not trivial, do not have a single way of doing them, and would require
buy-in and testing from various groups before being able to be considered
acceptable. But to be able to be easy to test and feel comfortable with, I
feel we need to have an easy way to clone/test it and a project is a good
way (yes a sandbox too but you have to be a committer and, for example, a
few engineers I'd like working on the TSAN feature are not yet OpenJDK
committers...)


>
> Sorry,  I really just try to understand. I think you have valid pain
> points you try to solve, I am just not sure if a whole new umbrella
> project is the best solution. Or, I could just not understanding the
> proposal.
>

No worries, I understand entirely. To be honest, I rather get this out of
the way now and we start (if we do) on solid footing.  I rather have
everyone on the same page and agreeing that:
  1) This is not trying to make things more complex
  2) This is not trying to add noise
  3) This is not trying to fragment

I'm only trying to have a place do add features that I believe are useful
in a way that makes sense and seems more open, easier to understand what we
are doing, easier to get feedback in the open and be traceable, etc. So let
us have these conversations to all try to figure out if:
  1) This could be something we can do
  2) This could be supported by the community
  3) What are the basic ground rules (which is what we are talking about in
this conversation)

My best estimate is to say "do not assume Atlantis is another route for
serviceability", imagine it is a potential route for items that
serviceability experts say "yeah sounds interesting but depends on how it
looks from an implementation point of view, why not go experiment a bit and
come back with something that some key-people agree to". But we are a long
ways from that; right now imagine it really as "a TSAN project with
potential to maybe perhaps at some point host other TSAN style projects
that engineers are interested in working on".


> Thanks & Kind Regards, Thomas
>
> p.s. if I as an author go for the third option, via atlantis-dev: I
> would like to be reasonably sure that once my contribution meets
> approval and gets incubated, chances are reasonably high that it will
> be merged into mainline later. Because, if atlantis-dev greenlights a
> feature and helps incubating it but later it gets barred from
> mainline, that would make Atlantis moot. This means that atlantis-dev
> must be monitored by the resident area experts.
>

I actually don't agree with this because that is true in the mainline
anyway:
  - Serviceability says yes that looks good
  - You get to a point where the implementation looks good
  - You get initial reviews, your JEP/CSR/test plan is well on its way and
something comes to derail it; that is part of life, the process; the system.

So for me, if something gets a green light in the Atlantis project, it does
not mean anything until it's in the mainline. And yes chances might be high
that it would get it in but it does not mean it will automatically get in;
if you stop working on it, if an alternative shows up, etc. why does it
make Atlantis moot? If this consistently happens, yes maybe (I could
advocate no but let's not).

Bottom line of the Atlantis project is: I would love to have a means to
work on the TSAN feature and other Google features in the open with the
community; I would love to work on serviceability projects that are not
Google specific in the open and see it happen :-);  and I'd really rather
have a project to do so in order to have a reasonable means to get other
engineers in the community to have a place to talk about them, test them,
talk about implementations, etc. And I think a project is actually the way
to go. I am not trying to create noise, fragment, derail, or make issues
with the way things are done; I'm only trying to do something a bit
different for the next potential JEP I would contribute that might be
easier on all involved :)

So to that effect, if we need to put the right words to make this cemented
in stone and explicit enough so that no one is worried that the Atlantis
project is trying to rock the boat, let us do that. Perhaps following the
Amber project and requiring a JEP candidate for any work done in the
Atlantis project will help ease certain minds and get this going. I think
it's not required because I'm not worried about it, I think that if we see
that this is not working with that requirement, we add it as a rule amongst
ourselves (I believe in iteratively fixing things like this).

Perhaps a middle ground would be to at least require the work to be under a
JEP draft which is signed off by one of the reviewers of Atlantis ? This
forces the work to be published in JBS, "signed" off by key members though
not to the point of a candidate JEP, and therefore help ease our minds?

Thanks again for your comments and hopefully my answers are helping,
Jc



>
> On Tue, Nov 20, 2018 at 9:36 PM JC Beyler <jcbeyler at google.com> wrote:
> >
> > Thanks all for the questions and feedback. I've tried to summarize the
> questions/concerns here:
> >
> > - I want to be really clear that I don't have all the answers, I believe
> very strongly in trial and error
> >     - Part of this project's motivation is to have these conversations :)
> >     - Another part is that I think there is something to be gained by
> this project and it is exactly what Mario has said above: that there be an
> "umbrella for experimentation; This will probably already cure most of the
> risk of such fragmentation, however, if it's a written statement in the JEP
> goals it makes more explicit that we are interested in enhancing the
> current frameworks rather than creating alternative versions (unless it
> *really* makes sense to do so)."
> >         -> Exactly, I'm not interested in creating the next "shiny
> thing" that actually does the same thing as existing items; I am interested
> in looking at an issue/monitoring feature; considering all existing
> frameworks; seeing where it would fit best; figure out if it is feasible
> acceptable; then move forward".
> >
> > - (From Thomas mainly but also Mario) Is there a risk of fragmentation
> of knowledge or fragmenting serviceability?
> >     - I don't think so nor do I want to; I'm actually against
> fragmentation in almost all cases to be very honest and clear; that is why
> I've tried to work closely with the serviceability team and engineers to
> figure out what could/should be the scope of this project
> >     - Let us not forget that in the end, the work done in this project
> still needs a JEP to get into the mainline and that is our last
> barrier/hurdle that ensures consistency in OpenJDK and not fragmentation
> >     - To finish this subject (and re-iterate), if an item is to do
> "yet-another-monitoring/performance-analysis tool" that could be done by
> the various ones we have already available, I believe it has no place in
> this project; on the other hand, if it can be proved that X cannot be done
> at all in a product environment with current systems then we can/should
> start asking the questions:
> >          - Is there a solution close, how could it be extended, what do
> the current developers of that mechanism think, should we try to prototype
> something to see the extent of changes, should we push it to being a JEP
> from this project? It is for those cases that I see a path forward via this
> project.
> >     - I think that the balance between what could go directly to
> mainline or via this project has to be found, I don't have the right
> answer, I have the items I know I'd like to work on openly via this project
> and then see what comes from them; but I'm open to suggestions :-)
> >        - Which means I don't have the right answer of what should go
> directly to serviceability-dev or here when it's a "Hey I have this" or
> "Hey could we that" type of question; I think we will see what happens,
> right now, I'm personally going to concentrate on the internal features we
> have at Google to start the conversation and the project work and see where
> that takes me/us if that helps. I'm open to any explicit guideline that is
> deemed necessary/warranted or we can do what I would hope is possible too:
> by having myself, Serguei, and core members of serviceability, we will work
> out the balance together as we move forward.
> >
> > From Andrew mainly: I'm going to quote Andrew's question/comment here:
> > "JC, can you provide assurance that Google staff involved in pushing any
> code upstream will also be available to help perform that maintenance
> effort, in this case presumably as members of the serviceability team?"
> >
> > (Pre-amble: I believe it is a valid question for any relatively
> new-comer to be asked this question and hopefully my answer does help have
> confidence in my actual resolve)
> >
> > I actually had a longer answer but thought I was diluting the actual
> answer that you are looking/hoping for:
> >    - I believe that this is true for any JEP being submitted: ie if you
> push code upstream, you should be available for maintaining that code
> onwards
> >    - I sincerely only can talk about myself and I think we are in
> agreement, any work coming in the mainline is additional burden and that is
> what the JEP process is about: defining the work, the testing, the risks
> and the maintenance; I believe that as we move forward, we would only
> shepherd JEPs to the mainline in which we are confident that they will be
> maintained right and not be added burden to the rest of the team
> >    - Finally, I think that this is the biggest reason we need to have
> the right people as reviewers, experts, and committed people to ensure this
> very fact; we do not want contribute-and-run scenarios; this not only hurts
> the code-base, it hurts the community at large.
> >
> > So I don't answer your question directly because I want this not to be a
> Google-specific project, nor do I want it to be Google is pushing code via
> this project. You have my assurance that any JEP that I contribute to and
> push forward has my commitment to be helping the maintenance of AND that
> I'll make very clear to whoever is trying to do work in Atlantis and trying
> to then go to the mainline via a JEP that the JEP must have a engineer
> maintenance assurance that we, the community, feel comfortable with. So I
> believe I do however answer that I think your sentence is the right one
> across the board and we should all abide by:
> > "There is a requirement that anything new of a significant size or
> complexity that gets developed under a JEP and then integrated into the
> mainline repos will have someone available to maintain it."; I don't think
> this needs to be stated in the project description but I can definitely add
> it as I believe in this probably more strongly than most ;-)
> >
> > To finish this first long answer, and as someone recommended to me as
> well, I am hoping to get serviceability core members and members from the
> various monitoring tools (JVMTI, JMX, JMC and JFR) to be interested and be
> able to follow up with opinions about key items:
> >     - "I think this could be done via JVMTI, JMX, JMC and JFR; we should
> either go do it via their projects/branches"
> >     - I have no problem a monitoring/perf. analysis team saying let's
> prototype/experiment this idea here in Atlantis; then we can see if will
> merge it with the mainline
> >     - This participation can be ad-hoc or via being an initial
> reviewer/committer
> >
> > And that should solve the questions that Mario had about that side.
> >
> > Hopefully my answer helps understand that this project is an attempt to
> have a venue to create conversations about current internal systems or
> non-existant ones and that we can see what "sticks" and what doesn't; where
> it would "stick" potentially, what it would like and then how could be best
> push it forward with the support of the whole community.
> >
> > I'll finish the way I started, I don't believe in fragmentation, I don't
> believe in trying to build "yet-another-X" when the existing systems
> suffice,
> > Jc
> >
> > On Tue, Nov 20, 2018 at 10:41 AM <jesper.wilhelmsson at oracle.com> wrote:
> >>
> >> On 20 Nov 2018, at 17:21, Mario Torre <neugens at redhat.com> wrote:
> >>
> >> On Tue, 2018-11-20 at 15:38 +0000, Andrew Dinn wrote:
> >>
> >> On 20/11/2018 15:14, Mario Torre wrote:
> >>
> >> On Tue, 2018-11-20 at 15:38 +0100, Thomas St?fe wrote:
> >>
> >> I'm worried about fragmentation though. Once we have this
> >> project,
> >> what is then the role of serviceability-dev? Just smallish
> >> everyday
> >> fixes? Also, the fact that VM developer feedback is hard to come
> >> by
> >> is
> >> mostly due to a lack of time on their part, and a new mailing
> >> list
> >> will not fix that.
> >>
> >>
> >> Generally projects have their own mailing lists for development
> >> until
> >> they are "finished", I would expect that for this project it would
> >> work
> >> the same, once it's integrated into mainline it becomes part of the
> >> serviceability code and the project mailing list is folded into
> >> serviceability-dev, until then it's somewhat an independent unit.
> >>
> >>
> >> Well, there is a requirement that anything new of a significant size
> >> orc
> >> complexity that gets developed under a JEP and then integrated into
> >> the
> >> mainline repos will have someone available to maintain it.
> >>
> >>
> >> Yes, indeed!
> >>
> >> JC, can you provide assurance that Google staff involved in pushing
> >> any
> >> code upstream will also be available to help perform that maintenance
> >> effort, in this case presumably as members of the serviceability
> >> team?
> >>
> >> n.b. Modulo the above detail I'm all for this proposal. I'd be very
> >> pleased if the project brings opportunities for Google either to
> >> upstream their existing work to everyone's benefit or, where for
> >> whatever reason that is not feasible, to help tailor existing
> >> upstream
> >> capabilities (such as JFR events) so they might encompass Google's
> >> downstream needs.
> >>
> >> It's very welcome to see Google so willing to contribute to OpenJDK.
> >>
> >>
> >> I agree!
> >>
> >> I actually wanted to ask that one written requirement in the JEP would
> >> be to extend Flight Recorder whenever it makes sense, even if this
> >> means going an extra step (in other words, the default option), so that
> >> we avoid fragmentation and the proliferation of a multitude of tools.
> >>
> >>
> >> +1
> >>
> >> Given that it is an explicit goal to extend and utilize existing
> frameworks rather than adding another one, I welcome this project.
> >> /Jesper
> >>
> >>
> >> I understand the project is meant to be some sort of umbrealla for
> >> experiementation? This will probably already cure most of the risk of
> >> such fragmentation, however, if it's a written statement in the JEP
> >> goals it makes more explicit that we are interested in enhancing the
> >> current frameworks rather than creating alternative versions (unless it
> >> *really* makes sense to do so).
> >>
> >> Cheers,
> >> Mario
> >> --
> >> Mario Torre
> >> Associate Manager, Software Engineering
> >> Red Hat GmbH <https://www.redhat.com>
> >> 9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898
> >>
> >>
> >
> >
> > --
> >
> > Thanks,
> > Jc
>


-- 

Thanks,
Jc

From zanglin5 at jd.com  Tue Nov 27 15:07:10 2018
From: zanglin5 at jd.com (=?gb2312?B?6rDB1Q==?=)
Date: Tue, 27 Nov 2018 15:07:10 +0000
Subject: Call for discussion: Extend JMap to support parallel and incremental
 heap scanning.
Message-ID: <46f011b2797a4a9ab0ef07153668b8a4@jd.com>

 Hello,

    This is Zang Lin,  JVM developer in JD.COM. My department just signed OCA and we will try to contribute to OpenJDK.   As I am newbie at openJDK, I'd like to discuss some of my ideas and ask for suggestions.


    Recently I tried to use "jmap -histo" to collect histogram info on a large heap at ~200G bytes.  it took about 160 seconds, which actually cause our online system suffering "non-responsive" issue.  In addition, if we stop jmap with "kill", jmap exits directly (even the attached JVM still working on heap iteration), leave no help info for analysis.



To address the issue above, I proposed following approaches:

1) Extend Jmap to be parallel and incremental, which means Jmap iterates heap in parallel and save the intermediate results incrementally during heap scanning (controlled by threshold to decide when to dump).

2) Make Jmap to accept arguments that enable/disable parallel and partial heap iterating, also make parallel thread number configurable.

3) Make Jmap to dump results to specified file path given by argument, rather than output to stdout directly.



    A prototype has been made internally with G1 upon JDK 11, and our preliminary measurement show that parallel iterate ~200GB heap with 4 threads can save ~30-40 seconds, and I am also analyzing the approach to make the time more scaled with parallel threads.  Moreover, with the implementation of incremental dump, we can get usefully partial dump info for analyzing, even when jmap hang for a long time, or get killed for some reason.



    Several problems were encountered during development. One problem is that it is not easy to dynamically allocate some data structures for thread local operations, such as KlassInfoTable,which is designed to be StackObj that not allowed to be allocated using "new", another problem is that some base classes definition in hotspot has to be modified to support parallel operations, such as ObjectClosure etc. I am not sure whether these modifications can be acceptable.



    Any suggestions/guidance would be much appreciated!



BRs,

Lin




From martijnverburg at gmail.com  Tue Nov 27 15:17:31 2018
From: martijnverburg at gmail.com (Martijn Verburg)
Date: Tue, 27 Nov 2018 15:17:31 +0000
Subject: Call for discussion: Extend JMap to support parallel and
 incremental heap scanning.
In-Reply-To: <46f011b2797a4a9ab0ef07153668b8a4@jd.com>
References: <46f011b2797a4a9ab0ef07153668b8a4@jd.com>
Message-ID: <CAP7YuAQho1ViT4s6X9uYZDETSGhg-kriGgXrC-ZNMj6pdAhP4Q@mail.gmail.com>

Hi Lin and welcome to OpenJDK!

There's an ongoing thread discussing project Atlantis (
http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html)
which I think will be looking at potential enhancements like this alongside
the serviceability group.

In the meantime, I think the serviceability-dev mailing list is the right
place to start your discussion.

Cheers,
Martijn


On Tue, 27 Nov 2018 at 15:08, ?? <zanglin5 at jd.com> wrote:

>  Hello,
>
>     This is Zang Lin,  JVM developer in JD.COM. My department just signed
> OCA and we will try to contribute to OpenJDK.   As I am newbie at openJDK,
> I'd like to discuss some of my ideas and ask for suggestions.
>
>
>     Recently I tried to use "jmap -histo" to collect histogram info on a
> large heap at ~200G bytes.  it took about 160 seconds, which actually cause
> our online system suffering "non-responsive" issue.  In addition, if we
> stop jmap with "kill", jmap exits directly (even the attached JVM still
> working on heap iteration), leave no help info for analysis.
>
>
>
> To address the issue above, I proposed following approaches:
>
> 1) Extend Jmap to be parallel and incremental, which means Jmap iterates
> heap in parallel and save the intermediate results incrementally during
> heap scanning (controlled by threshold to decide when to dump).
>
> 2) Make Jmap to accept arguments that enable/disable parallel and partial
> heap iterating, also make parallel thread number configurable.
>
> 3) Make Jmap to dump results to specified file path given by argument,
> rather than output to stdout directly.
>
>
>
>     A prototype has been made internally with G1 upon JDK 11, and our
> preliminary measurement show that parallel iterate ~200GB heap with 4
> threads can save ~30-40 seconds, and I am also analyzing the approach to
> make the time more scaled with parallel threads.  Moreover, with the
> implementation of incremental dump, we can get usefully partial dump info
> for analyzing, even when jmap hang for a long time, or get killed for some
> reason.
>
>
>
>     Several problems were encountered during development. One problem is
> that it is not easy to dynamically allocate some data structures for thread
> local operations, such as KlassInfoTable,which is designed to be StackObj
> that not allowed to be allocated using "new", another problem is that some
> base classes definition in hotspot has to be modified to support parallel
> operations, such as ObjectClosure etc. I am not sure whether these
> modifications can be acceptable.
>
>
>
>     Any suggestions/guidance would be much appreciated!
>
>
>
> BRs,
>
> Lin
>
>
>
>

From coleen.phillimore at oracle.com  Tue Nov 27 23:52:12 2018
From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com)
Date: Tue, 27 Nov 2018 18:52:12 -0500
Subject: JNI - question about jmethodid values.
In-Reply-To: <CAA-vtUwH_H1HcwHmEL8XDs3tQ0gJWgu84O=CqZbdXG5Jsb-3PQ@mail.gmail.com>
References: <CAK9Gx1ePUfKSNe6g7C7bfWZTD9R58v9Pm+OM4Ehs17JqR6-R3w@mail.gmail.com>
 <CAA-vtUztzLqMLssNWXmf78v3TGQmqMa8vGn0exEeV2f689=Mbw@mail.gmail.com>
 <CAK9Gx1ev6ov7iWt84s8whrAqP6Enb8kbY83Evz15d_KHmAh6nw@mail.gmail.com>
 <CAA-vtUyc-36Hzf0-KVAY_tN342zaqSnwkfhEvL517zsehjeuwQ@mail.gmail.com>
 <CAF9BGBxmN2TQuYvG=CVqXFSoDNCYYMe0TB9P7EBjASUfaw8PDg@mail.gmail.com>
 <CAA-vtUwH_H1HcwHmEL8XDs3tQ0gJWgu84O=CqZbdXG5Jsb-3PQ@mail.gmail.com>
Message-ID: <66b64194-cc03-f099-9a08-6f1d292f4e59@oracle.com>


Hi I have a couple of comments.

On 11/26/18 2:33 PM, Thomas St?fe wrote:
> Hey JC,
>
> On Mon, Nov 26, 2018 at 8:15 PM JC Beyler <jcbeyler at google.com> wrote:
>> Hi all,
>>
>> Just added my two cents on one comment:
>>
>> On Mon, Nov 26, 2018 at 5:00 AM Thomas St?fe <thomas.stuefe at gmail.com> wrote:
>>> Hi Peter,
>>>
>>> On Mon, Nov 26, 2018 at 1:02 PM Peter Hull <peterhull90 at gmail.com> wrote:
>>>> Hi Thomas,
>>>> Thank you very much for the detailed explanation. For your
>>>> information, the relevant NetBeans bug is
>>>> https://issues.apache.org/jira/browse/NETBEANS-1428
>>>>
>>>> On Sat, Nov 24, 2018 at 3:41 PM Thomas St?fe <thomas.stuefe at gmail.com> wrote:
>>>>> A jmethodid is a pointer to malloc'ed memory.
>>>> OK. Just in case I haven't understood it - does this mean a jmethodid,
>>>> once 'created', won't change (after garbage collection or whatever)?
>>> yes. It lives on, unchanged, forever. Even if the associated class is
>>> unloaded. That must be since outside JNI/JVMTI code may call in with
>>> outdated jmethodids which we must be able to handle gracefully.
>>>
>> This is the current design and most likely will remain for quite a bit but it is not defined by the SPEC to be like this so, in essence, it is an implementation detail that could be not true in future releases ;-)
> I politely disagree :)
>
> JNI spec says that you have a function like this:
>
> jmethodID GetMethodID(..)
>
> returning a jmethodid which you can then use in subsequent JNI
> functions. These functions are required to behave in a predictable way
> if the jmethodid has gotten invalid. The spec has no way to inform the
> caller if a jmethodid has gotten invalid by class unloading. Since the
> jmethodid is handed up to the caller and stored by him, there is no
> way to change its value on the fly either.
>
> So even though the spec does not specifically say that jmethodid lives
> forever, it is laid out in a way that prevents jmethodids from ever
> becoming invalid, and from changing their numerical value. This can of
> course change, but not without changing the JNI spec.
>
> I think my point is that Peter can rely on this behavior without fear
> that it will change in the future without him noticing. Before this
> behavior changes, the JNI spec would have to change.
>
> Cheers, Thomas

If the class is unloaded, the jmethodID is cleared.? Native code should 
first test whether it's NULL.? I think that is the predictable behavior 
that the spec requires.

Also, the jmethodIDs were never in the PermGen but in CHeap allocated 
memory blocks.? The Method* that was in the jmethodID was in the 
permgen.? I don't think there was any guarantee of contiguity (is that a 
word) but they are allocated together in a block.? After permgen was 
removed, the blocks were sorted by class loader, so that the class 
loader data could point to them for clearing.

Thanks,
Coleen

>
>
>
>
>
>
>> Thanks,
>> Jc
>>
>>>>> But it is not guaranteed to work. I would probably rather use a
>>>>> hashmap or similar.
>>>> I need to look at the implications on more detail but think it would
>>>> make sense to use long/jlong instead of int/jint on all platforms; the
>>>> extra memory use shouldn't be a problem. I think the IDs are just
>>>> stored on the Java side and used to get the method name and signature
>>>> later. That should be a loss-free cast, shouldn't it?
>>> Sure.
>>>
>>>>> If this is
>>>>> true, this 4x30bit assumption may actually have worked before jdk8,
>>>>> since the java heap is allocated as one continuous space, with the
>>>>> PermGen clustered in one part of it.
>>>> Indeed we did only start to get crashes on JDK9 and later (only
>>>> observed on Windows, macOS seems OK and other platforms have not been
>>>> tested)
>>>>
>>>> Yours sincerely,
>>>> Peter
>>> Cheers, Thomas
>>
>>
>> --
>>
>> Thanks,
>> Jc


From zanglin5 at jd.com  Wed Nov 28 02:30:40 2018
From: zanglin5 at jd.com (=?utf-8?B?6Ien55Cz?=)
Date: Wed, 28 Nov 2018 02:30:40 +0000
Subject: Call for discussion: Extend JMap to support parallel and
 incremental heap scanning.
Message-ID: <c3252c7579ab4f5cb5dcddf62ff4b06b@jd.com>

Hi Martijn,
       Thanks a lot for you guidance! I will go to serviceability-dev and try to join the Atlantis discussion.

Cheers,
Lin




???: Martijn Verburg [mailto:martijnverburg at gmail.com]
????: 2018?11?27? 23:18
???: ?? <zanglin5 at jd.com>
??: discuss at openjdk.java.net
??: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.

Hi Lin and welcome to OpenJDK!

There's an ongoing thread discussing project Atlantis (http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html) which I think will be looking at potential enhancements like this alongside the serviceability group.

In the meantime, I think the serviceability-dev mailing list is the right place to start your discussion.

Cheers,
Martijn


On Tue, 27 Nov 2018 at 15:08, ?? <zanglin5 at jd.com<mailto:zanglin5 at jd.com>> wrote:
 Hello,

    This is Zang Lin,  JVM developer in JD.COM<http://JD.COM>. My department just signed OCA and we will try to contribute to OpenJDK.   As I am newbie at openJDK, I'd like to discuss some of my ideas and ask for suggestions.


    Recently I tried to use "jmap -histo" to collect histogram info on a large heap at ~200G bytes.  it took about 160 seconds, which actually cause our online system suffering "non-responsive" issue.  In addition, if we stop jmap with "kill", jmap exits directly (even the attached JVM still working on heap iteration), leave no help info for analysis.



To address the issue above, I proposed following approaches:

1) Extend Jmap to be parallel and incremental, which means Jmap iterates heap in parallel and save the intermediate results incrementally during heap scanning (controlled by threshold to decide when to dump).

2) Make Jmap to accept arguments that enable/disable parallel and partial heap iterating, also make parallel thread number configurable.

3) Make Jmap to dump results to specified file path given by argument, rather than output to stdout directly.



    A prototype has been made internally with G1 upon JDK 11, and our preliminary measurement show that parallel iterate ~200GB heap with 4 threads can save ~30-40 seconds, and I am also analyzing the approach to make the time more scaled with parallel threads.  Moreover, with the implementation of incremental dump, we can get usefully partial dump info for analyzing, even when jmap hang for a long time, or get killed for some reason.



    Several problems were encountered during development. One problem is that it is not easy to dynamically allocate some data structures for thread local operations, such as KlassInfoTable,which is designed to be StackObj that not allowed to be allocated using "new", another problem is that some base classes definition in hotspot has to be modified to support parallel operations, such as ObjectClosure etc. I am not sure whether these modifications can be acceptable.



    Any suggestions/guidance would be much appreciated!



BRs,

Lin



From zanglin5 at jd.com  Wed Nov 28 02:47:38 2018
From: zanglin5 at jd.com (=?gb2312?B?6rDB1Q==?=)
Date: Wed, 28 Nov 2018 02:47:38 +0000
Subject: =?gb2312?B?tPC4tDogQ2FsbCBmb3IgZGlzY3Vzc2lvbjogRXh0ZW5kIEpNYXAgdG8gc3Vw?=
 =?gb2312?Q?port_parallel_and_incremental_heap_scanning.?=
In-Reply-To: <c3252c7579ab4f5cb5dcddf62ff4b06b@jd.com>
References: <c3252c7579ab4f5cb5dcddf62ff4b06b@jd.com>
Message-ID: <24357a4ebb1347cb8dcd5e4c199402b4@jd.com>

Add serviceability-dev.

Thanks.


________________________________
???: ??
????: 2018?11?28? 10:30
???: Martijn Verburg
??: discuss at openjdk.java.net
??: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.


Hi Martijn,

       Thanks a lot for you guidance! I will go to serviceability-dev and try to join the Atlantis discussion.



Cheers,

Lin









???: Martijn Verburg [mailto:martijnverburg at gmail.com]
????: 2018?11?27? 23:18
???: ?? <zanglin5 at jd.com>
??: discuss at openjdk.java.net
??: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.



Hi Lin and welcome to OpenJDK!



There's an ongoing thread discussing project Atlantis (http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html) which I think will be looking at potential enhancements like this alongside the serviceability group.



In the meantime, I think the serviceability-dev mailing list is the right place to start your discussion.


Cheers,
Martijn





On Tue, 27 Nov 2018 at 15:08, ?? <zanglin5 at jd.com<mailto:zanglin5 at jd.com>> wrote:

 Hello,

    This is Zang Lin,  JVM developer in JD.COM<http://JD.COM>. My department just signed OCA and we will try to contribute to OpenJDK.   As I am newbie at openJDK, I'd like to discuss some of my ideas and ask for suggestions.


    Recently I tried to use "jmap -histo" to collect histogram info on a large heap at ~200G bytes.  it took about 160 seconds, which actually cause our online system suffering "non-responsive" issue.  In addition, if we stop jmap with "kill", jmap exits directly (even the attached JVM still working on heap iteration), leave no help info for analysis.



To address the issue above, I proposed following approaches:

1) Extend Jmap to be parallel and incremental, which means Jmap iterates heap in parallel and save the intermediate results incrementally during heap scanning (controlled by threshold to decide when to dump).

2) Make Jmap to accept arguments that enable/disable parallel and partial heap iterating, also make parallel thread number configurable.

3) Make Jmap to dump results to specified file path given by argument, rather than output to stdout directly.



    A prototype has been made internally with G1 upon JDK 11, and our preliminary measurement show that parallel iterate ~200GB heap with 4 threads can save ~30-40 seconds, and I am also analyzing the approach to make the time more scaled with parallel threads.  Moreover, with the implementation of incremental dump, we can get usefully partial dump info for analyzing, even when jmap hang for a long time, or get killed for some reason.



    Several problems were encountered during development. One problem is that it is not easy to dynamically allocate some data structures for thread local operations, such as KlassInfoTable,which is designed to be StackObj that not allowed to be allocated using "new", another problem is that some base classes definition in hotspot has to be modified to support parallel operations, such as ObjectClosure etc. I am not sure whether these modifications can be acceptable.



    Any suggestions/guidance would be much appreciated!



BRs,

Lin



From magnus.ihse.bursie at oracle.com  Wed Nov 28 09:41:50 2018
From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie)
Date: Wed, 28 Nov 2018 10:41:50 +0100
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <CAF9BGBxLSeB4aVT5a8RuRmRtp47wwt_-OZ0pGEMXCFeOCF9a0w@mail.gmail.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <f92eb2fc-27fa-49d8-a111-8798f65029be.chuansheng.lcs@alibaba-inc.com>
 <CAF9BGBxLSeB4aVT5a8RuRmRtp47wwt_-OZ0pGEMXCFeOCF9a0w@mail.gmail.com>
Message-ID: <5632d395-1d13-3351-890d-2c736143aa51@oracle.com>

Hi JC,

Just a simple observation. Creating a project for the purpose of working 
on TSAN should really be a no-brainer. I don't think anyone would object 
to that. There are many projects like this, e.g. Portola, where there is 
a need for the infrastructure OpenJDK grants a project (repo, mailing 
list).


I think what maybe makes this proposal slightly more controversial is 
the broad and not well delimited scope. Maybe a simpler route would be 
to retarget your ideas for Atlantis from "any kind of servicability 
stuff that needs a project" to "trying to get TSAN in shape for 
integration into OpenJDK". And then, later on, when you feel you are 
ready to start tackling another serviceability feature as a project, ask 
the community if that should be a separate project Mu[1], or if Atlantis 
should have it scope widened to also handle this second project.

/Magnus

[1] https://en.wikipedia.org/wiki/Mu_(lost_continent) :-)

On 2018-11-26 20:22, JC Beyler wrote:
> Hi Jonathan,
>
> Thanks for the interest and sounds good to me! We could start conversations
> on this topic once the project opens and see what we could do in that
> problem-space :)
> Jc
>
> On Tue, Nov 20, 2018 at 7:46 PM Jonathan Lu <chuansheng.lcs at alibaba-inc.com>
> wrote:
>
>> Hi JC,
>>
>> Personally, I like this idea.
>>
>> And I'm especially interested in the heap dumping part.
>> It costs a lot to create, transfer, and analyze huge heap dump files for
>> my applications.
>> Maybe some fundamental changes can be explored in this project, I guess,
>> such as making an enhanced format of heap dump file :)
>>
>> Thank you
>> Jonathan
>>
>> ------------------------------------------------------------------
>> From:JC Beyler <jcbeyler at google.com>
>> Send Time:2018?11?20?(???) 02:48
>> To:discuss <discuss at openjdk.java.net>
>> Cc:serguei.spitsyn <serguei.spitsyn at oracle.com>
>> Subject:Call For Discussion: New Project: Atlantis
>>
>> Hi all,
>>
>> I would like to start a discussion about the possible creation of the
>> Atlantis Project with myself, Jean Christophe Beyler, as the Lead and the
>> Serviceability group as the sponsoring group.
>>
>> In accordance with the OpenJDK guidelines [1], this project would provide a
>> venue to explore and incubate monitoring and performance analysis features
>> that could be integrated into the Hotspot JVM and the JVM Tool Interface.
>> This includes working, evaluating, and incubating a Thread-Sanitizer
>> implementation [2] for Java.
>>
>> The Google platform team has historically worked on providing new features,
>> including monitoring hooks due to certain monitoring features not being
>> available in a low-overhead, turned on-by-default manner. Therefore, for
>> Google Java users, the platform team augmented or added mechanisms to
>> assess various metrics such as but not limited to:
>>
>> - Thread Sanitizer support for Java (see JDK-8208520 [2]) (though this one
>> specifically is not a low-overhead one, sorry ;-))
>> - Lock contention profiles to better understand where threads are spending
>> time in locks
>> - Heap dumping mechanisms to improve heap dump times for large heaps
>> - Efficient thread tagging to help filter profiling
>> - etc.
>>
>> All of the above have added changes to the JDK in various forms and the
>> Atlantis project proposes to investigate, in the open, how these changes
>> could be made general enough to be pushed into the mainline, or should be
>> dropped due to being too specific or could be done using other existing
>> mechanisms. The project would not be a Google-only project and will have a
>> mailing list to discuss and potentially accept any contribution trying to
>> extend and/or enhance current monitoring systems.
>>
>> The evaluation criteria to consider a feature/improvement includes but
>> would not be limited to:
>>
>>       * Performance: metrics and new hooks should not be hurting performance
>> when disabled
>>       * General usefulness for the community at large
>>       * No working alternatives: sometimes it may be that there are new
>> technologies available (ie JFR events for example) that might be leveraged
>> to deprecate existing solutions
>>
>> If one or more prototypes indicate a new metric, a new monitoring system,
>> or a feature offers substantial improvements, the Atlantis project would
>> shepherd a JEP to integrate the change into the mainline of OpenJDK.
>>
>> The current suggested initial Reviewers and Committers would be but is most
>> likely not going to be limited to:
>> * Jean Christophe Beyler
>> * Serguei Spitsyn
>> * Chuck Rasbold
>> * Liam Cushon
>>
>> What do you all think? :-)
>>
>> Thanks,
>> Jc
>>
>> [1] http://openjdk.java.net/projects/#new-project
>> [2] https://bugs.openjdk.java.net/browse/JDK-8208520
>>
>>
>>


From jcbeyler at google.com  Thu Nov 29 22:25:41 2018
From: jcbeyler at google.com (JC Beyler)
Date: Thu, 29 Nov 2018 14:25:41 -0800
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <5632d395-1d13-3351-890d-2c736143aa51@oracle.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <f92eb2fc-27fa-49d8-a111-8798f65029be.chuansheng.lcs@alibaba-inc.com>
 <CAF9BGBxLSeB4aVT5a8RuRmRtp47wwt_-OZ0pGEMXCFeOCF9a0w@mail.gmail.com>
 <5632d395-1d13-3351-890d-2c736143aa51@oracle.com>
Message-ID: <CAF9BGBwbh0cg0r8vB2ozRw2ciECB3eu1wd5gWcXYrXm83Hk0uA@mail.gmail.com>

Hi Magnus,

Thanks for the observation and it is much appreciated. You are probably
right and an easier means to get us going would be to start with a TSAN
specific project and then we can figure out what to do with the next
feature that might require to be hosted via a project (ie should it be
under Atlantis or under Mu).

This would mean that the CFD shifts itself to this new project description:

-------------------------------------------------------

I would like to start a discussion about the possible creation of the
Atlantis Project with myself, Jean Christophe Beyler, as the Lead and the
Serviceability group as the sponsoring group.

In accordance with the OpenJDK guidelines [1], this project would provide a
venue to explore and incubate a Thread Sanitizing feature that could be
integrated into the Hotspot JVM and the JVM Tool Interface. This includes
working, evaluating, and incubating a Thread-Sanitizer implementation [2]
for Java.

The Google platform team has historically worked on a TSAN project
internally. The Atlantis project proposes to investigate, in the open, how
these changes could be made general enough to be pushed into the mainline,
or should be dropped due to being too specific or could be done using other
existing mechanisms. The project would not be a Google-only project and
will have a mailing list to discuss and potentially accept any contribution
trying to extend and/or enhance the TSAN implementation.

The evaluation criteria for the TSAN implementation includes but would not
be limited to:
     * Performance: not adding overhead to non-tsan runs is paramount and
there are various means to do so
     * General usefulness for the community at large so that the TSAN
feature is deemed a good feature to have

If the TSAN implementation becomes stable and is deemed useful enough, the
Atlantis project would shepherd a JEP to integrate the change into the
mainline of OpenJDK.

The current suggested initial Reviewers and Committers would be but is most
likely not going to be limited to:
* Jean Christophe Beyler
* Serguei Spitsyn
* Chuck Rasbold
* Liam Cushon
  ... TBD ...

[1] http://openjdk.java.net/projects/#new-project
[2] https://bugs.openjdk.java.net/browse/JDK-8208520

---------------------------------

What do you all think? :-)

Thanks,
Jc

On Wed, Nov 28, 2018 at 1:41 AM Magnus Ihse Bursie <
magnus.ihse.bursie at oracle.com> wrote:

> Hi JC,
>
> Just a simple observation. Creating a project for the purpose of working
> on TSAN should really be a no-brainer. I don't think anyone would object
> to that. There are many projects like this, e.g. Portola, where there is
> a need for the infrastructure OpenJDK grants a project (repo, mailing
> list).
>
>
> I think what maybe makes this proposal slightly more controversial is
> the broad and not well delimited scope. Maybe a simpler route would be
> to retarget your ideas for Atlantis from "any kind of servicability
> stuff that needs a project" to "trying to get TSAN in shape for
> integration into OpenJDK". And then, later on, when you feel you are
> ready to start tackling another serviceability feature as a project, ask
> the community if that should be a separate project Mu[1], or if Atlantis
> should have it scope widened to also handle this second project.
>
> /Magnus
>
> [1] https://en.wikipedia.org/wiki/Mu_(lost_continent) :-)
>
> On 2018-11-26 20:22, JC Beyler wrote:
> > Hi Jonathan,
> >
> > Thanks for the interest and sounds good to me! We could start
> conversations
> > on this topic once the project opens and see what we could do in that
> > problem-space :)
> > Jc
> >
> > On Tue, Nov 20, 2018 at 7:46 PM Jonathan Lu <
> chuansheng.lcs at alibaba-inc.com>
> > wrote:
> >
> >> Hi JC,
> >>
> >> Personally, I like this idea.
> >>
> >> And I'm especially interested in the heap dumping part.
> >> It costs a lot to create, transfer, and analyze huge heap dump files for
> >> my applications.
> >> Maybe some fundamental changes can be explored in this project, I guess,
> >> such as making an enhanced format of heap dump file :)
> >>
> >> Thank you
> >> Jonathan
> >>
> >> ------------------------------------------------------------------
> >> From:JC Beyler <jcbeyler at google.com>
> >> Send Time:2018?11?20?(???) 02:48
> >> To:discuss <discuss at openjdk.java.net>
> >> Cc:serguei.spitsyn <serguei.spitsyn at oracle.com>
> >> Subject:Call For Discussion: New Project: Atlantis
> >>
> >> Hi all,
> >>
> >> I would like to start a discussion about the possible creation of the
> >> Atlantis Project with myself, Jean Christophe Beyler, as the Lead and
> the
> >> Serviceability group as the sponsoring group.
> >>
> >> In accordance with the OpenJDK guidelines [1], this project would
> provide a
> >> venue to explore and incubate monitoring and performance analysis
> features
> >> that could be integrated into the Hotspot JVM and the JVM Tool
> Interface.
> >> This includes working, evaluating, and incubating a Thread-Sanitizer
> >> implementation [2] for Java.
> >>
> >> The Google platform team has historically worked on providing new
> features,
> >> including monitoring hooks due to certain monitoring features not being
> >> available in a low-overhead, turned on-by-default manner. Therefore, for
> >> Google Java users, the platform team augmented or added mechanisms to
> >> assess various metrics such as but not limited to:
> >>
> >> - Thread Sanitizer support for Java (see JDK-8208520 [2]) (though this
> one
> >> specifically is not a low-overhead one, sorry ;-))
> >> - Lock contention profiles to better understand where threads are
> spending
> >> time in locks
> >> - Heap dumping mechanisms to improve heap dump times for large heaps
> >> - Efficient thread tagging to help filter profiling
> >> - etc.
> >>
> >> All of the above have added changes to the JDK in various forms and the
> >> Atlantis project proposes to investigate, in the open, how these changes
> >> could be made general enough to be pushed into the mainline, or should
> be
> >> dropped due to being too specific or could be done using other existing
> >> mechanisms. The project would not be a Google-only project and will
> have a
> >> mailing list to discuss and potentially accept any contribution trying
> to
> >> extend and/or enhance current monitoring systems.
> >>
> >> The evaluation criteria to consider a feature/improvement includes but
> >> would not be limited to:
> >>
> >>       * Performance: metrics and new hooks should not be hurting
> performance
> >> when disabled
> >>       * General usefulness for the community at large
> >>       * No working alternatives: sometimes it may be that there are new
> >> technologies available (ie JFR events for example) that might be
> leveraged
> >> to deprecate existing solutions
> >>
> >> If one or more prototypes indicate a new metric, a new monitoring
> system,
> >> or a feature offers substantial improvements, the Atlantis project would
> >> shepherd a JEP to integrate the change into the mainline of OpenJDK.
> >>
> >> The current suggested initial Reviewers and Committers would be but is
> most
> >> likely not going to be limited to:
> >> * Jean Christophe Beyler
> >> * Serguei Spitsyn
> >> * Chuck Rasbold
> >> * Liam Cushon
> >>
> >> What do you all think? :-)
> >>
> >> Thanks,
> >> Jc
> >>
> >> [1] http://openjdk.java.net/projects/#new-project
> >> [2] https://bugs.openjdk.java.net/browse/JDK-8208520
> >>
> >>
> >>
>
>

-- 

Thanks,
Jc

From neugens.limasoftware at gmail.com  Fri Nov 30 10:38:00 2018
From: neugens.limasoftware at gmail.com (Mario Torre)
Date: Fri, 30 Nov 2018 11:38:00 +0100
Subject: Free Java Devroom deadline extended!
Message-ID: <CAGUMyaTaNik7TC3YEFHBPb_=t6ppZn=JVi71JH4q-5Q+vMhkeQ@mail.gmail.com>

Hello all!

I hope you are recovering from the Black Friday rush that generated
almost as many emails as there are OpenJDK mailing lists ;)

To celebrate this surreal new found quiet, we decided to send you
another email with attachments: the deadline extension for the Java
DevRoom.

This offer will expire on the 12th of December 2018, one minute before
midnight: 23:59 CET, and there won't be any further extension after
that, so don't let the date pass without submitting a CFP, or all your
unread emails will turn into pumpkins[1].

More informations, as always, can be found on the following link, or
at least until that one also turns into something else ;)

https://wiki.debian.org/Java/DevJam/2019/Fosdem/CallForParticipation

Cheers,
Mario

[1] Or may not, well, that's a fine art, but it's not bug free. The
pumpkins I mean.
-- 
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/

From jesper.wilhelmsson at oracle.com  Fri Nov 30 12:01:10 2018
From: jesper.wilhelmsson at oracle.com (jesper.wilhelmsson at oracle.com)
Date: Fri, 30 Nov 2018 13:01:10 +0100
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <CAF9BGBwbh0cg0r8vB2ozRw2ciECB3eu1wd5gWcXYrXm83Hk0uA@mail.gmail.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <f92eb2fc-27fa-49d8-a111-8798f65029be.chuansheng.lcs@alibaba-inc.com>
 <CAF9BGBxLSeB4aVT5a8RuRmRtp47wwt_-OZ0pGEMXCFeOCF9a0w@mail.gmail.com>
 <5632d395-1d13-3351-890d-2c736143aa51@oracle.com>
 <CAF9BGBwbh0cg0r8vB2ozRw2ciECB3eu1wd5gWcXYrXm83Hk0uA@mail.gmail.com>
Message-ID: <9B5CAF76-0558-46D3-9E58-844A58DDA528@oracle.com>

Looks good to me.
/Jesper

> On 29 Nov 2018, at 23:25, JC Beyler <jcbeyler at google.com> wrote:
> 
> Hi Magnus,
> 
> Thanks for the observation and it is much appreciated. You are probably
> right and an easier means to get us going would be to start with a TSAN
> specific project and then we can figure out what to do with the next
> feature that might require to be hosted via a project (ie should it be
> under Atlantis or under Mu).
> 
> This would mean that the CFD shifts itself to this new project description:
> 
> -------------------------------------------------------
> 
> I would like to start a discussion about the possible creation of the
> Atlantis Project with myself, Jean Christophe Beyler, as the Lead and the
> Serviceability group as the sponsoring group.
> 
> In accordance with the OpenJDK guidelines [1], this project would provide a
> venue to explore and incubate a Thread Sanitizing feature that could be
> integrated into the Hotspot JVM and the JVM Tool Interface. This includes
> working, evaluating, and incubating a Thread-Sanitizer implementation [2]
> for Java.
> 
> The Google platform team has historically worked on a TSAN project
> internally. The Atlantis project proposes to investigate, in the open, how
> these changes could be made general enough to be pushed into the mainline,
> or should be dropped due to being too specific or could be done using other
> existing mechanisms. The project would not be a Google-only project and
> will have a mailing list to discuss and potentially accept any contribution
> trying to extend and/or enhance the TSAN implementation.
> 
> The evaluation criteria for the TSAN implementation includes but would not
> be limited to:
>     * Performance: not adding overhead to non-tsan runs is paramount and
> there are various means to do so
>     * General usefulness for the community at large so that the TSAN
> feature is deemed a good feature to have
> 
> If the TSAN implementation becomes stable and is deemed useful enough, the
> Atlantis project would shepherd a JEP to integrate the change into the
> mainline of OpenJDK.
> 
> The current suggested initial Reviewers and Committers would be but is most
> likely not going to be limited to:
> * Jean Christophe Beyler
> * Serguei Spitsyn
> * Chuck Rasbold
> * Liam Cushon
>  ... TBD ...
> 
> [1] http://openjdk.java.net/projects/#new-project
> [2] https://bugs.openjdk.java.net/browse/JDK-8208520
> 
> ---------------------------------
> 
> What do you all think? :-)
> 
> Thanks,
> Jc
> 
> On Wed, Nov 28, 2018 at 1:41 AM Magnus Ihse Bursie <
> magnus.ihse.bursie at oracle.com> wrote:
> 
>> Hi JC,
>> 
>> Just a simple observation. Creating a project for the purpose of working
>> on TSAN should really be a no-brainer. I don't think anyone would object
>> to that. There are many projects like this, e.g. Portola, where there is
>> a need for the infrastructure OpenJDK grants a project (repo, mailing
>> list).
>> 
>> 
>> I think what maybe makes this proposal slightly more controversial is
>> the broad and not well delimited scope. Maybe a simpler route would be
>> to retarget your ideas for Atlantis from "any kind of servicability
>> stuff that needs a project" to "trying to get TSAN in shape for
>> integration into OpenJDK". And then, later on, when you feel you are
>> ready to start tackling another serviceability feature as a project, ask
>> the community if that should be a separate project Mu[1], or if Atlantis
>> should have it scope widened to also handle this second project.
>> 
>> /Magnus
>> 
>> [1] https://en.wikipedia.org/wiki/Mu_(lost_continent) :-)
>> 
>> On 2018-11-26 20:22, JC Beyler wrote:
>>> Hi Jonathan,
>>> 
>>> Thanks for the interest and sounds good to me! We could start
>> conversations
>>> on this topic once the project opens and see what we could do in that
>>> problem-space :)
>>> Jc
>>> 
>>> On Tue, Nov 20, 2018 at 7:46 PM Jonathan Lu <
>> chuansheng.lcs at alibaba-inc.com>
>>> wrote:
>>> 
>>>> Hi JC,
>>>> 
>>>> Personally, I like this idea.
>>>> 
>>>> And I'm especially interested in the heap dumping part.
>>>> It costs a lot to create, transfer, and analyze huge heap dump files for
>>>> my applications.
>>>> Maybe some fundamental changes can be explored in this project, I guess,
>>>> such as making an enhanced format of heap dump file :)
>>>> 
>>>> Thank you
>>>> Jonathan
>>>> 
>>>> ------------------------------------------------------------------
>>>> From:JC Beyler <jcbeyler at google.com>
>>>> Send Time:2018?11?20?(???) 02:48
>>>> To:discuss <discuss at openjdk.java.net>
>>>> Cc:serguei.spitsyn <serguei.spitsyn at oracle.com>
>>>> Subject:Call For Discussion: New Project: Atlantis
>>>> 
>>>> Hi all,
>>>> 
>>>> I would like to start a discussion about the possible creation of the
>>>> Atlantis Project with myself, Jean Christophe Beyler, as the Lead and
>> the
>>>> Serviceability group as the sponsoring group.
>>>> 
>>>> In accordance with the OpenJDK guidelines [1], this project would
>> provide a
>>>> venue to explore and incubate monitoring and performance analysis
>> features
>>>> that could be integrated into the Hotspot JVM and the JVM Tool
>> Interface.
>>>> This includes working, evaluating, and incubating a Thread-Sanitizer
>>>> implementation [2] for Java.
>>>> 
>>>> The Google platform team has historically worked on providing new
>> features,
>>>> including monitoring hooks due to certain monitoring features not being
>>>> available in a low-overhead, turned on-by-default manner. Therefore, for
>>>> Google Java users, the platform team augmented or added mechanisms to
>>>> assess various metrics such as but not limited to:
>>>> 
>>>> - Thread Sanitizer support for Java (see JDK-8208520 [2]) (though this
>> one
>>>> specifically is not a low-overhead one, sorry ;-))
>>>> - Lock contention profiles to better understand where threads are
>> spending
>>>> time in locks
>>>> - Heap dumping mechanisms to improve heap dump times for large heaps
>>>> - Efficient thread tagging to help filter profiling
>>>> - etc.
>>>> 
>>>> All of the above have added changes to the JDK in various forms and the
>>>> Atlantis project proposes to investigate, in the open, how these changes
>>>> could be made general enough to be pushed into the mainline, or should
>> be
>>>> dropped due to being too specific or could be done using other existing
>>>> mechanisms. The project would not be a Google-only project and will
>> have a
>>>> mailing list to discuss and potentially accept any contribution trying
>> to
>>>> extend and/or enhance current monitoring systems.
>>>> 
>>>> The evaluation criteria to consider a feature/improvement includes but
>>>> would not be limited to:
>>>> 
>>>>      * Performance: metrics and new hooks should not be hurting
>> performance
>>>> when disabled
>>>>      * General usefulness for the community at large
>>>>      * No working alternatives: sometimes it may be that there are new
>>>> technologies available (ie JFR events for example) that might be
>> leveraged
>>>> to deprecate existing solutions
>>>> 
>>>> If one or more prototypes indicate a new metric, a new monitoring
>> system,
>>>> or a feature offers substantial improvements, the Atlantis project would
>>>> shepherd a JEP to integrate the change into the mainline of OpenJDK.
>>>> 
>>>> The current suggested initial Reviewers and Committers would be but is
>> most
>>>> likely not going to be limited to:
>>>> * Jean Christophe Beyler
>>>> * Serguei Spitsyn
>>>> * Chuck Rasbold
>>>> * Liam Cushon
>>>> 
>>>> What do you all think? :-)
>>>> 
>>>> Thanks,
>>>> Jc
>>>> 
>>>> [1] http://openjdk.java.net/projects/#new-project
>>>> [2] https://bugs.openjdk.java.net/browse/JDK-8208520
>>>> 
>>>> 
>>>> 
>> 
>> 
> 
> --
> 
> Thanks,
> Jc


From thomas.stuefe at gmail.com  Fri Nov 30 12:48:55 2018
From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=)
Date: Fri, 30 Nov 2018 13:48:55 +0100
Subject: Call For Discussion: New Project: Atlantis
In-Reply-To: <CAF9BGBwbh0cg0r8vB2ozRw2ciECB3eu1wd5gWcXYrXm83Hk0uA@mail.gmail.com>
References: <CAF9BGBzGWM9P9GN5GdBzj4VJxBraz5+VKkJyirn0iW9ENg4AdQ@mail.gmail.com>
 <f92eb2fc-27fa-49d8-a111-8798f65029be.chuansheng.lcs@alibaba-inc.com>
 <CAF9BGBxLSeB4aVT5a8RuRmRtp47wwt_-OZ0pGEMXCFeOCF9a0w@mail.gmail.com>
 <5632d395-1d13-3351-890d-2c736143aa51@oracle.com>
 <CAF9BGBwbh0cg0r8vB2ozRw2ciECB3eu1wd5gWcXYrXm83Hk0uA@mail.gmail.com>
Message-ID: <CAA-vtUz+hQT6BHAX_h4OtgokD4RQ9_eUH+yaDRtrHd=NSRt5Mw@mail.gmail.com>

Hi JC,

This looks fine and it is a pragmatic approach.

I still think parts of your original vision have merit, namely having
a new mailing list for discussing serviceability proposals which are
larger or span multiple areas.

Cheers, Thomas


On Thu, Nov 29, 2018 at 11:27 PM JC Beyler <jcbeyler at google.com> wrote:
>
> Hi Magnus,
>
> Thanks for the observation and it is much appreciated. You are probably
> right and an easier means to get us going would be to start with a TSAN
> specific project and then we can figure out what to do with the next
> feature that might require to be hosted via a project (ie should it be
> under Atlantis or under Mu).
>
> This would mean that the CFD shifts itself to this new project description:
>
> -------------------------------------------------------
>
> I would like to start a discussion about the possible creation of the
> Atlantis Project with myself, Jean Christophe Beyler, as the Lead and the
> Serviceability group as the sponsoring group.
>
> In accordance with the OpenJDK guidelines [1], this project would provide a
> venue to explore and incubate a Thread Sanitizing feature that could be
> integrated into the Hotspot JVM and the JVM Tool Interface. This includes
> working, evaluating, and incubating a Thread-Sanitizer implementation [2]
> for Java.
>
> The Google platform team has historically worked on a TSAN project
> internally. The Atlantis project proposes to investigate, in the open, how
> these changes could be made general enough to be pushed into the mainline,
> or should be dropped due to being too specific or could be done using other
> existing mechanisms. The project would not be a Google-only project and
> will have a mailing list to discuss and potentially accept any contribution
> trying to extend and/or enhance the TSAN implementation.
>
> The evaluation criteria for the TSAN implementation includes but would not
> be limited to:
>      * Performance: not adding overhead to non-tsan runs is paramount and
> there are various means to do so
>      * General usefulness for the community at large so that the TSAN
> feature is deemed a good feature to have
>
> If the TSAN implementation becomes stable and is deemed useful enough, the
> Atlantis project would shepherd a JEP to integrate the change into the
> mainline of OpenJDK.
>
> The current suggested initial Reviewers and Committers would be but is most
> likely not going to be limited to:
> * Jean Christophe Beyler
> * Serguei Spitsyn
> * Chuck Rasbold
> * Liam Cushon
>   ... TBD ...
>
> [1] http://openjdk.java.net/projects/#new-project
> [2] https://bugs.openjdk.java.net/browse/JDK-8208520
>
> ---------------------------------
>
> What do you all think? :-)
>
> Thanks,
> Jc
>
> On Wed, Nov 28, 2018 at 1:41 AM Magnus Ihse Bursie <
> magnus.ihse.bursie at oracle.com> wrote:
>
> > Hi JC,
> >
> > Just a simple observation. Creating a project for the purpose of working
> > on TSAN should really be a no-brainer. I don't think anyone would object
> > to that. There are many projects like this, e.g. Portola, where there is
> > a need for the infrastructure OpenJDK grants a project (repo, mailing
> > list).
> >
> >
> > I think what maybe makes this proposal slightly more controversial is
> > the broad and not well delimited scope. Maybe a simpler route would be
> > to retarget your ideas for Atlantis from "any kind of servicability
> > stuff that needs a project" to "trying to get TSAN in shape for
> > integration into OpenJDK". And then, later on, when you feel you are
> > ready to start tackling another serviceability feature as a project, ask
> > the community if that should be a separate project Mu[1], or if Atlantis
> > should have it scope widened to also handle this second project.
> >
> > /Magnus
> >
> > [1] https://en.wikipedia.org/wiki/Mu_(lost_continent) :-)
> >
> > On 2018-11-26 20:22, JC Beyler wrote:
> > > Hi Jonathan,
> > >
> > > Thanks for the interest and sounds good to me! We could start
> > conversations
> > > on this topic once the project opens and see what we could do in that
> > > problem-space :)
> > > Jc
> > >
> > > On Tue, Nov 20, 2018 at 7:46 PM Jonathan Lu <
> > chuansheng.lcs at alibaba-inc.com>
> > > wrote:
> > >
> > >> Hi JC,
> > >>
> > >> Personally, I like this idea.
> > >>
> > >> And I'm especially interested in the heap dumping part.
> > >> It costs a lot to create, transfer, and analyze huge heap dump files for
> > >> my applications.
> > >> Maybe some fundamental changes can be explored in this project, I guess,
> > >> such as making an enhanced format of heap dump file :)
> > >>
> > >> Thank you
> > >> Jonathan
> > >>
> > >> ------------------------------------------------------------------
> > >> From:JC Beyler <jcbeyler at google.com>
> > >> Send Time:2018?11?20?(???) 02:48
> > >> To:discuss <discuss at openjdk.java.net>
> > >> Cc:serguei.spitsyn <serguei.spitsyn at oracle.com>
> > >> Subject:Call For Discussion: New Project: Atlantis
> > >>
> > >> Hi all,
> > >>
> > >> I would like to start a discussion about the possible creation of the
> > >> Atlantis Project with myself, Jean Christophe Beyler, as the Lead and
> > the
> > >> Serviceability group as the sponsoring group.
> > >>
> > >> In accordance with the OpenJDK guidelines [1], this project would
> > provide a
> > >> venue to explore and incubate monitoring and performance analysis
> > features
> > >> that could be integrated into the Hotspot JVM and the JVM Tool
> > Interface.
> > >> This includes working, evaluating, and incubating a Thread-Sanitizer
> > >> implementation [2] for Java.
> > >>
> > >> The Google platform team has historically worked on providing new
> > features,
> > >> including monitoring hooks due to certain monitoring features not being
> > >> available in a low-overhead, turned on-by-default manner. Therefore, for
> > >> Google Java users, the platform team augmented or added mechanisms to
> > >> assess various metrics such as but not limited to:
> > >>
> > >> - Thread Sanitizer support for Java (see JDK-8208520 [2]) (though this
> > one
> > >> specifically is not a low-overhead one, sorry ;-))
> > >> - Lock contention profiles to better understand where threads are
> > spending
> > >> time in locks
> > >> - Heap dumping mechanisms to improve heap dump times for large heaps
> > >> - Efficient thread tagging to help filter profiling
> > >> - etc.
> > >>
> > >> All of the above have added changes to the JDK in various forms and the
> > >> Atlantis project proposes to investigate, in the open, how these changes
> > >> could be made general enough to be pushed into the mainline, or should
> > be
> > >> dropped due to being too specific or could be done using other existing
> > >> mechanisms. The project would not be a Google-only project and will
> > have a
> > >> mailing list to discuss and potentially accept any contribution trying
> > to
> > >> extend and/or enhance current monitoring systems.
> > >>
> > >> The evaluation criteria to consider a feature/improvement includes but
> > >> would not be limited to:
> > >>
> > >>       * Performance: metrics and new hooks should not be hurting
> > performance
> > >> when disabled
> > >>       * General usefulness for the community at large
> > >>       * No working alternatives: sometimes it may be that there are new
> > >> technologies available (ie JFR events for example) that might be
> > leveraged
> > >> to deprecate existing solutions
> > >>
> > >> If one or more prototypes indicate a new metric, a new monitoring
> > system,
> > >> or a feature offers substantial improvements, the Atlantis project would
> > >> shepherd a JEP to integrate the change into the mainline of OpenJDK.
> > >>
> > >> The current suggested initial Reviewers and Committers would be but is
> > most
> > >> likely not going to be limited to:
> > >> * Jean Christophe Beyler
> > >> * Serguei Spitsyn
> > >> * Chuck Rasbold
> > >> * Liam Cushon
> > >>
> > >> What do you all think? :-)
> > >>
> > >> Thanks,
> > >> Jc
> > >>
> > >> [1] http://openjdk.java.net/projects/#new-project
> > >> [2] https://bugs.openjdk.java.net/browse/JDK-8208520
> > >>
> > >>
> > >>
> >
> >
>
> --
>
> Thanks,
> Jc