Call For Discussion: New Project: Atlantis

[JC Beyler]

Hi Thomas,

Let me inline my answers and hopefully it will help understand what I'm
trying to accomplish.

On Mon, Nov 26, 2018 at 1:51 PM Thomas Stüfe <thomas.stuefe at gmail.com>
wrote:

> Hi JC, guys,
>
> (JC: I know we had a similar discussion offlist some days ago, sorry
> for raising the same points, still being unsure here).
>

No worries :). Please note that your comments are well received,
appreciated and I believe that they are useful to have in order to have all
involved agreeing on key parts, feeling good about it and not worried about
anything involved with this potential project.


>
> Here is what I think I understand:
>
> We want a "council" of sorts of experts in the general
> monitoring/serviceability area. A one-stop point to discuss larger
> planned contributions. They should advice on new developments,
> redirect them, shepherd them, curtail them when necessary. Impose a
> common strategy we all consent to (e.g. "JFR first").
>

Not at all to be honest. In a nutshell, I just want a place where I can
work on a few feature implementations that are either tricky/have multiple
implementation possibilities and that will take time to get right (seems to
me to be the reason to have a project :-)). And because I'd like to get it
right as soon as possible, I'd rather have conversations about the
implementations in the open with whoever is interested and able to follow.
And because I'd like to do it with people inter-company, it seems also the
way to go. For example, having conversations about Thread Sanitizing (TSAN)
details and where we want to go seems really difficult to me right now for
example.


>
> No one should be able to add unnecessary features or reinvent the
> wheel. We want to avoid fragmentation and keep the codebase simple and
> consistent.
>

Yes I believe that sums up my opinion on that as well.


>
> But also, no valuable contributions should find themselves barred. If
> anything, we want to make the process of contributing smoother, and
> that also includes the process of rejecting proposals.
>

No, it really has never been my plan for Atlantis to now be a new
"official" route/habit or system for serviceability/monitoring systems. It
actually has never been in my plans for Atlantis to be anything
automatically massive/long lasting. I have TSAN and a few other monitoring
items that might interest the community but where there has to be a
conversation about the implementation details and an easy way for others in
the community to use/test and play with these before we can ascertain
usefulness across the board and then feel comfortable trying to push it
forward.

To be honest, I have debated on having the same bar of entry as Amber that
works on "features that have been accepted as candidate JEPs under the
OpenJDK JEP process". I chose not because I thought that I would prefer to
keep things a bit more open to debate in the initial stages of this project
(and because I'd like to be able to work on items in the open that might
not make it to a JEP because we figure out they are not useful for the
community).


>
> And if a contribution is considered worthwhile, it would be incubated
> separately from mainline, until it is ready to be upmerged.
>

Yes, were it to be put in Atlantis then this would be a reasonable way.


>
> ..
>
> Here is where I start getting confused: are we not doing most of this
> already? I would have thought most of this is common sense and at
> least in theory no-one would argue these points? (e.g. noone would
> argue that reinventing the wheel is good :)
>

Not exactly. Basically the problem is that right now the flow is the
following:
  - For a feature you're insterested, you can send an email about it and
see if it "sticks" (sound familiar Thomas? ;-))
  - If it doesn't or is "determined" not useful for the community, then
it's probably dead in the water (maybe not if you can come back with a
counter?)
  - But if it is deemed interesting, the implementation you had internally
might be very different than what could/should be done for the mainline
      - So now what do you do? How do you work in the open with other
people interested in helping? Say you and I wanted to work on a similar
feature, how could we do so?
            (Except a sandbox without any open mailing list)

 It is that last question that I'm trying to do better with the TSAN
feature because JEP-331 was a bit difficult in that respect :)



> Currently, we have the JEP process, we have area-specific mailing
> lists (serviceability-dev, jfr-dev etc). We have the sandbox repo for
> test branches. And larger contributions are already spun off own into
> own feature-specific projects with own mailing lists and repos.
>

  - This is exactly right:
     - I want exactly the "larger contributions are already spun off own
into own feature-specific projects with own mailing lists and repos."
because the TSAN project is big enough and complex enough to require it in
my opinion
     - I also have the other features that I know I'd like to work under
this same umbrella before presenting it to the main serviceability



>
> Based on our earlier discussions, I thought the main thing missing was
> a way to reach the experts via a cross-area channel. Since posting
> contribution proposals to serviceability-dev has been considered
> "spamming" (which surprised me btw), svc-dev is out of the question.
> But would not just a new mailing list suffice for this? Why a whole
> new umbrella project, new reviewer roles etc?
>


Actually the main thing I'm trying to do right now is to have the right
avenue to work on the TSAN feature and others, in the open, where others
could come and contribute so that we can figure out if the feature(s) are
mainline-able, of interest, and what would be the implementation and
use-cases that would work for most of the users in an inter-company world.
To do that and to do that in the open, I find no real way than to have a
side project where we can have conversations about implementation details
without adding noise to the serviceability-dev mailing list.

Btw, I'm not sure that proposing a feature is considered "spamming"; I
could see you and I having regular technical conversations about how to
implement a given "big" feature and talking about various bugs we are
seeing with our current feature version would then become disruptive.

If we have only a mailing list, it's already a good start but the project
allows us to have our own repos and branches to better serve what I'm
trying to accomplish, which is to have a better avenue for implementing the
serviceability features that I currently think we should have in the
mainline but do it in the open. I am kind of using the Amber project (
https://openjdk.java.net/projects/amber/) as a template for this.


>
> Especially since this umbrella project now conflicts with the idea of
> feature-specific projects. Now we have three options for a larger
> contribution: We can either discuss it on the area-specific mailing
> lists and add it directly to mainline, optionally with a JEP; we can
> also create an own feature-specific project, incubate it an own
> repository with an own mailing list, then add it to mainline. Now,
> with Atlantis, we have the third option of adding it to the
> Atlantis-specific repository (?), as a new branch (?) and discuss it
> on the Atlantis-specific mailing list.
>

I think that is the kind of thing that is open to conversation of what
would/could be done. I believe in doing tiny baby steps and seeing what
happens. Currently, my only goal was to open a project, have a repo where I
could start working on the TSAN feature and have people follow it and be
able to test it :-). I'd like it to be a bit open so that if someone comes
with a feature that could be useful but might need a major rewrite, we have
a means to do that easier than what I saw with my JEP-331 work but in no
way am I trying to make a "third" route to getting something in.

It's the major reason why I've wanted to have some "expert" serviceability
engineers at the start of this project to help maintain cohesiveness,
coherent systems, and not make things more complicated. I'm actually trying
to just make it easier for me, JC, to work on the features that I know are
not trivial, do not have a single way of doing them, and would require
buy-in and testing from various groups before being able to be considered
acceptable. But to be able to be easy to test and feel comfortable with, I
feel we need to have an easy way to clone/test it and a project is a good
way (yes a sandbox too but you have to be a committer and, for example, a
few engineers I'd like working on the TSAN feature are not yet OpenJDK
committers...)


>
> Sorry,  I really just try to understand. I think you have valid pain
> points you try to solve, I am just not sure if a whole new umbrella
> project is the best solution. Or, I could just not understanding the
> proposal.
>

No worries, I understand entirely. To be honest, I rather get this out of
the way now and we start (if we do) on solid footing.  I rather have
everyone on the same page and agreeing that:
  1) This is not trying to make things more complex
  2) This is not trying to add noise
  3) This is not trying to fragment

I'm only trying to have a place do add features that I believe are useful
in a way that makes sense and seems more open, easier to understand what we
are doing, easier to get feedback in the open and be traceable, etc. So let
us have these conversations to all try to figure out if:
  1) This could be something we can do
  2) This could be supported by the community
  3) What are the basic ground rules (which is what we are talking about in
this conversation)

My best estimate is to say "do not assume Atlantis is another route for
serviceability", imagine it is a potential route for items that
serviceability experts say "yeah sounds interesting but depends on how it
looks from an implementation point of view, why not go experiment a bit and
come back with something that some key-people agree to". But we are a long
ways from that; right now imagine it really as "a TSAN project with
potential to maybe perhaps at some point host other TSAN style projects
that engineers are interested in working on".


> Thanks & Kind Regards, Thomas
>
> p.s. if I as an author go for the third option, via atlantis-dev: I
> would like to be reasonably sure that once my contribution meets
> approval and gets incubated, chances are reasonably high that it will
> be merged into mainline later. Because, if atlantis-dev greenlights a
> feature and helps incubating it but later it gets barred from
> mainline, that would make Atlantis moot. This means that atlantis-dev
> must be monitored by the resident area experts.
>

I actually don't agree with this because that is true in the mainline
anyway:
  - Serviceability says yes that looks good
  - You get to a point where the implementation looks good
  - You get initial reviews, your JEP/CSR/test plan is well on its way and
something comes to derail it; that is part of life, the process; the system.

So for me, if something gets a green light in the Atlantis project, it does
not mean anything until it's in the mainline. And yes chances might be high
that it would get it in but it does not mean it will automatically get in;
if you stop working on it, if an alternative shows up, etc. why does it
make Atlantis moot? If this consistently happens, yes maybe (I could
advocate no but let's not).

Bottom line of the Atlantis project is: I would love to have a means to
work on the TSAN feature and other Google features in the open with the
community; I would love to work on serviceability projects that are not
Google specific in the open and see it happen :-);  and I'd really rather
have a project to do so in order to have a reasonable means to get other
engineers in the community to have a place to talk about them, test them,
talk about implementations, etc. And I think a project is actually the way
to go. I am not trying to create noise, fragment, derail, or make issues
with the way things are done; I'm only trying to do something a bit
different for the next potential JEP I would contribute that might be
easier on all involved :)

So to that effect, if we need to put the right words to make this cemented
in stone and explicit enough so that no one is worried that the Atlantis
project is trying to rock the boat, let us do that. Perhaps following the
Amber project and requiring a JEP candidate for any work done in the
Atlantis project will help ease certain minds and get this going. I think
it's not required because I'm not worried about it, I think that if we see
that this is not working with that requirement, we add it as a rule amongst
ourselves (I believe in iteratively fixing things like this).

Perhaps a middle ground would be to at least require the work to be under a
JEP draft which is signed off by one of the reviewers of Atlantis ? This
forces the work to be published in JBS, "signed" off by key members though
not to the point of a candidate JEP, and therefore help ease our minds?

Thanks again for your comments and hopefully my answers are helping,
Jc



>
> On Tue, Nov 20, 2018 at 9:36 PM JC Beyler <jcbeyler at google.com> wrote:
> >
> > Thanks all for the questions and feedback. I've tried to summarize the
> questions/concerns here:
> >
> > - I want to be really clear that I don't have all the answers, I believe
> very strongly in trial and error
> >     - Part of this project's motivation is to have these conversations :)
> >     - Another part is that I think there is something to be gained by
> this project and it is exactly what Mario has said above: that there be an
> "umbrella for experimentation; This will probably already cure most of the
> risk of such fragmentation, however, if it's a written statement in the JEP
> goals it makes more explicit that we are interested in enhancing the
> current frameworks rather than creating alternative versions (unless it
> *really* makes sense to do so)."
> >         -> Exactly, I'm not interested in creating the next "shiny
> thing" that actually does the same thing as existing items; I am interested
> in looking at an issue/monitoring feature; considering all existing
> frameworks; seeing where it would fit best; figure out if it is feasible
> acceptable; then move forward".
> >
> > - (From Thomas mainly but also Mario) Is there a risk of fragmentation
> of knowledge or fragmenting serviceability?
> >     - I don't think so nor do I want to; I'm actually against
> fragmentation in almost all cases to be very honest and clear; that is why
> I've tried to work closely with the serviceability team and engineers to
> figure out what could/should be the scope of this project
> >     - Let us not forget that in the end, the work done in this project
> still needs a JEP to get into the mainline and that is our last
> barrier/hurdle that ensures consistency in OpenJDK and not fragmentation
> >     - To finish this subject (and re-iterate), if an item is to do
> "yet-another-monitoring/performance-analysis tool" that could be done by
> the various ones we have already available, I believe it has no place in
> this project; on the other hand, if it can be proved that X cannot be done
> at all in a product environment with current systems then we can/should
> start asking the questions:
> >          - Is there a solution close, how could it be extended, what do
> the current developers of that mechanism think, should we try to prototype
> something to see the extent of changes, should we push it to being a JEP
> from this project? It is for those cases that I see a path forward via this
> project.
> >     - I think that the balance between what could go directly to
> mainline or via this project has to be found, I don't have the right
> answer, I have the items I know I'd like to work on openly via this project
> and then see what comes from them; but I'm open to suggestions :-)
> >        - Which means I don't have the right answer of what should go
> directly to serviceability-dev or here when it's a "Hey I have this" or
> "Hey could we that" type of question; I think we will see what happens,
> right now, I'm personally going to concentrate on the internal features we
> have at Google to start the conversation and the project work and see where
> that takes me/us if that helps. I'm open to any explicit guideline that is
> deemed necessary/warranted or we can do what I would hope is possible too:
> by having myself, Serguei, and core members of serviceability, we will work
> out the balance together as we move forward.
> >
> > From Andrew mainly: I'm going to quote Andrew's question/comment here:
> > "JC, can you provide assurance that Google staff involved in pushing any
> code upstream will also be available to help perform that maintenance
> effort, in this case presumably as members of the serviceability team?"
> >
> > (Pre-amble: I believe it is a valid question for any relatively
> new-comer to be asked this question and hopefully my answer does help have
> confidence in my actual resolve)
> >
> > I actually had a longer answer but thought I was diluting the actual
> answer that you are looking/hoping for:
> >    - I believe that this is true for any JEP being submitted: ie if you
> push code upstream, you should be available for maintaining that code
> onwards
> >    - I sincerely only can talk about myself and I think we are in
> agreement, any work coming in the mainline is additional burden and that is
> what the JEP process is about: defining the work, the testing, the risks
> and the maintenance; I believe that as we move forward, we would only
> shepherd JEPs to the mainline in which we are confident that they will be
> maintained right and not be added burden to the rest of the team
> >    - Finally, I think that this is the biggest reason we need to have
> the right people as reviewers, experts, and committed people to ensure this
> very fact; we do not want contribute-and-run scenarios; this not only hurts
> the code-base, it hurts the community at large.
> >
> > So I don't answer your question directly because I want this not to be a
> Google-specific project, nor do I want it to be Google is pushing code via
> this project. You have my assurance that any JEP that I contribute to and
> push forward has my commitment to be helping the maintenance of AND that
> I'll make very clear to whoever is trying to do work in Atlantis and trying
> to then go to the mainline via a JEP that the JEP must have a engineer
> maintenance assurance that we, the community, feel comfortable with. So I
> believe I do however answer that I think your sentence is the right one
> across the board and we should all abide by:
> > "There is a requirement that anything new of a significant size or
> complexity that gets developed under a JEP and then integrated into the
> mainline repos will have someone available to maintain it."; I don't think
> this needs to be stated in the project description but I can definitely add
> it as I believe in this probably more strongly than most ;-)
> >
> > To finish this first long answer, and as someone recommended to me as
> well, I am hoping to get serviceability core members and members from the
> various monitoring tools (JVMTI, JMX, JMC and JFR) to be interested and be
> able to follow up with opinions about key items:
> >     - "I think this could be done via JVMTI, JMX, JMC and JFR; we should
> either go do it via their projects/branches"
> >     - I have no problem a monitoring/perf. analysis team saying let's
> prototype/experiment this idea here in Atlantis; then we can see if will
> merge it with the mainline
> >     - This participation can be ad-hoc or via being an initial
> reviewer/committer
> >
> > And that should solve the questions that Mario had about that side.
> >
> > Hopefully my answer helps understand that this project is an attempt to
> have a venue to create conversations about current internal systems or
> non-existant ones and that we can see what "sticks" and what doesn't; where
> it would "stick" potentially, what it would like and then how could be best
> push it forward with the support of the whole community.
> >
> > I'll finish the way I started, I don't believe in fragmentation, I don't
> believe in trying to build "yet-another-X" when the existing systems
> suffice,
> > Jc
> >
> > On Tue, Nov 20, 2018 at 10:41 AM <jesper.wilhelmsson at oracle.com> wrote:
> >>
> >> On 20 Nov 2018, at 17:21, Mario Torre <neugens at redhat.com> wrote:
> >>
> >> On Tue, 2018-11-20 at 15:38 +0000, Andrew Dinn wrote:
> >>
> >> On 20/11/2018 15:14, Mario Torre wrote:
> >>
> >> On Tue, 2018-11-20 at 15:38 +0100, Thomas Stüfe wrote:
> >>
> >> I'm worried about fragmentation though. Once we have this
> >> project,
> >> what is then the role of serviceability-dev? Just smallish
> >> everyday
> >> fixes? Also, the fact that VM developer feedback is hard to come
> >> by
> >> is
> >> mostly due to a lack of time on their part, and a new mailing
> >> list
> >> will not fix that.
> >>
> >>
> >> Generally projects have their own mailing lists for development
> >> until
> >> they are "finished", I would expect that for this project it would
> >> work
> >> the same, once it's integrated into mainline it becomes part of the
> >> serviceability code and the project mailing list is folded into
> >> serviceability-dev, until then it's somewhat an independent unit.
> >>
> >>
> >> Well, there is a requirement that anything new of a significant size
> >> orc
> >> complexity that gets developed under a JEP and then integrated into
> >> the
> >> mainline repos will have someone available to maintain it.
> >>
> >>
> >> Yes, indeed!
> >>
> >> JC, can you provide assurance that Google staff involved in pushing
> >> any
> >> code upstream will also be available to help perform that maintenance
> >> effort, in this case presumably as members of the serviceability
> >> team?
> >>
> >> n.b. Modulo the above detail I'm all for this proposal. I'd be very
> >> pleased if the project brings opportunities for Google either to
> >> upstream their existing work to everyone's benefit or, where for
> >> whatever reason that is not feasible, to help tailor existing
> >> upstream
> >> capabilities (such as JFR events) so they might encompass Google's
> >> downstream needs.
> >>
> >> It's very welcome to see Google so willing to contribute to OpenJDK.
> >>
> >>
> >> I agree!
> >>
> >> I actually wanted to ask that one written requirement in the JEP would
> >> be to extend Flight Recorder whenever it makes sense, even if this
> >> means going an extra step (in other words, the default option), so that
> >> we avoid fragmentation and the proliferation of a multitude of tools.
> >>
> >>
> >> +1
> >>
> >> Given that it is an explicit goal to extend and utilize existing
> frameworks rather than adding another one, I welcome this project.
> >> /Jesper
> >>
> >>
> >> I understand the project is meant to be some sort of umbrealla for
> >> experiementation? This will probably already cure most of the risk of
> >> such fragmentation, however, if it's a written statement in the JEP
> >> goals it makes more explicit that we are interested in enhancing the
> >> current frameworks rather than creating alternative versions (unless it
> >> *really* makes sense to do so).
> >>
> >> Cheers,
> >> Mario
> >> --
> >> Mario Torre
> >> Associate Manager, Software Engineering
> >> Red Hat GmbH <https://www.redhat.com>
> >> 9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898
> >>
> >>
> >
> >
> > --
> >
> > Thanks,
> > Jc
>


-- 

Thanks,
Jc