Call For Discussion: New Project: Atlantis

[Thomas Stüfe]

Hi JC, guys,

(JC: I know we had a similar discussion offlist some days ago, sorry
for raising the same points, still being unsure here).

Here is what I think I understand:

We want a "council" of sorts of experts in the general
monitoring/serviceability area. A one-stop point to discuss larger
planned contributions. They should advice on new developments,
redirect them, shepherd them, curtail them when necessary. Impose a
common strategy we all consent to (e.g. "JFR first").

No one should be able to add unnecessary features or reinvent the
wheel. We want to avoid fragmentation and keep the codebase simple and
consistent.

But also, no valuable contributions should find themselves barred. If
anything, we want to make the process of contributing smoother, and
that also includes the process of rejecting proposals.

And if a contribution is considered worthwhile, it would be incubated
separately from mainline, until it is ready to be upmerged.

..

Here is where I start getting confused: are we not doing most of this
already? I would have thought most of this is common sense and at
least in theory no-one would argue these points? (e.g. noone would
argue that reinventing the wheel is good :)

Currently, we have the JEP process, we have area-specific mailing
lists (serviceability-dev, jfr-dev etc). We have the sandbox repo for
test branches. And larger contributions are already spun off own into
own feature-specific projects with own mailing lists and repos.

Based on our earlier discussions, I thought the main thing missing was
a way to reach the experts via a cross-area channel. Since posting
contribution proposals to serviceability-dev has been considered
"spamming" (which surprised me btw), svc-dev is out of the question.
But would not just a new mailing list suffice for this? Why a whole
new umbrella project, new reviewer roles etc?

Especially since this umbrella project now conflicts with the idea of
feature-specific projects. Now we have three options for a larger
contribution: We can either discuss it on the area-specific mailing
lists and add it directly to mainline, optionally with a JEP; we can
also create an own feature-specific project, incubate it an own
repository with an own mailing list, then add it to mainline. Now,
with Atlantis, we have the third option of adding it to the
Atlantis-specific repository (?), as a new branch (?) and discuss it
on the Atlantis-specific mailing list.

Sorry,  I really just try to understand. I think you have valid pain
points you try to solve, I am just not sure if a whole new umbrella
project is the best solution. Or, I could just not understanding the
proposal.

Thanks & Kind Regards, Thomas

p.s. if I as an author go for the third option, via atlantis-dev: I
would like to be reasonably sure that once my contribution meets
approval and gets incubated, chances are reasonably high that it will
be merged into mainline later. Because, if atlantis-dev greenlights a
feature and helps incubating it but later it gets barred from
mainline, that would make Atlantis moot. This means that atlantis-dev
must be monitored by the resident area experts.


On Tue, Nov 20, 2018 at 9:36 PM JC Beyler <jcbeyler at google.com> wrote:
>
> Thanks all for the questions and feedback. I've tried to summarize the questions/concerns here:
>
> - I want to be really clear that I don't have all the answers, I believe very strongly in trial and error
>     - Part of this project's motivation is to have these conversations :)
>     - Another part is that I think there is something to be gained by this project and it is exactly what Mario has said above: that there be an "umbrella for experimentation; This will probably already cure most of the risk of such fragmentation, however, if it's a written statement in the JEP goals it makes more explicit that we are interested in enhancing the current frameworks rather than creating alternative versions (unless it *really* makes sense to do so)."
>         -> Exactly, I'm not interested in creating the next "shiny thing" that actually does the same thing as existing items; I am interested in looking at an issue/monitoring feature; considering all existing frameworks; seeing where it would fit best; figure out if it is feasible acceptable; then move forward".
>
> - (From Thomas mainly but also Mario) Is there a risk of fragmentation of knowledge or fragmenting serviceability?
>     - I don't think so nor do I want to; I'm actually against fragmentation in almost all cases to be very honest and clear; that is why I've tried to work closely with the serviceability team and engineers to figure out what could/should be the scope of this project
>     - Let us not forget that in the end, the work done in this project still needs a JEP to get into the mainline and that is our last barrier/hurdle that ensures consistency in OpenJDK and not fragmentation
>     - To finish this subject (and re-iterate), if an item is to do "yet-another-monitoring/performance-analysis tool" that could be done by the various ones we have already available, I believe it has no place in this project; on the other hand, if it can be proved that X cannot be done at all in a product environment with current systems then we can/should start asking the questions:
>          - Is there a solution close, how could it be extended, what do the current developers of that mechanism think, should we try to prototype something to see the extent of changes, should we push it to being a JEP from this project? It is for those cases that I see a path forward via this project.
>     - I think that the balance between what could go directly to mainline or via this project has to be found, I don't have the right answer, I have the items I know I'd like to work on openly via this project and then see what comes from them; but I'm open to suggestions :-)
>        - Which means I don't have the right answer of what should go directly to serviceability-dev or here when it's a "Hey I have this" or "Hey could we that" type of question; I think we will see what happens, right now, I'm personally going to concentrate on the internal features we have at Google to start the conversation and the project work and see where that takes me/us if that helps. I'm open to any explicit guideline that is deemed necessary/warranted or we can do what I would hope is possible too: by having myself, Serguei, and core members of serviceability, we will work out the balance together as we move forward.
>
> From Andrew mainly: I'm going to quote Andrew's question/comment here:
> "JC, can you provide assurance that Google staff involved in pushing any code upstream will also be available to help perform that maintenance effort, in this case presumably as members of the serviceability team?"
>
> (Pre-amble: I believe it is a valid question for any relatively new-comer to be asked this question and hopefully my answer does help have confidence in my actual resolve)
>
> I actually had a longer answer but thought I was diluting the actual answer that you are looking/hoping for:
>    - I believe that this is true for any JEP being submitted: ie if you push code upstream, you should be available for maintaining that code onwards
>    - I sincerely only can talk about myself and I think we are in agreement, any work coming in the mainline is additional burden and that is what the JEP process is about: defining the work, the testing, the risks and the maintenance; I believe that as we move forward, we would only shepherd JEPs to the mainline in which we are confident that they will be maintained right and not be added burden to the rest of the team
>    - Finally, I think that this is the biggest reason we need to have the right people as reviewers, experts, and committed people to ensure this very fact; we do not want contribute-and-run scenarios; this not only hurts the code-base, it hurts the community at large.
>
> So I don't answer your question directly because I want this not to be a Google-specific project, nor do I want it to be Google is pushing code via this project. You have my assurance that any JEP that I contribute to and push forward has my commitment to be helping the maintenance of AND that I'll make very clear to whoever is trying to do work in Atlantis and trying to then go to the mainline via a JEP that the JEP must have a engineer maintenance assurance that we, the community, feel comfortable with. So I believe I do however answer that I think your sentence is the right one across the board and we should all abide by:
> "There is a requirement that anything new of a significant size or complexity that gets developed under a JEP and then integrated into the mainline repos will have someone available to maintain it."; I don't think this needs to be stated in the project description but I can definitely add it as I believe in this probably more strongly than most ;-)
>
> To finish this first long answer, and as someone recommended to me as well, I am hoping to get serviceability core members and members from the various monitoring tools (JVMTI, JMX, JMC and JFR) to be interested and be able to follow up with opinions about key items:
>     - "I think this could be done via JVMTI, JMX, JMC and JFR; we should either go do it via their projects/branches"
>     - I have no problem a monitoring/perf. analysis team saying let's prototype/experiment this idea here in Atlantis; then we can see if will merge it with the mainline
>     - This participation can be ad-hoc or via being an initial reviewer/committer
>
> And that should solve the questions that Mario had about that side.
>
> Hopefully my answer helps understand that this project is an attempt to have a venue to create conversations about current internal systems or non-existant ones and that we can see what "sticks" and what doesn't; where it would "stick" potentially, what it would like and then how could be best push it forward with the support of the whole community.
>
> I'll finish the way I started, I don't believe in fragmentation, I don't believe in trying to build "yet-another-X" when the existing systems suffice,
> Jc
>
> On Tue, Nov 20, 2018 at 10:41 AM <jesper.wilhelmsson at oracle.com> wrote:
>>
>> On 20 Nov 2018, at 17:21, Mario Torre <neugens at redhat.com> wrote:
>>
>> On Tue, 2018-11-20 at 15:38 +0000, Andrew Dinn wrote:
>>
>> On 20/11/2018 15:14, Mario Torre wrote:
>>
>> On Tue, 2018-11-20 at 15:38 +0100, Thomas Stüfe wrote:
>>
>> I'm worried about fragmentation though. Once we have this
>> project,
>> what is then the role of serviceability-dev? Just smallish
>> everyday
>> fixes? Also, the fact that VM developer feedback is hard to come
>> by
>> is
>> mostly due to a lack of time on their part, and a new mailing
>> list
>> will not fix that.
>>
>>
>> Generally projects have their own mailing lists for development
>> until
>> they are "finished", I would expect that for this project it would
>> work
>> the same, once it's integrated into mainline it becomes part of the
>> serviceability code and the project mailing list is folded into
>> serviceability-dev, until then it's somewhat an independent unit.
>>
>>
>> Well, there is a requirement that anything new of a significant size
>> orc
>> complexity that gets developed under a JEP and then integrated into
>> the
>> mainline repos will have someone available to maintain it.
>>
>>
>> Yes, indeed!
>>
>> JC, can you provide assurance that Google staff involved in pushing
>> any
>> code upstream will also be available to help perform that maintenance
>> effort, in this case presumably as members of the serviceability
>> team?
>>
>> n.b. Modulo the above detail I'm all for this proposal. I'd be very
>> pleased if the project brings opportunities for Google either to
>> upstream their existing work to everyone's benefit or, where for
>> whatever reason that is not feasible, to help tailor existing
>> upstream
>> capabilities (such as JFR events) so they might encompass Google's
>> downstream needs.
>>
>> It's very welcome to see Google so willing to contribute to OpenJDK.
>>
>>
>> I agree!
>>
>> I actually wanted to ask that one written requirement in the JEP would
>> be to extend Flight Recorder whenever it makes sense, even if this
>> means going an extra step (in other words, the default option), so that
>> we avoid fragmentation and the proliferation of a multitude of tools.
>>
>>
>> +1
>>
>> Given that it is an explicit goal to extend and utilize existing frameworks rather than adding another one, I welcome this project.
>> /Jesper
>>
>>
>> I understand the project is meant to be some sort of umbrealla for
>> experiementation? This will probably already cure most of the risk of
>> such fragmentation, however, if it's a written statement in the JEP
>> goals it makes more explicit that we are interested in enhancing the
>> current frameworks rather than creating alternative versions (unless it
>> *really* makes sense to do so).
>>
>> Cheers,
>> Mario
>> --
>> Mario Torre
>> Associate Manager, Software Engineering
>> Red Hat GmbH <https://www.redhat.com>
>> 9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898
>>
>>
>
>
> --
>
> Thanks,
> Jc