JShell: source in langtools vs JDK?
Jonathan Gibbons
jonathan.gibbons at oracle.com
Thu Sep 17 15:59:26 UTC 2015
Hmmm.
Thanks for the write up.
Looking ahead, the environment will change when Jigsaw gets integrated
into JDK 9. If nothing else, the -Xbootclasspath/p: option goes away as
such, and will be rejected. The proposed replacement is -Xoverride, and
in the jigsaw world, we are looking at how to incorporate that into our
langtools build.xml world. At that time, folk will have to update
personal scripts etc as well.
With regard to testing, I'm not sure how this can be used to our
advantage, but these days we're tagging tests with the new @modules
line. There are two variants of this tag.
@modules M
@modules M/P
where M and P are a module name and package name respectively.
Both forms express a dependence on the specified module, such that if it
is not present in the testJDK, the test will not be selected for
execution. (The second option additionally arranges for the specified
package to be exported to the unnamed module in which the test is
loaded.) All of which is to say that if a module (like jdk.jdi) is not
available, any tests (like jshell tests) which may rely on it will
automatically not be run.
So, going forward, Jan has been looking at how to use -Xoverride in
langtools build.xml. We should continue that work, and then maybe
create a "play" forest contains both the kulla code and jigsaw code, to
investigate these issues in that context. And related to that, we
should be careful about investing too much effort into propagating
bootstrap options like -Xbootclasspath into any code, without having a
sense of what that code should be in the long term.
I also feel that long term, the problem gets easier when Jigsaw is
available as standard. In that world, it should be possible to take a
baseline build of JDK (with JDI and jline in it) and to build, run and
test just the module containing the jshell code, and use all other
modules from the baseline JDK.
In the short term, would it help those folk actively working on the
kulla code base to have recent full builds of the kulla forest available
from somewhere (i.e, with jline and jdi present), so that we can
minimize any short term inconvenience until kulla is integrated?
-- Jon
On 09/17/2015 02:43 AM, Maurizio Cimadamore wrote:
> Here's a more precise description of what you'd run into if you would
> like to run jshell on top of an existing JDK; assumption:
>
> <JDK9_HOME> : points to a valid JDK 9 binary
> <KULLA_JDK_BUILD> : points to a valid Kulla exploded build
>
> #1 run jshell - just add jshell (and langtools) to bootstrap classes:
>
> $ <JDK9_HOME>/bin/java
> -Xbootclasspath/p:<KULLA_JDK_BUILD>/jdk/modules/*jdk.jshell*:<KULLA_JDK_BUILD>/jdk/modules/*jdk.compiler*:<KULLA_JDK_BUILD>/jdk/modules/*java.compiler*jdk.internal.jshell.tool.JShellTool
> Exception in thread "main" java.lang.NoClassDefFoundError:
> jdk/internal/jline/TerminalFactory$Flavor
> at
> jdk.internal.jshell.tool.ConsoleIOContext.<init>(ConsoleIOContext.java:70)
> at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:242)
> at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233)
>
>
> #2 add jline
>
> $ <JDK9_HOME>/bin/java
> -Xbootclasspath/p:<KULLA_JDK_BUILD>/jdk/modules/jdk.jshell:<KULLA_JDK_BUILD>/jdk/modules/jdk.compiler:<KULLA_JDK_BUILD>/jdk/modules/java.compiler:<KULLA_JDK_BUILD>/jdk/modules/*jdk.internal.le*jdk.internal.jshell.tool.JShellTool
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/sun/jdi/Bootstrap
> at jdk.jshell.JDIConnection.findConnector(JDIConnection.java:77)
> at jdk.jshell.JDIConnection.<init>(JDIConnection.java:312)
> at jdk.jshell.JDIEnv.init(JDIEnv.java:45)
> at jdk.jshell.ExecutionControl.jdiGo(ExecutionControl.java:251)
> at jdk.jshell.ExecutionControl.launch(ExecutionControl.java:66)
> at jdk.jshell.JShell.executionControl(JShell.java:601)
> at jdk.jshell.Eval.load(Eval.java:729)
> at jdk.jshell.Eval.declare(Eval.java:591)
> at jdk.jshell.Eval.declare(Eval.java:494)
> at jdk.jshell.Eval.processMethod(Eval.java:393)
> at jdk.jshell.Eval.eval(Eval.java:137)
> at jdk.jshell.JShell.eval(JShell.java:349)
> at
> jdk.internal.jshell.tool.JShellTool.processCompleteSource(JShellTool.java:1360)
> at
> jdk.internal.jshell.tool.JShellTool.processSource(JShellTool.java:1349)
> at
> jdk.internal.jshell.tool.JShellTool.processSourceCatchingReset(JShellTool.java:479)
> at jdk.internal.jshell.tool.JShellTool.run(JShellTool.java:465)
> at jdk.internal.jshell.tool.JShellTool.resetState(JShellTool.java:399)
> at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:248)
> at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:243)
> at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233)
>
> #3 add JDI
>
> $ <JDK9_HOME>/bin/java
> -Xbootclasspath/p:<KULLA_JDK_BUILD>/jdk/modules/jdk.jshell:<KULLA_JDK_BUILD>/jdk/modules/jdk.compiler:<KULLA_JDK_BUILD>/jdk/modules/java.compiler:<KULLA_JDK_BUILD>/jdk/modules/jdk.internal.le:<KULLA_JDK_BUILD>/jdk/modules/*jdk.jdi*jdk.internal.jshell.tool.JShellTool
>
> Error: Could not find or load main class
> jdk.internal.jshell.remote.RemoteAgent
> | State engine terminated. See /history
>
> #4 Give up :-)
>
>
> My brief understanding of what goes wrong:
>
> #1 fails because you need jline. But wait - jline is already in the
> JDK I used to run jshell - so what? There seems to be an issue with
> jimage: if some bootstrap mentions 'jdk.internal', all contents of
> 'jdk.internal' coming from the jimage are nuked out - in other words,
> adding jshell to the bootclasspath prevents jline to be discovered.
> Uuugh. Not sure if this will go away with new modularized options.
>
> #2 fails because JDI is not part of rt.jar (or its new equivalent). So
> that must be manually added. Ugh. Means we need a full build available
> somewhere. Ok.
>
> #3 fails because, when jshell is forking the agent VM, it propagates
> the classpath options but not the bootclasspath options. This means
> that the new agent vm won't see the jshell package. Again, not sure if
> this is fixable or not.
>
> Because of the combination of #1, #2 and #3 it is impossible to
> (reliably) run jshell on top of an existing (and recent) JDK 9 binary.
> Note that the failures above are made more explicit by the fact that
> jshell is not in the binary snapshot used for launching it - if we
> were using a jshell-enabled JDK the failure would be much more subtle
> - as jshell would start, but the tool and the agent would be seeing
> two different versions of the jshell module - with a lot of potential
> for troubles.
>
> Running tests is basically more of the same.
>
> If we can fix #1, #2 and #3 (or show a path that leads us there), I
> agree that the decision of where to put jshell is largely irrelevant.
> But if any of these issues is there to stay, then I don't think having
> jshell in langtools is really buying anything: jshell will end up
> being a corner case of langtools that you need to build/run/test
> specially - meaning most developers won't even bother with it.
>
> Maurizio
>
>
> On 17/09/15 09:17, Maurizio Cimadamore wrote:
>>
>>
>> On 17/09/15 02:56, Jonathan Gibbons wrote:
>>> I guess I don't understand why testing won't work as expected,
>>> provided you have a sufficiently recent snapshot of JDK to use as a
>>> baseline. In this case, the requirement is that the JDK baseline
>>> has jline and whatever other recent API has been added to JDK.
>> There are many factors that contribute to this - the main one being
>> JShell too not passing bootstrapping setting on the remote agent VM,
>> plus some weird bug I encountered where if I set bootclasspath that
>> mentions some jdk.internal classes (but which doesn't have JLine -
>> because it comes from langtools), _every_ class from that package
>> will not be fetched from the jimage file - meaning that I'll be left
>> w/o jline. In other words, bootstrapping doesn't work - and I have
>> reasons to believe that bootstrapping with the new module options
>> (when they will become available - i.e. moduleOverride) won't work too.
>>
>> Maurizio
>>>
>>> If kulla is routinely adding and modifying API in the jdk repo, then
>>> I agree that would tend to suggest that the source bits should be in
>>> the jdk repo as well. But I would have thought that the kulla
>>> additions to the jdk repo will stabilize very quickly, and so after
>>> one or two builds and promotions, if JShell ends up in langtools,
>>> then it will be easy enough to run all the tests using a recent
>>> build of JDK.
>>>
>>> I agree that we should not expect langtools developers to have to
>>> use "make test" and I agree that we should not expect langtools
>>> developers to exclude the JShell tests. That implies we should look
>>> at and understand why the JShell tests might fail if they are
>>> co-located in the langtools directory. If it is just because jline
>>> is not yet standard in JDK builds, that's a temporary state of
>>> affairs that we will get past soon enough. If there's something
>>> else going on, I guess I'd like to better understand what that might
>>> be, so that we can see if we can fix the issue.
>>>
>>> -- Jon
>>>
>>> On 09/16/2015 05:38 PM, Maurizio Cimadamore wrote:
>>>> While I agree with many points you raise - I feel this discussion
>>>> is very abstract; please try to run all langtools tests in the
>>>> kulla langtools repo using your standard scripts - does everything
>>>> still work for you? I found that I could not run tests using my
>>>> usual scripts. The only way to run jshell tests reliably is through
>>>> make. Now, if we were in a world where 100% langtools developers
>>>> ran tests that way - I would agree with you; but the reality is
>>>> somewhat different, and many developers are used to run langtools
>>>> tests using some JDK N snapshot and bootstrapping the required
>>>> classes. That technique WILL cease to work here. I don't think we
>>>> can say to those developer - just don't run kulla tests and be
>>>> happy (what if some of their changes accidentally broke something
>>>> in kulla?). Are you saying that those developers should just stop
>>>> doing what they are doing and start running langtools tests as part
>>>> of a full JDK build and test cycle? That's a fine answer - I guess
>>>> I'm just trying to guess where you are coming from.
>>>>
>>>> Maurizio
>>>>
>>>> On 17/09/15 01:04, Jonathan Gibbons wrote:
>>>>> I didn't say I think JShell should be in langtools, and I thought
>>>>> I was being careful to be very explicit about that. I was just
>>>>> trying to make sure we were making the right choice for the right
>>>>> reasons.
>>>>>
>>>>> The fact that it depends on JDI and jline is IMO a red herring. If
>>>>> they were both already in JDK 8 (i.e. the boot JDK) would we be
>>>>> having this discussion?
>>>>>
>>>>> To me, the bottom line is that I don't think it matters where the
>>>>> code lives, and as a result I personally don't care and have no
>>>>> opinion where the code should live. I /do/ think that whereever we
>>>>> choose to put it, we can make everything work well from a
>>>>> developer standpoint, without inconveniencing any existing
>>>>> developer usage, although I concede that we might choose not to
>>>>> integrate it with the langtools make/build.xml infrastructure
>>>>> and/or the existing NetBeans project. And from a somewhat selfish
>>>>> standpoint, I think we have somewhat more latitude to set things
>>>>> up the way we like it in the langtools repo than we would in the
>>>>> jdk repo.
>>>>>
>>>>> I agree that JShell is higher up in the layers of abstraction than
>>>>> javac, but I also don't think that the langtools repo is defined
>>>>> to be "core tools to be able to work with the JDK". The provenance
>>>>> of the repo (as indicated by its name) is that it was code that
>>>>> was managed by the old Sun Language Tools Group, from way back
>>>>> when, meaning that it was more of a "team" repository than a
>>>>> repository defined by a layer of abstraction.
>>>>>
>>>>> So, for me, I think this comes down to, "where does the Kulla dev
>>>>> team think it would be most convenient for them to keep and work
>>>>> on the code?". Once we can answer that, we can move on to, "what
>>>>> is the most convenient way to satisfy all the requirements of all
>>>>> the interested parties?"
>>>>>
>>>>> -- Jon
>>>>>
>>>>>
>>>>> On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote:
>>>>>> Jon, let's reverse the question - why do you think it *should*
>>>>>> live in langtools? Honestly it feels very different from all
>>>>>> other tools that live there; there's dependencies on JDI, jline -
>>>>>> so to me it feels closer to other tools (i.e. jconsole) which
>>>>>> already are in the jdk repo. I guess my argument is - if
>>>>>> langtools defines the core tools to be able to work with the JDK,
>>>>>> jshell feels like one ore two layer of abstractions on top of that.
>>>>>>
>>>>>> Maurizio
>>>>>>
>>>>>> On 16/09/15 23:31, Jonathan Gibbons wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 09/16/2015 03:06 PM, Robert Field wrote:
>>>>>>>> As part of the JShell code review, Maurizio made the case that
>>>>>>>> JShell should be in the JDK repo rather than langtools repo
>>>>>>>> (where it is in the Kulla workspace). We would like additional
>>>>>>>> feedback on this before proceeding --
>>>>>>>>
>>>>>>>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore
>>>>>>>>> <maurizio.cimadamore at oracle.com> wrote:
>>>>>>>>>
>>>>>>>>> One general high-level comment, which I also pointed out
>>>>>>>>> elsewhere, is that I'm not sure jshell really belongs in
>>>>>>>>> langtools; while it's semantically (obviously) related to
>>>>>>>>> langtools - it is a rather different beasts w.r.t. all other
>>>>>>>>> tools in langtools-land; the fact that it depends on the JDK
>>>>>>>>> (for jline, and for JDI in general) makes it very hard to run
>>>>>>>>> on top of a random JDK and then bootstrapping classes - which
>>>>>>>>> is a technique widely used to be able to run langtools tools
>>>>>>>>> w/o having to do a full build cycle. More specifically,
>>>>>>>>> talking about IDE integration, I don't see how
>>>>>>>>> IntelliJ/Netbeans langtools projects (and the langtools
>>>>>>>>> internal developer ant build) could be updated to be able to
>>>>>>>>> run/debug jshell w/o a full build.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> In the Big Picture view of the universe, folk would like to
>>>>>>> restructure the repos in an OPenJDK forest, at which point the
>>>>>>> distinction between the langtools and jdk repos will likely
>>>>>>> disappear, or at least become very (very) blurred. While that
>>>>>>> change is not imminent, IMO opinion, it significantly reduces
>>>>>>> the impact of the reasons why JShell should not be in langtools.
>>>>>>>
>>>>>>> I also don't entirely agree with the IDE reasons either. If you
>>>>>>> want to work on JShell in an IDE, you're going to have to do
>>>>>>> what it takes to get rid of the red squiggly lines anyway,
>>>>>>> whatever repo the code is in. To keep JShell out of Langtools
>>>>>>> because it doesn't fit our current methodologies is a case of
>>>>>>> the tail wagging the dog.
>>>>>>>
>>>>>>> Also, I note that JDK 9 now builds the product module by module.
>>>>>>> In times past, we built the world repo by repo, and that would
>>>>>>> indeed have been good reason to keep JShell out of langtools,
>>>>>>> because of the dependencies of JShell on JDK 9 API, like jline.
>>>>>>> But now, the build will build "interim javac", and can then
>>>>>>> build JDK modules like java.base and whatever module jline ends
>>>>>>> up in, and can then build the module containing JShell.
>>>>>>>
>>>>>>> As to how to organize IDE projects, in the worst case, we keep
>>>>>>> the langtools IDE setup as it is now (no JShell) and have a
>>>>>>> separate project for JShell itself. That is tantamount to what
>>>>>>> we would do if we push JShell into the jdk repo.
>>>>>>>
>>>>>>> So, I'm saying all that while trying to be agnostic on which
>>>>>>> repo it should live in. I don't really agree with the stated
>>>>>>> reasons why it should not live in langtools, which means that we
>>>>>>> can look for other reasons to choose one repo or another.
>>>>>>>
>>>>>>> -- Jon
>>>>>>
>>>>>
>>>>
>>>
>>
>
More information about the kulla-dev
mailing list