From robert.field at oracle.com Tue Sep 1 23:01:09 2015 From: robert.field at oracle.com (Robert Field) Date: Tue, 1 Sep 2015 16:01:09 -0700 Subject: Implementation notes: Compilation and analysis within JShell Message-ID: <601059D1-5CF5-482C-B490-C7D37C9DB673@oracle.com> Compilation in JShell goes through three stages: parsing, wrapping & analysis, and code generation. Parsing is of one snippet (declaration, statement, expression, ?) of Java source. The most important result of parsing is determination of the kind of snippet. In wrapping & analysis, each kind of snippet is handled differently. The snippet is wrapped to make a valid Java source file. At a minimum, each snippet is wrapped in a class. Note that even class definitions are wrapped in a class, this is critical to replacing the snippet if overwritten. Each wrap starts with a package definition (always ?REPL?) followed by a set of imports that defines the complete context, including explicit imports of all previous active snippets. Wrapping uses information from the parse including tree structure, source positions and names. Once wrapped, the snippet may be analyzed and possibly rewrapped based on that analysis. Once the final wrap in determined, the wrapped source is compiled to an in-memory class file. That class file is shipped to the backend over a socket, loaded or redefined, and, if executable, executed. Classes and methods are just wrapped in a generated class. Statements are wrapped in a doit method within the class. Variable declarations are separated into their declaration part, which becomes a field, and their initializer which becomes an assignment in a doit method. Expressions become synthetic temporary variables in the same general form as variables. Imports follow a completely different mechanism, after being verified, they are simply recorded for use with subsequent snippets. Parsing uses the compiler API?s parse functionality, with one method in the parser overridden to allow stand-alone snippets as input. Analysis and compilation use the compiler API unmodified. JShell has its own FileManager so that input comes from Strings and output class files go to byte arrays. JShell?s interface to compiler API tasks is though the TaskManager, with separate subclasses of BaseTask for the cases above. Wrapping is accomplished with nested wraps that track location translation. Wrap is the abstract base class and holds its subclasses and other functionality. Nesting is with CompoundWrap which holds a sequence of Strings and Wraps. RangeWrap is the base wrap of user snippet sections. Information about the snippet, including its Wrap, found during parsing and analysis, is stored in subclasses of Snippet. Eval orchestrates these components. The JShell class is the entry point for JShell which delegates the whole eval() process to Eval. Please let me know if there are questions or areas I did not cover. Enjoy, Robert From sundararajan.athijegannathan at oracle.com Wed Sep 2 02:59:56 2015 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Wed, 2 Sep 2015 08:29:56 +0530 Subject: Implementation notes: Compilation and analysis within JShell In-Reply-To: <601059D1-5CF5-482C-B490-C7D37C9DB673@oracle.com> References: <601059D1-5CF5-482C-B490-C7D37C9DB673@oracle.com> Message-ID: <55E6662C.5030202@oracle.com> Nice! Thanks for the summary. Will you please put this notes in the repo itself? yes, it is difficult to keep it up to date with code -- but it is easy to miss this email in the kulla-dev archive. Thanks, -Sundar On 9/2/2015 4:31 AM, Robert Field wrote: > Compilation in JShell goes through three stages: parsing, wrapping & analysis, and code generation. Parsing is of one snippet (declaration, statement, expression, ?) of Java source. The most important result of parsing is determination of the kind of snippet. In wrapping & analysis, each kind of snippet is handled differently. The snippet is wrapped to make a valid Java source file. At a minimum, each snippet is wrapped in a class. Note that even class definitions are wrapped in a class, this is critical to replacing the snippet if overwritten. Each wrap starts with a package definition (always ?REPL?) followed by a set of imports that defines the complete context, including explicit imports of all previous active snippets. Wrapping uses information from the parse including tree structure, source positions and names. Once wrapped, the snippet may be analyzed and possibly rewrapped based on that analysis. Once the final wrap in determined, the wrapped source is compiled to an in-memory class file. That class file is shipped to the backend over a socket, loaded or redefined, and, if executable, executed. > > Classes and methods are just wrapped in a generated class. Statements are wrapped in a doit method within the class. Variable declarations are separated into their declaration part, which becomes a field, and their initializer which becomes an assignment in a doit method. Expressions become synthetic temporary variables in the same general form as variables. Imports follow a completely different mechanism, after being verified, they are simply recorded for use with subsequent snippets. > > Parsing uses the compiler API?s parse functionality, with one method in the parser overridden to allow stand-alone snippets as input. Analysis and compilation use the compiler API unmodified. JShell has its own FileManager so that input comes from Strings and output class files go to byte arrays. > > JShell?s interface to compiler API tasks is though the TaskManager, with separate subclasses of BaseTask for the cases above. Wrapping is accomplished with nested wraps that track location translation. Wrap is the abstract base class and holds its subclasses and other functionality. Nesting is with CompoundWrap which holds a sequence of Strings and Wraps. RangeWrap is the base wrap of user snippet sections. Information about the snippet, including its Wrap, found during parsing and analysis, is stored in subclasses of Snippet. Eval orchestrates these components. The JShell class is the entry point for JShell which delegates the whole eval() process to Eval. > > Please let me know if there are questions or areas I did not cover. > > Enjoy, > Robert > From robert.field at oracle.com Wed Sep 2 06:32:46 2015 From: robert.field at oracle.com (Robert Field) Date: Tue, 1 Sep 2015 23:32:46 -0700 Subject: Implementation notes: Compilation and analysis within JShell In-Reply-To: <55E6662C.5030202@oracle.com> References: <601059D1-5CF5-482C-B490-C7D37C9DB673@oracle.com> <55E6662C.5030202@oracle.com> Message-ID: <6EE58220-C5FA-4AA9-AE6B-B0E1C3BDCB25@oracle.com> > On Sep 1, 2015, at 7:59 PM, Sundararajan Athijegannathan wrote: > > Nice! Thanks for the summary. Will you please put this notes in the repo itself? yes, it is difficult to keep it up to date with code -- but it is easy to miss this email in the kulla-dev archive. True. Is there a standard place/form for such a document, I haven?t run across this yet. -Robert > > Thanks, > -Sundar > > On 9/2/2015 4:31 AM, Robert Field wrote: >> Compilation in JShell goes through three stages: parsing, wrapping & analysis, and code generation. Parsing is of one snippet (declaration, statement, expression, ?) of Java source. The most important result of parsing is determination of the kind of snippet. In wrapping & analysis, each kind of snippet is handled differently. The snippet is wrapped to make a valid Java source file. At a minimum, each snippet is wrapped in a class. Note that even class definitions are wrapped in a class, this is critical to replacing the snippet if overwritten. Each wrap starts with a package definition (always ?REPL?) followed by a set of imports that defines the complete context, including explicit imports of all previous active snippets. Wrapping uses information from the parse including tree structure, source positions and names. Once wrapped, the snippet may be analyzed and possibly rewrapped based on that analysis. Once the final wrap in determined, the wrapped source is compiled to an in-memory class file. That class file is shipped to the backend over a socket, loaded or redefined, and, if executable, executed. >> >> Classes and methods are just wrapped in a generated class. Statements are wrapped in a doit method within the class. Variable declarations are separated into their declaration part, which becomes a field, and their initializer which becomes an assignment in a doit method. Expressions become synthetic temporary variables in the same general form as variables. Imports follow a completely different mechanism, after being verified, they are simply recorded for use with subsequent snippets. >> >> Parsing uses the compiler API?s parse functionality, with one method in the parser overridden to allow stand-alone snippets as input. Analysis and compilation use the compiler API unmodified. JShell has its own FileManager so that input comes from Strings and output class files go to byte arrays. >> >> JShell?s interface to compiler API tasks is though the TaskManager, with separate subclasses of BaseTask for the cases above. Wrapping is accomplished with nested wraps that track location translation. Wrap is the abstract base class and holds its subclasses and other functionality. Nesting is with CompoundWrap which holds a sequence of Strings and Wraps. RangeWrap is the base wrap of user snippet sections. Information about the snippet, including its Wrap, found during parsing and analysis, is stored in subclasses of Snippet. Eval orchestrates these components. The JShell class is the entry point for JShell which delegates the whole eval() process to Eval. >> >> Please let me know if there are questions or areas I did not cover. >> >> Enjoy, >> Robert >> > > From sundararajan.athijegannathan at oracle.com Wed Sep 2 08:05:31 2015 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Wed, 2 Sep 2015 13:35:31 +0530 Subject: Implementation notes: Compilation and analysis within JShell In-Reply-To: <6EE58220-C5FA-4AA9-AE6B-B0E1C3BDCB25@oracle.com> References: <601059D1-5CF5-482C-B490-C7D37C9DB673@oracle.com> <55E6662C.5030202@oracle.com> <6EE58220-C5FA-4AA9-AE6B-B0E1C3BDCB25@oracle.com> Message-ID: <55E6ADCB.8060408@oracle.com> How about openjdk wiki "kulla" section? https://wiki.openjdk.java.net/display/kulla/Main -Sundar On 9/2/2015 12:02 PM, Robert Field wrote: >> On Sep 1, 2015, at 7:59 PM, Sundararajan Athijegannathan wrote: >> >> Nice! Thanks for the summary. Will you please put this notes in the repo itself? yes, it is difficult to keep it up to date with code -- but it is easy to miss this email in the kulla-dev archive. > True. > Is there a standard place/form for such a document, I haven?t run across this yet. > > -Robert > >> Thanks, >> -Sundar >> >> On 9/2/2015 4:31 AM, Robert Field wrote: >>> Compilation in JShell goes through three stages: parsing, wrapping & analysis, and code generation. Parsing is of one snippet (declaration, statement, expression, ?) of Java source. The most important result of parsing is determination of the kind of snippet. In wrapping & analysis, each kind of snippet is handled differently. The snippet is wrapped to make a valid Java source file. At a minimum, each snippet is wrapped in a class. Note that even class definitions are wrapped in a class, this is critical to replacing the snippet if overwritten. Each wrap starts with a package definition (always ?REPL?) followed by a set of imports that defines the complete context, including explicit imports of all previous active snippets. Wrapping uses information from the parse including tree structure, source positions and names. Once wrapped, the snippet may be analyzed and possibly rewrapped based on that analysis. Once the final wrap in determined, the wrapped source is compiled to an in-memory class file. That class file is shipped to the backend over a socket, loaded or redefined, and, if executable, executed. >>> >>> Classes and methods are just wrapped in a generated class. Statements are wrapped in a doit method within the class. Variable declarations are separated into their declaration part, which becomes a field, and their initializer which becomes an assignment in a doit method. Expressions become synthetic temporary variables in the same general form as variables. Imports follow a completely different mechanism, after being verified, they are simply recorded for use with subsequent snippets. >>> >>> Parsing uses the compiler API?s parse functionality, with one method in the parser overridden to allow stand-alone snippets as input. Analysis and compilation use the compiler API unmodified. JShell has its own FileManager so that input comes from Strings and output class files go to byte arrays. >>> >>> JShell?s interface to compiler API tasks is though the TaskManager, with separate subclasses of BaseTask for the cases above. Wrapping is accomplished with nested wraps that track location translation. Wrap is the abstract base class and holds its subclasses and other functionality. Nesting is with CompoundWrap which holds a sequence of Strings and Wraps. RangeWrap is the base wrap of user snippet sections. Information about the snippet, including its Wrap, found during parsing and analysis, is stored in subclasses of Snippet. Eval orchestrates these components. The JShell class is the entry point for JShell which delegates the whole eval() process to Eval. >>> >>> Please let me know if there are questions or areas I did not cover. >>> >>> Enjoy, >>> Robert >>> >> From jan.lahoda at oracle.com Wed Sep 2 12:46:01 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Wed, 02 Sep 2015 12:46:01 +0000 Subject: hg: kulla/dev/langtools: Code cleanup based on review comments Message-ID: <201509021246.t82Ck1QM012574@aojmv0008.oracle.com> Changeset: 5395dd59a4d6 Author: jlahoda Date: 2015-09-02 09:19 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/5395dd59a4d6 Code cleanup based on review comments ! src/jdk.jshell/share/classes/jdk/jshell/SourceCodeAnalysisImpl.java From paul.sandoz at oracle.com Fri Sep 4 17:19:27 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 4 Sep 2015 19:19:27 +0200 Subject: REPL tool implementation code review Message-ID: Hi, Here is the review of the REPL tool implementation. Generally it all looks very good. I ran out of time to analyse the tests more closely. Quite a few comments are suggestions related to using newer APIs or techniques. Paul. Implementation: jdk/internal/jshell/tool/JShellTool.java:91 > private static final String VERSION = "0.819?; > ? > case "-version": > cmdout.printf("jshell %s\n", VERSION); > return null; > Should this version be the JDK version, as is the case for other Java tools? jdk/internal/jshell/tool/JShellTool.java:128 > private IOContext input = null; > private boolean regenerateOnDeath = true; > private boolean live = false; > > SourceCodeAnalysis analysis; > JShell state = null; > Subscription shutdownSubscription = null; > > private boolean debug = false; > private boolean displayPrompt = true; > public boolean testPrompt = false; > private Feedback feedback = Feedback.Default; > private String cmdlineClasspath = null; > private String cmdlineStartup = null; > private String editor = null; > A small style thing, there is no need to initialize to default values (it?s also applied inconsistently) jdk/internal/jshell/tool/JShellTool.java:307 > byte[] encoded = Files.readAllBytes(Paths.get(filename)); > cmdlineStartup = new String(encoded); What character set is the file encoded in? should the platform's default charset be used or UTF-8? Same applies to processing the load list files. jdk/internal/jshell/tool/JShellTool.java:582 > int lastSlash = code.lastIndexOf('/'); //XXX Rogue comment. jdk/internal/jshell/tool/JShellTool.java:587 > try (DirectoryStream dir = Files.newDirectoryStream(current)) { > for (Path e : dir) { > if (!accept.test(e) || !e.getFileName().toString().startsWith(prefix)) > continue; > result.add(new Suggestion(e.getFileName() + (Files.isDirectory(e) ? "/" : ""), false)); > } > } catch (IOException ex) { > //ignore? > } See files.lines(path) that returns a stream with map.filter.collect(toCollection(ArrayList::new)) > if (path.isEmpty()) { > for (Path root : FileSystems.getDefault().getRootDirectories()) { > if (!accept.test(root) || !root.toString().startsWith(prefix)) > continue; > result.add(new Suggestion(root.toString(), false)); > } > } Alas we don?t have a FileSystem.rootDirectories method that returns a stream. jdk/internal/jshell/tool/JShellTool.java:645 > registerCommand(new Command("/list", "/l", "[all]", "list the source you have typed", > arg -> cmdList(arg), > new FixedCompletionProvider("all"))); You could use a method reference for "arg -> ??, same applies for other command declarations. jdk/internal/jshell/tool/JShellTool.java:716 > public List commandCompletionSuggestions(String code, int cursor, int[] anchor) { > code = code.substring(0, cursor); > int space = code.indexOf(' '); > List result = new ArrayList<>(); > > if (space == (-1)) { > for (Command cmd : new LinkedHashSet<>(commands.values())) { > if (cmd.kind == CommandKind.HIDDEN || cmd.kind == CommandKind.HELP_ONLY) > continue; > String key = cmd.aliases[0]; > if (key.startsWith(code)) { > result.add(new Suggestion(key + " ", false)); > } > } > anchor[0] = 0; > } else { > String arg = code.substring(space + 1); > String cmd = code.substring(0, space); > Command command = commands.get(cmd); > if (command != null) { > result.addAll(command.completions.completionSuggestions(arg, cursor - space, anchor)); > anchor[0] += space + 1; > } > } > > Collections.sort(result, (s1, s2) -> s1.continuation.compareTo(s2.continuation)); > return result; > } I believe you could use a stream here for both branches then do a sorted(?).collect(toList()) at the end of the method. For the first branch i think it might be something like values().distinct().filter(?).map(?).filter(?). For the second branch it is either an empty stream or the stream from the list. FYI: you can also to List.sort now. jdk/internal/jshell/tool/JShellTool.java:763 > if (arg == null || arg.isEmpty()) { I dunno if it is possible but it would be nice to normalise to empty strings rather than checking for null or an empty string. jdk/internal/jshell/tool/JShellTool.java:959 > Iterator it = keySet.iterator(); > while (it.hasNext()) { > if (!(it.next() instanceof PersistentSnippet)) { > it.remove(); > } > } See the bulk operation Collection.removeIf. > PersistentSnippet key = (PersistentSnippet) keySet.iterator().next(); > state.drop(key).forEach(e -> handleEvent(e)); For kicks you could also do it like this (not suggesting you actually do unless you really want to, it?s just instructive for new APIs in Java 8): keySet.stream().findFirst(). map(PersistentSnippet.class::cast). ifPresent(k -> state.drop(k).forEach(this::handleEvent)); jdk/internal/jshell/tool/JShellTool.java:981 > private void cmdEdit(String arg) { > Set keySet = argToKeySet(arg); > if (keySet == null) { > return; > } > Set srcSet = new LinkedHashSet<>(); > for (Snippet key : keySet) { > String src = key.source(); > switch (key.subKind()) { > case VAR_VALUE_SUBKIND: > break; > case ASSIGNMENT_SUBKIND: > case OTHER_EXPRESSION_SUBKIND: > case TEMP_VAR_EXPRESSION_SUBKIND: > if (!src.endsWith(";")) { > src = src + ";"; > } > srcSet.add(src); > break; > default: > srcSet.add(src); > break; > } > } > StringBuilder sb = new StringBuilder(); > for (String s : srcSet) { > sb.append(s); > sb.append('\n'); > } > String src = sb.toString(); There appears to be a reliance on an undefined iteration order, perhaps argToKeySet should return LinkedHashSet? I think this might be convertible to a stream with filter(?).map(?).collect(joining(?\n?, ??, ?\n")) jdk/internal/jshell/tool/JShellTool.java:1150 > try (BufferedWriter writer = new BufferedWriter(new FileWriter(resolveUserHome(filename)))) { See also Files.newBufferedWriter(?) i like that because it makes it more explicit whether you are appending to an existing file or not. jdk/internal/jshell/tool/JShellTool.java:1266 > List> errorsOnly(List> diagnostics) { > List> errors = new ArrayList<>(); > for (Diagnostic diag : diagnostics) { > switch (diag.getKind()) { > case ERROR: > errors.add(diag); > break; > case MANDATORY_WARNING: > case WARNING: > break; > case NOTE: > case OTHER: > break; > } > } > return errors; > } diagnostics.stream().filter(d -> d.getKind() == ERROR).collect(toList()); jdk/internal/jshell/tool/JShellTool.java:1710 > public ScannerIOContext(InputStream inStream, PrintStream pStream) { > this(new Scanner(inStream), pStream); > } Unused constructor. jdk/internal/jshell/tool/ExternalEditor.java:134 > private void saveFile() { > List lines; > try { > lines = Files.readAllLines(tmpfile); > } catch (IOException ex) { > errorHandler.accept("Failure read edit file: " + ex.getMessage()); > return ; > } > StringBuilder sb = new StringBuilder(); > for (String ln : lines) { > sb.append(ln); > sb.append('\n'); > } > saveHandler.accept(sb.toString()); > } Why don?t you just read as bytes then create a String from those bytes? > if (!key.pollEvents().isEmpty()) { > if (!input.terminalEditorRunning()) { > saveFile(); > } > } Why does the watcher thread need to call saveFile when it is also performed after the process running the external editor has terminated? I am guessing every time the file is saved in the editor it is processed by the REPL? jdk/internal/jshell/tool/EditPad.java:46 > public class EditPad extends JFrame implements Runnable { > private final Consumer errorHandler; > private final String initialText; > private final boolean[] closeLock; > private final Consumer saveHandler; > > EditPad(Consumer errorHandler, String initialText, > boolean[] closeLock, Consumer saveHandler) { > super("Edit Pad (Experimental)"); > this.errorHandler = errorHandler; > this.initialText = initialText; > this.closeLock = closeLock; > this.saveHandler = saveHandler; > } Unused field errorHandle. Is this still experimental? Is so should be it included? jdk/internal/jshell/tool/EditPad.java:114 > private void notifyClose() { > synchronized (closeLock) { > closeLock[0] = true; > closeLock.notify(); > } > } > > public static void edit(Consumer errorHandler, String initialText, > Consumer saveHandler) { > boolean[] closeLock = new boolean[1]; > SwingUtilities.invokeLater( > new EditPad(errorHandler, initialText, closeLock, saveHandler)); > synchronized (closeLock) { > while (!closeLock[0]) { > try { > closeLock.wait(); > } catch (InterruptedException ex) { > // ignore and loop > } > } > } > } You could use a shared CountDownLatch, rather than notify/wait/synchronization with a boolean array of one element. jdk/internal/jshell/tool/ConsoleIOContext.java:70 > TerminalFactory.registerFlavor(Flavor.WINDOWS, JShellWindowsTerminal :: new); > TerminalFactory.registerFlavor(Flavor.UNIX, JShellUnixTerminal :: new); The syntax for the constructor refs is a little unusual with the white space. Same further down when binding CRTL_UP/DOWN actions. jdk/internal/jshell/tool/ConsoleIOContext.java:136 > .forEach(s -> { result.add(s); }); forEach(result::add); > Optional prefix = > suggestions.stream() > .map(s -> s.continuation) > .reduce((s1, s2) -> commonPrefix(s1, s2)); .reduce(ConsoleIOContext::commonPrefix) > try { > Method setBuffer = in.getClass().getDeclaredMethod("setBuffer", String.class); > > setBuffer.setAccessible(true); > setBuffer.invoke(in, in.getHistory().current().toString()); > in.flush(); > } catch (ReflectiveOperationException | IOException ex) { > throw new IllegalStateException(ex); > } You might wanna comment more about the limitation in ConsoleReader and the necessity of calling a private method. > private static String commonPrefix(String str1, String str2) { > for (int i = 0; i < str2.length(); i++) { > if (!str1.startsWith(str2.substring(0, i + 1))) { > return str2.substring(0, i); > } > } > > return str2; > } What we need is a String mismatch method, from which to build a common prefix method, stay tuned?. jdk/internal/jshell/tool/ConsoleIOContext.java:505 > Set snippetsStart = new HashSet<>(); > for (String start : prefs.get(HISTORY_SNIPPET_START, "").split(";")) { > if (start.isEmpty()) > continue; > snippetsStart.add(Integer.parseInt(start)); > } Could streamify but perhaps not worth it. At least convert to if(!start.isEmpty()) snippetsStart.add(Integer.parseInt(start)); > List keys = new ArrayList<>(Arrays.asList(prefs.keys())); > Collections.sort(keys); Stream.of(prefs.keys()).sorted().collect(toList()); jdk/internal/jshell/tool/ConsoleIOContext.java: I am struggling to understand the interactions of the before/afterUse code blocks, the CircularBuffer, and the thread created for writing (or pushing) read values to the buffer. CircularBuffer feels like a blocking queue, but there is some additional communication signalling how much input is needed. Tests General point: (a) -> ? Can be rewritten as: a -> ... ReplToolTesting.java:302 > private void addKey(boolean after, T memberInfo, Map map) { > if (after) { > for (Iterator> iterator = map.entrySet().iterator(); iterator.hasNext(); ) { > Entry info = iterator.next(); > if (info.getValue().equals(memberInfo)) { > iterator.remove(); > break; > } > } > map.put(memberInfo.toString(), memberInfo); > } > } Just do: map.entrySet().removeIf(e -> e.getValue().equals(memberInfo)); ReplToolTesting.java:571 > public Consumer checkOutput() { > String fullType = type.equals("@interface")? "annotation interface" : type; > String expectedOutput = String.format("\\| *\\w+ %s %s", fullType, name); > Pattern outputPattern = Pattern.compile(expectedOutput); > return (s) -> { > Matcher matcher = outputPattern.matcher(s); > assertTrue(matcher.find(), "Expected: '" + expectedOutput + "', actual: " + s); > }; > } See Pattern.asPredicate, capture the returned predicate in the lambda instead of the pattern and call predicate.test. CommandCompletionTest.java:154 > private List listFiles(Path path, DirectoryStream.Filter filter) throws IOException { > List paths = new ArrayList<>(); > try (DirectoryStream stream = Files.newDirectoryStream(path, filter)) { > stream.iterator().forEachRemaining(p -> { > String fileName = p.getFileName().toString(); > if (Files.isDirectory(p)) { > fileName += "/"; > } > paths.add(fileName); > }); > } > Collections.sort(paths); > return paths; > } See Files.list which returns a Stream, you could probably reformulate using Predicate rather than DirectoryStream.Filter. StartOptionTest.java:68 > private void start(Consumer checkOutput, Consumer checkError, String...args) throws Exception { Space between ? and parameter name. Same goes for other cases in this file. From forax at univ-mlv.fr Fri Sep 4 18:46:57 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 4 Sep 2015 20:46:57 +0200 (CEST) Subject: REPL tool implementation code review In-Reply-To: References: Message-ID: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> Hi, just some comments on top of Paul's comments, On 09/04/2015 07:19 PM, Paul Sandoz wrote: > Hi, > > Here is the review of the REPL tool implementation. > > Generally it all looks very good. I ran out of time to analyse the tests more closely. Quite a few comments are suggestions related to using newer APIs or techniques. > > Paul. > > Implementation: > > > jdk/internal/jshell/tool/JShellTool.java:91 >> private static final String VERSION = "0.819?; >> ? >> case "-version": >> cmdout.printf("jshell %s\n", VERSION); >> return null; >> > > Should this version be the JDK version, as is the case for other Java tools? i agree. > > jdk/internal/jshell/tool/JShellTool.java:128 >> private IOContext input = null; >> private boolean regenerateOnDeath = true; >> private boolean live = false; >> >> SourceCodeAnalysis analysis; >> JShell state = null; >> Subscription shutdownSubscription = null; >> >> private boolean debug = false; >> private boolean displayPrompt = true; >> public boolean testPrompt = false; >> private Feedback feedback = Feedback.Default; >> private String cmdlineClasspath = null; >> private String cmdlineStartup = null; >> private String editor = null; >> > > A small style thing, there is no need to initialize to default values (it?s also applied inconsistently) > > jdk/internal/jshell/tool/JShellTool.java:307 >> byte[] encoded = Files.readAllBytes(Paths.get(filename)); >> cmdlineStartup = new String(encoded); > > > What character set is the file encoded in? should the platform's default charset be used or UTF-8? Same applies to processing the load list files. > > > jdk/internal/jshell/tool/JShellTool.java:582 >> int lastSlash = code.lastIndexOf('/'); //XXX > > Rogue comment. > > > jdk/internal/jshell/tool/JShellTool.java:587 >> try (DirectoryStream dir = Files.newDirectoryStream(current)) { >> for (Path e : dir) { >> if (!accept.test(e) || !e.getFileName().toString().startsWith(prefix)) >> continue; >> result.add(new Suggestion(e.getFileName() + (Files.isDirectory(e) ? "/" : ""), false)); >> } >> } catch (IOException ex) { >> //ignore? >> } > > See files.lines(path) that returns a stream with map.filter.collect(toCollection(ArrayList::new)) > > >> if (path.isEmpty()) { >> for (Path root : FileSystems.getDefault().getRootDirectories()) { >> if (!accept.test(root) || !root.toString().startsWith(prefix)) >> continue; >> result.add(new Suggestion(root.toString(), false)); >> } >> } > > Alas we don?t have a FileSystem.rootDirectories method that returns a stream. The array is two small to worth a dedicated stream method, no ? and one can use, Arrays.stream(...). > > > > jdk/internal/jshell/tool/JShellTool.java:645 >> registerCommand(new Command("/list", "/l", "[all]", "list the source you have typed", >> arg -> cmdList(arg), >> new FixedCompletionProvider("all"))); > > > You could use a method reference for "arg -> ??, same applies for other command declarations. Also, it's a good idea to put the lambda as last argument so one can write, new Command (foo, bar, baz, arg -> { ... }) > > > > jdk/internal/jshell/tool/JShellTool.java:716 >> public List commandCompletionSuggestions(String code, int cursor, int[] anchor) { >> code = code.substring(0, cursor); >> int space = code.indexOf(' '); >> List result = new ArrayList<>(); >> >> if (space == (-1)) { >> for (Command cmd : new LinkedHashSet<>(commands.values())) { >> if (cmd.kind == CommandKind.HIDDEN || cmd.kind == CommandKind.HELP_ONLY) >> continue; >> String key = cmd.aliases[0]; >> if (key.startsWith(code)) { >> result.add(new Suggestion(key + " ", false)); >> } >> } >> anchor[0] = 0; >> } else { >> String arg = code.substring(space + 1); >> String cmd = code.substring(0, space); >> Command command = commands.get(cmd); >> if (command != null) { >> result.addAll(command.completions.completionSuggestions(arg, cursor - space, anchor)); >> anchor[0] += space + 1; >> } >> } >> >> Collections.sort(result, (s1, s2) -> s1.continuation.compareTo(s2.continuation)); >> return result; >> } > > > > I believe you could use a stream here for both branches then do a sorted(?).collect(toList()) at the end of the method. For the first branch i think it might be something like values().distinct().filter(?).map(?).filter(?). For the second branch it is either an empty stream or the stream from the list. > > FYI: you can also to List.sort now. > > > jdk/internal/jshell/tool/JShellTool.java:763 >> if (arg == null || arg.isEmpty()) { > > I dunno if it is possible but it would be nice to normalise to empty strings rather than checking for null or an empty string. > > > jdk/internal/jshell/tool/JShellTool.java:959 >> Iterator it = keySet.iterator(); >> while (it.hasNext()) { >> if (!(it.next() instanceof PersistentSnippet)) { >> it.remove(); >> } >> } > > See the bulk operation Collection.removeIf. > > >> PersistentSnippet key = (PersistentSnippet) keySet.iterator().next(); >> state.drop(key).forEach(e -> handleEvent(e)); > > > For kicks you could also do it like this (not suggesting you actually do unless you really want to, it?s just instructive for new APIs in Java 8): > > keySet.stream().findFirst(). > map(PersistentSnippet.class::cast). > ifPresent(k -> state.drop(k).forEach(this::handleEvent)); > > > jdk/internal/jshell/tool/JShellTool.java:981 >> private void cmdEdit(String arg) { >> Set keySet = argToKeySet(arg); >> if (keySet == null) { >> return; >> } >> Set srcSet = new LinkedHashSet<>(); >> for (Snippet key : keySet) { >> String src = key.source(); >> switch (key.subKind()) { >> case VAR_VALUE_SUBKIND: >> break; >> case ASSIGNMENT_SUBKIND: >> case OTHER_EXPRESSION_SUBKIND: >> case TEMP_VAR_EXPRESSION_SUBKIND: >> if (!src.endsWith(";")) { >> src = src + ";"; >> } >> srcSet.add(src); >> break; >> default: >> srcSet.add(src); >> break; >> } >> } >> StringBuilder sb = new StringBuilder(); >> for (String s : srcSet) { >> sb.append(s); >> sb.append('\n'); >> } >> String src = sb.toString(); > > > There appears to be a reliance on an undefined iteration order, perhaps argToKeySet should return LinkedHashSet? > > I think this might be convertible to a stream with filter(?).map(?).collect(joining(?\n?, ??, ?\n")) you need to use distinct() between map() and collect(). > > > > jdk/internal/jshell/tool/JShellTool.java:1150 >> try (BufferedWriter writer = new BufferedWriter(new FileWriter(resolveUserHome(filename)))) { > > See also Files.newBufferedWriter(?) i like that because it makes it more explicit whether you are appending to an existing file or not. and more correct because with "new BufferedWriter(new FileWriter" if the constructor of BufferedWriter raises an exception, the Filewriter will not be closed :( > > > > jdk/internal/jshell/tool/JShellTool.java:1266 >> List> errorsOnly(List> diagnostics) { >> List> errors = new ArrayList<>(); >> for (Diagnostic diag : diagnostics) { >> switch (diag.getKind()) { >> case ERROR: >> errors.add(diag); >> break; >> case MANDATORY_WARNING: >> case WARNING: >> break; >> case NOTE: >> case OTHER: >> break; >> } >> } >> return errors; >> } > > diagnostics.stream().filter(d -> d.getKind() == ERROR).collect(toList()); and instead of having a method that takes a List and returns a List, it's better to have a method that take a Stream and returns a Stream to avoid to create intermediary collections. > > > > jdk/internal/jshell/tool/JShellTool.java:1710 >> public ScannerIOContext(InputStream inStream, PrintStream pStream) { >> this(new Scanner(inStream), pStream); >> } > > Unused constructor. > > > jdk/internal/jshell/tool/ExternalEditor.java:134 >> private void saveFile() { >> List lines; >> try { >> lines = Files.readAllLines(tmpfile); >> } catch (IOException ex) { >> errorHandler.accept("Failure read edit file: " + ex.getMessage()); >> return ; >> } >> StringBuilder sb = new StringBuilder(); >> for (String ln : lines) { >> sb.append(ln); >> sb.append('\n'); >> } >> saveHandler.accept(sb.toString()); >> } > > Why don?t you just read as bytes then create a String from those bytes? or Files.lines(tmpFile).collect(Collectors.joining("\n")) > > >> if (!key.pollEvents().isEmpty()) { >> if (!input.terminalEditorRunning()) { >> saveFile(); >> } >> } > > Why does the watcher thread need to call saveFile when it is also performed after the process running the external editor has terminated? I am guessing every time the file is saved in the editor it is processed by the REPL? > > > jdk/internal/jshell/tool/EditPad.java:46 >> public class EditPad extends JFrame implements Runnable { >> private final Consumer errorHandler; >> private final String initialText; >> private final boolean[] closeLock; >> private final Consumer saveHandler; >> >> EditPad(Consumer errorHandler, String initialText, >> boolean[] closeLock, Consumer saveHandler) { >> super("Edit Pad (Experimental)"); >> this.errorHandler = errorHandler; >> this.initialText = initialText; >> this.closeLock = closeLock; >> this.saveHandler = saveHandler; >> } > > > Unused field errorHandle. > > Is this still experimental? Is so should be it included? > > > jdk/internal/jshell/tool/EditPad.java:114 >> private void notifyClose() { >> synchronized (closeLock) { >> closeLock[0] = true; >> closeLock.notify(); >> } >> } >> >> public static void edit(Consumer errorHandler, String initialText, >> Consumer saveHandler) { >> boolean[] closeLock = new boolean[1]; >> SwingUtilities.invokeLater( >> new EditPad(errorHandler, initialText, closeLock, saveHandler)); >> synchronized (closeLock) { >> while (!closeLock[0]) { >> try { >> closeLock.wait(); >> } catch (InterruptedException ex) { >> // ignore and loop >> } >> } >> } >> } > > You could use a shared CountDownLatch, rather than notify/wait/synchronization with a boolean array of one element. > > > jdk/internal/jshell/tool/ConsoleIOContext.java:70 >> TerminalFactory.registerFlavor(Flavor.WINDOWS, JShellWindowsTerminal :: new); >> TerminalFactory.registerFlavor(Flavor.UNIX, JShellUnixTerminal :: new); > > The syntax for the constructor refs is a little unusual with the white space. Same further down when binding CRTL_UP/DOWN actions. > > > jdk/internal/jshell/tool/ConsoleIOContext.java:136 >> .forEach(s -> { result.add(s); }); > > forEach(result::add); > >> Optional prefix = >> suggestions.stream() >> .map(s -> s.continuation) >> .reduce((s1, s2) -> commonPrefix(s1, s2)); > > .reduce(ConsoleIOContext::commonPrefix) > > > >> try { >> Method setBuffer = in.getClass().getDeclaredMethod("setBuffer", String.class); >> >> setBuffer.setAccessible(true); >> setBuffer.invoke(in, in.getHistory().current().toString()); >> in.flush(); >> } catch (ReflectiveOperationException | IOException ex) { >> throw new IllegalStateException(ex); >> } > > You might wanna comment more about the limitation in ConsoleReader and the necessity of calling a private method. > > >> private static String commonPrefix(String str1, String str2) { >> for (int i = 0; i < str2.length(); i++) { >> if (!str1.startsWith(str2.substring(0, i + 1))) { >> return str2.substring(0, i); >> } >> } >> >> return str2; >> } > > What we need is a String mismatch method, from which to build a common prefix method, stay tuned?. > > > jdk/internal/jshell/tool/ConsoleIOContext.java:505 >> Set snippetsStart = new HashSet<>(); >> for (String start : prefs.get(HISTORY_SNIPPET_START, "").split(";")) { >> if (start.isEmpty()) >> continue; >> snippetsStart.add(Integer.parseInt(start)); >> } > > Could streamify but perhaps not worth it. At least convert to > > if(!start.isEmpty()) > snippetsStart.add(Integer.parseInt(start)); > > >> List keys = new ArrayList<>(Arrays.asList(prefs.keys())); >> Collections.sort(keys); > > Stream.of(prefs.keys()).sorted().collect(toList()); > > > jdk/internal/jshell/tool/ConsoleIOContext.java: > > I am struggling to understand the interactions of the before/afterUse code blocks, the CircularBuffer, and the thread created for writing (or pushing) read values to the buffer. > > CircularBuffer feels like a blocking queue, but there is some additional communication signalling how much input is needed. > > > > Tests > > General point: > > (a) -> ? > > Can be rewritten as: > > a -> ... > > > ReplToolTesting.java:302 >> private void addKey(boolean after, T memberInfo, Map map) { >> if (after) { >> for (Iterator> iterator = map.entrySet().iterator(); iterator.hasNext(); ) { >> Entry info = iterator.next(); >> if (info.getValue().equals(memberInfo)) { >> iterator.remove(); >> break; >> } >> } >> map.put(memberInfo.toString(), memberInfo); >> } >> } > > Just do: > > map.entrySet().removeIf(e -> e.getValue().equals(memberInfo)); > > > ReplToolTesting.java:571 >> public Consumer checkOutput() { >> String fullType = type.equals("@interface")? "annotation interface" : type; >> String expectedOutput = String.format("\\| *\\w+ %s %s", fullType, name); >> Pattern outputPattern = Pattern.compile(expectedOutput); >> return (s) -> { >> Matcher matcher = outputPattern.matcher(s); >> assertTrue(matcher.find(), "Expected: '" + expectedOutput + "', actual: " + s); >> }; >> } > > > See Pattern.asPredicate, capture the returned predicate in the lambda instead of the pattern and call predicate.test. > > > CommandCompletionTest.java:154 >> private List listFiles(Path path, DirectoryStream.Filter filter) throws IOException { >> List paths = new ArrayList<>(); >> try (DirectoryStream stream = Files.newDirectoryStream(path, filter)) { >> stream.iterator().forEachRemaining(p -> { >> String fileName = p.getFileName().toString(); >> if (Files.isDirectory(p)) { >> fileName += "/"; >> } >> paths.add(fileName); >> }); >> } >> Collections.sort(paths); >> return paths; >> } > > > See Files.list which returns a Stream, you could probably reformulate using Predicate rather than DirectoryStream.Filter. and try to do not use stream.iterator() which is usually slow, stream.forEach() is fine here. > > > > StartOptionTest.java:68 >> private void start(Consumer checkOutput, Consumer checkError, String...args) throws Exception { > > > Space between ? and parameter name. Same goes for other cases in this file. > cheers, R?mi From robert.field at oracle.com Fri Sep 4 19:10:19 2015 From: robert.field at oracle.com (Robert Field) Date: Fri, 4 Sep 2015 12:10:19 -0700 Subject: REPL tool implementation code review In-Reply-To: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> Message-ID: <4081C2A8-5950-40BA-A0ED-A3CD27A303AA@oracle.com> Thank you Paul, and R?mi! -Robert > On Sep 4, 2015, at 11:46 AM, Remi Forax wrote: > > Hi, > just some comments on top of Paul's comments, > > On 09/04/2015 07:19 PM, Paul Sandoz wrote: >> Hi, >> >> Here is the review of the REPL tool implementation. >> >> Generally it all looks very good. I ran out of time to analyse the tests more closely. Quite a few comments are suggestions related to using newer APIs or techniques. >> >> Paul. >> >> Implementation: >> >> >> jdk/internal/jshell/tool/JShellTool.java:91 >>> private static final String VERSION = "0.819?; >>> ? >>> case "-version": >>> cmdout.printf("jshell %s\n", VERSION); >>> return null; >>> >> >> Should this version be the JDK version, as is the case for other Java tools? > > i agree. > >> >> jdk/internal/jshell/tool/JShellTool.java:128 >>> private IOContext input = null; >>> private boolean regenerateOnDeath = true; >>> private boolean live = false; >>> >>> SourceCodeAnalysis analysis; >>> JShell state = null; >>> Subscription shutdownSubscription = null; >>> >>> private boolean debug = false; >>> private boolean displayPrompt = true; >>> public boolean testPrompt = false; >>> private Feedback feedback = Feedback.Default; >>> private String cmdlineClasspath = null; >>> private String cmdlineStartup = null; >>> private String editor = null; >>> >> >> A small style thing, there is no need to initialize to default values (it?s also applied inconsistently) >> >> jdk/internal/jshell/tool/JShellTool.java:307 >>> byte[] encoded = Files.readAllBytes(Paths.get(filename)); >>> cmdlineStartup = new String(encoded); >> >> >> What character set is the file encoded in? should the platform's default charset be used or UTF-8? Same applies to processing the load list files. >> >> >> jdk/internal/jshell/tool/JShellTool.java:582 >>> int lastSlash = code.lastIndexOf('/'); //XXX >> >> Rogue comment. >> >> >> jdk/internal/jshell/tool/JShellTool.java:587 >>> try (DirectoryStream dir = Files.newDirectoryStream(current)) { >>> for (Path e : dir) { >>> if (!accept.test(e) || !e.getFileName().toString().startsWith(prefix)) >>> continue; >>> result.add(new Suggestion(e.getFileName() + (Files.isDirectory(e) ? "/" : ""), false)); >>> } >>> } catch (IOException ex) { >>> //ignore? >>> } >> >> See files.lines(path) that returns a stream with map.filter.collect(toCollection(ArrayList::new)) >> >> >>> if (path.isEmpty()) { >>> for (Path root : FileSystems.getDefault().getRootDirectories()) { >>> if (!accept.test(root) || !root.toString().startsWith(prefix)) >>> continue; >>> result.add(new Suggestion(root.toString(), false)); >>> } >>> } >> >> Alas we don?t have a FileSystem.rootDirectories method that returns a stream. > > The array is two small to worth a dedicated stream method, no ? > and one can use, Arrays.stream(...). > >> >> >> >> jdk/internal/jshell/tool/JShellTool.java:645 >>> registerCommand(new Command("/list", "/l", "[all]", "list the source you have typed", >>> arg -> cmdList(arg), >>> new FixedCompletionProvider("all"))); >> >> >> You could use a method reference for "arg -> ??, same applies for other command declarations. > > Also, it's a good idea to put the lambda as last argument so one can write, > > new Command (foo, bar, baz, arg -> { > ... > }) > >> >> >> >> jdk/internal/jshell/tool/JShellTool.java:716 >>> public List commandCompletionSuggestions(String code, int cursor, int[] anchor) { >>> code = code.substring(0, cursor); >>> int space = code.indexOf(' '); >>> List result = new ArrayList<>(); >>> >>> if (space == (-1)) { >>> for (Command cmd : new LinkedHashSet<>(commands.values())) { >>> if (cmd.kind == CommandKind.HIDDEN || cmd.kind == CommandKind.HELP_ONLY) >>> continue; >>> String key = cmd.aliases[0]; >>> if (key.startsWith(code)) { >>> result.add(new Suggestion(key + " ", false)); >>> } >>> } >>> anchor[0] = 0; >>> } else { >>> String arg = code.substring(space + 1); >>> String cmd = code.substring(0, space); >>> Command command = commands.get(cmd); >>> if (command != null) { >>> result.addAll(command.completions.completionSuggestions(arg, cursor - space, anchor)); >>> anchor[0] += space + 1; >>> } >>> } >>> >>> Collections.sort(result, (s1, s2) -> s1.continuation.compareTo(s2.continuation)); >>> return result; >>> } >> >> >> >> I believe you could use a stream here for both branches then do a sorted(?).collect(toList()) at the end of the method. For the first branch i think it might be something like values().distinct().filter(?).map(?).filter(?). For the second branch it is either an empty stream or the stream from the list. >> >> FYI: you can also to List.sort now. >> >> >> jdk/internal/jshell/tool/JShellTool.java:763 >>> if (arg == null || arg.isEmpty()) { >> >> I dunno if it is possible but it would be nice to normalise to empty strings rather than checking for null or an empty string. >> >> >> jdk/internal/jshell/tool/JShellTool.java:959 >>> Iterator it = keySet.iterator(); >>> while (it.hasNext()) { >>> if (!(it.next() instanceof PersistentSnippet)) { >>> it.remove(); >>> } >>> } >> >> See the bulk operation Collection.removeIf. >> >> >>> PersistentSnippet key = (PersistentSnippet) keySet.iterator().next(); >>> state.drop(key).forEach(e -> handleEvent(e)); >> >> >> For kicks you could also do it like this (not suggesting you actually do unless you really want to, it?s just instructive for new APIs in Java 8): >> >> keySet.stream().findFirst(). >> map(PersistentSnippet.class::cast). >> ifPresent(k -> state.drop(k).forEach(this::handleEvent)); >> >> >> jdk/internal/jshell/tool/JShellTool.java:981 >>> private void cmdEdit(String arg) { >>> Set keySet = argToKeySet(arg); >>> if (keySet == null) { >>> return; >>> } >>> Set srcSet = new LinkedHashSet<>(); >>> for (Snippet key : keySet) { >>> String src = key.source(); >>> switch (key.subKind()) { >>> case VAR_VALUE_SUBKIND: >>> break; >>> case ASSIGNMENT_SUBKIND: >>> case OTHER_EXPRESSION_SUBKIND: >>> case TEMP_VAR_EXPRESSION_SUBKIND: >>> if (!src.endsWith(";")) { >>> src = src + ";"; >>> } >>> srcSet.add(src); >>> break; >>> default: >>> srcSet.add(src); >>> break; >>> } >>> } >>> StringBuilder sb = new StringBuilder(); >>> for (String s : srcSet) { >>> sb.append(s); >>> sb.append('\n'); >>> } >>> String src = sb.toString(); >> >> >> There appears to be a reliance on an undefined iteration order, perhaps argToKeySet should return LinkedHashSet? >> >> I think this might be convertible to a stream with filter(?).map(?).collect(joining(?\n?, ??, ?\n")) > > you need to use distinct() between map() and collect(). > >> >> >> >> jdk/internal/jshell/tool/JShellTool.java:1150 >>> try (BufferedWriter writer = new BufferedWriter(new FileWriter(resolveUserHome(filename)))) { >> >> See also Files.newBufferedWriter(?) i like that because it makes it more explicit whether you are appending to an existing file or not. > > and more correct because with "new BufferedWriter(new FileWriter" if the constructor of BufferedWriter raises an exception, the Filewriter will not be closed :( >> >> >> >> jdk/internal/jshell/tool/JShellTool.java:1266 >>> List> errorsOnly(List> diagnostics) { >>> List> errors = new ArrayList<>(); >>> for (Diagnostic diag : diagnostics) { >>> switch (diag.getKind()) { >>> case ERROR: >>> errors.add(diag); >>> break; >>> case MANDATORY_WARNING: >>> case WARNING: >>> break; >>> case NOTE: >>> case OTHER: >>> break; >>> } >>> } >>> return errors; >>> } >> >> diagnostics.stream().filter(d -> d.getKind() == ERROR).collect(toList()); > > and instead of having a method that takes a List and returns a List, it's better to have > a method that take a Stream and returns a Stream to avoid to create intermediary collections. > >> >> >> >> jdk/internal/jshell/tool/JShellTool.java:1710 >>> public ScannerIOContext(InputStream inStream, PrintStream pStream) { >>> this(new Scanner(inStream), pStream); >>> } >> >> Unused constructor. >> >> >> jdk/internal/jshell/tool/ExternalEditor.java:134 >>> private void saveFile() { >>> List lines; >>> try { >>> lines = Files.readAllLines(tmpfile); >>> } catch (IOException ex) { >>> errorHandler.accept("Failure read edit file: " + ex.getMessage()); >>> return ; >>> } >>> StringBuilder sb = new StringBuilder(); >>> for (String ln : lines) { >>> sb.append(ln); >>> sb.append('\n'); >>> } >>> saveHandler.accept(sb.toString()); >>> } >> >> Why don?t you just read as bytes then create a String from those bytes? > > or Files.lines(tmpFile).collect(Collectors.joining("\n")) > >> >> >>> if (!key.pollEvents().isEmpty()) { >>> if (!input.terminalEditorRunning()) { >>> saveFile(); >>> } >>> } >> >> Why does the watcher thread need to call saveFile when it is also performed after the process running the external editor has terminated? I am guessing every time the file is saved in the editor it is processed by the REPL? >> >> >> jdk/internal/jshell/tool/EditPad.java:46 >>> public class EditPad extends JFrame implements Runnable { >>> private final Consumer errorHandler; >>> private final String initialText; >>> private final boolean[] closeLock; >>> private final Consumer saveHandler; >>> >>> EditPad(Consumer errorHandler, String initialText, >>> boolean[] closeLock, Consumer saveHandler) { >>> super("Edit Pad (Experimental)"); >>> this.errorHandler = errorHandler; >>> this.initialText = initialText; >>> this.closeLock = closeLock; >>> this.saveHandler = saveHandler; >>> } >> >> >> Unused field errorHandle. >> >> Is this still experimental? Is so should be it included? >> >> >> jdk/internal/jshell/tool/EditPad.java:114 >>> private void notifyClose() { >>> synchronized (closeLock) { >>> closeLock[0] = true; >>> closeLock.notify(); >>> } >>> } >>> >>> public static void edit(Consumer errorHandler, String initialText, >>> Consumer saveHandler) { >>> boolean[] closeLock = new boolean[1]; >>> SwingUtilities.invokeLater( >>> new EditPad(errorHandler, initialText, closeLock, saveHandler)); >>> synchronized (closeLock) { >>> while (!closeLock[0]) { >>> try { >>> closeLock.wait(); >>> } catch (InterruptedException ex) { >>> // ignore and loop >>> } >>> } >>> } >>> } >> >> You could use a shared CountDownLatch, rather than notify/wait/synchronization with a boolean array of one element. >> >> >> jdk/internal/jshell/tool/ConsoleIOContext.java:70 >>> TerminalFactory.registerFlavor(Flavor.WINDOWS, JShellWindowsTerminal :: new); >>> TerminalFactory.registerFlavor(Flavor.UNIX, JShellUnixTerminal :: new); >> >> The syntax for the constructor refs is a little unusual with the white space. Same further down when binding CRTL_UP/DOWN actions. >> >> >> jdk/internal/jshell/tool/ConsoleIOContext.java:136 >>> .forEach(s -> { result.add(s); }); >> >> forEach(result::add); >> >>> Optional prefix = >>> suggestions.stream() >>> .map(s -> s.continuation) >>> .reduce((s1, s2) -> commonPrefix(s1, s2)); >> >> .reduce(ConsoleIOContext::commonPrefix) >> >> >> >>> try { >>> Method setBuffer = in.getClass().getDeclaredMethod("setBuffer", String.class); >>> >>> setBuffer.setAccessible(true); >>> setBuffer.invoke(in, in.getHistory().current().toString()); >>> in.flush(); >>> } catch (ReflectiveOperationException | IOException ex) { >>> throw new IllegalStateException(ex); >>> } >> >> You might wanna comment more about the limitation in ConsoleReader and the necessity of calling a private method. >> >> >>> private static String commonPrefix(String str1, String str2) { >>> for (int i = 0; i < str2.length(); i++) { >>> if (!str1.startsWith(str2.substring(0, i + 1))) { >>> return str2.substring(0, i); >>> } >>> } >>> >>> return str2; >>> } >> >> What we need is a String mismatch method, from which to build a common prefix method, stay tuned?. >> >> >> jdk/internal/jshell/tool/ConsoleIOContext.java:505 >>> Set snippetsStart = new HashSet<>(); >>> for (String start : prefs.get(HISTORY_SNIPPET_START, "").split(";")) { >>> if (start.isEmpty()) >>> continue; >>> snippetsStart.add(Integer.parseInt(start)); >>> } >> >> Could streamify but perhaps not worth it. At least convert to >> >> if(!start.isEmpty()) >> snippetsStart.add(Integer.parseInt(start)); >> >> >>> List keys = new ArrayList<>(Arrays.asList(prefs.keys())); >>> Collections.sort(keys); >> >> Stream.of(prefs.keys()).sorted().collect(toList()); >> >> >> jdk/internal/jshell/tool/ConsoleIOContext.java: >> >> I am struggling to understand the interactions of the before/afterUse code blocks, the CircularBuffer, and the thread created for writing (or pushing) read values to the buffer. >> >> CircularBuffer feels like a blocking queue, but there is some additional communication signalling how much input is needed. >> >> >> >> Tests >> >> General point: >> >> (a) -> ? >> >> Can be rewritten as: >> >> a -> ... >> >> >> ReplToolTesting.java:302 >>> private void addKey(boolean after, T memberInfo, Map map) { >>> if (after) { >>> for (Iterator> iterator = map.entrySet().iterator(); iterator.hasNext(); ) { >>> Entry info = iterator.next(); >>> if (info.getValue().equals(memberInfo)) { >>> iterator.remove(); >>> break; >>> } >>> } >>> map.put(memberInfo.toString(), memberInfo); >>> } >>> } >> >> Just do: >> >> map.entrySet().removeIf(e -> e.getValue().equals(memberInfo)); >> >> >> ReplToolTesting.java:571 >>> public Consumer checkOutput() { >>> String fullType = type.equals("@interface")? "annotation interface" : type; >>> String expectedOutput = String.format("\\| *\\w+ %s %s", fullType, name); >>> Pattern outputPattern = Pattern.compile(expectedOutput); >>> return (s) -> { >>> Matcher matcher = outputPattern.matcher(s); >>> assertTrue(matcher.find(), "Expected: '" + expectedOutput + "', actual: " + s); >>> }; >>> } >> >> >> See Pattern.asPredicate, capture the returned predicate in the lambda instead of the pattern and call predicate.test. >> >> >> CommandCompletionTest.java:154 >>> private List listFiles(Path path, DirectoryStream.Filter filter) throws IOException { >>> List paths = new ArrayList<>(); >>> try (DirectoryStream stream = Files.newDirectoryStream(path, filter)) { >>> stream.iterator().forEachRemaining(p -> { >>> String fileName = p.getFileName().toString(); >>> if (Files.isDirectory(p)) { >>> fileName += "/"; >>> } >>> paths.add(fileName); >>> }); >>> } >>> Collections.sort(paths); >>> return paths; >>> } >> >> >> See Files.list which returns a Stream, you could probably reformulate using Predicate rather than DirectoryStream.Filter. > > and try to do not use stream.iterator() which is usually slow, stream.forEach() is fine here. > >> >> >> >> StartOptionTest.java:68 >>> private void start(Consumer checkOutput, Consumer checkError, String...args) throws Exception { >> >> >> Space between ? and parameter name. Same goes for other cases in this file. >> > > cheers, > R?mi > > From paul.sandoz at oracle.com Mon Sep 7 08:14:43 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 7 Sep 2015 10:14:43 +0200 Subject: REPL tool implementation code review In-Reply-To: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> Message-ID: Hi, Some comments on Remi?s comments, don?t worry the set keeps getting smaller :-) On 4 Sep 2015, at 20:46, Remi Forax wrote: >>> if (path.isEmpty()) { >>> for (Path root : FileSystems.getDefault().getRootDirectories()) { >>> if (!accept.test(root) || !root.toString().startsWith(prefix)) >>> continue; >>> result.add(new Suggestion(root.toString(), false)); >>> } >>> } >> >> Alas we don?t have a FileSystem.rootDirectories method that returns a stream. > > The array is two small to worth a dedicated stream method, no ? > and one can use, Arrays.stream(?) Yes, good point. >> >> jdk/internal/jshell/tool/JShellTool.java:981 >>> private void cmdEdit(String arg) { >>> Set keySet = argToKeySet(arg); >>> if (keySet == null) { >>> return; >>> } >>> Set srcSet = new LinkedHashSet<>(); >>> for (Snippet key : keySet) { >>> String src = key.source(); >>> switch (key.subKind()) { >>> case VAR_VALUE_SUBKIND: >>> break; >>> case ASSIGNMENT_SUBKIND: >>> case OTHER_EXPRESSION_SUBKIND: >>> case TEMP_VAR_EXPRESSION_SUBKIND: >>> if (!src.endsWith(";")) { >>> src = src + ";"; >>> } >>> srcSet.add(src); >>> break; >>> default: >>> srcSet.add(src); >>> break; >>> } >>> } >>> StringBuilder sb = new StringBuilder(); >>> for (String s : srcSet) { >>> sb.append(s); >>> sb.append('\n'); >>> } >>> String src = sb.toString(); >> >> >> There appears to be a reliance on an undefined iteration order, perhaps argToKeySet should return LinkedHashSet? >> >> I think this might be convertible to a stream with filter(?).map(?).collect(joining(?\n?, ??, ?\n")) > > you need to use distinct() between map() and collect(). > Yes, i missed the transition key.source(). >> >> jdk/internal/jshell/tool/ExternalEditor.java:134 >>> private void saveFile() { >>> List lines; >>> try { >>> lines = Files.readAllLines(tmpfile); >>> } catch (IOException ex) { >>> errorHandler.accept("Failure read edit file: " + ex.getMessage()); >>> return ; >>> } >>> StringBuilder sb = new StringBuilder(); >>> for (String ln : lines) { >>> sb.append(ln); >>> sb.append('\n'); >>> } >>> saveHandler.accept(sb.toString()); >>> } >> >> Why don?t you just read as bytes then create a String from those bytes? > > or Files.lines(tmpFile).collect(Collectors.joining("\n?)) > I thought that initially too, but then IIRC the list of lines is consumed to reproduce the file as a string, AFAICT the only difference being is a CR/LF is guaranteed at the end. >> CommandCompletionTest.java:154 >>> private List listFiles(Path path, DirectoryStream.Filter filter) throws IOException { >>> List paths = new ArrayList<>(); >>> try (DirectoryStream stream = Files.newDirectoryStream(path, filter)) { >>> stream.iterator().forEachRemaining(p -> { >>> String fileName = p.getFileName().toString(); >>> if (Files.isDirectory(p)) { >>> fileName += "/"; >>> } >>> paths.add(fileName); >>> }); >>> } >>> Collections.sort(paths); >>> return paths; >>> } >> >> >> See Files.list which returns a Stream, you could probably reformulate using Predicate rather than DirectoryStream.Filter. > > and try to do not use stream.iterator() which is usually slow, stream.forEach() is fine here. > Yes, try to avoid using stream.iterator() unless absolutely necessary. Paul. From forax at univ-mlv.fr Mon Sep 7 09:43:19 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Mon, 7 Sep 2015 11:43:19 +0200 (CEST) Subject: REPL tool implementation code review In-Reply-To: References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> Message-ID: <366865632.261257.1441618999693.JavaMail.zimbra@u-pem.fr> one comment on Paul's comments :) ----- Mail original ----- > De: "Paul Sandoz" > Cc: kulla-dev at openjdk.java.net > Envoy?: Lundi 7 Septembre 2015 10:14:43 > Objet: Re: REPL tool implementation code review > > Hi, > > Some comments on Remi?s comments, don?t worry the set keeps getting smaller > :-) > > > On 4 Sep 2015, at 20:46, Remi Forax wrote: > >> > >> jdk/internal/jshell/tool/ExternalEditor.java:134 > >>> private void saveFile() { > >>> List lines; > >>> try { > >>> lines = Files.readAllLines(tmpfile); > >>> } catch (IOException ex) { > >>> errorHandler.accept("Failure read edit file: " + ex.getMessage()); > >>> return ; > >>> } > >>> StringBuilder sb = new StringBuilder(); > >>> for (String ln : lines) { > >>> sb.append(ln); > >>> sb.append('\n'); > >>> } > >>> saveHandler.accept(sb.toString()); > >>> } > >> > >> Why don?t you just read as bytes then create a String from those bytes? > > > > or Files.lines(tmpFile).collect(Collectors.joining("\n?)) > > > > I thought that initially too, but then IIRC the list of lines is consumed to > reproduce the file as a string, AFAICT the only difference being is a CR/LF > is guaranteed at the end. with '\n' at the end of the last line: Files.lines(tmpFile).collect(Collectors.joining("\n", "", "\n")) the other difference is that the catch block needs to catch IOException and UncheckedIOException (or IOError, i don't remember) if an IOExcepton occurs after the file has been opened. > > Paul. > Remi From paul.sandoz at oracle.com Mon Sep 7 09:50:44 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 7 Sep 2015 11:50:44 +0200 Subject: REPL tool implementation code review In-Reply-To: <366865632.261257.1441618999693.JavaMail.zimbra@u-pem.fr> References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> <366865632.261257.1441618999693.JavaMail.zimbra@u-pem.fr> Message-ID: <5DA74EA0-3BA6-488E-803C-39F6FEEC3B82@oracle.com> On 7 Sep 2015, at 11:43, Remi Forax wrote: > one comment on Paul's comments :) > Last one, promise :-) > ----- Mail original ----- >> De: "Paul Sandoz" >> Cc: kulla-dev at openjdk.java.net >> Envoy?: Lundi 7 Septembre 2015 10:14:43 >> Objet: Re: REPL tool implementation code review >> >> Hi, >> >> Some comments on Remi?s comments, don?t worry the set keeps getting smaller >> :-) >> >> >> On 4 Sep 2015, at 20:46, Remi Forax wrote: >>>> >>>> jdk/internal/jshell/tool/ExternalEditor.java:134 >>>>> private void saveFile() { >>>>> List lines; >>>>> try { >>>>> lines = Files.readAllLines(tmpfile); >>>>> } catch (IOException ex) { >>>>> errorHandler.accept("Failure read edit file: " + ex.getMessage()); >>>>> return ; >>>>> } >>>>> StringBuilder sb = new StringBuilder(); >>>>> for (String ln : lines) { >>>>> sb.append(ln); >>>>> sb.append('\n'); >>>>> } >>>>> saveHandler.accept(sb.toString()); >>>>> } >>>> >>>> Why don?t you just read as bytes then create a String from those bytes? >>> >>> or Files.lines(tmpFile).collect(Collectors.joining("\n?)) >>> >> >> I thought that initially too, but then IIRC the list of lines is consumed to >> reproduce the file as a string, AFAICT the only difference being is a CR/LF >> is guaranteed at the end. > > with '\n' at the end of the last line: > Files.lines(tmpFile).collect(Collectors.joining("\n", "", "\n?) Yes, you can do that, but my point is there may be no need to create a list of lines only to then reconstitute as a string containing those lines using a string builder, but perhaps there is some normalisation of line feeds going on? Paul. From jan.lahoda at oracle.com Mon Sep 7 09:52:12 2015 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Mon, 07 Sep 2015 11:52:12 +0200 Subject: REPL tool implementation code review In-Reply-To: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> Message-ID: <55ED5E4C.3080700@oracle.com> Hi, Thanks for the review. On 4.9.2015 20:46, Remi Forax wrote: > Hi, > just some comments on top of Paul's comments, > > On 09/04/2015 07:19 PM, Paul Sandoz wrote: >> Hi, >> >> Here is the review of the REPL tool implementation. >> >> Generally it all looks very good. I ran out of time to analyse the tests more closely. Quite a few comments are suggestions related to using newer APIs or techniques. >> >> Paul. >> >> Implementation: >> >> >> jdk/internal/jshell/tool/JShellTool.java:307 >>> byte[] encoded = Files.readAllBytes(Paths.get(filename)); >>> cmdlineStartup = new String(encoded); >> >> >> What character set is the file encoded in? should the platform's default charset be used or UTF-8? Same applies to processing the load list files. I'll leave the final answer on Robert, but I'd expect the platform's default charset to be used - the file is provided by the user on the command line, so I'd assume it is in the platform's encoding. >> >> >> jdk/internal/jshell/tool/JShellTool.java:582 >>> int lastSlash = code.lastIndexOf('/'); //XXX >> >> Rogue comment. Will fix. >> >> >> jdk/internal/jshell/tool/JShellTool.java:587 >>> try (DirectoryStream dir = Files.newDirectoryStream(current)) { >>> for (Path e : dir) { >>> if (!accept.test(e) || !e.getFileName().toString().startsWith(prefix)) >>> continue; >>> result.add(new Suggestion(e.getFileName() + (Files.isDirectory(e) ? "/" : ""), false)); >>> } >>> } catch (IOException ex) { >>> //ignore? >>> } >> >> See files.lines(path) that returns a stream with map.filter.collect(toCollection(ArrayList::new)) Will fix. >> >> >>> if (path.isEmpty()) { >>> for (Path root : FileSystems.getDefault().getRootDirectories()) { >>> if (!accept.test(root) || !root.toString().startsWith(prefix)) >>> continue; >>> result.add(new Suggestion(root.toString(), false)); >>> } >>> } >> >> Alas we don?t have a FileSystem.rootDirectories method that returns a stream. > > The array is two small to worth a dedicated stream method, no ? > and one can use, Arrays.stream(...). I'll use StreamSupport.stream to get the Stream. > >> >> >> >> jdk/internal/jshell/tool/JShellTool.java:645 >>> registerCommand(new Command("/list", "/l", "[all]", "list the source you have typed", >>> arg -> cmdList(arg), >>> new FixedCompletionProvider("all"))); >> >> >> You could use a method reference for "arg -> ??, same applies for other command declarations. Not all of the command actions can be converted to method references currently (some of the methods don't need "arg"), so I'd slightly prefer to keep the code consistent and keep the lambdas there. But, if method refs would be strongly preferred, I can put them there. > > Also, it's a good idea to put the lambda as last argument so one can write, > > new Command (foo, bar, baz, arg -> { > ... > }) > The current order is: 1. the command name (and help/description) 2. what the command does 3. the completion for the command 4. (optional) extra attributes To me, this order seems reasonable (going from more important things to less important things). So, I'd personally keep it, but I can change the order if strongly preferred. >> >> >> >> jdk/internal/jshell/tool/JShellTool.java:716 >>> public List commandCompletionSuggestions(String code, int cursor, int[] anchor) { >>> code = code.substring(0, cursor); >>> int space = code.indexOf(' '); >>> List result = new ArrayList<>(); >>> >>> if (space == (-1)) { >>> for (Command cmd : new LinkedHashSet<>(commands.values())) { >>> if (cmd.kind == CommandKind.HIDDEN || cmd.kind == CommandKind.HELP_ONLY) >>> continue; >>> String key = cmd.aliases[0]; >>> if (key.startsWith(code)) { >>> result.add(new Suggestion(key + " ", false)); >>> } >>> } >>> anchor[0] = 0; >>> } else { >>> String arg = code.substring(space + 1); >>> String cmd = code.substring(0, space); >>> Command command = commands.get(cmd); >>> if (command != null) { >>> result.addAll(command.completions.completionSuggestions(arg, cursor - space, anchor)); >>> anchor[0] += space + 1; >>> } >>> } >>> >>> Collections.sort(result, (s1, s2) -> s1.continuation.compareTo(s2.continuation)); >>> return result; >>> } >> >> >> >> I believe you could use a stream here for both branches then do a sorted(?).collect(toList()) at the end of the method. For the first branch i think it might be something like values().distinct().filter(?).map(?).filter(?). For the second branch it is either an empty stream or the stream from the list. Will fix. >> >> FYI: you can also to List.sort now. >> >> [snip] >> jdk/internal/jshell/tool/JShellTool.java:1710 >>> public ScannerIOContext(InputStream inStream, PrintStream pStream) { >>> this(new Scanner(inStream), pStream); >>> } >> >> Unused constructor. Will fix. [snip] >> >>> if (!key.pollEvents().isEmpty()) { >>> if (!input.terminalEditorRunning()) { >>> saveFile(); >>> } >>> } >> >> Why does the watcher thread need to call saveFile when it is also performed after the process running the external editor has terminated? I am guessing every time the file is saved in the editor it is processed by the REPL? Yes, I think the intent is that when the (GUI) editor saves the file, it is immediately processed by REPL, so the user can see the outcomes without quiting the editor. Note this processing is disabled when the editor is detected to be a terminal editor (as processing the input while the terminal editor is running would make a mess of the screen). [snip] >> jdk/internal/jshell/tool/ConsoleIOContext.java:70 >>> TerminalFactory.registerFlavor(Flavor.WINDOWS, JShellWindowsTerminal :: new); >>> TerminalFactory.registerFlavor(Flavor.UNIX, JShellUnixTerminal :: new); >> >> The syntax for the constructor refs is a little unusual with the white space. Same further down when binding CRTL_UP/DOWN actions. Will fix. >> >> >> jdk/internal/jshell/tool/ConsoleIOContext.java:136 >>> .forEach(s -> { result.add(s); }); >> >> forEach(result::add); Will fix. >> >>> Optional prefix = >>> suggestions.stream() >>> .map(s -> s.continuation) >>> .reduce((s1, s2) -> commonPrefix(s1, s2)); >> >> .reduce(ConsoleIOContext::commonPrefix) Will fix. >> >> >> >>> try { >>> Method setBuffer = in.getClass().getDeclaredMethod("setBuffer", String.class); >>> >>> setBuffer.setAccessible(true); >>> setBuffer.invoke(in, in.getHistory().current().toString()); >>> in.flush(); >>> } catch (ReflectiveOperationException | IOException ex) { >>> throw new IllegalStateException(ex); >>> } >> >> You might wanna comment more about the limitation in ConsoleReader and the necessity of calling a private method. I'll add a comment there. >> >> >>> private static String commonPrefix(String str1, String str2) { >>> for (int i = 0; i < str2.length(); i++) { >>> if (!str1.startsWith(str2.substring(0, i + 1))) { >>> return str2.substring(0, i); >>> } >>> } >>> >>> return str2; >>> } >> >> What we need is a String mismatch method, from which to build a common prefix method, stay tuned?. Ok. >> >> >> jdk/internal/jshell/tool/ConsoleIOContext.java:505 >>> Set snippetsStart = new HashSet<>(); >>> for (String start : prefs.get(HISTORY_SNIPPET_START, "").split(";")) { >>> if (start.isEmpty()) >>> continue; >>> snippetsStart.add(Integer.parseInt(start)); >>> } >> >> Could streamify but perhaps not worth it. At least convert to >> >> if(!start.isEmpty()) >> snippetsStart.add(Integer.parseInt(start)); Will fix. >> >> >>> List keys = new ArrayList<>(Arrays.asList(prefs.keys())); >>> Collections.sort(keys); >> >> Stream.of(prefs.keys()).sorted().collect(toList()); Will fix. >> >> >> jdk/internal/jshell/tool/ConsoleIOContext.java: >> >> I am struggling to understand the interactions of the before/afterUse code blocks, the CircularBuffer, and the thread created for writing (or pushing) read values to the buffer. >> >> CircularBuffer feels like a blocking queue, but there is some additional communication signalling how much input is needed. This is a tricky code trying to resolve interaction between two of our "special" features: a) ability to stop/kill user's code by pressing Ctrl-C b) run terminal editor in /edit (as opposed to GUI editors, which don't need to use the terminal) For "stop", we need to read from the input to detect the Ctrl-C - in particular we need to read from the input while the user's code is running to detect the stop request. For terminal editors, we need to let the terminal editor work with the terminal as it wants - in particular, we must not read from the input (otherwise, not all of the user's input would reach the terminal editor). A complicating aspect is that (to my knowledge), we can't do a non-blocking read from System.in. So the solution (not saying there can't be a better one) is to only read from the input if we know we will process the input character: -during ConsoleReader.readLine only if/when the ConsoleReader actually requests to read from the input (because it will then process the character) - see the InputStream returned from ConsoleIOContext.InputHandler.wrapInIfNeeded. -when the user's code is running (i.e. inside the beforeUserCode-afterUserCode pair) - but, in this case, we only need to process the Ctrl-C character (i.e. (int) 3) - we need to do something with the rest, and the solution is to stash it into a buffer, so that it is not lost and can be processed in the future. This way we are not doing System.in.read() (or alike) while the terminal editor is running, so that it gets the input characters that the user types, but still can read as much as we need while the user's code is running. [snip] >> CommandCompletionTest.java:154 >>> private List listFiles(Path path, DirectoryStream.Filter filter) throws IOException { >>> List paths = new ArrayList<>(); >>> try (DirectoryStream stream = Files.newDirectoryStream(path, filter)) { >>> stream.iterator().forEachRemaining(p -> { >>> String fileName = p.getFileName().toString(); >>> if (Files.isDirectory(p)) { >>> fileName += "/"; >>> } >>> paths.add(fileName); >>> }); >>> } >>> Collections.sort(paths); >>> return paths; >>> } >> >> >> See Files.list which returns a Stream, you could probably reformulate using Predicate rather than DirectoryStream.Filter. Will fix. Thanks, Jan > > and try to do not use stream.iterator() which is usually slow, stream.forEach() is fine here. > >> >> >> >> StartOptionTest.java:68 >>> private void start(Consumer checkOutput, Consumer checkError, String...args) throws Exception { >> >> >> Space between ? and parameter name. Same goes for other cases in this file. >> > > cheers, > R?mi > > From paul.sandoz at oracle.com Mon Sep 7 12:59:46 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 7 Sep 2015 14:59:46 +0200 Subject: REPL tool implementation code review In-Reply-To: <55ED5E4C.3080700@oracle.com> References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> <55ED5E4C.3080700@oracle.com> Message-ID: On 7 Sep 2015, at 11:52, Jan Lahoda wrote: >>> jdk/internal/jshell/tool/JShellTool.java:307 >>>> byte[] encoded = Files.readAllBytes(Paths.get(filename)); >>>> cmdlineStartup = new String(encoded); >>> >>> >>> What character set is the file encoded in? should the platform's default charset be used or UTF-8? Same applies to processing the load list files. > > I'll leave the final answer on Robert, but I'd expect the platform's default charset to be used - the file is provided by the user on the command line, so I'd assume it is in the platform's encoding. > I would assume so too, but just wanted to double check. >>> >>> >>>> if (path.isEmpty()) { >>>> for (Path root : FileSystems.getDefault().getRootDirectories()) { >>>> if (!accept.test(root) || !root.toString().startsWith(prefix)) >>>> continue; >>>> result.add(new Suggestion(root.toString(), false)); >>>> } >>>> } >>> >>> Alas we don?t have a FileSystem.rootDirectories method that returns a stream. >> >> The array is two small to worth a dedicated stream method, no ? >> and one can use, Arrays.stream(...). > > I'll use StreamSupport.stream to get the Stream. > Ah, i recall now it returns an Iterable. I dunno if it is worth it, at least transform to if (X && Y) and remove the ?continue. >> >>> >>> >>> >>> jdk/internal/jshell/tool/JShellTool.java:645 >>>> registerCommand(new Command("/list", "/l", "[all]", "list the source you have typed", >>>> arg -> cmdList(arg), >>>> new FixedCompletionProvider("all"))); >>> >>> >>> You could use a method reference for "arg -> ??, same applies for other command declarations. > > Not all of the command actions can be converted to method references currently (some of the methods don't need "arg"), so I'd slightly prefer to keep the code consistent and keep the lambdas there. But, if method refs would be strongly preferred, I can put them there. > Up to you, consistency is a good reason to leave as is. > >>> >>>> if (!key.pollEvents().isEmpty()) { >>>> if (!input.terminalEditorRunning()) { >>>> saveFile(); >>>> } >>>> } >>> >>> Why does the watcher thread need to call saveFile when it is also performed after the process running the external editor has terminated? I am guessing every time the file is saved in the editor it is processed by the REPL? > > Yes, I think the intent is that when the (GUI) editor saves the file, it is immediately processed by REPL, so the user can see the outcomes without quiting the editor. > > Note this processing is disabled when the editor is detected to be a terminal editor (as processing the input while the terminal editor is running would make a mess of the screen). > Ok. >>> >>> >>> jdk/internal/jshell/tool/ConsoleIOContext.java: >>> >>> I am struggling to understand the interactions of the before/afterUse code blocks, the CircularBuffer, and the thread created for writing (or pushing) read values to the buffer. >>> >>> CircularBuffer feels like a blocking queue, but there is some additional communication signalling how much input is needed. > It would be useful to clear up all the access levels of ConsoleIOContext and the inner class as that would help in terms of knowing who can do what e.g. make some/all inner classes of ConsoleIOContext private and then make ConsoleIOContext package private, since it?s constructor is. Perhaps some unit tests require certain access, mainly related to EditingHistory? if so could that be made a top-level class with a comment as to why it is public? You might need to make InputHandler.reading volatile as it is accessed by two different threads (the thread reading InputHandler.reading may never observer the write in another thread). > This is a tricky code trying to resolve interaction between two of our "special" features: Indeed. Have i got the following correct? 1) reading a line, processing user code, and terminal editing are three mutually exclusive states 2) the underlying input.read may block since a request to read is not an indication of there being anything available to be read 3) when processing user code the waiting until requested input is effectively disabled, thus any characters are buffered asynchronously ready for processing when the next line is requested. 4) when processing user code if a CTRL-C is observed then the REPL is asynchronously stopped. I wonder if it is possible to replace InputHandler.reading and CircularBuffer.inputNeeded with an enum representing those three states. enum { // The state of waiting for a request to read from the underlying input WAIT, // The state of reading from the underlying input stream and propagating READ_AND_PROPAGATE, // The state of reading from the underlying input stream and buffering READ_AND_BUFFER; } That assumes you can do something like the following in InputHandle: public String readLine(String prompt) throws IOException { inputBuffer.s = BufferState.READ_AND_PROPAGATE; try { return consoleReader.readLine(prompt); } finally { inputBuffer.s = BufferState.WAIT; } } public void beforeUserCode() { inputBuffer.s = BufferState.BUFFER; } public void afterUserCode() { inputBuffer.s = BufferState.WAIT; } And on CircularBuffer: public synchronized void waitInputNeeded() { while (s == BufferState.WAIT) { try { wait(); } catch (InterruptedException ex) { //ignore } } } Then you can map those states via some comments to the REPL operations. I hacked together something real quick along those lines and it seemed to work. Is the asynchronous operation on the REPL in 4 an issue? My inclination is you should try and merge the thread logic in InputBuffer.wrapInIfNeeded into CircularBuffer, then modify and rename CircularBuffer to extend from InputStream, thereby containing all the relevant multi-threaded logic and it becomes an obvious thing being deferred to. You could also add another state CLOSED, which i think more clearly indicates closure rather than continually writing -1 every time you read -1. Sorry for labouring the point here, but i think this kind of thing is worth spending some time on. Consider someone trying to modify the code in a few years time. Review-wise we could spin this off as a separate bug so as not to block the code going in. Paul. > a) ability to stop/kill user's code by pressing Ctrl-C > b) run terminal editor in /edit (as opposed to GUI editors, which don't need to use the terminal) > > For "stop", we need to read from the input to detect the Ctrl-C - in particular we need to read from the input while the user's code is running to detect the stop request. > > For terminal editors, we need to let the terminal editor work with the terminal as it wants - in particular, we must not read from the input (otherwise, not all of the user's input would reach the terminal editor). > > A complicating aspect is that (to my knowledge), we can't do a non-blocking read from System.in. > > So the solution (not saying there can't be a better one) is to only read from the input if we know we will process the input character: > -during ConsoleReader.readLine only if/when the ConsoleReader actually requests to read from the input (because it will then process the character) - see the InputStream returned from ConsoleIOContext.InputHandler.wrapInIfNeeded. > > -when the user's code is running (i.e. inside the beforeUserCode-afterUserCode pair) - but, in this case, we only need to process the Ctrl-C character (i.e. (int) 3) - we need to do something with the rest, and the solution is to stash it into a buffer, so that it is not lost and can be processed in the future. > > This way we are not doing System.in.read() (or alike) while the terminal editor is running, so that it gets the input characters that the user types, but still can read as much as we need while the user's code is running. > From jan.lahoda at oracle.com Mon Sep 7 15:07:27 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Mon, 07 Sep 2015 15:07:27 +0000 Subject: hg: kulla/dev/langtools: Reflecting some of the review comments by Paul and Remi. Message-ID: <201509071507.t87F7RF7006661@aojmv0008.oracle.com> Changeset: 45e459d45f64 Author: jlahoda Date: 2015-09-07 17:06 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/45e459d45f64 Reflecting some of the review comments by Paul and Remi. ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/ConsoleIOContext.java ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/JShellTool.java ! test/jdk/jshell/CommandCompletionTest.java From jan.lahoda at oracle.com Mon Sep 7 15:31:22 2015 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Mon, 07 Sep 2015 17:31:22 +0200 Subject: REPL tool implementation code review In-Reply-To: References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> <55ED5E4C.3080700@oracle.com> Message-ID: <55EDADCA.8020109@oracle.com> I've tried to resolve some of the comments from the original e-mail here: http://hg.openjdk.java.net/kulla/dev/langtools/rev/45e459d45f64 On 7.9.2015 14:59, Paul Sandoz wrote: > On 7 Sep 2015, at 11:52, Jan Lahoda wrote: [snip] >>>> jdk/internal/jshell/tool/ConsoleIOContext.java: >>>> >>>> I am struggling to understand the interactions of the before/afterUse code blocks, the CircularBuffer, and the thread created for writing (or pushing) read values to the buffer. >>>> >>>> CircularBuffer feels like a blocking queue, but there is some additional communication signalling how much input is needed. >> > > It would be useful to clear up all the access levels of ConsoleIOContext and the inner class as that would help in terms of knowing who can do what e.g. make some/all inner classes of ConsoleIOContext private and then make ConsoleIOContext package private, since it?s constructor is. Perhaps some unit tests require certain access, mainly related to EditingHistory? if so could that be made a top-level class with a comment as to why it is public? Ok, I'll look into that. > > You might need to make InputHandler.reading volatile as it is accessed by two different threads (the thread reading InputHandler.reading may never observer the write in another thread). Right, it should be volatile. > > >> This is a tricky code trying to resolve interaction between two of our "special" features: > > Indeed. > > Have i got the following correct? > > 1) reading a line, processing user code, and terminal editing are three mutually exclusive states Yes (just processing user code is more "running user's code in the main thread" - some processing of the user's code by JShell may happen even while reading the input, for e.g. the tab completion, but we don't support stopping that). > > 2) the underlying input.read may block since a request to read is not an indication of there being anything available to be read > > 3) when processing user code the waiting until requested input is effectively disabled, thus any characters are buffered asynchronously ready for processing when the next line is requested. > > 4) when processing user code if a CTRL-C is observed then the REPL is asynchronously stopped. Yes. > > I wonder if it is possible to replace InputHandler.reading and CircularBuffer.inputNeeded with an enum representing those three states. > > enum { > // The state of waiting for a request to read from the underlying input > WAIT, > // The state of reading from the underlying input stream and propagating > READ_AND_PROPAGATE, > // The state of reading from the underlying input stream and buffering > READ_AND_BUFFER; > } > > That assumes you can do something like the following in InputHandle: > > public String readLine(String prompt) throws IOException { > inputBuffer.s = BufferState.READ_AND_PROPAGATE; > try { > return consoleReader.readLine(prompt); > } finally { > inputBuffer.s = BufferState.WAIT; > } > } > > public void beforeUserCode() { > inputBuffer.s = BufferState.BUFFER; > } > > public void afterUserCode() { > inputBuffer.s = BufferState.WAIT; > } > > And on CircularBuffer: > > public synchronized void waitInputNeeded() { > while (s == BufferState.WAIT) { > try { > wait(); > } catch (InterruptedException ex) { > //ignore > } > } > } > > Then you can map those states via some comments to the REPL operations. I hacked together something real quick along those lines and it seemed to work. > > Is the asynchronous operation on the REPL in 4 an issue? The asynchronous operation is also somewhat tricky (but necessary, I think). I have another patch in my queue that reflects (some of) Brian comments on that. > > My inclination is you should try and merge the thread logic in InputBuffer.wrapInIfNeeded into CircularBuffer, then modify and rename CircularBuffer to extend from InputStream, thereby containing all the relevant multi-threaded logic and it becomes an obvious thing being deferred to. You could also add another state CLOSED, which i think more clearly indicates closure rather than continually writing -1 every time you read -1. > > Sorry for labouring the point here, but i think this kind of thing is worth spending some time on. Consider someone trying to modify the code in a few years time. Review-wise we could spin this off as a separate bug so as not to block the code going in. No problem. I'll look into doing the changes as you describe. Thanks for the comments! Jan > > Paul. > > >> a) ability to stop/kill user's code by pressing Ctrl-C >> b) run terminal editor in /edit (as opposed to GUI editors, which don't need to use the terminal) >> >> For "stop", we need to read from the input to detect the Ctrl-C - in particular we need to read from the input while the user's code is running to detect the stop request. >> >> For terminal editors, we need to let the terminal editor work with the terminal as it wants - in particular, we must not read from the input (otherwise, not all of the user's input would reach the terminal editor). >> >> A complicating aspect is that (to my knowledge), we can't do a non-blocking read from System.in. >> >> So the solution (not saying there can't be a better one) is to only read from the input if we know we will process the input character: >> -during ConsoleReader.readLine only if/when the ConsoleReader actually requests to read from the input (because it will then process the character) - see the InputStream returned from ConsoleIOContext.InputHandler.wrapInIfNeeded. >> >> -when the user's code is running (i.e. inside the beforeUserCode-afterUserCode pair) - but, in this case, we only need to process the Ctrl-C character (i.e. (int) 3) - we need to do something with the rest, and the solution is to stash it into a buffer, so that it is not lost and can be processed in the future. >> >> This way we are not doing System.in.read() (or alike) while the terminal editor is running, so that it gets the input characters that the user types, but still can read as much as we need while the user's code is running. >> > > > From paul.sandoz at oracle.com Mon Sep 7 16:26:12 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 7 Sep 2015 18:26:12 +0200 Subject: REPL tool implementation code review In-Reply-To: <55EDADCA.8020109@oracle.com> References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> <55ED5E4C.3080700@oracle.com> <55EDADCA.8020109@oracle.com> Message-ID: <8FA1BE5E-5E66-4223-9658-CD1EF0CD7C6C@oracle.com> On 7 Sep 2015, at 17:31, Jan Lahoda wrote: > I've tried to resolve some of the comments from the original e-mail here: > http://hg.openjdk.java.net/kulla/dev/langtools/rev/45e459d45f64 > Looks good, it?s ok to do: forEach(results::add) rather than create additional collections if you know what you are doing. > On 7.9.2015 14:59, Paul Sandoz wrote: >> On 7 Sep 2015, at 11:52, Jan Lahoda wrote: >> >> Is the asynchronous operation on the REPL in 4 an issue? > > The asynchronous operation is also somewhat tricky (but necessary, I think). I have another patch in my queue that reflects (some of) Brian comments on that. > Ok, i was more questioning whether it is safe rather than necessary to perform on the reading thread, since i could not determine if stopping the REPL concurrently in the reader thread would interfere with executing user code performed in the main thread. >> >> My inclination is you should try and merge the thread logic in InputBuffer.wrapInIfNeeded into CircularBuffer, then modify and rename CircularBuffer to extend from InputStream, thereby containing all the relevant multi-threaded logic and it becomes an obvious thing being deferred to. You could also add another state CLOSED, which i think more clearly indicates closure rather than continually writing -1 every time you read -1. >> >> Sorry for labouring the point here, but i think this kind of thing is worth spending some time on. Consider someone trying to modify the code in a few years time. Review-wise we could spin this off as a separate bug so as not to block the code going in. > > No problem. I'll look into doing the changes as you describe. Thanks for the comments! > Thanks, Paul. From robert.field at oracle.com Tue Sep 8 16:39:11 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Tue, 08 Sep 2015 16:39:11 +0000 Subject: hg: kulla/dev/langtools: Per review, remove unneeded argument null checks. Message-ID: <201509081639.t88GdGqq017738@aojmv0008.oracle.com> Changeset: 8525c8582cd6 Author: rfield Date: 2015-09-08 09:39 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/8525c8582cd6 Per review, remove unneeded argument null checks. ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/JShellTool.java From robert.field at oracle.com Tue Sep 8 17:51:18 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Tue, 08 Sep 2015 17:51:18 +0000 Subject: hg: kulla/dev/langtools: Per JShellTool review, use Files.newBufferedWriter. Also, related, change resolveUserHome to return a Path 'toPathResolvingUserHome' Message-ID: <201509081751.t88HpIUA002730@aojmv0008.oracle.com> Changeset: f2b74c96cb01 Author: rfield Date: 2015-09-08 10:51 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/f2b74c96cb01 Per JShellTool review, use Files.newBufferedWriter. Also, related, change resolveUserHome to return a Path 'toPathResolvingUserHome' ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/JShellTool.java From robert.field at oracle.com Tue Sep 8 18:46:23 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Tue, 08 Sep 2015 18:46:23 +0000 Subject: hg: kulla/dev/langtools: Per JShellTool review, streamify errorsOnly(). Thanks Paul. Message-ID: <201509081846.t88IkNAM013988@aojmv0008.oracle.com> Changeset: 65e671a437d0 Author: rfield Date: 2015-09-08 11:46 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/65e671a437d0 Per JShellTool review, streamify errorsOnly(). Thanks Paul. ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/JShellTool.java From robert.field at oracle.com Tue Sep 8 20:18:47 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Tue, 08 Sep 2015 20:18:47 +0000 Subject: hg: kulla/dev/langtools: Per JShellTool review, streamify saveFile(). Thanks Remi. Message-ID: <201509082018.t88KIl1T003902@aojmv0008.oracle.com> Changeset: f2e1744c62d0 Author: rfield Date: 2015-09-08 13:18 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/f2e1744c62d0 Per JShellTool review, streamify saveFile(). Thanks Remi. ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/ExternalEditor.java From robert.field at oracle.com Tue Sep 8 21:40:46 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Tue, 08 Sep 2015 21:40:46 +0000 Subject: hg: kulla/dev/langtools: Per JShellTool review, switch EditPad's closeLock to CountDownLatch. Message-ID: <201509082140.t88LeklN021032@aojmv0008.oracle.com> Changeset: 5906e079b087 Author: rfield Date: 2015-09-08 14:40 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/5906e079b087 Per JShellTool review, switch EditPad's closeLock to CountDownLatch. ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/EditPad.java ! test/jdk/jshell/EditorPadTest.java From robert.field at oracle.com Wed Sep 9 01:32:33 2015 From: robert.field at oracle.com (Robert Field) Date: Tue, 08 Sep 2015 18:32:33 -0700 Subject: REPL tool implementation code review In-Reply-To: References: Message-ID: <55EF8C31.2040209@oracle.com> On 09/04/15 10:19, Paul Sandoz wrote: > Hi, > > Here is the review of the REPL tool implementation. > > Generally it all looks very good. I ran out of time to analyse the tests more closely. Quite a few comments are suggestions related to using newer APIs or techniques. > > Paul. Thanks again, Paul for the detailed review. I've responded inline or fixed the issue for all the items in my areas except the first two. > > Implementation: > > > jdk/internal/jshell/tool/JShellTool.java:91 >> private static final String VERSION = "0.819?; >> ? >> case "-version": >> cmdout.printf("jshell %s\n", VERSION); >> return null; >> > Should this version be the JDK version, as is the case for other Java tools? Yes. Working on this. > > > jdk/internal/jshell/tool/JShellTool.java:128 >> private IOContext input = null; >> private boolean regenerateOnDeath = true; >> private boolean live = false; >> >> SourceCodeAnalysis analysis; >> JShell state = null; >> Subscription shutdownSubscription = null; >> >> private boolean debug = false; >> private boolean displayPrompt = true; >> public boolean testPrompt = false; >> private Feedback feedback = Feedback.Default; >> private String cmdlineClasspath = null; >> private String cmdlineStartup = null; >> private String editor = null; >> > A small style thing, there is no need to initialize to default values (it?s also applied inconsistently) See separate email. > > jdk/internal/jshell/tool/JShellTool.java:307 >> byte[] encoded = Files.readAllBytes(Paths.get(filename)); >> cmdlineStartup = new String(encoded); > > What character set is the file encoded in? should the platform's default charset be used or UTF-8? Same applies to processing the load list files. Should be default charset. > > > jdk/internal/jshell/tool/JShellTool.java:582 >> int lastSlash = code.lastIndexOf('/'); //XXX > Rogue comment. Jan fixed. > > > jdk/internal/jshell/tool/JShellTool.java:587 >> try (DirectoryStream dir = Files.newDirectoryStream(current)) { >> for (Path e : dir) { >> if (!accept.test(e) || !e.getFileName().toString().startsWith(prefix)) >> continue; >> result.add(new Suggestion(e.getFileName() + (Files.isDirectory(e) ? "/" : ""), false)); >> } >> } catch (IOException ex) { >> //ignore? >> } > See files.lines(path) that returns a stream with map.filter.collect(toCollection(ArrayList::new)) Jan has updated. > > >> if (path.isEmpty()) { >> for (Path root : FileSystems.getDefault().getRootDirectories()) { >> if (!accept.test(root) || !root.toString().startsWith(prefix)) >> continue; >> result.add(new Suggestion(root.toString(), false)); >> } >> } > Alas we don?t have a FileSystem.rootDirectories method that returns a stream. Jan has updated. > > > jdk/internal/jshell/tool/JShellTool.java:645 >> registerCommand(new Command("/list", "/l", "[all]", "list the source you have typed", >> arg -> cmdList(arg), >> new FixedCompletionProvider("all"))); > > You could use a method reference for "arg -> ??, same applies for other command declarations. Half the commands take no args, consistency might be more important than using method reference (in half the cases). > > > jdk/internal/jshell/tool/JShellTool.java:716 >> public List commandCompletionSuggestions(String code, int cursor, int[] anchor) { >> code = code.substring(0, cursor); >> int space = code.indexOf(' '); >> List result = new ArrayList<>(); >> >> if (space == (-1)) { >> for (Command cmd : new LinkedHashSet<>(commands.values())) { >> if (cmd.kind == CommandKind.HIDDEN || cmd.kind == CommandKind.HELP_ONLY) >> continue; >> String key = cmd.aliases[0]; >> if (key.startsWith(code)) { >> result.add(new Suggestion(key + " ", false)); >> } >> } >> anchor[0] = 0; >> } else { >> String arg = code.substring(space + 1); >> String cmd = code.substring(0, space); >> Command command = commands.get(cmd); >> if (command != null) { >> result.addAll(command.completions.completionSuggestions(arg, cursor - space, anchor)); >> anchor[0] += space + 1; >> } >> } >> >> Collections.sort(result, (s1, s2) -> s1.continuation.compareTo(s2.continuation)); >> return result; >> } > > > I believe you could use a stream here for both branches then do a sorted(?).collect(toList()) at the end of the method. For the first branch i think it might be something like values().distinct().filter(?).map(?).filter(?). For the second branch it is either an empty stream or the stream from the list. > > FYI: you can also to List.sort now. Jan has updated. > > > jdk/internal/jshell/tool/JShellTool.java:763 >> if (arg == null || arg.isEmpty()) { > I dunno if it is possible but it would be nice to normalise to empty strings rather than checking for null or an empty string. It cannot be, currently, be null. This was defensive programming. But it was applied inconsistently. Removed. > > > jdk/internal/jshell/tool/JShellTool.java:959 >> Iterator it = keySet.iterator(); >> while (it.hasNext()) { >> if (!(it.next() instanceof PersistentSnippet)) { >> it.remove(); >> } >> } > See the bulk operation Collection.removeIf. Used, also fixed naming. > > >> PersistentSnippet key = (PersistentSnippet) keySet.iterator().next(); >> state.drop(key).forEach(e -> handleEvent(e)); > > For kicks you could also do it like this (not suggesting you actually do unless you really want to, it?s just instructive for new APIs in Java 8): > > keySet.stream().findFirst(). > map(PersistentSnippet.class::cast). > ifPresent(k -> state.drop(k).forEach(this::handleEvent)); > Well, I did change to using a method reference and fixed naming. > jdk/internal/jshell/tool/JShellTool.java:981 >> private void cmdEdit(String arg) { >> Set keySet = argToKeySet(arg); >> if (keySet == null) { >> return; >> } >> Set srcSet = new LinkedHashSet<>(); >> for (Snippet key : keySet) { >> String src = key.source(); >> switch (key.subKind()) { >> case VAR_VALUE_SUBKIND: >> break; >> case ASSIGNMENT_SUBKIND: >> case OTHER_EXPRESSION_SUBKIND: >> case TEMP_VAR_EXPRESSION_SUBKIND: >> if (!src.endsWith(";")) { >> src = src + ";"; >> } >> srcSet.add(src); >> break; >> default: >> srcSet.add(src); >> break; >> } >> } >> StringBuilder sb = new StringBuilder(); >> for (String s : srcSet) { >> sb.append(s); >> sb.append('\n'); >> } >> String src = sb.toString(); > > There appears to be a reliance on an undefined iteration order, perhaps argToKeySet should return LinkedHashSet? It does use LinkedHashSet. > > I think this might be convertible to a stream with filter(?).map(?).collect(joining(?\n?, ??, ?\n")) > > > jdk/internal/jshell/tool/JShellTool.java:1150 >> try (BufferedWriter writer = new BufferedWriter(new FileWriter(resolveUserHome(filename)))) { > See also Files.newBufferedWriter(?) i like that because it makes it more explicit whether you are appending to an existing file or not. Done, used explicit list of StandardOpenOption to make write options explicit and did related clean-up using Path instead of String. > > > jdk/internal/jshell/tool/JShellTool.java:1266 >> List> errorsOnly(List> diagnostics) { >> List> errors = new ArrayList<>(); >> for (Diagnostic diag : diagnostics) { >> switch (diag.getKind()) { >> case ERROR: >> errors.add(diag); >> break; >> case MANDATORY_WARNING: >> case WARNING: >> break; >> case NOTE: >> case OTHER: >> break; >> } >> } >> return errors; >> } > diagnostics.stream().filter(d -> d.getKind() == ERROR).collect(toList()); Cool! Done. > > > jdk/internal/jshell/tool/JShellTool.java:1710 >> public ScannerIOContext(InputStream inStream, PrintStream pStream) { >> this(new Scanner(inStream), pStream); >> } > Unused constructor. I'll leave for Jan. > > > jdk/internal/jshell/tool/ExternalEditor.java:134 >> private void saveFile() { >> List lines; >> try { >> lines = Files.readAllLines(tmpfile); >> } catch (IOException ex) { >> errorHandler.accept("Failure read edit file: " + ex.getMessage()); >> return ; >> } >> StringBuilder sb = new StringBuilder(); >> for (String ln : lines) { >> sb.append(ln); >> sb.append('\n'); >> } >> saveHandler.accept(sb.toString()); >> } > Why don?t you just read as bytes then create a String from those bytes? I want the normalization and termination. Used Remi's suggestion, making it: private void saveFile() { try { saveHandler.accept(Files.lines(tmpfile).collect(Collectors.joining("\n", "", "\n"))); } catch (IOException ex) { errorHandler.accept("Failure in read edit file: " + ex.getMessage()); } } > >> if (!key.pollEvents().isEmpty()) { >> if (!input.terminalEditorRunning()) { >> saveFile(); >> } >> } > Why does the watcher thread need to call saveFile when it is also performed after the process running the external editor has terminated? I am guessing every time the file is saved in the editor it is processed by the REPL? Exactly! > > > jdk/internal/jshell/tool/EditPad.java:46 >> public class EditPad extends JFrame implements Runnable { >> private final Consumer errorHandler; >> private final String initialText; >> private final boolean[] closeLock; >> private final Consumer saveHandler; >> >> EditPad(Consumer errorHandler, String initialText, >> boolean[] closeLock, Consumer saveHandler) { >> super("Edit Pad (Experimental)"); >> this.errorHandler = errorHandler; >> this.initialText = initialText; >> this.closeLock = closeLock; >> this.saveHandler = saveHandler; >> } > > Unused field errorHandle. I'd like to leave it for possible future use and symmetry with ExternalEditor. > > Is this still experimental? Is so should be it included? I think it is a useful default., removed "Experimental" > > > jdk/internal/jshell/tool/EditPad.java:114 >> private void notifyClose() { >> synchronized (closeLock) { >> closeLock[0] = true; >> closeLock.notify(); >> } >> } >> >> public static void edit(Consumer errorHandler, String initialText, >> Consumer saveHandler) { >> boolean[] closeLock = new boolean[1]; >> SwingUtilities.invokeLater( >> new EditPad(errorHandler, initialText, closeLock, saveHandler)); >> synchronized (closeLock) { >> while (!closeLock[0]) { >> try { >> closeLock.wait(); >> } catch (InterruptedException ex) { >> // ignore and loop >> } >> } >> } >> } > You could use a shared CountDownLatch, rather than notify/wait/synchronization with a boolean array of one element. Clean! I rewrote as this (not sure about interrupt handling): public static void edit(Consumer errorHandler, String initialText, Consumer saveHandler) { CountDownLatch closeLock = new CountDownLatch(1); SwingUtilities.invokeLater( new EditPad(errorHandler, initialText, closeLock, saveHandler)); do { try { closeLock.await(); break; } catch (InterruptedException ex) { // ignore and loop } } while (true); } The rest are Jan's and Andrei's... -Robert From paul.sandoz at oracle.com Wed Sep 9 08:16:49 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 9 Sep 2015 10:16:49 +0200 Subject: REPL tool implementation code review In-Reply-To: <55EF8C31.2040209@oracle.com> References: <55EF8C31.2040209@oracle.com> Message-ID: <221D0FF5-9BEB-411C-B822-9B6E41C710D9@oracle.com> Hi Robert, The changesets look good. On 9 Sep 2015, at 03:32, Robert Field wrote: >> jdk/internal/jshell/tool/JShellTool.java:981 >>> private void cmdEdit(String arg) { >>> Set keySet = argToKeySet(arg); >>> if (keySet == null) { >>> return; >>> } >>> Set srcSet = new LinkedHashSet<>(); >>> for (Snippet key : keySet) { >>> String src = key.source(); >>> switch (key.subKind()) { >>> case VAR_VALUE_SUBKIND: >>> break; >>> case ASSIGNMENT_SUBKIND: >>> case OTHER_EXPRESSION_SUBKIND: >>> case TEMP_VAR_EXPRESSION_SUBKIND: >>> if (!src.endsWith(";")) { >>> src = src + ";"; >>> } >>> srcSet.add(src); >>> break; >>> default: >>> srcSet.add(src); >>> break; >>> } >>> } >>> StringBuilder sb = new StringBuilder(); >>> for (String s : srcSet) { >>> sb.append(s); >>> sb.append('\n'); >>> } >>> String src = sb.toString(); >> >> There appears to be a reliance on an undefined iteration order, perhaps argToKeySet should return LinkedHashSet? > > It does use LinkedHashSet. > But the argToKeySet() method returns a Set, which may or may not have a defined encounter order, it?s not clear and would be more so if argToKeySet returned LinkedHashSet. >> >> >> jdk/internal/jshell/tool/ExternalEditor.java:134 >>> private void saveFile() { >>> List lines; >>> try { >>> lines = Files.readAllLines(tmpfile); >>> } catch (IOException ex) { >>> errorHandler.accept("Failure read edit file: " + ex.getMessage()); >>> return ; >>> } >>> StringBuilder sb = new StringBuilder(); >>> for (String ln : lines) { >>> sb.append(ln); >>> sb.append('\n'); >>> } >>> saveHandler.accept(sb.toString()); >>> } >> Why don?t you just read as bytes then create a String from those bytes? > > I want the normalization and termination. Used Remi's suggestion, making it: > > private void saveFile() { > try { > saveHandler.accept(Files.lines(tmpfile).collect(Collectors.joining("\n", "", "\n"))); > } catch (IOException ex) { > errorHandler.accept("Failure in read edit file: " + ex.getMessage()); > } > } > Ok. I cannot recall if there was or you added a comment regarding normalization and termination, if not it might be useful to do so, in case someone comes along later on and inadvertently changes it. Paul. From robert.field at oracle.com Wed Sep 9 16:05:06 2015 From: robert.field at oracle.com (Robert Field) Date: Wed, 09 Sep 2015 09:05:06 -0700 Subject: REPL tool implementation code review In-Reply-To: <221D0FF5-9BEB-411C-B822-9B6E41C710D9@oracle.com> References: <55EF8C31.2040209@oracle.com> <221D0FF5-9BEB-411C-B822-9B6E41C710D9@oracle.com> Message-ID: <55F058B2.3040901@oracle.com> On 09/09/15 01:16, Paul Sandoz wrote: > Hi Robert, > > The changesets look good. Cool! > > > On 9 Sep 2015, at 03:32, Robert Field wrote: >>> jdk/internal/jshell/tool/JShellTool.java:981 >>>> private void cmdEdit(String arg) { >>>> Set keySet = argToKeySet(arg); >>>> if (keySet == null) { >>>> return; >>>> } >>>> Set srcSet = new LinkedHashSet<>(); >>>> for (Snippet key : keySet) { >>>> String src = key.source(); >>>> switch (key.subKind()) { >>>> case VAR_VALUE_SUBKIND: >>>> break; >>>> case ASSIGNMENT_SUBKIND: >>>> case OTHER_EXPRESSION_SUBKIND: >>>> case TEMP_VAR_EXPRESSION_SUBKIND: >>>> if (!src.endsWith(";")) { >>>> src = src + ";"; >>>> } >>>> srcSet.add(src); >>>> break; >>>> default: >>>> srcSet.add(src); >>>> break; >>>> } >>>> } >>>> StringBuilder sb = new StringBuilder(); >>>> for (String s : srcSet) { >>>> sb.append(s); >>>> sb.append('\n'); >>>> } >>>> String src = sb.toString(); >>> There appears to be a reliance on an undefined iteration order, perhaps argToKeySet should return LinkedHashSet? >> It does use LinkedHashSet. >> > But the argToKeySet() method returns a Set, which may or may not have a defined encounter order, it?s not clear and would be more so if argToKeySet returned LinkedHashSet. So, you are saying, for documentation purposes, I should declare it (now called argToSnippetSet) as returning LinkedHashSet, and the sets within the method (keySet now snippetSet and srcSet) as LinkedHashSet too? I thought the standard was to declare the interface and that one shouldn't declare them as their implementation type. > > >>> >>> jdk/internal/jshell/tool/ExternalEditor.java:134 >>>> private void saveFile() { >>>> List lines; >>>> try { >>>> lines = Files.readAllLines(tmpfile); >>>> } catch (IOException ex) { >>>> errorHandler.accept("Failure read edit file: " + ex.getMessage()); >>>> return ; >>>> } >>>> StringBuilder sb = new StringBuilder(); >>>> for (String ln : lines) { >>>> sb.append(ln); >>>> sb.append('\n'); >>>> } >>>> saveHandler.accept(sb.toString()); >>>> } >>> Why don?t you just read as bytes then create a String from those bytes? >> I want the normalization and termination. Used Remi's suggestion, making it: >> >> private void saveFile() { >> try { >> saveHandler.accept(Files.lines(tmpfile).collect(Collectors.joining("\n", "", "\n"))); >> } catch (IOException ex) { >> errorHandler.accept("Failure in read edit file: " + ex.getMessage()); >> } >> } >> > Ok. I cannot recall if there was or you added a comment regarding normalization and termination, if not it might be useful to do so, in case someone comes along later on and inadvertently changes it. OK, will do. Thanks, Robert > > Paul. From paul.sandoz at oracle.com Thu Sep 10 08:12:02 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 10 Sep 2015 10:12:02 +0200 Subject: REPL tool implementation code review In-Reply-To: <55F058B2.3040901@oracle.com> References: <55EF8C31.2040209@oracle.com> <221D0FF5-9BEB-411C-B822-9B6E41C710D9@oracle.com> <55F058B2.3040901@oracle.com> Message-ID: <5EAE1185-8C4A-42D7-B52D-E9A6676E904A@oracle.com> On 9 Sep 2015, at 18:05, Robert Field wrote: >> On 9 Sep 2015, at 03:32, Robert Field wrote: >>>> jdk/internal/jshell/tool/JShellTool.java:981 >>>>> private void cmdEdit(String arg) { >>>>> Set keySet = argToKeySet(arg); >>>>> if (keySet == null) { >>>>> return; >>>>> } >>>>> Set srcSet = new LinkedHashSet<>(); >>>>> for (Snippet key : keySet) { >>>>> String src = key.source(); >>>>> switch (key.subKind()) { >>>>> case VAR_VALUE_SUBKIND: >>>>> break; >>>>> case ASSIGNMENT_SUBKIND: >>>>> case OTHER_EXPRESSION_SUBKIND: >>>>> case TEMP_VAR_EXPRESSION_SUBKIND: >>>>> if (!src.endsWith(";")) { >>>>> src = src + ";"; >>>>> } >>>>> srcSet.add(src); >>>>> break; >>>>> default: >>>>> srcSet.add(src); >>>>> break; >>>>> } >>>>> } >>>>> StringBuilder sb = new StringBuilder(); >>>>> for (String s : srcSet) { >>>>> sb.append(s); >>>>> sb.append('\n'); >>>>> } >>>>> String src = sb.toString(); >>>> There appears to be a reliance on an undefined iteration order, perhaps argToKeySet should return LinkedHashSet? >>> It does use LinkedHashSet. >>> >> But the argToKeySet() method returns a Set, which may or may not have a defined encounter order, it?s not clear and would be more so if argToKeySet returned LinkedHashSet. > > So, you are saying, for documentation purposes, I should declare it (now called argToSnippetSet) as returning LinkedHashSet, and the sets within the method (keySet now snippetSet and srcSet) as LinkedHashSet too? I thought the standard was to declare the interface and that one shouldn't declare them as their implementation type. > When i looked at the cmdEdit method i could see it depends on an encounter order [*] of the Set returned from argToSnippetSet, but i don?t immediately know if the returned Set has an encounter order, it?s not defined in the method signature, i can look into the implementation and see it?s using LinkedHashSet and i know that state.snippets() returns a List so all is good, but what if some future developer inadvertently decides to change the argToSnippetSet implementation to use a HashSet instead of a LinkedHashSet? It?s a subtle and smallish thing, but i think useful to capture the intent, as such encounter order bugs can be hard to track down. Paul. [*] i.e. guarantees concerning the order in which the elements are reported (or encountered) when the collection is traversed. From jan.lahoda at oracle.com Thu Sep 10 08:43:37 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Thu, 10 Sep 2015 08:43:37 +0000 Subject: hg: kulla/dev/langtools: Review feedback - moving input-related threading into CircularBuffer, renaming it to StopDetectingInput, more cleanup around ConsoleIOContext. Message-ID: <201509100843.t8A8hbWB025535@aojmv0008.oracle.com> Changeset: 4141dc296d4d Author: jlahoda Date: 2015-09-10 10:43 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/4141dc296d4d Review feedback - moving input-related threading into CircularBuffer, renaming it to StopDetectingInput, more cleanup around ConsoleIOContext. ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/ConsoleIOContext.java + src/jdk.jshell/share/classes/jdk/internal/jshell/tool/EditingHistory.java ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/IOContext.java ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/JShellTool.java + src/jdk.jshell/share/classes/jdk/internal/jshell/tool/StopDetectingInput.java ! test/jdk/jshell/HistoryTest.java ! test/jdk/jshell/StopExecutionTest.java From jan.lahoda at oracle.com Thu Sep 10 09:07:34 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Thu, 10 Sep 2015 09:07:34 +0000 Subject: hg: kulla/dev/langtools: Review comment - using forEach(result:add) instead of collect(toList()) to avoid unnecessary instance creation Message-ID: <201509100907.t8A97YIJ000718@aojmv0008.oracle.com> Changeset: dbbedddaf7fe Author: jlahoda Date: 2015-09-10 11:02 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/dbbedddaf7fe Review comment - using forEach(result:add) instead of collect(toList()) to avoid unnecessary instance creation ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/JShellTool.java From jan.lahoda at oracle.com Thu Sep 10 09:13:10 2015 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Thu, 10 Sep 2015 11:13:10 +0200 Subject: REPL tool implementation code review In-Reply-To: <8FA1BE5E-5E66-4223-9658-CD1EF0CD7C6C@oracle.com> References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> <55ED5E4C.3080700@oracle.com> <55EDADCA.8020109@oracle.com> <8FA1BE5E-5E66-4223-9658-CD1EF0CD7C6C@oracle.com> Message-ID: <55F149A6.8090906@oracle.com> Hi, On 7.9.2015 18:26, Paul Sandoz wrote: > On 7 Sep 2015, at 17:31, Jan Lahoda wrote: >> I've tried to resolve some of the comments from the original e-mail here: >> http://hg.openjdk.java.net/kulla/dev/langtools/rev/45e459d45f64 >> > > Looks good, it?s ok to do: > > forEach(results::add) > > rather than create additional collections if you know what you are doing. Done here: http://hg.openjdk.java.net/kulla/dev/langtools/rev/dbbedddaf7fe > > >> On 7.9.2015 14:59, Paul Sandoz wrote: >>> On 7 Sep 2015, at 11:52, Jan Lahoda wrote: >>> >>> Is the asynchronous operation on the REPL in 4 an issue? >> >> The asynchronous operation is also somewhat tricky (but necessary, I think). I have another patch in my queue that reflects (some of) Brian comments on that. >> > > Ok, i was more questioning whether it is safe rather than necessary to perform on the reading thread, since i could not determine if stopping the REPL concurrently in the reader thread would interfere with executing user code performed in the main thread. > > >>> >>> My inclination is you should try and merge the thread logic in InputBuffer.wrapInIfNeeded into CircularBuffer, then modify and rename CircularBuffer to extend from InputStream, thereby containing all the relevant multi-threaded logic and it becomes an obvious thing being deferred to. You could also add another state CLOSED, which i think more clearly indicates closure rather than continually writing -1 every time you read -1. >>> >>> Sorry for labouring the point here, but i think this kind of thing is worth spending some time on. Consider someone trying to modify the code in a few years time. Review-wise we could spin this off as a separate bug so as not to block the code going in. >> >> No problem. I'll look into doing the changes as you describe. Thanks for the comments! I've done some improvements to the CircularBuffer (including renaming it) here: http://hg.openjdk.java.net/kulla/dev/langtools/rev/4141dc296d4d I've also cleaned up the related code a little in the change, to simplify how the ConsoleIOContext works. How does it look? Thanks, Jan >> > > Thanks, > Paul. > > From andrei.eremeev at oracle.com Thu Sep 10 12:26:44 2015 From: andrei.eremeev at oracle.com (andrei.eremeev at oracle.com) Date: Thu, 10 Sep 2015 12:26:44 +0000 Subject: hg: kulla/dev/langtools: Review comment: test code clean-up Message-ID: <201509101226.t8ACQiSU000998@aojmv0008.oracle.com> Changeset: 3c512b76ef9c Author: aeremeev Date: 2015-09-10 15:25 +0300 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/3c512b76ef9c Review comment: test code clean-up ! test/jdk/jshell/CommandCompletionTest.java ! test/jdk/jshell/EditorTest.java ! test/jdk/jshell/EditorTestBase.java ! test/jdk/jshell/ErrorTranslationTest.java ! test/jdk/jshell/ReplToolTesting.java ! test/jdk/jshell/ShutdownTest.java ! test/jdk/jshell/SnippetStatusListenerTest.java ! test/jdk/jshell/StartOptionTest.java ! test/jdk/jshell/ToolBasicTest.java From andrei.eremeev at oracle.com Thu Sep 10 12:27:41 2015 From: andrei.eremeev at oracle.com (Andrei Eremeev) Date: Thu, 10 Sep 2015 15:27:41 +0300 Subject: REPL tool implementation code review In-Reply-To: References: Message-ID: <55F1773D.6090703@oracle.com> Thank for your comments, Paul! The test code was fixed. Andrei On 04.09.2015 20:19, Paul Sandoz wrote: > Hi, > > Here is the review of the REPL tool implementation. > > Generally it all looks very good. I ran out of time to analyse the tests more closely. Quite a few comments are suggestions related to using newer APIs or techniques. > > Paul. > > Implementation: > > > jdk/internal/jshell/tool/JShellTool.java:91 >> private static final String VERSION = "0.819?; >> ? >> case "-version": >> cmdout.printf("jshell %s\n", VERSION); >> return null; >> > Should this version be the JDK version, as is the case for other Java tools? > > > jdk/internal/jshell/tool/JShellTool.java:128 >> private IOContext input = null; >> private boolean regenerateOnDeath = true; >> private boolean live = false; >> >> SourceCodeAnalysis analysis; >> JShell state = null; >> Subscription shutdownSubscription = null; >> >> private boolean debug = false; >> private boolean displayPrompt = true; >> public boolean testPrompt = false; >> private Feedback feedback = Feedback.Default; >> private String cmdlineClasspath = null; >> private String cmdlineStartup = null; >> private String editor = null; >> > A small style thing, there is no need to initialize to default values (it?s also applied inconsistently) > > jdk/internal/jshell/tool/JShellTool.java:307 >> byte[] encoded = Files.readAllBytes(Paths.get(filename)); >> cmdlineStartup = new String(encoded); > > What character set is the file encoded in? should the platform's default charset be used or UTF-8? Same applies to processing the load list files. > > > jdk/internal/jshell/tool/JShellTool.java:582 >> int lastSlash = code.lastIndexOf('/'); //XXX > Rogue comment. > > > jdk/internal/jshell/tool/JShellTool.java:587 >> try (DirectoryStream dir = Files.newDirectoryStream(current)) { >> for (Path e : dir) { >> if (!accept.test(e) || !e.getFileName().toString().startsWith(prefix)) >> continue; >> result.add(new Suggestion(e.getFileName() + (Files.isDirectory(e) ? "/" : ""), false)); >> } >> } catch (IOException ex) { >> //ignore? >> } > See files.lines(path) that returns a stream with map.filter.collect(toCollection(ArrayList::new)) > > >> if (path.isEmpty()) { >> for (Path root : FileSystems.getDefault().getRootDirectories()) { >> if (!accept.test(root) || !root.toString().startsWith(prefix)) >> continue; >> result.add(new Suggestion(root.toString(), false)); >> } >> } > Alas we don?t have a FileSystem.rootDirectories method that returns a stream. > > > jdk/internal/jshell/tool/JShellTool.java:645 >> registerCommand(new Command("/list", "/l", "[all]", "list the source you have typed", >> arg -> cmdList(arg), >> new FixedCompletionProvider("all"))); > > You could use a method reference for "arg -> ??, same applies for other command declarations. > > > jdk/internal/jshell/tool/JShellTool.java:716 >> public List commandCompletionSuggestions(String code, int cursor, int[] anchor) { >> code = code.substring(0, cursor); >> int space = code.indexOf(' '); >> List result = new ArrayList<>(); >> >> if (space == (-1)) { >> for (Command cmd : new LinkedHashSet<>(commands.values())) { >> if (cmd.kind == CommandKind.HIDDEN || cmd.kind == CommandKind.HELP_ONLY) >> continue; >> String key = cmd.aliases[0]; >> if (key.startsWith(code)) { >> result.add(new Suggestion(key + " ", false)); >> } >> } >> anchor[0] = 0; >> } else { >> String arg = code.substring(space + 1); >> String cmd = code.substring(0, space); >> Command command = commands.get(cmd); >> if (command != null) { >> result.addAll(command.completions.completionSuggestions(arg, cursor - space, anchor)); >> anchor[0] += space + 1; >> } >> } >> >> Collections.sort(result, (s1, s2) -> s1.continuation.compareTo(s2.continuation)); >> return result; >> } > > > I believe you could use a stream here for both branches then do a sorted(?).collect(toList()) at the end of the method. For the first branch i think it might be something like values().distinct().filter(?).map(?).filter(?). For the second branch it is either an empty stream or the stream from the list. > > FYI: you can also to List.sort now. > > > jdk/internal/jshell/tool/JShellTool.java:763 >> if (arg == null || arg.isEmpty()) { > I dunno if it is possible but it would be nice to normalise to empty strings rather than checking for null or an empty string. > > > jdk/internal/jshell/tool/JShellTool.java:959 >> Iterator it = keySet.iterator(); >> while (it.hasNext()) { >> if (!(it.next() instanceof PersistentSnippet)) { >> it.remove(); >> } >> } > See the bulk operation Collection.removeIf. > > >> PersistentSnippet key = (PersistentSnippet) keySet.iterator().next(); >> state.drop(key).forEach(e -> handleEvent(e)); > > For kicks you could also do it like this (not suggesting you actually do unless you really want to, it?s just instructive for new APIs in Java 8): > > keySet.stream().findFirst(). > map(PersistentSnippet.class::cast). > ifPresent(k -> state.drop(k).forEach(this::handleEvent)); > > > jdk/internal/jshell/tool/JShellTool.java:981 >> private void cmdEdit(String arg) { >> Set keySet = argToKeySet(arg); >> if (keySet == null) { >> return; >> } >> Set srcSet = new LinkedHashSet<>(); >> for (Snippet key : keySet) { >> String src = key.source(); >> switch (key.subKind()) { >> case VAR_VALUE_SUBKIND: >> break; >> case ASSIGNMENT_SUBKIND: >> case OTHER_EXPRESSION_SUBKIND: >> case TEMP_VAR_EXPRESSION_SUBKIND: >> if (!src.endsWith(";")) { >> src = src + ";"; >> } >> srcSet.add(src); >> break; >> default: >> srcSet.add(src); >> break; >> } >> } >> StringBuilder sb = new StringBuilder(); >> for (String s : srcSet) { >> sb.append(s); >> sb.append('\n'); >> } >> String src = sb.toString(); > > There appears to be a reliance on an undefined iteration order, perhaps argToKeySet should return LinkedHashSet? > > I think this might be convertible to a stream with filter(?).map(?).collect(joining(?\n?, ??, ?\n")) > > > jdk/internal/jshell/tool/JShellTool.java:1150 >> try (BufferedWriter writer = new BufferedWriter(new FileWriter(resolveUserHome(filename)))) { > See also Files.newBufferedWriter(?) i like that because it makes it more explicit whether you are appending to an existing file or not. > > > jdk/internal/jshell/tool/JShellTool.java:1266 >> List> errorsOnly(List> diagnostics) { >> List> errors = new ArrayList<>(); >> for (Diagnostic diag : diagnostics) { >> switch (diag.getKind()) { >> case ERROR: >> errors.add(diag); >> break; >> case MANDATORY_WARNING: >> case WARNING: >> break; >> case NOTE: >> case OTHER: >> break; >> } >> } >> return errors; >> } > diagnostics.stream().filter(d -> d.getKind() == ERROR).collect(toList()); > > > jdk/internal/jshell/tool/JShellTool.java:1710 >> public ScannerIOContext(InputStream inStream, PrintStream pStream) { >> this(new Scanner(inStream), pStream); >> } > Unused constructor. > > > jdk/internal/jshell/tool/ExternalEditor.java:134 >> private void saveFile() { >> List lines; >> try { >> lines = Files.readAllLines(tmpfile); >> } catch (IOException ex) { >> errorHandler.accept("Failure read edit file: " + ex.getMessage()); >> return ; >> } >> StringBuilder sb = new StringBuilder(); >> for (String ln : lines) { >> sb.append(ln); >> sb.append('\n'); >> } >> saveHandler.accept(sb.toString()); >> } > Why don?t you just read as bytes then create a String from those bytes? > >> if (!key.pollEvents().isEmpty()) { >> if (!input.terminalEditorRunning()) { >> saveFile(); >> } >> } > Why does the watcher thread need to call saveFile when it is also performed after the process running the external editor has terminated? I am guessing every time the file is saved in the editor it is processed by the REPL? > > > jdk/internal/jshell/tool/EditPad.java:46 >> public class EditPad extends JFrame implements Runnable { >> private final Consumer errorHandler; >> private final String initialText; >> private final boolean[] closeLock; >> private final Consumer saveHandler; >> >> EditPad(Consumer errorHandler, String initialText, >> boolean[] closeLock, Consumer saveHandler) { >> super("Edit Pad (Experimental)"); >> this.errorHandler = errorHandler; >> this.initialText = initialText; >> this.closeLock = closeLock; >> this.saveHandler = saveHandler; >> } > > Unused field errorHandle. > > Is this still experimental? Is so should be it included? > > > jdk/internal/jshell/tool/EditPad.java:114 >> private void notifyClose() { >> synchronized (closeLock) { >> closeLock[0] = true; >> closeLock.notify(); >> } >> } >> >> public static void edit(Consumer errorHandler, String initialText, >> Consumer saveHandler) { >> boolean[] closeLock = new boolean[1]; >> SwingUtilities.invokeLater( >> new EditPad(errorHandler, initialText, closeLock, saveHandler)); >> synchronized (closeLock) { >> while (!closeLock[0]) { >> try { >> closeLock.wait(); >> } catch (InterruptedException ex) { >> // ignore and loop >> } >> } >> } >> } > You could use a shared CountDownLatch, rather than notify/wait/synchronization with a boolean array of one element. > > > jdk/internal/jshell/tool/ConsoleIOContext.java:70 >> TerminalFactory.registerFlavor(Flavor.WINDOWS, JShellWindowsTerminal :: new); >> TerminalFactory.registerFlavor(Flavor.UNIX, JShellUnixTerminal :: new); > The syntax for the constructor refs is a little unusual with the white space. Same further down when binding CRTL_UP/DOWN actions. > > > jdk/internal/jshell/tool/ConsoleIOContext.java:136 >> .forEach(s -> { result.add(s); }); > forEach(result::add); > >> Optional prefix = >> suggestions.stream() >> .map(s -> s.continuation) >> .reduce((s1, s2) -> commonPrefix(s1, s2)); > .reduce(ConsoleIOContext::commonPrefix) > > > >> try { >> Method setBuffer = in.getClass().getDeclaredMethod("setBuffer", String.class); >> >> setBuffer.setAccessible(true); >> setBuffer.invoke(in, in.getHistory().current().toString()); >> in.flush(); >> } catch (ReflectiveOperationException | IOException ex) { >> throw new IllegalStateException(ex); >> } > You might wanna comment more about the limitation in ConsoleReader and the necessity of calling a private method. > > >> private static String commonPrefix(String str1, String str2) { >> for (int i = 0; i < str2.length(); i++) { >> if (!str1.startsWith(str2.substring(0, i + 1))) { >> return str2.substring(0, i); >> } >> } >> >> return str2; >> } > What we need is a String mismatch method, from which to build a common prefix method, stay tuned?. > > > jdk/internal/jshell/tool/ConsoleIOContext.java:505 >> Set snippetsStart = new HashSet<>(); >> for (String start : prefs.get(HISTORY_SNIPPET_START, "").split(";")) { >> if (start.isEmpty()) >> continue; >> snippetsStart.add(Integer.parseInt(start)); >> } > Could streamify but perhaps not worth it. At least convert to > > if(!start.isEmpty()) > snippetsStart.add(Integer.parseInt(start)); > > >> List keys = new ArrayList<>(Arrays.asList(prefs.keys())); >> Collections.sort(keys); > Stream.of(prefs.keys()).sorted().collect(toList()); > > > jdk/internal/jshell/tool/ConsoleIOContext.java: > > I am struggling to understand the interactions of the before/afterUse code blocks, the CircularBuffer, and the thread created for writing (or pushing) read values to the buffer. > > CircularBuffer feels like a blocking queue, but there is some additional communication signalling how much input is needed. > > > > Tests > > General point: > > (a) -> ? > > Can be rewritten as: > > a -> ... > > > ReplToolTesting.java:302 >> private void addKey(boolean after, T memberInfo, Map map) { >> if (after) { >> for (Iterator> iterator = map.entrySet().iterator(); iterator.hasNext(); ) { >> Entry info = iterator.next(); >> if (info.getValue().equals(memberInfo)) { >> iterator.remove(); >> break; >> } >> } >> map.put(memberInfo.toString(), memberInfo); >> } >> } > Just do: > > map.entrySet().removeIf(e -> e.getValue().equals(memberInfo)); > > > ReplToolTesting.java:571 >> public Consumer checkOutput() { >> String fullType = type.equals("@interface")? "annotation interface" : type; >> String expectedOutput = String.format("\\| *\\w+ %s %s", fullType, name); >> Pattern outputPattern = Pattern.compile(expectedOutput); >> return (s) -> { >> Matcher matcher = outputPattern.matcher(s); >> assertTrue(matcher.find(), "Expected: '" + expectedOutput + "', actual: " + s); >> }; >> } > > See Pattern.asPredicate, capture the returned predicate in the lambda instead of the pattern and call predicate.test. > > > CommandCompletionTest.java:154 >> private List listFiles(Path path, DirectoryStream.Filter filter) throws IOException { >> List paths = new ArrayList<>(); >> try (DirectoryStream stream = Files.newDirectoryStream(path, filter)) { >> stream.iterator().forEachRemaining(p -> { >> String fileName = p.getFileName().toString(); >> if (Files.isDirectory(p)) { >> fileName += "/"; >> } >> paths.add(fileName); >> }); >> } >> Collections.sort(paths); >> return paths; >> } > > See Files.list which returns a Stream, you could probably reformulate using Predicate rather than DirectoryStream.Filter. > > > StartOptionTest.java:68 >> private void start(Consumer checkOutput, Consumer checkError, String...args) throws Exception { > > Space between ? and parameter name. Same goes for other cases in this file. > From paul.sandoz at oracle.com Thu Sep 10 15:11:02 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 10 Sep 2015 17:11:02 +0200 Subject: REPL tool implementation code review In-Reply-To: <55F1773D.6090703@oracle.com> References: <55F1773D.6090703@oracle.com> Message-ID: <98892083-E798-44A1-B61C-F433CEB946DB@oracle.com> On 10 Sep 2015, at 14:27, Andrei Eremeev wrote: > Thank for your comments, Paul! > The test code was fixed. > Changset looks good, thanks, Paul. From paul.sandoz at oracle.com Thu Sep 10 16:14:18 2015 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 10 Sep 2015 18:14:18 +0200 Subject: REPL tool implementation code review In-Reply-To: <55F149A6.8090906@oracle.com> References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> <55ED5E4C.3080700@oracle.com> <55EDADCA.8020109@oracle.com> <8FA1BE5E-5E66-4223-9658-CD1EF0CD7C6C@oracle.com> <55F149A6.8090906@oracle.com> Message-ID: On 10 Sep 2015, at 11:13, Jan Lahoda wrote: > Hi, > On 7.9.2015 18:26, Paul Sandoz wrote: >> On 7 Sep 2015, at 17:31, Jan Lahoda wrote: >>> I've tried to resolve some of the comments from the original e-mail here: >>> http://hg.openjdk.java.net/kulla/dev/langtools/rev/45e459d45f64 >>> >> >> Looks good, it?s ok to do: >> >> forEach(results::add) >> >> rather than create additional collections if you know what you are doing. > > Done here: > > http://hg.openjdk.java.net/kulla/dev/langtools/rev/dbbedddaf7fe > +1 >> >> >>> On 7.9.2015 14:59, Paul Sandoz wrote: >>>> On 7 Sep 2015, at 11:52, Jan Lahoda wrote: >>>> >>>> Is the asynchronous operation on the REPL in 4 an issue? >>> >>> The asynchronous operation is also somewhat tricky (but necessary, I think). I have another patch in my queue that reflects (some of) Brian comments on that. >>> >> >> Ok, i was more questioning whether it is safe rather than necessary to perform on the reading thread, since i could not determine if stopping the REPL concurrently in the reader thread would interfere with executing user code performed in the main thread. >> >> >>>> >>>> My inclination is you should try and merge the thread logic in InputBuffer.wrapInIfNeeded into CircularBuffer, then modify and rename CircularBuffer to extend from InputStream, thereby containing all the relevant multi-threaded logic and it becomes an obvious thing being deferred to. You could also add another state CLOSED, which i think more clearly indicates closure rather than continually writing -1 every time you read -1. >>>> >>>> Sorry for labouring the point here, but i think this kind of thing is worth spending some time on. Consider someone trying to modify the code in a few years time. Review-wise we could spin this off as a separate bug so as not to block the code going in. >>> >>> No problem. I'll look into doing the changes as you describe. Thanks for the comments! > > I've done some improvements to the CircularBuffer (including renaming it) here: > http://hg.openjdk.java.net/kulla/dev/langtools/rev/4141dc296d4d > > I've also cleaned up the related code a little in the change, to simplify how the ConsoleIOContext works. > > How does it look? > This looks much better! StopDetectingInput is now a nicely contained piece of logic. Perhaps StopDetectingInput should be renamed to StopDetectingInputStream? In ConsoleIOContext you do: if (System.getProperty("os.name").toLowerCase(Locale.US).contains(TerminalFactory.WINDOWS)) { term = new JShellWindowsTerminal(this); } else { term = new JShellUnixTerminal(this); } You are passing a partially constructed instance of ConsoleIOContext as an argument to the constructor of the terminal-based classes. Perhaps you can just pass the StopDetectingInput instance instead? Paul. From jan.lahoda at oracle.com Thu Sep 10 18:26:15 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Thu, 10 Sep 2015 18:26:15 +0000 Subject: hg: kulla/dev/langtools: Renaming StopDetectingInput to StopDetectingInputStream; not passing unfinished this to another object Message-ID: <201509101826.t8AIQFu9008833@aojmv0008.oracle.com> Changeset: d76df662d99d Author: jlahoda Date: 2015-09-10 20:25 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/d76df662d99d Renaming StopDetectingInput to StopDetectingInputStream; not passing unfinished this to another object ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/ConsoleIOContext.java - src/jdk.jshell/share/classes/jdk/internal/jshell/tool/StopDetectingInput.java + src/jdk.jshell/share/classes/jdk/internal/jshell/tool/StopDetectingInputStream.java ! test/jdk/jshell/StopExecutionTest.java From jan.lahoda at oracle.com Thu Sep 10 18:27:07 2015 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Thu, 10 Sep 2015 20:27:07 +0200 Subject: REPL tool implementation code review In-Reply-To: References: <557959520.2159948.1441392417543.JavaMail.zimbra@u-pem.fr> <55ED5E4C.3080700@oracle.com> <55EDADCA.8020109@oracle.com> <8FA1BE5E-5E66-4223-9658-CD1EF0CD7C6C@oracle.com> <55F149A6.8090906@oracle.com> Message-ID: <55F1CB7B.5090800@oracle.com> On 10.9.2015 18:14, Paul Sandoz wrote: > > On 10 Sep 2015, at 11:13, Jan Lahoda wrote: >>>> On 7.9.2015 14:59, Paul Sandoz wrote: >>>>> On 7 Sep 2015, at 11:52, Jan Lahoda wrote: >>>>> >>>>> Is the asynchronous operation on the REPL in 4 an issue? >>>> >>>> The asynchronous operation is also somewhat tricky (but necessary, I think). I have another patch in my queue that reflects (some of) Brian comments on that. >>>> >>> >>> Ok, i was more questioning whether it is safe rather than necessary to perform on the reading thread, since i could not determine if stopping the REPL concurrently in the reader thread would interfere with executing user code performed in the main thread. >>> >>> >>>>> >>>>> My inclination is you should try and merge the thread logic in InputBuffer.wrapInIfNeeded into CircularBuffer, then modify and rename CircularBuffer to extend from InputStream, thereby containing all the relevant multi-threaded logic and it becomes an obvious thing being deferred to. You could also add another state CLOSED, which i think more clearly indicates closure rather than continually writing -1 every time you read -1. >>>>> >>>>> Sorry for labouring the point here, but i think this kind of thing is worth spending some time on. Consider someone trying to modify the code in a few years time. Review-wise we could spin this off as a separate bug so as not to block the code going in. >>>> >>>> No problem. I'll look into doing the changes as you describe. Thanks for the comments! >> >> I've done some improvements to the CircularBuffer (including renaming it) here: >> http://hg.openjdk.java.net/kulla/dev/langtools/rev/4141dc296d4d >> >> I've also cleaned up the related code a little in the change, to simplify how the ConsoleIOContext works. >> >> How does it look? >> > > This looks much better! StopDetectingInput is now a nicely contained piece of logic. > > Perhaps StopDetectingInput should be renamed to StopDetectingInputStream? > > In ConsoleIOContext you do: > > if (System.getProperty("os.name").toLowerCase(Locale.US).contains(TerminalFactory.WINDOWS)) { > term = new JShellWindowsTerminal(this); > } else { > term = new JShellUnixTerminal(this); > } > > You are passing a partially constructed instance of ConsoleIOContext as an argument to the constructor of the terminal-based classes. Perhaps you can just pass the StopDetectingInput instance instead? Right, sorry for that. Fixed: http://hg.openjdk.java.net/kulla/dev/langtools/rev/d76df662d99d Thanks, Jan > > Paul. > From jan.lahoda at oracle.com Fri Sep 11 08:19:51 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Fri, 11 Sep 2015 08:19:51 +0000 Subject: hg: kulla/dev/langtools: Review comment: improving synchronization of JShell.stop() Message-ID: <201509110819.t8B8JqoH011959@aojmv0008.oracle.com> Changeset: 9fd4ba1a78d3 Author: jlahoda Date: 2015-09-11 10:19 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/9fd4ba1a78d3 Review comment: improving synchronization of JShell.stop() ! src/jdk.jshell/share/classes/jdk/jshell/ExecutionControl.java ! src/jdk.jshell/share/classes/jdk/jshell/JShell.java ! test/jdk/jshell/StopExecutionTest.java From maurizio.cimadamore at oracle.com Fri Sep 11 15:25:36 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Fri, 11 Sep 2015 16:25:36 +0100 Subject: REPL code review Message-ID: <55F2F270.8020803@oracle.com> Hi, this is a review for REPL - I focused on the javac/language-related parts and I think I came up with some interesting examples. I tried to cover as many areas as possible and, as a result, the comments are not as deep. I can dive more into specific sections if required. Overall the feeling I get from this is very good. The overall machinery seems to be cleanly defined - I like the fact that you use JDI to completely hide the REPL frontend from how user-defined classes are loaded/redefined. I did run into some bugs - some of them minor, some of them more worrisome - I hope this is the kind of feedback you were looking for. One general high-level comment, which I also pointed out elsewhere, is that I'm not sure jshell really belongs in langtools; while it's semantically (obviously) related to langtools - it is a rather different beasts w.r.t. all other tools in langtools-land; the fat that it depends on the JDK (for jline, and for JDI in general) makes it very hard to run on top of a random JDK and then bootstrapping classes - which is a technique widely used to be able to run langtools tools w/o having to do a full build cycle. More specifically, talking about IDE integration, I don't see how IntelliJ/Netbeans langtools projects (and the langtools internal developer ant build) could be updated to be able to run/debug jshell w/o a full build. Below are the more detailed review comments (organized by class): * TreeDependencyScanner - not sure why Object is being used as a return type for the visitor methods; you could probably just use Void and then use scan() instead of scanAndReduce() [given there's nothing to reduce] ? - I'm not sure you need the scanDeclaration method - it is only used in Scan.processVariable - which runs it on a variable's type - but variable types are already scanned by TDS references with the right 'decl' set? * Eval - just curious - you seem to do a lot of String-based analysis (see example in computeQualifiedParameterTypes()) - why is it not possible to rely on the more robust Element API to do that? I.e. couldn't a snippet corresponding to a declaration have an Element? Or, even more generally, why not just create real AST nodes for wraps? - UnresolvedExtractor - it would probably help to use DiagnosticFlag.RECOVERABLE instead of hard-coded strings to filter out resolution error (I also noted this somewhere below). However, I also note how the logic for extracting the unresolved symbol is inherently fragile and dependent on what is the shape of the javac diagnostics. I wonder if looking at the analyzed tree might not provide a more robust solution here. - the Range logic is clever - but, again, it seems like a big workaround for the fact that wrapping is defined in terms of strings - not trees; any reason you went down this path? - corralled - I found the Snippet hierarchy underused when it comes to corralling; the task of corralling a snippet is basically performed by Eval through some helper routines - couldn't we have each snippet define how corralling should be done? You go half-way there by having a corral() method in the Snippet base class, but it doesn't do much, other then retrieving whatever value has been set from outside. - the corralling logic doesn't seem to work on supertypes using FQN: -> class A extends B.Inner { static class Inner { } } | Error: | package B does not exist | class A extends B.Inner { static class Inner { } } | ^-----^ I would have expected the snippet to wait for B. - the updates of corralled snippets seems to happen at the wrong time - this could lead to chain of symbols that are not usable: -> class D extends E { } | Added class D, however, it cannot be referenced until class E is declared -> class E { D d; } | Added class E, however, it cannot be referenced until class D is declared I guess what happens here is that, since the decision on whether to corral or not is based on the fact that we can compile cleanly , here we run into troubles. The first snippet obviously cannot be compiled cleanly, so it's corralled; the second one cannot compile cleanly because the first one is not 'defined' - so you get another resolution error and another corralled snippet. Unfortunately, the two snippets taken together form a legal source. Btw, I think the current behavior is preventing stuff like this: -> class A extends B { } | Added class A, however, it cannot be referenced until class B is declared -> class B extends A { } | Added class B, however, it cannot be referenced until class A is declared from looping forever in Eval.declare; perhaps the code could be made more robust by looking explicitly for update loops (i.e. this can be done by keeping a stack of all the updated snippets and see if we encounter the same snippet twice). - on update loops - I guess it is indeed possible to have loops, even with current corralling limitations (by exploiting redefinition): -> class Outer { class Inner extends Foo { } } | Added class Outer, however, it cannot be referenced until class Foo is declared -> class Foo { } | Added class Foo -> class Foo extends Outer { } Update loop check logic is in order I think. Also, interestingly - I'm not able to stop the REPL once I get into such a loop. Actually, later I found out it doesn't loop forever - I guess eventually it runs out of stack (after several minutes) and exits saying this: | Replaced class Foo | Update overwrote class Foo Which seems bogus anyway - given there's cyclic inheritance (no error is reported!). * ReplParser - in general it looks good. The question I have here is more design-oriented: why do we need it? I believe the main use case is that you need to parse the input source to be able to decide between expressions and declarations. Can't we wrap the snippet in the following two ways: class Temp$Decl { } and: class Temp$Expr { Object o = } and then see who wins? This would also require also to move the wrapping logic from Eval to some earlier stage (when the input is deemed complete). It's probably a very big change, so I'm merely throwing this as a suggestion for future improvements - it's outside the scope of the review. * TypePrinter - so, this is needed when you need to print the type of some expression; it seems the exact copy of the printer used by the RichDiagnosticFormatter, except for the fact that you want to omit the captured type-variable details. So, maybe you could split this process in two: first you map the original type onto a type where all captured variables have been replaced by their bounds (i.e from List<#CAP1> to List) - then you call the RichDiagnosticFormatter printer onto it (the best way to do that would probably be to create a fake diagnostic with a type as its argument, and then feed that to the diagnostic formatter - that should give you back a string). The RichDiagnosticFormatter, by default will also generate stuff that you probably don't want, such as 'where' clauses to disambiguate type-variable names with the same names but different owners - but such features can be selectively disabled (via javac command line options). * MemoryFileManager - Maybe the code of 'list' could be more optimized/lazy by returning a compound iterator - i.e. an iterator that keeps scanning the items in the first iterator and, when finished moves onto the next iterator. - comments: is it normal to repeat all comments from the JavaFileManager interface? - watch out for unused methods: dumpClasses, findGeneratedClass, findGeneratedBytes, inferModuleName, listModueLocations (maybe this stuff is for Jigsaw? If so, maybe add a comment) * CompletenessAnalyzer - Location position constants appear more complex than necessary (or I'm missing something); it appears that constants like XEXPR1o, XDECL1o, XSTMT1o are never used? The only uages I see are two: - to build other constants (such as XEXPR1, XDECL1 and XSTMT1) - in a switch inside parseUnit Since no token is really using those constants explicitly, maybe they could be removed in favor of their more general counterparts (this also kind of means that i.e. XEXPR1 == XEXPR, etc.) - I believe in general the design of location codes + 'belongs' field feels weird, and given this code is quite convoluted (because the task is hard!) the code might benefit from some cleanup. My suggestions: (1) come up with a cleaner enum which contains: { EXPR, DECL, STMT, AMBIGUOUS, ERR } - use this for classification purposes in parseUnit and related predicates (i.e. what is this token like?) (2) extract the 'isOkToTerminate' property onto a separate boolean flag on the token. After all, this is an orthogonal property of all tokens. - watch out for commented out code/dead code (here and elsewhere) Sometimes the completeness analysis doesn't behave as expected: -> new >> ; >> ; >> ; >> ; >> ; >> ; >> Foo(); | Error: | expected | new | ^ | Error: | '(' or '[' expected | ; | ^ In the above, I found it confusing that the ';' is not enough to tell the analysis logic that I'm done with the statement - and the analysis will keep waiting until it matches a '()'. -SourceCodeAnalysisImpl - generally looks good, and the tab completion logic is quite clever: - in scopeContent() what is the difference between the 'scopeIterable' and a call to Scope.getLocalElements (which is routed via Scope.getSymbols)? - the logic for filtering based on target type is cool - but could use some cleanup - for instance there are two smart filters, and it's not clear why those are needed; it seems like smartTypeFilter is only used in imports, and the main difference is that it is allowed to operate on type elements, while the smartFilter can't; but the typing logic is the same (except for isAssignable vs. isSubtype - which shouldn't be important in the smartTypeFilter case anyway). I suggest just using smartFilter (without the class filtering logic), and the add the proper extra filtering logic (i.e. must be a class, must be a non-class) as a separate 'kind' filter. -found a bug with varargs (varargs expansion does not take place): -> class Foo { static void m(int... i) { } } | Replaced class Foo | Update overwrote class Foo -> Foo.m(Display all 395 possibilities? (y or n) -found bugs around constructors; asMemberOf doesn't get called, and that leads to useless smart filters: -> class Baz { >> Baz(X x) { } >> } | Added class Baz -> String s = ""; | Added variable s of type String with initial value "" -> new Baz(Display all 391 possibilities? (y or n) -> new Baz(Display all 391 possibilities? (y or n) -> Baz bz = new Baz<>(Display all 391 possibilities? (y or n) The problem is that the type of the constructor is Baz(X) and no variable in scope can possibly have type X, so all elements in the scope are cleared out. In the first two cases should be fixable by simply adding a call to asMemberOf() on the constructor element. The third case is more difficult - perhaps some heuristics could be attempted, but the general case is hard. -smart completion doesn't always work as expected - in particular, target-type-based suggestions seem to occur only once: -> fs.m( ls -> fs.m | Error: | cannot find symbol | symbol: variable m | fs.m | ^--^ -> fs.m(Display all 389 possibilities? (y or n) After first time, all members will be displayed (unless you start working on a different instance...) * TaskFactory: - I see that cntResolve counts the recoverable errors - javac does have a notion of recoverable errors that is used in annotation processing - all recoverable diagnostics are marked with a special flag (DiagnosticFlag.RECOVERABLE). I wonder if it would be more robust to rely on that, rather than using raw diagnostic text (as the latter might change over time). - I like the hierarchy of tasks - it is very easy to see which tasks are meant to operate on source code and which on synthetic wraps. Maurizio From jan.lahoda at oracle.com Fri Sep 11 17:01:01 2015 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Fri, 11 Sep 2015 19:01:01 +0200 Subject: REPL code review In-Reply-To: <55F2F270.8020803@oracle.com> References: <55F2F270.8020803@oracle.com> Message-ID: <55F308CD.1010203@oracle.com> Hi Maurizio, Thanks for the review and comments! On 11.9.2015 17:25, Maurizio Cimadamore wrote: [snip] > -SourceCodeAnalysisImpl - generally looks good, and the tab completion > logic is quite clever: > > - in scopeContent() what is the difference between the > 'scopeIterable' and a call to Scope.getLocalElements (which is routed > via Scope.getSymbols)? The general intent of scopeContent is to gather all elements that are defined at the given place (for which the Scope was created). Scope.getLocalElements only returns elements that are local to the scope - for example for a Scope that was created for a location inside a method, the getLocalElements won't include the imported elements. So, generally, it is necessary to recursively use Scope.getEnclosingScope() and use getLocalElements on all the Scopes from the given Scope up. The scopeIterable is used to do that - it iterates up the Scope hierarchy - on each item of the iterable the getLocalElements is called and the results are concatenated (using Stream.flatMap). I'll work on the other comments for SourceCodeAnalysisImpl. Thanks, Jan > > - the logic for filtering based on target type is cool - but could > use some cleanup - for instance there are two smart filters, and it's > not clear why those are needed; it seems like smartTypeFilter is only > used in imports, and the main difference is that it is allowed to > operate on type elements, while the smartFilter can't; but the typing > logic is the same (except for isAssignable vs. isSubtype - which > shouldn't be important in the smartTypeFilter case anyway). I suggest > just using smartFilter (without the class filtering logic), and the add > the proper extra filtering logic > (i.e. must be a class, must be a non-class) as a separate 'kind' > filter. > > -found a bug with varargs (varargs expansion does not take place): > > -> class Foo { static void m(int... i) { } } > | Replaced class Foo > | Update overwrote class Foo > -> Foo.m(Display all 395 possibilities? (y or n) > > -found bugs around constructors; asMemberOf doesn't get called, and > that leads to useless smart filters: > > -> class Baz { > >> Baz(X x) { } > >> } > | Added class Baz > > -> String s = ""; > | Added variable s of type String with initial value "" > > -> new Baz(Display all 391 possibilities? (y or n) > > -> new Baz(Display all 391 possibilities? (y or n) > > -> Baz bz = new Baz<>(Display all 391 possibilities? (y or n) > > The problem is that the type of the constructor is Baz(X) and no > variable in scope can possibly have type X, so all elements in the scope > are cleared out. In the first two cases should be fixable by simply > adding a call to asMemberOf() on the constructor element. The third case > is more difficult - perhaps some heuristics could be attempted, but the > general case is hard. > > -smart completion doesn't always work as expected - in particular, > target-type-based suggestions seem to occur only once: > > -> fs.m( > ls > > -> fs.m > | Error: > | cannot find symbol > | symbol: variable m > | fs.m > | ^--^ > > -> fs.m(Display all 389 possibilities? (y or n) > > After first time, all members will be displayed (unless you start > working on a different instance...) > > * TaskFactory: > - I see that cntResolve counts the recoverable errors - javac does > have a notion of recoverable errors that is used in annotation > processing - all recoverable diagnostics are marked with a special flag > (DiagnosticFlag.RECOVERABLE). I wonder if it would be more robust to > rely on that, rather than using raw diagnostic text (as the latter might > change over time). > > - I like the hierarchy of tasks - it is very easy to see which tasks > are meant to operate on source code and which on synthetic wraps. > > > Maurizio > From maurizio.cimadamore at oracle.com Fri Sep 11 17:27:50 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Fri, 11 Sep 2015 18:27:50 +0100 Subject: REPL code review In-Reply-To: <55F308CD.1010203@oracle.com> References: <55F2F270.8020803@oracle.com> <55F308CD.1010203@oracle.com> Message-ID: <55F30F16.9080201@oracle.com> On 11/09/15 18:01, Jan Lahoda wrote: > The general intent of scopeContent is to gather all elements that are > defined at the given place (for which the Scope was created). > Scope.getLocalElements only returns elements that are local to the > scope - for example for a Scope that was created for a location inside > a method, the getLocalElements won't include the imported elements. ah - thx for the clarification; I get that imported elements will be missing, but I believe elements declared in enclosing scopes will still be there? I.e. after all, a scope will have its 'next' field pointing to the enclosing scope (which is the scope of the outer env); but I get that there are corner cases in which this 1-1 mapping breaks. Maurizio From jan.lahoda at oracle.com Fri Sep 11 19:23:54 2015 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Fri, 11 Sep 2015 21:23:54 +0200 Subject: REPL code review In-Reply-To: <55F30F16.9080201@oracle.com> References: <55F2F270.8020803@oracle.com> <55F308CD.1010203@oracle.com> <55F30F16.9080201@oracle.com> Message-ID: <55F32A4A.5060707@oracle.com> On 11.9.2015 19:27, Maurizio Cimadamore wrote: > > > On 11/09/15 18:01, Jan Lahoda wrote: >> The general intent of scopeContent is to gather all elements that are >> defined at the given place (for which the Scope was created). >> Scope.getLocalElements only returns elements that are local to the >> scope - for example for a Scope that was created for a location inside >> a method, the getLocalElements won't include the imported elements. > ah - thx for the clarification; I get that imported elements will be > missing, but I believe elements declared in enclosing scopes will still > be there? I.e. after all, a scope will have its 'next' field pointing to > the enclosing scope (which is the scope of the outer env); but I get > that there are corner cases in which this 1-1 mapping breaks. The hierarchies of api Scopes (com.sun.tools.tree.Scope) and implementation Scopes (com.sun.tools.javac.code.Scope) differ. Consider this case: class X { int i; public void test() { int j; //here } } The implementation Scope at the marked position will have several outer Scopes with less and less elements defined. But the "outer/next Scope" hierarchy will stop at the class declaration. For API Scope at the marked position, the API Scope.getEnclosingScope will jump directly out of the class declaration - the API Scope.getEnclosingScope is based on Env.outer (see JavacScope.getEnclosingScope). The implementation Scope kept internally in the API Scope (via an Env) is flattened in Attr.copyScope, so it does not have any outer Scope (but it indeed contains all the elements defined on the path from the enclosing class to the point where the Scope was requested). So, in the API Scope, doing getEnclosingScope and getLocalElements should not lead to duplicated elements in the result. Jan > > Maurizio From robert.field at oracle.com Fri Sep 11 20:32:16 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Fri, 11 Sep 2015 20:32:16 +0000 Subject: hg: kulla/dev/langtools: Resolve issue raised in JShell tool review of command arguments processing using sets, by adjusting logic for future named ids, and switching to List. Message-ID: <201509112032.t8BKWGL6021244@aojmv0008.oracle.com> Changeset: 4a799a74083e Author: rfield Date: 2015-09-11 13:31 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/4a799a74083e Resolve issue raised in JShell tool review of command arguments processing using sets, by adjusting logic for future named ids, and switching to List. ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/JShellTool.java ! test/jdk/jshell/ToolBasicTest.java From robert.field at oracle.com Sat Sep 12 05:56:31 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Sat, 12 Sep 2015 05:56:31 +0000 Subject: hg: kulla/dev/langtools: 8136423: JShell: rationalize TreeDependencyScanner. Per review, changed TreeDependencyScanner to Void returns and removed reduce and scanDeclaration. Also changed decl/body logic to nest 'correctly' even though this distinction is not currently used. And other clean-up. Message-ID: <201509120556.t8C5uVpO008226@aojmv0008.oracle.com> Changeset: ffc330edec8b Author: rfield Date: 2015-09-11 22:56 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/ffc330edec8b 8136423: JShell: rationalize TreeDependencyScanner. Per review, changed TreeDependencyScanner to Void returns and removed reduce and scanDeclaration. Also changed decl/body logic to nest 'correctly' even though this distinction is not currently used. And other clean-up. ! src/jdk.jshell/share/classes/jdk/jshell/DeclarationSnippet.java ! src/jdk.jshell/share/classes/jdk/jshell/Eval.java ! src/jdk.jshell/share/classes/jdk/jshell/TreeDependencyScanner.java From jan.lahoda at oracle.com Mon Sep 14 04:47:10 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Mon, 14 Sep 2015 04:47:10 +0000 Subject: hg: kulla/dev/langtools: Close the reader properly, and restore terminal, when finishing. Message-ID: <201509140447.t8E4lAAt020832@aojmv0008.oracle.com> Changeset: 59241cba77e8 Author: jlahoda Date: 2015-09-14 06:46 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/59241cba77e8 Close the reader properly, and restore terminal, when finishing. ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/ConsoleIOContext.java ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/IOContext.java From jan.lahoda at oracle.com Mon Sep 14 14:43:04 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Mon, 14 Sep 2015 14:43:04 +0000 Subject: hg: kulla/dev/langtools: 4 new changesets Message-ID: <201509141443.t8EEh4hQ020923@aojmv0008.oracle.com> Changeset: 946ca38ccc49 Author: jlahoda Date: 2015-09-14 14:19 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/946ca38ccc49 Review comment: tab completion should handle vararg method invocations. ! src/jdk.jshell/share/classes/jdk/jshell/SourceCodeAnalysisImpl.java ! test/jdk/jshell/CompletionSuggestionTest.java Changeset: ea1fd645628d Author: jlahoda Date: 2015-09-14 14:19 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/ea1fd645628d Review comment: must use asMemberOf on constructors. ! src/jdk.jshell/share/classes/jdk/jshell/SourceCodeAnalysisImpl.java ! test/jdk/jshell/CompletionSuggestionTest.java Changeset: 214544e27c9a Author: jlahoda Date: 2015-09-14 14:19 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/214544e27c9a Tab completion cycles between the smart and non-smart tab completion now. (Review comment.) ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/ConsoleIOContext.java Changeset: 1fdd34dd23bc Author: jlahoda Date: 2015-09-14 15:23 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/1fdd34dd23bc An attempt to simplify smartFilter&smartTypeFilter. (Review comment). ! src/jdk.jshell/share/classes/jdk/jshell/SourceCodeAnalysisImpl.java From jan.lahoda at oracle.com Mon Sep 14 15:06:49 2015 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Mon, 14 Sep 2015 17:06:49 +0200 Subject: REPL code review In-Reply-To: <55F308CD.1010203@oracle.com> References: <55F2F270.8020803@oracle.com> <55F308CD.1010203@oracle.com> Message-ID: <55F6E289.2010902@oracle.com> Hi Maurizio, I've tried to tweak the code according to the comments below. Links inline. On 11.9.2015 19:01, Jan Lahoda wrote: > Hi Maurizio, > > Thanks for the review and comments! > > On 11.9.2015 17:25, Maurizio Cimadamore wrote: > [snip] > >> -SourceCodeAnalysisImpl - generally looks good, and the tab completion >> logic is quite clever: >> >> - in scopeContent() what is the difference between the >> 'scopeIterable' and a call to Scope.getLocalElements (which is routed >> via Scope.getSymbols)? > > The general intent of scopeContent is to gather all elements that are > defined at the given place (for which the Scope was created). > Scope.getLocalElements only returns elements that are local to the scope > - for example for a Scope that was created for a location inside a > method, the getLocalElements won't include the imported elements. > > So, generally, it is necessary to recursively use > Scope.getEnclosingScope() and use getLocalElements on all the Scopes > from the given Scope up. The scopeIterable is used to do that - it > iterates up the Scope hierarchy - on each item of the iterable the > getLocalElements is called and the results are concatenated (using > Stream.flatMap). > > I'll work on the other comments for SourceCodeAnalysisImpl. > > Thanks, > Jan > >> >> - the logic for filtering based on target type is cool - but could >> use some cleanup - for instance there are two smart filters, and it's >> not clear why those are needed; it seems like smartTypeFilter is only >> used in imports, and the main difference is that it is allowed to >> operate on type elements, while the smartFilter can't; but the typing >> logic is the same (except for isAssignable vs. isSubtype - which >> shouldn't be important in the smartTypeFilter case anyway). I suggest >> just using smartFilter (without the class filtering logic), and the add >> the proper extra filtering logic >> (i.e. must be a class, must be a non-class) as a separate 'kind' >> filter. http://hg.openjdk.java.net/kulla/dev/langtools/rev/1fdd34dd23bc I kept smartTypeFilter and smartFilter, but the only the first one is doing the actual type check now - the second one adds the kind check (to get a full filter). >> >> -found a bug with varargs (varargs expansion does not take place): >> >> -> class Foo { static void m(int... i) { } } >> | Replaced class Foo >> | Update overwrote class Foo >> -> Foo.m(Display all 395 possibilities? (y or n) http://hg.openjdk.java.net/kulla/dev/langtools/rev/946ca38ccc49 >> >> -found bugs around constructors; asMemberOf doesn't get called, and >> that leads to useless smart filters: >> >> -> class Baz { >> >> Baz(X x) { } >> >> } >> | Added class Baz >> >> -> String s = ""; >> | Added variable s of type String with initial value "" >> >> -> new Baz(Display all 391 possibilities? (y or n) >> >> -> new Baz(Display all 391 possibilities? (y or n) >> >> -> Baz bz = new Baz<>(Display all 391 possibilities? (y or n) >> >> The problem is that the type of the constructor is Baz(X) and no >> variable in scope can possibly have type X, so all elements in the scope >> are cleared out. In the first two cases should be fixable by simply >> adding a call to asMemberOf() on the constructor element. The third case >> is more difficult - perhaps some heuristics could be attempted, but the >> general case is hard. http://hg.openjdk.java.net/kulla/dev/langtools/rev/ea1fd645628d It first tries to find the type in the AST, and if there is no usable type, it resorts to the targetType machinery that already exists. I hope this will work reasonably, I think. >> >> -smart completion doesn't always work as expected - in particular, >> target-type-based suggestions seem to occur only once: >> >> -> fs.m( >> ls >> >> -> fs.m >> | Error: >> | cannot find symbol >> | symbol: variable m >> | fs.m >> | ^--^ >> >> -> fs.m(Display all 389 possibilities? (y or n) >> >> After first time, all members will be displayed (unless you start >> working on a different instance...) http://hg.openjdk.java.net/kulla/dev/langtools/rev/214544e27c9a Pressing should now cycle between the "smart" and ordinary completion. (I was looking if I could improve typing detection - so that the completion state would reset when the user would e.g. type and then delete something, but so far, I didn't find a reasonable way to do that.) How do these look? Thanks! Jan >> >> * TaskFactory: >> - I see that cntResolve counts the recoverable errors - javac does >> have a notion of recoverable errors that is used in annotation >> processing - all recoverable diagnostics are marked with a special flag >> (DiagnosticFlag.RECOVERABLE). I wonder if it would be more robust to >> rely on that, rather than using raw diagnostic text (as the latter might >> change over time). >> >> - I like the hierarchy of tasks - it is very easy to see which tasks >> are meant to operate on source code and which on synthetic wraps. >> >> >> Maurizio >> From maurizio.cimadamore at oracle.com Mon Sep 14 15:58:46 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 14 Sep 2015 16:58:46 +0100 Subject: REPL code review In-Reply-To: <55F6E289.2010902@oracle.com> References: <55F2F270.8020803@oracle.com> <55F308CD.1010203@oracle.com> <55F6E289.2010902@oracle.com> Message-ID: <55F6EEB6.5050302@oracle.com> The changes look good; I like how you handled the diamond case by reusing the target-type machinery to find a suitable target-type. The smart filter simplification is also welcome. Thanks Maurizio On 14/09/15 16:06, Jan Lahoda wrote: > Hi Maurizio, > > I've tried to tweak the code according to the comments below. Links > inline. > > On 11.9.2015 19:01, Jan Lahoda wrote: >> Hi Maurizio, >> >> Thanks for the review and comments! >> >> On 11.9.2015 17:25, Maurizio Cimadamore wrote: >> [snip] >> >>> -SourceCodeAnalysisImpl - generally looks good, and the tab completion >>> logic is quite clever: >>> >>> - in scopeContent() what is the difference between the >>> 'scopeIterable' and a call to Scope.getLocalElements (which is routed >>> via Scope.getSymbols)? >> >> The general intent of scopeContent is to gather all elements that are >> defined at the given place (for which the Scope was created). >> Scope.getLocalElements only returns elements that are local to the scope >> - for example for a Scope that was created for a location inside a >> method, the getLocalElements won't include the imported elements. >> >> So, generally, it is necessary to recursively use >> Scope.getEnclosingScope() and use getLocalElements on all the Scopes >> from the given Scope up. The scopeIterable is used to do that - it >> iterates up the Scope hierarchy - on each item of the iterable the >> getLocalElements is called and the results are concatenated (using >> Stream.flatMap). >> >> I'll work on the other comments for SourceCodeAnalysisImpl. >> >> Thanks, >> Jan >> >>> >>> - the logic for filtering based on target type is cool - but could >>> use some cleanup - for instance there are two smart filters, and it's >>> not clear why those are needed; it seems like smartTypeFilter is only >>> used in imports, and the main difference is that it is allowed to >>> operate on type elements, while the smartFilter can't; but the typing >>> logic is the same (except for isAssignable vs. isSubtype - which >>> shouldn't be important in the smartTypeFilter case anyway). I suggest >>> just using smartFilter (without the class filtering logic), and the add >>> the proper extra filtering logic >>> (i.e. must be a class, must be a non-class) as a separate 'kind' >>> filter. > > http://hg.openjdk.java.net/kulla/dev/langtools/rev/1fdd34dd23bc > > I kept smartTypeFilter and smartFilter, but the only the first one is > doing the actual type check now - the second one adds the kind check > (to get a full filter). > >>> >>> -found a bug with varargs (varargs expansion does not take place): >>> >>> -> class Foo { static void m(int... i) { } } >>> | Replaced class Foo >>> | Update overwrote class Foo >>> -> Foo.m(Display all 395 possibilities? (y or n) > > http://hg.openjdk.java.net/kulla/dev/langtools/rev/946ca38ccc49 > >>> >>> -found bugs around constructors; asMemberOf doesn't get called, and >>> that leads to useless smart filters: >>> >>> -> class Baz { >>> >> Baz(X x) { } >>> >> } >>> | Added class Baz >>> >>> -> String s = ""; >>> | Added variable s of type String with initial value "" >>> >>> -> new Baz(Display all 391 possibilities? (y or n) >>> >>> -> new Baz(Display all 391 possibilities? (y or n) >>> >>> -> Baz bz = new Baz<>(Display all 391 possibilities? (y >>> or n) >>> >>> The problem is that the type of the constructor is Baz(X) and no >>> variable in scope can possibly have type X, so all elements in the >>> scope >>> are cleared out. In the first two cases should be fixable by simply >>> adding a call to asMemberOf() on the constructor element. The third >>> case >>> is more difficult - perhaps some heuristics could be attempted, but the >>> general case is hard. > > http://hg.openjdk.java.net/kulla/dev/langtools/rev/ea1fd645628d > > It first tries to find the type in the AST, and if there is no usable > type, it resorts to the targetType machinery that already exists. I > hope this will work reasonably, I think. > >>> >>> -smart completion doesn't always work as expected - in particular, >>> target-type-based suggestions seem to occur only once: >>> >>> -> fs.m( >>> ls >>> >>> -> fs.m >>> | Error: >>> | cannot find symbol >>> | symbol: variable m >>> | fs.m >>> | ^--^ >>> >>> -> fs.m(Display all 389 possibilities? (y or n) >>> >>> After first time, all members will be displayed (unless you start >>> working on a different instance...) > > http://hg.openjdk.java.net/kulla/dev/langtools/rev/214544e27c9a > > Pressing should now cycle between the "smart" and ordinary > completion. (I was looking if I could improve typing detection - so > that the completion state would reset when the user would e.g. type > and then delete something, but so far, I didn't find a reasonable way > to do that.) > > > How do these look? > > Thanks! > Jan > >>> >>> * TaskFactory: >>> - I see that cntResolve counts the recoverable errors - javac does >>> have a notion of recoverable errors that is used in annotation >>> processing - all recoverable diagnostics are marked with a special flag >>> (DiagnosticFlag.RECOVERABLE). I wonder if it would be more robust to >>> rely on that, rather than using raw diagnostic text (as the latter >>> might >>> change over time). >>> >>> - I like the hierarchy of tasks - it is very easy to see which >>> tasks >>> are meant to operate on source code and which on synthetic wraps. >>> >>> >>> Maurizio >>> From robert.field at oracle.com Wed Sep 16 22:06:48 2015 From: robert.field at oracle.com (Robert Field) Date: Wed, 16 Sep 2015 15:06:48 -0700 Subject: JShell: source in langtools vs JDK? Message-ID: As part of the JShell code review, Maurizio made the case that JShell should be in the JDK repo rather than langtools repo (where it is in the Kulla workspace). We would like additional feedback on this before proceeding -- > On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore wrote: > > One general high-level comment, which I also pointed out elsewhere, is that I'm not sure jshell really belongs in langtools; while it's semantically (obviously) related to langtools - it is a rather different beasts w.r.t. all other tools in langtools-land; the fact that it depends on the JDK (for jline, and for JDI in general) makes it very hard to run on top of a random JDK and then bootstrapping classes - which is a technique widely used to be able to run langtools tools w/o having to do a full build cycle. More specifically, talking about IDE integration, I don't see how IntelliJ/Netbeans langtools projects (and the langtools internal developer ant build) could be updated to be able to run/debug jshell w/o a full build. From jonathan.gibbons at oracle.com Wed Sep 16 22:31:50 2015 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 16 Sep 2015 15:31:50 -0700 Subject: JShell: source in langtools vs JDK? In-Reply-To: References: Message-ID: <55F9EDD6.9050702@oracle.com> On 09/16/2015 03:06 PM, Robert Field wrote: > As part of the JShell code review, Maurizio made the case that JShell should be in the JDK repo rather than langtools repo (where it is in the Kulla workspace). We would like additional feedback on this before proceeding -- > >> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore wrote: >> >> One general high-level comment, which I also pointed out elsewhere, is that I'm not sure jshell really belongs in langtools; while it's semantically (obviously) related to langtools - it is a rather different beasts w.r.t. all other tools in langtools-land; the fact that it depends on the JDK (for jline, and for JDI in general) makes it very hard to run on top of a random JDK and then bootstrapping classes - which is a technique widely used to be able to run langtools tools w/o having to do a full build cycle. More specifically, talking about IDE integration, I don't see how IntelliJ/Netbeans langtools projects (and the langtools internal developer ant build) could be updated to be able to run/debug jshell w/o a full build. > > In the Big Picture view of the universe, folk would like to restructure the repos in an OPenJDK forest, at which point the distinction between the langtools and jdk repos will likely disappear, or at least become very (very) blurred. While that change is not imminent, IMO opinion, it significantly reduces the impact of the reasons why JShell should not be in langtools. I also don't entirely agree with the IDE reasons either. If you want to work on JShell in an IDE, you're going to have to do what it takes to get rid of the red squiggly lines anyway, whatever repo the code is in. To keep JShell out of Langtools because it doesn't fit our current methodologies is a case of the tail wagging the dog. Also, I note that JDK 9 now builds the product module by module. In times past, we built the world repo by repo, and that would indeed have been good reason to keep JShell out of langtools, because of the dependencies of JShell on JDK 9 API, like jline. But now, the build will build "interim javac", and can then build JDK modules like java.base and whatever module jline ends up in, and can then build the module containing JShell. As to how to organize IDE projects, in the worst case, we keep the langtools IDE setup as it is now (no JShell) and have a separate project for JShell itself. That is tantamount to what we would do if we push JShell into the jdk repo. So, I'm saying all that while trying to be agnostic on which repo it should live in. I don't really agree with the stated reasons why it should not live in langtools, which means that we can look for other reasons to choose one repo or another. -- Jon From maurizio.cimadamore at oracle.com Wed Sep 16 23:02:11 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Thu, 17 Sep 2015 00:02:11 +0100 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55F9EDD6.9050702@oracle.com> References: <55F9EDD6.9050702@oracle.com> Message-ID: <55F9F4F3.8030609@oracle.com> Jon, let's reverse the question - why do you think it *should* live in langtools? Honestly it feels very different from all other tools that live there; there's dependencies on JDI, jline - so to me it feels closer to other tools (i.e. jconsole) which already are in the jdk repo. I guess my argument is - if langtools defines the core tools to be able to work with the JDK, jshell feels like one ore two layer of abstractions on top of that. Maurizio On 16/09/15 23:31, Jonathan Gibbons wrote: > > > On 09/16/2015 03:06 PM, Robert Field wrote: >> As part of the JShell code review, Maurizio made the case that JShell >> should be in the JDK repo rather than langtools repo (where it is in >> the Kulla workspace). We would like additional feedback on this >> before proceeding -- >> >>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>> wrote: >>> >>> One general high-level comment, which I also pointed out elsewhere, >>> is that I'm not sure jshell really belongs in langtools; while it's >>> semantically (obviously) related to langtools - it is a rather >>> different beasts w.r.t. all other tools in langtools-land; the fact >>> that it depends on the JDK (for jline, and for JDI in general) makes >>> it very hard to run on top of a random JDK and then bootstrapping >>> classes - which is a technique widely used to be able to run >>> langtools tools w/o having to do a full build cycle. More >>> specifically, talking about IDE integration, I don't see how >>> IntelliJ/Netbeans langtools projects (and the langtools internal >>> developer ant build) could be updated to be able to run/debug jshell >>> w/o a full build. >> >> > > In the Big Picture view of the universe, folk would like to > restructure the repos in an OPenJDK forest, at which point the > distinction between the langtools and jdk repos will likely disappear, > or at least become very (very) blurred. While that change is not > imminent, IMO opinion, it significantly reduces the impact of the > reasons why JShell should not be in langtools. > > I also don't entirely agree with the IDE reasons either. If you want > to work on JShell in an IDE, you're going to have to do what it takes > to get rid of the red squiggly lines anyway, whatever repo the code is > in. To keep JShell out of Langtools because it doesn't fit our current > methodologies is a case of the tail wagging the dog. > > Also, I note that JDK 9 now builds the product module by module. In > times past, we built the world repo by repo, and that would indeed > have been good reason to keep JShell out of langtools, because of the > dependencies of JShell on JDK 9 API, like jline. But now, the build > will build "interim javac", and can then build JDK modules like > java.base and whatever module jline ends up in, and can then build the > module containing JShell. > > As to how to organize IDE projects, in the worst case, we keep the > langtools IDE setup as it is now (no JShell) and have a separate > project for JShell itself. That is tantamount to what we would do if > we push JShell into the jdk repo. > > So, I'm saying all that while trying to be agnostic on which repo it > should live in. I don't really agree with the stated reasons why it > should not live in langtools, which means that we can look for other > reasons to choose one repo or another. > > -- Jon From joe.darcy at oracle.com Wed Sep 16 23:28:38 2015 From: joe.darcy at oracle.com (Joseph D. Darcy) Date: Wed, 16 Sep 2015 16:28:38 -0700 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55F9EDD6.9050702@oracle.com> References: <55F9EDD6.9050702@oracle.com> Message-ID: <55F9FB26.9030903@oracle.com> On 9/16/2015 3:31 PM, Jonathan Gibbons wrote: > > > On 09/16/2015 03:06 PM, Robert Field wrote: >> As part of the JShell code review, Maurizio made the case that JShell >> should be in the JDK repo rather than langtools repo (where it is in >> the Kulla workspace). We would like additional feedback on this >> before proceeding -- >> >>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>> wrote: >>> >>> One general high-level comment, which I also pointed out elsewhere, >>> is that I'm not sure jshell really belongs in langtools; while it's >>> semantically (obviously) related to langtools - it is a rather >>> different beasts w.r.t. all other tools in langtools-land; the fact >>> that it depends on the JDK (for jline, and for JDI in general) makes >>> it very hard to run on top of a random JDK and then bootstrapping >>> classes - which is a technique widely used to be able to run >>> langtools tools w/o having to do a full build cycle. More >>> specifically, talking about IDE integration, I don't see how >>> IntelliJ/Netbeans langtools projects (and the langtools internal >>> developer ant build) could be updated to be able to run/debug jshell >>> w/o a full build. >> >> > > In the Big Picture view of the universe, folk would like to > restructure the repos in an OPenJDK forest, at which point the > distinction between the langtools and jdk repos will likely disappear, > or at least become very (very) blurred. While that change is not > imminent, IMO opinion, it significantly reduces the impact of the > reasons why JShell should not be in langtools. As an aside, I am one of those folk who favor such a restructuring ;-) As a strawman proposal, I think the root, jdk, langtools, and hotspot repos are one logical unit that should be combined together into a single repo. However, I don't expect this to change any time before JDK 10. Cheers, -Joe From jonathan.gibbons at oracle.com Thu Sep 17 00:04:00 2015 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 16 Sep 2015 17:04:00 -0700 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55F9F4F3.8030609@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> Message-ID: <55FA0370.2000805@oracle.com> I didn't say I think JShell should be in langtools, and I thought I was being careful to be very explicit about that. I was just trying to make sure we were making the right choice for the right reasons. The fact that it depends on JDI and jline is IMO a red herring. If they were both already in JDK 8 (i.e. the boot JDK) would we be having this discussion? To me, the bottom line is that I don't think it matters where the code lives, and as a result I personally don't care and have no opinion where the code should live. I /do/ think that whereever we choose to put it, we can make everything work well from a developer standpoint, without inconveniencing any existing developer usage, although I concede that we might choose not to integrate it with the langtools make/build.xml infrastructure and/or the existing NetBeans project. And from a somewhat selfish standpoint, I think we have somewhat more latitude to set things up the way we like it in the langtools repo than we would in the jdk repo. I agree that JShell is higher up in the layers of abstraction than javac, but I also don't think that the langtools repo is defined to be "core tools to be able to work with the JDK". The provenance of the repo (as indicated by its name) is that it was code that was managed by the old Sun Language Tools Group, from way back when, meaning that it was more of a "team" repository than a repository defined by a layer of abstraction. So, for me, I think this comes down to, "where does the Kulla dev team think it would be most convenient for them to keep and work on the code?". Once we can answer that, we can move on to, "what is the most convenient way to satisfy all the requirements of all the interested parties?" -- Jon On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote: > Jon, let's reverse the question - why do you think it *should* live in > langtools? Honestly it feels very different from all other tools that > live there; there's dependencies on JDI, jline - so to me it feels > closer to other tools (i.e. jconsole) which already are in the jdk > repo. I guess my argument is - if langtools defines the core tools to > be able to work with the JDK, jshell feels like one ore two layer of > abstractions on top of that. > > Maurizio > > On 16/09/15 23:31, Jonathan Gibbons wrote: >> >> >> On 09/16/2015 03:06 PM, Robert Field wrote: >>> As part of the JShell code review, Maurizio made the case that >>> JShell should be in the JDK repo rather than langtools repo (where >>> it is in the Kulla workspace). We would like additional feedback on >>> this before proceeding -- >>> >>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>>> wrote: >>>> >>>> One general high-level comment, which I also pointed out elsewhere, >>>> is that I'm not sure jshell really belongs in langtools; while it's >>>> semantically (obviously) related to langtools - it is a rather >>>> different beasts w.r.t. all other tools in langtools-land; the fact >>>> that it depends on the JDK (for jline, and for JDI in general) >>>> makes it very hard to run on top of a random JDK and then >>>> bootstrapping classes - which is a technique widely used to be able >>>> to run langtools tools w/o having to do a full build cycle. More >>>> specifically, talking about IDE integration, I don't see how >>>> IntelliJ/Netbeans langtools projects (and the langtools internal >>>> developer ant build) could be updated to be able to run/debug >>>> jshell w/o a full build. >>> >>> >> >> In the Big Picture view of the universe, folk would like to >> restructure the repos in an OPenJDK forest, at which point the >> distinction between the langtools and jdk repos will likely >> disappear, or at least become very (very) blurred. While that change >> is not imminent, IMO opinion, it significantly reduces the impact of >> the reasons why JShell should not be in langtools. >> >> I also don't entirely agree with the IDE reasons either. If you want >> to work on JShell in an IDE, you're going to have to do what it takes >> to get rid of the red squiggly lines anyway, whatever repo the code >> is in. To keep JShell out of Langtools because it doesn't fit our >> current methodologies is a case of the tail wagging the dog. >> >> Also, I note that JDK 9 now builds the product module by module. In >> times past, we built the world repo by repo, and that would indeed >> have been good reason to keep JShell out of langtools, because of the >> dependencies of JShell on JDK 9 API, like jline. But now, the build >> will build "interim javac", and can then build JDK modules like >> java.base and whatever module jline ends up in, and can then build >> the module containing JShell. >> >> As to how to organize IDE projects, in the worst case, we keep the >> langtools IDE setup as it is now (no JShell) and have a separate >> project for JShell itself. That is tantamount to what we would do if >> we push JShell into the jdk repo. >> >> So, I'm saying all that while trying to be agnostic on which repo it >> should live in. I don't really agree with the stated reasons why >> it should not live in langtools, which means that we can look for >> other reasons to choose one repo or another. >> >> -- Jon > From maurizio.cimadamore at oracle.com Thu Sep 17 00:38:52 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Thu, 17 Sep 2015 01:38:52 +0100 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FA0370.2000805@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> Message-ID: <55FA0B9C.3090206@oracle.com> While I agree with many points you raise - I feel this discussion is very abstract; please try to run all langtools tests in the kulla langtools repo using your standard scripts - does everything still work for you? I found that I could not run tests using my usual scripts. The only way to run jshell tests reliably is through make. Now, if we were in a world where 100% langtools developers ran tests that way - I would agree with you; but the reality is somewhat different, and many developers are used to run langtools tests using some JDK N snapshot and bootstrapping the required classes. That technique WILL cease to work here. I don't think we can say to those developer - just don't run kulla tests and be happy (what if some of their changes accidentally broke something in kulla?). Are you saying that those developers should just stop doing what they are doing and start running langtools tests as part of a full JDK build and test cycle? That's a fine answer - I guess I'm just trying to guess where you are coming from. Maurizio On 17/09/15 01:04, Jonathan Gibbons wrote: > I didn't say I think JShell should be in langtools, and I thought I > was being careful to be very explicit about that. I was just trying to > make sure we were making the right choice for the right reasons. > > The fact that it depends on JDI and jline is IMO a red herring. If > they were both already in JDK 8 (i.e. the boot JDK) would we be having > this discussion? > > To me, the bottom line is that I don't think it matters where the code > lives, and as a result I personally don't care and have no opinion > where the code should live. I /do/ think that whereever we choose to > put it, we can make everything work well from a developer standpoint, > without inconveniencing any existing developer usage, although I > concede that we might choose not to integrate it with the langtools > make/build.xml infrastructure and/or the existing NetBeans project. > And from a somewhat selfish standpoint, I think we have somewhat more > latitude to set things up the way we like it in the langtools repo > than we would in the jdk repo. > > I agree that JShell is higher up in the layers of abstraction than > javac, but I also don't think that the langtools repo is defined to be > "core tools to be able to work with the JDK". The provenance of the > repo (as indicated by its name) is that it was code that was managed > by the old Sun Language Tools Group, from way back when, meaning that > it was more of a "team" repository than a repository defined by a > layer of abstraction. > > So, for me, I think this comes down to, "where does the Kulla dev team > think it would be most convenient for them to keep and work on the > code?". Once we can answer that, we can move on to, "what is the most > convenient way to satisfy all the requirements of all the interested > parties?" > > -- Jon > > > On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote: >> Jon, let's reverse the question - why do you think it *should* live >> in langtools? Honestly it feels very different from all other tools >> that live there; there's dependencies on JDI, jline - so to me it >> feels closer to other tools (i.e. jconsole) which already are in the >> jdk repo. I guess my argument is - if langtools defines the core >> tools to be able to work with the JDK, jshell feels like one ore two >> layer of abstractions on top of that. >> >> Maurizio >> >> On 16/09/15 23:31, Jonathan Gibbons wrote: >>> >>> >>> On 09/16/2015 03:06 PM, Robert Field wrote: >>>> As part of the JShell code review, Maurizio made the case that >>>> JShell should be in the JDK repo rather than langtools repo (where >>>> it is in the Kulla workspace). We would like additional feedback >>>> on this before proceeding -- >>>> >>>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>>>> wrote: >>>>> >>>>> One general high-level comment, which I also pointed out >>>>> elsewhere, is that I'm not sure jshell really belongs in >>>>> langtools; while it's semantically (obviously) related to >>>>> langtools - it is a rather different beasts w.r.t. all other tools >>>>> in langtools-land; the fact that it depends on the JDK (for jline, >>>>> and for JDI in general) makes it very hard to run on top of a >>>>> random JDK and then bootstrapping classes - which is a technique >>>>> widely used to be able to run langtools tools w/o having to do a >>>>> full build cycle. More specifically, talking about IDE >>>>> integration, I don't see how IntelliJ/Netbeans langtools projects >>>>> (and the langtools internal developer ant build) could be updated >>>>> to be able to run/debug jshell w/o a full build. >>>> >>>> >>> >>> In the Big Picture view of the universe, folk would like to >>> restructure the repos in an OPenJDK forest, at which point the >>> distinction between the langtools and jdk repos will likely >>> disappear, or at least become very (very) blurred. While that >>> change is not imminent, IMO opinion, it significantly reduces the >>> impact of the reasons why JShell should not be in langtools. >>> >>> I also don't entirely agree with the IDE reasons either. If you >>> want to work on JShell in an IDE, you're going to have to do what it >>> takes to get rid of the red squiggly lines anyway, whatever repo the >>> code is in. To keep JShell out of Langtools because it doesn't fit >>> our current methodologies is a case of the tail wagging the dog. >>> >>> Also, I note that JDK 9 now builds the product module by module. In >>> times past, we built the world repo by repo, and that would indeed >>> have been good reason to keep JShell out of langtools, because of >>> the dependencies of JShell on JDK 9 API, like jline. But now, the >>> build will build "interim javac", and can then build JDK modules >>> like java.base and whatever module jline ends up in, and can then >>> build the module containing JShell. >>> >>> As to how to organize IDE projects, in the worst case, we keep the >>> langtools IDE setup as it is now (no JShell) and have a separate >>> project for JShell itself. That is tantamount to what we would do if >>> we push JShell into the jdk repo. >>> >>> So, I'm saying all that while trying to be agnostic on which repo it >>> should live in. I don't really agree with the stated reasons why >>> it should not live in langtools, which means that we can look for >>> other reasons to choose one repo or another. >>> >>> -- Jon >> > From robert.field at oracle.com Thu Sep 17 01:26:06 2015 From: robert.field at oracle.com (Robert Field) Date: Wed, 16 Sep 2015 18:26:06 -0700 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FA0B9C.3090206@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> Message-ID: <55FA16AE.3030002@oracle.com> Testing -- point point. Testing seems like a good perspective to analyze this from. The grouping (langtools vs JDK) is more likely to be tested together. The chance that a change in JDI would break JShell is very unlikely. JShell uses JDI only through the API and only a small part of it. JDI is stable. JShell uses jline only through the API. It in an internal copy of a public API for use by Nashorn and JShell. If the jline version got updated and was incompatible that would be the first thing checked. JShell uses the compiler API but also subclasses several javac classes and uses internal javac data types directly. The chance that some subtle change to javac could break JShell in an unexpected way is very real. JShell testing should be part of testing a change to javac just as testing javadoc and other directly downstream tools. The degree to which JShell interfaces with the compiler is vastly greater than the degree to which it interfaces with JDI or jline. -Robert On 09/16/15 17:38, Maurizio Cimadamore wrote: > While I agree with many points you raise - I feel this discussion is > very abstract; please try to run all langtools tests in the kulla > langtools repo using your standard scripts - does everything still > work for you? I found that I could not run tests using my usual > scripts. The only way to run jshell tests reliably is through make. > Now, if we were in a world where 100% langtools developers ran tests > that way - I would agree with you; but the reality is somewhat > different, and many developers are used to run langtools tests using > some JDK N snapshot and bootstrapping the required classes. That > technique WILL cease to work here. I don't think we can say to those > developer - just don't run kulla tests and be happy (what if some of > their changes accidentally broke something in kulla?). Are you saying > that those developers should just stop doing what they are doing and > start running langtools tests as part of a full JDK build and test > cycle? That's a fine answer - I guess I'm just trying to guess where > you are coming from. > > Maurizio > > On 17/09/15 01:04, Jonathan Gibbons wrote: >> I didn't say I think JShell should be in langtools, and I thought I >> was being careful to be very explicit about that. I was just trying >> to make sure we were making the right choice for the right reasons. >> >> The fact that it depends on JDI and jline is IMO a red herring. If >> they were both already in JDK 8 (i.e. the boot JDK) would we be >> having this discussion? >> >> To me, the bottom line is that I don't think it matters where the >> code lives, and as a result I personally don't care and have no >> opinion where the code should live. I /do/ think that whereever we >> choose to put it, we can make everything work well from a developer >> standpoint, without inconveniencing any existing developer usage, >> although I concede that we might choose not to integrate it with the >> langtools make/build.xml infrastructure and/or the existing NetBeans >> project. And from a somewhat selfish standpoint, I think we have >> somewhat more latitude to set things up the way we like it in the >> langtools repo than we would in the jdk repo. >> >> I agree that JShell is higher up in the layers of abstraction than >> javac, but I also don't think that the langtools repo is defined to >> be "core tools to be able to work with the JDK". The provenance of >> the repo (as indicated by its name) is that it was code that was >> managed by the old Sun Language Tools Group, from way back when, >> meaning that it was more of a "team" repository than a repository >> defined by a layer of abstraction. >> >> So, for me, I think this comes down to, "where does the Kulla dev >> team think it would be most convenient for them to keep and work on >> the code?". Once we can answer that, we can move on to, "what is the >> most convenient way to satisfy all the requirements of all the >> interested parties?" >> >> -- Jon >> >> >> On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote: >>> Jon, let's reverse the question - why do you think it *should* live >>> in langtools? Honestly it feels very different from all other tools >>> that live there; there's dependencies on JDI, jline - so to me it >>> feels closer to other tools (i.e. jconsole) which already are in the >>> jdk repo. I guess my argument is - if langtools defines the core >>> tools to be able to work with the JDK, jshell feels like one ore two >>> layer of abstractions on top of that. >>> >>> Maurizio >>> >>> On 16/09/15 23:31, Jonathan Gibbons wrote: >>>> >>>> >>>> On 09/16/2015 03:06 PM, Robert Field wrote: >>>>> As part of the JShell code review, Maurizio made the case that >>>>> JShell should be in the JDK repo rather than langtools repo (where >>>>> it is in the Kulla workspace). We would like additional feedback >>>>> on this before proceeding -- >>>>> >>>>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>>>>> wrote: >>>>>> >>>>>> One general high-level comment, which I also pointed out >>>>>> elsewhere, is that I'm not sure jshell really belongs in >>>>>> langtools; while it's semantically (obviously) related to >>>>>> langtools - it is a rather different beasts w.r.t. all other >>>>>> tools in langtools-land; the fact that it depends on the JDK (for >>>>>> jline, and for JDI in general) makes it very hard to run on top >>>>>> of a random JDK and then bootstrapping classes - which is a >>>>>> technique widely used to be able to run langtools tools w/o >>>>>> having to do a full build cycle. More specifically, talking about >>>>>> IDE integration, I don't see how IntelliJ/Netbeans langtools >>>>>> projects (and the langtools internal developer ant build) could >>>>>> be updated to be able to run/debug jshell w/o a full build. >>>>> >>>>> >>>> >>>> In the Big Picture view of the universe, folk would like to >>>> restructure the repos in an OPenJDK forest, at which point the >>>> distinction between the langtools and jdk repos will likely >>>> disappear, or at least become very (very) blurred. While that >>>> change is not imminent, IMO opinion, it significantly reduces the >>>> impact of the reasons why JShell should not be in langtools. >>>> >>>> I also don't entirely agree with the IDE reasons either. If you >>>> want to work on JShell in an IDE, you're going to have to do what >>>> it takes to get rid of the red squiggly lines anyway, whatever repo >>>> the code is in. To keep JShell out of Langtools because it doesn't >>>> fit our current methodologies is a case of the tail wagging the dog. >>>> >>>> Also, I note that JDK 9 now builds the product module by module. In >>>> times past, we built the world repo by repo, and that would indeed >>>> have been good reason to keep JShell out of langtools, because of >>>> the dependencies of JShell on JDK 9 API, like jline. But now, the >>>> build will build "interim javac", and can then build JDK modules >>>> like java.base and whatever module jline ends up in, and can then >>>> build the module containing JShell. >>>> >>>> As to how to organize IDE projects, in the worst case, we keep the >>>> langtools IDE setup as it is now (no JShell) and have a separate >>>> project for JShell itself. That is tantamount to what we would do >>>> if we push JShell into the jdk repo. >>>> >>>> So, I'm saying all that while trying to be agnostic on which repo >>>> it should live in. I don't really agree with the stated reasons >>>> why it should not live in langtools, which means that we can look >>>> for other reasons to choose one repo or another. >>>> >>>> -- Jon >>> >> > From jonathan.gibbons at oracle.com Thu Sep 17 01:56:34 2015 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 16 Sep 2015 18:56:34 -0700 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FA0B9C.3090206@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> Message-ID: <55FA1DD2.5010602@oracle.com> I guess I don't understand why testing won't work as expected, provided you have a sufficiently recent snapshot of JDK to use as a baseline. In this case, the requirement is that the JDK baseline has jline and whatever other recent API has been added to JDK. If kulla is routinely adding and modifying API in the jdk repo, then I agree that would tend to suggest that the source bits should be in the jdk repo as well. But I would have thought that the kulla additions to the jdk repo will stabilize very quickly, and so after one or two builds and promotions, if JShell ends up in langtools, then it will be easy enough to run all the tests using a recent build of JDK. I agree that we should not expect langtools developers to have to use "make test" and I agree that we should not expect langtools developers to exclude the JShell tests. That implies we should look at and understand why the JShell tests might fail if they are co-located in the langtools directory. If it is just because jline is not yet standard in JDK builds, that's a temporary state of affairs that we will get past soon enough. If there's something else going on, I guess I'd like to better understand what that might be, so that we can see if we can fix the issue. -- Jon On 09/16/2015 05:38 PM, Maurizio Cimadamore wrote: > While I agree with many points you raise - I feel this discussion is > very abstract; please try to run all langtools tests in the kulla > langtools repo using your standard scripts - does everything still > work for you? I found that I could not run tests using my usual > scripts. The only way to run jshell tests reliably is through make. > Now, if we were in a world where 100% langtools developers ran tests > that way - I would agree with you; but the reality is somewhat > different, and many developers are used to run langtools tests using > some JDK N snapshot and bootstrapping the required classes. That > technique WILL cease to work here. I don't think we can say to those > developer - just don't run kulla tests and be happy (what if some of > their changes accidentally broke something in kulla?). Are you saying > that those developers should just stop doing what they are doing and > start running langtools tests as part of a full JDK build and test > cycle? That's a fine answer - I guess I'm just trying to guess where > you are coming from. > > Maurizio > > On 17/09/15 01:04, Jonathan Gibbons wrote: >> I didn't say I think JShell should be in langtools, and I thought I >> was being careful to be very explicit about that. I was just trying >> to make sure we were making the right choice for the right reasons. >> >> The fact that it depends on JDI and jline is IMO a red herring. If >> they were both already in JDK 8 (i.e. the boot JDK) would we be >> having this discussion? >> >> To me, the bottom line is that I don't think it matters where the >> code lives, and as a result I personally don't care and have no >> opinion where the code should live. I /do/ think that whereever we >> choose to put it, we can make everything work well from a developer >> standpoint, without inconveniencing any existing developer usage, >> although I concede that we might choose not to integrate it with the >> langtools make/build.xml infrastructure and/or the existing NetBeans >> project. And from a somewhat selfish standpoint, I think we have >> somewhat more latitude to set things up the way we like it in the >> langtools repo than we would in the jdk repo. >> >> I agree that JShell is higher up in the layers of abstraction than >> javac, but I also don't think that the langtools repo is defined to >> be "core tools to be able to work with the JDK". The provenance of >> the repo (as indicated by its name) is that it was code that was >> managed by the old Sun Language Tools Group, from way back when, >> meaning that it was more of a "team" repository than a repository >> defined by a layer of abstraction. >> >> So, for me, I think this comes down to, "where does the Kulla dev >> team think it would be most convenient for them to keep and work on >> the code?". Once we can answer that, we can move on to, "what is the >> most convenient way to satisfy all the requirements of all the >> interested parties?" >> >> -- Jon >> >> >> On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote: >>> Jon, let's reverse the question - why do you think it *should* live >>> in langtools? Honestly it feels very different from all other tools >>> that live there; there's dependencies on JDI, jline - so to me it >>> feels closer to other tools (i.e. jconsole) which already are in the >>> jdk repo. I guess my argument is - if langtools defines the core >>> tools to be able to work with the JDK, jshell feels like one ore two >>> layer of abstractions on top of that. >>> >>> Maurizio >>> >>> On 16/09/15 23:31, Jonathan Gibbons wrote: >>>> >>>> >>>> On 09/16/2015 03:06 PM, Robert Field wrote: >>>>> As part of the JShell code review, Maurizio made the case that >>>>> JShell should be in the JDK repo rather than langtools repo (where >>>>> it is in the Kulla workspace). We would like additional feedback >>>>> on this before proceeding -- >>>>> >>>>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>>>>> wrote: >>>>>> >>>>>> One general high-level comment, which I also pointed out >>>>>> elsewhere, is that I'm not sure jshell really belongs in >>>>>> langtools; while it's semantically (obviously) related to >>>>>> langtools - it is a rather different beasts w.r.t. all other >>>>>> tools in langtools-land; the fact that it depends on the JDK (for >>>>>> jline, and for JDI in general) makes it very hard to run on top >>>>>> of a random JDK and then bootstrapping classes - which is a >>>>>> technique widely used to be able to run langtools tools w/o >>>>>> having to do a full build cycle. More specifically, talking about >>>>>> IDE integration, I don't see how IntelliJ/Netbeans langtools >>>>>> projects (and the langtools internal developer ant build) could >>>>>> be updated to be able to run/debug jshell w/o a full build. >>>>> >>>>> >>>> >>>> In the Big Picture view of the universe, folk would like to >>>> restructure the repos in an OPenJDK forest, at which point the >>>> distinction between the langtools and jdk repos will likely >>>> disappear, or at least become very (very) blurred. While that >>>> change is not imminent, IMO opinion, it significantly reduces the >>>> impact of the reasons why JShell should not be in langtools. >>>> >>>> I also don't entirely agree with the IDE reasons either. If you >>>> want to work on JShell in an IDE, you're going to have to do what >>>> it takes to get rid of the red squiggly lines anyway, whatever repo >>>> the code is in. To keep JShell out of Langtools because it doesn't >>>> fit our current methodologies is a case of the tail wagging the dog. >>>> >>>> Also, I note that JDK 9 now builds the product module by module. In >>>> times past, we built the world repo by repo, and that would indeed >>>> have been good reason to keep JShell out of langtools, because of >>>> the dependencies of JShell on JDK 9 API, like jline. But now, the >>>> build will build "interim javac", and can then build JDK modules >>>> like java.base and whatever module jline ends up in, and can then >>>> build the module containing JShell. >>>> >>>> As to how to organize IDE projects, in the worst case, we keep the >>>> langtools IDE setup as it is now (no JShell) and have a separate >>>> project for JShell itself. That is tantamount to what we would do >>>> if we push JShell into the jdk repo. >>>> >>>> So, I'm saying all that while trying to be agnostic on which repo >>>> it should live in. I don't really agree with the stated reasons >>>> why it should not live in langtools, which means that we can look >>>> for other reasons to choose one repo or another. >>>> >>>> -- Jon >>> >> > From robert.field at oracle.com Thu Sep 17 02:03:11 2015 From: robert.field at oracle.com (Robert Field) Date: Wed, 16 Sep 2015 19:03:11 -0700 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FA1DD2.5010602@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> Message-ID: <55FA1F5F.70800@oracle.com> See the email I just sent speaking to some of this. As to JShell causing changes in the JDK repo, that would be rare. JDI is existing, stable, and unchanged for JShell. jline is a snapshot of the external jline with a tiny tweak to integrate it cleanly. Only a version update would likely change that. In contrast, JShell has required changes to javac code. -Robert On 09/16/15 18:56, Jonathan Gibbons wrote: > I guess I don't understand why testing won't work as expected, > provided you have a sufficiently recent snapshot of JDK to use as a > baseline. In this case, the requirement is that the JDK baseline has > jline and whatever other recent API has been added to JDK. > > If kulla is routinely adding and modifying API in the jdk repo, then I > agree that would tend to suggest that the source bits should be in the > jdk repo as well. But I would have thought that the kulla additions > to the jdk repo will stabilize very quickly, and so after one or two > builds and promotions, if JShell ends up in langtools, then it will be > easy enough to run all the tests using a recent build of JDK. > > I agree that we should not expect langtools developers to have to use > "make test" and I agree that we should not expect langtools developers > to exclude the JShell tests. That implies we should look at and > understand why the JShell tests might fail if they are co-located in > the langtools directory. If it is just because jline is not yet > standard in JDK builds, that's a temporary state of affairs that we > will get past soon enough. If there's something else going on, I > guess I'd like to better understand what that might be, so that we can > see if we can fix the issue. > > -- Jon > > On 09/16/2015 05:38 PM, Maurizio Cimadamore wrote: >> While I agree with many points you raise - I feel this discussion is >> very abstract; please try to run all langtools tests in the kulla >> langtools repo using your standard scripts - does everything still >> work for you? I found that I could not run tests using my usual >> scripts. The only way to run jshell tests reliably is through make. >> Now, if we were in a world where 100% langtools developers ran tests >> that way - I would agree with you; but the reality is somewhat >> different, and many developers are used to run langtools tests using >> some JDK N snapshot and bootstrapping the required classes. That >> technique WILL cease to work here. I don't think we can say to those >> developer - just don't run kulla tests and be happy (what if some of >> their changes accidentally broke something in kulla?). Are you saying >> that those developers should just stop doing what they are doing and >> start running langtools tests as part of a full JDK build and test >> cycle? That's a fine answer - I guess I'm just trying to guess where >> you are coming from. >> >> Maurizio >> >> On 17/09/15 01:04, Jonathan Gibbons wrote: >>> I didn't say I think JShell should be in langtools, and I thought I >>> was being careful to be very explicit about that. I was just trying >>> to make sure we were making the right choice for the right reasons. >>> >>> The fact that it depends on JDI and jline is IMO a red herring. If >>> they were both already in JDK 8 (i.e. the boot JDK) would we be >>> having this discussion? >>> >>> To me, the bottom line is that I don't think it matters where the >>> code lives, and as a result I personally don't care and have no >>> opinion where the code should live. I /do/ think that whereever we >>> choose to put it, we can make everything work well from a developer >>> standpoint, without inconveniencing any existing developer usage, >>> although I concede that we might choose not to integrate it with the >>> langtools make/build.xml infrastructure and/or the existing NetBeans >>> project. And from a somewhat selfish standpoint, I think we have >>> somewhat more latitude to set things up the way we like it in the >>> langtools repo than we would in the jdk repo. >>> >>> I agree that JShell is higher up in the layers of abstraction than >>> javac, but I also don't think that the langtools repo is defined to >>> be "core tools to be able to work with the JDK". The provenance of >>> the repo (as indicated by its name) is that it was code that was >>> managed by the old Sun Language Tools Group, from way back when, >>> meaning that it was more of a "team" repository than a repository >>> defined by a layer of abstraction. >>> >>> So, for me, I think this comes down to, "where does the Kulla dev >>> team think it would be most convenient for them to keep and work on >>> the code?". Once we can answer that, we can move on to, "what is the >>> most convenient way to satisfy all the requirements of all the >>> interested parties?" >>> >>> -- Jon >>> >>> >>> On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote: >>>> Jon, let's reverse the question - why do you think it *should* live >>>> in langtools? Honestly it feels very different from all other tools >>>> that live there; there's dependencies on JDI, jline - so to me it >>>> feels closer to other tools (i.e. jconsole) which already are in >>>> the jdk repo. I guess my argument is - if langtools defines the >>>> core tools to be able to work with the JDK, jshell feels like one >>>> ore two layer of abstractions on top of that. >>>> >>>> Maurizio >>>> >>>> On 16/09/15 23:31, Jonathan Gibbons wrote: >>>>> >>>>> >>>>> On 09/16/2015 03:06 PM, Robert Field wrote: >>>>>> As part of the JShell code review, Maurizio made the case that >>>>>> JShell should be in the JDK repo rather than langtools repo >>>>>> (where it is in the Kulla workspace). We would like additional >>>>>> feedback on this before proceeding -- >>>>>> >>>>>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>>>>>> wrote: >>>>>>> >>>>>>> One general high-level comment, which I also pointed out >>>>>>> elsewhere, is that I'm not sure jshell really belongs in >>>>>>> langtools; while it's semantically (obviously) related to >>>>>>> langtools - it is a rather different beasts w.r.t. all other >>>>>>> tools in langtools-land; the fact that it depends on the JDK >>>>>>> (for jline, and for JDI in general) makes it very hard to run on >>>>>>> top of a random JDK and then bootstrapping classes - which is a >>>>>>> technique widely used to be able to run langtools tools w/o >>>>>>> having to do a full build cycle. More specifically, talking >>>>>>> about IDE integration, I don't see how IntelliJ/Netbeans >>>>>>> langtools projects (and the langtools internal developer ant >>>>>>> build) could be updated to be able to run/debug jshell w/o a >>>>>>> full build. >>>>>> >>>>>> >>>>> >>>>> In the Big Picture view of the universe, folk would like to >>>>> restructure the repos in an OPenJDK forest, at which point the >>>>> distinction between the langtools and jdk repos will likely >>>>> disappear, or at least become very (very) blurred. While that >>>>> change is not imminent, IMO opinion, it significantly reduces the >>>>> impact of the reasons why JShell should not be in langtools. >>>>> >>>>> I also don't entirely agree with the IDE reasons either. If you >>>>> want to work on JShell in an IDE, you're going to have to do what >>>>> it takes to get rid of the red squiggly lines anyway, whatever >>>>> repo the code is in. To keep JShell out of Langtools because it >>>>> doesn't fit our current methodologies is a case of the tail >>>>> wagging the dog. >>>>> >>>>> Also, I note that JDK 9 now builds the product module by module. >>>>> In times past, we built the world repo by repo, and that would >>>>> indeed have been good reason to keep JShell out of langtools, >>>>> because of the dependencies of JShell on JDK 9 API, like jline. >>>>> But now, the build will build "interim javac", and can then build >>>>> JDK modules like java.base and whatever module jline ends up in, >>>>> and can then build the module containing JShell. >>>>> >>>>> As to how to organize IDE projects, in the worst case, we keep the >>>>> langtools IDE setup as it is now (no JShell) and have a separate >>>>> project for JShell itself. That is tantamount to what we would do >>>>> if we push JShell into the jdk repo. >>>>> >>>>> So, I'm saying all that while trying to be agnostic on which repo >>>>> it should live in. I don't really agree with the stated reasons >>>>> why it should not live in langtools, which means that we can look >>>>> for other reasons to choose one repo or another. >>>>> >>>>> -- Jon >>>> >>> >> > From jonathan.gibbons at oracle.com Thu Sep 17 02:08:21 2015 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 16 Sep 2015 19:08:21 -0700 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FA1F5F.70800@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA1F5F.70800@oracle.com> Message-ID: <55FA2095.5030808@oracle.com> Yes, I generally agree with your comments that testing is a good perspective from which to decide where to put the jshell bits, and I agree with your comments that because jshell is tightly integrated with javac, that would indicate that langtools is the preferable home. But I also want to better understand Maurizio's comments as to whether and why he thinks there may be long term testing issues, so that we can work to address those issues. I'm not so fussed about any short term inconvenience for a couple of builds after kulla is integrated into jdk9/dev. -- Jon On 09/16/2015 07:03 PM, Robert Field wrote: > See the email I just sent speaking to some of this. > > As to JShell causing changes in the JDK repo, that would be rare. JDI > is existing, stable, and unchanged for JShell. jline is a snapshot of > the external jline with a tiny tweak to integrate it cleanly. Only a > version update would likely change that. In contrast, JShell has > required changes to javac code. > > -Robert > > On 09/16/15 18:56, Jonathan Gibbons wrote: >> I guess I don't understand why testing won't work as expected, >> provided you have a sufficiently recent snapshot of JDK to use as a >> baseline. In this case, the requirement is that the JDK baseline >> has jline and whatever other recent API has been added to JDK. >> >> If kulla is routinely adding and modifying API in the jdk repo, then >> I agree that would tend to suggest that the source bits should be in >> the jdk repo as well. But I would have thought that the kulla >> additions to the jdk repo will stabilize very quickly, and so after >> one or two builds and promotions, if JShell ends up in langtools, >> then it will be easy enough to run all the tests using a recent build >> of JDK. >> >> I agree that we should not expect langtools developers to have to use >> "make test" and I agree that we should not expect langtools >> developers to exclude the JShell tests. That implies we should look >> at and understand why the JShell tests might fail if they are >> co-located in the langtools directory. If it is just because jline >> is not yet standard in JDK builds, that's a temporary state of >> affairs that we will get past soon enough. If there's something else >> going on, I guess I'd like to better understand what that might be, >> so that we can see if we can fix the issue. >> >> -- Jon >> >> On 09/16/2015 05:38 PM, Maurizio Cimadamore wrote: >>> While I agree with many points you raise - I feel this discussion is >>> very abstract; please try to run all langtools tests in the kulla >>> langtools repo using your standard scripts - does everything still >>> work for you? I found that I could not run tests using my usual >>> scripts. The only way to run jshell tests reliably is through make. >>> Now, if we were in a world where 100% langtools developers ran tests >>> that way - I would agree with you; but the reality is somewhat >>> different, and many developers are used to run langtools tests using >>> some JDK N snapshot and bootstrapping the required classes. That >>> technique WILL cease to work here. I don't think we can say to those >>> developer - just don't run kulla tests and be happy (what if some of >>> their changes accidentally broke something in kulla?). Are you >>> saying that those developers should just stop doing what they are >>> doing and start running langtools tests as part of a full JDK build >>> and test cycle? That's a fine answer - I guess I'm just trying to >>> guess where you are coming from. >>> >>> Maurizio >>> >>> On 17/09/15 01:04, Jonathan Gibbons wrote: >>>> I didn't say I think JShell should be in langtools, and I thought I >>>> was being careful to be very explicit about that. I was just trying >>>> to make sure we were making the right choice for the right reasons. >>>> >>>> The fact that it depends on JDI and jline is IMO a red herring. If >>>> they were both already in JDK 8 (i.e. the boot JDK) would we be >>>> having this discussion? >>>> >>>> To me, the bottom line is that I don't think it matters where the >>>> code lives, and as a result I personally don't care and have no >>>> opinion where the code should live. I /do/ think that whereever we >>>> choose to put it, we can make everything work well from a developer >>>> standpoint, without inconveniencing any existing developer usage, >>>> although I concede that we might choose not to integrate it with >>>> the langtools make/build.xml infrastructure and/or the existing >>>> NetBeans project. And from a somewhat selfish standpoint, I think >>>> we have somewhat more latitude to set things up the way we like it >>>> in the langtools repo than we would in the jdk repo. >>>> >>>> I agree that JShell is higher up in the layers of abstraction than >>>> javac, but I also don't think that the langtools repo is defined to >>>> be "core tools to be able to work with the JDK". The provenance of >>>> the repo (as indicated by its name) is that it was code that was >>>> managed by the old Sun Language Tools Group, from way back when, >>>> meaning that it was more of a "team" repository than a repository >>>> defined by a layer of abstraction. >>>> >>>> So, for me, I think this comes down to, "where does the Kulla dev >>>> team think it would be most convenient for them to keep and work on >>>> the code?". Once we can answer that, we can move on to, "what is >>>> the most convenient way to satisfy all the requirements of all the >>>> interested parties?" >>>> >>>> -- Jon >>>> >>>> >>>> On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote: >>>>> Jon, let's reverse the question - why do you think it *should* >>>>> live in langtools? Honestly it feels very different from all other >>>>> tools that live there; there's dependencies on JDI, jline - so to >>>>> me it feels closer to other tools (i.e. jconsole) which already >>>>> are in the jdk repo. I guess my argument is - if langtools defines >>>>> the core tools to be able to work with the JDK, jshell feels like >>>>> one ore two layer of abstractions on top of that. >>>>> >>>>> Maurizio >>>>> >>>>> On 16/09/15 23:31, Jonathan Gibbons wrote: >>>>>> >>>>>> >>>>>> On 09/16/2015 03:06 PM, Robert Field wrote: >>>>>>> As part of the JShell code review, Maurizio made the case that >>>>>>> JShell should be in the JDK repo rather than langtools repo >>>>>>> (where it is in the Kulla workspace). We would like additional >>>>>>> feedback on this before proceeding -- >>>>>>> >>>>>>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>>>>>>> wrote: >>>>>>>> >>>>>>>> One general high-level comment, which I also pointed out >>>>>>>> elsewhere, is that I'm not sure jshell really belongs in >>>>>>>> langtools; while it's semantically (obviously) related to >>>>>>>> langtools - it is a rather different beasts w.r.t. all other >>>>>>>> tools in langtools-land; the fact that it depends on the JDK >>>>>>>> (for jline, and for JDI in general) makes it very hard to run >>>>>>>> on top of a random JDK and then bootstrapping classes - which >>>>>>>> is a technique widely used to be able to run langtools tools >>>>>>>> w/o having to do a full build cycle. More specifically, talking >>>>>>>> about IDE integration, I don't see how IntelliJ/Netbeans >>>>>>>> langtools projects (and the langtools internal developer ant >>>>>>>> build) could be updated to be able to run/debug jshell w/o a >>>>>>>> full build. >>>>>>> >>>>>>> >>>>>> >>>>>> In the Big Picture view of the universe, folk would like to >>>>>> restructure the repos in an OPenJDK forest, at which point the >>>>>> distinction between the langtools and jdk repos will likely >>>>>> disappear, or at least become very (very) blurred. While that >>>>>> change is not imminent, IMO opinion, it significantly reduces the >>>>>> impact of the reasons why JShell should not be in langtools. >>>>>> >>>>>> I also don't entirely agree with the IDE reasons either. If you >>>>>> want to work on JShell in an IDE, you're going to have to do what >>>>>> it takes to get rid of the red squiggly lines anyway, whatever >>>>>> repo the code is in. To keep JShell out of Langtools because it >>>>>> doesn't fit our current methodologies is a case of the tail >>>>>> wagging the dog. >>>>>> >>>>>> Also, I note that JDK 9 now builds the product module by module. >>>>>> In times past, we built the world repo by repo, and that would >>>>>> indeed have been good reason to keep JShell out of langtools, >>>>>> because of the dependencies of JShell on JDK 9 API, like jline. >>>>>> But now, the build will build "interim javac", and can then build >>>>>> JDK modules like java.base and whatever module jline ends up in, >>>>>> and can then build the module containing JShell. >>>>>> >>>>>> As to how to organize IDE projects, in the worst case, we keep >>>>>> the langtools IDE setup as it is now (no JShell) and have a >>>>>> separate project for JShell itself. That is tantamount to what we >>>>>> would do if we push JShell into the jdk repo. >>>>>> >>>>>> So, I'm saying all that while trying to be agnostic on which repo >>>>>> it should live in. I don't really agree with the stated >>>>>> reasons why it should not live in langtools, which means that we >>>>>> can look for other reasons to choose one repo or another. >>>>>> >>>>>> -- Jon >>>>> >>>> >>> >> > From jan.lahoda at oracle.com Thu Sep 17 06:26:33 2015 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Thu, 17 Sep 2015 08:26:33 +0200 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FA2095.5030808@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA1F5F.70800@oracle.com> <55FA2095.5030808@oracle.com> Message-ID: <55FA5D19.6040708@oracle.com> On 17.9.2015 04:08, Jonathan Gibbons wrote: > Yes, I generally agree with your comments that testing is a good > perspective from which to decide where to put the jshell bits, and I > agree with your comments that because jshell is tightly integrated with > javac, that would indicate that langtools is the preferable home. > > But I also want to better understand Maurizio's comments as to whether > and why he thinks there may be long term testing issues, so that we can > work to address those issues. I think that if we want to run jshell tests standalone similarly to e.g. javac tests (i.e. without doing the full JDK build), then we need to have a way to do a standalone build of jshell as well, similar to what we have for the other langtools modules. If we wouldn't have the standalone build, then the up-to-date jshell tests would run against the jshell code from the target JDK, and tests for bugs fixed recently could fail (if the bugfixes would be missing in the target JDK but would be present in the langtools checkout). Building jdk.jshell standalone is a little bit different from the other langtools modules, as the other langtools modules are built against the boot JDK, while jdk.jshell needs to be built against (presumably) the target JDK. My personal approach so far was to treat jdk.jshell as a module that requires full JDK build to run and test. Jan > > I'm not so fussed about any short term inconvenience for a couple of > builds after kulla is integrated into jdk9/dev. > > -- Jon > > On 09/16/2015 07:03 PM, Robert Field wrote: >> See the email I just sent speaking to some of this. >> >> As to JShell causing changes in the JDK repo, that would be rare. JDI >> is existing, stable, and unchanged for JShell. jline is a snapshot of >> the external jline with a tiny tweak to integrate it cleanly. Only a >> version update would likely change that. In contrast, JShell has >> required changes to javac code. >> >> -Robert >> >> On 09/16/15 18:56, Jonathan Gibbons wrote: >>> I guess I don't understand why testing won't work as expected, >>> provided you have a sufficiently recent snapshot of JDK to use as a >>> baseline. In this case, the requirement is that the JDK baseline >>> has jline and whatever other recent API has been added to JDK. >>> >>> If kulla is routinely adding and modifying API in the jdk repo, then >>> I agree that would tend to suggest that the source bits should be in >>> the jdk repo as well. But I would have thought that the kulla >>> additions to the jdk repo will stabilize very quickly, and so after >>> one or two builds and promotions, if JShell ends up in langtools, >>> then it will be easy enough to run all the tests using a recent build >>> of JDK. >>> >>> I agree that we should not expect langtools developers to have to use >>> "make test" and I agree that we should not expect langtools >>> developers to exclude the JShell tests. That implies we should look >>> at and understand why the JShell tests might fail if they are >>> co-located in the langtools directory. If it is just because jline >>> is not yet standard in JDK builds, that's a temporary state of >>> affairs that we will get past soon enough. If there's something else >>> going on, I guess I'd like to better understand what that might be, >>> so that we can see if we can fix the issue. >>> >>> -- Jon >>> >>> On 09/16/2015 05:38 PM, Maurizio Cimadamore wrote: >>>> While I agree with many points you raise - I feel this discussion is >>>> very abstract; please try to run all langtools tests in the kulla >>>> langtools repo using your standard scripts - does everything still >>>> work for you? I found that I could not run tests using my usual >>>> scripts. The only way to run jshell tests reliably is through make. >>>> Now, if we were in a world where 100% langtools developers ran tests >>>> that way - I would agree with you; but the reality is somewhat >>>> different, and many developers are used to run langtools tests using >>>> some JDK N snapshot and bootstrapping the required classes. That >>>> technique WILL cease to work here. I don't think we can say to those >>>> developer - just don't run kulla tests and be happy (what if some of >>>> their changes accidentally broke something in kulla?). Are you >>>> saying that those developers should just stop doing what they are >>>> doing and start running langtools tests as part of a full JDK build >>>> and test cycle? That's a fine answer - I guess I'm just trying to >>>> guess where you are coming from. >>>> >>>> Maurizio >>>> >>>> On 17/09/15 01:04, Jonathan Gibbons wrote: >>>>> I didn't say I think JShell should be in langtools, and I thought I >>>>> was being careful to be very explicit about that. I was just trying >>>>> to make sure we were making the right choice for the right reasons. >>>>> >>>>> The fact that it depends on JDI and jline is IMO a red herring. If >>>>> they were both already in JDK 8 (i.e. the boot JDK) would we be >>>>> having this discussion? >>>>> >>>>> To me, the bottom line is that I don't think it matters where the >>>>> code lives, and as a result I personally don't care and have no >>>>> opinion where the code should live. I /do/ think that whereever we >>>>> choose to put it, we can make everything work well from a developer >>>>> standpoint, without inconveniencing any existing developer usage, >>>>> although I concede that we might choose not to integrate it with >>>>> the langtools make/build.xml infrastructure and/or the existing >>>>> NetBeans project. And from a somewhat selfish standpoint, I think >>>>> we have somewhat more latitude to set things up the way we like it >>>>> in the langtools repo than we would in the jdk repo. >>>>> >>>>> I agree that JShell is higher up in the layers of abstraction than >>>>> javac, but I also don't think that the langtools repo is defined to >>>>> be "core tools to be able to work with the JDK". The provenance of >>>>> the repo (as indicated by its name) is that it was code that was >>>>> managed by the old Sun Language Tools Group, from way back when, >>>>> meaning that it was more of a "team" repository than a repository >>>>> defined by a layer of abstraction. >>>>> >>>>> So, for me, I think this comes down to, "where does the Kulla dev >>>>> team think it would be most convenient for them to keep and work on >>>>> the code?". Once we can answer that, we can move on to, "what is >>>>> the most convenient way to satisfy all the requirements of all the >>>>> interested parties?" >>>>> >>>>> -- Jon >>>>> >>>>> >>>>> On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote: >>>>>> Jon, let's reverse the question - why do you think it *should* >>>>>> live in langtools? Honestly it feels very different from all other >>>>>> tools that live there; there's dependencies on JDI, jline - so to >>>>>> me it feels closer to other tools (i.e. jconsole) which already >>>>>> are in the jdk repo. I guess my argument is - if langtools defines >>>>>> the core tools to be able to work with the JDK, jshell feels like >>>>>> one ore two layer of abstractions on top of that. >>>>>> >>>>>> Maurizio >>>>>> >>>>>> On 16/09/15 23:31, Jonathan Gibbons wrote: >>>>>>> >>>>>>> >>>>>>> On 09/16/2015 03:06 PM, Robert Field wrote: >>>>>>>> As part of the JShell code review, Maurizio made the case that >>>>>>>> JShell should be in the JDK repo rather than langtools repo >>>>>>>> (where it is in the Kulla workspace). We would like additional >>>>>>>> feedback on this before proceeding -- >>>>>>>> >>>>>>>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> One general high-level comment, which I also pointed out >>>>>>>>> elsewhere, is that I'm not sure jshell really belongs in >>>>>>>>> langtools; while it's semantically (obviously) related to >>>>>>>>> langtools - it is a rather different beasts w.r.t. all other >>>>>>>>> tools in langtools-land; the fact that it depends on the JDK >>>>>>>>> (for jline, and for JDI in general) makes it very hard to run >>>>>>>>> on top of a random JDK and then bootstrapping classes - which >>>>>>>>> is a technique widely used to be able to run langtools tools >>>>>>>>> w/o having to do a full build cycle. More specifically, talking >>>>>>>>> about IDE integration, I don't see how IntelliJ/Netbeans >>>>>>>>> langtools projects (and the langtools internal developer ant >>>>>>>>> build) could be updated to be able to run/debug jshell w/o a >>>>>>>>> full build. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> In the Big Picture view of the universe, folk would like to >>>>>>> restructure the repos in an OPenJDK forest, at which point the >>>>>>> distinction between the langtools and jdk repos will likely >>>>>>> disappear, or at least become very (very) blurred. While that >>>>>>> change is not imminent, IMO opinion, it significantly reduces the >>>>>>> impact of the reasons why JShell should not be in langtools. >>>>>>> >>>>>>> I also don't entirely agree with the IDE reasons either. If you >>>>>>> want to work on JShell in an IDE, you're going to have to do what >>>>>>> it takes to get rid of the red squiggly lines anyway, whatever >>>>>>> repo the code is in. To keep JShell out of Langtools because it >>>>>>> doesn't fit our current methodologies is a case of the tail >>>>>>> wagging the dog. >>>>>>> >>>>>>> Also, I note that JDK 9 now builds the product module by module. >>>>>>> In times past, we built the world repo by repo, and that would >>>>>>> indeed have been good reason to keep JShell out of langtools, >>>>>>> because of the dependencies of JShell on JDK 9 API, like jline. >>>>>>> But now, the build will build "interim javac", and can then build >>>>>>> JDK modules like java.base and whatever module jline ends up in, >>>>>>> and can then build the module containing JShell. >>>>>>> >>>>>>> As to how to organize IDE projects, in the worst case, we keep >>>>>>> the langtools IDE setup as it is now (no JShell) and have a >>>>>>> separate project for JShell itself. That is tantamount to what we >>>>>>> would do if we push JShell into the jdk repo. >>>>>>> >>>>>>> So, I'm saying all that while trying to be agnostic on which repo >>>>>>> it should live in. I don't really agree with the stated >>>>>>> reasons why it should not live in langtools, which means that we >>>>>>> can look for other reasons to choose one repo or another. >>>>>>> >>>>>>> -- Jon >>>>>> >>>>> >>>> >>> >> > From maurizio.cimadamore at oracle.com Thu Sep 17 08:17:22 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Thu, 17 Sep 2015 09:17:22 +0100 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FA1DD2.5010602@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> Message-ID: <55FA7712.4010704@oracle.com> On 17/09/15 02:56, Jonathan Gibbons wrote: > I guess I don't understand why testing won't work as expected, > provided you have a sufficiently recent snapshot of JDK to use as a > baseline. In this case, the requirement is that the JDK baseline has > jline and whatever other recent API has been added to JDK. There are many factors that contribute to this - the main one being JShell too not passing bootstrapping setting on the remote agent VM, plus some weird bug I encountered where if I set bootclasspath that mentions some jdk.internal classes (but which doesn't have JLine - because it comes from langtools), _every_ class from that package will not be fetched from the jimage file - meaning that I'll be left w/o jline. In other words, bootstrapping doesn't work - and I have reasons to believe that bootstrapping with the new module options (when they will become available - i.e. moduleOverride) won't work too. Maurizio > > If kulla is routinely adding and modifying API in the jdk repo, then I > agree that would tend to suggest that the source bits should be in the > jdk repo as well. But I would have thought that the kulla additions > to the jdk repo will stabilize very quickly, and so after one or two > builds and promotions, if JShell ends up in langtools, then it will be > easy enough to run all the tests using a recent build of JDK. > > I agree that we should not expect langtools developers to have to use > "make test" and I agree that we should not expect langtools developers > to exclude the JShell tests. That implies we should look at and > understand why the JShell tests might fail if they are co-located in > the langtools directory. If it is just because jline is not yet > standard in JDK builds, that's a temporary state of affairs that we > will get past soon enough. If there's something else going on, I > guess I'd like to better understand what that might be, so that we can > see if we can fix the issue. > > -- Jon > > On 09/16/2015 05:38 PM, Maurizio Cimadamore wrote: >> While I agree with many points you raise - I feel this discussion is >> very abstract; please try to run all langtools tests in the kulla >> langtools repo using your standard scripts - does everything still >> work for you? I found that I could not run tests using my usual >> scripts. The only way to run jshell tests reliably is through make. >> Now, if we were in a world where 100% langtools developers ran tests >> that way - I would agree with you; but the reality is somewhat >> different, and many developers are used to run langtools tests using >> some JDK N snapshot and bootstrapping the required classes. That >> technique WILL cease to work here. I don't think we can say to those >> developer - just don't run kulla tests and be happy (what if some of >> their changes accidentally broke something in kulla?). Are you saying >> that those developers should just stop doing what they are doing and >> start running langtools tests as part of a full JDK build and test >> cycle? That's a fine answer - I guess I'm just trying to guess where >> you are coming from. >> >> Maurizio >> >> On 17/09/15 01:04, Jonathan Gibbons wrote: >>> I didn't say I think JShell should be in langtools, and I thought I >>> was being careful to be very explicit about that. I was just trying >>> to make sure we were making the right choice for the right reasons. >>> >>> The fact that it depends on JDI and jline is IMO a red herring. If >>> they were both already in JDK 8 (i.e. the boot JDK) would we be >>> having this discussion? >>> >>> To me, the bottom line is that I don't think it matters where the >>> code lives, and as a result I personally don't care and have no >>> opinion where the code should live. I /do/ think that whereever we >>> choose to put it, we can make everything work well from a developer >>> standpoint, without inconveniencing any existing developer usage, >>> although I concede that we might choose not to integrate it with the >>> langtools make/build.xml infrastructure and/or the existing NetBeans >>> project. And from a somewhat selfish standpoint, I think we have >>> somewhat more latitude to set things up the way we like it in the >>> langtools repo than we would in the jdk repo. >>> >>> I agree that JShell is higher up in the layers of abstraction than >>> javac, but I also don't think that the langtools repo is defined to >>> be "core tools to be able to work with the JDK". The provenance of >>> the repo (as indicated by its name) is that it was code that was >>> managed by the old Sun Language Tools Group, from way back when, >>> meaning that it was more of a "team" repository than a repository >>> defined by a layer of abstraction. >>> >>> So, for me, I think this comes down to, "where does the Kulla dev >>> team think it would be most convenient for them to keep and work on >>> the code?". Once we can answer that, we can move on to, "what is the >>> most convenient way to satisfy all the requirements of all the >>> interested parties?" >>> >>> -- Jon >>> >>> >>> On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote: >>>> Jon, let's reverse the question - why do you think it *should* live >>>> in langtools? Honestly it feels very different from all other tools >>>> that live there; there's dependencies on JDI, jline - so to me it >>>> feels closer to other tools (i.e. jconsole) which already are in >>>> the jdk repo. I guess my argument is - if langtools defines the >>>> core tools to be able to work with the JDK, jshell feels like one >>>> ore two layer of abstractions on top of that. >>>> >>>> Maurizio >>>> >>>> On 16/09/15 23:31, Jonathan Gibbons wrote: >>>>> >>>>> >>>>> On 09/16/2015 03:06 PM, Robert Field wrote: >>>>>> As part of the JShell code review, Maurizio made the case that >>>>>> JShell should be in the JDK repo rather than langtools repo >>>>>> (where it is in the Kulla workspace). We would like additional >>>>>> feedback on this before proceeding -- >>>>>> >>>>>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>>>>>> wrote: >>>>>>> >>>>>>> One general high-level comment, which I also pointed out >>>>>>> elsewhere, is that I'm not sure jshell really belongs in >>>>>>> langtools; while it's semantically (obviously) related to >>>>>>> langtools - it is a rather different beasts w.r.t. all other >>>>>>> tools in langtools-land; the fact that it depends on the JDK >>>>>>> (for jline, and for JDI in general) makes it very hard to run on >>>>>>> top of a random JDK and then bootstrapping classes - which is a >>>>>>> technique widely used to be able to run langtools tools w/o >>>>>>> having to do a full build cycle. More specifically, talking >>>>>>> about IDE integration, I don't see how IntelliJ/Netbeans >>>>>>> langtools projects (and the langtools internal developer ant >>>>>>> build) could be updated to be able to run/debug jshell w/o a >>>>>>> full build. >>>>>> >>>>>> >>>>> >>>>> In the Big Picture view of the universe, folk would like to >>>>> restructure the repos in an OPenJDK forest, at which point the >>>>> distinction between the langtools and jdk repos will likely >>>>> disappear, or at least become very (very) blurred. While that >>>>> change is not imminent, IMO opinion, it significantly reduces the >>>>> impact of the reasons why JShell should not be in langtools. >>>>> >>>>> I also don't entirely agree with the IDE reasons either. If you >>>>> want to work on JShell in an IDE, you're going to have to do what >>>>> it takes to get rid of the red squiggly lines anyway, whatever >>>>> repo the code is in. To keep JShell out of Langtools because it >>>>> doesn't fit our current methodologies is a case of the tail >>>>> wagging the dog. >>>>> >>>>> Also, I note that JDK 9 now builds the product module by module. >>>>> In times past, we built the world repo by repo, and that would >>>>> indeed have been good reason to keep JShell out of langtools, >>>>> because of the dependencies of JShell on JDK 9 API, like jline. >>>>> But now, the build will build "interim javac", and can then build >>>>> JDK modules like java.base and whatever module jline ends up in, >>>>> and can then build the module containing JShell. >>>>> >>>>> As to how to organize IDE projects, in the worst case, we keep the >>>>> langtools IDE setup as it is now (no JShell) and have a separate >>>>> project for JShell itself. That is tantamount to what we would do >>>>> if we push JShell into the jdk repo. >>>>> >>>>> So, I'm saying all that while trying to be agnostic on which repo >>>>> it should live in. I don't really agree with the stated reasons >>>>> why it should not live in langtools, which means that we can look >>>>> for other reasons to choose one repo or another. >>>>> >>>>> -- Jon >>>> >>> >> > From maurizio.cimadamore at oracle.com Thu Sep 17 08:18:33 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Thu, 17 Sep 2015 09:18:33 +0100 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FA5D19.6040708@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA1F5F.70800@oracle.com> <55FA2095.5030808@oracle.com> <55FA5D19.6040708@oracle.com> Message-ID: <55FA7759.7030205@oracle.com> On 17/09/15 07:26, Jan Lahoda wrote: > My personal approach so far was to treat jdk.jshell as a module that > requires full JDK build to run and test. I ended up there too. And that's why I started this thread ;-) Maurizio From maurizio.cimadamore at oracle.com Thu Sep 17 09:43:06 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Thu, 17 Sep 2015 10:43:06 +0100 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FA7712.4010704@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA7712.4010704@oracle.com> Message-ID: <55FA8B2A.5080907@oracle.com> Here's a more precise description of what you'd run into if you would like to run jshell on top of an existing JDK; assumption: : points to a valid JDK 9 binary : points to a valid Kulla exploded build #1 run jshell - just add jshell (and langtools) to bootstrap classes: $ /bin/java -Xbootclasspath/p:/jdk/modules/*jdk.jshell*:/jdk/modules/*jdk.compiler*:/jdk/modules/*java.compiler*jdk.internal.jshell.tool.JShellTool Exception in thread "main" java.lang.NoClassDefFoundError: jdk/internal/jline/TerminalFactory$Flavor at jdk.internal.jshell.tool.ConsoleIOContext.(ConsoleIOContext.java:70) at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:242) at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233) #2 add jline $ /bin/java -Xbootclasspath/p:/jdk/modules/jdk.jshell:/jdk/modules/jdk.compiler:/jdk/modules/java.compiler:/jdk/modules/*jdk.internal.le*jdk.internal.jshell.tool.JShellTool Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jdi/Bootstrap at jdk.jshell.JDIConnection.findConnector(JDIConnection.java:77) at jdk.jshell.JDIConnection.(JDIConnection.java:312) at jdk.jshell.JDIEnv.init(JDIEnv.java:45) at jdk.jshell.ExecutionControl.jdiGo(ExecutionControl.java:251) at jdk.jshell.ExecutionControl.launch(ExecutionControl.java:66) at jdk.jshell.JShell.executionControl(JShell.java:601) at jdk.jshell.Eval.load(Eval.java:729) at jdk.jshell.Eval.declare(Eval.java:591) at jdk.jshell.Eval.declare(Eval.java:494) at jdk.jshell.Eval.processMethod(Eval.java:393) at jdk.jshell.Eval.eval(Eval.java:137) at jdk.jshell.JShell.eval(JShell.java:349) at jdk.internal.jshell.tool.JShellTool.processCompleteSource(JShellTool.java:1360) at jdk.internal.jshell.tool.JShellTool.processSource(JShellTool.java:1349) at jdk.internal.jshell.tool.JShellTool.processSourceCatchingReset(JShellTool.java:479) at jdk.internal.jshell.tool.JShellTool.run(JShellTool.java:465) at jdk.internal.jshell.tool.JShellTool.resetState(JShellTool.java:399) at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:248) at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:243) at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233) #3 add JDI $ /bin/java -Xbootclasspath/p:/jdk/modules/jdk.jshell:/jdk/modules/jdk.compiler:/jdk/modules/java.compiler:/jdk/modules/jdk.internal.le:/jdk/modules/*jdk.jdi*jdk.internal.jshell.tool.JShellTool Error: Could not find or load main class jdk.internal.jshell.remote.RemoteAgent | State engine terminated. See /history #4 Give up :-) My brief understanding of what goes wrong: #1 fails because you need jline. But wait - jline is already in the JDK I used to run jshell - so what? There seems to be an issue with jimage: if some bootstrap mentions 'jdk.internal', all contents of 'jdk.internal' coming from the jimage are nuked out - in other words, adding jshell to the bootclasspath prevents jline to be discovered. Uuugh. Not sure if this will go away with new modularized options. #2 fails because JDI is not part of rt.jar (or its new equivalent). So that must be manually added. Ugh. Means we need a full build available somewhere. Ok. #3 fails because, when jshell is forking the agent VM, it propagates the classpath options but not the bootclasspath options. This means that the new agent vm won't see the jshell package. Again, not sure if this is fixable or not. Because of the combination of #1, #2 and #3 it is impossible to (reliably) run jshell on top of an existing (and recent) JDK 9 binary. Note that the failures above are made more explicit by the fact that jshell is not in the binary snapshot used for launching it - if we were using a jshell-enabled JDK the failure would be much more subtle - as jshell would start, but the tool and the agent would be seeing two different versions of the jshell module - with a lot of potential for troubles. Running tests is basically more of the same. If we can fix #1, #2 and #3 (or show a path that leads us there), I agree that the decision of where to put jshell is largely irrelevant. But if any of these issues is there to stay, then I don't think having jshell in langtools is really buying anything: jshell will end up being a corner case of langtools that you need to build/run/test specially - meaning most developers won't even bother with it. Maurizio On 17/09/15 09:17, Maurizio Cimadamore wrote: > > > On 17/09/15 02:56, Jonathan Gibbons wrote: >> I guess I don't understand why testing won't work as expected, >> provided you have a sufficiently recent snapshot of JDK to use as a >> baseline. In this case, the requirement is that the JDK baseline >> has jline and whatever other recent API has been added to JDK. > There are many factors that contribute to this - the main one being > JShell too not passing bootstrapping setting on the remote agent VM, > plus some weird bug I encountered where if I set bootclasspath that > mentions some jdk.internal classes (but which doesn't have JLine - > because it comes from langtools), _every_ class from that package will > not be fetched from the jimage file - meaning that I'll be left w/o > jline. In other words, bootstrapping doesn't work - and I have reasons > to believe that bootstrapping with the new module options (when they > will become available - i.e. moduleOverride) won't work too. > > Maurizio >> >> If kulla is routinely adding and modifying API in the jdk repo, then >> I agree that would tend to suggest that the source bits should be in >> the jdk repo as well. But I would have thought that the kulla >> additions to the jdk repo will stabilize very quickly, and so after >> one or two builds and promotions, if JShell ends up in langtools, >> then it will be easy enough to run all the tests using a recent build >> of JDK. >> >> I agree that we should not expect langtools developers to have to use >> "make test" and I agree that we should not expect langtools >> developers to exclude the JShell tests. That implies we should look >> at and understand why the JShell tests might fail if they are >> co-located in the langtools directory. If it is just because jline >> is not yet standard in JDK builds, that's a temporary state of >> affairs that we will get past soon enough. If there's something else >> going on, I guess I'd like to better understand what that might be, >> so that we can see if we can fix the issue. >> >> -- Jon >> >> On 09/16/2015 05:38 PM, Maurizio Cimadamore wrote: >>> While I agree with many points you raise - I feel this discussion is >>> very abstract; please try to run all langtools tests in the kulla >>> langtools repo using your standard scripts - does everything still >>> work for you? I found that I could not run tests using my usual >>> scripts. The only way to run jshell tests reliably is through make. >>> Now, if we were in a world where 100% langtools developers ran tests >>> that way - I would agree with you; but the reality is somewhat >>> different, and many developers are used to run langtools tests using >>> some JDK N snapshot and bootstrapping the required classes. That >>> technique WILL cease to work here. I don't think we can say to those >>> developer - just don't run kulla tests and be happy (what if some of >>> their changes accidentally broke something in kulla?). Are you >>> saying that those developers should just stop doing what they are >>> doing and start running langtools tests as part of a full JDK build >>> and test cycle? That's a fine answer - I guess I'm just trying to >>> guess where you are coming from. >>> >>> Maurizio >>> >>> On 17/09/15 01:04, Jonathan Gibbons wrote: >>>> I didn't say I think JShell should be in langtools, and I thought I >>>> was being careful to be very explicit about that. I was just trying >>>> to make sure we were making the right choice for the right reasons. >>>> >>>> The fact that it depends on JDI and jline is IMO a red herring. If >>>> they were both already in JDK 8 (i.e. the boot JDK) would we be >>>> having this discussion? >>>> >>>> To me, the bottom line is that I don't think it matters where the >>>> code lives, and as a result I personally don't care and have no >>>> opinion where the code should live. I /do/ think that whereever we >>>> choose to put it, we can make everything work well from a developer >>>> standpoint, without inconveniencing any existing developer usage, >>>> although I concede that we might choose not to integrate it with >>>> the langtools make/build.xml infrastructure and/or the existing >>>> NetBeans project. And from a somewhat selfish standpoint, I think >>>> we have somewhat more latitude to set things up the way we like it >>>> in the langtools repo than we would in the jdk repo. >>>> >>>> I agree that JShell is higher up in the layers of abstraction than >>>> javac, but I also don't think that the langtools repo is defined to >>>> be "core tools to be able to work with the JDK". The provenance of >>>> the repo (as indicated by its name) is that it was code that was >>>> managed by the old Sun Language Tools Group, from way back when, >>>> meaning that it was more of a "team" repository than a repository >>>> defined by a layer of abstraction. >>>> >>>> So, for me, I think this comes down to, "where does the Kulla dev >>>> team think it would be most convenient for them to keep and work on >>>> the code?". Once we can answer that, we can move on to, "what is >>>> the most convenient way to satisfy all the requirements of all the >>>> interested parties?" >>>> >>>> -- Jon >>>> >>>> >>>> On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote: >>>>> Jon, let's reverse the question - why do you think it *should* >>>>> live in langtools? Honestly it feels very different from all other >>>>> tools that live there; there's dependencies on JDI, jline - so to >>>>> me it feels closer to other tools (i.e. jconsole) which already >>>>> are in the jdk repo. I guess my argument is - if langtools defines >>>>> the core tools to be able to work with the JDK, jshell feels like >>>>> one ore two layer of abstractions on top of that. >>>>> >>>>> Maurizio >>>>> >>>>> On 16/09/15 23:31, Jonathan Gibbons wrote: >>>>>> >>>>>> >>>>>> On 09/16/2015 03:06 PM, Robert Field wrote: >>>>>>> As part of the JShell code review, Maurizio made the case that >>>>>>> JShell should be in the JDK repo rather than langtools repo >>>>>>> (where it is in the Kulla workspace). We would like additional >>>>>>> feedback on this before proceeding -- >>>>>>> >>>>>>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>>>>>>> wrote: >>>>>>>> >>>>>>>> One general high-level comment, which I also pointed out >>>>>>>> elsewhere, is that I'm not sure jshell really belongs in >>>>>>>> langtools; while it's semantically (obviously) related to >>>>>>>> langtools - it is a rather different beasts w.r.t. all other >>>>>>>> tools in langtools-land; the fact that it depends on the JDK >>>>>>>> (for jline, and for JDI in general) makes it very hard to run >>>>>>>> on top of a random JDK and then bootstrapping classes - which >>>>>>>> is a technique widely used to be able to run langtools tools >>>>>>>> w/o having to do a full build cycle. More specifically, talking >>>>>>>> about IDE integration, I don't see how IntelliJ/Netbeans >>>>>>>> langtools projects (and the langtools internal developer ant >>>>>>>> build) could be updated to be able to run/debug jshell w/o a >>>>>>>> full build. >>>>>>> >>>>>>> >>>>>> >>>>>> In the Big Picture view of the universe, folk would like to >>>>>> restructure the repos in an OPenJDK forest, at which point the >>>>>> distinction between the langtools and jdk repos will likely >>>>>> disappear, or at least become very (very) blurred. While that >>>>>> change is not imminent, IMO opinion, it significantly reduces the >>>>>> impact of the reasons why JShell should not be in langtools. >>>>>> >>>>>> I also don't entirely agree with the IDE reasons either. If you >>>>>> want to work on JShell in an IDE, you're going to have to do what >>>>>> it takes to get rid of the red squiggly lines anyway, whatever >>>>>> repo the code is in. To keep JShell out of Langtools because it >>>>>> doesn't fit our current methodologies is a case of the tail >>>>>> wagging the dog. >>>>>> >>>>>> Also, I note that JDK 9 now builds the product module by module. >>>>>> In times past, we built the world repo by repo, and that would >>>>>> indeed have been good reason to keep JShell out of langtools, >>>>>> because of the dependencies of JShell on JDK 9 API, like jline. >>>>>> But now, the build will build "interim javac", and can then build >>>>>> JDK modules like java.base and whatever module jline ends up in, >>>>>> and can then build the module containing JShell. >>>>>> >>>>>> As to how to organize IDE projects, in the worst case, we keep >>>>>> the langtools IDE setup as it is now (no JShell) and have a >>>>>> separate project for JShell itself. That is tantamount to what we >>>>>> would do if we push JShell into the jdk repo. >>>>>> >>>>>> So, I'm saying all that while trying to be agnostic on which repo >>>>>> it should live in. I don't really agree with the stated >>>>>> reasons why it should not live in langtools, which means that we >>>>>> can look for other reasons to choose one repo or another. >>>>>> >>>>>> -- Jon >>>>> >>>> >>> >> > From jonathan.gibbons at oracle.com Thu Sep 17 15:59:26 2015 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 17 Sep 2015 08:59:26 -0700 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FA8B2A.5080907@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA7712.4010704@oracle.com> <55FA8B2A.5080907@oracle.com> Message-ID: <55FAE35E.5080801@oracle.com> Hmmm. Thanks for the write up. Looking ahead, the environment will change when Jigsaw gets integrated into JDK 9. If nothing else, the -Xbootclasspath/p: option goes away as such, and will be rejected. The proposed replacement is -Xoverride, and in the jigsaw world, we are looking at how to incorporate that into our langtools build.xml world. At that time, folk will have to update personal scripts etc as well. With regard to testing, I'm not sure how this can be used to our advantage, but these days we're tagging tests with the new @modules line. There are two variants of this tag. @modules M @modules M/P where M and P are a module name and package name respectively. Both forms express a dependence on the specified module, such that if it is not present in the testJDK, the test will not be selected for execution. (The second option additionally arranges for the specified package to be exported to the unnamed module in which the test is loaded.) All of which is to say that if a module (like jdk.jdi) is not available, any tests (like jshell tests) which may rely on it will automatically not be run. So, going forward, Jan has been looking at how to use -Xoverride in langtools build.xml. We should continue that work, and then maybe create a "play" forest contains both the kulla code and jigsaw code, to investigate these issues in that context. And related to that, we should be careful about investing too much effort into propagating bootstrap options like -Xbootclasspath into any code, without having a sense of what that code should be in the long term. I also feel that long term, the problem gets easier when Jigsaw is available as standard. In that world, it should be possible to take a baseline build of JDK (with JDI and jline in it) and to build, run and test just the module containing the jshell code, and use all other modules from the baseline JDK. In the short term, would it help those folk actively working on the kulla code base to have recent full builds of the kulla forest available from somewhere (i.e, with jline and jdi present), so that we can minimize any short term inconvenience until kulla is integrated? -- Jon On 09/17/2015 02:43 AM, Maurizio Cimadamore wrote: > Here's a more precise description of what you'd run into if you would > like to run jshell on top of an existing JDK; assumption: > > : points to a valid JDK 9 binary > : points to a valid Kulla exploded build > > #1 run jshell - just add jshell (and langtools) to bootstrap classes: > > $ /bin/java > -Xbootclasspath/p:/jdk/modules/*jdk.jshell*:/jdk/modules/*jdk.compiler*:/jdk/modules/*java.compiler*jdk.internal.jshell.tool.JShellTool > Exception in thread "main" java.lang.NoClassDefFoundError: > jdk/internal/jline/TerminalFactory$Flavor > at > jdk.internal.jshell.tool.ConsoleIOContext.(ConsoleIOContext.java:70) > at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:242) > at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233) > > > #2 add jline > > $ /bin/java > -Xbootclasspath/p:/jdk/modules/jdk.jshell:/jdk/modules/jdk.compiler:/jdk/modules/java.compiler:/jdk/modules/*jdk.internal.le*jdk.internal.jshell.tool.JShellTool > Exception in thread "main" java.lang.NoClassDefFoundError: > com/sun/jdi/Bootstrap > at jdk.jshell.JDIConnection.findConnector(JDIConnection.java:77) > at jdk.jshell.JDIConnection.(JDIConnection.java:312) > at jdk.jshell.JDIEnv.init(JDIEnv.java:45) > at jdk.jshell.ExecutionControl.jdiGo(ExecutionControl.java:251) > at jdk.jshell.ExecutionControl.launch(ExecutionControl.java:66) > at jdk.jshell.JShell.executionControl(JShell.java:601) > at jdk.jshell.Eval.load(Eval.java:729) > at jdk.jshell.Eval.declare(Eval.java:591) > at jdk.jshell.Eval.declare(Eval.java:494) > at jdk.jshell.Eval.processMethod(Eval.java:393) > at jdk.jshell.Eval.eval(Eval.java:137) > at jdk.jshell.JShell.eval(JShell.java:349) > at > jdk.internal.jshell.tool.JShellTool.processCompleteSource(JShellTool.java:1360) > at > jdk.internal.jshell.tool.JShellTool.processSource(JShellTool.java:1349) > at > jdk.internal.jshell.tool.JShellTool.processSourceCatchingReset(JShellTool.java:479) > at jdk.internal.jshell.tool.JShellTool.run(JShellTool.java:465) > at jdk.internal.jshell.tool.JShellTool.resetState(JShellTool.java:399) > at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:248) > at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:243) > at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233) > > #3 add JDI > > $ /bin/java > -Xbootclasspath/p:/jdk/modules/jdk.jshell:/jdk/modules/jdk.compiler:/jdk/modules/java.compiler:/jdk/modules/jdk.internal.le:/jdk/modules/*jdk.jdi*jdk.internal.jshell.tool.JShellTool > > Error: Could not find or load main class > jdk.internal.jshell.remote.RemoteAgent > | State engine terminated. See /history > > #4 Give up :-) > > > My brief understanding of what goes wrong: > > #1 fails because you need jline. But wait - jline is already in the > JDK I used to run jshell - so what? There seems to be an issue with > jimage: if some bootstrap mentions 'jdk.internal', all contents of > 'jdk.internal' coming from the jimage are nuked out - in other words, > adding jshell to the bootclasspath prevents jline to be discovered. > Uuugh. Not sure if this will go away with new modularized options. > > #2 fails because JDI is not part of rt.jar (or its new equivalent). So > that must be manually added. Ugh. Means we need a full build available > somewhere. Ok. > > #3 fails because, when jshell is forking the agent VM, it propagates > the classpath options but not the bootclasspath options. This means > that the new agent vm won't see the jshell package. Again, not sure if > this is fixable or not. > > Because of the combination of #1, #2 and #3 it is impossible to > (reliably) run jshell on top of an existing (and recent) JDK 9 binary. > Note that the failures above are made more explicit by the fact that > jshell is not in the binary snapshot used for launching it - if we > were using a jshell-enabled JDK the failure would be much more subtle > - as jshell would start, but the tool and the agent would be seeing > two different versions of the jshell module - with a lot of potential > for troubles. > > Running tests is basically more of the same. > > If we can fix #1, #2 and #3 (or show a path that leads us there), I > agree that the decision of where to put jshell is largely irrelevant. > But if any of these issues is there to stay, then I don't think having > jshell in langtools is really buying anything: jshell will end up > being a corner case of langtools that you need to build/run/test > specially - meaning most developers won't even bother with it. > > Maurizio > > > On 17/09/15 09:17, Maurizio Cimadamore wrote: >> >> >> On 17/09/15 02:56, Jonathan Gibbons wrote: >>> I guess I don't understand why testing won't work as expected, >>> provided you have a sufficiently recent snapshot of JDK to use as a >>> baseline. In this case, the requirement is that the JDK baseline >>> has jline and whatever other recent API has been added to JDK. >> There are many factors that contribute to this - the main one being >> JShell too not passing bootstrapping setting on the remote agent VM, >> plus some weird bug I encountered where if I set bootclasspath that >> mentions some jdk.internal classes (but which doesn't have JLine - >> because it comes from langtools), _every_ class from that package >> will not be fetched from the jimage file - meaning that I'll be left >> w/o jline. In other words, bootstrapping doesn't work - and I have >> reasons to believe that bootstrapping with the new module options >> (when they will become available - i.e. moduleOverride) won't work too. >> >> Maurizio >>> >>> If kulla is routinely adding and modifying API in the jdk repo, then >>> I agree that would tend to suggest that the source bits should be in >>> the jdk repo as well. But I would have thought that the kulla >>> additions to the jdk repo will stabilize very quickly, and so after >>> one or two builds and promotions, if JShell ends up in langtools, >>> then it will be easy enough to run all the tests using a recent >>> build of JDK. >>> >>> I agree that we should not expect langtools developers to have to >>> use "make test" and I agree that we should not expect langtools >>> developers to exclude the JShell tests. That implies we should look >>> at and understand why the JShell tests might fail if they are >>> co-located in the langtools directory. If it is just because jline >>> is not yet standard in JDK builds, that's a temporary state of >>> affairs that we will get past soon enough. If there's something >>> else going on, I guess I'd like to better understand what that might >>> be, so that we can see if we can fix the issue. >>> >>> -- Jon >>> >>> On 09/16/2015 05:38 PM, Maurizio Cimadamore wrote: >>>> While I agree with many points you raise - I feel this discussion >>>> is very abstract; please try to run all langtools tests in the >>>> kulla langtools repo using your standard scripts - does everything >>>> still work for you? I found that I could not run tests using my >>>> usual scripts. The only way to run jshell tests reliably is through >>>> make. Now, if we were in a world where 100% langtools developers >>>> ran tests that way - I would agree with you; but the reality is >>>> somewhat different, and many developers are used to run langtools >>>> tests using some JDK N snapshot and bootstrapping the required >>>> classes. That technique WILL cease to work here. I don't think we >>>> can say to those developer - just don't run kulla tests and be >>>> happy (what if some of their changes accidentally broke something >>>> in kulla?). Are you saying that those developers should just stop >>>> doing what they are doing and start running langtools tests as part >>>> of a full JDK build and test cycle? That's a fine answer - I guess >>>> I'm just trying to guess where you are coming from. >>>> >>>> Maurizio >>>> >>>> On 17/09/15 01:04, Jonathan Gibbons wrote: >>>>> I didn't say I think JShell should be in langtools, and I thought >>>>> I was being careful to be very explicit about that. I was just >>>>> trying to make sure we were making the right choice for the right >>>>> reasons. >>>>> >>>>> The fact that it depends on JDI and jline is IMO a red herring. If >>>>> they were both already in JDK 8 (i.e. the boot JDK) would we be >>>>> having this discussion? >>>>> >>>>> To me, the bottom line is that I don't think it matters where the >>>>> code lives, and as a result I personally don't care and have no >>>>> opinion where the code should live. I /do/ think that whereever we >>>>> choose to put it, we can make everything work well from a >>>>> developer standpoint, without inconveniencing any existing >>>>> developer usage, although I concede that we might choose not to >>>>> integrate it with the langtools make/build.xml infrastructure >>>>> and/or the existing NetBeans project. And from a somewhat selfish >>>>> standpoint, I think we have somewhat more latitude to set things >>>>> up the way we like it in the langtools repo than we would in the >>>>> jdk repo. >>>>> >>>>> I agree that JShell is higher up in the layers of abstraction than >>>>> javac, but I also don't think that the langtools repo is defined >>>>> to be "core tools to be able to work with the JDK". The provenance >>>>> of the repo (as indicated by its name) is that it was code that >>>>> was managed by the old Sun Language Tools Group, from way back >>>>> when, meaning that it was more of a "team" repository than a >>>>> repository defined by a layer of abstraction. >>>>> >>>>> So, for me, I think this comes down to, "where does the Kulla dev >>>>> team think it would be most convenient for them to keep and work >>>>> on the code?". Once we can answer that, we can move on to, "what >>>>> is the most convenient way to satisfy all the requirements of all >>>>> the interested parties?" >>>>> >>>>> -- Jon >>>>> >>>>> >>>>> On 09/16/2015 04:02 PM, Maurizio Cimadamore wrote: >>>>>> Jon, let's reverse the question - why do you think it *should* >>>>>> live in langtools? Honestly it feels very different from all >>>>>> other tools that live there; there's dependencies on JDI, jline - >>>>>> so to me it feels closer to other tools (i.e. jconsole) which >>>>>> already are in the jdk repo. I guess my argument is - if >>>>>> langtools defines the core tools to be able to work with the JDK, >>>>>> jshell feels like one ore two layer of abstractions on top of that. >>>>>> >>>>>> Maurizio >>>>>> >>>>>> On 16/09/15 23:31, Jonathan Gibbons wrote: >>>>>>> >>>>>>> >>>>>>> On 09/16/2015 03:06 PM, Robert Field wrote: >>>>>>>> As part of the JShell code review, Maurizio made the case that >>>>>>>> JShell should be in the JDK repo rather than langtools repo >>>>>>>> (where it is in the Kulla workspace). We would like additional >>>>>>>> feedback on this before proceeding -- >>>>>>>> >>>>>>>>> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> One general high-level comment, which I also pointed out >>>>>>>>> elsewhere, is that I'm not sure jshell really belongs in >>>>>>>>> langtools; while it's semantically (obviously) related to >>>>>>>>> langtools - it is a rather different beasts w.r.t. all other >>>>>>>>> tools in langtools-land; the fact that it depends on the JDK >>>>>>>>> (for jline, and for JDI in general) makes it very hard to run >>>>>>>>> on top of a random JDK and then bootstrapping classes - which >>>>>>>>> is a technique widely used to be able to run langtools tools >>>>>>>>> w/o having to do a full build cycle. More specifically, >>>>>>>>> talking about IDE integration, I don't see how >>>>>>>>> IntelliJ/Netbeans langtools projects (and the langtools >>>>>>>>> internal developer ant build) could be updated to be able to >>>>>>>>> run/debug jshell w/o a full build. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> In the Big Picture view of the universe, folk would like to >>>>>>> restructure the repos in an OPenJDK forest, at which point the >>>>>>> distinction between the langtools and jdk repos will likely >>>>>>> disappear, or at least become very (very) blurred. While that >>>>>>> change is not imminent, IMO opinion, it significantly reduces >>>>>>> the impact of the reasons why JShell should not be in langtools. >>>>>>> >>>>>>> I also don't entirely agree with the IDE reasons either. If you >>>>>>> want to work on JShell in an IDE, you're going to have to do >>>>>>> what it takes to get rid of the red squiggly lines anyway, >>>>>>> whatever repo the code is in. To keep JShell out of Langtools >>>>>>> because it doesn't fit our current methodologies is a case of >>>>>>> the tail wagging the dog. >>>>>>> >>>>>>> Also, I note that JDK 9 now builds the product module by module. >>>>>>> In times past, we built the world repo by repo, and that would >>>>>>> indeed have been good reason to keep JShell out of langtools, >>>>>>> because of the dependencies of JShell on JDK 9 API, like jline. >>>>>>> But now, the build will build "interim javac", and can then >>>>>>> build JDK modules like java.base and whatever module jline ends >>>>>>> up in, and can then build the module containing JShell. >>>>>>> >>>>>>> As to how to organize IDE projects, in the worst case, we keep >>>>>>> the langtools IDE setup as it is now (no JShell) and have a >>>>>>> separate project for JShell itself. That is tantamount to what >>>>>>> we would do if we push JShell into the jdk repo. >>>>>>> >>>>>>> So, I'm saying all that while trying to be agnostic on which >>>>>>> repo it should live in. I don't really agree with the stated >>>>>>> reasons why it should not live in langtools, which means that we >>>>>>> can look for other reasons to choose one repo or another. >>>>>>> >>>>>>> -- Jon >>>>>> >>>>> >>>> >>> >> > From maurizio.cimadamore at oracle.com Thu Sep 17 16:25:02 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Thu, 17 Sep 2015 17:25:02 +0100 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FAE35E.5080801@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA7712.4010704@oracle.com> <55FA8B2A.5080907@oracle.com> <55FAE35E.5080801@oracle.com> Message-ID: <55FAE95E.3030308@oracle.com> On 17/09/15 16:59, Jonathan Gibbons wrote: > In the short term, would it help those folk actively working on the > kulla code base to have recent full builds of the kulla forest > available from somewhere (i.e, with jline and jdi present), so that we > can minimize any short term inconvenience until kulla is integrated? I don't think that would help; note that the issues I pointed out in my writeup essentially boils down to the fact that part of the jshell environment (the remote agent) is insensitive to whatever parameter you pass on the command line when you run jshell/run tests (this is problem #3 in my email). As a consequence, you will be rnning some weird two headed beast which has some bits coming from the jshell under development, while other bits will fall back to the jshell version available in the 'recent build' (which presumably doesn't contain the fix/enhancement the dev is working on). I somewhat agree that #1 and #2 *should* go away in the long run when modules will be fully supported; that is, assuming that -override will do the right thing and not (as currently with -Xbootclasspath) omit all classes in same package available in jimage but not in the overriding module (problem #1). Maurizio From robert.field at oracle.com Thu Sep 17 16:28:22 2015 From: robert.field at oracle.com (Robert Field) Date: Thu, 17 Sep 2015 09:28:22 -0700 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FAE95E.3030308@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA7712.4010704@oracle.com> <55FA8B2A.5080907@oracle.com> <55FAE35E.5080801@oracle.com> <55FAE95E.3030308@oracle.com> Message-ID: <55FAEA26.8000600@oracle.com> On 09/17/15 09:25, Maurizio Cimadamore wrote: > > > On 17/09/15 16:59, Jonathan Gibbons wrote: >> In the short term, would it help those folk actively working on the >> kulla code base to have recent full builds of the kulla forest >> available from somewhere (i.e, with jline and jdi present), so that >> we can minimize any short term inconvenience until kulla is integrated? > I don't think that would help; note that the issues I pointed out in > my writeup essentially boils down to the fact that part of the jshell > environment (the remote agent) is insensitive to whatever parameter > you pass on the command line when you run jshell/run tests (this is > problem #3 in my email). As a consequence, you will be rnning some > weird two headed beast which has some bits coming from the jshell > under development, while other bits will fall back to the jshell > version available in the 'recent build' (which presumably doesn't > contain the fix/enhancement the dev is working on). That seems a bug that needs to be addressed rather than a factor in the location of the source. -Robert > > I somewhat agree that #1 and #2 *should* go away in the long run when > modules will be fully supported; that is, assuming that -override will > do the right thing and not (as currently with -Xbootclasspath) omit > all classes in same package available in jimage but not in the > overriding module (problem #1). > > Maurizio From maurizio.cimadamore at oracle.com Thu Sep 17 16:36:53 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Thu, 17 Sep 2015 17:36:53 +0100 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FAEA26.8000600@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA7712.4010704@oracle.com> <55FA8B2A.5080907@oracle.com> <55FAE35E.5080801@oracle.com> <55FAE95E.3030308@oracle.com> <55FAEA26.8000600@oracle.com> Message-ID: <55FAEC25.5080201@oracle.com> On 17/09/15 17:28, Robert Field wrote: > > On 09/17/15 09:25, Maurizio Cimadamore wrote: >> >> >> On 17/09/15 16:59, Jonathan Gibbons wrote: >>> In the short term, would it help those folk actively working on the >>> kulla code base to have recent full builds of the kulla forest >>> available from somewhere (i.e, with jline and jdi present), so that >>> we can minimize any short term inconvenience until kulla is integrated? >> I don't think that would help; note that the issues I pointed out in >> my writeup essentially boils down to the fact that part of the jshell >> environment (the remote agent) is insensitive to whatever parameter >> you pass on the command line when you run jshell/run tests (this is >> problem #3 in my email). As a consequence, you will be rnning some >> weird two headed beast which has some bits coming from the jshell >> under development, while other bits will fall back to the jshell >> version available in the 'recent build' (which presumably doesn't >> contain the fix/enhancement the dev is working on). > > That seems a bug that needs to be addressed rather than a factor in > the location of the source. If it's a bug great - I wasn't sure if the lack of bootclasspath propagation was a necessary/design choice as I'm not an expert with JDI. Maurizio > > -Robert > >> >> I somewhat agree that #1 and #2 *should* go away in the long run when >> modules will be fully supported; that is, assuming that -override >> will do the right thing and not (as currently with -Xbootclasspath) >> omit all classes in same package available in jimage but not in the >> overriding module (problem #1). >> >> Maurizio > From maurizio.cimadamore at oracle.com Thu Sep 17 16:39:49 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Thu, 17 Sep 2015 17:39:49 +0100 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FAEC25.5080201@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA7712.4010704@oracle.com> <55FA8B2A.5080907@oracle.com> <55FAE35E.5080801@oracle.com> <55FAE95E.3030308@oracle.com> <55FAEA26.8000600@oracle.com> <55FAEC25.5080201@oracle.com> Message-ID: <55FAECD5.9040708@oracle.com> On 17/09/15 17:36, Maurizio Cimadamore wrote: > > > On 17/09/15 17:28, Robert Field wrote: >> >> On 09/17/15 09:25, Maurizio Cimadamore wrote: >>> >>> >>> On 17/09/15 16:59, Jonathan Gibbons wrote: >>>> In the short term, would it help those folk actively working on the >>>> kulla code base to have recent full builds of the kulla forest >>>> available from somewhere (i.e, with jline and jdi present), so that >>>> we can minimize any short term inconvenience until kulla is >>>> integrated? >>> I don't think that would help; note that the issues I pointed out in >>> my writeup essentially boils down to the fact that part of the >>> jshell environment (the remote agent) is insensitive to whatever >>> parameter you pass on the command line when you run jshell/run tests >>> (this is problem #3 in my email). As a consequence, you will be >>> rnning some weird two headed beast which has some bits coming from >>> the jshell under development, while other bits will fall back to the >>> jshell version available in the 'recent build' (which presumably >>> doesn't contain the fix/enhancement the dev is working on). >> >> That seems a bug that needs to be addressed rather than a factor in >> the location of the source. > If it's a bug great - I wasn't sure if the lack of bootclasspath > propagation was a necessary/design choice as I'm not an expert with JDI. So, assuming that gets fixed, we are left with #1 - Jon, what is your feeling about the behavior of -override? I.e. assuming I have two modules like this: jdk.internal.le jdk internal jline and jdk.jshell jdk internal jshell If I run with -override set to override jdk.jshell, will I still be able to see jdk.internal.jline ? Maurizio > > Maurizio >> >> -Robert >> >>> >>> I somewhat agree that #1 and #2 *should* go away in the long run >>> when modules will be fully supported; that is, assuming that >>> -override will do the right thing and not (as currently with >>> -Xbootclasspath) omit all classes in same package available in >>> jimage but not in the overriding module (problem #1). >>> >>> Maurizio >> > From jonathan.gibbons at oracle.com Thu Sep 17 17:14:00 2015 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 17 Sep 2015 10:14:00 -0700 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FAECD5.9040708@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA7712.4010704@oracle.com> <55FA8B2A.5080907@oracle.com> <55FAE35E.5080801@oracle.com> <55FAE95E.3030308@oracle.com> <55FAEA26.8000600@oracle.com> <55FAEC25.5080201@oracle.com> <55FAECD5.9040708@oracle.com> Message-ID: <55FAF4D8.2020600@oracle.com> On 09/17/2015 09:39 AM, Maurizio Cimadamore wrote: > > > On 17/09/15 17:36, Maurizio Cimadamore wrote: >> >> >> On 17/09/15 17:28, Robert Field wrote: >>> >>> On 09/17/15 09:25, Maurizio Cimadamore wrote: >>>> >>>> >>>> On 17/09/15 16:59, Jonathan Gibbons wrote: >>>>> In the short term, would it help those folk actively working on >>>>> the kulla code base to have recent full builds of the kulla forest >>>>> available from somewhere (i.e, with jline and jdi present), so >>>>> that we can minimize any short term inconvenience until kulla is >>>>> integrated? >>>> I don't think that would help; note that the issues I pointed out >>>> in my writeup essentially boils down to the fact that part of the >>>> jshell environment (the remote agent) is insensitive to whatever >>>> parameter you pass on the command line when you run jshell/run >>>> tests (this is problem #3 in my email). As a consequence, you will >>>> be rnning some weird two headed beast which has some bits coming >>>> from the jshell under development, while other bits will fall back >>>> to the jshell version available in the 'recent build' (which >>>> presumably doesn't contain the fix/enhancement the dev is working on). >>> >>> That seems a bug that needs to be addressed rather than a factor in >>> the location of the source. >> If it's a bug great - I wasn't sure if the lack of bootclasspath >> propagation was a necessary/design choice as I'm not an expert with JDI. > So, assuming that gets fixed, we are left with #1 - Jon, what is your > feeling about the behavior of -override? I.e. assuming I have two > modules like this: > > jdk.internal.le > jdk > internal > jline > > and > > jdk.jshell > jdk > internal > jshell > > > If I run with -override set to override jdk.jshell, will I still be > able to see jdk.internal.jline ? It's not clear from your example whether "jline" and "jshell" are classes or packages. (They're both all lower case). With today's jigsaw bits, you can't add packages, you can just add/modify classes in existing packages. But the Jigsaw team is set to discuss -Xoverride in the near future, so this is a good time to ensure that -Xoverride will be able to do the right thing. -- Jon > > Maurizio > >> >> Maurizio >>> >>> -Robert >>> >>>> >>>> I somewhat agree that #1 and #2 *should* go away in the long run >>>> when modules will be fully supported; that is, assuming that >>>> -override will do the right thing and not (as currently with >>>> -Xbootclasspath) omit all classes in same package available in >>>> jimage but not in the overriding module (problem #1). >>>> >>>> Maurizio >>> >> > From maurizio.cimadamore at oracle.com Thu Sep 17 17:24:59 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Thu, 17 Sep 2015 18:24:59 +0100 Subject: JShell: source in langtools vs JDK? In-Reply-To: <55FAECD5.9040708@oracle.com> References: <55F9EDD6.9050702@oracle.com> <55F9F4F3.8030609@oracle.com> <55FA0370.2000805@oracle.com> <55FA0B9C.3090206@oracle.com> <55FA1DD2.5010602@oracle.com> <55FA7712.4010704@oracle.com> <55FA8B2A.5080907@oracle.com> <55FAE35E.5080801@oracle.com> <55FAE95E.3030308@oracle.com> <55FAEA26.8000600@oracle.com> <55FAEC25.5080201@oracle.com> <55FAECD5.9040708@oracle.com> Message-ID: <55FAF76B.60809@oracle.com> Ok, I chatted with Jon, here's an action item for this: * workaround problem #1 by putting jshell in a package other than jdk.internal (i.e. jdk.jshell could be good); this will avoid -Xbootstrap accidentally making jline unavailable * fix problem #3 - i.e. by using -bootclasspath System.getProperty(?sun.boot.class.path?) to launch the agent VM (will need to change once -Xoverride arrive) * leave jshell in langtools - stuff like problem #2 can easily be fixed up by looking up the jdi library in the target.java.home Deal? Maurizio On 17/09/15 17:39, Maurizio Cimadamore wrote: > > > On 17/09/15 17:36, Maurizio Cimadamore wrote: >> >> >> On 17/09/15 17:28, Robert Field wrote: >>> >>> On 09/17/15 09:25, Maurizio Cimadamore wrote: >>>> >>>> >>>> On 17/09/15 16:59, Jonathan Gibbons wrote: >>>>> In the short term, would it help those folk actively working on >>>>> the kulla code base to have recent full builds of the kulla forest >>>>> available from somewhere (i.e, with jline and jdi present), so >>>>> that we can minimize any short term inconvenience until kulla is >>>>> integrated? >>>> I don't think that would help; note that the issues I pointed out >>>> in my writeup essentially boils down to the fact that part of the >>>> jshell environment (the remote agent) is insensitive to whatever >>>> parameter you pass on the command line when you run jshell/run >>>> tests (this is problem #3 in my email). As a consequence, you will >>>> be rnning some weird two headed beast which has some bits coming >>>> from the jshell under development, while other bits will fall back >>>> to the jshell version available in the 'recent build' (which >>>> presumably doesn't contain the fix/enhancement the dev is working on). >>> >>> That seems a bug that needs to be addressed rather than a factor in >>> the location of the source. >> If it's a bug great - I wasn't sure if the lack of bootclasspath >> propagation was a necessary/design choice as I'm not an expert with JDI. > So, assuming that gets fixed, we are left with #1 - Jon, what is your > feeling about the behavior of -override? I.e. assuming I have two > modules like this: > > jdk.internal.le > jdk > internal > jline > > and > > jdk.jshell > jdk > internal > jshell > > > If I run with -override set to override jdk.jshell, will I still be > able to see jdk.internal.jline ? > > Maurizio > >> >> Maurizio >>> >>> -Robert >>> >>>> >>>> I somewhat agree that #1 and #2 *should* go away in the long run >>>> when modules will be fully supported; that is, assuming that >>>> -override will do the right thing and not (as currently with >>>> -Xbootclasspath) omit all classes in same package available in >>>> jimage but not in the overriding module (problem #1). >>>> >>>> Maurizio >>> >> > From brian.goetz at oracle.com Thu Sep 17 17:32:00 2015 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 17 Sep 2015 13:32:00 -0400 Subject: JShell: source in langtools vs JDK? In-Reply-To: References: Message-ID: <55FAF910.3080007@oracle.com> One additional comment: because jshell is so tightly coupled to javac, every time we add new node types to javac, we're going to have to add support to them for jshell. While this does not mean it must be in langtools, it does seem we're more likely to remember to update both together if they share a repo and a test base. (Of course, if langtools+jdk merge, this issue goes away.) On 9/16/2015 6:06 PM, Robert Field wrote: > As part of the JShell code review, Maurizio made the case that JShell should be in the JDK repo rather than langtools repo (where it is in the Kulla workspace). We would like additional feedback on this before proceeding -- > >> On Sep 11, 2015, at 8:25 AM, Maurizio Cimadamore wrote: >> >> One general high-level comment, which I also pointed out elsewhere, is that I'm not sure jshell really belongs in langtools; while it's semantically (obviously) related to langtools - it is a rather different beasts w.r.t. all other tools in langtools-land; the fact that it depends on the JDK (for jline, and for JDI in general) makes it very hard to run on top of a random JDK and then bootstrapping classes - which is a technique widely used to be able to run langtools tools w/o having to do a full build cycle. More specifically, talking about IDE integration, I don't see how IntelliJ/Netbeans langtools projects (and the langtools internal developer ant build) could be updated to be able to run/debug jshell w/o a full build. > > > From jonathan.gibbons at oracle.com Thu Sep 17 18:52:55 2015 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 17 Sep 2015 11:52:55 -0700 Subject: JShell: packages In-Reply-To: <55FB0182.2030800@oracle.com> References: <55FB0182.2030800@oracle.com> Message-ID: <55FB0C07.5040403@oracle.com> We should not change any of these packages. My understanding from Maurizio is that there is a bug or anti-feature in the system today such that the choice of package for jline is problematic, and so we were simply discussing the possibility of using a different package for jline for the immediate short term as a workaround for the -Xbootclasspath/p: issue. When Jigsaw gets integrated, the -Xbootclasspath/p: option will go away, and we will have to use the new -Xoverride option instead. In advance of that, we should ensure that -Xoverride works in a way that is satisfactory with jline in its propoer long term home. -- Jon On 09/17/2015 11:08 AM, Robert Field wrote: > There were extensive discussions and approvals around package names. > This is what we have now: > > jdk.jshell > > JShell API and core implementation. > > jdk.internal.jshell.remote > > The remote side of the implementation > > jdk.internal.jshell.debug > > Single class supporting debugging of the implementation through > external tools > > jdk.internal.jshell.tool > > The JShell tool (built on the API) > > > The package jdk.jshell has all the public API. > > jdk.internal.jshell.* includes some classes with public access > modifiers but these packages are not public APIs (we want the ability > to change them as needed) -- my understanding is that packages with > those characteristics should be named "jdk.internal.*". > > ----- > > However we have problems -- > >> some weird bug I encountered where if I set bootclasspath that >> mentions some jdk.internal classes (but which doesn't have JLine >> - because it comes from langtools), _every_ class from that >> package will not be fetched from the jimage file - meaning that >> I'll be left w/o jline. In other words, bootstrapping doesn't >> work - and I have reasons to believe that bootstrapping with the >> new module options (when they will become available - i.e. >> moduleOverride) won't work too. > > > and the proposal -- > >> * workaround problem #1 by putting jshell in a package other than >> jdk.internal (i.e. jdk.jshell could be good); this will avoid >> -Xbootstrap accidentally making jline unavailable > > > -Robert > > From robert.field at oracle.com Thu Sep 17 20:17:49 2015 From: robert.field at oracle.com (Robert Field) Date: Thu, 17 Sep 2015 13:17:49 -0700 Subject: REPL code review -- Eval In-Reply-To: <55F2F270.8020803@oracle.com> References: <55F2F270.8020803@oracle.com> Message-ID: <55FB1FED.4010705@oracle.com> On 09/11/15 08:25, Maurizio Cimadamore wrote: > * Eval > > - just curious - you seem to do a lot of String-based analysis > (see example in computeQualifiedParameterTypes()) - why is it not > possible to rely on the more robust Element API to do that? Not sure what you mean by "String-based analysis". The final return is a String which is stored in the Snippet for equality comparison with incoming Snippets. But all the processing using compiler API constructs: a compiler analysis pass is run "AnalyzeTask", TreeDissector walks the tree to extract the the MethodType. Then a subclass of the compiler's SignatureGenerator converts it to a String. > I.e. couldn't a snippet corresponding to a declaration have an Element? Well, it does have an Element intermediate. It is not stored in the Snippet because there is no on-going need to analyze the Snippet once it is place. I use Type directly because Symbol hiding behind Element does not have the functionality I need. > Or, even more generally, why not just create real AST nodes for wraps? A high-level design decision was that wraps would be source-to-source this is for specification and design-docs. Also, in some cases trees do not break the source down sufficiently. > > - UnresolvedExtractor - it would probably help to use > DiagnosticFlag.RECOVERABLE instead of hard-coded strings to filter out > resolution error (I also noted this somewhere below). However, I also > note how the logic for extracting the unresolved symbol is inherently > fragile and dependent on what is the shape of the javac diagnostics. Using a flag seems better indeed. I see DiagnosticFlag.RESOLVE_ERROR which seems to be what matches resolution errors (there is no doc on these enum values). Yes? What is DiagnosticFlag.RECOVERABLE? > I wonder if looking at the analyzed tree might not provide a more > robust solution here. I would not want to replicate javac's error detection logic, but after the error is detected I'd have to use trees to find the unresolved if I can't depend on the error format. > > - the Range logic is clever - but, again, it seems like a big > workaround for the fact that wrapping is defined in terms of strings - > not trees; any reason you went down this path? See above. > > - corralled - I found the Snippet hierarchy underused when it > comes to corralling; the task of corralling a snippet is basically > performed by Eval through some helper routines - couldn't we have each > snippet define how corralling should be done? You go half-way there by > having a corral() method in the Snippet base class, but it doesn't do > much, other then retrieving whatever value has been set from outside. Corralling logic is in two parts. In Eval.processMethod() the corralled wrap is generated and used as one of the values to create the MethodSnippet (currently only methods can be corralled). In Eval.declare(), on failing compilation, if a corralled wrap is available, it attempts to compile the corralled wrap. The *Snippet classes present as immutable values, though some internal values are mutable. This is an internal immutable value. So, it could be computed after creation. But the tree dissection, range, and wrapping code has everything in common with Eval, and nothing in common with *Snippet. The corralled compilation logic is independent of Snippet kind. > > - the corralling logic doesn't seem to work on supertypes using FQN: > > -> class A extends B.Inner { static class Inner { } } > | Error: > | package B does not exist > | class A extends B.Inner { static class Inner { } } > | ^-----^ > I would have expected the snippet to wait for B. Hmmm,.. It is a good thing you suggested changing those confusing names. I use the old term "corralling" still in the code, which refers to RECOVERABLE_DEFINED, which this isn't, but seems it should be RECOVERABLE_NOT_DEFINED. The problem is that the error is not a "compiler.err.cant.resolve" error but a "package does not exist" error. It seems, except for import that this error should be handled like resolution errors. The change to using a flag and more general extractor (as above) should address this. > > - the updates of corralled snippets seems to happen at the wrong > time - this could lead to chain of symbols that are not usable: > > -> class D extends E { } > | Added class D, however, it cannot be referenced until class E is > declared > > -> class E { D d; } > | Added class E, however, it cannot be referenced until class D is > declared > > I guess what happens here is that, since the decision on whether > to corral or not is based on the fact that we can compile cleanly , > here we run into troubles. The first snippet obviously cannot be > compiled cleanly, so it's corralled; the second one cannot compile > cleanly because the first one is not 'defined' - so you get another > resolution error and another corralled snippet. Unfortunately, the two > snippets taken together form a legal source. Btw, I think the current > behavior is preventing stuff like this: > > -> class A extends B { } > | Added class A, however, it cannot be referenced until class B is > declared > > -> class B extends A { } > | Added class B, however, it cannot be referenced until class A is > declared > > from looping forever in Eval.declare; perhaps the code could be made > more robust by looking explicitly for update loops (i.e. this can be > done by keeping a stack of all the updated snippets and see if we > encounter the same snippet twice). > > - on update loops - I guess it is indeed possible to have loops, > even with current corralling limitations (by exploiting redefinition): > > -> class Outer { class Inner extends Foo { } } > | Added class Outer, however, it cannot be referenced until class Foo > is declared > > -> class Foo { } > | Added class Foo > > -> class Foo extends Outer { } > > Update loop check logic is in order I think. Also, interestingly - I'm > not able to stop the REPL once I get into such a loop. Actually, later > I found out it doesn't loop forever - I guess eventually it runs out > of stack (after several minutes) and exits saying this: > > | Replaced class Foo > | Update overwrote class Foo > > > Which seems bogus anyway - given there's cyclic inheritance (no error > is reported!). > Oy! Wrapped snippets are compiled one at a time -- simple and clean, but that fails in this case. One problem is that the set of updates is not known before compilation. If a new snippet is or causes an update to an existing snippet its dependencies need only be updated in the case that redefinition fails. Redefinition usually succeeds, and certainly does for the false positive updates. I think the initial snippet and set of updates will need to be recompiled as full sets in each iteration, at least in the case of failure. Thanks, Robert From maurizio.cimadamore at oracle.com Fri Sep 18 00:16:33 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Fri, 18 Sep 2015 01:16:33 +0100 Subject: REPL code review -- Eval In-Reply-To: <55FB1FED.4010705@oracle.com> References: <55F2F270.8020803@oracle.com> <55FB1FED.4010705@oracle.com> Message-ID: <55FB57E1.2050504@oracle.com> On 17/09/15 21:17, Robert Field wrote: > > On 09/11/15 08:25, Maurizio Cimadamore wrote: >> * Eval >> >> - just curious - you seem to do a lot of String-based analysis >> (see example in computeQualifiedParameterTypes()) - why is it not >> possible to rely on the more robust Element API to do that? > > Not sure what you mean by "String-based analysis". The final return > is a String which is stored in the Snippet for equality comparison > with incoming Snippets. But all the processing using compiler API > constructs: a compiler analysis pass is run "AnalyzeTask", > TreeDissector walks the tree to extract the the MethodType. Then a > subclass of the compiler's SignatureGenerator converts it to a String. > >> I.e. couldn't a snippet corresponding to a declaration have an Element? > > Well, it does have an Element intermediate. It is not stored in the > Snippet because there is no on-going need to analyze the Snippet once > it is place. I use Type directly because Symbol hiding behind Element > does not have the functionality I need. > >> Or, even more generally, why not just create real AST nodes for wraps? > > A high-level design decision was that wraps would be source-to-source > this is for specification and design-docs. Also, in some cases trees > do not break the source down sufficiently. Not sure I understand what you mean here - but I understand this was a deliberate choice; I just note that, at the end of the day, I'm not sure if it buys you simplicity. > >> >> - UnresolvedExtractor - it would probably help to use >> DiagnosticFlag.RECOVERABLE instead of hard-coded strings to filter >> out resolution error (I also noted this somewhere below). However, I >> also note how the logic for extracting the unresolved symbol is >> inherently fragile and dependent on what is the shape of the javac >> diagnostics. > > Using a flag seems better indeed. I see DiagnosticFlag.RESOLVE_ERROR > which seems to be what matches resolution errors (there is no doc on > these enum values). Yes? What is DiagnosticFlag.RECOVERABLE? Sorry - I messed them up - RESOLVE_ERROR is the flag associated with recoverable errors during annotation processing - bottom line, annotation processing has a very similar definition of recoverable as you do. > >> I wonder if looking at the analyzed tree might not provide a more >> robust solution here. > > I would not want to replicate javac's error detection logic, but after > the error is detected I'd have to use trees to find the unresolved if > I can't depend on the error format. > >> >> - the Range logic is clever - but, again, it seems like a big >> workaround for the fact that wrapping is defined in terms of strings >> - not trees; any reason you went down this path? > > See above. > >> >> - corralled - I found the Snippet hierarchy underused when it >> comes to corralling; the task of corralling a snippet is basically >> performed by Eval through some helper routines - couldn't we have >> each snippet define how corralling should be done? You go half-way >> there by having a corral() method in the Snippet base class, but it >> doesn't do much, other then retrieving whatever value has been set >> from outside. > > Corralling logic is in two parts. In Eval.processMethod() the > corralled wrap is generated and used as one of the values to create > the MethodSnippet (currently only methods can be corralled). In > Eval.declare(), on failing compilation, if a corralled wrap is > available, it attempts to compile the corralled wrap. > > The *Snippet classes present as immutable values, though some > internal values are mutable. This is an internal immutable value. So, > it could be computed after creation. But the tree dissection, range, > and wrapping code has everything in common with Eval, and nothing in > common with *Snippet. > > The corralled compilation logic is independent of Snippet kind. > >> >> - the corralling logic doesn't seem to work on supertypes using FQN: >> >> -> class A extends B.Inner { static class Inner { } } >> | Error: >> | package B does not exist >> | class A extends B.Inner { static class Inner { } } >> | ^-----^ > >> I would have expected the snippet to wait for B. > > > Hmmm,.. > > It is a good thing you suggested changing those confusing names. I use > the old term "corralling" still in the code, which refers to > RECOVERABLE_DEFINED, which this isn't, but seems it should be > RECOVERABLE_NOT_DEFINED. > > The problem is that the error is not a "compiler.err.cant.resolve" > error but a "package does not exist" error. It seems, except for > import that this error should be handled like resolution errors. The > change to using a flag and more general extractor (as above) should > address this. If the flag is set for such errors (finger crossed) - anyway if it's not I guess we can make it so (but we'd need to consult the annotation processing gods). > >> >> - the updates of corralled snippets seems to happen at the wrong >> time - this could lead to chain of symbols that are not usable: >> >> -> class D extends E { } >> | Added class D, however, it cannot be referenced until class E is >> declared >> >> -> class E { D d; } >> | Added class E, however, it cannot be referenced until class D is >> declared >> >> I guess what happens here is that, since the decision on whether >> to corral or not is based on the fact that we can compile cleanly , >> here we run into troubles. The first snippet obviously cannot be >> compiled cleanly, so it's corralled; the second one cannot compile >> cleanly because the first one is not 'defined' - so you get another >> resolution error and another corralled snippet. Unfortunately, the >> two snippets taken together form a legal source. Btw, I think the >> current behavior is preventing stuff like this: >> >> -> class A extends B { } >> | Added class A, however, it cannot be referenced until class B is >> declared >> >> -> class B extends A { } >> | Added class B, however, it cannot be referenced until class A is >> declared >> >> from looping forever in Eval.declare; perhaps the code could be made >> more robust by looking explicitly for update loops (i.e. this can be >> done by keeping a stack of all the updated snippets and see if we >> encounter the same snippet twice). >> >> - on update loops - I guess it is indeed possible to have loops, >> even with current corralling limitations (by exploiting redefinition): >> >> -> class Outer { class Inner extends Foo { } } >> | Added class Outer, however, it cannot be referenced until class >> Foo is declared >> >> -> class Foo { } >> | Added class Foo >> >> -> class Foo extends Outer { } >> >> Update loop check logic is in order I think. Also, interestingly - >> I'm not able to stop the REPL once I get into such a loop. Actually, >> later I found out it doesn't loop forever - I guess eventually it >> runs out of stack (after several minutes) and exits saying this: >> >> | Replaced class Foo >> | Update overwrote class Foo >> >> >> Which seems bogus anyway - given there's cyclic inheritance (no error >> is reported!). >> > > Oy! > > Wrapped snippets are compiled one at a time -- simple and clean, but > that fails in this case. One problem is that the set of updates is > not known before compilation. If a new snippet is or causes an update > to an existing snippet its dependencies need only be updated in the > case that redefinition fails. Redefinition usually succeeds, and > certainly does for the false positive updates. I think the initial > snippet and set of updates will need to be recompiled as full sets in > each iteration, at least in the case of failure. Yeah - that seems a promising approach - bundle them up and recompile them together. Maurizio > > Thanks, > Robert > From maurizio.cimadamore at oracle.com Fri Sep 18 00:20:39 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Fri, 18 Sep 2015 01:20:39 +0100 Subject: JShell: packages In-Reply-To: <55FB0C07.5040403@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> Message-ID: <55FB58D7.5080702@oracle.com> On 17/09/15 19:52, Jonathan Gibbons wrote: > We should not change any of these packages. Now I'm confused :-) As you say, we need a workaround for the Xbootclasspath issue - the simpler workaround would be to change the way jshell uses jdk.internal package names; I'm assuming fixing the bootstrap vs. jimage issue would take much longer. So, what do you mean by "we should not change any of these packages" ? Maurizio > > My understanding from Maurizio is that there is a bug or anti-feature > in the system today such that the choice of package for jline is > problematic, and so we were simply discussing the possibility of using > a different package for jline for the immediate short term as a > workaround for the -Xbootclasspath/p: issue. When Jigsaw gets > integrated, the -Xbootclasspath/p: option will go away, and we will > have to use the new -Xoverride option instead. In advance of that, > we should ensure that -Xoverride works in a way that is satisfactory > with jline in its propoer long term home. > > -- Jon > > On 09/17/2015 11:08 AM, Robert Field wrote: >> There were extensive discussions and approvals around package names. >> This is what we have now: >> >> jdk.jshell >> >> JShell API and core implementation. >> >> jdk.internal.jshell.remote >> >> The remote side of the implementation >> >> jdk.internal.jshell.debug >> >> Single class supporting debugging of the implementation through >> external tools >> >> jdk.internal.jshell.tool >> >> The JShell tool (built on the API) >> >> >> The package jdk.jshell has all the public API. >> >> jdk.internal.jshell.* includes some classes with public access >> modifiers but these packages are not public APIs (we want the ability >> to change them as needed) -- my understanding is that packages with >> those characteristics should be named "jdk.internal.*". >> >> ----- >> >> However we have problems -- >> >>> some weird bug I encountered where if I set bootclasspath that >>> mentions some jdk.internal classes (but which doesn't have JLine >>> - because it comes from langtools), _every_ class from that >>> package will not be fetched from the jimage file - meaning that >>> I'll be left w/o jline. In other words, bootstrapping doesn't >>> work - and I have reasons to believe that bootstrapping with the >>> new module options (when they will become available - i.e. >>> moduleOverride) won't work too. >> >> >> and the proposal -- >> >>> * workaround problem #1 by putting jshell in a package other >>> than jdk.internal (i.e. jdk.jshell could be good); this will >>> avoid -Xbootstrap accidentally making jline unavailable >> >> >> -Robert >> >> > From maurizio.cimadamore at oracle.com Fri Sep 18 00:25:14 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Fri, 18 Sep 2015 01:25:14 +0100 Subject: JShell: packages In-Reply-To: <55FB58D7.5080702@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> Message-ID: <55FB59EA.5020108@oracle.com> On 18/09/15 01:20, Maurizio Cimadamore wrote: > Now I'm confused Less confused now - you prefer changing jline's package instead of jshell's. That should work too (dunno if jshell uses other stuff inside other jdk.internal sub-packages). Whatever works - as long as jshell, jline and any other JDK dependencies end up with same package name. Maurizio From jonathan.gibbons at oracle.com Fri Sep 18 00:25:26 2015 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 17 Sep 2015 17:25:26 -0700 Subject: JShell: packages In-Reply-To: <55FB58D7.5080702@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> Message-ID: <55FB59F6.5010308@oracle.com> If the problem is just with jline, I was thinking we should temporarily put it in its own package, such as jdk.internal.jline, and leave alone the packages that Robert listed. Or am I missing something here? (again?) -- Jon On 09/17/2015 05:20 PM, Maurizio Cimadamore wrote: > > > On 17/09/15 19:52, Jonathan Gibbons wrote: >> We should not change any of these packages. > Now I'm confused :-) > > As you say, we need a workaround for the Xbootclasspath issue - the > simpler workaround would be to change the way jshell uses jdk.internal > package names; I'm assuming fixing the bootstrap vs. jimage issue > would take much longer. > > So, what do you mean by "we should not change any of these packages" ? > > Maurizio >> >> My understanding from Maurizio is that there is a bug or anti-feature >> in the system today such that the choice of package for jline is >> problematic, and so we were simply discussing the possibility of >> using a different package for jline for the immediate short term as a >> workaround for the -Xbootclasspath/p: issue. When Jigsaw gets >> integrated, the -Xbootclasspath/p: option will go away, and we will >> have to use the new -Xoverride option instead. In advance of that, >> we should ensure that -Xoverride works in a way that is satisfactory >> with jline in its propoer long term home. >> >> -- Jon >> >> On 09/17/2015 11:08 AM, Robert Field wrote: >>> There were extensive discussions and approvals around package >>> names. This is what we have now: >>> >>> jdk.jshell >>> >>> JShell API and core implementation. >>> >>> jdk.internal.jshell.remote >>> >>> The remote side of the implementation >>> >>> jdk.internal.jshell.debug >>> >>> Single class supporting debugging of the implementation through >>> external tools >>> >>> jdk.internal.jshell.tool >>> >>> The JShell tool (built on the API) >>> >>> >>> The package jdk.jshell has all the public API. >>> >>> jdk.internal.jshell.* includes some classes with public access >>> modifiers but these packages are not public APIs (we want the >>> ability to change them as needed) -- my understanding is that >>> packages with those characteristics should be named "jdk.internal.*". >>> >>> ----- >>> >>> However we have problems -- >>> >>>> some weird bug I encountered where if I set bootclasspath that >>>> mentions some jdk.internal classes (but which doesn't have >>>> JLine - because it comes from langtools), _every_ class from >>>> that package will not be fetched from the jimage file - meaning >>>> that I'll be left w/o jline. In other words, bootstrapping >>>> doesn't work - and I have reasons to believe that bootstrapping >>>> with the new module options (when they will become available - >>>> i.e. moduleOverride) won't work too. >>> >>> >>> and the proposal -- >>> >>>> * workaround problem #1 by putting jshell in a package other >>>> than jdk.internal (i.e. jdk.jshell could be good); this will >>>> avoid -Xbootstrap accidentally making jline unavailable >>> >>> >>> -Robert >>> >>> >> > From robert.field at oracle.com Fri Sep 18 00:55:32 2015 From: robert.field at oracle.com (Robert Field) Date: Thu, 17 Sep 2015 17:55:32 -0700 Subject: JShell: packages In-Reply-To: <55FB59F6.5010308@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> <55FB59F6.5010308@oracle.com> Message-ID: <14fddf2f3b8.2784.4011f3a8741ca2aabce58b8b81f42d24@oracle.com> Jline is already in its own jdk.internal package. On September 17, 2015 5:25:27 PM Jonathan Gibbons wrote: > If the problem is just with jline, I was thinking we should temporarily > put it in its own package, such as jdk.internal.jline, and leave alone > the packages that Robert listed. > > Or am I missing something here? (again?) > > -- Jon > > On 09/17/2015 05:20 PM, Maurizio Cimadamore wrote: >> >> >> On 17/09/15 19:52, Jonathan Gibbons wrote: >>> We should not change any of these packages. >> Now I'm confused :-) >> >> As you say, we need a workaround for the Xbootclasspath issue - the >> simpler workaround would be to change the way jshell uses jdk.internal >> package names; I'm assuming fixing the bootstrap vs. jimage issue >> would take much longer. >> >> So, what do you mean by "we should not change any of these packages" ? >> >> Maurizio >>> >>> My understanding from Maurizio is that there is a bug or anti-feature >>> in the system today such that the choice of package for jline is >>> problematic, and so we were simply discussing the possibility of >>> using a different package for jline for the immediate short term as a >>> workaround for the -Xbootclasspath/p: issue. When Jigsaw gets >>> integrated, the -Xbootclasspath/p: option will go away, and we will >>> have to use the new -Xoverride option instead. In advance of that, >>> we should ensure that -Xoverride works in a way that is satisfactory >>> with jline in its propoer long term home. >>> >>> -- Jon >>> >>> On 09/17/2015 11:08 AM, Robert Field wrote: >>>> There were extensive discussions and approvals around package >>>> names. This is what we have now: >>>> >>>> jdk.jshell >>>> >>>> JShell API and core implementation. >>>> >>>> jdk.internal.jshell.remote >>>> >>>> The remote side of the implementation >>>> >>>> jdk.internal.jshell.debug >>>> >>>> Single class supporting debugging of the implementation through >>>> external tools >>>> >>>> jdk.internal.jshell.tool >>>> >>>> The JShell tool (built on the API) >>>> >>>> >>>> The package jdk.jshell has all the public API. >>>> >>>> jdk.internal.jshell.* includes some classes with public access >>>> modifiers but these packages are not public APIs (we want the >>>> ability to change them as needed) -- my understanding is that >>>> packages with those characteristics should be named "jdk.internal.*". >>>> >>>> ----- >>>> >>>> However we have problems -- >>>> >>>>> some weird bug I encountered where if I set bootclasspath that >>>>> mentions some jdk.internal classes (but which doesn't have >>>>> JLine - because it comes from langtools), _every_ class from >>>>> that package will not be fetched from the jimage file - meaning >>>>> that I'll be left w/o jline. In other words, bootstrapping >>>>> doesn't work - and I have reasons to believe that bootstrapping >>>>> with the new module options (when they will become available - >>>>> i.e. moduleOverride) won't work too. >>>> >>>> >>>> and the proposal -- >>>> >>>>> * workaround problem #1 by putting jshell in a package other >>>>> than jdk.internal (i.e. jdk.jshell could be good); this will >>>>> avoid -Xbootstrap accidentally making jline unavailable >>>> >>>> >>>> -Robert >>>> >>>> >>> >> > From jonathan.gibbons at oracle.com Fri Sep 18 01:08:26 2015 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 17 Sep 2015 18:08:26 -0700 Subject: JShell: packages In-Reply-To: <14fddf2f3b8.2784.4011f3a8741ca2aabce58b8b81f42d24@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> <55FB59F6.5010308@oracle.com> <14fddf2f3b8.2784.4011f3a8741ca2aabce58b8b81f42d24@oracle.com> Message-ID: <55FB640A.2060409@oracle.com> Then I am indeed confused as to the issue/interaction between -Xbootclasspath/p: and .jimage files. -- Jon On 09/17/2015 05:55 PM, Robert Field wrote: > > Jline is already in its own jdk.internal package. > > On September 17, 2015 5:25:27 PM Jonathan Gibbons > wrote: > >> If the problem is just with jline, I was thinking we should >> temporarily put it in its own package, such as jdk.internal.jline, >> and leave alone the packages that Robert listed. >> >> Or am I missing something here? (again?) >> >> -- Jon >> >> On 09/17/2015 05:20 PM, Maurizio Cimadamore wrote: >>> >>> >>> On 17/09/15 19:52, Jonathan Gibbons wrote: >>>> We should not change any of these packages. >>> Now I'm confused :-) >>> >>> As you say, we need a workaround for the Xbootclasspath issue - the >>> simpler workaround would be to change the way jshell uses >>> jdk.internal package names; I'm assuming fixing the bootstrap vs. >>> jimage issue would take much longer. >>> >>> So, what do you mean by "we should not change any of these packages" ? >>> >>> Maurizio >>>> >>>> My understanding from Maurizio is that there is a bug or >>>> anti-feature in the system today such that the choice of package >>>> for jline is problematic, and so we were simply discussing the >>>> possibility of using a different package for jline for the >>>> immediate short term as a workaround for the -Xbootclasspath/p: >>>> issue. When Jigsaw gets integrated, the -Xbootclasspath/p: option >>>> will go away, and we will have to use the new -Xoverride option >>>> instead. In advance of that, we should ensure that -Xoverride >>>> works in a way that is satisfactory with jline in its propoer long >>>> term home. >>>> >>>> -- Jon >>>> >>>> On 09/17/2015 11:08 AM, Robert Field wrote: >>>>> There were extensive discussions and approvals around package >>>>> names. This is what we have now: >>>>> >>>>> jdk.jshell >>>>> >>>>> JShell API and core implementation. >>>>> >>>>> jdk.internal.jshell.remote >>>>> >>>>> The remote side of the implementation >>>>> >>>>> jdk.internal.jshell.debug >>>>> >>>>> Single class supporting debugging of the implementation >>>>> through external tools >>>>> >>>>> jdk.internal.jshell.tool >>>>> >>>>> The JShell tool (built on the API) >>>>> >>>>> >>>>> The package jdk.jshell has all the public API. >>>>> >>>>> jdk.internal.jshell.* includes some classes with public access >>>>> modifiers but these packages are not public APIs (we want the >>>>> ability to change them as needed) -- my understanding is that >>>>> packages with those characteristics should be named "jdk.internal.*". >>>>> >>>>> ----- >>>>> >>>>> However we have problems -- >>>>> >>>>>> some weird bug I encountered where if I set bootclasspath >>>>>> that mentions some jdk.internal classes (but which doesn't >>>>>> have JLine - because it comes from langtools), _every_ class >>>>>> from that package will not be fetched from the jimage file - >>>>>> meaning that I'll be left w/o jline. In other words, >>>>>> bootstrapping doesn't work - and I have reasons to believe >>>>>> that bootstrapping with the new module options (when they >>>>>> will become available - i.e. moduleOverride) won't work too. >>>>> >>>>> >>>>> and the proposal -- >>>>> >>>>>> * workaround problem #1 by putting jshell in a package other >>>>>> than jdk.internal (i.e. jdk.jshell could be good); this will >>>>>> avoid -Xbootstrap accidentally making jline unavailable >>>>> >>>>> >>>>> -Robert >>>>> >>>>> >>>> >>> >> From sundararajan.athijegannathan at oracle.com Fri Sep 18 04:14:16 2015 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Fri, 18 Sep 2015 09:44:16 +0530 Subject: JShell: packages In-Reply-To: <55FB59EA.5020108@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> <55FB59EA.5020108@oracle.com> Message-ID: <55FB8F98.80208@oracle.com> jline is used by nashorn's jjs tools as well (jdk.nashorn.tools.jjs.Main). Why should jline be on the same package as jshell? It is under jdk.internal.jline package (in jdk.internal.le module that is exposed to both jshell and nashorn's shell module). -Sundar On 9/18/2015 5:55 AM, Maurizio Cimadamore wrote: > > > On 18/09/15 01:20, Maurizio Cimadamore wrote: >> Now I'm confused > Less confused now - you prefer changing jline's package instead of > jshell's. That should work too (dunno if jshell uses other stuff > inside other jdk.internal sub-packages). > > Whatever works - as long as jshell, jline and any other JDK > dependencies end up with same package name. > > Maurizio From maurizio.cimadamore at oracle.com Fri Sep 18 10:22:03 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Fri, 18 Sep 2015 11:22:03 +0100 Subject: JShell: packages In-Reply-To: <55FB640A.2060409@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> <55FB59F6.5010308@oracle.com> <14fddf2f3b8.2784.4011f3a8741ca2aabce58b8b81f42d24@oracle.com> <55FB640A.2060409@oracle.com> Message-ID: <55FBE5CB.9060702@oracle.com> I suggest we restart from scratch - I don't know where the problem is exactly, but I don't see why the following is failing: $ /bin/java -Xbootclasspath/p:/jdk/modules/*jdk.jshell*:/jdk/modules/*jdk.compiler*:/jdk/modules/*java.compiler*jdk.internal.jshell.tool.JShellTool Exception in thread "main" java.lang.NoClassDefFoundError: jdk/internal/jline/TerminalFactory$Flavor at jdk.internal.jshell.tool.ConsoleIOContext.(ConsoleIOContext.java:70) at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:242) at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233) Why is the tool failing to locate jline? Note that JDK9 home is a recent JDK9 repo which has the jdk.internal.le package. By any chance, maybe jline is not part of rt.jar? Maurizio On 18/09/15 02:08, Jonathan Gibbons wrote: > Then I am indeed confused as to the issue/interaction between > -Xbootclasspath/p: and .jimage files. > > -- Jon > > On 09/17/2015 05:55 PM, Robert Field wrote: >> >> Jline is already in its own jdk.internal package. >> >> On September 17, 2015 5:25:27 PM Jonathan Gibbons >> wrote: >> >>> If the problem is just with jline, I was thinking we should >>> temporarily put it in its own package, such as jdk.internal.jline, >>> and leave alone the packages that Robert listed. >>> >>> Or am I missing something here? (again?) >>> >>> -- Jon >>> >>> On 09/17/2015 05:20 PM, Maurizio Cimadamore wrote: >>>> >>>> >>>> On 17/09/15 19:52, Jonathan Gibbons wrote: >>>>> We should not change any of these packages. >>>> Now I'm confused :-) >>>> >>>> As you say, we need a workaround for the Xbootclasspath issue - the >>>> simpler workaround would be to change the way jshell uses >>>> jdk.internal package names; I'm assuming fixing the bootstrap vs. >>>> jimage issue would take much longer. >>>> >>>> So, what do you mean by "we should not change any of these packages" ? >>>> >>>> Maurizio >>>>> >>>>> My understanding from Maurizio is that there is a bug or >>>>> anti-feature in the system today such that the choice of package >>>>> for jline is problematic, and so we were simply discussing the >>>>> possibility of using a different package for jline for the >>>>> immediate short term as a workaround for the -Xbootclasspath/p: >>>>> issue. When Jigsaw gets integrated, the -Xbootclasspath/p: >>>>> option will go away, and we will have to use the new -Xoverride >>>>> option instead. In advance of that, we should ensure that >>>>> -Xoverride works in a way that is satisfactory with jline in its >>>>> propoer long term home. >>>>> >>>>> -- Jon >>>>> >>>>> On 09/17/2015 11:08 AM, Robert Field wrote: >>>>>> There were extensive discussions and approvals around package >>>>>> names. This is what we have now: >>>>>> >>>>>> jdk.jshell >>>>>> >>>>>> JShell API and core implementation. >>>>>> >>>>>> jdk.internal.jshell.remote >>>>>> >>>>>> The remote side of the implementation >>>>>> >>>>>> jdk.internal.jshell.debug >>>>>> >>>>>> Single class supporting debugging of the implementation >>>>>> through external tools >>>>>> >>>>>> jdk.internal.jshell.tool >>>>>> >>>>>> The JShell tool (built on the API) >>>>>> >>>>>> >>>>>> The package jdk.jshell has all the public API. >>>>>> >>>>>> jdk.internal.jshell.* includes some classes with public access >>>>>> modifiers but these packages are not public APIs (we want the >>>>>> ability to change them as needed) -- my understanding is that >>>>>> packages with those characteristics should be named "jdk.internal.*". >>>>>> >>>>>> ----- >>>>>> >>>>>> However we have problems -- >>>>>> >>>>>>> some weird bug I encountered where if I set bootclasspath >>>>>>> that mentions some jdk.internal classes (but which doesn't >>>>>>> have JLine - because it comes from langtools), _every_ class >>>>>>> from that package will not be fetched from the jimage file - >>>>>>> meaning that I'll be left w/o jline. In other words, >>>>>>> bootstrapping doesn't work - and I have reasons to believe >>>>>>> that bootstrapping with the new module options (when they >>>>>>> will become available - i.e. moduleOverride) won't work too. >>>>>> >>>>>> >>>>>> and the proposal -- >>>>>> >>>>>>> * workaround problem #1 by putting jshell in a package other >>>>>>> than jdk.internal (i.e. jdk.jshell could be good); this will >>>>>>> avoid -Xbootstrap accidentally making jline unavailable >>>>>> >>>>>> >>>>>> -Robert >>>>>> >>>>>> >>>>> >>>> >>> > From maurizio.cimadamore at oracle.com Fri Sep 18 12:48:31 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Fri, 18 Sep 2015 13:48:31 +0100 Subject: JShell: packages In-Reply-To: <55FBE5CB.9060702@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> <55FB59F6.5010308@oracle.com> <14fddf2f3b8.2784.4011f3a8741ca2aabce58b8b81f42d24@oracle.com> <55FB640A.2060409@oracle.com> <55FBE5CB.9060702@oracle.com> Message-ID: <55FC081F.7000405@oracle.com> I had a productive chat with Alan - there's no bug here. The problem is caused by the fact that when you use bootclasspath, you force jshell to be loaded by the bootloader and, unfortunately, both jline and jdi are NOT in the bootloader (they are on the app loader); since bootclassloader doesn't delegate, you then get errors. As I understand, there's an hack to make this sort of things work with the exploded build, but it's just an accident that things work there. In other words, using -Xbootclasspath to launch jshell is NOT the right way to do things, as jshell depends on stuff that's not in the boot loader. Unfortunately, until -Xoverride is available, this means there's no solution, as what we really need here is a way to use a -classpath that overrides the JDK classes, but that is clearly not possible. To verify this, I've added both jdk.jdi and jdk.internal.le to this file in my vanilla JDK 9 repo: jdk/make/src/classes/build/tools/module/boot.modules And, after rebuilding, all the problems were gone, demonstrating that the issues were really originating from the classloader mismatch. Any thoughts? Maurizio On 18/09/15 11:22, Maurizio Cimadamore wrote: > I suggest we restart from scratch - I don't know where the problem is > exactly, but I don't see why the following is failing: > > > $ /bin/java > -Xbootclasspath/p:/jdk/modules/*jdk.jshell*:/jdk/modules/*jdk.compiler*:/jdk/modules/*java.compiler*jdk.internal.jshell.tool.JShellTool > Exception in thread "main" java.lang.NoClassDefFoundError: > jdk/internal/jline/TerminalFactory$Flavor > at > jdk.internal.jshell.tool.ConsoleIOContext.(ConsoleIOContext.java:70) > at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:242) > at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233) > > > Why is the tool failing to locate jline? Note that JDK9 home is a > recent JDK9 repo which has the jdk.internal.le package. > > By any chance, maybe jline is not part of rt.jar? > > Maurizio > > On 18/09/15 02:08, Jonathan Gibbons wrote: >> Then I am indeed confused as to the issue/interaction between >> -Xbootclasspath/p: and .jimage files. >> >> -- Jon >> >> On 09/17/2015 05:55 PM, Robert Field wrote: >>> >>> Jline is already in its own jdk.internal package. >>> >>> On September 17, 2015 5:25:27 PM Jonathan Gibbons >>> wrote: >>> >>>> If the problem is just with jline, I was thinking we should >>>> temporarily put it in its own package, such as jdk.internal.jline, >>>> and leave alone the packages that Robert listed. >>>> >>>> Or am I missing something here? (again?) >>>> >>>> -- Jon >>>> >>>> On 09/17/2015 05:20 PM, Maurizio Cimadamore wrote: >>>>> >>>>> >>>>> On 17/09/15 19:52, Jonathan Gibbons wrote: >>>>>> We should not change any of these packages. >>>>> Now I'm confused :-) >>>>> >>>>> As you say, we need a workaround for the Xbootclasspath issue - >>>>> the simpler workaround would be to change the way jshell uses >>>>> jdk.internal package names; I'm assuming fixing the bootstrap vs. >>>>> jimage issue would take much longer. >>>>> >>>>> So, what do you mean by "we should not change any of these >>>>> packages" ? >>>>> >>>>> Maurizio >>>>>> >>>>>> My understanding from Maurizio is that there is a bug or >>>>>> anti-feature in the system today such that the choice of package >>>>>> for jline is problematic, and so we were simply discussing the >>>>>> possibility of using a different package for jline for the >>>>>> immediate short term as a workaround for the -Xbootclasspath/p: >>>>>> issue. When Jigsaw gets integrated, the -Xbootclasspath/p: >>>>>> option will go away, and we will have to use the new -Xoverride >>>>>> option instead. In advance of that, we should ensure that >>>>>> -Xoverride works in a way that is satisfactory with jline in its >>>>>> propoer long term home. >>>>>> >>>>>> -- Jon >>>>>> >>>>>> On 09/17/2015 11:08 AM, Robert Field wrote: >>>>>>> There were extensive discussions and approvals around package >>>>>>> names. This is what we have now: >>>>>>> >>>>>>> jdk.jshell >>>>>>> >>>>>>> JShell API and core implementation. >>>>>>> >>>>>>> jdk.internal.jshell.remote >>>>>>> >>>>>>> The remote side of the implementation >>>>>>> >>>>>>> jdk.internal.jshell.debug >>>>>>> >>>>>>> Single class supporting debugging of the implementation >>>>>>> through external tools >>>>>>> >>>>>>> jdk.internal.jshell.tool >>>>>>> >>>>>>> The JShell tool (built on the API) >>>>>>> >>>>>>> >>>>>>> The package jdk.jshell has all the public API. >>>>>>> >>>>>>> jdk.internal.jshell.* includes some classes with public access >>>>>>> modifiers but these packages are not public APIs (we want the >>>>>>> ability to change them as needed) -- my understanding is that >>>>>>> packages with those characteristics should be named >>>>>>> "jdk.internal.*". >>>>>>> >>>>>>> ----- >>>>>>> >>>>>>> However we have problems -- >>>>>>> >>>>>>>> some weird bug I encountered where if I set bootclasspath >>>>>>>> that mentions some jdk.internal classes (but which doesn't >>>>>>>> have JLine - because it comes from langtools), _every_ class >>>>>>>> from that package will not be fetched from the jimage file - >>>>>>>> meaning that I'll be left w/o jline. In other words, >>>>>>>> bootstrapping doesn't work - and I have reasons to believe >>>>>>>> that bootstrapping with the new module options (when they >>>>>>>> will become available - i.e. moduleOverride) won't work too. >>>>>>> >>>>>>> >>>>>>> and the proposal -- >>>>>>> >>>>>>>> * workaround problem #1 by putting jshell in a package other >>>>>>>> than jdk.internal (i.e. jdk.jshell could be good); this will >>>>>>>> avoid -Xbootstrap accidentally making jline unavailable >>>>>>> >>>>>>> >>>>>>> -Robert >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> > From maurizio.cimadamore at oracle.com Fri Sep 18 12:49:22 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Fri, 18 Sep 2015 13:49:22 +0100 Subject: JShell: packages In-Reply-To: <55FC081F.7000405@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> <55FB59F6.5010308@oracle.com> <14fddf2f3b8.2784.4011f3a8741ca2aabce58b8b81f42d24@oracle.com> <55FB640A.2060409@oracle.com> <55FBE5CB.9060702@oracle.com> <55FC081F.7000405@oracle.com> Message-ID: <55FC0852.5060804@oracle.com> Adding Alan On 18/09/15 13:48, Maurizio Cimadamore wrote: > I had a productive chat with Alan - there's no bug here. The problem > is caused by the fact that when you use bootclasspath, you force > jshell to be loaded by the bootloader and, unfortunately, both jline > and jdi are NOT in the bootloader (they are on the app loader); since > bootclassloader doesn't delegate, you then get errors. As I > understand, there's an hack to make this sort of things work with the > exploded build, but it's just an accident that things work there. > > In other words, using -Xbootclasspath to launch jshell is NOT the > right way to do things, as jshell depends on stuff that's not in the > boot loader. Unfortunately, until -Xoverride is available, this means > there's no solution, as what we really need here is a way to use a > -classpath that overrides the JDK classes, but that is clearly not > possible. > > To verify this, I've added both jdk.jdi and jdk.internal.le to this > file in my vanilla JDK 9 repo: > > jdk/make/src/classes/build/tools/module/boot.modules > > And, after rebuilding, all the problems were gone, demonstrating that > the issues were really originating from the classloader mismatch. > > Any thoughts? > > Maurizio > > On 18/09/15 11:22, Maurizio Cimadamore wrote: >> I suggest we restart from scratch - I don't know where the problem is >> exactly, but I don't see why the following is failing: >> >> >> $ /bin/java >> -Xbootclasspath/p:/jdk/modules/*jdk.jshell*:/jdk/modules/*jdk.compiler*:/jdk/modules/*java.compiler*jdk.internal.jshell.tool.JShellTool >> Exception in thread "main" java.lang.NoClassDefFoundError: >> jdk/internal/jline/TerminalFactory$Flavor >> at >> jdk.internal.jshell.tool.ConsoleIOContext.(ConsoleIOContext.java:70) >> at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:242) >> at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233) >> >> >> Why is the tool failing to locate jline? Note that JDK9 home is a >> recent JDK9 repo which has the jdk.internal.le package. >> >> By any chance, maybe jline is not part of rt.jar? >> >> Maurizio >> >> On 18/09/15 02:08, Jonathan Gibbons wrote: >>> Then I am indeed confused as to the issue/interaction between >>> -Xbootclasspath/p: and .jimage files. >>> >>> -- Jon >>> >>> On 09/17/2015 05:55 PM, Robert Field wrote: >>>> >>>> Jline is already in its own jdk.internal package. >>>> >>>> On September 17, 2015 5:25:27 PM Jonathan Gibbons >>>> wrote: >>>> >>>>> If the problem is just with jline, I was thinking we should >>>>> temporarily put it in its own package, such as jdk.internal.jline, >>>>> and leave alone the packages that Robert listed. >>>>> >>>>> Or am I missing something here? (again?) >>>>> >>>>> -- Jon >>>>> >>>>> On 09/17/2015 05:20 PM, Maurizio Cimadamore wrote: >>>>>> >>>>>> >>>>>> On 17/09/15 19:52, Jonathan Gibbons wrote: >>>>>>> We should not change any of these packages. >>>>>> Now I'm confused :-) >>>>>> >>>>>> As you say, we need a workaround for the Xbootclasspath issue - >>>>>> the simpler workaround would be to change the way jshell uses >>>>>> jdk.internal package names; I'm assuming fixing the bootstrap vs. >>>>>> jimage issue would take much longer. >>>>>> >>>>>> So, what do you mean by "we should not change any of these >>>>>> packages" ? >>>>>> >>>>>> Maurizio >>>>>>> >>>>>>> My understanding from Maurizio is that there is a bug or >>>>>>> anti-feature in the system today such that the choice of package >>>>>>> for jline is problematic, and so we were simply discussing the >>>>>>> possibility of using a different package for jline for the >>>>>>> immediate short term as a workaround for the -Xbootclasspath/p: >>>>>>> issue. When Jigsaw gets integrated, the -Xbootclasspath/p: >>>>>>> option will go away, and we will have to use the new -Xoverride >>>>>>> option instead. In advance of that, we should ensure that >>>>>>> -Xoverride works in a way that is satisfactory with jline in its >>>>>>> propoer long term home. >>>>>>> >>>>>>> -- Jon >>>>>>> >>>>>>> On 09/17/2015 11:08 AM, Robert Field wrote: >>>>>>>> There were extensive discussions and approvals around package >>>>>>>> names. This is what we have now: >>>>>>>> >>>>>>>> jdk.jshell >>>>>>>> >>>>>>>> JShell API and core implementation. >>>>>>>> >>>>>>>> jdk.internal.jshell.remote >>>>>>>> >>>>>>>> The remote side of the implementation >>>>>>>> >>>>>>>> jdk.internal.jshell.debug >>>>>>>> >>>>>>>> Single class supporting debugging of the implementation >>>>>>>> through external tools >>>>>>>> >>>>>>>> jdk.internal.jshell.tool >>>>>>>> >>>>>>>> The JShell tool (built on the API) >>>>>>>> >>>>>>>> >>>>>>>> The package jdk.jshell has all the public API. >>>>>>>> >>>>>>>> jdk.internal.jshell.* includes some classes with public access >>>>>>>> modifiers but these packages are not public APIs (we want the >>>>>>>> ability to change them as needed) -- my understanding is that >>>>>>>> packages with those characteristics should be named >>>>>>>> "jdk.internal.*". >>>>>>>> >>>>>>>> ----- >>>>>>>> >>>>>>>> However we have problems -- >>>>>>>> >>>>>>>>> some weird bug I encountered where if I set bootclasspath >>>>>>>>> that mentions some jdk.internal classes (but which doesn't >>>>>>>>> have JLine - because it comes from langtools), _every_ class >>>>>>>>> from that package will not be fetched from the jimage file - >>>>>>>>> meaning that I'll be left w/o jline. In other words, >>>>>>>>> bootstrapping doesn't work - and I have reasons to believe >>>>>>>>> that bootstrapping with the new module options (when they >>>>>>>>> will become available - i.e. moduleOverride) won't work too. >>>>>>>> >>>>>>>> >>>>>>>> and the proposal -- >>>>>>>> >>>>>>>>> * workaround problem #1 by putting jshell in a package other >>>>>>>>> than jdk.internal (i.e. jdk.jshell could be good); this will >>>>>>>>> avoid -Xbootstrap accidentally making jline unavailable >>>>>>>> >>>>>>>> >>>>>>>> -Robert >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >> > From sundararajan.athijegannathan at oracle.com Fri Sep 18 13:00:59 2015 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Fri, 18 Sep 2015 18:30:59 +0530 Subject: JShell: packages In-Reply-To: <55FBE5CB.9060702@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> <55FB59F6.5010308@oracle.com> <14fddf2f3b8.2784.4011f3a8741ca2aabce58b8b81f42d24@oracle.com> <55FB640A.2060409@oracle.com> <55FBE5CB.9060702@oracle.com> Message-ID: <55FC0B0B.9000200@oracle.com> I think I got the issue here. There is no bug in the way -Xbootclasspath/p: works. There are three .jimage files in jdk9-dev - bootmodules.jimage, extmodules.jimage, appmodules.jimage. Boot, extension and "app"/"launcher" loaders load classes from these .jimage files respectively. jdk.internal.le modules (which contains jline classes in jdk.internal.jline.* packages) is part of appmodules.jimage - and hence would be loaded by app/launcher loader. jshell classes also live in the same .jimage file. But, with your command line, you're putting jshell classes alone in bootstrap class path - leaving jline classes in appmodules.jimage. Bootstrap loader will only search classes in your prepended boot path and bootmodules.jimage and so won't find jline classes. You need to put all dependencies of jshell in bootclasspath/p: -Sundar On 9/18/2015 3:52 PM, Maurizio Cimadamore wrote: > I suggest we restart from scratch - I don't know where the problem is > exactly, but I don't see why the following is failing: > > > $ /bin/java > -Xbootclasspath/p:/jdk/modules/*jdk.jshell*:/jdk/modules/*jdk.compiler*:/jdk/modules/*java.compiler*jdk.internal.jshell.tool.JShellTool > Exception in thread "main" java.lang.NoClassDefFoundError: > jdk/internal/jline/TerminalFactory$Flavor > at > jdk.internal.jshell.tool.ConsoleIOContext.(ConsoleIOContext.java:70) > at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:242) > at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233) > > > Why is the tool failing to locate jline? Note that JDK9 home is a > recent JDK9 repo which has the jdk.internal.le package. > > By any chance, maybe jline is not part of rt.jar? > > Maurizio > > On 18/09/15 02:08, Jonathan Gibbons wrote: >> Then I am indeed confused as to the issue/interaction between >> -Xbootclasspath/p: and .jimage files. >> >> -- Jon >> >> On 09/17/2015 05:55 PM, Robert Field wrote: >>> >>> Jline is already in its own jdk.internal package. >>> >>> On September 17, 2015 5:25:27 PM Jonathan Gibbons >>> wrote: >>> >>>> If the problem is just with jline, I was thinking we should >>>> temporarily put it in its own package, such as jdk.internal.jline, >>>> and leave alone the packages that Robert listed. >>>> >>>> Or am I missing something here? (again?) >>>> >>>> -- Jon >>>> >>>> On 09/17/2015 05:20 PM, Maurizio Cimadamore wrote: >>>>> >>>>> >>>>> On 17/09/15 19:52, Jonathan Gibbons wrote: >>>>>> We should not change any of these packages. >>>>> Now I'm confused :-) >>>>> >>>>> As you say, we need a workaround for the Xbootclasspath issue - >>>>> the simpler workaround would be to change the way jshell uses >>>>> jdk.internal package names; I'm assuming fixing the bootstrap vs. >>>>> jimage issue would take much longer. >>>>> >>>>> So, what do you mean by "we should not change any of these >>>>> packages" ? >>>>> >>>>> Maurizio >>>>>> >>>>>> My understanding from Maurizio is that there is a bug or >>>>>> anti-feature in the system today such that the choice of package >>>>>> for jline is problematic, and so we were simply discussing the >>>>>> possibility of using a different package for jline for the >>>>>> immediate short term as a workaround for the -Xbootclasspath/p: >>>>>> issue. When Jigsaw gets integrated, the -Xbootclasspath/p: >>>>>> option will go away, and we will have to use the new -Xoverride >>>>>> option instead. In advance of that, we should ensure that >>>>>> -Xoverride works in a way that is satisfactory with jline in its >>>>>> propoer long term home. >>>>>> >>>>>> -- Jon >>>>>> >>>>>> On 09/17/2015 11:08 AM, Robert Field wrote: >>>>>>> There were extensive discussions and approvals around package >>>>>>> names. This is what we have now: >>>>>>> >>>>>>> jdk.jshell >>>>>>> >>>>>>> JShell API and core implementation. >>>>>>> >>>>>>> jdk.internal.jshell.remote >>>>>>> >>>>>>> The remote side of the implementation >>>>>>> >>>>>>> jdk.internal.jshell.debug >>>>>>> >>>>>>> Single class supporting debugging of the implementation >>>>>>> through external tools >>>>>>> >>>>>>> jdk.internal.jshell.tool >>>>>>> >>>>>>> The JShell tool (built on the API) >>>>>>> >>>>>>> >>>>>>> The package jdk.jshell has all the public API. >>>>>>> >>>>>>> jdk.internal.jshell.* includes some classes with public access >>>>>>> modifiers but these packages are not public APIs (we want the >>>>>>> ability to change them as needed) -- my understanding is that >>>>>>> packages with those characteristics should be named >>>>>>> "jdk.internal.*". >>>>>>> >>>>>>> ----- >>>>>>> >>>>>>> However we have problems -- >>>>>>> >>>>>>>> some weird bug I encountered where if I set bootclasspath >>>>>>>> that mentions some jdk.internal classes (but which doesn't >>>>>>>> have JLine - because it comes from langtools), _every_ class >>>>>>>> from that package will not be fetched from the jimage file - >>>>>>>> meaning that I'll be left w/o jline. In other words, >>>>>>>> bootstrapping doesn't work - and I have reasons to believe >>>>>>>> that bootstrapping with the new module options (when they >>>>>>>> will become available - i.e. moduleOverride) won't work too. >>>>>>> >>>>>>> >>>>>>> and the proposal -- >>>>>>> >>>>>>>> * workaround problem #1 by putting jshell in a package other >>>>>>>> than jdk.internal (i.e. jdk.jshell could be good); this will >>>>>>>> avoid -Xbootstrap accidentally making jline unavailable >>>>>>> >>>>>>> >>>>>>> -Robert >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> > From sundararajan.athijegannathan at oracle.com Fri Sep 18 13:05:23 2015 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Fri, 18 Sep 2015 18:35:23 +0530 Subject: JShell: packages In-Reply-To: <55FC0B0B.9000200@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> <55FB59F6.5010308@oracle.com> <14fddf2f3b8.2784.4011f3a8741ca2aabce58b8b81f42d24@oracle.com> <55FB640A.2060409@oracle.com> <55FBE5CB.9060702@oracle.com> <55FC0B0B.9000200@oracle.com> Message-ID: <55FC0C13.3030104@oracle.com> Oops.. sorry I just saw Maurizio's email which was sent earlier. Please ignore my email.. (similar content). No bug in -Xbootclasspath/p: -Sundar On 9/18/2015 6:30 PM, Sundararajan Athijegannathan wrote: > I think I got the issue here. There is no bug in the way > -Xbootclasspath/p: works. > > There are three .jimage files in jdk9-dev - bootmodules.jimage, > extmodules.jimage, appmodules.jimage. Boot, extension and > "app"/"launcher" loaders load classes from these .jimage files > respectively. > > jdk.internal.le modules (which contains jline classes in > jdk.internal.jline.* packages) is part of appmodules.jimage - and > hence would be loaded by app/launcher loader. jshell classes also live > in the same .jimage file. > > But, with your command line, you're putting jshell classes alone in > bootstrap class path - leaving jline classes in appmodules.jimage. > Bootstrap loader will only search classes in your prepended boot path > and bootmodules.jimage and so won't find jline classes. You need to > put all dependencies of jshell in bootclasspath/p: > > -Sundar > > On 9/18/2015 3:52 PM, Maurizio Cimadamore wrote: >> I suggest we restart from scratch - I don't know where the problem is >> exactly, but I don't see why the following is failing: >> >> >> $ /bin/java >> -Xbootclasspath/p:/jdk/modules/*jdk.jshell*:/jdk/modules/*jdk.compiler*:/jdk/modules/*java.compiler*jdk.internal.jshell.tool.JShellTool >> Exception in thread "main" java.lang.NoClassDefFoundError: >> jdk/internal/jline/TerminalFactory$Flavor >> at >> jdk.internal.jshell.tool.ConsoleIOContext.(ConsoleIOContext.java:70) >> at jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:242) >> at jdk.internal.jshell.tool.JShellTool.main(JShellTool.java:233) >> >> >> Why is the tool failing to locate jline? Note that JDK9 home is a >> recent JDK9 repo which has the jdk.internal.le package. >> >> By any chance, maybe jline is not part of rt.jar? >> >> Maurizio >> >> On 18/09/15 02:08, Jonathan Gibbons wrote: >>> Then I am indeed confused as to the issue/interaction between >>> -Xbootclasspath/p: and .jimage files. >>> >>> -- Jon >>> >>> On 09/17/2015 05:55 PM, Robert Field wrote: >>>> >>>> Jline is already in its own jdk.internal package. >>>> >>>> On September 17, 2015 5:25:27 PM Jonathan Gibbons >>>> wrote: >>>> >>>>> If the problem is just with jline, I was thinking we should >>>>> temporarily put it in its own package, such as jdk.internal.jline, >>>>> and leave alone the packages that Robert listed. >>>>> >>>>> Or am I missing something here? (again?) >>>>> >>>>> -- Jon >>>>> >>>>> On 09/17/2015 05:20 PM, Maurizio Cimadamore wrote: >>>>>> >>>>>> >>>>>> On 17/09/15 19:52, Jonathan Gibbons wrote: >>>>>>> We should not change any of these packages. >>>>>> Now I'm confused :-) >>>>>> >>>>>> As you say, we need a workaround for the Xbootclasspath issue - >>>>>> the simpler workaround would be to change the way jshell uses >>>>>> jdk.internal package names; I'm assuming fixing the bootstrap vs. >>>>>> jimage issue would take much longer. >>>>>> >>>>>> So, what do you mean by "we should not change any of these >>>>>> packages" ? >>>>>> >>>>>> Maurizio >>>>>>> >>>>>>> My understanding from Maurizio is that there is a bug or >>>>>>> anti-feature in the system today such that the choice of package >>>>>>> for jline is problematic, and so we were simply discussing the >>>>>>> possibility of using a different package for jline for the >>>>>>> immediate short term as a workaround for the -Xbootclasspath/p: >>>>>>> issue. When Jigsaw gets integrated, the -Xbootclasspath/p: >>>>>>> option will go away, and we will have to use the new -Xoverride >>>>>>> option instead. In advance of that, we should ensure that >>>>>>> -Xoverride works in a way that is satisfactory with jline in its >>>>>>> propoer long term home. >>>>>>> >>>>>>> -- Jon >>>>>>> >>>>>>> On 09/17/2015 11:08 AM, Robert Field wrote: >>>>>>>> There were extensive discussions and approvals around package >>>>>>>> names. This is what we have now: >>>>>>>> >>>>>>>> jdk.jshell >>>>>>>> >>>>>>>> JShell API and core implementation. >>>>>>>> >>>>>>>> jdk.internal.jshell.remote >>>>>>>> >>>>>>>> The remote side of the implementation >>>>>>>> >>>>>>>> jdk.internal.jshell.debug >>>>>>>> >>>>>>>> Single class supporting debugging of the implementation >>>>>>>> through external tools >>>>>>>> >>>>>>>> jdk.internal.jshell.tool >>>>>>>> >>>>>>>> The JShell tool (built on the API) >>>>>>>> >>>>>>>> >>>>>>>> The package jdk.jshell has all the public API. >>>>>>>> >>>>>>>> jdk.internal.jshell.* includes some classes with public access >>>>>>>> modifiers but these packages are not public APIs (we want the >>>>>>>> ability to change them as needed) -- my understanding is that >>>>>>>> packages with those characteristics should be named >>>>>>>> "jdk.internal.*". >>>>>>>> >>>>>>>> ----- >>>>>>>> >>>>>>>> However we have problems -- >>>>>>>> >>>>>>>>> some weird bug I encountered where if I set bootclasspath >>>>>>>>> that mentions some jdk.internal classes (but which doesn't >>>>>>>>> have JLine - because it comes from langtools), _every_ class >>>>>>>>> from that package will not be fetched from the jimage file - >>>>>>>>> meaning that I'll be left w/o jline. In other words, >>>>>>>>> bootstrapping doesn't work - and I have reasons to believe >>>>>>>>> that bootstrapping with the new module options (when they >>>>>>>>> will become available - i.e. moduleOverride) won't work too. >>>>>>>> >>>>>>>> >>>>>>>> and the proposal -- >>>>>>>> >>>>>>>>> * workaround problem #1 by putting jshell in a package other >>>>>>>>> than jdk.internal (i.e. jdk.jshell could be good); this will >>>>>>>>> avoid -Xbootstrap accidentally making jline unavailable >>>>>>>> >>>>>>>> >>>>>>>> -Robert >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >> > From robert.field at oracle.com Sat Sep 19 19:00:22 2015 From: robert.field at oracle.com (Robert Field) Date: Sat, 19 Sep 2015 12:00:22 -0700 Subject: REPL code review -- ReplParser In-Reply-To: <55F2F270.8020803@oracle.com> References: <55F2F270.8020803@oracle.com> Message-ID: <55FDB0C6.7040209@oracle.com> On 09/11/15 08:25, Maurizio Cimadamore wrote: > > * ReplParser - in general it looks good. The question I have here is > more design-oriented: why do we need it? I believe the main use case > is that you need to parse the input source to be able to decide > between expressions and declarations. Can't we wrap the snippet in the > following two ways: > > class Temp$Decl { > > } > > and: > > class Temp$Expr { > Object o = > } > > and then see who wins? This would also require also to move the > wrapping logic from Eval to some earlier stage (when the input is > deemed complete). It's probably a very big change, so I'm merely > throwing this as a suggestion for future improvements - it's outside > the scope of the review. Were it so easy ;-) Trial-and-error was the first approach I took. Remember there are statements and imports. But much more importantly the problem is determining "who wins". You are likely to get errors for any number of reasons other than not having the snippet shoved in the right context. Within the processing of each type you will notice that the trail-and-error approach is still used in some cases to get the full context. Nesting trail-and-error in trail-and-error was just too wobbly a foundation. -Robert From robert.field at oracle.com Sat Sep 19 19:29:59 2015 From: robert.field at oracle.com (Robert Field) Date: Sat, 19 Sep 2015 12:29:59 -0700 Subject: REPL code review -- TypePrinter In-Reply-To: <55F2F270.8020803@oracle.com> References: <55F2F270.8020803@oracle.com> Message-ID: <55FDB7B7.40105@oracle.com> On 09/11/15 08:25, Maurizio Cimadamore wrote: > * TypePrinter - so, this is needed when you need to print the type of > some expression; it seems the exact copy of the printer used by the > RichDiagnosticFormatter, except for the fact that you want to omit the > captured type-variable details. So, maybe you could split this process > in two: first you map the original type onto a type where all captured > variables have been replaced by their bounds (i.e from List<#CAP1> to > List) - then you call the RichDiagnosticFormatter printer onto > it (the best way to do that would probably be to create a fake > diagnostic with a type as its argument, and then feed that to the > diagnostic formatter - that should give you back a string). The > RichDiagnosticFormatter, by default will also generate stuff that you > probably don't want, such as 'where' clauses to disambiguate > type-variable names with the same names but different owners - but > such features can be selectively disabled (via javac command line > options). By "the printer used by the RichDiagnosticFormatter" do you mean RichDiagnosticFormatter.RichPrinter? They are both subclasses of Printer and share similar structures. That approach seems convoluted and fragile. Am I missing something? -Robert From robert.field at oracle.com Sat Sep 19 21:38:32 2015 From: robert.field at oracle.com (Robert Field) Date: Sat, 19 Sep 2015 14:38:32 -0700 Subject: REPL code review In-Reply-To: <55F2F270.8020803@oracle.com> References: <55F2F270.8020803@oracle.com> Message-ID: <55FDD5D8.6050500@oracle.com> On 09/11/15 08:25, Maurizio Cimadamore wrote: > > * MemoryFileManager > > - Maybe the code of 'list' could be more optimized/lazy by > returning a compound iterator - i.e. an iterator that keeps scanning > the items in the first iterator and, when finished moves onto the next > iterator. How's this? It could be more or less lazy: "it" could be pre-filled, or "stdList" could be called at the last minute. @Override public Iterable list(JavaFileManager.Location location, String packageName, Set kinds, boolean recurse) throws IOException { Iterable stdList = stdFileManager.list(location, packageName, kinds, recurse); if (location==CLASS_PATH && packageName.equals("REPL")) { // if the desired list is for our JShell package, lazily iterate over // first the standard list then any generated classes. return () -> new Iterator() { boolean stdDone = false; Iterator it; @Override public boolean hasNext() { if (it == null) { it = stdList.iterator(); } if (it.hasNext()) { return true; } if (stdDone) { return false; } else { stdDone = true; it = generatedClasses().iterator(); return it.hasNext(); } } @Override public JavaFileObject next() { if (!hasNext()) { throw new NoSuchElementException(); } return it.next(); } }; } else { return stdList; } } > > - comments: is it normal to repeat all comments from the > JavaFileManager interface? I don't think it is normal. I've found it very useful to have them there both during implementation and maintenance. If you feel they should be removed, I will. > > - watch out for unused methods: dumpClasses, findGeneratedClass, > findGeneratedBytes, inferModuleName, listModueLocations (maybe this > stuff is for Jigsaw? If so, maybe add a comment) I've added comment (in my local repo). // For debugging dumps dumpClasses // For restoring process-local execution support findGeneratedClass, findGeneratedBytes and several others // Make compatible with Jigsaw inferModuleName, listModueLocations Thanks, Robert From maurizio.cimadamore at oracle.com Mon Sep 21 00:06:30 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 21 Sep 2015 01:06:30 +0100 Subject: REPL code review -- TypePrinter In-Reply-To: <55FDB7B7.40105@oracle.com> References: <55F2F270.8020803@oracle.com> <55FDB7B7.40105@oracle.com> Message-ID: <55FF4A06.90409@oracle.com> On 19/09/15 20:29, Robert Field wrote: > > On 09/11/15 08:25, Maurizio Cimadamore wrote: >> * TypePrinter - so, this is needed when you need to print the type of >> some expression; it seems the exact copy of the printer used by the >> RichDiagnosticFormatter, except for the fact that you want to omit >> the captured type-variable details. So, maybe you could split this >> process in two: first you map the original type onto a type where all >> captured variables have been replaced by their bounds (i.e from >> List<#CAP1> to List) - then you call the >> RichDiagnosticFormatter printer onto it (the best way to do that >> would probably be to create a fake diagnostic with a type as its >> argument, and then feed that to the diagnostic formatter - that >> should give you back a string). The RichDiagnosticFormatter, by >> default will also generate stuff that you probably don't want, such >> as 'where' clauses to disambiguate type-variable names with the same >> names but different owners - but such features can be selectively >> disabled (via javac command line options). > > By "the printer used by the RichDiagnosticFormatter" do you mean > RichDiagnosticFormatter.RichPrinter? They are both subclasses of > Printer and share similar structures. > > That approach seems convoluted and fragile. Am I missing something? I'm just saying that there's a very sophisticated and configurable type/symbol printer (RichDiagnosticFormatter.RichPrinter) - so it seems a bit odd to see some of the same goodies available in that printer being almost exactly replicated there. If there was a way to reuse the rich printer code in some way, I would say that should lead to more maintainable code in the long run. Maurizio > > -Robert > From maurizio.cimadamore at oracle.com Mon Sep 21 00:10:23 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 21 Sep 2015 01:10:23 +0100 Subject: REPL code review In-Reply-To: <55FDD5D8.6050500@oracle.com> References: <55F2F270.8020803@oracle.com> <55FDD5D8.6050500@oracle.com> Message-ID: <55FF4AEF.1050600@oracle.com> On 19/09/15 22:38, Robert Field wrote: > > On 09/11/15 08:25, Maurizio Cimadamore wrote: >> >> * MemoryFileManager >> >> - Maybe the code of 'list' could be more optimized/lazy by >> returning a compound iterator - i.e. an iterator that keeps scanning >> the items in the first iterator and, when finished moves onto the >> next iterator. > > How's this? It could be more or less lazy: "it" could be pre-filled, > or "stdList" could be called at the last minute. This looks exactly what I was thinking - thanks. > > @Override > public Iterable list(JavaFileManager.Location > location, > String packageName, > Set kinds, > boolean recurse) > throws IOException { > Iterable stdList = > stdFileManager.list(location, packageName, kinds, recurse); > if (location==CLASS_PATH && packageName.equals("REPL")) { > // if the desired list is for our JShell package, lazily > iterate over > // first the standard list then any generated classes. > return () -> new Iterator() { > boolean stdDone = false; > Iterator it; > > @Override > public boolean hasNext() { > if (it == null) { > it = stdList.iterator(); > } > if (it.hasNext()) { > return true; > } > if (stdDone) { > return false; > } else { > stdDone = true; > it = generatedClasses().iterator(); > return it.hasNext(); > } > } > > @Override > public JavaFileObject next() { > if (!hasNext()) { > throw new NoSuchElementException(); > } > return it.next(); > } > > }; > } else { > return stdList; > } > } > >> >> - comments: is it normal to repeat all comments from the >> JavaFileManager interface? > > I don't think it is normal. > > I've found it very useful to have them there both during > implementation and maintenance. If you feel they should be removed, I > will. I believe most of the code I've seen around tends not to repeat the documentation, unless something significantly different occurs in the method, so that you feel like you need to extend the javadoc in some way. But if you feel there's some value in keeping the docs, then again I'm not strongly against it - just noting it is mildly unusual. > >> >> - watch out for unused methods: dumpClasses, findGeneratedClass, >> findGeneratedBytes, inferModuleName, listModueLocations (maybe this >> stuff is for Jigsaw? If so, maybe add a comment) > > I've added comment (in my local repo). > > // For debugging dumps > dumpClasses > > // For restoring process-local execution support > findGeneratedClass, findGeneratedBytes > and several others > > // Make compatible with Jigsaw > inferModuleName, listModueLocations Thanks Maurizio > > Thanks, > Robert > From andrei.eremeev at oracle.com Mon Sep 21 10:02:16 2015 From: andrei.eremeev at oracle.com (andrei.eremeev at oracle.com) Date: Mon, 21 Sep 2015 10:02:16 +0000 Subject: hg: kulla/dev/langtools: 8136822: JShell tool: Fix tests for EditorPad and ExternalEditor Message-ID: <201509211002.t8LA2Gbs018501@aojmv0008.oracle.com> Changeset: 316b3da0f027 Author: aeremeev Date: 2015-09-21 13:00 +0300 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/316b3da0f027 8136822: JShell tool: Fix tests for EditorPad and ExternalEditor - test/jdk/jshell/EditorTest.java ! test/jdk/jshell/EditorTestBase.java + test/jdk/jshell/ExternalEditorTest.java ! test/jdk/jshell/testng.xml From robert.field at oracle.com Mon Sep 21 15:09:53 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Mon, 21 Sep 2015 15:09:53 +0000 Subject: hg: kulla/dev/langtools: Per review of MemoryFileManager, compound iterator for lazy list(), and comments on unused methods Message-ID: <201509211509.t8LF9r9l027532@aojmv0008.oracle.com> Changeset: b7e1316cb68c Author: rfield Date: 2015-09-21 08:09 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/b7e1316cb68c Per review of MemoryFileManager, compound iterator for lazy list(), and comments on unused methods ! src/jdk.jshell/share/classes/jdk/jshell/MemoryFileManager.java From robert.field at oracle.com Mon Sep 21 15:11:05 2015 From: robert.field at oracle.com (Robert Field) Date: Mon, 21 Sep 2015 08:11:05 -0700 Subject: REPL code review -- MemoryFileManager In-Reply-To: <55FDD5D8.6050500@oracle.com> References: <55F2F270.8020803@oracle.com> <55FDD5D8.6050500@oracle.com> Message-ID: <56001E09.4020707@oracle.com> Fix pushed. -Robert On 09/19/15 14:38, Robert Field wrote: > > On 09/11/15 08:25, Maurizio Cimadamore wrote: >> >> * MemoryFileManager >> >> - Maybe the code of 'list' could be more optimized/lazy by >> returning a compound iterator - i.e. an iterator that keeps scanning >> the items in the first iterator and, when finished moves onto the >> next iterator. > > How's this? It could be more or less lazy: "it" could be pre-filled, > or "stdList" could be called at the last minute. > > @Override > public Iterable list(JavaFileManager.Location > location, > String packageName, > Set kinds, > boolean recurse) > throws IOException { > Iterable stdList = > stdFileManager.list(location, packageName, kinds, recurse); > if (location==CLASS_PATH && packageName.equals("REPL")) { > // if the desired list is for our JShell package, lazily > iterate over > // first the standard list then any generated classes. > return () -> new Iterator() { > boolean stdDone = false; > Iterator it; > > @Override > public boolean hasNext() { > if (it == null) { > it = stdList.iterator(); > } > if (it.hasNext()) { > return true; > } > if (stdDone) { > return false; > } else { > stdDone = true; > it = generatedClasses().iterator(); > return it.hasNext(); > } > } > > @Override > public JavaFileObject next() { > if (!hasNext()) { > throw new NoSuchElementException(); > } > return it.next(); > } > > }; > } else { > return stdList; > } > } > >> >> - comments: is it normal to repeat all comments from the >> JavaFileManager interface? > > I don't think it is normal. > > I've found it very useful to have them there both during > implementation and maintenance. If you feel they should be removed, I > will. > >> >> - watch out for unused methods: dumpClasses, findGeneratedClass, >> findGeneratedBytes, inferModuleName, listModueLocations (maybe this >> stuff is for Jigsaw? If so, maybe add a comment) > > I've added comment (in my local repo). > > // For debugging dumps > dumpClasses > > // For restoring process-local execution support > findGeneratedClass, findGeneratedBytes > and several others > > // Make compatible with Jigsaw > inferModuleName, listModueLocations > > Thanks, > Robert > From robert.field at oracle.com Tue Sep 22 05:46:15 2015 From: robert.field at oracle.com (Robert Field) Date: Mon, 21 Sep 2015 22:46:15 -0700 Subject: REPL code review -- CompletenessAnalyzer In-Reply-To: <55F2F270.8020803@oracle.com> References: <55F2F270.8020803@oracle.com> Message-ID: <5600EB27.6040104@oracle.com> On 09/11/15 08:25, Maurizio Cimadamore wrote: > * CompletenessAnalyzer > > - Location position constants appear more complex than necessary > (or I'm missing something); it appears that constants like XEXPR1o, > XDECL1o, XSTMT1o are never used? The only uages I see are two: > > - to build other constants (such as XEXPR1, XDECL1 and XSTMT1) > > - in a switch inside parseUnit Exactly, they define the bits and the semantics of those bits. They are applied as a set of bits. The comments were broken, I'm updating as such -- // Location position kinds -- a token is ... private static final int XEXPR = 0b1; // OK in expression (not first) private static final int XDECL = 0b10; // OK in declaration (not first) private static final int XSTMT = 0b100; // OK in statement framework (not first) private static final int XEXPR1o = 0b1000; // OK first in expression private static final int XDECL1o = 0b10000; // OK first in declaration private static final int XSTMT1o = 0b100000; // OK first or only in statement framework private static final int XEXPR1 = XEXPR1o | XEXPR; // OK in expression (anywhere) private static final int XDECL1 = XDECL1o | XDECL; // OK in declaration (anywhere) private static final int XSTMT1 = XSTMT1o | XSTMT; // OK in statement framework (anywhere) private static final int XANY1 = XEXPR1o | XDECL1o | XSTMT1o; // Mask: first in statement, declaration, or expression private static final int XTERM = 0b100000000; // Can terminate (last before EOF) private static final int XERRO = 0b1000000000; // Is an error So, for example "boolean" is XEXPR | XDECL1, which is XEXPR | XDECL | XDECL1o -- meaning it can occur in an expression (in a cast), or at the beginning or later in a declaration. If we encounter "boolean" at the beginning of a snippet, parseUnit() masks to just look at allowed in first position, in this case XDECL1o. So, it knows it is the beginning of a declaration. > > Since no token is really using those constants explicitly, maybe they > could be removed in favor of their more general counterparts (this > also kind of means that i.e. XEXPR1 == XEXPR, etc.) See above, it is being used. > > - I believe in general the design of location codes + 'belongs' > field feels weird, and given this code is quite convoluted (because > the task is hard!) the code might benefit from some cleanup. My > suggestions: > (1) come up with a cleaner enum which contains: { EXPR, DECL, > STMT, AMBIGUOUS, ERR } - use this for classification purposes in > parseUnit and related predicates (i.e. what is this token like?) > (2) extract the 'isOkToTerminate' property onto a separate boolean > flag on the token. After all, this is an orthogonal property of all > tokens. Does my explanation clarify? Am I still missing a simplification? An enum won't work as they are unions of possibilities. > > - watch out for commented out code/dead code (here and elsewhere) > > Sometimes the completeness analysis doesn't behave as expected: > > -> new > >> ; > >> ; > >> ; > >> ; > >> ; > >> ; > >> Foo(); > | Error: > | expected > | new > | ^ > | Error: > | '(' or '[' expected > | ; > | ^ > > In the above, I found it confusing that the ';' is not enough to tell > the analysis logic that I'm done with the statement - and the analysis > will keep waiting until it matches a '()'. > OK, "new" has special processing that has makes it error prone. In the webrev below I remove that special processing at a small loss of error detection, and add a general check for bad sequencing. http://cr.openjdk.java.net/~rfield/CompletenessAnalyzer_v0/ This fixes "new ;" but does not fix "(;". The lexical analysis does not have enough context to know this isn't the beginning of a for-statement. Basically, everything between matching parenthesis, brackets, or braces is currently ignored -- effectively "(...)", "[...]", or "{...}". A result of this is, for example, that "else" is returned from completion as "unknown" (erroneous) so it is immediately parsed and an error returned; Whereas "(else" is returned from completion as "incomplete" prompting for more input. To fix this, the nesting analysis would need to be moved into the parser (where is typically belongs) which would be a fairly significant change. I will poke at that and see how complex it is. Your thoughts on its importance? Thanks, Robert From maurizio.cimadamore at oracle.com Tue Sep 22 09:50:21 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Tue, 22 Sep 2015 10:50:21 +0100 Subject: REPL code review -- CompletenessAnalyzer In-Reply-To: <5600EB27.6040104@oracle.com> References: <55F2F270.8020803@oracle.com> <5600EB27.6040104@oracle.com> Message-ID: <5601245D.5050902@oracle.com> On 22/09/15 06:46, Robert Field wrote: > > On 09/11/15 08:25, Maurizio Cimadamore wrote: >> * CompletenessAnalyzer >> >> - Location position constants appear more complex than necessary >> (or I'm missing something); it appears that constants like XEXPR1o, >> XDECL1o, XSTMT1o are never used? The only uages I see are two: >> >> - to build other constants (such as XEXPR1, XDECL1 and XSTMT1) >> >> - in a switch inside parseUnit > > Exactly, they define the bits and the semantics of those bits. They > are applied as a set of bits. > > The comments were broken, I'm updating as such -- > > // Location position kinds -- a token is ... > private static final int XEXPR = > 0b1; // OK in expression (not first) > private static final int XDECL = > 0b10; // OK in declaration (not first) > private static final int XSTMT = > 0b100; // OK in statement framework (not first) > private static final int XEXPR1o = > 0b1000; // OK first in expression > private static final int XDECL1o = > 0b10000; // OK first in declaration > private static final int XSTMT1o = > 0b100000; // OK first or only in statement framework > private static final int XEXPR1 = XEXPR1o | > XEXPR; // OK in expression (anywhere) > private static final int XDECL1 = XDECL1o | > XDECL; // OK in declaration (anywhere) > private static final int XSTMT1 = XSTMT1o | > XSTMT; // OK in statement framework (anywhere) > private static final int XANY1 = XEXPR1o | XDECL1o | > XSTMT1o; // Mask: first in statement, declaration, or expression > private static final int XTERM = 0b100000000; > // Can terminate (last before EOF) > private static final int XERRO = 0b1000000000; > // Is an error > > So, for example "boolean" is XEXPR | XDECL1, which is XEXPR | XDECL | > XDECL1o -- meaning it can occur in an expression (in a cast), or at > the beginning or later in a declaration. If we encounter "boolean" at > the beginning of a snippet, parseUnit() masks to just look at allowed > in first position, in this case XDECL1o. So, it knows it is the > beginning of a declaration. > >> >> Since no token is really using those constants explicitly, maybe they >> could be removed in favor of their more general counterparts (this >> also kind of means that i.e. XEXPR1 == XEXPR, etc.) > > See above, it is being used. I see now - what I was missing is that 'is at the beginning?' property is not a property of the token as a whole - but it is a context-dependent property; i.e. a boolean can either be at the beginning or not depending on whether it is parsed as a statement or as an expression - hence the need for multiple flags (where in my mind only one would have been sufficient). Thanks for taking the time to explain - the example was useful! > >> >> - I believe in general the design of location codes + 'belongs' >> field feels weird, and given this code is quite convoluted (because >> the task is hard!) the code might benefit from some cleanup. My >> suggestions: >> (1) come up with a cleaner enum which contains: { EXPR, DECL, >> STMT, AMBIGUOUS, ERR } - use this for classification purposes in >> parseUnit and related predicates (i.e. what is this token like?) >> (2) extract the 'isOkToTerminate' property onto a separate >> boolean flag on the token. After all, this is an orthogonal property >> of all tokens. > > Does my explanation clarify? Am I still missing a simplification? > > An enum won't work as they are unions of possibilities. Yes, your explanation helps - my aim was to separate between three orthogonal properties: * what is the kind of this token? * is this token a beginning? * is this token a valid termination? So that the last two could be moved onto separate boolean flags. But, as you point out, you need a richer set of flags to express things like 'boolean'. As for enums and union, EnumSet is still your friend, if needs be. > >> >> - watch out for commented out code/dead code (here and elsewhere) >> >> Sometimes the completeness analysis doesn't behave as expected: >> >> -> new >> >> ; >> >> ; >> >> ; >> >> ; >> >> ; >> >> ; >> >> Foo(); >> | Error: >> | expected >> | new >> | ^ >> | Error: >> | '(' or '[' expected >> | ; >> | ^ >> >> In the above, I found it confusing that the ';' is not enough to tell >> the analysis logic that I'm done with the statement - and the >> analysis will keep waiting until it matches a '()'. >> > > OK, "new" has special processing that has makes it error prone. In the > webrev below I remove that special processing at a small loss of error > detection, and add a general check for bad sequencing. > > http://cr.openjdk.java.net/~rfield/CompletenessAnalyzer_v0/ > > This fixes "new ;" but does not fix "(;". The lexical analysis does > not have enough context to know this isn't the beginning of a > for-statement. Basically, everything between matching parenthesis, > brackets, or braces is currently ignored -- effectively "(...)", > "[...]", or "{...}". A result of this is, for example, that "else" > is returned from completion as "unknown" (erroneous) so it is > immediately parsed and an error returned; Whereas "(else" is returned > from completion as "incomplete" prompting for more input. > > To fix this, the nesting analysis would need to be moved into the > parser (where is typically belongs) which would be a fairly > significant change. I will poke at that and see how complex it is. > Your thoughts on its importance? I don't think it's very important - the confusing part is that when you make a mistake (as I did in the case above) it is hard to realize what the REPL is expecting from you in order to mark the statement as 'complete'. Would some hint help here - i.e. "I'm waiting for (...)" Also, while you might not be able to discard "(;" immediately, we should at least try to discard "(;;;", as that's not compatible with a for anyway. Maurizio > > Thanks, > Robert > From jan.lahoda at oracle.com Tue Sep 22 15:36:53 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Tue, 22 Sep 2015 15:36:53 +0000 Subject: hg: kulla/dev/langtools: Adding ability to build jdk.jshell using the langtools/make/build.xml script. Message-ID: <201509221536.t8MFarOD015205@aojmv0008.oracle.com> Changeset: 4219c62e4a34 Author: jlahoda Date: 2015-09-22 17:36 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/4219c62e4a34 Adding ability to build jdk.jshell using the langtools/make/build.xml script. An important limitation of this support is that the target.java.home needs to be a recent JDK 9 build; and it needs to be the full JDK image, not an exploded JDK build. ! make/build.properties ! make/build.xml ! make/launcher.sh-template ! make/netbeans/langtools/nbproject/project.xml + make/tools/anttasks/DumpClassesTask.java From maurizio.cimadamore at oracle.com Tue Sep 22 17:29:12 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Tue, 22 Sep 2015 18:29:12 +0100 Subject: REPL code review -- CompletenessAnalyzer In-Reply-To: <56018B9F.4000803@oracle.com> References: <55F2F270.8020803@oracle.com> <5600EB27.6040104@oracle.com> <5601245D.5050902@oracle.com> <56018B9F.4000803@oracle.com> Message-ID: <56018FE8.3090701@oracle.com> Back on the completeness analyzer flags - I guess I still think there's some redundancy - i.e. you have: private static final int XEXPR = 0b1; // OK in expression private static final int XEXPR1o = 0b1000; // First in statement private static final int XEXPR1 = XEXPR1o | XEXPR; // or later But in reality you end up using either XEXPR or XEXPR1. So, can we at least get rid of the 'XEXPR1o' flags (and similar) on the basis that when 'XEXPR1o' is used, you really mean 'XEXPR1' ? Maurizio On 22/09/15 18:10, Robert Field wrote: > > On 09/22/15 02:50, Maurizio Cimadamore wrote: >> >> >> On 22/09/15 06:46, Robert Field wrote: >>> >>> On 09/11/15 08:25, Maurizio Cimadamore wrote: >>>> * CompletenessAnalyzer >>>> >>>> - Location position constants appear more complex than necessary >>>> (or I'm missing something); it appears that constants like XEXPR1o, >>>> XDECL1o, XSTMT1o are never used? The only uages I see are two: >>>> >>>> - to build other constants (such as XEXPR1, XDECL1 and XSTMT1) >>>> >>>> - in a switch inside parseUnit >>> >>> Exactly, they define the bits and the semantics of those bits. They >>> are applied as a set of bits. >>> >>> The comments were broken, I'm updating as such -- >>> >>> // Location position kinds -- a token is ... >>> private static final int XEXPR = >>> 0b1; // OK in expression (not first) >>> private static final int XDECL = >>> 0b10; // OK in declaration (not first) >>> private static final int XSTMT = >>> 0b100; // OK in statement framework (not first) >>> private static final int XEXPR1o = >>> 0b1000; // OK first in expression >>> private static final int XDECL1o = >>> 0b10000; // OK first in declaration >>> private static final int XSTMT1o = >>> 0b100000; // OK first or only in statement framework >>> private static final int XEXPR1 = XEXPR1o | >>> XEXPR; // OK in expression (anywhere) >>> private static final int XDECL1 = XDECL1o | >>> XDECL; // OK in declaration (anywhere) >>> private static final int XSTMT1 = XSTMT1o | >>> XSTMT; // OK in statement framework (anywhere) >>> private static final int XANY1 = XEXPR1o | XDECL1o | >>> XSTMT1o; // Mask: first in statement, declaration, or expression >>> private static final int XTERM = >>> 0b100000000; // Can terminate (last before EOF) >>> private static final int XERRO = >>> 0b1000000000; // Is an error >>> >>> So, for example "boolean" is XEXPR | XDECL1, which is XEXPR | XDECL >>> | XDECL1o -- meaning it can occur in an expression (in a cast), or >>> at the beginning or later in a declaration. If we encounter >>> "boolean" at the beginning of a snippet, parseUnit() masks to just >>> look at allowed in first position, in this case XDECL1o. So, it >>> knows it is the beginning of a declaration. >>> >>>> >>>> Since no token is really using those constants explicitly, maybe >>>> they could be removed in favor of their more general counterparts >>>> (this also kind of means that i.e. XEXPR1 == XEXPR, etc.) >>> >>> See above, it is being used. >> I see now - what I was missing is that 'is at the beginning?' >> property is not a property of the token as a whole - but it is a >> context-dependent property; i.e. a boolean can either be at the >> beginning or not depending on whether it is parsed as a statement or >> as an expression - hence the need for multiple flags (where in my >> mind only one would have been sufficient). Thanks for taking the time >> to explain - the example was useful! >>> >>>> >>>> - I believe in general the design of location codes + 'belongs' >>>> field feels weird, and given this code is quite convoluted (because >>>> the task is hard!) the code might benefit from some cleanup. My >>>> suggestions: >>>> (1) come up with a cleaner enum which contains: { EXPR, DECL, >>>> STMT, AMBIGUOUS, ERR } - use this for classification purposes in >>>> parseUnit and related predicates (i.e. what is this token like?) >>>> (2) extract the 'isOkToTerminate' property onto a separate >>>> boolean flag on the token. After all, this is an orthogonal >>>> property of all tokens. >>> >>> Does my explanation clarify? Am I still missing a simplification? >>> >>> An enum won't work as they are unions of possibilities. >> Yes, your explanation helps - my aim was to separate between three >> orthogonal properties: >> >> * what is the kind of this token? >> * is this token a beginning? >> * is this token a valid termination? >> >> So that the last two could be moved onto separate boolean flags. But, >> as you point out, you need a richer set of flags to express things >> like 'boolean'. >> >> As for enums and union, EnumSet is still your friend, if needs be. >>> >>>> >>>> - watch out for commented out code/dead code (here and elsewhere) >>>> >>>> Sometimes the completeness analysis doesn't behave as expected: >>>> >>>> -> new >>>> >> ; >>>> >> ; >>>> >> ; >>>> >> ; >>>> >> ; >>>> >> ; >>>> >> Foo(); >>>> | Error: >>>> | expected >>>> | new >>>> | ^ >>>> | Error: >>>> | '(' or '[' expected >>>> | ; >>>> | ^ >>>> >>>> In the above, I found it confusing that the ';' is not enough to >>>> tell the analysis logic that I'm done with the statement - and the >>>> analysis will keep waiting until it matches a '()'. >>>> >>> >>> OK, "new" has special processing that has makes it error prone. In >>> the webrev below I remove that special processing at a small loss of >>> error detection, and add a general check for bad sequencing. >>> >>> http://cr.openjdk.java.net/~rfield/CompletenessAnalyzer_v0/ >>> >>> This fixes "new ;" but does not fix "(;". The lexical analysis does >>> not have enough context to know this isn't the beginning of a >>> for-statement. Basically, everything between matching parenthesis, >>> brackets, or braces is currently ignored -- effectively "(...)", >>> "[...]", or "{...}". A result of this is, for example, that "else" >>> is returned from completion as "unknown" (erroneous) so it is >>> immediately parsed and an error returned; Whereas "(else" is >>> returned from completion as "incomplete" prompting for more input. >>> >>> To fix this, the nesting analysis would need to be moved into the >>> parser (where is typically belongs) which would be a fairly >>> significant change. I will poke at that and see how complex it is. >>> Your thoughts on its importance? >> I don't think it's very important - the confusing part is that when >> you make a mistake (as I did in the case above) it is hard to realize >> what the REPL is expecting from you in order to mark the statement as >> 'complete'. >> Would some hint help here - i.e. "I'm waiting for (...)" >> >> Also, while you might not be able to discard "(;" immediately, we >> should at least try to discard "(;;;", as that's not compatible with >> a for anyway. > > That too would require context that the matcher's previous-token > context does not have. Pondered parser implementation last night, > while I should have been sleeping, giving it a try now.... > > Thanks, > Robert > >> >> Maurizio >>> >>> Thanks, >>> Robert >>> >> > From maurizio.cimadamore at oracle.com Tue Sep 22 23:25:03 2015 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Wed, 23 Sep 2015 00:25:03 +0100 Subject: REPL code review -- CompletenessAnalyzer In-Reply-To: <56019412.3010105@oracle.com> References: <55F2F270.8020803@oracle.com> <5600EB27.6040104@oracle.com> <5601245D.5050902@oracle.com> <56018B9F.4000803@oracle.com> <56018FE8.3090701@oracle.com> <56019412.3010105@oracle.com> Message-ID: <5601E34F.8020701@oracle.com> On 22/09/15 18:46, Robert Field wrote: > Each token is either set as XEXPR or XEXPR1, but tests are of XEXPR1o > or XEXPR. So if you changed the flags to mean > can-be-in-expr-but-not-first and can-be-in-expr-but-including-first > then you would still need an or'ed value for the can-be-in-expr > checks. That doesn't simplify anything, and would make XANY1 more complex. Looking at code usages, the number of tests are rather limited - you only check for XEXPR1o or XEXPR in few places: inside parseUnit (where you can use XEXPR1) and inside the predicate isExpression() inside TK - in which you can use an or. In other words, with a bunch of bitwise-or you can get rid of the XYZo flags. I guess whether it's better or not is in the eye of the beholder; personally, I try to use disjoint flags only where you can really have cases where A and B can occur independently (either or/both) - this is not exactly the case here. Maurizio From robert.field at oracle.com Wed Sep 23 01:26:48 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Wed, 23 Sep 2015 01:26:48 +0000 Subject: hg: kulla/dev/langtools: 8136981: JShell API: completion analysis of new is inconsistent Message-ID: <201509230126.t8N1Qmq4010615@aojmv0008.oracle.com> Changeset: 34a76abe2c71 Author: rfield Date: 2015-09-22 18:26 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/34a76abe2c71 8136981: JShell API: completion analysis of new is inconsistent ! src/jdk.jshell/share/classes/jdk/jshell/CompletenessAnalyzer.java ! test/jdk/jshell/CompletenessStressTest.java ! test/jdk/jshell/CompletenessTest.java From jan.lahoda at oracle.com Wed Sep 23 08:55:42 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Wed, 23 Sep 2015 08:55:42 +0000 Subject: hg: kulla/dev/langtools: 2 new changesets Message-ID: <201509230855.t8N8tgam021292@aojmv0008.oracle.com> Changeset: 67c5a3d343fd Author: mcimadamore Date: 2015-09-23 10:37 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/67c5a3d343fd Support for IntelliJ for jshell. Passing proper bootclasspath to the RemoteAgent. ! make/intellij/langtools.iml ! make/intellij/workspace.xml ! src/jdk.jshell/share/classes/jdk/jshell/ExecutionControl.java Changeset: 53af9cc364c6 Author: jlahoda Date: 2015-09-23 10:37 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/53af9cc364c6 Improving NB support for jshell - ability to run inside NB. ! make/netbeans/langtools/build.xml ! make/tools/anttasks/SelectToolTask.java From robert.field at oracle.com Thu Sep 24 06:10:51 2015 From: robert.field at oracle.com (Robert Field) Date: Wed, 23 Sep 2015 23:10:51 -0700 Subject: Dan: JDK 9 RFR of JDK-8134254 JShell API/tool: REPL for Java into JDK9 In-Reply-To: <56033931.5040509@oracle.com> References: <55D8114E.6070506@oracle.com> <55F1D4D1.5090600@oracle.com> <560335AC.2050203@oracle.com> <56033931.5040509@oracle.com> Message-ID: <3D5B9DF0-1BC1-43C1-A5E8-CC67A71C1638@oracle.com> Thanks Dan! Forwarding to Kulla. -Robert > On Sep 23, 2015, at 4:43 PM, Michel Trudeau wrote: > > Thank you Dan for the code review. Very appreciated. > > Thanks, > Michel > > > > > Daniel D. Daugherty wrote: >> > http://cr.openjdk.java.net/~rfield/jshell_langtools_webrev_v0/ >> >> src/jdk.jshell/share/classes/jdk/internal/jshell/remote/RemoteAgent.java >> L212: //thrown by the main process via JDI: >> Space after '//'. >> >> L226: void clientCodeLeave() { >> >> L228: while (expectingStop); >> Tight loop here is not kind. A very short sleep would be nice >> if you expect 'expectingStop' to transition from 'true' -> >> 'false' at some point. If you don't expect this thread to ever >> return from this loop, then 'Thread.sleep()' >> is good. >> >> Update: looks like expectingStop is set to 'false' in a few >> places so short sleep would be nice. If you call Thread.sleep(0) >> then that typically translates into the shortest sleep on the >> platform. >> >> src/jdk.jshell/share/classes/jdk/jshell/Eval.java >> L564: // First try Redefine >> L565: Map mp = new TreeMap<>(); >> L566: List newNameBytesList = new ArrayList<>(); >> L567: for (ClassNameBytes nb : nameBytesList) { >> L568: ReferenceType rt = state.executionControl().nameToRef(nb.className); >> L569: if (rt != null) { >> L570: mp.put(rt, nb.bytes); >> L571: } else { >> L572: newNameBytesList.add(nb); >> L573: } >> L574: } >> >> I'm not quite understanding this use of com.sun.jdi.ReferenceType, >> but I suspect that the mentions of 'Redefine' here are not >> related to JVM/TI RedefineClasses(). >> >> src/jdk.jshell/share/classes/jdk/jshell/ExecutionControl.java >> L68: // out before in >> L69: out = new ObjectOutputStream(socket.getOutputStream()); >> L70: in = new ObjectInputStream(socket.getInputStream()); >> >> Yes, the comment on L68 is right, but would better if it >> stated _why_ we create 'out' before 'in'. >> >> L154: boolean commandRedefine(Map mp) { >> L155: try { >> L156: env.vm().redefineClasses(mp); >> >> Maybe I was wrong in my Eval.java comment about it not being >> related to JVM/TI RedefineClasses(). :-) >> >> L268: //could also tag the thread (e.g. using name), to find it easier >> Space after '//'. >> >> src/jdk.jshell/share/classes/jdk/jshell/JDIConnection.java >> L47: * Adapted from jdb JDIConnection. >> What are the significant differences? Did the original code get >> deleted as part of 'jdb' removal? >> >> Perhaps a short summary of the differences... >> >> L321: synchronized VirtualMachine open() { >> >> L332: if (vm.canBeModified()){ >> L333: // setEventRequests(vm); >> L334: // resolveEventRequests(); >> L335: } >> >> Why commented out instead of deleted? >> >> L397: /*** >> L398: private void setEventRequests(VirtualMachine vm) { >> >> L419: private void resolveEventRequests() { >> L420: Env.specList.resolveAll(); >> L421: } >> L422: ***/ >> >> Why commented out instead of deleted? >> >> L429: pStream.print((char)i); >> L434: throw ex; >> The indents are wrong for these two lines. >> >> L506: /* We should never do this: attach to running target vm */ >> L507: private VirtualMachine attachTarget() { >> So delete the function or have it throw an exception >> if used or... >> >> L517: /* We should never do this: alisten for connection from target vm */ >> L518: private VirtualMachine listenTarget() { >> So delete the function or have it throw an exception >> if used or... >> >> src/jdk.jshell/share/classes/jdk/jshell/JDIEnv.java >> L33: * Extracted from jdb Env. >> Not quite 'Extracted'. Perhaps 'Adapted'. What are the differences? >> Did the original code get deleted as part of 'jdb' removal? >> >> Perhaps a short summary of the differences... >> >> src/jdk.jshell/share/classes/jdk/jshell/JDIEventHandler.java >> L33: * Adapted from jdb EventHandler. >> What are the significant differences? Did the original code get >> deleted as part of 'jdb' removal? >> >> Perhaps a short summary of the differences... >> >> src/jdk.jshell/share/classes/jdk/jshell/JDINotConnectedException.java >> L30: * Adapted from JDI JDINotConnectedException. >> >> Did the original code get deleted as part of 'jdb' removal? >> >> Perhaps a short summary of the differences... >> >> src/jdk.jshell/share/classes/jdk/jshell/JShell.java >> L595 // --- priave / package-private implementation support --- >> Typo? 'priave' Perhaps 'private' >> >> >> So if I understand the code correctly, JDI only comes into play in >> the "Eval" portion of the "Read-Eval-Print Loop". It's used to evaluate >> the code, to stop a currently executing evaluation either in a normal >> way or in a regain control way... Looks like just the exception handling >> piece is there and there's no use of JDI debugging capability to try >> and debug the code being evaluated. >> >> There is some mention of RedefineClasses(), but I suspect that is >> simply to redefine the code being evaluated so there's something >> to run... >> >> Please let me know if I didn't review something that you needed >> me to review. Also let me know if I'm even close to understanding >> what you're using JDI for in this area... :-) >> >> Dan >> >> >> >> On 9/10/15 1:06 PM, Robert Field wrote: >>> Hi Dan, >>> >>> Long time again. >>> >>> I've been working on adding a Read-Eval-Print Loop Java tool. We are on the verge of pushing it into JDK9, it has had an API review but needs a code review. It is a lot of new code, so we are breaking the task into areas. We have reviewers for the tool (built on the API), and for the compilation related areas. But we need someone for the JDI / remote interaction areas. Not much experience on this team in JDI. I'm hoping you would be will to do the review of that area. Let me know and I'll point you at the appropriate bits. >>> >>> Thanks, >>> Robert >>> >>> >>> >>> -------- Original Message -------- >>> Subject: JDK 9 RFR of JDK-8134254 JShell API/tool: REPL for Java into JDK9 >>> Date: Fri, 21 Aug 2015 23:06:06 -0700 >>> From: Robert Field >>> To: kulla-dev >>> >>> Please review: >>> >>> 8134254 JShell API/tool: REPL for Java into JDK9 >>> https://bugs.openjdk.java.net/browse/JDK-8134254 >>> >>> http://cr.openjdk.java.net/~rfield/jshell_langtools_webrev_v0/ >>> >>> This webrev is of all of the JShell changes in langtools, which is the addition of the source code and all the tests. >>> For context, please familiarize yourself with the API first: >>> >>> http://cr.openjdk.java.net/~rfield/doc/ >>> >>> Since the webrev is all adds, and the order and access is far from ideal, it is almost certainly easier to clone the kulla repo: >>> >>> http://hg.openjdk.java.net/kulla/dev >>> >>> This webrev is the contents of these langtools directories >>> >>> src/jdk.jshell >>> test/jdk/jshell >>> >>> added to jdk9/dev/langtools. >>> >>> JShell.java and JShellImpl.java in src/jdk.jshell/share/classes/jdk/jshell/ would make a logical starting point for understanding the code. >>> >>> There are also small changes to the base jdk9 repo for make files and to add the module, and to the jdk repo to add the launcher, I will send these separately. >>> >>> Thanks, >>> Robert >>> >>> >>> >> > From jan.lahoda at oracle.com Thu Sep 24 09:08:50 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Thu, 24 Sep 2015 09:08:50 +0000 Subject: hg: kulla/dev/langtools: 8136422: the version of jshell should now follow the same conventions as javac version does (based on Robert's patch). Message-ID: <201509240908.t8O98oM6011591@aojmv0008.oracle.com> Changeset: 45209b819d27 Author: jlahoda Date: 2015-09-24 10:27 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/45209b819d27 8136422: the version of jshell should now follow the same conventions as javac version does (based on Robert's patch). + make/gensrc/Gensrc-jdk.jshell.gmk ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/JShellTool.java + src/jdk.jshell/share/classes/jdk/internal/jshell/tool/resources/version.properties-template From robert.field at oracle.com Thu Sep 24 17:53:23 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Thu, 24 Sep 2015 17:53:23 +0000 Subject: hg: kulla/dev/langtools: Per review, fix comments. Message-ID: <201509241753.t8OHrO5d023191@aojmv0008.oracle.com> Changeset: 6e4ef25b4419 Author: rfield Date: 2015-09-24 10:53 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/6e4ef25b4419 Per review, fix comments. ! src/jdk.jshell/share/classes/jdk/internal/jshell/remote/RemoteAgent.java ! src/jdk.jshell/share/classes/jdk/jshell/Eval.java ! src/jdk.jshell/share/classes/jdk/jshell/ExecutionControl.java ! src/jdk.jshell/share/classes/jdk/jshell/JDIConnection.java ! src/jdk.jshell/share/classes/jdk/jshell/JDIEnv.java ! src/jdk.jshell/share/classes/jdk/jshell/JDIEventHandler.java ! src/jdk.jshell/share/classes/jdk/jshell/JDINotConnectedException.java ! src/jdk.jshell/share/classes/jdk/jshell/JShell.java From jan.lahoda at oracle.com Fri Sep 25 09:46:55 2015 From: jan.lahoda at oracle.com (jan.lahoda at oracle.com) Date: Fri, 25 Sep 2015 09:46:55 +0000 Subject: hg: kulla/dev/langtools: Per review, adding a short sleep into the loop that waits for the stop exception. Message-ID: <201509250946.t8P9ktDq023683@aojmv0008.oracle.com> Changeset: eb209acb2451 Author: jlahoda Date: 2015-09-25 11:41 +0200 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/eb209acb2451 Per review, adding a short sleep into the loop that waits for the stop exception. ! src/jdk.jshell/share/classes/jdk/internal/jshell/remote/RemoteAgent.java From jan.lahoda at oracle.com Fri Sep 25 09:49:19 2015 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Fri, 25 Sep 2015 11:49:19 +0200 Subject: Dan: JDK 9 RFR of JDK-8134254 JShell API/tool: REPL for Java into JDK9 In-Reply-To: <3D5B9DF0-1BC1-43C1-A5E8-CC67A71C1638@oracle.com> References: <55D8114E.6070506@oracle.com> <55F1D4D1.5090600@oracle.com> <560335AC.2050203@oracle.com> <56033931.5040509@oracle.com> <3D5B9DF0-1BC1-43C1-A5E8-CC67A71C1638@oracle.com> Message-ID: <5605189F.1040404@oracle.com> Hi Dan, Thanks for the comments! I've resolved the RemoteAgent comments: http://hg.openjdk.java.net/kulla/dev/langtools/rev/eb209acb2451 Please let me know if there are any issues with that. Thanks! Jan On 24.9.2015 08:10, Robert Field wrote: > Thanks Dan! > > Forwarding to Kulla. > > -Robert > >> On Sep 23, 2015, at 4:43 PM, Michel Trudeau wrote: >> >> Thank you Dan for the code review. Very appreciated. >> >> Thanks, >> Michel >> >> >> >> >> Daniel D. Daugherty wrote: >>>> http://cr.openjdk.java.net/~rfield/jshell_langtools_webrev_v0/ >>> >>> src/jdk.jshell/share/classes/jdk/internal/jshell/remote/RemoteAgent.java >>> L212: //thrown by the main process via JDI: >>> Space after '//'. >>> >>> L226: void clientCodeLeave() { >>> >>> L228: while (expectingStop); >>> Tight loop here is not kind. A very short sleep would be nice >>> if you expect 'expectingStop' to transition from 'true' -> >>> 'false' at some point. If you don't expect this thread to ever >>> return from this loop, then 'Thread.sleep()' >>> is good. >>> >>> Update: looks like expectingStop is set to 'false' in a few >>> places so short sleep would be nice. If you call Thread.sleep(0) >>> then that typically translates into the shortest sleep on the >>> platform. >>> >>> src/jdk.jshell/share/classes/jdk/jshell/Eval.java >>> L564: // First try Redefine >>> L565: Map mp = new TreeMap<>(); >>> L566: List newNameBytesList = new ArrayList<>(); >>> L567: for (ClassNameBytes nb : nameBytesList) { >>> L568: ReferenceType rt = state.executionControl().nameToRef(nb.className); >>> L569: if (rt != null) { >>> L570: mp.put(rt, nb.bytes); >>> L571: } else { >>> L572: newNameBytesList.add(nb); >>> L573: } >>> L574: } >>> >>> I'm not quite understanding this use of com.sun.jdi.ReferenceType, >>> but I suspect that the mentions of 'Redefine' here are not >>> related to JVM/TI RedefineClasses(). >>> >>> src/jdk.jshell/share/classes/jdk/jshell/ExecutionControl.java >>> L68: // out before in >>> L69: out = new ObjectOutputStream(socket.getOutputStream()); >>> L70: in = new ObjectInputStream(socket.getInputStream()); >>> >>> Yes, the comment on L68 is right, but would better if it >>> stated _why_ we create 'out' before 'in'. >>> >>> L154: boolean commandRedefine(Map mp) { >>> L155: try { >>> L156: env.vm().redefineClasses(mp); >>> >>> Maybe I was wrong in my Eval.java comment about it not being >>> related to JVM/TI RedefineClasses(). :-) >>> >>> L268: //could also tag the thread (e.g. using name), to find it easier >>> Space after '//'. >>> >>> src/jdk.jshell/share/classes/jdk/jshell/JDIConnection.java >>> L47: * Adapted from jdb JDIConnection. >>> What are the significant differences? Did the original code get >>> deleted as part of 'jdb' removal? >>> >>> Perhaps a short summary of the differences... >>> >>> L321: synchronized VirtualMachine open() { >>> >>> L332: if (vm.canBeModified()){ >>> L333: // setEventRequests(vm); >>> L334: // resolveEventRequests(); >>> L335: } >>> >>> Why commented out instead of deleted? >>> >>> L397: /*** >>> L398: private void setEventRequests(VirtualMachine vm) { >>> >>> L419: private void resolveEventRequests() { >>> L420: Env.specList.resolveAll(); >>> L421: } >>> L422: ***/ >>> >>> Why commented out instead of deleted? >>> >>> L429: pStream.print((char)i); >>> L434: throw ex; >>> The indents are wrong for these two lines. >>> >>> L506: /* We should never do this: attach to running target vm */ >>> L507: private VirtualMachine attachTarget() { >>> So delete the function or have it throw an exception >>> if used or... >>> >>> L517: /* We should never do this: alisten for connection from target vm */ >>> L518: private VirtualMachine listenTarget() { >>> So delete the function or have it throw an exception >>> if used or... >>> >>> src/jdk.jshell/share/classes/jdk/jshell/JDIEnv.java >>> L33: * Extracted from jdb Env. >>> Not quite 'Extracted'. Perhaps 'Adapted'. What are the differences? >>> Did the original code get deleted as part of 'jdb' removal? >>> >>> Perhaps a short summary of the differences... >>> >>> src/jdk.jshell/share/classes/jdk/jshell/JDIEventHandler.java >>> L33: * Adapted from jdb EventHandler. >>> What are the significant differences? Did the original code get >>> deleted as part of 'jdb' removal? >>> >>> Perhaps a short summary of the differences... >>> >>> src/jdk.jshell/share/classes/jdk/jshell/JDINotConnectedException.java >>> L30: * Adapted from JDI JDINotConnectedException. >>> >>> Did the original code get deleted as part of 'jdb' removal? >>> >>> Perhaps a short summary of the differences... >>> >>> src/jdk.jshell/share/classes/jdk/jshell/JShell.java >>> L595 // --- priave / package-private implementation support --- >>> Typo? 'priave' Perhaps 'private' >>> >>> >>> So if I understand the code correctly, JDI only comes into play in >>> the "Eval" portion of the "Read-Eval-Print Loop". It's used to evaluate >>> the code, to stop a currently executing evaluation either in a normal >>> way or in a regain control way... Looks like just the exception handling >>> piece is there and there's no use of JDI debugging capability to try >>> and debug the code being evaluated. >>> >>> There is some mention of RedefineClasses(), but I suspect that is >>> simply to redefine the code being evaluated so there's something >>> to run... >>> >>> Please let me know if I didn't review something that you needed >>> me to review. Also let me know if I'm even close to understanding >>> what you're using JDI for in this area... :-) >>> >>> Dan >>> >>> >>> >>> On 9/10/15 1:06 PM, Robert Field wrote: >>>> Hi Dan, >>>> >>>> Long time again. >>>> >>>> I've been working on adding a Read-Eval-Print Loop Java tool. We are on the verge of pushing it into JDK9, it has had an API review but needs a code review. It is a lot of new code, so we are breaking the task into areas. We have reviewers for the tool (built on the API), and for the compilation related areas. But we need someone for the JDI / remote interaction areas. Not much experience on this team in JDI. I'm hoping you would be will to do the review of that area. Let me know and I'll point you at the appropriate bits. >>>> >>>> Thanks, >>>> Robert >>>> >>>> >>>> >>>> -------- Original Message -------- >>>> Subject: JDK 9 RFR of JDK-8134254 JShell API/tool: REPL for Java into JDK9 >>>> Date: Fri, 21 Aug 2015 23:06:06 -0700 >>>> From: Robert Field >>>> To: kulla-dev >>>> >>>> Please review: >>>> >>>> 8134254 JShell API/tool: REPL for Java into JDK9 >>>> https://bugs.openjdk.java.net/browse/JDK-8134254 >>>> >>>> http://cr.openjdk.java.net/~rfield/jshell_langtools_webrev_v0/ >>>> >>>> This webrev is of all of the JShell changes in langtools, which is the addition of the source code and all the tests. >>>> For context, please familiarize yourself with the API first: >>>> >>>> http://cr.openjdk.java.net/~rfield/doc/ >>>> >>>> Since the webrev is all adds, and the order and access is far from ideal, it is almost certainly easier to clone the kulla repo: >>>> >>>> http://hg.openjdk.java.net/kulla/dev >>>> >>>> This webrev is the contents of these langtools directories >>>> >>>> src/jdk.jshell >>>> test/jdk/jshell >>>> >>>> added to jdk9/dev/langtools. >>>> >>>> JShell.java and JShellImpl.java in src/jdk.jshell/share/classes/jdk/jshell/ would make a logical starting point for understanding the code. >>>> >>>> There are also small changes to the base jdk9 repo for make files and to add the module, and to the jdk repo to add the launcher, I will send these separately. >>>> >>>> Thanks, >>>> Robert >>>> >>>> >>>> >>> >> > From Alan.Bateman at oracle.com Fri Sep 18 13:03:09 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 18 Sep 2015 14:03:09 +0100 Subject: JShell: packages In-Reply-To: <55FC0852.5060804@oracle.com> References: <55FB0182.2030800@oracle.com> <55FB0C07.5040403@oracle.com> <55FB58D7.5080702@oracle.com> <55FB59F6.5010308@oracle.com> <14fddf2f3b8.2784.4011f3a8741ca2aabce58b8b81f42d24@oracle.com> <55FB640A.2060409@oracle.com> <55FBE5CB.9060702@oracle.com> <55FC081F.7000405@oracle.com> <55FC0852.5060804@oracle.com> Message-ID: <55FC0B8D.6050904@oracle.com> On 18/09/2015 13:49, Maurizio Cimadamore wrote: > Adding Alan > > On 18/09/15 13:48, Maurizio Cimadamore wrote: >> I had a productive chat with Alan - there's no bug here. The problem >> is caused by the fact that when you use bootclasspath, you force >> jshell to be loaded by the bootloader and, unfortunately, both jline >> and jdi are NOT in the bootloader (they are on the app loader); since >> bootclassloader doesn't delegate, you then get errors. As I >> understand, there's an hack to make this sort of things work with the >> exploded build, but it's just an accident that things work there. Just to add to the point about why this works with exploded builds ... With exploded builds then all types in all modules are defined to the boot loader. This is temporary but consistent with how exploded have always worked. It's completely replaced in the jake forest so that modules are defined to their appropriate loader irrespective of whether it's an exploded/developer or images build. Also the -Xoverride option in jake is independent of class loader and so can be used to override classes in any module (not just modules defined to the boot loader). -Alan From daniel.daugherty at oracle.com Fri Sep 25 13:35:34 2015 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 25 Sep 2015 07:35:34 -0600 Subject: Dan: JDK 9 RFR of JDK-8134254 JShell API/tool: REPL for Java into JDK9 In-Reply-To: <5605189F.1040404@oracle.com> References: <55D8114E.6070506@oracle.com> <55F1D4D1.5090600@oracle.com> <560335AC.2050203@oracle.com> <56033931.5040509@oracle.com> <3D5B9DF0-1BC1-43C1-A5E8-CC67A71C1638@oracle.com> <5605189F.1040404@oracle.com> Message-ID: <56054DA6.9010605@oracle.com> Thanks for the update. Looks good. Dan On 9/25/15 3:49 AM, Jan Lahoda wrote: > Hi Dan, > > Thanks for the comments! I've resolved the RemoteAgent comments: > http://hg.openjdk.java.net/kulla/dev/langtools/rev/eb209acb2451 > > Please let me know if there are any issues with that. > > Thanks! > > Jan > > On 24.9.2015 08:10, Robert Field wrote: >> Thanks Dan! >> >> Forwarding to Kulla. >> >> -Robert >> >>> On Sep 23, 2015, at 4:43 PM, Michel Trudeau >>> wrote: >>> >>> Thank you Dan for the code review. Very appreciated. >>> >>> Thanks, >>> Michel >>> >>> >>> >>> >>> Daniel D. Daugherty wrote: >>>>> http://cr.openjdk.java.net/~rfield/jshell_langtools_webrev_v0/ >>>>> >>>> >>>> src/jdk.jshell/share/classes/jdk/internal/jshell/remote/RemoteAgent.java >>>> >>>> L212: //thrown by the main process via >>>> JDI: >>>> Space after '//'. >>>> >>>> L226: void clientCodeLeave() { >>>> >>>> L228: while (expectingStop); >>>> Tight loop here is not kind. A very short sleep would be nice >>>> if you expect 'expectingStop' to transition from 'true' -> >>>> 'false' at some point. If you don't expect this thread to >>>> ever >>>> return from this loop, then >>>> 'Thread.sleep()' >>>> is good. >>>> >>>> Update: looks like expectingStop is set to 'false' in a few >>>> places so short sleep would be nice. If you call >>>> Thread.sleep(0) >>>> then that typically translates into the shortest sleep on the >>>> platform. >>>> >>>> src/jdk.jshell/share/classes/jdk/jshell/Eval.java >>>> L564: // First try Redefine >>>> L565: Map mp = >>>> new TreeMap<>(); >>>> L566: List newNameBytesList = new ArrayList<>(); >>>> L567: for (ClassNameBytes nb : >>>> nameBytesList) { >>>> L568: ReferenceType rt = >>>> state.executionControl().nameToRef(nb.className); >>>> L569: if (rt != null) { >>>> L570: mp.put(rt, nb.bytes); >>>> L571: } else { >>>> L572: newNameBytesList.add(nb); >>>> L573: } >>>> L574: } >>>> >>>> I'm not quite understanding this use of >>>> com.sun.jdi.ReferenceType, >>>> but I suspect that the mentions of 'Redefine' here are not >>>> related to JVM/TI RedefineClasses(). >>>> >>>> src/jdk.jshell/share/classes/jdk/jshell/ExecutionControl.java >>>> L68: // out before in >>>> L69: out = new >>>> ObjectOutputStream(socket.getOutputStream()); >>>> L70: in = new >>>> ObjectInputStream(socket.getInputStream()); >>>> >>>> Yes, the comment on L68 is right, but would better if it >>>> stated _why_ we create 'out' before 'in'. >>>> >>>> L154: boolean commandRedefine(Map >>>> mp) { >>>> L155: try { >>>> L156: env.vm().redefineClasses(mp); >>>> >>>> Maybe I was wrong in my Eval.java comment about it not being >>>> related to JVM/TI RedefineClasses(). :-) >>>> >>>> L268: //could also tag the thread (e.g. using >>>> name), to find it easier >>>> Space after '//'. >>>> >>>> src/jdk.jshell/share/classes/jdk/jshell/JDIConnection.java >>>> L47: * Adapted from jdb JDIConnection. >>>> What are the significant differences? Did the original >>>> code get >>>> deleted as part of 'jdb' removal? >>>> >>>> Perhaps a short summary of the differences... >>>> >>>> L321: synchronized VirtualMachine open() { >>>> >>>> L332: if (vm.canBeModified()){ >>>> L333: // setEventRequests(vm); >>>> L334: // resolveEventRequests(); >>>> L335: } >>>> >>>> Why commented out instead of deleted? >>>> >>>> L397: /*** >>>> L398: private void setEventRequests(VirtualMachine vm) { >>>> >>>> L419: private void resolveEventRequests() { >>>> L420: Env.specList.resolveAll(); >>>> L421: } >>>> L422: ***/ >>>> >>>> Why commented out instead of deleted? >>>> >>>> L429: pStream.print((char)i); >>>> L434: throw ex; >>>> The indents are wrong for these two lines. >>>> >>>> L506: /* We should never do this: attach to running target >>>> vm */ >>>> L507: private VirtualMachine attachTarget() { >>>> So delete the function or have it throw an exception >>>> if used or... >>>> >>>> L517: /* We should never do this: alisten for connection >>>> from target vm */ >>>> L518: private VirtualMachine listenTarget() { >>>> So delete the function or have it throw an exception >>>> if used or... >>>> >>>> src/jdk.jshell/share/classes/jdk/jshell/JDIEnv.java >>>> L33: * Extracted from jdb Env. >>>> Not quite 'Extracted'. Perhaps 'Adapted'. What are the >>>> differences? >>>> Did the original code get deleted as part of 'jdb' removal? >>>> >>>> Perhaps a short summary of the differences... >>>> >>>> src/jdk.jshell/share/classes/jdk/jshell/JDIEventHandler.java >>>> L33: * Adapted from jdb EventHandler. >>>> What are the significant differences? Did the original >>>> code get >>>> deleted as part of 'jdb' removal? >>>> >>>> Perhaps a short summary of the differences... >>>> >>>> src/jdk.jshell/share/classes/jdk/jshell/JDINotConnectedException.java >>>> L30: * Adapted from JDI JDINotConnectedException. >>>> >>>> Did the original code get deleted as part of 'jdb' removal? >>>> >>>> Perhaps a short summary of the differences... >>>> >>>> src/jdk.jshell/share/classes/jdk/jshell/JShell.java >>>> L595 // --- priave / package-private implementation >>>> support --- >>>> Typo? 'priave' Perhaps 'private' >>>> >>>> >>>> So if I understand the code correctly, JDI only comes into play in >>>> the "Eval" portion of the "Read-Eval-Print Loop". It's used to >>>> evaluate >>>> the code, to stop a currently executing evaluation either in a normal >>>> way or in a regain control way... Looks like just the exception >>>> handling >>>> piece is there and there's no use of JDI debugging capability to try >>>> and debug the code being evaluated. >>>> >>>> There is some mention of RedefineClasses(), but I suspect that is >>>> simply to redefine the code being evaluated so there's something >>>> to run... >>>> >>>> Please let me know if I didn't review something that you needed >>>> me to review. Also let me know if I'm even close to understanding >>>> what you're using JDI for in this area... :-) >>>> >>>> Dan >>>> >>>> >>>> >>>> On 9/10/15 1:06 PM, Robert Field wrote: >>>>> Hi Dan, >>>>> >>>>> Long time again. >>>>> >>>>> I've been working on adding a Read-Eval-Print Loop Java tool. We >>>>> are on the verge of pushing it into JDK9, it has had an API review >>>>> but needs a code review. It is a lot of new code, so we are >>>>> breaking the task into areas. We have reviewers for the tool >>>>> (built on the API), and for the compilation related areas. But we >>>>> need someone for the JDI / remote interaction areas. Not much >>>>> experience on this team in JDI. I'm hoping you would be will to >>>>> do the review of that area. Let me know and I'll point you at the >>>>> appropriate bits. >>>>> >>>>> Thanks, >>>>> Robert >>>>> >>>>> >>>>> >>>>> -------- Original Message -------- >>>>> Subject: JDK 9 RFR of JDK-8134254 JShell API/tool: REPL for >>>>> Java into JDK9 >>>>> Date: Fri, 21 Aug 2015 23:06:06 -0700 >>>>> From: Robert Field >>>>> >>>>> To: kulla-dev >>>>> >>>>> >>>>> Please review: >>>>> >>>>> 8134254 JShell API/tool: REPL for Java into JDK9 >>>>> https://bugs.openjdk.java.net/browse/JDK-8134254 >>>>> >>>>> >>>>> http://cr.openjdk.java.net/~rfield/jshell_langtools_webrev_v0/ >>>>> >>>>> >>>>> This webrev is of all of the JShell changes in langtools, which is >>>>> the addition of the source code and all the tests. >>>>> For context, please familiarize yourself with the API first: >>>>> >>>>> http://cr.openjdk.java.net/~rfield/doc/ >>>>> >>>>> >>>>> Since the webrev is all adds, and the order and access is far from >>>>> ideal, it is almost certainly easier to clone the kulla repo: >>>>> >>>>> http://hg.openjdk.java.net/kulla/dev >>>>> >>>>> >>>>> This webrev is the contents of these langtools directories >>>>> >>>>> src/jdk.jshell >>>>> test/jdk/jshell >>>>> >>>>> added to jdk9/dev/langtools. >>>>> >>>>> JShell.java and JShellImpl.java in >>>>> src/jdk.jshell/share/classes/jdk/jshell/ would make a logical >>>>> starting point for understanding the code. >>>>> >>>>> There are also small changes to the base jdk9 repo for make files >>>>> and to add the module, and to the jdk repo to add the launcher, I >>>>> will send these separately. >>>>> >>>>> Thanks, >>>>> Robert >>>>> >>>>> >>>>> >>>> >>> >> From daniel.daugherty at oracle.com Fri Sep 25 13:38:07 2015 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 25 Sep 2015 07:38:07 -0600 Subject: Dan: JDK 9 RFR of JDK-8134254 JShell API/tool: REPL for Java into JDK9 In-Reply-To: <56043660.2010801@oracle.com> References: <55D8114E.6070506@oracle.com> <55F1D4D1.5090600@oracle.com> <560335AC.2050203@oracle.com> <56043660.2010801@oracle.com> Message-ID: <56054E3F.90103@oracle.com> > > Please let me know if I didn't review something that you needed > > me to review. Also let me know if I'm even close to understanding > > what you're using JDI for in this area... :-) > > What I'd be most interested from you would be any interprocess and synchronization stuff. > Clashes between JDI threads and remote, risk of hangs, races, ... This will take more mulling. I'll likely have more questions about how JShell works (1 VM or 2 VMs or ...) Dan On 9/24/15 11:44 AM, Robert Field wrote: > > On 09/23/15 16:28, Daniel D. Daugherty wrote: >> > http://cr.openjdk.java.net/~rfield/jshell_langtools_webrev_v0/ >> >> src/jdk.jshell/share/classes/jdk/internal/jshell/remote/RemoteAgent.java >> L212: //thrown by the main process via JDI: >> Space after '//'. >> >> L226: void clientCodeLeave() { >> >> L228: while (expectingStop); >> Tight loop here is not kind. A very short sleep would be nice >> if you expect 'expectingStop' to transition from 'true' -> >> 'false' at some point. If you don't expect this thread to ever >> return from this loop, then 'Thread.sleep()' >> is good. >> >> Update: looks like expectingStop is set to 'false' in a few >> places so short sleep would be nice. If you call Thread.sleep(0) >> then that typically translates into the shortest sleep on the >> platform. > > Jan will answer this. > >> >> src/jdk.jshell/share/classes/jdk/jshell/Eval.java >> L564: // First try Redefine >> L565: Map mp = new >> TreeMap<>(); >> L566: List >> newNameBytesList = new ArrayList<>(); >> L567: for (ClassNameBytes nb : >> nameBytesList) { >> L568: ReferenceType rt = >> state.executionControl().nameToRef(nb.className); >> L569: if (rt != null) { >> L570: mp.put(rt, nb.bytes); >> L571: } else { >> L572: newNameBytesList.add(nb); >> L573: } >> L574: } >> >> I'm not quite understanding this use of >> com.sun.jdi.ReferenceType, >> but I suspect that the mentions of 'Redefine' here are not >> related to JVM/TI RedefineClasses(). > > That is indeed JDI ReferenceType being used to a JDI RedefineClasses > >> >> src/jdk.jshell/share/classes/jdk/jshell/ExecutionControl.java >> L68: // out before in >> L69: out = new >> ObjectOutputStream(socket.getOutputStream()); >> L70: in = new ObjectInputStream(socket.getInputStream()); >> >> Yes, the comment on L68 is right, but would better if it >> stated _why_ we create 'out' before 'in'. > > Comment added on here and match remote side. > >> >> L154: boolean commandRedefine(Map mp) { >> L155: try { >> L156: env.vm().redefineClasses(mp); >> >> Maybe I was wrong in my Eval.java comment about it not being >> related to JVM/TI RedefineClasses(). :-) > > ;-) > > Yep, that's why you are here > > >> >> L268: //could also tag the thread (e.g. using >> name), to find it easier >> Space after '//'. > > Added > >> >> src/jdk.jshell/share/classes/jdk/jshell/JDIConnection.java >> L47: * Adapted from jdb JDIConnection. >> What are the significant differences? Did the original code get >> deleted as part of 'jdb' removal? >> >> Perhaps a short summary of the differences... > > Added: > > * Adapted from jdb VMConnection. Message handling, exception > handling, and I/O > * redirection changed. Interface to JShell added. > >> >> L321: synchronized VirtualMachine open() { >> >> L332: if (vm.canBeModified()){ >> L333: // setEventRequests(vm); >> L334: // resolveEventRequests(); >> L335: } >> >> Why commented out instead of deleted? > > Moved the test and calls into a new method, and made this: > > // Uncomment here and below to enable event requests > // installEventRequests(vm); > > There are features that have been discussed that would require adding > event requests > >> >> L397: /*** >> L398: private void setEventRequests(VirtualMachine vm) { >> >> L419: private void resolveEventRequests() { >> L420: Env.specList.resolveAll(); >> L421: } >> L422: ***/ >> >> Why commented out instead of deleted? > > /*** Preserved for possible future support of event requests > >> >> L429: pStream.print((char)i); >> L434: throw ex; >> The indents are wrong for these two lines. > > Fixed > >> >> L506: /* We should never do this: attach to running target vm */ >> L507: private VirtualMachine attachTarget() { >> So delete the function or have it throw an exception >> if used or... >> >> L517: /* We should never do this: alisten for connection from >> target vm */ >> L518: private VirtualMachine listenTarget() { >> So delete the function or have it throw an exception >> if used or... > > Yeah, that is in the stupid comment department ;-) > > and it is no longer definitely true, there has been considerable > interest in > features that would require this. > > Changed the comments to: > > /* JShell currently uses only launch, preserved for futures: */ > >> >> src/jdk.jshell/share/classes/jdk/jshell/JDIEnv.java >> L33: * Extracted from jdb Env. >> Not quite 'Extracted'. Perhaps 'Adapted'. What are the >> differences? >> Did the original code get deleted as part of 'jdb' removal? >> >> Perhaps a short summary of the differences... > > Well, pretty much is extracted, Env is this big complex thing. I just > pulled out > four little methods. Changed the comment to: > > * Select methods extracted from jdb Env; shutdown() adapted to JShell > shutdown. > > Just checked, the jdb code did not get removed, it is still sitting > happily in: > > dev/jdk/src/jdk.jdi/share/classes/com/sun/tools/example/debug/tty > >> >> src/jdk.jshell/share/classes/jdk/jshell/JDIEventHandler.java >> L33: * Adapted from jdb EventHandler. >> What are the significant differences? Did the original code get >> deleted as part of 'jdb' removal? >> >> Perhaps a short summary of the differences... > > * Adapted from jdb EventHandler; Handling of events not used by > JShell stubbed out. > >> >> src/jdk.jshell/share/classes/jdk/jshell/JDINotConnectedException.java >> L30: * Adapted from JDI JDINotConnectedException. >> >> Did the original code get deleted as part of 'jdb' removal? >> >> Perhaps a short summary of the differences... > > * Copy of jdb VMNotConnectedException. > >> >> src/jdk.jshell/share/classes/jdk/jshell/JShell.java >> L595 // --- priave / package-private implementation support --- >> Typo? 'priave' Perhaps 'private' > > Fixed. > >> >> >> So if I understand the code correctly, JDI only comes into play in >> the "Eval" portion of the "Read-Eval-Print Loop". It's used to evaluate >> the code, to stop a currently executing evaluation either in a normal >> way or in a regain control way... > > Yes. Also, and most significantly to redefine classes holding the > user's snippets of code as the user evolves them. > >> Looks like just the exception handling >> piece is there and there's no use of JDI debugging capability to try >> and debug the code being evaluated. > > Yes, we don't want to tread on the IDE/debugger space. > > I would like to add tracing at some point. > >> >> There is some mention of RedefineClasses(), but I suspect that is >> simply to redefine the code being evaluated so there's something >> to run... > > If the user enters: > > int f(int x) { return x; } > > It is wrapped in a class, compiled, and just loaded. > > But if they then enter: > > int f(int x) { return x * x; } > > It is wrapped in a class (of the same name), compiled, and redefined > with JDI RedefineClasses. > (failing that, it does less graceful things) > > And that is why I'm using JDI. > >> >> Please let me know if I didn't review something that you needed >> me to review. Also let me know if I'm even close to understanding >> what you're using JDI for in this area... :-) > > What I'd be most interested from you would be any interprocess and > synchronization stuff. > Clashes between JDI threads and remote, risk of hangs, races, ... > > Thanks, > Robert > >> >> Dan >> >> >> >> On 9/10/15 1:06 PM, Robert Field wrote: >>> Hi Dan, >>> >>> Long time again. >>> >>> I've been working on adding a Read-Eval-Print Loop Java tool. We are >>> on the verge of pushing it into JDK9, it has had an API review but >>> needs a code review. It is a lot of new code, so we are breaking >>> the task into areas. We have reviewers for the tool (built on the >>> API), and for the compilation related areas. But we need someone >>> for the JDI / remote interaction areas. Not much experience on this >>> team in JDI. I'm hoping you would be will to do the review of that >>> area. Let me know and I'll point you at the appropriate bits. >>> >>> Thanks, >>> Robert >>> >>> >>> >>> -------- Original Message -------- >>> Subject: JDK 9 RFR of JDK-8134254 JShell API/tool: REPL for Java >>> into JDK9 >>> Date: Fri, 21 Aug 2015 23:06:06 -0700 >>> From: Robert Field >>> To: kulla-dev >>> >>> >>> >>> Please review: >>> >>> 8134254 JShell API/tool: REPL for Java into JDK9 >>> https://bugs.openjdk.java.net/browse/JDK-8134254 >>> >>> http://cr.openjdk.java.net/~rfield/jshell_langtools_webrev_v0/ >>> >>> This webrev is of all of the JShell changes in langtools, which is >>> the addition of the source code and all the tests. >>> For context, please familiarize yourself with the API first: >>> >>> http://cr.openjdk.java.net/~rfield/doc/ >>> >>> Since the webrev is all adds, and the order and access is far from >>> ideal, it is almost certainly easier to clone the kulla repo: >>> >>> http://hg.openjdk.java.net/kulla/dev >>> >>> This webrev is the contents of these langtools directories >>> >>> src/jdk.jshell >>> test/jdk/jshell >>> >>> added to jdk9/dev/langtools. >>> >>> JShell.java and JShellImpl.java in >>> src/jdk.jshell/share/classes/jdk/jshell/ would make a logical >>> starting point for understanding the code. >>> >>> There are also small changes to the base jdk9 repo for make files >>> and to add the module, and to the jdk repo to add the launcher, I >>> will send these separately. >>> >>> Thanks, >>> Robert >>> >>> >>> >> > From robert.field at oracle.com Sat Sep 26 04:42:10 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Sat, 26 Sep 2015 04:42:10 +0000 Subject: hg: kulla/dev/langtools: Partial 8137064: JShell API: circular dependencies cause infinite circular update. Steps to simultaneous compiles: Our own Diag interface. Generated classes no longer public, not package-private. Message-ID: <201509260442.t8Q4gAWc005389@aojmv0008.oracle.com> Changeset: 9ae2799dfeb3 Author: rfield Date: 2015-09-25 21:41 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/9ae2799dfeb3 Partial 8137064: JShell API: circular dependencies cause infinite circular update. Steps to simultaneous compiles: Our own Diag interface. Generated classes no longer public, not package-private. ! src/jdk.jshell/share/classes/jdk/internal/jshell/remote/RemoteAgent.java ! src/jdk.jshell/share/classes/jdk/internal/jshell/tool/JShellTool.java + src/jdk.jshell/share/classes/jdk/jshell/Diag.java ! src/jdk.jshell/share/classes/jdk/jshell/Eval.java ! src/jdk.jshell/share/classes/jdk/jshell/JShell.java ! src/jdk.jshell/share/classes/jdk/jshell/MemoryFileManager.java ! src/jdk.jshell/share/classes/jdk/jshell/OuterWrap.java ! src/jdk.jshell/share/classes/jdk/jshell/Snippet.java ! src/jdk.jshell/share/classes/jdk/jshell/TaskFactory.java ! test/jdk/jshell/ExpectedDiagnostic.java ! test/jdk/jshell/KullaTesting.java ! test/jdk/jshell/RejectedFailedTest.java From robert.field at oracle.com Sun Sep 27 03:54:58 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Sun, 27 Sep 2015 03:54:58 +0000 Subject: hg: kulla/dev/langtools: Partial 8137064: JShell API: circular dependencies cause infinite circular update. More steps to simultaneous compiles: TaskFactory restructured to allow multiple inputs and to track the Snippet in the JavaFileObject for later Diagnostic demultiplexing. Message-ID: <201509270354.t8R3sw61007778@aojmv0008.oracle.com> Changeset: 46699721f34e Author: rfield Date: 2015-09-26 20:54 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/46699721f34e Partial 8137064: JShell API: circular dependencies cause infinite circular update. More steps to simultaneous compiles: TaskFactory restructured to allow multiple inputs and to track the Snippet in the JavaFileObject for later Diagnostic demultiplexing. ! src/jdk.jshell/share/classes/jdk/jshell/CompletenessAnalyzer.java ! src/jdk.jshell/share/classes/jdk/jshell/Eval.java ! src/jdk.jshell/share/classes/jdk/jshell/MemoryFileManager.java ! src/jdk.jshell/share/classes/jdk/jshell/Snippet.java ! src/jdk.jshell/share/classes/jdk/jshell/SourceCodeAnalysisImpl.java ! src/jdk.jshell/share/classes/jdk/jshell/TaskFactory.java From robert.field at oracle.com Sun Sep 27 21:02:37 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Sun, 27 Sep 2015 21:02:37 +0000 Subject: hg: kulla/dev/langtools: Partial 8137064: JShell API: circular dependencies cause infinite circular update. Decompose compilation, individually tracking Snippet compilation state. Message-ID: <201509272102.t8RL2bOZ023271@aojmv0008.oracle.com> Changeset: 85ea062798c4 Author: rfield Date: 2015-09-27 14:02 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/85ea062798c4 Partial 8137064: JShell API: circular dependencies cause infinite circular update. Decompose compilation, individually tracking Snippet compilation state. + src/jdk.jshell/share/classes/jdk/jshell/Compile.java ! src/jdk.jshell/share/classes/jdk/jshell/Eval.java ! src/jdk.jshell/share/classes/jdk/jshell/JShell.java ! src/jdk.jshell/share/classes/jdk/jshell/Snippet.java From robert.field at oracle.com Mon Sep 28 17:25:17 2015 From: robert.field at oracle.com (robert.field at oracle.com) Date: Mon, 28 Sep 2015 17:25:17 +0000 Subject: hg: kulla/dev/langtools: Partial 8137064: JShell API: circular dependencies cause infinite circular update. Demultiplex diagnostics using specialized diagnostic list. Message-ID: <201509281725.t8SHPHqk007451@aojmv0008.oracle.com> Changeset: 94a7406584f4 Author: rfield Date: 2015-09-28 10:25 -0700 URL: http://hg.openjdk.java.net/kulla/dev/langtools/rev/94a7406584f4 Partial 8137064: JShell API: circular dependencies cause infinite circular update. Demultiplex diagnostics using specialized diagnostic list. ! src/jdk.jshell/share/classes/jdk/jshell/Compile.java ! src/jdk.jshell/share/classes/jdk/jshell/Diag.java + src/jdk.jshell/share/classes/jdk/jshell/DiagList.java ! src/jdk.jshell/share/classes/jdk/jshell/Eval.java ! src/jdk.jshell/share/classes/jdk/jshell/OuterWrap.java ! src/jdk.jshell/share/classes/jdk/jshell/Snippet.java ! src/jdk.jshell/share/classes/jdk/jshell/TaskFactory.java