From jan.lahoda at oracle.com Wed May 6 15:30:52 2020 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Wed, 6 May 2020 17:30:52 +0200 Subject: Faster incremental OpenJDK compilation Message-ID: Hi, Triggered by Magnus' recent e-mail on adjusting the location of the IDE files, I looked at possibilities to improve speed of incremental compilation using make. About 3 years ago, we have sped up incremental build by only rebuilding modules when API of modules they depend on changed. But the module which contains modified sources is always compiled in full. So, for changes in java.base, this change improved the incremental build time significantly (by not recompiling e.g. java.desktop), but it can still take many seconds to build java.base after a trivial change. So, this time, I am thinking of speeding up module builds by not rebuilding all the source if possible. What I am thinking of is a relatively simple approach: detect changed files in a module and check if their "interface" changed. If it did, recompile the whole module. If it didn't, only compile the modified files. As a consequence, a small change inside a method body should lead to a fast build. Changes outside of method bodies may trigger longer build, but my hope would be that these would be less common. So far, I unfortunately don't know how to efficiently do this as nicely as the API digests used for inter-module dependencies. The approach that seems might work is this: the Depend plugin hooks itself into javac internals, and filters the incoming files - it first parses the modified ones, and if it cannot find a significant change, it will throw away the unmodified files, and only compile the modified ones. (In principle, it could also do dependency analysis, if we at some point decide it is important.) For a simple "touch Object.java && make", the wall-clock time is less then 5s, which sounds interesting: --- $ touch src/java.base/share/classes/java/lang/Object.java && time make Building target 'default (exploded-image)' in configuration 'linux-x86_64-server-release' Compiling 3028 files for java.base Stopping sjavac server Finished building target 'default (exploded-image)' in configuration 'linux-x86_64-server-release' real 0m3,104s user 0m5,731s sys 0m1,086s --- My current prototype is in the jdk/sandbox repository, branch "jlahoda-depend-in-module": http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module I wonder if this would sound interesting to developers working on base modules, like java.base. What do you think? Any ideas/feedback? Thanks, Jan From jonathan.gibbons at oracle.com Wed May 6 16:07:34 2020 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 6 May 2020 09:07:34 -0700 Subject: Faster incremental OpenJDK compilation In-Reply-To: References: Message-ID: Jan, This seems like an interesting approach. How are you determining "significant change"?? I could imagine trying to do this by looking at the changed lines, to see if they only within method bodies and comments (for example), or by doing some sort of lexical hash on the signatures, assuming that folk aren't messing with imports to change the resolution of simple type names in the signature. -- Jon On 5/6/20 8:30 AM, Jan Lahoda wrote: > Hi, > > Triggered by Magnus' recent e-mail on adjusting the location of the > IDE files, I looked at possibilities to improve speed of incremental > compilation using make. About 3 years ago, we have sped up incremental > build by only rebuilding modules when API of modules they depend on > changed. But the module which contains modified sources is always > compiled in full. So, for changes in java.base, this change improved > the incremental build time significantly (by not recompiling e.g. > java.desktop), but it can still take many seconds to build java.base > after a trivial change. So, this time, I am thinking of speeding up > module builds by not rebuilding all the source if possible. > > What I am thinking of is a relatively simple approach: detect changed > files in a module and check if their "interface" changed. If it did, > recompile the whole module. If it didn't, only compile the modified > files. As a consequence, a small change inside a method body should > lead to a fast build. Changes outside of method bodies may trigger > longer build, but my hope would be that these would be less common. > > So far, I unfortunately don't know how to efficiently do this as > nicely as the API digests used for inter-module dependencies. The > approach that seems might work is this: the Depend plugin hooks itself > into javac internals, and filters the incoming files - it first parses > the modified ones, and if it cannot find a significant change, it will > throw away the unmodified files, and only compile the modified ones. > (In principle, it could also do dependency analysis, if we at some > point decide it is important.) > > For a simple "touch Object.java && make", the wall-clock time is less > then 5s, which sounds interesting: > --- > $ touch src/java.base/share/classes/java/lang/Object.java && time make > Building target 'default (exploded-image)' in configuration > 'linux-x86_64-server-release' > Compiling 3028 files for java.base > Stopping sjavac server > Finished building target 'default (exploded-image)' in configuration > 'linux-x86_64-server-release' > > real??? 0m3,104s > user??? 0m5,731s > sys???? 0m1,086s > --- > > My current prototype is in the jdk/sandbox repository, branch > "jlahoda-depend-in-module": > http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module > > I wonder if this would sound interesting to developers working on base > modules, like java.base. > > What do you think? Any ideas/feedback? > > Thanks, > ???? Jan > From amaembo at gmail.com Wed May 6 16:05:47 2020 From: amaembo at gmail.com (Tagir Valeev) Date: Wed, 6 May 2020 23:05:47 +0700 Subject: Faster incremental OpenJDK compilation In-Reply-To: <234188178.1766036.1588780598274.JavaMail.zimbra@u-pem.fr> References: <234188178.1766036.1588780598274.JavaMail.zimbra@u-pem.fr> Message-ID: Hello! I think you are missing the dependencies that are changing the generated > code, > - in a switch, javac generates a different code depending on the order of > the enum constants, > - static final are inlined in the code, > - for valhalla, changing a class to an inline class or vice versa change > all the method/field descriptors > > and i'm sure there are more ... > I think changing the annotation retention policy affects the compilation of every use site of that annotation. Changing target (e.g. adding or removing TYPE_USE) may also change the generated code for the use sites. > > > > For a simple "touch Object.java && make", the wall-clock time is less > > then 5s, which sounds interesting: > > --- > > $ touch src/java.base/share/classes/java/lang/Object.java && time make > > Building target 'default (exploded-image)' in configuration > > 'linux-x86_64-server-release' > > Compiling 3028 files for java.base > > Stopping sjavac server > > Finished building target 'default (exploded-image)' in configuration > > 'linux-x86_64-server-release' > > > > real 0m3,104s > > user 0m5,731s > > sys 0m1,086s > > --- > > > > My current prototype is in the jdk/sandbox repository, branch > > "jlahoda-depend-in-module": > > http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module > > > > I wonder if this would sound interesting to developers working on base > > modules, like java.base. > > > > What do you think? Any ideas/feedback? > > > > Thanks, > > Jan > > R?mi > From jan.lahoda at oracle.com Wed May 6 16:37:50 2020 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Wed, 6 May 2020 18:37:50 +0200 Subject: Faster incremental OpenJDK compilation In-Reply-To: <234188178.1766036.1588780598274.JavaMail.zimbra@u-pem.fr> References: <234188178.1766036.1588780598274.JavaMail.zimbra@u-pem.fr> Message-ID: <0275d0eb-cde5-dafd-51ca-ac2836cf4074@oracle.com> On 06. 05. 20 17:56, Remi Forax wrote: > > > ----- Mail original ----- >> De: "jan lahoda" >> ?: ide-support-dev at openjdk.java.net, "compiler-dev" >> Envoy?: Mercredi 6 Mai 2020 17:30:52 >> Objet: Faster incremental OpenJDK compilation > >> Hi, >> >> Triggered by Magnus' recent e-mail on adjusting the location of the IDE >> files, I looked at possibilities to improve speed of incremental >> compilation using make. About 3 years ago, we have sped up incremental >> build by only rebuilding modules when API of modules they depend on >> changed. But the module which contains modified sources is always >> compiled in full. So, for changes in java.base, this change improved the >> incremental build time significantly (by not recompiling e.g. >> java.desktop), but it can still take many seconds to build java.base >> after a trivial change. So, this time, I am thinking of speeding up >> module builds by not rebuilding all the source if possible. >> >> What I am thinking of is a relatively simple approach: detect changed >> files in a module and check if their "interface" changed. If it did, >> recompile the whole module. If it didn't, only compile the modified >> files. As a consequence, a small change inside a method body should lead >> to a fast build. Changes outside of method bodies may trigger longer >> build, but my hope would be that these would be less common. >> >> So far, I unfortunately don't know how to efficiently do this as nicely >> as the API digests used for inter-module dependencies. The approach that >> seems might work is this: the Depend plugin hooks itself into javac >> internals, and filters the incoming files - it first parses the modified >> ones, and if it cannot find a significant change, it will throw away the >> unmodified files, and only compile the modified ones. (In principle, it >> could also do dependency analysis, if we at some point decide it is >> important.) > > > I think you are missing the dependencies that are changing the generated code, Do you mean for the current hashing, or for the new hashing? The new hashing is more sketchy now (I just realized modifiers are not reflected in the hash properly). But in both cases, I believe the intent is to make the hashing conservative (and easy), potentially causing more rebuilds than necessary, although I could see some arguments for being more optimistic. (This is basically only for people developing OpenJDK, after all.) > - in a switch, javac generates a different code depending on the order of the enum constants, I believe that any change to order of significant members will change the API hashes. (Which, admittedly, might be overly conservative.) > - static final are inlined in the code, For the existing API hash, changes to compile-time constants should change the hash. For the new intra-module hash, currently, (significant) fields initializers are included in full (this is probably overly pessimistic.) > - for valhalla, changing a class to an inline class or vice versa change all the method/field descriptors As long as that is reflected in the modifiers, we should be able to handle fine. If not, then the hashing may need some more tweaks. > > and i'm sure there are more .. Possibly - and it is entirely possible the current hashes are way too conservative. But what I wonder about is if people developing e.g. java.base would find such faster incremental builds useful. Jan > >> >> For a simple "touch Object.java && make", the wall-clock time is less >> then 5s, which sounds interesting: >> --- >> $ touch src/java.base/share/classes/java/lang/Object.java && time make >> Building target 'default (exploded-image)' in configuration >> 'linux-x86_64-server-release' >> Compiling 3028 files for java.base >> Stopping sjavac server >> Finished building target 'default (exploded-image)' in configuration >> 'linux-x86_64-server-release' >> >> real 0m3,104s >> user 0m5,731s >> sys 0m1,086s >> --- >> >> My current prototype is in the jdk/sandbox repository, branch >> "jlahoda-depend-in-module": >> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module >> >> I wonder if this would sound interesting to developers working on base >> modules, like java.base. >> >> What do you think? Any ideas/feedback? >> >> Thanks, >> Jan > > R?mi > From maurizio.cimadamore at oracle.com Wed May 6 16:48:18 2020 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Wed, 6 May 2020 17:48:18 +0100 Subject: Faster incremental OpenJDK compilation In-Reply-To: References: Message-ID: On 06/05/2020 16:30, Jan Lahoda wrote: > I wonder if this would sound interesting to developers working on base > modules, like java.base. This would be extremely good. When working on Valhalla/Panama it happened to me quite frequently that, when changing something in the _implementation_ of one of the core classes (var handle, method handle, you name it), as a result, everything else needs to be recompiled. After having witnessed some of the attempts with dependency tracking with sjavac, my personal feeling is that simpler is better - and what you have might well be a sweet spot in terms of complexity vs. benefits ratio. Maurizio From jan.lahoda at oracle.com Wed May 6 16:49:28 2020 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Wed, 6 May 2020 18:49:28 +0200 Subject: Faster incremental OpenJDK compilation In-Reply-To: References: Message-ID: <2f8b5f2f-f037-22fe-544b-4fc37617fa35@oracle.com> Hi Jon, Good question. I was first experimenting with hashing Elements, as we do currently for the module API hashes, but it was too slow to enter the sources (although it is not completely impossible). So I did a new hashing based on the AST - basically, it is a TreeScanner sending Tree kinds, names and sub-trees into a MessageDigest. It ignores things like method bodies, class initializers, and private class members. Does not currently skip imports, although that may be seen as too conservative. But tweaks to the AST hashing are surely still needed, so we can tweak the exact meaning of a "significant change". Jan On 06. 05. 20 18:07, Jonathan Gibbons wrote: > Jan, > > This seems like an interesting approach. > > How are you determining "significant change"?? I could imagine trying to > do this by looking at the changed lines, to see if they only within > method bodies and comments (for example), or by doing some sort of > lexical hash on the signatures, assuming that folk aren't messing with > imports to change the resolution of simple type names in the signature. > > -- Jon > > On 5/6/20 8:30 AM, Jan Lahoda wrote: >> Hi, >> >> Triggered by Magnus' recent e-mail on adjusting the location of the >> IDE files, I looked at possibilities to improve speed of incremental >> compilation using make. About 3 years ago, we have sped up incremental >> build by only rebuilding modules when API of modules they depend on >> changed. But the module which contains modified sources is always >> compiled in full. So, for changes in java.base, this change improved >> the incremental build time significantly (by not recompiling e.g. >> java.desktop), but it can still take many seconds to build java.base >> after a trivial change. So, this time, I am thinking of speeding up >> module builds by not rebuilding all the source if possible. >> >> What I am thinking of is a relatively simple approach: detect changed >> files in a module and check if their "interface" changed. If it did, >> recompile the whole module. If it didn't, only compile the modified >> files. As a consequence, a small change inside a method body should >> lead to a fast build. Changes outside of method bodies may trigger >> longer build, but my hope would be that these would be less common. >> >> So far, I unfortunately don't know how to efficiently do this as >> nicely as the API digests used for inter-module dependencies. The >> approach that seems might work is this: the Depend plugin hooks itself >> into javac internals, and filters the incoming files - it first parses >> the modified ones, and if it cannot find a significant change, it will >> throw away the unmodified files, and only compile the modified ones. >> (In principle, it could also do dependency analysis, if we at some >> point decide it is important.) >> >> For a simple "touch Object.java && make", the wall-clock time is less >> then 5s, which sounds interesting: >> --- >> $ touch src/java.base/share/classes/java/lang/Object.java && time make >> Building target 'default (exploded-image)' in configuration >> 'linux-x86_64-server-release' >> Compiling 3028 files for java.base >> Stopping sjavac server >> Finished building target 'default (exploded-image)' in configuration >> 'linux-x86_64-server-release' >> >> real??? 0m3,104s >> user??? 0m5,731s >> sys???? 0m1,086s >> --- >> >> My current prototype is in the jdk/sandbox repository, branch >> "jlahoda-depend-in-module": >> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module >> >> I wonder if this would sound interesting to developers working on base >> modules, like java.base. >> >> What do you think? Any ideas/feedback? >> >> Thanks, >> ???? Jan >> From jan.lahoda at oracle.com Wed May 6 16:52:54 2020 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Wed, 6 May 2020 18:52:54 +0200 Subject: Faster incremental OpenJDK compilation In-Reply-To: <0275d0eb-cde5-dafd-51ca-ac2836cf4074@oracle.com> References: <234188178.1766036.1588780598274.JavaMail.zimbra@u-pem.fr> <0275d0eb-cde5-dafd-51ca-ac2836cf4074@oracle.com> Message-ID: FWIW, I should explain this only relates to the OpenJDK build. This is not a proposal to do any kind of tweaks like this in javac for other compilations. I realized I only wrote that into the subject, but not the body. Sorry for that. Jan On 06. 05. 20 18:37, Jan Lahoda wrote: > On 06. 05. 20 17:56, Remi Forax wrote: >> >> >> ----- Mail original ----- >>> De: "jan lahoda" >>> ?: ide-support-dev at openjdk.java.net, "compiler-dev" >>> >>> Envoy?: Mercredi 6 Mai 2020 17:30:52 >>> Objet: Faster incremental OpenJDK compilation >> >>> Hi, >>> >>> Triggered by Magnus' recent e-mail on adjusting the location of the IDE >>> files, I looked at possibilities to improve speed of incremental >>> compilation using make. About 3 years ago, we have sped up incremental >>> build by only rebuilding modules when API of modules they depend on >>> changed. But the module which contains modified sources is always >>> compiled in full. So, for changes in java.base, this change improved the >>> incremental build time significantly (by not recompiling e.g. >>> java.desktop), but it can still take many seconds to build java.base >>> after a trivial change. So, this time, I am thinking of speeding up >>> module builds by not rebuilding all the source if possible. >>> >>> What I am thinking of is a relatively simple approach: detect changed >>> files in a module and check if their "interface" changed. If it did, >>> recompile the whole module. If it didn't, only compile the modified >>> files. As a consequence, a small change inside a method body should lead >>> to a fast build. Changes outside of method bodies may trigger longer >>> build, but my hope would be that these would be less common. >>> >>> So far, I unfortunately don't know how to efficiently do this as nicely >>> as the API digests used for inter-module dependencies. The approach that >>> seems might work is this: the Depend plugin hooks itself into javac >>> internals, and filters the incoming files - it first parses the modified >>> ones, and if it cannot find a significant change, it will throw away the >>> unmodified files, and only compile the modified ones. (In principle, it >>> could also do dependency analysis, if we at some point decide it is >>> important.) >> >> >> I think you are missing the dependencies that are changing the >> generated code, > > Do you mean for the current hashing, or for the new hashing? > > The new hashing is more sketchy now (I just realized modifiers are not > reflected in the hash properly). But in both cases, I believe the intent > is to make the hashing conservative (and easy), potentially causing more > rebuilds than necessary, although I could see some arguments for being > more optimistic. (This is basically only for people developing OpenJDK, > after all.) > >> - in a switch, javac generates a different code depending on the order >> of the enum constants, > > I believe that any change to order of significant members will change > the API hashes. (Which, admittedly, might be overly conservative.) > >> - static final are inlined in the code, > > For the existing API hash, changes to compile-time constants should > change the hash. For the new intra-module hash, currently, (significant) > fields initializers are included in full (this is probably overly > pessimistic.) > >> - for valhalla, changing a class to an inline class or vice versa >> change all the method/field descriptors > > As long as that is reflected in the modifiers, we should be able to > handle fine. If not, then the hashing may need some more tweaks. > >> >> and i'm sure there are more .. > > Possibly - and it is entirely possible the current hashes are way too > conservative. But what I wonder about is if people developing e.g. > java.base would find such faster incremental builds useful. > > Jan > >> >>> >>> For a simple "touch Object.java && make", the wall-clock time is less >>> then 5s, which sounds interesting: >>> --- >>> $ touch src/java.base/share/classes/java/lang/Object.java && time make >>> Building target 'default (exploded-image)' in configuration >>> 'linux-x86_64-server-release' >>> Compiling 3028 files for java.base >>> Stopping sjavac server >>> Finished building target 'default (exploded-image)' in configuration >>> 'linux-x86_64-server-release' >>> >>> real??? 0m3,104s >>> user??? 0m5,731s >>> sys???? 0m1,086s >>> --- >>> >>> My current prototype is in the jdk/sandbox repository, branch >>> "jlahoda-depend-in-module": >>> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module >>> >>> I wonder if this would sound interesting to developers working on base >>> modules, like java.base. >>> >>> What do you think? Any ideas/feedback? >>> >>> Thanks, >>> ?????? Jan >> >> R?mi >> From jonathan.gibbons at oracle.com Wed May 6 17:23:46 2020 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 6 May 2020 10:23:46 -0700 Subject: Faster incremental OpenJDK compilation In-Reply-To: <2f8b5f2f-f037-22fe-544b-4fc37617fa35@oracle.com> References: <2f8b5f2f-f037-22fe-544b-4fc37617fa35@oracle.com> Message-ID: You could refine the imports by filtering them according to whether they are 'star-imports' or the simple name matches a simple name that has been seen by the TreeScanner.? In other words, try and filter out imports that are definitely not relevant. -- Jon On 5/6/20 9:49 AM, Jan Lahoda wrote: > Hi Jon, > > Good question. I was first experimenting with hashing Elements, as we > do currently for the module API hashes, but it was too slow to enter > the sources (although it is not completely impossible). So I did a new > hashing based on the AST - basically, it is a TreeScanner sending Tree > kinds, names and sub-trees into a MessageDigest. It ignores things > like method bodies, class initializers, and private class members. > Does not currently skip imports, although that may be seen as too > conservative. But tweaks to the AST hashing are surely still needed, > so we can tweak the exact meaning of a "significant change". > > Jan > > On 06. 05. 20 18:07, Jonathan Gibbons wrote: >> Jan, >> >> This seems like an interesting approach. >> >> How are you determining "significant change"?? I could imagine trying >> to do this by looking at the changed lines, to see if they only >> within method bodies and comments (for example), or by doing some >> sort of lexical hash on the signatures, assuming that folk aren't >> messing with imports to change the resolution of simple type names in >> the signature. >> >> -- Jon >> >> On 5/6/20 8:30 AM, Jan Lahoda wrote: >>> Hi, >>> >>> Triggered by Magnus' recent e-mail on adjusting the location of the >>> IDE files, I looked at possibilities to improve speed of incremental >>> compilation using make. About 3 years ago, we have sped up >>> incremental build by only rebuilding modules when API of modules >>> they depend on changed. But the module which contains modified >>> sources is always compiled in full. So, for changes in java.base, >>> this change improved the incremental build time significantly (by >>> not recompiling e.g. java.desktop), but it can still take many >>> seconds to build java.base after a trivial change. So, this time, I >>> am thinking of speeding up module builds by not rebuilding all the >>> source if possible. >>> >>> What I am thinking of is a relatively simple approach: detect >>> changed files in a module and check if their "interface" changed. If >>> it did, recompile the whole module. If it didn't, only compile the >>> modified files. As a consequence, a small change inside a method >>> body should lead to a fast build. Changes outside of method bodies >>> may trigger longer build, but my hope would be that these would be >>> less common. >>> >>> So far, I unfortunately don't know how to efficiently do this as >>> nicely as the API digests used for inter-module dependencies. The >>> approach that seems might work is this: the Depend plugin hooks >>> itself into javac internals, and filters the incoming files - it >>> first parses the modified ones, and if it cannot find a significant >>> change, it will throw away the unmodified files, and only compile >>> the modified ones. (In principle, it could also do dependency >>> analysis, if we at some point decide it is important.) >>> >>> For a simple "touch Object.java && make", the wall-clock time is >>> less then 5s, which sounds interesting: >>> --- >>> $ touch src/java.base/share/classes/java/lang/Object.java && time make >>> Building target 'default (exploded-image)' in configuration >>> 'linux-x86_64-server-release' >>> Compiling 3028 files for java.base >>> Stopping sjavac server >>> Finished building target 'default (exploded-image)' in configuration >>> 'linux-x86_64-server-release' >>> >>> real??? 0m3,104s >>> user??? 0m5,731s >>> sys???? 0m1,086s >>> --- >>> >>> My current prototype is in the jdk/sandbox repository, branch >>> "jlahoda-depend-in-module": >>> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module >>> >>> >>> I wonder if this would sound interesting to developers working on >>> base modules, like java.base. >>> >>> What do you think? Any ideas/feedback? >>> >>> Thanks, >>> ???? Jan >>> From bradford.wetmore at oracle.com Wed May 6 23:40:59 2020 From: bradford.wetmore at oracle.com (Bradford Wetmore) Date: Wed, 6 May 2020 16:40:59 -0700 Subject: RFR: JDK-8244093 Move all IDE support into coherent structure in make directory In-Reply-To: <1f8df2d9-0547-0a2b-34ca-d0e8de8b8c61@oracle.com> References: <1e9d4486-351b-742b-766f-609c2d360abb@oracle.com> <1f8df2d9-0547-0a2b-34ca-d0e8de8b8c61@oracle.com> Message-ID: <829b49c4-0789-804d-72e6-8e449a0498fa@oracle.com> On 4/30/2020 12:28 AM, Jan Lahoda wrote: > Hi Brad, > > This is very similar to what I do. Some small comments inline. > > On 29. 04. 20 21:02, Bradford Wetmore wrote: >> Jan, >> >> What is your current recommended technique to use NetBeans to >> build/edit/test OpenJDK for normal OpenJDK library developers? >> >> After many versions of Netbeans, my current setup is to: >> >> 1.? Do an external "exploded" build >> >> 2.? Run Netbeans, open the src/java.base module project and any other >> modules needed. >> >> 3a. For OpenJDK test files under test/jdk:? Open the java.base test >> dirs for the JTREG tests I need.? Then run "Debug Test File" to invoke >> JTREG >> >> 3b. For tests not in test/jdk (external to repo): >> ???????? 1.? Add a Java platform for the exploded-build > > I believe a platform for the exploded build should be added > automatically when the project is opened. Its name is > "-", e.g. "jdk - > linux-x86_64-server-release". If deleted, this platform should be > re-created on next start/project open. But creating one manually works > as well. Hmmm...I'll have to try that next time. >> ???????? 2.? Create a project, assign the platform >> ???????? 3.? Create standard Debug project. >> >> 4.? For incremental builds, edit source file, build from the command >> line something like: >> >> ???? % make JDK_FILTER=java/security java.base-java-only > > If there is something that could help here, please let me know. I am > peeking a little at possibilities to speed up incremental build (so that > plain "make" would be faster), but not much success so far. This works for me, because I know about it. But for newbies, it's not always obvious. There was talk about removing JDK_FILTER a while ago, but it's incredibly useful to avoid excess builds. Glad it's still here. When I get a few minutes, I'll try to update/resend my semi-annual email with the latest info. Brad >> Is there another way of working I'm missing? >> >> Things have been really solid for me lately (OpenJDK/JTREG plugins not >> needed), thanks for all your hard work on this. > > Thank you for your support! > > Jan > >> >> Cheers, >> >> Brad >> >> >> On 4/29/2020 4:06 AM, Jan Lahoda wrote: >>> I am not sure if anyone is still using make/jdk/netbeans. Apache >>> NetBeans does not (should not) need these config files, it supports >>> OpenJDK modules out of the box (with some tweaks/dependencies on >>> make/langtools/** to speed up langtools build). >>> >>> As Maurizio, there may be some need to move the "fast" langtools >>> build more carefully. I'll try to take a look later, unless some else >>> wants to. Although, a little independently, I wonder somewhat if >>> there's an opportunity to further speed up the ordinary make build in >>> incremental environment to reduce the need for a "fast" langtools >>> build. E.g. by enhancing the current Depend javac plugin, and >>> possibly optionally disabling the interim langtools build. This could >>> improve incremental build behavior for other modules (like java.base >>> or java.desktop) as well. >>> >>> Jan >>> >>> On 29. 04. 20 12:36, Magnus Ihse Bursie wrote: >>>> The IDE support in OpenJDK unfortunately leaves a lot to be desired. >>>> There have been a garden variety of attempt to support a specific >>>> IDE for a specific part of the code base, cluttered all over the >>>> code base. >>>> >>>> This patch is a first attempt go get one ring, eh..., structure, to >>>> rule them all. >>>> >>>> I have moved all IDE project creators into the following structure: >>>> >>>> make/ide// >>>> >>>> where is one of currently "hotspot", "langtools" >>>> or "jdk", and is one of "vscode", "idea", "netbeans" or >>>> "vistualstudio". >>>> >>>> This will not magically improve IDE support, but will at least make >>>> it clearer what we have and what we are missing. >>>> >>>> Ownership of the IDE support is notoriously vague. I've cc:ed a >>>> bunch of people who has shown interest and/or submitted fixes to >>>> some of the IDE projects according to the hg history. I'd appreciate >>>> it if anyone who is interested in a particular case for IDE support >>>> can verify that it still works. I've tried my best to make sure all >>>> targets can run without errors, but I cannot verify that the IDE >>>> environment themselves are correct. >>>> >>>> If you know about an IDE project that is no longer relevant, and >>>> should be removed instead of shuffled around, please let me know! >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8244093 >>>> WebRev: >>>> http://cr.openjdk.java.net/~ihse/JDK-8244093-move-ide-support/webrev.01 >>>> >>>> /Magnus From jan.lahoda at oracle.com Thu May 7 08:26:04 2020 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Thu, 7 May 2020 10:26:04 +0200 Subject: Faster incremental OpenJDK compilation In-Reply-To: References: <2f8b5f2f-f037-22fe-544b-4fc37617fa35@oracle.com> Message-ID: On 06. 05. 20 19:23, Jonathan Gibbons wrote: > You could refine the imports by filtering them according to whether they > are 'star-imports' or the simple name matches a simple name that has > been seen by the TreeScanner.? In other words, try and filter out > imports that are definitely not relevant. Good idea. I've tried to do that, and also fixed some bugs in the AST hashing, and enhanced tests. All is still on "jlahoda-depend-in-module" branch in jdk/sandbox. Jan > > -- Jon > > > On 5/6/20 9:49 AM, Jan Lahoda wrote: >> Hi Jon, >> >> Good question. I was first experimenting with hashing Elements, as we >> do currently for the module API hashes, but it was too slow to enter >> the sources (although it is not completely impossible). So I did a new >> hashing based on the AST - basically, it is a TreeScanner sending Tree >> kinds, names and sub-trees into a MessageDigest. It ignores things >> like method bodies, class initializers, and private class members. >> Does not currently skip imports, although that may be seen as too >> conservative. But tweaks to the AST hashing are surely still needed, >> so we can tweak the exact meaning of a "significant change". >> >> Jan >> >> On 06. 05. 20 18:07, Jonathan Gibbons wrote: >>> Jan, >>> >>> This seems like an interesting approach. >>> >>> How are you determining "significant change"?? I could imagine trying >>> to do this by looking at the changed lines, to see if they only >>> within method bodies and comments (for example), or by doing some >>> sort of lexical hash on the signatures, assuming that folk aren't >>> messing with imports to change the resolution of simple type names in >>> the signature. >>> >>> -- Jon >>> >>> On 5/6/20 8:30 AM, Jan Lahoda wrote: >>>> Hi, >>>> >>>> Triggered by Magnus' recent e-mail on adjusting the location of the >>>> IDE files, I looked at possibilities to improve speed of incremental >>>> compilation using make. About 3 years ago, we have sped up >>>> incremental build by only rebuilding modules when API of modules >>>> they depend on changed. But the module which contains modified >>>> sources is always compiled in full. So, for changes in java.base, >>>> this change improved the incremental build time significantly (by >>>> not recompiling e.g. java.desktop), but it can still take many >>>> seconds to build java.base after a trivial change. So, this time, I >>>> am thinking of speeding up module builds by not rebuilding all the >>>> source if possible. >>>> >>>> What I am thinking of is a relatively simple approach: detect >>>> changed files in a module and check if their "interface" changed. If >>>> it did, recompile the whole module. If it didn't, only compile the >>>> modified files. As a consequence, a small change inside a method >>>> body should lead to a fast build. Changes outside of method bodies >>>> may trigger longer build, but my hope would be that these would be >>>> less common. >>>> >>>> So far, I unfortunately don't know how to efficiently do this as >>>> nicely as the API digests used for inter-module dependencies. The >>>> approach that seems might work is this: the Depend plugin hooks >>>> itself into javac internals, and filters the incoming files - it >>>> first parses the modified ones, and if it cannot find a significant >>>> change, it will throw away the unmodified files, and only compile >>>> the modified ones. (In principle, it could also do dependency >>>> analysis, if we at some point decide it is important.) >>>> >>>> For a simple "touch Object.java && make", the wall-clock time is >>>> less then 5s, which sounds interesting: >>>> --- >>>> $ touch src/java.base/share/classes/java/lang/Object.java && time make >>>> Building target 'default (exploded-image)' in configuration >>>> 'linux-x86_64-server-release' >>>> Compiling 3028 files for java.base >>>> Stopping sjavac server >>>> Finished building target 'default (exploded-image)' in configuration >>>> 'linux-x86_64-server-release' >>>> >>>> real??? 0m3,104s >>>> user??? 0m5,731s >>>> sys???? 0m1,086s >>>> --- >>>> >>>> My current prototype is in the jdk/sandbox repository, branch >>>> "jlahoda-depend-in-module": >>>> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module >>>> >>>> >>>> I wonder if this would sound interesting to developers working on >>>> base modules, like java.base. >>>> >>>> What do you think? Any ideas/feedback? >>>> >>>> Thanks, >>>> ???? Jan >>>> From magnus.ihse.bursie at oracle.com Tue May 19 11:33:58 2020 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Tue, 19 May 2020 13:33:58 +0200 Subject: RFR: JDK-8244093 Move all IDE support into coherent structure in make directory In-Reply-To: <37db738a-e1ea-2ba6-a2bd-839569437be8@oracle.com> References: <37db738a-e1ea-2ba6-a2bd-839569437be8@oracle.com> Message-ID: <9978edea-6891-d2d6-5d25-68e134f4b89f@oracle.com> How derpy of me. I forgot to post the link to the updated webrev! http://cr.openjdk.java.net/~ihse/JDK-8244093-move-ide-support/webrev.02/ /Magnus On 2020-05-18 16:50, Magnus Ihse Bursie wrote: > Hi all, > > Sorry for the long time to follow up on this patch. > > From the reviewers' feedback, I have now restored all files to > make/langtools, except for those in the netbeans and intellij > directories. This also meant that in the make/ide/idea/langtools > directory there were now only an "intellij" directory. This seemed > awkward, so I moved everything up one level, effectively scrapping the > "intellij" directory. Paths has been changed accordingly in > make/langtools and make/ide/*/langtools. > > I also removed make/ide/netbeans/jdk. No-one seemed to be countering > Jan's claim that it's unused. Instead, we got testimony on how > Netbeans can be setup without any specific support from the build > system. I think this excellent information should be properly > documented. I've created JDK-8245210 for this, and I hope I can enroll > the help of Jan and Brad to get this correctly documented. > > No other changes, outside the langtools files and > make/ide/netbeans/jdk has been made, compared to the previous version. > > /Magnus > > On 2020-04-29 13:06, Jan Lahoda wrote: >> I am not sure if anyone is still using make/jdk/netbeans. Apache >> NetBeans does not (should not) need these config files, it supports >> OpenJDK modules out of the box (with some tweaks/dependencies on >> make/langtools/** to speed up langtools build). >> >> As Maurizio, there may be some need to move the "fast" langtools >> build more carefully. I'll try to take a look later, unless some else >> wants to. Although, a little independently, I wonder somewhat if >> there's an opportunity to further speed up the ordinary make build in >> incremental environment to reduce the need for a "fast" langtools >> build. E.g. by enhancing the current Depend javac plugin, and >> possibly optionally disabling the interim langtools build. This could >> improve incremental build behavior for other modules (like java.base >> or java.desktop) as well. >> >> Jan >> >> On 29. 04. 20 12:36, Magnus Ihse Bursie wrote: >>> The IDE support in OpenJDK unfortunately leaves a lot to be desired. >>> There have been a garden variety of attempt to support a specific >>> IDE for a specific part of the code base, cluttered all over the >>> code base. >>> >>> This patch is a first attempt go get one ring, eh..., structure, to >>> rule them all. >>> >>> I have moved all IDE project creators into the following structure: >>> >>> make/ide// >>> >>> where is one of currently "hotspot", "langtools" >>> or "jdk", and is one of "vscode", "idea", "netbeans" or >>> "vistualstudio". >>> >>> This will not magically improve IDE support, but will at least make >>> it clearer what we have and what we are missing. >>> >>> Ownership of the IDE support is notoriously vague. I've cc:ed a >>> bunch of people who has shown interest and/or submitted fixes to >>> some of the IDE projects according to the hg history. I'd appreciate >>> it if anyone who is interested in a particular case for IDE support >>> can verify that it still works. I've tried my best to make sure all >>> targets can run without errors, but I cannot verify that the IDE >>> environment themselves are correct. >>> >>> If you know about an IDE project that is no longer relevant, and >>> should be removed instead of shuffled around, please let me know! >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8244093 >>> WebRev: >>> http://cr.openjdk.java.net/~ihse/JDK-8244093-move-ide-support/webrev.01 >>> >>> /Magnus > From erik.joelsson at oracle.com Tue May 19 13:27:03 2020 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Tue, 19 May 2020 06:27:03 -0700 Subject: RFR: JDK-8244093 Move all IDE support into coherent structure in make directory In-Reply-To: <9978edea-6891-d2d6-5d25-68e134f4b89f@oracle.com> References: <37db738a-e1ea-2ba6-a2bd-839569437be8@oracle.com> <9978edea-6891-d2d6-5d25-68e134f4b89f@oracle.com> Message-ID: <032a7f02-449e-f592-ded4-cef55a8c8da2@oracle.com> I like the new structure. Looks good. /Erik On 2020-05-19 04:33, Magnus Ihse Bursie wrote: > How derpy of me. I forgot to post the link to the updated webrev! > > http://cr.openjdk.java.net/~ihse/JDK-8244093-move-ide-support/webrev.02/ > > /Magnus > > On 2020-05-18 16:50, Magnus Ihse Bursie wrote: >> Hi all, >> >> Sorry for the long time to follow up on this patch. >> >> From the reviewers' feedback, I have now restored all files to >> make/langtools, except for those in the netbeans and intellij >> directories. This also meant that in the make/ide/idea/langtools >> directory there were now only an "intellij" directory. This seemed >> awkward, so I moved everything up one level, effectively scrapping >> the "intellij" directory. Paths has been changed accordingly in >> make/langtools and make/ide/*/langtools. >> >> I also removed make/ide/netbeans/jdk. No-one seemed to be countering >> Jan's claim that it's unused. Instead, we got testimony on how >> Netbeans can be setup without any specific support from the build >> system. I think this excellent information should be properly >> documented. I've created JDK-8245210 for this, and I hope I can >> enroll the help of Jan and Brad to get this correctly documented. >> >> No other changes, outside the langtools files and >> make/ide/netbeans/jdk has been made, compared to the previous version. >> >> /Magnus >> >> On 2020-04-29 13:06, Jan Lahoda wrote: >>> I am not sure if anyone is still using make/jdk/netbeans. Apache >>> NetBeans does not (should not) need these config files, it supports >>> OpenJDK modules out of the box (with some tweaks/dependencies on >>> make/langtools/** to speed up langtools build). >>> >>> As Maurizio, there may be some need to move the "fast" langtools >>> build more carefully. I'll try to take a look later, unless some >>> else wants to. Although, a little independently, I wonder somewhat >>> if there's an opportunity to further speed up the ordinary make >>> build in incremental environment to reduce the need for a "fast" >>> langtools build. E.g. by enhancing the current Depend javac plugin, >>> and possibly optionally disabling the interim langtools build. This >>> could improve incremental build behavior for other modules (like >>> java.base or java.desktop) as well. >>> >>> Jan >>> >>> On 29. 04. 20 12:36, Magnus Ihse Bursie wrote: >>>> The IDE support in OpenJDK unfortunately leaves a lot to be >>>> desired. There have been a garden variety of attempt to support a >>>> specific IDE for a specific part of the code base, cluttered all >>>> over the code base. >>>> >>>> This patch is a first attempt go get one ring, eh..., structure, to >>>> rule them all. >>>> >>>> I have moved all IDE project creators into the following structure: >>>> >>>> make/ide// >>>> >>>> where is one of currently "hotspot", "langtools" >>>> or "jdk", and is one of "vscode", "idea", "netbeans" or >>>> "vistualstudio". >>>> >>>> This will not magically improve IDE support, but will at least make >>>> it clearer what we have and what we are missing. >>>> >>>> Ownership of the IDE support is notoriously vague. I've cc:ed a >>>> bunch of people who has shown interest and/or submitted fixes to >>>> some of the IDE projects according to the hg history. I'd >>>> appreciate it if anyone who is interested in a particular case for >>>> IDE support can verify that it still works. I've tried my best to >>>> make sure all targets can run without errors, but I cannot verify >>>> that the IDE environment themselves are correct. >>>> >>>> If you know about an IDE project that is no longer relevant, and >>>> should be removed instead of shuffled around, please let me know! >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8244093 >>>> WebRev: >>>> http://cr.openjdk.java.net/~ihse/JDK-8244093-move-ide-support/webrev.01 >>>> >>>> >>>> /Magnus >> > From jan.lahoda at oracle.com Wed May 20 08:20:42 2020 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Wed, 20 May 2020 10:20:42 +0200 Subject: RFR: JDK-8245445: Langtools NetBeans ant build broken after JDK-8244093 Message-ID: Hi, The langtools-specific ant build for NetBeans appears to be broken after JDK-8244093: --- $ ant -f make/ide/netbeans/langtools/build.xml -Dlangtools.jdk.home=$PWD/build/linux-x86_64-server-release/jdk/ build Buildfile: /home/jlahoda/src/jdk/jdk/make/ide/netbeans/langtools/build.xml BUILD FAILED /home/jlahoda/src/jdk/jdk/make/ide/netbeans/langtools/build.xml:47: Cannot find /home/jlahoda/src/jdk/jdk/make/ide/build.xml imported from /home/jlahoda/src/jdk/jdk/make/ide/netbeans/langtools/build.xml Total time: 0 seconds --- The proposed fix is to correct the location of the imported build script, and basedir. Webrev: http://cr.openjdk.java.net/~jlahoda/8245445/webrev.00/ JBS: https://bugs.openjdk.java.net/browse/JDK-8245445 What do you think? Thanks, Jan From maurizio.cimadamore at oracle.com Wed May 20 09:50:39 2020 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Wed, 20 May 2020 10:50:39 +0100 Subject: RFR: JDK-8245445: Langtools NetBeans ant build broken after JDK-8244093 In-Reply-To: References: Message-ID: <994b2f17-fff5-5247-02e7-4b290bb15137@oracle.com> Looks good Maurizio On 20/05/2020 09:20, Jan Lahoda wrote: > Hi, > > The langtools-specific ant build for NetBeans appears to be broken > after JDK-8244093: > --- > $ ant -f make/ide/netbeans/langtools/build.xml > -Dlangtools.jdk.home=$PWD/build/linux-x86_64-server-release/jdk/ build > Buildfile: > /home/jlahoda/src/jdk/jdk/make/ide/netbeans/langtools/build.xml > > BUILD FAILED > /home/jlahoda/src/jdk/jdk/make/ide/netbeans/langtools/build.xml:47: > Cannot find /home/jlahoda/src/jdk/jdk/make/ide/build.xml imported from > /home/jlahoda/src/jdk/jdk/make/ide/netbeans/langtools/build.xml > > Total time: 0 seconds > --- > > The proposed fix is to correct the location of the imported build > script, and basedir. > > Webrev: http://cr.openjdk.java.net/~jlahoda/8245445/webrev.00/ > JBS: https://bugs.openjdk.java.net/browse/JDK-8245445 > > What do you think? > > Thanks, > ??? Jan From erik.joelsson at oracle.com Wed May 20 13:24:16 2020 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Wed, 20 May 2020 06:24:16 -0700 Subject: RFR: JDK-8245445: Langtools NetBeans ant build broken after JDK-8244093 In-Reply-To: References: Message-ID: Looks good. /Erik On 2020-05-20 01:20, Jan Lahoda wrote: > Hi, > > The langtools-specific ant build for NetBeans appears to be broken > after JDK-8244093: > --- > $ ant -f make/ide/netbeans/langtools/build.xml > -Dlangtools.jdk.home=$PWD/build/linux-x86_64-server-release/jdk/ build > Buildfile: > /home/jlahoda/src/jdk/jdk/make/ide/netbeans/langtools/build.xml > > BUILD FAILED > /home/jlahoda/src/jdk/jdk/make/ide/netbeans/langtools/build.xml:47: > Cannot find /home/jlahoda/src/jdk/jdk/make/ide/build.xml imported from > /home/jlahoda/src/jdk/jdk/make/ide/netbeans/langtools/build.xml > > Total time: 0 seconds > --- > > The proposed fix is to correct the location of the imported build > script, and basedir. > > Webrev: http://cr.openjdk.java.net/~jlahoda/8245445/webrev.00/ > JBS: https://bugs.openjdk.java.net/browse/JDK-8245445 > > What do you think? > > Thanks, > ??? Jan