[jdk8u-dev] RFR: 8327440: "bad source file" error during beaninfo generation

fitzsim duke at openjdk.org
Tue Mar 12 20:07:42 UTC 2024 

The `gensrc_no_srczip/_the.generated_beaninfo` build rule does not use any sources in `gensrc`.  Remove `$(JDK_OUTPUTDIR)/gensrc` from the `javadoc` `-sourcepath` argument, because its presence can cause build failures.

Other build targets (for example,
`gensrc/java/nio/_the.exceptions.dir`), generate source code under that path.  When make is invoked with more than one job, sometimes one of those other targets will be in the process of writing a source file when the `_the.generated_beaninfo` build rule invokes `javadoc`.  The result is build failures of the form:


Generating beaninfo
[...]\jdk8u-dev\jdk\src\share\classes\java\io\PrintStream.java:32: error: cannot access UnsupportedCharsetException
import java.nio.charset.UnsupportedCharsetException;
                       ^
  bad source file: [...]\jdk8u-dev\build\windows-x64\jdk\gensrc\java\nio\charset\UnsupportedCharsetException.java


Background
==========

This patch fixes a parallel make race condition during beaninfo generation.  I investigated this issue because I was seeing GitHub Actions failures on https://github.com/openjdk/jdk8u-dev/pull/452 and wanted to make sure they were not caused by the ZipFile backport.

This type of failure happens when generated source files are partially written to `$(JDK_OUTPUTDIR)/gensrc`.  The failing invocation is `javadoc` (i.e., `com.sun.tools.javadoc.Main`, i.e., the expansion of `$(NEW_JAVADOC)`).  `$(JDK_OUTPUTDIR)/gensrc` is listed in the `-sourcepath` javadoc argument.  Therefore javadoc tries to parse source files it finds while traversing `gensrc` and its subdirectories.

Other build rules write source files to `$(JDK_OUTPUTDIR)/gensrc`, and if any such source file is partially written, the `javadoc` sourcepath parsing fails with messages like:


d:\a\jdk8u-dev\jdk8u-dev\jdk\jdk\src\share\classes\java\nio\channels\spi\AbstractSelectableChannel.java:33: error: cannot access IllegalSelectorException
import java.nio.channels.IllegalSelectorException;
                        ^
  bad source file: d:\a\jdk8u-dev\jdk8u-dev\jdk\build\windows-x64\jdk\gensrc\java\nio\channels\IllegalSelectorException.java
    file does not contain class java.nio.channels.IllegalSelectorException
    Please remove or make sure it appears in the correct subdirectory of the sourcepath.


which occurred in
https://github.com/gnu-andrew/jdk8u-dev/actions/runs/7979614913/job/21787468883.

So far I have seen this type of error happen twice on Windows GitHub Actions runs, with the same error message.  Likely the specific error message can be different depending on which file is being written under `gensrc` when `javadoc` reads it.  I have been able to replicate this type of error message locally by manipulating relative timing on a `Fedora 38 x86_64` machine with this patch:


diff --git a/jdk/make/scripts/genExceptions.sh b/jdk/make/scripts/genExceptions.sh
index 66ef805691..81108ebb8e 100644
--- a/jdk/make/scripts/genExceptions.sh
+++ b/jdk/make/scripts/genExceptions.sh
@@ -43,6 +43,7 @@ gen() {
 
   $SH ${SCRIPTS}/addNotices.sh "$COPYRIGHT_YEARS" > $out
 
+  sleep 0.25
   cat >>$out <<__END__
 
 // -- This file was mechanically generated: Do not edit! -- //


To accelerate testing I added this patch too:


diff --git a/jdk/make/gensrc/GensrcSwing.gmk b/jdk/make/gensrc/GensrcSwing.gmk
index ba73623ca0..dc68756f45 100644
--- a/jdk/make/gensrc/GensrcSwing.gmk
+++ b/jdk/make/gensrc/GensrcSwing.gmk
@@ -86,6 +86,7 @@ $(JDK_OUTPUTDIR)/gensrc_no_srczip/_the.generated_beaninfo: $(BEANS_SRC) \
         # Move the JTextComponent into its proper package directory.
 	$(MKDIR) -p $(JDK_OUTPUTDIR)/gensrc_no_srczip/javax/swing/text
 	$(MV) $(JDK_OUTPUTDIR)/gensrc_no_srczip/javax/swing/JTextComponentBeanInfo.java $(JDK_OUTPUTDIR)/gensrc_no_srczip/javax/swing/text/JTextComponentBeanInfo.java
+	success-here
 	$(TOUCH) $@
 
 # This file is the part of dt.jar


After a successful normal build, I applied that patch, then tested like this:


rm ./build/linux-x86_64-normal-server-release/jdk/gensrc_no_srczip/_the.generated_beaninfo
for i in `seq 1 100`
do
    echo === $i
    rm ./build/linux-x86_64-normal-server-release/jdk/gensrc/java/nio/_the.exceptions.dir
    make JOBS=4
done


Some runs get to "`success-here`":


=== 5
Building OpenJDK for target 'default' in configuration 'linux-x86_64-normal-server-release'

## Starting langtools
## Finished langtools (build time 00:00:00)

## Starting hotspot
## Finished hotspot (build time 00:00:00)

## Starting corba
## Finished corba (build time 00:00:00)

## Starting jaxp
## Finished jaxp (build time 00:00:00)

## Starting jaxws
## Finished jaxws (build time 00:00:01)

## Starting jdk
Generating exceptions classes
Generating beaninfo
gmake[2]: success-here: No such file or directory
gmake[2]: *** [gensrc/GensrcSwing.gmk:82: [...]/jdk8u-dev/build/linux-x86_64-normal-server-release/jdk/gensrc_no_srczip/_the.generated_beaninfo] Error 127
[...]


Others fail with "`bad source file`" type errors:


=== 1
Building OpenJDK for target 'default' in configuration 'linux-x86_64-normal-server-release'

## Starting langtools
## Finished langtools (build time 00:00:00)

## Starting hotspot
## Finished hotspot (build time 00:00:00)

## Starting corba
## Finished corba (build time 00:00:00)

## Starting jaxp
## Finished jaxp (build time 00:00:00)

## Starting jaxws
## Finished jaxws (build time 00:00:00)

## Starting jdk
Generating exceptions classes
Generating beaninfo
[...]/jdk8u-dev/jdk/src/share/classes/java/io/PrintStream.java:32: error: cannot access UnsupportedCharsetException
import java.nio.charset.UnsupportedCharsetException;
                       ^
  bad source file: [...]/jdk8u-dev/build/linux-x86_64-normal-server-release/jdk/gensrc/java/nio/charset/UnsupportedCharsetException.java
    file does not contain class java.nio.charset.UnsupportedCharsetException
    Please remove or make sure it appears in the correct subdirectory of the sourcepath.
[...]/jdk8u-dev/jdk/src/share/classes/java/io/ObjectOutputStream.java:37: error: cannot find symbol
import static java.io.ObjectStreamClass.processQueue;
^
  symbol:   static processQueue
  location: class


I also tested on `Windows x64`, and there, I can get the failure without the `genExceptions.sh` sleep.  I suspect this is just due to environment differences such as different storage speeds.  `OpenJDK 11` and newer are unaffected because they use different build logic in these areas, but not in easily-backportable ways.

Since I have only seen these errors with generated exception sources so far, I tried this minimal patch to improve the situation:


diff --git a/jdk/make/gensrc/GensrcSwing.gmk b/jdk/make/gensrc/GensrcSwing.gmk
index ba73623ca0..1ec983cf56 100644
--- a/jdk/make/gensrc/GensrcSwing.gmk
+++ b/jdk/make/gensrc/GensrcSwing.gmk
@@ -73,7 +73,7 @@ SWINGBEAN_DEBUG_FLAG = false
 $(JDK_OUTPUTDIR)/gensrc_no_srczip/_the.generated_beaninfo: $(BEANS_SRC) \
     $(JDK_OUTPUTDIR)/gensrc_no_srczip/javax/swing/SwingBeanInfoBase.java \
     $(JDK_OUTPUTDIR)/gensrc/sun/swing/BeanInfoUtils.java $(BUILD_TOOLS) \
-    | $(GENSRC_LOCALEDATAMETAINFO)
+    | $(GENSRC_LOCALEDATAMETAINFO) $(GENSRC_EXCEPTIONS)
 	$(ECHO) Generating beaninfo
 	$(MKDIR) -p $(JDK_OUTPUTDIR)/gensrc_no_srczip/javax/swing
 	$(JAVA) -Djava.awt.headless=true $(NEW_JAVADOC) \


That patch produces 100 straight clean runs on `Windows x64` and `Fedora 38 x86_64`.

I suspect it is possible for this to happen with other generated sources that end up under gensrc, though.  I could attempt to list all of the build-contributors to that directory in the order-only prerequisites, as was done for `$(GENSRC_LOCALEDATAMETAINFO)` in 43a9c5b0d3a8bf3a8182c2b4d2cdba05ded096cb.

However, with this pull request, I propose that a better approach is to remove `$(PATH_SEP)$(JDK_OUTPUTDIR)/gensrc` from the `-sourcepath` argument.

Only `GensrcSwing.gmk` writes to `$(JDK_OUTPUTDIR)/gensrc_no_srczip`.  And as far as I can tell, it does not write files anywhere else (though I have not come up with a good way to verify this).  Therefore, I tried on `Fedora 38 x86_64`:


[git at f63643b8a085ef51d3e3ca6788ce116bc769cb1d]
make clean
bash configure
make JOBS=64 WARNINGS_ARE_ERRORS=-Wno-register
mv build/linux-x86_64-normal-server-release/jdk/gensrc_no_srczip/ gensrc_no_srczip.1
make JOBS=64 WARNINGS_ARE_ERRORS=-Wno-register
diff -ur gensrc_no_srczip.1/ build/linux-x86_64-normal-server-release/jdk/gensrc_no_srczip/
echo $?
[result: 0]
[apply above patch]
rm -r build/linux-x86_64-normal-server-release/jdk/gensrc_no_srczip/
make JOBS=64 WARNINGS_ARE_ERRORS=-Wno-register
diff -ur gensrc_no_srczip.1/ build/linux-x86_64-normal-server-release/jdk/gensrc_no_srczip/
echo $?
[result: 0]


Similarly, on `Windows x64`:


make reconfigure
make clean
make JOBS=4
mv build/windows-x64/jdk/gensrc_no_srczip/ gensrc_no_srczip.1
make JOBS=4
diff -ur gensrc_no_srczip.1/ build/windows-x64/jdk/gensrc_no_srczip/
echo $?
[result: 0]
[apply above patch]
rm -r build/windows-x64/jdk/gensrc_no_srczip/
make JOBS=64
diff -ur gensrc_no_srczip.1/ build/windows-x64/jdk/gensrc_no_srczip/
echo $?
[result: 0]


Moreover, the contents of `gensrc_no_srczip.1` are identical between `Windows` and `Fedora`.

The patch in this pull request passes the aforementioned 100-cycle stress test on both the test systems I mentioned, with "`sleep 0.25`" in `genExceptions.sh`.

-------------

Commit messages:
 - GensrcSwing.gmk: Remove gensrc from javadoc source path

Changes: https://git.openjdk.org/jdk8u-dev/pull/465/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk8u-dev&pr=465&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8327440
  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk8u-dev/pull/465.diff
  Fetch: git fetch https://git.openjdk.org/jdk8u-dev.git pull/465/head:pull/465

PR: https://git.openjdk.org/jdk8u-dev/pull/465

More information about the jdk8u-dev mailing list