Possible open file leak in com.sun.tools.javac.file.JavacFileManager

[Jonathan Gibbons]

Andreas,

I investigated this using your test code, changing the location to use a 
search path given on the command line.

I see the effect are describing when the search path contains JAR files. 
There does not seem to be an issue when listing directories or system files.

You are right in that there is a missing 'close()', but I don't think it 
is in the list() method, as you suggested.  You don't give any 
significant amount of detail about the environment in which you are 
seeing this, although you do say "heavy use of compiling classes during 
runtime". That suggests to me that you are using the javax.tools API to 
run the compiler (as compared to command-line javac), which suggests 
that you may be setting up a file manager, as in your code example. If 
that is the case, a likely cause of your problems is that you are not 
closing any file managers created and used in your application.  When I 
modified my version of your code sample to close the file manager and 
run countOpenFiles() again, the number of open files reverted to the 
number at the beginning of the test.

You also note the change in behavior between JDK 8 and JDK 11. In JDK 9, 
javac was changed to use the new NIO-based APIs (java.nio.file.*) 
instead of the old File-based APIs (java.io.*). As part of this work, 
JAR files are now opened as NIO filesystems, and for performance 
reasons, they are held open until the file manager itself is closed. 
(Previously, javac had a custom implementation for handling ZIP files 
which did not rely on holding the JAR file open, meaning that you could 
get away with not bothering to close the file manager.)

So, you might try checking your code to ensure you are either reusing 
file managers, or else closing them when they are no longer required.

-- Jon

On 11/05/2018 06:24 AM, Andreas Fey wrote:
> Hi all,
>
> we maybe found a bug in the com.sun.tools.javac.file.JavacFileManager; 
> our tool makes heavy use of compiling classes during runtime, and 
> after switching from JDK 8 to 11, we noticed hundreds of open files 
> being created during list() and only closed when JVM exists. The 
> concerning lines of code are:
>
>  @Override @DefinedBy(Api.COMPILER)
>     public Iterable<JavaFileObject> list(Location location,
>                                          String packageName,
>                                          Set<JavaFileObject.Kind> kinds,
>                                          boolean recurse)
>         throws IOException
>     {
>         checkNotModuleOrientedLocation(location);
>         // validatePackageName(packageName);
>         nullCheck(packageName);
>         nullCheck(kinds);
>
>         Iterable<? extends Path> path = getLocationAsPaths(location);
>         if (path == null)
>             return List.nil();
>         RelativeDirectory subdirectory = 
> RelativeDirectory.forPackage(packageName);
>         ListBuffer<JavaFileObject> results = new ListBuffer<>();
>
>         for (Path directory : path) {
>             Container container = getContainer(directory);
>
>             container.list(directory, subdirectory, kinds, recurse, 
> results);
>         }
>
>         return results.toList();
>     }
>
> We think, a container.close() is missing in the for loop. Without 
> this, a filehandle is created for every container/directiry found 
> here. To test this, the following code snipped can be used, but the 
> location must be set property to find a valid path != null:
>
> @Test
> public void testFileManager( ) throws IOException
> {
> countOpenFiles( );
>
> final Set<javax.tools.JavaFileObject.Kind> kinds = new HashSet<>( );
> kinds.add( javax.tools.JavaFileObject.Kind.OTHER );
> kinds.add( javax.tools.JavaFileObject.Kind.SOURCE );
> kinds.add( javax.tools.JavaFileObject.Kind.CLASS );
> kinds.add( javax.tools.JavaFileObject.Kind.HTML );
>
> final StandardJavaFileManager sfm =
> ToolProvider.getSystemJavaCompiler( ).getStandardFileManager( null, 
> null, null );
> sfm.list( new JavaFileManager.Location( )
> {
> @Override
> public boolean isOutputLocation( )
> {
> return false;
> }
>
> @Override
> public String getName( )
> {
> return "CLASS_NAME";
> }
> }, "com", kinds, true );
>
> countOpenFiles( );
> }
>
> private void countOpenFiles( )
> {
> final String processName = 
> java.lang.management.ManagementFactory.getRuntimeMXBean( ).getName( );
> final long pid = Long.parseLong( processName.split( "@" )[ 0 ] );
> try
> {
> final Runtime rt = Runtime.getRuntime( );
> final Process pr = rt.exec( "lsof -p " + pid );
>
> int ctr = 0;
> final BufferedReader br = new BufferedReader(
> new InputStreamReader( pr.getInputStream( ) ) );
> while ( ( br.readLine( ) ) != null )
> ctr++;
> pr.waitFor( );
> pr.destroy( );
>
> System.out.println( "Open files: " + ctr );
> }
> catch ( final Exception e )
> {
> e.printStackTrace( );
> }
> }
>
> Can anybody confirm this?
> Best,
> Andreas