Problems creating *.obj.log files

[Schmelter, Ralf]

Hi,

while building openjdk on Windows, I've stumbled over a problem which leads to truncated *.obj.log files, which in turn leads to empty dependency files (which are generated from the contents of the *.obj.log files on Windows). In the end, the incremental build will not be reliable.

The reason seems to be this code from the ExecuteWithLog macro in common/MakeBase.gmk:
( $(strip $2) > >($(TEE) $(strip $1).log) 2> >($(TEE) $(strip $1).log >&2) || \
    ...

This executes the command given in $2 and tries to write stdout/stderr to the file given by $1 and to 'normal' stdout/stderr via process substitution. But it has a few problems.

The first problem occurs when the stdout-tee truncates the file, then it writes some of the input it got into the file and only after this the stderr-tee has run far enough to truncate the same file, which leads to the loss of at least a part of stdout.

Another problem appears when the command writes to stdout and stderr. Since each tee has its own file position, they would overwrite one another.

The third problem arises because when the command $2 has ended, the bash shell will not wait for the process substitutions (the two tee commands) to end and instead will process the next command. Here we try to copy the log file in case of an error. But the tee commands might not have written the whole input they have got, so not the whole content of the file might get copied.


In my case it seems to be the first problem which I see most. When doing a fresh build on my Windows machine (which has quite an aggressive virus scanner active), I get something like the following:
> find . -name "*.obj.log" -size 0 | wc -l
424

> find . -name "*.obj.log" -not -size 0 | wc -l
1603

Which means about 20% of the *.obj.log files are truncated.

When I replace the code above with:
  ( $(strip $2) > >($(TEE) $(strip $1).log) 2>&1 || \
    ...

I see no more truncated *.obj.log files. The cost is that stderr is redirected to stdout (since the >&2 of the second tee is missing). The last problem can be avoided by using something like:
  ($(RM) -f $(strip $1).log && $(strip $2) > >($(TEE) -a $(strip $1).log) 2> >($(TEE) -a $(strip $1).log >&2) || \
    ...

This also leads to no truncated files, but the relative order of stdout/stderr might be wrong.

If the third problem should be solved too, the only solution I've found was to redierect via >  $(strip $1).log  2>&1 to the file and then later use 'cat' to write the output to stdout (losing the distinction between stdout and stderr).

Best regards,
Ralf