Proposal of Outputting Unified Logging to TCP socket

[Johan Sjölén]

On 2022-10-04 16:21, Koichi Sakata wrote:
> Hello,
> 
> I would like to propose outputting unified logging (UL) to the TCP 
> socket. I know that this has been discussed before [1]. However, 
> circumstances have changed. Asynchronous logging was added to UL. If we 
> output logs to TCP with async mode, we will be able to solve some issues 
> raised in the past discussion (e.g. lag of log() calls). So I will take 
> up this subject again.
> 
> This feature is useful, especially in the container environment. Take 
> Kubernetes for example, there are many containers. It is recommended 
> that containers send logs to standard output, but the mixture of Java 
> application logs and UL logs in stdout is not handy to parse or search 
> in a log aggregator like Elasticsearch. I believe that it's beneficial 
> to deal with each type of log individually.
> 
> When we have files of UL logs, we have another issue. That is where to 
> persist those files. As you know, we lose logs when pods are removed. I 
> would rather not use volumes like Persistent Volume because of the cost 
> of using volumes and the effort of managing files. When using the 
> volume, we need to determine the file size and the volume size and put 
> many log files of containers in it.
> 
> UL via TCP socket answers these issues. In the Kubernetes example, we 
> use a log forwarder like Fluent Bit, Filebeat or something as a 
> DaemonSet. It gets UL logs from the socket, gets application logs from 
> stdout, and sends each log to corresponding log aggregators. As a 
> result, we don't have UL log files and volumes to persist them anymore. 
> This is premised on asynchronous logging. So the lag will not be a 
> problem, I think.
> 
> For these reasons, I would like to propose outputting UL logs to TCP 
> inputs. It seems to be unnecessary for traditional java applications, 
> but it's of great use for applications in container environments. This 
> makes it easier to build and operate applications on Kubernetes. There 
> is also a bonus that this is os independent.
> 
> I will create a prototype of this proposal if it is worth trying.
> 
> By the way, there is a network stream class that is used to send node 
> information to Ideal Graph Visualizer via socket. I'm not sure whether 
> we can divert it or not for this new feature.
> 
> Best Regards,
> Koichi Sakata
> 
> [1] 
> https://mail.openjdk.org/pipermail/hotspot-runtime-dev/2020-November/043221.html 
> 

Hi Koichi Sokata,

Thank you for your suggestion. I think that you're right, UL is in a 
better position to accept TCP sockets as output devices now than it was 
previously thanks to the introduction of asynchronous logging. However, 
I am skeptical to the idea of adding these to UL. It creates a rather 
large attack vector for us, it also creates a maintenance burden and 
finally getting this right for everyone is probably close to impossible.

Please note that the Ideal Graph Visualizer is a debug tool, it is not 
meant for use in production. Therefore, its requirements are very 
different to UL, which needs to be robust.

My suggestion would be to use an external tool. The most basic of these 
would look like this:

Process 0:
$ run_tcp_server localhost 4040
Process 1:
$ mkfifo my.log
$ tail -f my.log | netcat localhost 4040
Process 2:
$ java -Xlog:disable -Xlog::my.log

Where the initial -Xlog:disable disables any default configuration which 
would output to stdout.

I am sure that there are more sophisticated tools which works 
essentially like this.

Best regards,
Johan Sjölén