Proposal of Outputting Unified Logging to TCP socket

[David Holmes]

I second Johan's suggestion of files backed by sockets here. I think 
this has been raised before and the view was that we keep UL focused on 
"files" and then let the files map to other kinds of output devices 
rather than trying to support them all directly in UL.

Thanks,
David

On 6/10/2022 2:15 am, Johan Sjölén wrote:
> On 2022-10-04 16:21, Koichi Sakata wrote:
>> Hello,
>>
>> I would like to propose outputting unified logging (UL) to the TCP 
>> socket. I know that this has been discussed before [1]. However, 
>> circumstances have changed. Asynchronous logging was added to UL. If 
>> we output logs to TCP with async mode, we will be able to solve some 
>> issues raised in the past discussion (e.g. lag of log() calls). So I 
>> will take up this subject again.
>>
>> This feature is useful, especially in the container environment. Take 
>> Kubernetes for example, there are many containers. It is recommended 
>> that containers send logs to standard output, but the mixture of Java 
>> application logs and UL logs in stdout is not handy to parse or search 
>> in a log aggregator like Elasticsearch. I believe that it's beneficial 
>> to deal with each type of log individually.
>>
>> When we have files of UL logs, we have another issue. That is where to 
>> persist those files. As you know, we lose logs when pods are removed. 
>> I would rather not use volumes like Persistent Volume because of the 
>> cost of using volumes and the effort of managing files. When using the 
>> volume, we need to determine the file size and the volume size and put 
>> many log files of containers in it.
>>
>> UL via TCP socket answers these issues. In the Kubernetes example, we 
>> use a log forwarder like Fluent Bit, Filebeat or something as a 
>> DaemonSet. It gets UL logs from the socket, gets application logs from 
>> stdout, and sends each log to corresponding log aggregators. As a 
>> result, we don't have UL log files and volumes to persist them 
>> anymore. This is premised on asynchronous logging. So the lag will not 
>> be a problem, I think.
>>
>> For these reasons, I would like to propose outputting UL logs to TCP 
>> inputs. It seems to be unnecessary for traditional java applications, 
>> but it's of great use for applications in container environments. This 
>> makes it easier to build and operate applications on Kubernetes. There 
>> is also a bonus that this is os independent.
>>
>> I will create a prototype of this proposal if it is worth trying.
>>
>> By the way, there is a network stream class that is used to send node 
>> information to Ideal Graph Visualizer via socket. I'm not sure whether 
>> we can divert it or not for this new feature.
>>
>> Best Regards,
>> Koichi Sakata
>>
>> [1] 
>> https://mail.openjdk.org/pipermail/hotspot-runtime-dev/2020-November/043221.html
> 
> Hi Koichi Sokata,
> 
> Thank you for your suggestion. I think that you're right, UL is in a 
> better position to accept TCP sockets as output devices now than it was 
> previously thanks to the introduction of asynchronous logging. However, 
> I am skeptical to the idea of adding these to UL. It creates a rather 
> large attack vector for us, it also creates a maintenance burden and 
> finally getting this right for everyone is probably close to impossible.
> 
> Please note that the Ideal Graph Visualizer is a debug tool, it is not 
> meant for use in production. Therefore, its requirements are very 
> different to UL, which needs to be robust.
> 
> My suggestion would be to use an external tool. The most basic of these 
> would look like this:
> 
> Process 0:
> $ run_tcp_server localhost 4040
> Process 1:
> $ mkfifo my.log
> $ tail -f my.log | netcat localhost 4040
> Process 2:
> $ java -Xlog:disable -Xlog::my.log
> 
> Where the initial -Xlog:disable disables any default configuration which 
> would output to stdout.
> 
> I am sure that there are more sophisticated tools which works 
> essentially like this.
> 
> Best regards,
> Johan Sjölén