Unified Logging for network

[Yasumasa Suenaga]

Hi,

I agree this proposal might occur performance issue. However I think it is the responsibility of the user.
If this proposal is implemented, I think it would be transferred to local log shipper process (fluentd, logstash on 127.0.0.1) in most case because HotSpot does not send log with JSON. And also log shipper may support message buffering and message queue persistence.
We can avoid (in part of) performance/reliability issues with log shipper.

Even if current implementation, performance issues is occurs when the disk is very slow (e.g. storage is broken).


Cheers,

Yasumasa


On 2020/11/03 6:31, Thomas Stüfe wrote:
> Hi Ioi,
> 
> I dimly remember proposals like this from the past. Main problem I see is
> how large would you dimension the buffer, and what do you do if the buffer
> cannot be drained rapidly enough. Discard log output? Hold? The former
> sounds bad, the latter negates the advantages of such a buffer.
> 
> Then, access to such a buffer would probably have to be synchronized,
> whereas today AFAIK the log calls do not have to be.
> 
> Cheers, Thomas
> 
> On Mon 2. Nov 2020 at 22:18, Ioi Lam <ioi.lam at oracle.com> wrote:
> 
>> For performance, maybe the implementation can log into a memory buffer,
>> and use a worker thread to send the output over the network? That way we
>> can minimize the overhead per log_xxx() call.
>>
>> I agree that using "-Xlog:foo=debug:network=xyz.com:1234" would be quite
>> handy when you have lots of containers. You don't need to enable remote
>> access to the container's file system just to get to the log file.
>>
>> Thanks
>> - Ioi
>>
>> On 11/2/20 11:10 AM, Kirk Pepperdine wrote:
>>> Hi Thomas,
>>>
>>> I appreciate Yasumasa’s desire to be able to redirect UL output to
>> somewhere other than… I also appreciate that the highly granular nature of
>> how UL messages are currently structure can be and indeed are an issue.
>> That said, I’d also like the ability to push the data to some where other
>> than a file on disk.
>>>
>>> To the point of granularity, UL might benefit from some message
>> coarsening. This might also help in with other logging related performance
>> issues that I’ve noted here and there. Quite frankly dealing with logs in
>> containers isn’t a wonderful experience. And while I firmly believe that
>> there is more that containers can do to ease this, being able to redirect
>> output to something other than a log file does feel like it would be
>> helpful. That said, I’m also concerned about the potential performance
>> impacts but I think for this things that one would generally log, this
>> should be minimal.
>>>
>>> Kind regards,
>>> Kirk Pepperdine
>>>
>>>
>>>> On Nov 2, 2020, at 4:26 AM, Thomas Stüfe <thomas.stuefe at gmail.com>
>> wrote:
>>>>
>>>> Hi Yasumasa,
>>>>
>>>> one problem I see is that this could introduce a surprising amount of
>> lag
>>>> into log() calls which do look inconspicuous, thereby distorting timing
>>>> behavior or even create timeout effects. We already have that problem
>> now
>>>> to some degree when logging to network shares.
>>>>
>>>> Another thing, log output can be very fine granular, which would create
>> a
>>>> lot of network traffic.
>>>>
>>>> Such an addition may also open some security questions.
>>>>
>>>>   From a more philosophical standpoint, I like the "do one thing and do
>> it
>>>> right" Unix way and this seems more like something an outside tool
>> should
>>>> be doing. Which could also aggregate log output better. But I admit that
>>>> argument is weak.
>>>>
>>>> Cheers, Thomas
>>>>
>>>>
>>>>
>>>> On Mon, Nov 2, 2020 at 12:21 PM Yasumasa Suenaga <
>> suenaga at oss.nttdata.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> We need to out UL to stdout and/or file. If we can out it to TCP
>> socket, I
>>>>> think it is useful.
>>>>>
>>>>> For example, some system gather all logs to document oriented databases
>>>>> (e.g. Elasticsearch) and/or cloud monitoring platform (e.g.
>> CloudWatch). If
>>>>> HotSpot can out UL to TCP socket, we can send all logs to them via TCP
>>>>> input plugin (Fluentd, Logstash).
>>>>>
>>>>> I think it is useful for container platform. What do you think?
>>>>> If it is worth to work, I will add CSR and JBS ticket, and also will
>>>>> create patch.
>>>>>
>>
>>