Long log lines are not handled correctly

[crassirostris]

Couldn't fine an existing issue, so I'll start this one

Currently when a container logs a line that is longer than ~4KB, it is being split into multiple log entries in the container log. The information about this split in then dropped, AFAIU because current log format is not extensible. Therefore a 8KB line will be indistinguishable from two 4KB lines, which will break people who depend on the long lines longer than 4KB.

There should be at least enough information in the logs to reconstruct the original log stream, even if the lines are long. It seems like this is impossible to do without changing the log format though.

/cc @piosz @fgrzadkowski @Random-Liu @yujuhong

[Random-Liu]

There should be at least enough information in the logs to reconstruct the original log stream, even if the lines are long. It seems like this is impossible to do without changing the log format though.

Yeah. If we change the log format to something like docker json log, or add some other prefix to indicate partial lines, will fluentd be able to handle it?

[crassirostris]

Yeah. If we change the log format to something like docker json log, or add some other prefix to indicate partial lines, will fluentd be able to handle it?

Yes, it will be possible. There's no such plugin yet, but at least it will be possible to write one

Edit: JSON is more convent than a custom format, so if it's not a serious performance issue, I'd prefer it

[cpuguy83]

json is pretty bad at this.
Even with tons of optimizations, the json logger in Docker is requires 3 allocations per log entry, and on top of that just the slow encoding/decoding speed of json in general.

Might I suggest something like https://github.com/moby/moby/tree/master/api/types/plugins/logdriver as an alternative?

[crassirostris]

Problem is, json format is highly extensible and widely understood by logging agents like fluentd or logstash, it works there out of the box and this is an important advantage

Thanks for referring to the pb log driver though, that's interesting!

[Random-Liu]

We've switched to the new CRI log format this release. #421

Future work will be based on our decision about the logging format.

Move out of the milestone.