Repository navigation
in_tcp, in_syslog: fix message_length_limit dropping valid lines and repeating its log - #5624
Conversation
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for this. Logging the head only at debug with String#dump looks good to me.
I found two cases where the new "log only once" check hides a clear that master logs. Please see the inline comments.
Before: the flag to discard data until the next delimiter is shared by all connections. When a long message on one connection sets it, a valid message on another connection can be dropped, and the rest of the long message can be handled as a new message. After: each connection has its own flag, the same as in_syslog. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
Before: when message_length_limit is set and the delimiter has two or more bytes, such as "\r\n", clearing the buffer can remove the first part of a delimiter that is split between reads. Then the end of the long message is missed, so the next valid message is dropped. In in_syslog, this happens with the traditional frame. After: the last bytes that can be the first part of the delimiter are kept when the buffer is cleared, so the end of the long message is found. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
Before: the logs for message_length_limit have the head of the received bytes as is. When the head has binary bytes, fluentd's own log event can not be made into JSON, and "failed to emit fluentd's log event" (JSON::GeneratorError) happens with <label @FLUENT_LOG>. in_tcp writes the head in info logs, and in_syslog writes it in debug logs. After: the head is escaped so that the log event can be made into JSON. in_tcp shows only the limit and the size in info logs, and logs the head at debug level like in_syslog. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
Before: when message_length_limit is set and a client sends data without a delimiter, an info log is written each time the buffer goes over the limit. The data until the next delimiter fills the buffer again, so the log repeats, and how often depends on the read size. In in_syslog, this happens with the traditional frame. After: the buffer clear is logged only once for one message. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
1e7fc2d to
c9bedcb
Compare
There was a problem hiding this comment.
Thanks for the update. Both points from my previous review are fixed:
- The discard flag of in_tcp is now kept for each connection (#5624 (comment)).
- A delimiter split between reads is no longer cut by the clear (#5624 (comment)).
I also fed random streams with random read splits to both paths of in_tcp and to in_syslog (traditional). I used the delimiters ;, \r\n, abc, € and aa. The emitted records and the number of limit logs were as expected. With <label @FLUENT_LOG> and a JSON formatter, a binary head no longer causes failed to emit fluentd's log event.
There is one more case, and it is the same on master. A line of exactly message_length_limit bytes is still dropped when a read ends inside its \r\n, because the buffer then has the line and \r. This PR makes it better, because the next line is no longer dropped. So it can be a follow-up.
… valid lines and repeating its log (#5624) (#5653) **Which issue(s) this PR fixes**: Backport #5624 Fixes #5647 **What this PR does / why we need it**: This PR fixes problems with `message_length_limit` in in_tcp and in_syslog. Two of them drop valid messages, and two are about the logs. 1. in_tcp: a long message on one connection drops a valid message on another connection. Before: the flag to discard data until the next delimiter is shared by all connections. When a long message on one connection sets it, a valid message on another connection can be dropped, and the rest of the long message can be emitted as a record (#5647). After: each connection has its own flag, the same as in_syslog. 2. A delimiter split between reads drops the next message. Before: when the delimiter has two or more bytes, such as `\r\n`, clearing the buffer can remove the first part of a delimiter that is split between reads. Then the end of the long message is missed, and the next valid message is dropped. In in_syslog, this happens with the traditional frame. After: the last bytes that can be the first part of the delimiter are kept when the buffer is cleared, so the end of the long message is found. 3. The info log for a buffer clear repeats. Before: when `message_length_limit` is set and a client sends data without a delimiter, an info log is written each time the buffer goes over the limit. The data until the next delimiter fills the buffer again, so the log repeats, and how often depends on the read size. For example, sending 64MiB without a newline to in_tcp with `message_length_limit 4096` wrote about 4,200 info lines. After: the buffer clear is logged only once for one message. 4. The head of the received data breaks fluentd's own log events. Before: the logs have the head of the received bytes as is. in_tcp writes it at info level, and in_syslog writes it at debug level. When the head has binary bytes, fluentd's own log event can not be made into JSON, and `failed to emit fluentd's log event` (`JSON::GeneratorError`) happens with `<label @FLUENT_LOG>`. After: the info logs of in_tcp show only the limit and the size, like in_syslog. The head is logged at debug level, and it is escaped with `String#dump` like the existing debug log of in_syslog for a message length that is not a number. The tests wait for the log instead of sleeping where it matters, so they do not depend on how the server splits the reads. **Docs Changes**: **Release Note**: in_tcp, in_syslog: fix `message_length_limit` dropping valid messages, log a buffer clear for `message_length_limit` only once per message, and log the head of the received data only at debug level as an escaped string Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com> Signed-off-by: Shizuo Fujita <fujita@clear-code.com> Co-authored-by: Kentaro Hayashi <hayashi@clear-code.com> Co-authored-by: Claude <noreply@anthropic.com>
Which issue(s) this PR fixes:
Fixes #5647
What this PR does / why we need it:
This PR fixes problems with
message_length_limitin in_tcp and in_syslog. Two of them drop valid messages, and two are about the logs.in_tcp: a long message on one connection drops a valid message on another connection.
Before: the flag to discard data until the next delimiter is shared by all connections. When a long message on one connection sets it, a valid message on another connection can be dropped, and the rest of the long message can be emitted as a record (in_tcp: message_length_limit makes another connection lose a line and emits the rest of the long message #5647).
After: each connection has its own flag, the same as in_syslog.
A delimiter split between reads drops the next message.
Before: when the delimiter has two or more bytes, such as
\r\n, clearing the buffer can remove the first part of a delimiter that is split between reads. Then the end of the long message is missed, and the next valid message is dropped. In in_syslog, this happens with the traditional frame.After: the last bytes that can be the first part of the delimiter are kept when the buffer is cleared, so the end of the long message is found.
The info log for a buffer clear repeats.
Before: when
message_length_limitis set and a client sends data without a delimiter, an info log is written each time the buffer goes over the limit. The data until the next delimiter fills the buffer again, so the log repeats, and how often depends on the read size. For example, sending 64MiB without a newline to in_tcp withmessage_length_limit 4096wrote about 4,200 info lines.After: the buffer clear is logged only once for one message.
The head of the received data breaks fluentd's own log events.
Before: the logs have the head of the received bytes as is. in_tcp writes it at info level, and in_syslog writes it at debug level. When the head has binary bytes, fluentd's own log event can not be made into JSON, and
failed to emit fluentd's log event(JSON::GeneratorError) happens with<label @FLUENT_LOG>.After: the info logs of in_tcp show only the limit and the size, like in_syslog. The head is logged at debug level, and it is escaped with
String#dumplike the existing debug log of in_syslog for a message length that is not a number.The tests wait for the log instead of sleeping where it matters, so they do not depend on how the server splits the reads.
Docs Changes:
Release Note:
in_tcp, in_syslog: fix
message_length_limitdropping valid messages, log a buffer clear formessage_length_limitonly once per message, and log the head of the received data only at debug level as an escaped string