Skip to content

in_tcp, in_syslog: fix message_length_limit dropping valid lines and repeating its log - #5624

Merged
Watson1978 merged 4 commits into
fluent:masterfrom
kenhys:in_tcp-quiet-message-length-limit-log
Nov 18, 2026
Merged

Watson1978 merged 4 commits into
fluent:masterfrom
kenhys:in_tcp-quiet-message-length-limit-log

Conversation

@kenhys

@kenhys kenhys commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Which issue(s) this PR fixes:
Fixes #5647

What this PR does / why we need it:

This PR fixes problems with message_length_limit in in_tcp and in_syslog. Two of them drop valid messages, and two are about the logs.

  1. in_tcp: a long message on one connection drops a valid message on another connection.

    Before: the flag to discard data until the next delimiter is shared by all connections. When a long message on one connection sets it, a valid message on another connection can be dropped, and the rest of the long message can be emitted as a record (in_tcp: message_length_limit makes another connection lose a line and emits the rest of the long message #5647).

    After: each connection has its own flag, the same as in_syslog.

  2. A delimiter split between reads drops the next message.

    Before: when the delimiter has two or more bytes, such as \r\n, clearing the buffer can remove the first part of a delimiter that is split between reads. Then the end of the long message is missed, and the next valid message is dropped. In in_syslog, this happens with the traditional frame.

    After: the last bytes that can be the first part of the delimiter are kept when the buffer is cleared, so the end of the long message is found.

  3. The info log for a buffer clear repeats.

    Before: when message_length_limit is set and a client sends data without a delimiter, an info log is written each time the buffer goes over the limit. The data until the next delimiter fills the buffer again, so the log repeats, and how often depends on the read size. For example, sending 64MiB without a newline to in_tcp with message_length_limit 4096 wrote about 4,200 info lines.

    After: the buffer clear is logged only once for one message.

  4. The head of the received data breaks fluentd's own log events.

    Before: the logs have the head of the received bytes as is. in_tcp writes it at info level, and in_syslog writes it at debug level. When the head has binary bytes, fluentd's own log event can not be made into JSON, and failed to emit fluentd's log event (JSON::GeneratorError) happens with <label @FLUENT_LOG>.

    After: the info logs of in_tcp show only the limit and the size, like in_syslog. The head is logged at debug level, and it is escaped with String#dump like the existing debug log of in_syslog for a message length that is not a number.

The tests wait for the log instead of sleeping where it matters, so they do not depend on how the server splits the reads.

Docs Changes:

Release Note:

in_tcp, in_syslog: fix message_length_limit dropping valid messages, log a buffer clear for message_length_limit only once per message, and log the head of the received data only at debug level as an escaped string

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Logging the head only at debug with String#dump looks good to me.

I found two cases where the new "log only once" check hides a clear that master logs. Please see the inline comments.

Comment thread lib/fluent/plugin/in_tcp.rb Resolved
Comment thread lib/fluent/plugin/in_syslog.rb Resolved
kenhys and others added 4 commits November 17, 2026 06:39
Before: the flag to discard data until the next delimiter is shared by
all connections. When a long message on one connection sets it, a
valid message on another connection can be dropped, and the rest of
the long message can be handled as a new message.

After: each connection has its own flag, the same as in_syslog.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
Before: when message_length_limit is set and the delimiter has two or
more bytes, such as "\r\n", clearing the buffer can remove the first
part of a delimiter that is split between reads. Then the end of the
long message is missed, so the next valid message is dropped. In
in_syslog, this happens with the traditional frame.

After: the last bytes that can be the first part of the delimiter are
kept when the buffer is cleared, so the end of the long message is
found.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
Before: the logs for message_length_limit have the head of the
received bytes as is. When the head has binary bytes, fluentd's own
log event can not be made into JSON, and "failed to emit fluentd's log
event" (JSON::GeneratorError) happens with <label @FLUENT_LOG>. in_tcp
writes the head in info logs, and in_syslog writes it in debug logs.

After: the head is escaped so that the log event can be made into
JSON. in_tcp shows only the limit and the size in info logs, and logs
the head at debug level like in_syslog.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
Before: when message_length_limit is set and a client sends data
without a delimiter, an info log is written each time the buffer goes
over the limit. The data until the next delimiter fills the buffer
again, so the log repeats, and how often depends on the read size.
In in_syslog, this happens with the traditional frame.

After: the buffer clear is logged only once for one message.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
@kenhys
kenhys force-pushed the in_tcp-quiet-message-length-limit-log branch from 1e7fc2d to c9bedcb Compare November 17, 2026 09:55
@Watson1978 Watson1978 added bug Something isn't working backport to v1.19 We will backport this fix to the LTS branch labels Nov 18, 2026
@Watson1978 Watson1978 added this to the v1.21.0 milestone Nov 18, 2026

@Watson1978 Watson1978 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. Both points from my previous review are fixed:

  • The discard flag of in_tcp is now kept for each connection (#5624 (comment)).
  • A delimiter split between reads is no longer cut by the clear (#5624 (comment)).

I also fed random streams with random read splits to both paths of in_tcp and to in_syslog (traditional). I used the delimiters ;, \r\n, abc, € and aa. The emitted records and the number of limit logs were as expected. With <label @FLUENT_LOG> and a JSON formatter, a binary head no longer causes failed to emit fluentd's log event.

There is one more case, and it is the same on master. A line of exactly message_length_limit bytes is still dropped when a read ends inside its \r\n, because the buffer then has the line and \r. This PR makes it better, because the next line is no longer dropped. So it can be a follow-up.

@Watson1978 Watson1978 changed the title in_tcp,in_syslog: quiet message length limit log in_tcp, in_syslog: fix message_length_limit dropping valid lines and repeating its log Nov 18, 2026
@Watson1978
Watson1978 merged commit cd1f8da into fluent:master Nov 18, 2026
34 of 35 checks passed
@kenhys
kenhys deleted the in_tcp-quiet-message-length-limit-log branch November 19, 2026 01:25
kenhys added a commit that referenced this pull request Nov 19, 2026
… valid lines and repeating its log (#5624) (#5653)

**Which issue(s) this PR fixes**:
Backport #5624
Fixes #5647

**What this PR does / why we need it**:

This PR fixes problems with `message_length_limit` in in_tcp and
in_syslog. Two of them drop valid messages, and two are about the logs.

1. in_tcp: a long message on one connection drops a valid message on
another connection.

Before: the flag to discard data until the next delimiter is shared by
all connections. When a long message on one connection sets it, a valid
message on another connection can be dropped, and the rest of the long
message can be emitted as a record (#5647).

   After: each connection has its own flag, the same as in_syslog.

2. A delimiter split between reads drops the next message.

Before: when the delimiter has two or more bytes, such as `\r\n`,
clearing the buffer can remove the first part of a delimiter that is
split between reads. Then the end of the long message is missed, and the
next valid message is dropped. In in_syslog, this happens with the
traditional frame.

After: the last bytes that can be the first part of the delimiter are
kept when the buffer is cleared, so the end of the long message is
found.

3. The info log for a buffer clear repeats.

Before: when `message_length_limit` is set and a client sends data
without a delimiter, an info log is written each time the buffer goes
over the limit. The data until the next delimiter fills the buffer
again, so the log repeats, and how often depends on the read size. For
example, sending 64MiB without a newline to in_tcp with
`message_length_limit 4096` wrote about 4,200 info lines.

   After: the buffer clear is logged only once for one message.

4. The head of the received data breaks fluentd's own log events.

Before: the logs have the head of the received bytes as is. in_tcp
writes it at info level, and in_syslog writes it at debug level. When
the head has binary bytes, fluentd's own log event can not be made into
JSON, and `failed to emit fluentd's log event` (`JSON::GeneratorError`)
happens with `<label @FLUENT_LOG>`.

After: the info logs of in_tcp show only the limit and the size, like
in_syslog. The head is logged at debug level, and it is escaped with
`String#dump` like the existing debug log of in_syslog for a message
length that is not a number.

The tests wait for the log instead of sleeping where it matters, so they
do not depend on how the server splits the reads.


**Docs Changes**:

**Release Note**:

in_tcp, in_syslog: fix `message_length_limit` dropping valid messages,
log a buffer clear for `message_length_limit` only once per message, and
log the head of the received data only at debug level as an escaped
string

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
Co-authored-by: Kentaro Hayashi <hayashi@clear-code.com>
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added the backported "backport to LTS" is done label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport to v1.19 We will backport this fix to the LTS branch backported "backport to LTS" is done bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

in_tcp: message_length_limit makes another connection lose a line and emits the rest of the long message

2 participants