Skip to content

output: Pass update_retry as a keyword in the secondary rollback_write - #5636

Open
lbarrosby wants to merge 3 commits into
fluent:masterfrom
lbarrosby:fix-secondary-rollback-write
Open

lbarrosby wants to merge 3 commits into
fluent:masterfrom
lbarrosby:fix-secondary-rollback-write

Conversation

@lbarrosby

Copy link
Copy Markdown
Contributor

Which issue(s) this PR fixes:
Fixes #5633

What this PR does / why we need it:

acts_as_secondary defines rollback_write on the secondary so that it delegates to the primary, but it passed update_retry as a positional argument. Output#rollback_write(chunk_id, update_retry: true) has taken it as a keyword since 59a47a1, so the delegation always raised ArgumentError: wrong number of arguments (given 2, expected 1).

A delayed commit output used as <secondary>, such as out_forward with require_ack_response true, calls rollback_write when the ack times out. The error was logged as unexpected error while receiving ack. The chunk was not taken back, so it was not sent again.

This passes update_retry: update_retry, as the issue suggests.

Tests:

  • New data-driven test in test_output_as_buffered_secondary.rb. The secondary dequeues a chunk and calls rollback_write with update_retry: false and with the default. The chunk must go back to the queue, and update_retry_state must be called only with the default. On master both cases raise the ArgumentError.
  • test/plugin/test_output_as_buffered_secondary.rb 15 tests, test_output.rb 141 tests, test_output_as_buffered_retries.rb 14 tests: 0 failures.
  • The reproduction from the issue (primary port closed, secondary to a server that never acks), 22 s:
master: silent server received 1134 bytes, 9 x "unexpected error while receiving ack ... ArgumentError"
branch: silent server received 3285 bytes, no error

Docs Changes:
None.

Release Note:
Fix rollback_write on a <secondary> output, which always raised ArgumentError.

acts_as_secondary defines rollback_write to delegate to the primary
with rollback_write(chunk_id, update_retry). Output#rollback_write has
taken update_retry as a keyword argument since v0.14.25, so the call
always raised ArgumentError. A delayed commit output used as
<secondary>, such as out_forward with require_ack_response, could not
roll back a chunk after an ack timeout, and the chunk was not sent
again.

Fixes fluent#5633

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@Watson1978 Watson1978 added the bug Something isn't working label Nov 16, 2026

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fix and the clear reproduction. Passing update_retry as a keyword is correct, and the new test fails on master as described.

In one configuration, making this rollback work changes behavior that works on master. If the secondary is out_forward with heartbeat_type none and its destination never acks, the retry no longer reaches retry_timeout or retry_max_times. Fluentd then stays on the secondary, even after the primary recovers. Details are in the inline comment. I also left a note on the new test.

Comment thread lib/fluent/plugin/output.rb Outdated
singleton_class.module_eval do
define_method(:commit_write){ |chunk_id| @primary_instance.commit_write(chunk_id, delayed: delayed_commit, secondary: true) }
define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry) }
define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry: update_retry) }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: This fix changes how the retry ends in this configuration:

  • The secondary is out_forward with require_ack_response true and heartbeat_type none.
  • Its destination accepts data but never acks.
  • The secondary's ack_response_timeout is shorter than the primary's delayed_commit_timeout. This holds with the default values when the primary is also out_forward with require_ack_response: 190 seconds against 192 seconds.

On master, the delegation raises ArgumentError before the primary's rollback_write runs. So the chunk's entry stays in the primary's @dequeued_chunks. After the primary's delayed_commit_timeout, try_rollback_write takes the chunk back and calls update_retry_state. When the retry reaches its limit, master drops the queued chunks and clears the retry state. Later chunks then go to the primary.

With this PR, the ack timeout calls rollback_write(chunk_id, update_retry: false) on the primary. It returns the chunk to the queue and removes its entry, but it does not update the retry state. Nothing else updates it in this configuration. So retry_timeout and retry_max_times are never reached, and Fluentd keeps using the secondary, even after the primary recovers. With the default heartbeat_type transport, this does not happen. The disabled node raises NoNodesAvailable, and that error moves the retry forward.

update_retry: false comes from #1665, so that an ack failure on the primary does not count as a flush error. In the secondary phase, though, the retry state already exists. If the secondary never succeeds, only the retry limit ends that phase.

I ran Fluentd for 25 seconds with the configuration below. Nothing listens on port 24224 for the first 6 seconds. Then a server starts there and acks every chunk. The server on port 24225 reads the data but never acks.

<source>
  @type sample
  tag test
  size 1
  rate 1
</source>
<match test>
  @type forward
  heartbeat_type none
  require_ack_response true
  ack_response_timeout 1
  <server>
    host 127.0.0.1
    port 24224
  </server>
  <buffer>
    @type memory
    flush_interval 1s
    retry_type periodic
    retry_wait 1s
    retry_timeout 4s
    retry_secondary_threshold 0.25
  </buffer>
  <secondary>
    @type forward
    heartbeat_type none
    require_ack_response true
    ack_response_timeout 1
    <server>
      host 127.0.0.1
      port 24225
    </server>
  </secondary>
</match>
  • master (d2c7287): The retry ends, and chunks go to the primary from about 9 seconds after start. The primary received 5 chunks.
  • This PR: The primary received nothing. The secondary kept receiving the same 6 chunks, 24 sends in total.

I also checked the retry state with a script that runs the same plugins directly. With this PR, retry.steps stays at 0. With retry_type exponential_backoff and retry_max_times 2, master clears the queue after 2 steps. This PR is still at 0 steps after 16 seconds.

Could you keep retry_timeout and retry_max_times effective in this configuration, without changing how an ack failure on the primary is handled? Please add a test that goes through out_forward's ack timeout on the secondary, and check that each limit still ends the retry. The new test calls rollback_write directly, so it does not cover this.

Comment on lines +461 to +473
priconf = config_element('buffer','tag',{'flush_interval' => 1, 'retry_type' => :periodic, 'retry_wait' => 3, 'retry_timeout' => 60, 'retry_randomize' => false})
secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'})
@i.configure(config_element('ROOT','',{},[priconf,secconf]))
@i.register(:prefer_buffered_processing){ true }
@i.register(:prefer_delayed_commit){ false }
@i.register(:format){|tag,time,record| [tag,time.to_i,record].to_json + "\n" }
@i.register(:write){|chunk| raise "yay, your #write must fail" }
@i.secondary.register(:prefer_delayed_commit){ true }
@i.start
@i.after_start

@i.interrupt_flushes

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHOULD: This test leaves the flush thread running. interrupt_flushes stops automatic enqueueing and the timeout rollback, but the flush thread still calls try_flush. It wakes up about 1 second after start. If the test thread is that slow, the flush thread can take the rolled-back chunk and call the failing write. It then calls update_retry_state with three arguments. The mock rejects that call, and the flush thread dies. Teardown then loops in submit_flush_all forever, because @retry is still nil and the chunk is still queued.

I reproduced this by adding sleep 4 right after rollback_write. The update_retry false case failed, and the process hung until timeout killed it.

The test calls rollback_write directly, so it does not need the flush thread or the failing write. With the change below, it passed with the same sleep 4. It still fails on master, and also when the delegation ignores update_retry. queued_chunks_limit_size defaults to flush_thread_count, so it has to be set here. The update_retry false case also gets about 1 second faster, because teardown no longer flushes the chunk through the failing write.

Suggested change
priconf = config_element('buffer','tag',{'flush_interval' => 1, 'retry_type' => :periodic, 'retry_wait' => 3, 'retry_timeout' => 60, 'retry_randomize' => false})
secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'})
@i.configure(config_element('ROOT','',{},[priconf,secconf]))
@i.register(:prefer_buffered_processing){ true }
@i.register(:prefer_delayed_commit){ false }
@i.register(:format){|tag,time,record| [tag,time.to_i,record].to_json + "\n" }
@i.register(:write){|chunk| raise "yay, your #write must fail" }
@i.secondary.register(:prefer_delayed_commit){ true }
@i.start
@i.after_start
@i.interrupt_flushes
priconf = config_element('buffer','tag',{'flush_thread_count' => 0, 'queued_chunks_limit_size' => 1})
secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'})
@i.configure(config_element('ROOT','',{},[priconf,secconf]))
@i.register(:prefer_buffered_processing){ true }
@i.start
@i.after_start

A secondary runs while the primary is retrying. With out_forward as the
secondary, an ack timeout rolls the chunk back with update_retry: false,
so retry_timeout and retry_max_times were never reached when the
secondary never acked, and Fluentd stayed on the secondary even after
the primary recovered.

A rollback through the secondary now always updates the retry state of
the primary. An ack failure on the primary still does not.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@lbarrosby

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed case. Pushed 2bd73f2.

Retry limit with out_forward as the secondary. The rollback that the secondary delegates to the primary now always updates the retry state, even when the plugin passes update_retry: false. In the secondary phase the retry state already exists, and only its limit ends that phase, as you wrote. The primary's own rollback_write is unchanged, so an ack failure on the primary still does not count as a flush error. On master the same update happens later, through try_rollback_write after the primary's delayed_commit_timeout.

I ran your configuration for 25 seconds, with a fluentd in_forward started on port 24224 after 6 seconds and a server on 24225 that reads and never acks:

  • master (d2c7287): the limit was reached, 7 records reached the primary, 13 connections to the secondary.
  • previous head (2200f8a): 0 records reached the primary, 56 connections to the secondary.
  • this commit: the limit was reached at about 4 seconds, 23 records reached the primary, 4 connections to the secondary.

New test in test_out_forward.rb, as secondary with a destination that never acks. The primary is out_forward to a closed port, and the secondary is out_forward with heartbeat_type none, require_ack_response true and ack_response_timeout 1s to a TCP server that never acks. One case uses exponential_backoff with retry_max_times 2, and the other uses periodic with retry_timeout 4s. Each waits until the retry state is cleared and the buffer is empty. Both pass here and on master, and time out on 2200f8a.

Unit test. Applied your suggestion: flush_thread_count 0 and queued_chunks_limit_size 1, without the failing write or interrupt_flushes. With the change above, both the update_retry: false case and the default case now expect one update_retry_state call, and the name says so. It still fails on master with ArgumentError.

test_output_as_buffered_secondary.rb, test_output_as_buffered_retries.rb, test_out_forward.rb, test_output.rb and test_output_as_buffered.rb pass. test_out_secondary_file.rb has 3 encoding failures here, and they are the same on master.

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the update and the measurements. Both points from my last review are addressed. The retry limits now end the secondary phase, and an ack failure on the primary is handled as before. The new out_forward test fails on 2200f8a and passes here and on master. The unit test now passes even when the test thread stalls for 4 seconds after rollback_write.

However, my last request did not consider a secondary rollback that comes after the retry has ended. Such a rollback can now start a new retry. Then the queue can be dropped even though the primary has recovered. Details are in the inline comment.

Comment thread lib/fluent/plugin/output.rb Outdated
# A secondary runs while the primary is already retrying, and only the retry limit
# ends that phase if the secondary never succeeds. So a rollback from the secondary
# always updates the retry state, even when the plugin passes update_retry: false.
define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry: true) }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: A secondary rollback can now start a new retry after the previous retry has ended. My last request did not consider this case, sorry.

A chunk can still be waiting for its ack on the secondary when the retry ends. The retry ends when another chunk succeeds through the secondary, or when the retry limit clears the queue. When the waiting chunk's ack times out, this delegation passes update_retry: true. update_retry_state finds no retry state, so it creates a new one.

The new retry may switch to the secondary before its first flush. Then that flush goes to the secondary without trying the recovered primary. In the example below, the switch comes 1.2 seconds after the retry starts (2s × 3 × 0.2), and the first flush comes after 2 seconds. If the secondary still does not ack, the limit clears the queue again. With a memory buffer, those chunks are lost.

In this reproduction, master also creates a new retry, but only after the primary's delayed_commit_timeout, through try_rollback_write. On 2200f8a, this rollback passed update_retry: false, so it returned the chunk without touching the retry state.

I reproduced it with the out_forward test driver and this configuration:

heartbeat_type none
require_ack_response true
ack_response_timeout 60s
<server>
  host 127.0.0.1
  port 24224
</server>
<buffer tag>
  @type memory
  flush_mode immediate
  flush_thread_interval 0.05
  flush_thread_burst_interval 0.05
  queued_chunks_limit_size 20
  flush_at_shutdown false
  retry_type periodic
  retry_wait 2s
  retry_max_times 3
  retry_timeout 30s
  retry_randomize false
  retry_secondary_threshold 0.2
  delayed_commit_timeout 60s
</buffer>
<secondary>
  @type forward
  heartbeat_type none
  require_ack_response true
  ack_response_timeout 1s
  <server>
    host 127.0.0.1
    port 24225
  </server>
</secondary>

The server on port 24225 acks chunks tagged good, but never acks slow. I fed slow and good while nothing listened on port 24224. After good succeeded through the secondary, I started a server on port 24224 that acks every chunk. I waited until the ack timeout of slow was handled, then fed fresh and watched for 30 seconds.

  • master: fresh reached the primary. slow was still waiting for the 60-second timeout.
  • 2200f8a: slow and fresh reached the primary.
  • This PR: Neither reached the primary. Both went to the secondary again and were dropped at the limit.

The configuration from my last comment shows it too, but the result depends on whether a chunk is still waiting for its ack when the retry ends. I ran it 6 times on each version, for 25 seconds each.

  • master: The primary received 3 to 11 chunks. In 3 runs, the secondary was no longer used after 15 seconds.
  • This PR: The primary received 2 to 4 chunks. In all runs, the secondary was still used after 23 seconds.

For a secondary rollback with update_retry: false, could you keep updating the retry while it exists, but not create a new one when none remains? Please keep the current behavior for update_retry: true and for ack failures on the primary. The check and the update need to happen together under the primary's @retry_mutex. If the delegation checks the retry first, another chunk's success can clear it before the update. I confirmed this window by pausing a prototype at that point. Apart from that window, the prototype went back to the primary in both cases above, and your out_forward test still passed.

Please add a test where a chunk's ack times out on the secondary after another chunk succeeded through it. Wait until that rollback is handled. Then check that no new retry was created and that the next chunk goes to the primary. The update_retry false case of the unit test then needs an active retry state.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that case makes sense. Pushed 0440151.

  • The secondary's rollback_write passes update_retry through to the primary again. With update_retry: false, after the chunk is taken back, it calls update_retry_state(chunk_id, false, create_retry: false) on the primary.
  • update_retry_state got a create_retry: keyword, true by default. With false, it returns without changing anything when @retry is nil. The check and the update run in the same @retry_mutex.synchronize block, so another chunk's success cannot clear the retry between them.
  • update_retry: true and the primary's own rollback_write are unchanged, so an ack failure on the primary is handled as before.

Tests:

  • New out_forward test as secondary after the retry has ended. The secondary server acks good but never slow, with ack_response_timeout 3s on the secondary and your buffer settings. It feeds slow and good while the primary port is closed, waits until both reached the secondary and the retry is cleared, then starts a primary server that acks everything. After the rollback of slow (rollback_count goes up), it checks that retry is nil, feeds fresh, and waits until both slow and fresh reach the primary. On 2bd73f2 it fails with a new PeriodicRetry after the rollback.
  • The unit test now has four cases: update_retry: false and the default, each with and without an active retry. It checks retry_count and whether a retry exists. update_retry false without a retry fails on 2bd73f2.
  • Your earlier out_forward test (as secondary with a destination that never acks) still passes.

test_output_as_buffered_secondary.rb (17 tests), test_output_as_buffered_retries.rb (14), test_output.rb (141), test_out_forward.rb (81, 3 omissions) and test_output_as_buffered.rb (56) pass with 0 failures.

…try has ended

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the update. The case from my last review is fixed. When a secondary ack timeout returns a chunk with update_retry: false after the retry has ended, no new retry is created. The check for an existing retry and the update now happen together under @retry_mutex. Your new out_forward test fails on 2bd73f2 and passes on 0440151 on Linux.

I am leaving this review as a comment this time. Each round has found another problem with a late ack timeout, and my last request caused one of them. I would like to hear your view on the scope before asking for another change. The first inline comment describes the two cases I reproduced.

Separately, the out_forward retry-limit test added in 2bd73f2 timed out in all five Windows jobs of the test matrix. The second inline comment has the details.

Comment on lines +261 to +262
rolled_back = @primary_instance.rollback_write(chunk_id, update_retry: update_retry)
@primary_instance.update_retry_state(chunk_id, false, create_retry: false) if rolled_back && !update_retry

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two cases remain with an ack timeout that arrives late on the secondary. The callback only has the chunk ID. So it cannot tell which send, or which retry, the timeout belongs to.

  1. The callback can step a different retry. I reproduced this with test outputs and a frozen clock. good succeeded through the secondary while slow was still waiting for its ack, so the retry ended. Then the primary failed again, and a new retry started. The timeout for slow took the chunk back before the core did, and line 262 stepped the new retry from 1 to 2. With retry_type periodic, retry_wait 1s, retry_max_times 3 and retry_secondary_threshold 0.9, this moved the next flush to the switch to the secondary. The recovered primary was not tried. The retry limit later dropped slow and a chunk that was never sent. On master, slow, fresh and the unsent chunk all reached the recovered primary. If the core takes the chunk back first, its own rollback in try_rollback_write can step the newer retry too, on master and here. My last request to keep updating an existing retry did not consider this. I am sorry for that.

  2. The callback can take back a later send of the same chunk. I reproduced this with out_forward over TCP, with delayed_commit_timeout 5s on the primary and ack_response_timeout 8s on the secondary. The core took the first send back, and the secondary sent the chunk again. Then the first send's ack timed out, and line 261 took back the second send while it waited for its ack. When that ack arrived, there was nothing to purge, so the recovered primary received the chunk too. On master, the first callback raised ArgumentError, and the second ack completed the chunk. All three versions of this PR send the chunk again. This order is possible with the defaults: 60 seconds for a primary that does not use delayed commit, and 190 seconds for an out_forward ack.

I tried two prototypes on top of 0440151 to see how they affect these cases.

  • A recorded the retry when a chunk was sent through the secondary. The delegated update then stepped only that retry. This kept the fix from my last review, and it avoided case 1 while the original send was still recorded. But once the same chunk had been sent again, the lookup by chunk ID returned the new send's retry. The late callback still took that send back and stepped its retry. Your out_forward tests passed. The update_retry false while retrying unit case failed, because its chunk was dequeued by hand and had no recorded send.
  • B gave a chunk sent through the secondary the longer of the two delayed_commit_timeout values. For out_forward, the secondary's value is ack_response_timeout + 2. This avoided the extra delivery in the 5s/8s run. But I could still reproduce it by delaying the first send, because the core's timer starts before try_write and the ack timer starts after it. Case 1 also remained. In a run with the default timeouts and a 2-second periodic retry, the time between sends to a secondary that never acks went from about 62 to about 193 seconds. The existing test secondary plugin can do delayed commit even if primary does not do it, and non-committed chunks will be rollbacked by primary failed. Also, try_rollback_write checks the list from the front. So a longer deadline at the front delayed the rollback of a later primary chunk whose own deadline had passed.

Neither prototype covers both cases. My preference would be to handle case 1 in this PR if the change can stay small, and to discuss resend handling and the timeout semantics separately. Keeping the current fix for #5633 and tracking both cases in a follow-up issue also seems reasonable to me. Which scope would you prefer?

end
d.run(default_tag: 'test', timeout: 20, wait_flush_completion: false, shutdown: false, flush: false, force_flush_retry: true) do
d.feed(event_time, {"a" => 1})
waiting(15) { sleep 0.1 until limit_reached.call }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHOULD: Both cases of this test timed out in all five Windows jobs of the test matrix, in the CI run for 2bd73f2 (https://github.com/fluent/fluentd/actions/runs/43796615296). Each case failed with TestTimedOut after about 17 seconds. On Ubuntu and macOS, each case passed in about 9 seconds.

The test is unchanged in 0440151, and its Windows jobs have not run yet. I do not have a Windows machine, so I could not find which part of the wait condition stayed false. Could you look into it? Please also check the new test an ack timeout on the secondary does not start a new retry on Windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A <secondary> output cannot roll back a chunk: rollback_write always raises ArgumentError

2 participants