Repository navigation
Conversation
acts_as_secondary defines rollback_write to delegate to the primary with rollback_write(chunk_id, update_retry). Output#rollback_write has taken update_retry as a keyword argument since v0.14.25, so the call always raised ArgumentError. A delayed commit output used as <secondary>, such as out_forward with require_ack_response, could not roll back a chunk after an ack timeout, and the chunk was not sent again. Fixes fluent#5633 Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thank you for the fix and the clear reproduction. Passing update_retry as a keyword is correct, and the new test fails on master as described.
In one configuration, making this rollback work changes behavior that works on master. If the secondary is out_forward with heartbeat_type none and its destination never acks, the retry no longer reaches retry_timeout or retry_max_times. Fluentd then stays on the secondary, even after the primary recovers. Details are in the inline comment. I also left a note on the new test.
| singleton_class.module_eval do | ||
| define_method(:commit_write){ |chunk_id| @primary_instance.commit_write(chunk_id, delayed: delayed_commit, secondary: true) } | ||
| define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry) } | ||
| define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry: update_retry) } |
There was a problem hiding this comment.
MUST: This fix changes how the retry ends in this configuration:
- The secondary is
out_forwardwithrequire_ack_response trueandheartbeat_type none. - Its destination accepts data but never acks.
- The secondary's
ack_response_timeoutis shorter than the primary'sdelayed_commit_timeout. This holds with the default values when the primary is alsoout_forwardwithrequire_ack_response: 190 seconds against 192 seconds.
On master, the delegation raises ArgumentError before the primary's rollback_write runs. So the chunk's entry stays in the primary's @dequeued_chunks. After the primary's delayed_commit_timeout, try_rollback_write takes the chunk back and calls update_retry_state. When the retry reaches its limit, master drops the queued chunks and clears the retry state. Later chunks then go to the primary.
With this PR, the ack timeout calls rollback_write(chunk_id, update_retry: false) on the primary. It returns the chunk to the queue and removes its entry, but it does not update the retry state. Nothing else updates it in this configuration. So retry_timeout and retry_max_times are never reached, and Fluentd keeps using the secondary, even after the primary recovers. With the default heartbeat_type transport, this does not happen. The disabled node raises NoNodesAvailable, and that error moves the retry forward.
update_retry: false comes from #1665, so that an ack failure on the primary does not count as a flush error. In the secondary phase, though, the retry state already exists. If the secondary never succeeds, only the retry limit ends that phase.
I ran Fluentd for 25 seconds with the configuration below. Nothing listens on port 24224 for the first 6 seconds. Then a server starts there and acks every chunk. The server on port 24225 reads the data but never acks.
<source>
@type sample
tag test
size 1
rate 1
</source>
<match test>
@type forward
heartbeat_type none
require_ack_response true
ack_response_timeout 1
<server>
host 127.0.0.1
port 24224
</server>
<buffer>
@type memory
flush_interval 1s
retry_type periodic
retry_wait 1s
retry_timeout 4s
retry_secondary_threshold 0.25
</buffer>
<secondary>
@type forward
heartbeat_type none
require_ack_response true
ack_response_timeout 1
<server>
host 127.0.0.1
port 24225
</server>
</secondary>
</match>
- master (d2c7287): The retry ends, and chunks go to the primary from about 9 seconds after start. The primary received 5 chunks.
- This PR: The primary received nothing. The secondary kept receiving the same 6 chunks, 24 sends in total.
I also checked the retry state with a script that runs the same plugins directly. With this PR, retry.steps stays at 0. With retry_type exponential_backoff and retry_max_times 2, master clears the queue after 2 steps. This PR is still at 0 steps after 16 seconds.
Could you keep retry_timeout and retry_max_times effective in this configuration, without changing how an ack failure on the primary is handled? Please add a test that goes through out_forward's ack timeout on the secondary, and check that each limit still ends the retry. The new test calls rollback_write directly, so it does not cover this.
| priconf = config_element('buffer','tag',{'flush_interval' => 1, 'retry_type' => :periodic, 'retry_wait' => 3, 'retry_timeout' => 60, 'retry_randomize' => false}) | ||
| secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'}) | ||
| @i.configure(config_element('ROOT','',{},[priconf,secconf])) | ||
| @i.register(:prefer_buffered_processing){ true } | ||
| @i.register(:prefer_delayed_commit){ false } | ||
| @i.register(:format){|tag,time,record| [tag,time.to_i,record].to_json + "\n" } | ||
| @i.register(:write){|chunk| raise "yay, your #write must fail" } | ||
| @i.secondary.register(:prefer_delayed_commit){ true } | ||
| @i.start | ||
| @i.after_start | ||
|
|
||
| @i.interrupt_flushes | ||
|
|
There was a problem hiding this comment.
SHOULD: This test leaves the flush thread running. interrupt_flushes stops automatic enqueueing and the timeout rollback, but the flush thread still calls try_flush. It wakes up about 1 second after start. If the test thread is that slow, the flush thread can take the rolled-back chunk and call the failing write. It then calls update_retry_state with three arguments. The mock rejects that call, and the flush thread dies. Teardown then loops in submit_flush_all forever, because @retry is still nil and the chunk is still queued.
I reproduced this by adding sleep 4 right after rollback_write. The update_retry false case failed, and the process hung until timeout killed it.
The test calls rollback_write directly, so it does not need the flush thread or the failing write. With the change below, it passed with the same sleep 4. It still fails on master, and also when the delegation ignores update_retry. queued_chunks_limit_size defaults to flush_thread_count, so it has to be set here. The update_retry false case also gets about 1 second faster, because teardown no longer flushes the chunk through the failing write.
| priconf = config_element('buffer','tag',{'flush_interval' => 1, 'retry_type' => :periodic, 'retry_wait' => 3, 'retry_timeout' => 60, 'retry_randomize' => false}) | |
| secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'}) | |
| @i.configure(config_element('ROOT','',{},[priconf,secconf])) | |
| @i.register(:prefer_buffered_processing){ true } | |
| @i.register(:prefer_delayed_commit){ false } | |
| @i.register(:format){|tag,time,record| [tag,time.to_i,record].to_json + "\n" } | |
| @i.register(:write){|chunk| raise "yay, your #write must fail" } | |
| @i.secondary.register(:prefer_delayed_commit){ true } | |
| @i.start | |
| @i.after_start | |
| @i.interrupt_flushes | |
| priconf = config_element('buffer','tag',{'flush_thread_count' => 0, 'queued_chunks_limit_size' => 1}) | |
| secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'}) | |
| @i.configure(config_element('ROOT','',{},[priconf,secconf])) | |
| @i.register(:prefer_buffered_processing){ true } | |
| @i.start | |
| @i.after_start | |
A secondary runs while the primary is retrying. With out_forward as the secondary, an ack timeout rolls the chunk back with update_retry: false, so retry_timeout and retry_max_times were never reached when the secondary never acked, and Fluentd stayed on the secondary even after the primary recovered. A rollback through the secondary now always updates the retry state of the primary. An ack failure on the primary still does not. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
|
Thanks for the detailed case. Pushed 2bd73f2. Retry limit with out_forward as the secondary. The rollback that the secondary delegates to the primary now always updates the retry state, even when the plugin passes I ran your configuration for 25 seconds, with a fluentd
New test in Unit test. Applied your suggestion:
|
Watson1978
left a comment
There was a problem hiding this comment.
Thank you for the update and the measurements. Both points from my last review are addressed. The retry limits now end the secondary phase, and an ack failure on the primary is handled as before. The new out_forward test fails on 2200f8a and passes here and on master. The unit test now passes even when the test thread stalls for 4 seconds after rollback_write.
However, my last request did not consider a secondary rollback that comes after the retry has ended. Such a rollback can now start a new retry. Then the queue can be dropped even though the primary has recovered. Details are in the inline comment.
| # A secondary runs while the primary is already retrying, and only the retry limit | ||
| # ends that phase if the secondary never succeeds. So a rollback from the secondary | ||
| # always updates the retry state, even when the plugin passes update_retry: false. | ||
| define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry: true) } |
There was a problem hiding this comment.
MUST: A secondary rollback can now start a new retry after the previous retry has ended. My last request did not consider this case, sorry.
A chunk can still be waiting for its ack on the secondary when the retry ends. The retry ends when another chunk succeeds through the secondary, or when the retry limit clears the queue. When the waiting chunk's ack times out, this delegation passes update_retry: true. update_retry_state finds no retry state, so it creates a new one.
The new retry may switch to the secondary before its first flush. Then that flush goes to the secondary without trying the recovered primary. In the example below, the switch comes 1.2 seconds after the retry starts (2s × 3 × 0.2), and the first flush comes after 2 seconds. If the secondary still does not ack, the limit clears the queue again. With a memory buffer, those chunks are lost.
In this reproduction, master also creates a new retry, but only after the primary's delayed_commit_timeout, through try_rollback_write. On 2200f8a, this rollback passed update_retry: false, so it returned the chunk without touching the retry state.
I reproduced it with the out_forward test driver and this configuration:
heartbeat_type none
require_ack_response true
ack_response_timeout 60s
<server>
host 127.0.0.1
port 24224
</server>
<buffer tag>
@type memory
flush_mode immediate
flush_thread_interval 0.05
flush_thread_burst_interval 0.05
queued_chunks_limit_size 20
flush_at_shutdown false
retry_type periodic
retry_wait 2s
retry_max_times 3
retry_timeout 30s
retry_randomize false
retry_secondary_threshold 0.2
delayed_commit_timeout 60s
</buffer>
<secondary>
@type forward
heartbeat_type none
require_ack_response true
ack_response_timeout 1s
<server>
host 127.0.0.1
port 24225
</server>
</secondary>
The server on port 24225 acks chunks tagged good, but never acks slow. I fed slow and good while nothing listened on port 24224. After good succeeded through the secondary, I started a server on port 24224 that acks every chunk. I waited until the ack timeout of slow was handled, then fed fresh and watched for 30 seconds.
- master:
freshreached the primary.slowwas still waiting for the 60-second timeout. - 2200f8a:
slowandfreshreached the primary. - This PR: Neither reached the primary. Both went to the secondary again and were dropped at the limit.
The configuration from my last comment shows it too, but the result depends on whether a chunk is still waiting for its ack when the retry ends. I ran it 6 times on each version, for 25 seconds each.
- master: The primary received 3 to 11 chunks. In 3 runs, the secondary was no longer used after 15 seconds.
- This PR: The primary received 2 to 4 chunks. In all runs, the secondary was still used after 23 seconds.
For a secondary rollback with update_retry: false, could you keep updating the retry while it exists, but not create a new one when none remains? Please keep the current behavior for update_retry: true and for ack failures on the primary. The check and the update need to happen together under the primary's @retry_mutex. If the delegation checks the retry first, another chunk's success can clear it before the update. I confirmed this window by pausing a prototype at that point. Apart from that window, the prototype went back to the primary in both cases above, and your out_forward test still passed.
Please add a test where a chunk's ack times out on the secondary after another chunk succeeded through it. Wait until that rollback is handled. Then check that no new retry was created and that the next chunk goes to the primary. The update_retry false case of the unit test then needs an active retry state.
There was a problem hiding this comment.
Thanks, that case makes sense. Pushed 0440151.
- The secondary's
rollback_writepassesupdate_retrythrough to the primary again. Withupdate_retry: false, after the chunk is taken back, it callsupdate_retry_state(chunk_id, false, create_retry: false)on the primary. update_retry_stategot acreate_retry:keyword,trueby default. Withfalse, it returns without changing anything when@retryis nil. The check and the update run in the same@retry_mutex.synchronizeblock, so another chunk's success cannot clear the retry between them.update_retry: trueand the primary's ownrollback_writeare unchanged, so an ack failure on the primary is handled as before.
Tests:
- New out_forward test
as secondary after the retry has ended. The secondary server acksgoodbut neverslow, withack_response_timeout 3son the secondary and your buffer settings. It feedsslowandgoodwhile the primary port is closed, waits until both reached the secondary and the retry is cleared, then starts a primary server that acks everything. After the rollback ofslow(rollback_countgoes up), it checks thatretryis nil, feedsfresh, and waits until bothslowandfreshreach the primary. On 2bd73f2 it fails with a newPeriodicRetryafter the rollback. - The unit test now has four cases:
update_retry: falseand the default, each with and without an active retry. It checksretry_countand whether a retry exists.update_retry false without a retryfails on 2bd73f2. - Your earlier out_forward test (
as secondary with a destination that never acks) still passes.
test_output_as_buffered_secondary.rb (17 tests), test_output_as_buffered_retries.rb (14), test_output.rb (141), test_out_forward.rb (81, 3 omissions) and test_output_as_buffered.rb (56) pass with 0 failures.
…try has ended Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thank you for the update. The case from my last review is fixed. When a secondary ack timeout returns a chunk with update_retry: false after the retry has ended, no new retry is created. The check for an existing retry and the update now happen together under @retry_mutex. Your new out_forward test fails on 2bd73f2 and passes on 0440151 on Linux.
I am leaving this review as a comment this time. Each round has found another problem with a late ack timeout, and my last request caused one of them. I would like to hear your view on the scope before asking for another change. The first inline comment describes the two cases I reproduced.
Separately, the out_forward retry-limit test added in 2bd73f2 timed out in all five Windows jobs of the test matrix. The second inline comment has the details.
| rolled_back = @primary_instance.rollback_write(chunk_id, update_retry: update_retry) | ||
| @primary_instance.update_retry_state(chunk_id, false, create_retry: false) if rolled_back && !update_retry |
There was a problem hiding this comment.
Two cases remain with an ack timeout that arrives late on the secondary. The callback only has the chunk ID. So it cannot tell which send, or which retry, the timeout belongs to.
-
The callback can step a different retry. I reproduced this with test outputs and a frozen clock.
goodsucceeded through the secondary whileslowwas still waiting for its ack, so the retry ended. Then the primary failed again, and a new retry started. The timeout forslowtook the chunk back before the core did, and line 262 stepped the new retry from 1 to 2. Withretry_type periodic,retry_wait 1s,retry_max_times 3andretry_secondary_threshold 0.9, this moved the next flush to the switch to the secondary. The recovered primary was not tried. The retry limit later droppedslowand a chunk that was never sent. On master,slow,freshand the unsent chunk all reached the recovered primary. If the core takes the chunk back first, its own rollback intry_rollback_writecan step the newer retry too, on master and here. My last request to keep updating an existing retry did not consider this. I am sorry for that. -
The callback can take back a later send of the same chunk. I reproduced this with out_forward over TCP, with
delayed_commit_timeout 5son the primary andack_response_timeout 8son the secondary. The core took the first send back, and the secondary sent the chunk again. Then the first send's ack timed out, and line 261 took back the second send while it waited for its ack. When that ack arrived, there was nothing to purge, so the recovered primary received the chunk too. On master, the first callback raisedArgumentError, and the second ack completed the chunk. All three versions of this PR send the chunk again. This order is possible with the defaults: 60 seconds for a primary that does not use delayed commit, and 190 seconds for an out_forward ack.
I tried two prototypes on top of 0440151 to see how they affect these cases.
- A recorded the retry when a chunk was sent through the secondary. The delegated update then stepped only that retry. This kept the fix from my last review, and it avoided case 1 while the original send was still recorded. But once the same chunk had been sent again, the lookup by chunk ID returned the new send's retry. The late callback still took that send back and stepped its retry. Your out_forward tests passed. The
update_retry false while retryingunit case failed, because its chunk was dequeued by hand and had no recorded send. - B gave a chunk sent through the secondary the longer of the two
delayed_commit_timeoutvalues. For out_forward, the secondary's value isack_response_timeout + 2. This avoided the extra delivery in the 5s/8s run. But I could still reproduce it by delaying the first send, because the core's timer starts beforetry_writeand the ack timer starts after it. Case 1 also remained. In a run with the default timeouts and a 2-second periodic retry, the time between sends to a secondary that never acks went from about 62 to about 193 seconds. The existing testsecondary plugin can do delayed commit even if primary does not do it, and non-committed chunks will be rollbacked by primaryfailed. Also,try_rollback_writechecks the list from the front. So a longer deadline at the front delayed the rollback of a later primary chunk whose own deadline had passed.
Neither prototype covers both cases. My preference would be to handle case 1 in this PR if the change can stay small, and to discuss resend handling and the timeout semantics separately. Keeping the current fix for #5633 and tracking both cases in a follow-up issue also seems reasonable to me. Which scope would you prefer?
| end | ||
| d.run(default_tag: 'test', timeout: 20, wait_flush_completion: false, shutdown: false, flush: false, force_flush_retry: true) do | ||
| d.feed(event_time, {"a" => 1}) | ||
| waiting(15) { sleep 0.1 until limit_reached.call } |
There was a problem hiding this comment.
SHOULD: Both cases of this test timed out in all five Windows jobs of the test matrix, in the CI run for 2bd73f2 (https://github.com/fluent/fluentd/actions/runs/43796615296). Each case failed with TestTimedOut after about 17 seconds. On Ubuntu and macOS, each case passed in about 9 seconds.
The test is unchanged in 0440151, and its Windows jobs have not run yet. I do not have a Windows machine, so I could not find which part of the wait condition stayed false. Could you look into it? Please also check the new test an ack timeout on the secondary does not start a new retry on Windows.
Which issue(s) this PR fixes:
Fixes #5633
What this PR does / why we need it:
acts_as_secondarydefinesrollback_writeon the secondary so that it delegates to the primary, but it passedupdate_retryas a positional argument.Output#rollback_write(chunk_id, update_retry: true)has taken it as a keyword since 59a47a1, so the delegation always raisedArgumentError: wrong number of arguments (given 2, expected 1).A delayed commit output used as
<secondary>, such asout_forwardwithrequire_ack_response true, callsrollback_writewhen the ack times out. The error was logged asunexpected error while receiving ack. The chunk was not taken back, so it was not sent again.This passes
update_retry: update_retry, as the issue suggests.Tests:
test_output_as_buffered_secondary.rb. The secondary dequeues a chunk and callsrollback_writewithupdate_retry: falseand with the default. The chunk must go back to the queue, andupdate_retry_statemust be called only with the default. On master both cases raise theArgumentError.test/plugin/test_output_as_buffered_secondary.rb15 tests,test_output.rb141 tests,test_output_as_buffered_retries.rb14 tests: 0 failures.Docs Changes:
None.
Release Note:
Fix
rollback_writeon a<secondary>output, which always raised ArgumentError.