[evgeny-simplex]

Plan: https://github.com/evgeny-simplex/simplexmq/blob/ep/smp-sub-race/plans/2026-10-26-sub-send-race.md

Scope

This pull request is split into three (plan, section "Pull requests"):

• this one: the race fix, with D4 and D8, and tests T1, T2 and T5;
• smp server: deliver later messages after SUB while a delivery waits #2006, from master: D7 with test T4, independent of the fix;
• smp server: shorten row locks of PostgreSQL SUB batches #2007, stacked on this one: D9 and D10, with the V5 and D10 measurements.

Commits:

1. The plan.
2. Tests T1, T2 and T5; T1 fails on master.
3. Move of the subscriber registration functions to top level, without behaviour changes.
4. The fix, with D4, D8, D9 and D10.
5. D7, with test T4.
6. Corrections of the plan and of a code comment.
7. Merge of master.
8. Removal of D7 and T4 (now in smp server: deliver later messages after SUB while a delivery waits #2006).
9. Removal of D9 and D10 (now in smp server: shorten row locks of PostgreSQL SUB batches #2007).
10. The split, recorded in the plan.
11. Links to the split pull requests in the plan.

Problem

• A SUB batch reads the first message of each queue, and only later is the subscriber registered (by serverThread).
• SEND pushes a message only to a registered subscriber, and only when the queue was empty.
• A message written between the read and the registration is neither returned nor pushed; it waits for the next SUB, and later messages wait behind it.
• SUBS has the same window.

Change

• PostgreSQL: one transaction per SUB batch, on the priority connection pool: queue rows locked with FOR UPDATE in recipient_id order, the first message of each queue read, credentials verified against the locked records, subscribers registered, commit. write_message takes the same row lock.
• STM store: the same steps in one STM transaction per queue.
• Registration moves from serverThread to the commands thread for SUB, NEW with subscription, SUBS and DEL; the SMP serverThread is removed; the notification serverThread stays.
• SUBS message fold: the Sub is inserted with delivered set in one STM transaction, so that a message read by the fold is pushed once, either by SEND or by the fold.
• setQueueServices locks the queue rows in recipient_id order before the association UPDATE (D4).

Tests

1. T1, should deliver message sent in the same block as SUB: SEND processed between the SUB read and the registration. Fails on master.
2. T2, should deliver message sent in the same block as SUBS once: SEND in the same block as SUBS; it can be processed during the SUBS message fold and, on master, before serverThread registers the service. One MSG expected.
3. T5, should end service subscription with queues remaining after DEL: service subscription count after DEL and SUBS from other connections of the same service. It checks the final outcome and passes on master; the remaining DEL window is described in plan §4 item 9.

Questions

1. D1. Lock primitive: FOR UPDATE or advisory transaction locks. Implemented as recommended: FOR UPDATE (a), kept after V5: ≈ 395 B of WAL per subscription in a full restart storm (measurements in smp server: shorten row locks of PostgreSQL SUB batches #2007).
2. D2. Association races between service and non-service connections. Not applied, as recommended: a follow-up pull request (needs a migration).
3. D3. SUBS registers before its message fold. Implemented as recommended (accept).
4. D4. Ordered row lock in setQueueServices. Implemented as recommended (include).
5. D5. Credential re-check by key equality against the locked record. Implemented as recommended (accept).
6. D6. Same-queue batch order changes ([SUB q, GET q]). Implemented as recommended (accept).
7. D7. Stale SubThread state after a delivery thread exits (present on master). Moved to smp server: deliver later messages after SUB while a delivery waits #2006, with test T4.
8. D8. SUB transaction on the priority connection pool. Implemented as recommended (include).
9. D9. synchronous_commit off for the SUB transaction. Moved to smp server: shorten row locks of PostgreSQL SUB batches #2007.
10. D10. First-message query as a lateral LIMIT 1 per queue. Moved to smp server: shorten row locks of PostgreSQL SUB batches #2007.
11. D11. Existing lock cycle on services rows. Not applied, as recommended (leave unchanged).
12. D12. Contention fallbacks. Not applied, as recommended: apply only if V4 shows contention; the reduced V4 shows none in the 8-connection storm (section "Measurements: V4 contention, reduced").

The V4 measurements below were taken before the split, on dd436b0 and 9d9e8e0, which include D7, D9 and D10.

Measurements: V4 contention, reduced (plan §5 item 7)

In-memory store; server on 4 cores, load generator on 4 others; 4 senders × 8 threads without a rate cap; A = master c0c416d, C = dd436b0 (this PR before the review commits); 3 alternating pairs per scenario; median [min-max]; host shared with other workloads.

Scenario	Metric	A	C	C vs A
storm, 50,000 queues, 8 connections subscribing at once	subscribe window, s	23.8 [20.8-26.6]	21.6 [21.5-23.8]	-9%
storm	delivered/s	2163 [1847-2369]	2253 [2037-2273]	+4%
storm	ACK p50 / p99, ms	381 / 995	371 / 988	-2%
storm	server CPU per delivered, µs	1134 [1082-1259]	1084 [1073-1110]	-4%
individual, 10,000 queues, one connection	SUB response, s	4.02 [3.63-4.72]	5.03 [4.50-6.09]	+25%
individual	server CPU per delivered, µs	6561 [6428-6808]	7197 [6657-7504]	+10%

• Concurrent registration from 8 connections shows no contention: C's storm medians are at or below A's.
• Individual SUB responses of C were slower in all 3 pairs (one-sided p = 0.10), within the 4.44-6.77 s range of five earlier runs of A. In that phase one connection subscribes alone; the registry write runs inside each queue's transaction in its commands thread. Not yet measured further.
• Not covered: PostgreSQL store with pool and lock-wait figures, the registration re-execution counter, the displace scenario, per-batch SUB p50/p99.

Measurements: V4 contention, full (plan §5 item 7)

A = master c0c416d, C = 9d9e8e0 (this PR), both with benchmark-only counters (registration transaction executions, pool acquisition and hold times); SMP server (-N4) and PostgreSQL 16 (fresh cluster, pool size 10, defaults for fsync and synchronous_commit) on 4 cores, load generator on 4 others; 4 senders × 8 threads without a rate cap; 50,000 queues (individual: 10,000), one prefilled message each unless noted; storm = all connections subscribe at once; displace = set A subscribes all queues, then set B subscribes them while A stays connected; 3 alternating pairs per configuration; A / C medians; 1-minute load 4-14 at run start.

PostgreSQL:

Configuration	window, s	SUB call p99, s	ACK p99, ms	re-executions, % of SUB	SUB-batch hold p50, ms	SUB-batch acquisition p99, ms	WAL per SUB, B
storm, 8 connections	35.8 / 36.0	7.88 / 6.83	1241 / 1098	0 / 0.75	11.6 / 32.8	46 / 0.003	542 / 612
storm, 32	33.2 / 32.4	21.4 / 20.5	4359 / 3480	0 / 1.01	14.4 / 39.0	274 / 0.004	447 / 527
displace, set A, 8	34.2 / 34.3	6.21 / 6.54	1218 / 1130	0 / 0.39	12.6 / 37.3	33 / 0.005	397 / 454
displace, set A, 32	32.4 / 35.0	19.6 / 21.8	4431 / 3887	0 / 1.39	14.4 / 39.0	262 / 0.004	489 / 447
displace, set B, 8	13.0 / 10.9	2.92 / 2.00	452 / 354	0 / 0	6.05 / 30.0	26 / 0.006	292 / 334
displace, set B, 32	8.77 / 9.77	5.45 / 6.33	1125 / 1130	0 / 0	5.09 / 55.1	101 / 21	116 / 293
individual, 1 connection	9.91 / 11.1	1.36 / 1.45	234 / 235	0 / 0	6.05 / 14.4	0.002 / 0.004	412 / 487
individual, no prefill	2.56 / 4.10	0.28 / 0.57	-	0 / 0	1.45 / 12.6	0.004 / 0.020	0.7 / 263

In-memory store:

Configuration	window, s	SUB call p99, s	ACK p99, ms	re-executions, % of SUB
storm, 8 connections	27.2 / 28.5	5.54 / 6.68	1526 / 1670	0 / 1.47
storm, 32	31.6 / 33.0	21.3 / 20.4	4334 / 4186	0 / 1.79
storm, 32, no prefill (6 / 5 runs)	8.52 / 8.36	5.23 / 5.06	1016 / 944	0 / 4.45
displace, set A, 8	26.2 / 24.2	5.80 / 5.43	1418 / 1345	0 / 2.16
displace, set A, 32	32.0 / 29.8	20.2 / 20.3	3894 / 4646	0 / 2.08
displace, set B, 8	9.34 / 8.11	1.97 / 1.64	344 / 340	0 / 0
displace, set B, 32	7.66 / 8.40	5.30 / 5.40	925 / 960	0 / 0

• Contention from concurrent registration is visible but small: 0.4-4.5% of registration transactions re-execute when they insert new registry keys (≤ 0.006% when the keys exist), PostgreSQL lock waits appear in ≤ 0.6% of 10 ms samples with at most one waiter, and C's medians of window, SUB call p99 and ACK p99 lie within A's range or below it in every storm and displace configuration.
• The cost is the PostgreSQL SUB transaction: connection hold per batch is 2.4-11× of A, so one subscribing connection is slower (+12% with prefill, ranges overlapping; +60% without prefill, ranges disjoint, with SMP server CPU 1.36 → 2.24 s and PostgreSQL CPU 0.53 → 1.22 s per 10,000 SUBs), and SUB-only WAL rises from 0.7 to 263 B per SUB. On the priority pool C's batches wait ≤ 20 µs for a connection (A: 26-274 ms under load), except 21 ms with 32 connections in displace set B.
• One SUB of C was answered with ERR AUTH (in-memory, 32 connections, no prefill; 1 of 300,000 SUBs in 6 runs of C, 0 of 300,000 in 6 runs of A, 0 in other configurations); the server does not log which AUTH path it took, and the cause is not yet determined.
• Against the reduced run: the in-memory 8-connection storm no longer favours C (27.2 / 28.5 s against 23.8 / 21.6 s, ranges overlapping, higher host load); the counter quantifies the contention the reduced run did not show; the slower single-connection SUB of C recurs on PostgreSQL and is traced to the per-batch transaction. Not covered: service-association storm, PostgreSQL storm without prefill.

Measurements: V4, remaining scenarios

The two configurations left out of the full V4 run (plan §5 item 7): the PostgreSQL storm without prefill, and the service-association storm. Setup as in the full run: A = master c0c416d, C = 9d9e8e0 (this PR), the same benchmark-only counters, SMP server (-N4) and PostgreSQL 16 (fresh cluster, pool size 10) on 4 cores, load generator on 4 others, 4 senders × 8 threads without a rate cap, 50,000 queues without prefill, 3 alternating pairs per configuration, A / C medians, 1-minute load 7-15 at run start. In the service-association storm each connection has its own service certificate, and each SUB associates a queue created without a service with that service.

PostgreSQL:

Configuration	window, s	SUB call p99, s	ACK p99, ms	re-executions, % of SUB	SUB-batch hold p50, ms	other acquisition p99, ms	WAL per SUB, B
storm, 8 connections	12.2 / 13.7	2.28 / 2.99	392 / 516	0 / 1.18	5.31 / 28.8	27.6 / 32.8	232 / 325
storm, 32	8.15 / 8.66	5.32 / 5.85	1079 / 1118	0 / 2.61	6.32 / 39.0	96.8 / 170	148 / 271
service association, 8	21.5 / 22.0	4.62 / 4.89	532 / 548	0 / 1.81	5.31 / 24.2	30.0 / 31.4	1260 / 1362
service association, 32 (6 / 5 runs)	10.8 / 13.5	7.68 / 9.59	908 / 1112	0 / 2.62	5.67 / 37.3	101 / 194	1152 / 1254

In-memory store:

Configuration	window, s	SUB call p99, s	ACK p99, ms	re-executions, % of SUB
service association, 8 connections	14.2 / 14.2	2.80 / 3.05	329 / 319	0 / 3.18
service association, 32	9.57 / 11.0	6.26 / 7.67	890 / 1081	0 / 6.69

• Without prefill, C's PostgreSQL storm windows are 6-13% longer and the PostgreSQL association storm with 32 connections 25% longer (SUB call p50 5.38 → 6.52 s, delivered/s 194 → 147); A's and C's ranges overlap in every configuration. With prefill (full run) the storm windows did not differ. SUB-batch hold p50 is 24-39 ms in C against 5-6 ms in A.
• Service registrations raise registration re-executions in the in-memory store to 3.2% and 6.7% (full run: 1.5-4.5%); on PostgreSQL they are 1.8% and 2.6% (non-service storm without prefill: 1.2% and 2.6%). Each service registration writes the server-wide totalServiceSubs (D12 fallback 2).
• The association itself adds about 1 KB of WAL per SUB in both versions; C's WAL per SUB exceeds A's by 102 B, as in the non-service storm (93-123 B).
• ERR AUTH for one SUB of C again (PostgreSQL association storm, 32 connections; the run stopped and has no window): 1 of 21 C runs, 0 of 21 A runs. Both occurrences in V4 are in C, with 32 connections and no prefill, one per store. PostgreSQL logged no error; the server does not log which AUTH path was taken (verification, association, missing tryPeekMsgs entry, locked-key mismatch, updateQueueTime), and the cause is not yet determined.
• In one run of A (in-memory association storm, 8 connections) the registration counter stayed at 49,999 for 50,000 SUBs for 60 s; the cause is not determined.

CI with all tests after the split passed in https://github.com/simplex-chat/simplexmq/actions/runs/41097274652 (1449 examples, 0 failures).

/

[epoberezkin]

why? aren't they identical?