Fetch returns empty/EOF (v13 decode) after a proxy restart — consume silently stalls until a coordinated restart

[kamir]

Summary

After a routine proxy restart (host reboot, rollout), record Fetch through the kafSCALE proxy returns 0 records to every consumer, while metadata / ListOffsets keep working correctly. No client error is raised — the consumer just idle-times-out. The data is durable (segments in S3, high-watermark correct); only the record-Fetch read path is broken, and it stays broken until a manual coordinated broker + proxy restart.

This is a release-grade reliability concern: a routine ops action silently stalls the core read path for all consumers.

Observed on er1-bdr (KIND on master2): broker v1.6.0-readfix + proxy v1.6.0. Reproduced from the host, a compose container, and a remote client.

Per ADR-0002 the Kafka entrypoint is always the proxy; clients never talk to the broker directly. We do not want a Kafka wire-protocol change — the fix should land in the proxy (and/or as a version/build alignment), not in a broker wire widening.

Symptom

ListOffsets / kafka-get-offsets: correct HWM (e.g. er1a 495/521/6 = 1022). OK.
Consume (franz-go group-less, cp-kafka, remote): 0 records, no client error.

Proxy log on each fetch:

WARN fetch forward failed target=scalytics-primary-broker...:9092 error="read frame size: EOF"
WARN fetch forward failed target=scalytics-primary-broker...:9092 error="decode fetch response v13: response did not contain enough data to be valid"

The proxy forwards to the correct internal broker DNS (not a self-loop). The broker returns a partial/EOF v13 response.

Root cause analysis (from source)

Three factors compound:

1. The proxy is a full Fetch codec, not a byte forwarder.
forwardFetch (cmd/proxy/main.go) decodes the broker's FetchResponse, fans sub-requests out per broker, merges them, and re-encodes the response (protocol.EncodeResponse). So any mismatch between broker-side encode and proxy-side decode of a Fetch response breaks consume — the proxy cannot parse the broker's reply.

2. Fetch v13 + a build/version drift between broker and proxy.
The broker advertises Fetch min 11 / max 13 (cmd/broker/main.go: {key: protocol.APIKeyFetch, minVersion: 11, maxVersion: 13}), so clients negotiate v13. In the failing setup the broker ran a fork tag (v1.6.0-readfix) while the proxy was plain v1.6.0 — exactly the v13 (de)serialization is not guaranteed to match between those builds:

• read frame size: EOF (pkg/protocol/frame.go ReadFrame) = the broker returned nothing / closed the connection.
• decode fetch response v13: not enough data = the proxy read a frame, but it was too short to be a valid v13 FetchResponse.

A matched official v1.6.0 pair was previously byte-clean in a group-less franz-go round-trip (our read-path report, 2026-06-04), which points to the drift as the trigger.

3. The proxy does not recover — it retries only the wrong error class.
forwardFetch's retry loop retries only NOT_LEADER_OR_FOLLOWER partitions. A connection / decode failure is mapped to REQUEST_TIMED_OUT and is not retried on a fresh connection. The bad connection is closed (fanOutFetch), but the fetch is reported as failed, so the client sees an empty read. The coordinated restart only "heals" it by forcing broker and proxy back into a freshly-matched state.

Conclusion: not data loss, not a broken storage path. It is a Fetch-v13 compatibility / connection-state gap between a fork-broker and a release-proxy that the proxy does not absorb.

Workaround (verified)

Coordinated broker + proxy restart re-negotiates a matched pair and restores consume:

kubectl -n kafscale delete pod scalytics-primary-broker-0
kubectl -n kafscale wait --for=condition=ready pod/scalytics-primary-broker-0 --timeout=120s
kubectl -n kafscale rollout restart deploy/kafscale-proxy
# -> consume returns records again (verified: 1022 records, 0 failed)

What we want to achieve / proposed direction (discussion)

No wire-protocol change; fix in proxy / build / chart; always via the proxy.

1. Matched build pair. Proxy and broker built from the same source/tag; eliminate the -readfix fork-vs-release drift so v13 Fetch (de)serialization is guaranteed compatible. Cheapest, highest-leverage, likely the actual trigger.
2. Proxy resilience to a stale/half-open backend connection. On a frame / EOF / decode error, discard the connection and retry on a fresh one (extend the retry beyond NOT_LEADER_OR_FOLLOWER to connection-level failures) instead of silently returning REQUEST_TIMED_OUT.
3. (Optional) Down-negotiate the forwarded Fetch version to the broker's reliably-served version rather than blindly forwarding v13.
4. Regression smoke: restart the proxy, then assert a consume round-trip still returns records.

Questions for discussion

• Is the -readfix broker tag an outdated fork, and is the matched-pair drift the accepted root cause — or is there a known v13 encode/decode incompatibility in the current release pair regardless of build?
• Would maintainers prefer the proxy to (2) auto-recover connection-level Fetch failures, and/or (3) pin/down-negotiate the forwarded Fetch version? Either keeps the wire protocol unchanged.
• Is there appetite for the proxy-restart consume-regression smoke as a CI gate?

Happy to provide the full read-path report and reproduction artifacts.

[kamir]

Validation update (2026-06-10): drift confirmed, but the empty-Fetch does NOT reproduce on clean restarts

Re-tested on the live er1-bdr pair to validate the hypotheses. Honest result: the symptom did not reproduce on demand.

Running images (drift confirmed):

• broker kafscale-broker:v1.6.0-readfix
• proxy kafscale-proxy:v1.6.0

So the fork-tag-broker vs release-proxy drift (factor 2) is real in this deployment.

Tests via the proxy (kafka-get-offsets / kafka-console-consumer, cp-kafka 7.6.0):

Action	Result
ListOffsets via proxy	OK — HWM 495/521/6 = 1022
Consume via proxy (baseline)	returns records
Proxy-only rollout restart, then consume	still returns records, no fetch errors in proxy log
Abrupt broker kill (delete pod --force --grace-period=0), proxy untouched, then consume	broker recovers, still returns records, no fetch errors

Conclusion:

• The build drift exists but is not sufficient to trigger the empty-Fetch.
• A clean proxy-only restart and an abrupt broker churn both recover cleanly — the proxy re-establishes its backend connection and consume works. The "proxy does not retry connection-level Fetch failures" path (fix direction 2) did not bite in these tests.
• The original symptom was observed only after an uncoordinated host reboot (etcd/leadership churn + connection severing across all three components in arbitrary startup order). The coordinated broker+proxy restart healed it; the two controlled experiments above together effectively re-applied that coordinated restart, leaving the cluster healthy.

So the trigger narrows to the messy-reboot startup state, not a clean single-component restart. A rollout restart does not recreate it.

Questions this raises:

• Is there known proxy↔broker state that only goes bad when etcd leadership and broker readiness race during an uncoordinated startup (vs a clean rolling restart)?
• Would a proxy readiness gate that waits for broker Fetch to actually serve records (not just TCP/metadata) before going Ready close this, by enforcing coordinated ordering on startup?
• Still worth doing regardless: ship proxy and broker as a matched build pair (kill the -readfix vs v1.6.0 drift), and add a restart/reboot consume-regression smoke.

Happy to run a full-reboot reproduction next time the lab is power-cycled and attach the proxy logs from that window (that is when it last occurred).

[novatechflow]

The observed logs and behavior map directly to these paths in the current code (no other paths produce the exact messages or "0 records + no recovery" symptom for normal v11–v13 client fetches).

1. Fetch always takes the full codec path (not byte-forward) for supported versions

// cmd/proxy/main.go:1572
_, req, err := protocol.ParseRequest(payload)
if err != nil {
    return p.forwardFetchRaw(ctx, payload, pool)   // only for unparseable
}
...
groups := p.groupFetchPartitionsByBroker(ctx, fetchReq, nil)
return p.forwardFetch(ctx, header, fetchReq, payload, groups, pool)

forwardFetch + fanOutFetch then:
• forwards the (possibly re-encoded) sub-request,
• reads the broker reply,
• parses it,
• merges,
• and re-encodes the response to the client.

Raw byte forwarding only happens for unparseable requests or non-routed APIs (e.g. ListOffsets takes the simple path in handleConnection).

2. Exact log lines and error strings

// cmd/proxy/main.go:1706 (inside forwardFetch)
if r.err != nil {
    p.logger.Warn("fetch forward failed", "target", r.target, "error", r.err)
    addFetchErrorForAllPartitions(merged, r.subReq, protocol.REQUEST_TIMED_OUT)
    continue
}

// pkg/protocol/frame.go:34
if _, err := io.ReadFull(r, lengthBuf[:]); err != nil {
    return nil, fmt.Errorf("read frame size: %w", err)   // matches first log
}

// cmd/proxy/main.go:1889
func parseFetchResponse(data []byte, version int16) (*kmsg.FetchResponse, error) {
    ...
    if err := resp.ReadFrom(body); err != nil {
        return nil, fmt.Errorf("decode fetch response v%d: %w", version, err)  // matches second log
    }

Called from fanOutFetch:1819. On either error the conn is closed and the result is marked failed.

3. Retry logic only covers NOT_LEADER_OR_FOLLOWER

// cmd/proxy/main.go:1704
for _, r := range subResults {
    if r.err != nil {
        ... REQUEST_TIMED_OUT ...   // connection / decode / forward errors
        continue
    }
    ...
    if part.ErrorCode == protocol.NOT_LEADER_OR_FOLLOWER {  // 1718
        ... collect for retry ...
    } else {
        ... append part ...
    }
}

Only a successfully parsed NOT_LEADER_OR_FOLLOWER populates failedPartitions and drives the regroup/retry loop (up to 3 attempts). All other failures (including the ones producing the reported logs) immediately synthesize REQUEST_TIMED_OUT partitions for the client and return. No fresh-connection retry for that sub-fetch.

4. v13 + flexible encoding dependency
   Fetch v13 responses are flexible (kmsg.FetchResponse.IsFlexible() is true for v12+). The proxy's response handling therefore depends on:
   • protocol.EncodeResponse (broker side when replying to proxy)
   • SkipResponseHeader + parseFetchResponse (proxy side when reading broker reply)
   • protocol.EncodeResponse again (proxy → real client)

Both sides use the kmsg version present at build time. Any drift (different tags, different go.mod resolution, fork vs release) between the broker binary and proxy binary makes the proxy's parse of the broker's v13 reply fail with the exact "not enough data" error.

Why ListOffsets works and why only coordinated restart "heals" it
• ListOffsets and most other APIs do not go through handleFetchRouting / forwardFetch / parseFetchResponse.
• Once a fetch sub-request hits the error path above, the client receives a valid FetchResponse containing only REQUEST_TIMED_OUT (or empty) partitions. Subsequent client fetches repeat the same routing + fan-out and hit the same terminal error path. Nothing in the retry logic clears or retries the connection/decode failure class.

found in cmd/proxy/main.go:1570-1908 + pkg/protocol/frame.go + pkg/protocol/response.go:85.

[kamir]

Separate latent bug surfaced while building the readyz Fetch-probe: proxy re-encode path double-frames the request

While adding the /readyz Fetch-probe (gate on a real Fetch round-trip), the probe kept getting read frame size: EOF and the broker logged parse request: decode Produce v112 (payload bytes=116). Root cause is general, not probe-specific:

• encodeFetchRequest → kmsg RequestFormatter.AppendRequest already prepends the 4-byte length (it reserves dst = append(dst, 0,0,0,0) and writes len(dst[4:]) at the end).
• forwardToBackend then calls protocol.WriteFrame, which prepends another 4-byte length.

So any request built via encodeFetchRequest and sent through forwardToBackend is double-length-framed. The broker reads the outer length, then parses the payload starting at the inner length bytes — reading them as API key (0 = Produce) / version (here 112 = the body length) — and closes the connection.

Why it's normally invisible: the single-backend fetch path uses canUseOriginal and forwards the original client payload, whose length prefix ReadFrame already stripped — so exactly one prefix. The re-encode path (fanOutFetch when fanning a Fetch across multiple brokers, and forwardFetchRaw) builds via encodeFetchRequest and would double-frame.

Impact (needs confirmation on a multi-broker topology): Fetch fanout across >1 broker likely sends malformed double-framed requests. Single-broker clusters never hit it.

Fix options: have encodeFetchRequest return the body without the length prefix (and let WriteFrame own framing), or have the re-encode path write the already-framed bytes directly instead of through WriteFrame.

In the proxy readyz probe I worked around it by stripping the inner prefix (framed[4:]) before forwardToBackend. Filing here so the fanout path gets a real fix.

[kamir]

Reproduced on demand (2026-06-10): it's the REAL-DATA v13 Fetch, not the restart

The earlier validation couldn't reproduce the empty-Fetch on a clean restart. A live er1-brain end-to-end test pinned it down: the trigger is a real-data v13 Fetch, not the restart per se.

Setup: er1-bdr, broker v1.6.0-readfix-based + proxy from the readyz branch, all freshly restarted (coordinated).

1. Produced a fresh Avro record to er1a.events.er1_items via the proxy → it landed (ListOffsets 495/521/6 → 496/521/6).
2. The franz-go consumer (unfolder) read 0 records — on the compose network and on --network host.
3. cp-kafka (Java) consumes the same topic/partitions fine (records returned).
4. Proxy log during the franz-go run:

   fetch forward failed ... error="decode fetch response v13: response did not contain enough data to be valid"
   fetch forward failed ... error="read frame size: EOF"
   

So: metadata/ListOffsets fine; cp-kafka fine; franz-go v13 real-data Fetch → broker returns a short/EOF v13 response the proxy can't decode. Both clients are capped at the broker's advertised Fetch max (v13), so it's not a version-negotiation difference between clients — it's the fetch shape (franz-go's default max.partition.fetch.bytes / batch handling vs cp-kafka's) hitting a broker v13 real-data serialization fault. Strong overlap with BUG-0024 (segment larger than max.partition.fetch.bytes returned empty) against the 4 MiB segmentBytes.

Caveat for anyone adding a readiness/health Fetch probe

A Fetch probe against an unknown/empty topic returns a decodable empty response and passes — it does NOT exercise this real-data path. A meaningful probe must fetch a real partition with data at a low offset.

Fix direction

• Broker: fix the v13 real-data Fetch serialization (the short/EOF response), and/or honour max.partition.fetch.bytes by returning at least one message (BUG-0024).
• Proxy: down-negotiate / pin the forwarded Fetch to a version+shape the broker serves reliably (what cp-kafka effectively gets).

Happy to share the exact franz-go fetch config and a minimal repro.