kafka traffic breaks with Linkerd Edge 25.8.5+ due to HTTP/2 protocol hint on TCP (opaque) connections

[bezarsnba]

What is the issue?

After upgrading to Linkerd edge-25.8.5, Kafka clients started failing with NoBrokersAvailable.
Even when Kafka traffic is explicitly configured as opaque TCP (opaque-ports: 9092) and proxyInit.iptablesMode is set to legacy, the Linkerd proxy still assigns protocol_hint: Http2 to Kafka traffic and attempts to establish a meshed HTTP/2 transport.

This results in frequent connection resets (CANCEL) on long-lived TCP connections, which Kafka does not tolerate.

The issue disappears immediately when bypassing the proxy using skip-outbound-ports, indicating the problem occurs inside the Linkerd data plane.

How can it be reproduced?

• Deploy a Kafka cluster with brokers running on port 9092.
• Inject Linkerd sidecars into both Kafka clients and brokers.
• Configure Kafka traffic as opaque: config.linkerd.io/opaque-ports: "9092"
• Use neutral service port naming (e.g. tcp-client).

Observed result:

• Kafka client fails with NoBrokersAvailable
• Proxy logs show protocol_hint: Http2 and stream resets

Expected result:

• kafka traffic handled as pure TCP (opaque)
• No HTTP/2 hints
  -Stable long-lived connections

Logs, error output, etc

[DEBUG] Dropping cache entry
key: Opaq(OpaqSidecar {
  orig_dst: OrigDstAddr(<POD_IP>:9092),
  routes: Receiver {
    routes: Route {
      backend: Forward(
        <POD_IP>:9092,
        Metadata {
          namespace: "<namespace>",
          pod: "<kafka-broker>",
          serviceaccount: "<serviceaccount>"
        },
        protocol_hint: Http2,
        tagged_transport_port: Some(4143),
        identity: Some(ClientTls {
          server_name: "<identity.linkerd.cluster>"
        })
      )
    }
  }
})

[DEBUG] send Reset { error_code: CANCEL }
[DEBUG] Policy watch closed; terminating

output of linkerd check -o short

linkerd-version
---------------
‼ cli is up-to-date
    is running version 25.8.5 but the latest edge version is 26.1.4
    see https://linkerd.io/2/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 25.8.5 but the latest edge version is 26.1.4
    see https://linkerd.io/2/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
	* linkerd-destination-6b496bbcd9-8rvwg (edge-25.8.5)
	* linkerd-destination-6b496bbcd9-qn9rh (edge-25.8.5)
	* linkerd-destination-6b496bbcd9-tbqhp (edge-25.8.5)
	* linkerd-identity-597d6d7467-jthc6 (edge-25.8.5)
	* linkerd-identity-597d6d7467-p2bfd (edge-25.8.5)
	* linkerd-identity-597d6d7467-pz2v2 (edge-25.8.5)
	* linkerd-proxy-injector-6f6dc4fdfc-46w79 (edge-25.8.5)
	* linkerd-proxy-injector-6f6dc4fdfc-s8pvz (edge-25.8.5)
	* linkerd-proxy-injector-6f6dc4fdfc-wr7bm (edge-25.8.5)
    see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints

Environment

• Linkerd version: edge-25.8.5
• Kubernetes: GKE, Kubernetes 1.33
• CNI: GKE default
• iptables backend: nftables (also tested with iptablesMode: legacy)
• Workload type: Kafka (long-lived TCP connections)
• Sidecar injection: enabled on both Kafka clients and brokers

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None

[bezarsnba]

I believe have identified the root cause of the behavior observed after upgrading to 25.8.5.

Starting from recent edge releases, the default value of outboundTransportMode is transport-header. In this mode, traffic between meshed workloads no longer preserves the original destination port in the same way as transparent mode, as described in the following discussion:

• https://linkerd.buoyant.io/t/clarification-on-outbound-transport-mode/768/2

In our case, this change interacted poorly with Kafka workloads (TCP, long-lived connections). As part of the initial mitigation, we applied a global skip-outbound-ports: 9092, which restored connectivity but bypassed the proxy entirely.

After switching outboundTransportMode to transparent and removing both the global skip and opaque port configuration for 9092, Kafka connectivity stabilized while remaining fully meshed.

This suggests the issue is not specific to nftables or iptables mode, but rather to the change in outbound transport behavior combined with protocol detection and TCP workloads like Kafka.

[mehrdadbn9]

@mehrdadbn9 I would like to work on this issue. Please assign it to me.