[yuechao-qin]

What

A container log is read through the API server's proxy to the node's kubelet. When the
cluster-autoscaler reclaims the node, that proxy fails with a 500 — and the log died with the
node, so every retry is spent on something that cannot succeed.

_get_log_by_pod_key now returns None for that error, joining the branches that already exist
for a deleted Pod and a container that has not started.

The error

Observed response body, with cluster identifiers replaced:

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure",
 "message":"Get \\"https://<node-ip>:10250/containerLogs/<namespace>/<pod>/main?timestamps=true\\":
             dial tcp <node-ip>:10250: i/o timeout","code":500}

Raised as ServiceException, a subclass of ApiException, so it arrives at the existing
except ApiException. The test builds it from a response whose data is bytes, the way the
client does, because the client decodes the body itself — and older clients store it undecoded,
which is why _api_exception_body accepts either.

Every error on this path, measured

One deployment, 72 hours:

body	count	handling
dial tcp <node-ip>:10250: i/o timeout	2065	this PR: no log + warning
pods "<pod>" not found	9	existing 404 branch
container "main" in pod "<pod>" is waiting to start: PodInitializing	1	existing 400 branch
anything else — 401/403/429/503, read timeout, storage error	0	still raises

Those three are the whole census, so the narrow match costs no coverage. Identifiers are replaced
because this repository is public; the unredacted bodies and the queries behind each count are in
Shopify/oasis-backend#827, an internal
repository.

Why match narrowly

The wording comes from the API server's log proxy and is not a documented contract. A 500 is only
treated as a lost log when its body names the container-log endpoint and a failed dial;
anything else raises, so a reworded error surfaces rather than being silently swallowed. A test
covers that near miss: a kubelet failure reported as a TLS error still propagates.

Verification

pytest --ignore=tests/instrumentation -q → 433 passed. Neutering the classifier turns exactly
the kubelet case red. The warning logs the full response body, so a wording change is visible in
the logs rather than only in a metric.