Fall back to the org's default CAS backend on artifact download failures #3190

Problem

CAS mappings are recorded per backend. When a backend's credentials become invalid (e.g. AWS keys rotated by creating a new backend instead of updating the existing one), every artifact mapped to the old backend becomes undownloadable — even when the new, healthy backend points at the same bucket and the bytes are still there.

Real-world scenario:

1. Org has backend A (S3, bucket my-bucket). Artifacts uploaded → mappings point to A.
2. Credentials are rotated by creating backend B (same bucket my-bucket, new keys) and making it the default. Backend A is left with revoked keys (InvalidAccessKeyId).
3. Downloads of pre-rotation artifacts resolve their mapping to A and fail, even though B could serve the exact same content.

Proposal

Extend the download path with a fallback mechanism, similar in spirit to the existing write-time FindDefaultOrFallbackBackend logic:

• When a download through the mapped backend fails (invalid credentials / validation status not OK), retry the download through the org's current default backend (and/or the fallback backend).
• Since storage is content-addressable, the digest verification we already perform guarantees correctness: if the default backend happens to contain the same content (same bucket or replicated storage), the download succeeds and is verified; otherwise it fails as before.
• A cheap heuristic could short-circuit the retry: prefer backends with the same location as the broken mapped one, or simply attempt the default backend when the mapped backend's validationStatus is invalid.

Benefits

• Credential rotation done via a new backend (a common operational pattern) no longer strands historical artifacts.
• No data migration or mapping rewrite needed — content addressing makes the retry safe by construction.

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri