feat(db-bigquery): accept the service account key as a string, from the environment

[lloydtabb]

The gap

BigQuery has two credential slots today, and neither one works for a server that holds its service account key in an environment variable:

• serviceAccountKeyPath — file-typed, so {env: …} resolves, but it needs a file.
• serviceAccountKey — takes the key itself, but it is json-typed, and a json slot holds its value literally. config_compile.ts refuses reference indirection into structured config on purpose ("the security invariant that prevents reference injection into structured config"), so {env: …} is never resolved there.

The result is that this config:

"bq": {
  "is": "bigquery",
  "projectId": "my-project",
  "serviceAccountKey": {"env": "BIGQUERY_CREDENTIALS_JSON"}
}

hands the SDK the literal object {env: "BIGQUERY_CREDENTIALS_JSON"} as its credentials, which fails as:

The incoming JSON object does not contain a client_email field

The environment variable is never read, so the error is identical whether it holds the key, a path, or nothing — and it names neither the property nor the reason. client_email/private_key are honored by the constructor but are not declared properties, so the compiler drops them with an "unknown property" warning before the connection sees them.

The change

One new secret-typed (string) property, where references do resolve:

"serviceAccountKeyJson": {"env": "BIGQUERY_CREDENTIALS_JSON"}

It takes the entire key file as either raw JSON or base64, detected rather than declared — JSON always starts with {, and { is not in the base64 alphabet, so the two can be told apart with certainty. Base64 is worth accepting because it is a single unquoted token: a JSON object full of braces and quotes survives a shell, a CI secret editor, and a .env file poorly. (An earlier revision of this PR had two properties and justified base64 by newline mangling. That was wrong — a downloaded key file carries the private key's newlines as \n escapes, so the JSON is one line with no newlines to mangle. One property, correct reason.)

Details worth a second look in review:

• The parsed value is validated, not just type-checked as an object. It must carry client_email + private_key, or be an external_account config — the other shape the SDK's credentials option accepts (GoogleAuthOptions.credentials: CredentialBody | ExternalAccountClientOptions), which carries no key of its own. Without this, {} and any unrelated JSON reach the SDK and come back as the very client_email error this property exists to eliminate.
• Precedence: serviceAccountKey → serviceAccountKeyJson → client_email/private_key. Existing behavior is unchanged when the new property is unset.
• Values are trimmed, and empty or whitespace-only ones are skipped: {env: X} with X set-but-empty resolves to "" and must not read as "the key is here", and a key that arrived via a here-doc or $(cat key.json) carries a trailing newline that would otherwise break the encoding sniff.
• Errors quote nothing. JSON.parse's own SyntaxError embeds the input it choked on, which here is a private key. One message covers both encodings, because Buffer's base64 decoder drops unrecognized characters rather than failing — by the time the parse fails, there is no way to know which encoding was meant.
• Raw key material never reaches this.config; every key-bearing property is destructured out.

Tests

packages/malloy-db-bigquery/src/bigquery_credentials.unit.spec.ts, 20 hermetic tests (no warehouse): both encodings through one property, whitespace tolerance, newline restoration from \n escapes, external-account acceptance, key material staying out of the retained config, empty/whitespace fallthrough, the precedence ladder, and each error path — including that the malformed message contains neither the key nor the text it came from, and that {} and a half-a-pair key are rejected here rather than at the first query.

Three drive the real overlay stack through MalloyConfig with a set environment variable — the end-to-end claim this PR makes. One pins the old behavior for contrast: serviceAccountKey: {env: …} still lands the reference object in the credentials, so if malloy ever resolves references into json slots, that test fails and says so.

Two more read the registration back: that the property is secret (retyping it to json would silently restore the whole gap — verified by mutation that exactly those tests fail when it does), and that no second base64 property exists.

The live db-bigquery suite (149 tests) also passes against a real warehouse with this change.

Two follow-ups included

• connection/CONTEXT.md now documents the property and the precedence order. Its BigQuery list was also missing serviceAccountKey entirely; that is fixed. Per the cross-repo obligation in that file, the companion docs PR is docs(config): supplying a BigQuery service account key from the environment malloydata.github.io#339.
• connector-unit jest project (separate commit, droppable on its own). packages/malloy-db-*/**/*.unit.spec.ts are hermetic but live beside tests that need a warehouse, so they ran only in that backend's credentialed CI job — four specs across three connector packages, including this one. They now run in ci-core on every PR. The backend projects exclude the same pattern so each file runs in exactly one project; a file matched by two would appear twice in jest --listTests and fail scripts/ci-test-sanity-check.sh. Verified locally: sanity check passes, no duplicates, all 54 connector unit tests run with no credentials present.

[lloydtabb]

@mtoy-googly-moogly do you think "serviceAccountKeyJsonBase64": {"env": "BIGQUERY_CREDENTIALS_JSON_B64"} is overkill? I saw it being used on the web so I thought to do it, but maybe the plain json is enough.