Tolerate inaccessible datasets when loading BigQuery schema - #7763
Open
Umekawa wants to merge 1 commit into
Open
Tolerate inaccessible datasets when loading BigQuery schema#7763Umekawa wants to merge 1 commit into
Umekawa wants to merge 1 commit into
Conversation
Umekawa
force-pushed
the
bigquery-tolerate-inaccessible-datasets-in-get-schema
branch
from
June 26, 2026 06:45
1e28ba6 to
bfae549
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Description
get_schema()for BigQuery lists every dataset in the project and queries theirINFORMATION_SCHEMAin a singleUNION ALL. If any one dataset returns an error,_handle_run_query_error()raises and the whole schema refresh aborts, so the schema browser ends up empty.This breaks environments that use dataset-level IAM (per-team isolation): a service account is granted only a subset of datasets, and the datasets it cannot read return
403 PERMISSION_DENIEDonINFORMATION_SCHEMA.COLUMN_FIELD_PATHS/TABLE_OPTIONS. A single inaccessible dataset then hides the schema for the datasets the account can read.The
locationfilter (#7289) only helps when the inaccessible datasets live in a different location; when every dataset shares one location it excludes nothing.This PR makes schema loading tolerant of per-dataset failures:
UNION ALLquery first (fast path, unchanged for fully-accessible projects).Per-dataset fallback queries are bounded by dataset count (not table count), so the performance win from #5632 is preserved for the common case.
Compatible with the existing BigQuery query runner; no configuration or schema changes.
How is this tested?
Added
tests/query_runner/test_bigquery.py:test_get_schema_skips_datasets_without_access: an inaccessible dataset is skipped and the accessible dataset's schema still loads.test_get_schema_uses_single_batched_query_when_all_accessible: when all datasets are accessible, only the batched queries run (no per-dataset fallback).Related Tickets & Documents
Closes #7760
Refs #5632, #7289