Skip to content

Tolerate inaccessible datasets when loading BigQuery schema - #7763

Open
Umekawa wants to merge 1 commit into
getredash:masterfrom
Umekawa:bigquery-tolerate-inaccessible-datasets-in-get-schema
Open

Tolerate inaccessible datasets when loading BigQuery schema#7763
Umekawa wants to merge 1 commit into
getredash:masterfrom
Umekawa:bigquery-tolerate-inaccessible-datasets-in-get-schema

Conversation

@Umekawa

@Umekawa Umekawa commented Jun 26, 2026

Copy link
Copy Markdown

What type of PR is this?

  • Bug Fix

Description

get_schema() for BigQuery lists every dataset in the project and queries their INFORMATION_SCHEMA in a single UNION ALL. If any one dataset returns an error, _handle_run_query_error() raises and the whole schema refresh aborts, so the schema browser ends up empty.

This breaks environments that use dataset-level IAM (per-team isolation): a service account is granted only a subset of datasets, and the datasets it cannot read return 403 PERMISSION_DENIED on INFORMATION_SCHEMA.COLUMN_FIELD_PATHS / TABLE_OPTIONS. A single inaccessible dataset then hides the schema for the datasets the account can read.

The location filter (#7289) only helps when the inaccessible datasets live in a different location; when every dataset shares one location it excludes nothing.

This PR makes schema loading tolerant of per-dataset failures:

  • Send the batched UNION ALL query first (fast path, unchanged for fully-accessible projects).
  • If it fails, fall back to querying each dataset individually and skip the ones that error.

Per-dataset fallback queries are bounded by dataset count (not table count), so the performance win from #5632 is preserved for the common case.

Compatible with the existing BigQuery query runner; no configuration or schema changes.

How is this tested?

  • Unit tests (pytest, jest)

Added tests/query_runner/test_bigquery.py:

  • test_get_schema_skips_datasets_without_access: an inaccessible dataset is skipped and the accessible dataset's schema still loads.
  • test_get_schema_uses_single_batched_query_when_all_accessible: when all datasets are accessible, only the batched queries run (no per-dataset fallback).

Related Tickets & Documents

Closes #7760
Refs #5632, #7289

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

@Umekawa
Umekawa force-pushed the bigquery-tolerate-inaccessible-datasets-in-get-schema branch from 1e28ba6 to bfae549 Compare June 26, 2026 06:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BigQuery: schema browser is empty when the service account lacks access to some datasets in the project

1 participant