feat: implement storage context and billing project for gcs

[project-defiant]

This PR involves changes to opentargets/issues#4321

Feature

• Introduced a generic StorageContext context manager so backend-specific options can be passed through execution the task to the storage handler classes without changing storage API signatures.
• Added a generic StorageSettings model that shall be extended with custom settings that are needed by the specific StorageHandlers.
• Implemented the GoogleStorageSettings model that introduces the billing_project used by GoogleStorage classes to inject the project in case the bucket is requester pays.

Example config used to test

  study:
    - name: copy study
      source: gs://open-targets-data-releases/26.03/output/study/part-00000-a8a8bfa5-c19c-4fc9-8736-91c89327ec8e-c000.snappy.parquet
  studies:
      destination: output/study/study.parquet
    - name: copy_many studies
      source: gs://open-targets-data-releases/26.03/output/study/*.parquet
      destination: output/study

Google Backend Behavior

• Synchronous Google storage now reads billing_project from StorageContext and passes it as user_project when creating Bucket object.
• Asynchronous Google storage now injects:
  • x-goog-user-project header
  • userProject query parameter
    Async listing logic was updated to use list_objects and context-aware headers/params. (code taken from original async implementation that uses the headers, that can not be passed otherwise)

Task-Level Changes

• copy task now accepts optional settings and runs copy/validation inside storage context.
• copy_many task now accepts optional settings and executes source resolution and async copies inside storage context.

Tests

• Expanded Google storage tests (sync + async) to verify billing_project is correctly applied to requests.
• Tested on VM and downloading data from gs://open-taragets-data-releases/26.03/study
  • without the settings specified with billing_project key, the failure with requester_pays issue
  • with the settings specified, data is copied properly.

Next steps

• bump otter in pis and include the settings to gwas and credible_set steps.

[Copilot]

Pull request overview

This PR introduces a generic storage context mechanism to pass backend-specific options (notably GCS requester-pays billing_project) through tasks into storage backends without changing existing storage API signatures.

Changes:

• Added a storage_context ContextVar-based context manager plus settings models for backend-specific configuration.
• Updated Google Cloud Storage backends (sync + async) to apply billing_project to bucket creation / request headers and query params.
• Updated copy and copy_many tasks to accept optional settings and execute within the storage context; added/expanded tests for context usage and GCS request injection.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
uv.lock	Bumped editable package version recorded in the lockfile.
pyproject.toml	Updated project version metadata.
src/otter/storage/storage_context.py	Added ContextVar-backed storage context manager and accessor.
src/otter/storage/model.py	Introduced base StorageSettings Pydantic model for context settings validation.
src/otter/storage/settings/init.py	Added GoogleStorageSettings (billing project).
src/otter/storage/synchronous/model.py	Added hook for backends to provide a context settings validation model.
src/otter/storage/asynchronous/model.py	Added hook for async backends to provide a context settings validation model.
src/otter/storage/synchronous/handle.py	Validates active context settings against the backend model when creating handles.
src/otter/storage/asynchronous/handle.py	Validates active context settings against the backend model when creating async handles.
src/otter/storage/synchronous/google.py	Uses storage context billing project when creating buckets (user_project).
src/otter/storage/asynchronous/google.py	Injects billing project into headers/params; refactors listing logic to list_objects.
src/otter/tasks/copy.py	Adds optional settings and runs copy/validation inside storage_context.
src/otter/tasks/copy_many.py	Adds optional settings and resolves/copies sources inside storage_context.
src/otter/util/errors.py	Added StorageContextSettingsError.
test/task/test_copy.py	Added tests asserting storage_context is used in copy run/validate.
test/task/test_copy_many.py	Added tests asserting storage_context is used in copy_many.
test/storage/synchronous/test_storage_google.py	Added sync GCS test verifying user_project uses billing project context.
test/storage/asynchronous/test_storage_google.py	Updated async GCS tests and added context header/param assertions for read/copy.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.