[Issue #799] Transform PoC

[jcrichlake]

Summary

• Fixes [SDK] Draft transforms PoC: Python #799
• Time to review: 15 minutes

Changes proposed

Implements a bidirectional transform framework as a proof-of-concept for the plugin system. Plugin authors can now define declarative mapping dicts that compile into to_common /
from_common callables, enabling data translation between a source system's native format and the CommonGrants format without writing imperative transform code.

New utilities:

• build_transforms() — compiles a pair of mapping dicts into typed (to_common, from_common) callables with structural validation at call time
• TransformResult — unconditional return shape carrying a result dict and a list of PluginErrors (non-fatal; partial result always returned)
• PluginError — structured error type with path, handler, source_value, and cause fields for programmatic error handling
• ObjectSchemasInput — bundles to_common / from_common for a single object type; passed to define_plugin() via transform_schemas
• PluginMeta — optional metadata attached to a plugin (name, version, source_system, capabilities)

Built-in mapping handlers (in utils/transformation.py):

Handler	Description
field	Extracts a value via dot-notation path
const	Returns a fixed literal value
match	Case-based lookup (canonical ADR name)
switch	Alias for match (backward compat)
numberToString	Extracts numeric value, coerces to string
stringToNumber	Extracts string value, coerces to int/float

Custom handlers can be registered per build_transforms() call and cannot override built-in names.

Sample grants.gov plugin (examples/plugins/grants_gov/): demonstrates the full interface — bidirectional field mapping, status case conversion, funding amounts, key dates, and
plugin metadata.

Documentation (extensions/README.md): new "Bidirectional Transforms" section covering mapping format, handler reference, custom handlers, and roundtrip usage.

Tests (tests/extensions/, tests/utils/): coverage for build_transforms, TransformResult, PluginError, ObjectSchemasInput, PluginMeta, and transform_from_mapping.

Context for reviewers

This is a proof-of-concept scoped to the transform layer only. It is intentionally limited:

• build_transforms() validates mapping structure at call time but cannot validate that field paths resolve against real data (deferred to full SDK with schema introspection).
• The PoC does not auto-invert one mapping from the other — both directions must be authored explicitly, because handlers like match are not reversible.
• The TODO (full SDK) comments in transforms.py call out where model_validate integration belongs (in define_plugin(), which knows the target Pydantic model).

Verification:

Run the working example end-to-end:

cd lib/python-sdk
poetry run python examples/transforms.py

This executes a grants.gov → CommonGrants → grants.gov roundtrip and prints a PASS/FAIL check for each mapped field.

Run the test suite:
cd lib/python-sdk
poetry run pytest tests/extensions/ tests/utils/ -v

Additional information

Sample output from examples/transforms.py:

============================================================
SOURCE DATA (grants.gov format)
============================================================
{ "data": { "opportunity_title": "Research into conservation techniques", "opportunity_status": "posted", ... } }

============================================================
to_common: grants.gov → CommonGrants
============================================================
Errors: none

Result:
{ "title": "Research into conservation techniques", "status": { "value": "open", ... }, "funding": { "minAwardAmount": { "amount": 10000, "currency": "USD" }, ... } }

============================================================
from_common: CommonGrants → grants.gov
============================================================
Errors: none

============================================================
ROUNDTRIP CHECK
============================================================
  [PASS] title: 'Research into conservation techniques' -> 'Research into conservation techniques'
  [PASS] status: 'posted' -> 'posted'
  [PASS] award_floor: 10000 -> 10000
  [PASS] award_ceiling: 100000 -> 100000

Roundtrip result: ALL PASS

[jcrichlake]

I'm not sure if we should handle this TODO here or later down the road. Would love thoughts on this.

[github-actions]

🚀 Website Preview Deployed!

Preview your changes at: https://cg-pr-810.billy-daly.workers.dev

This preview will be automatically deleted when the PR is closed.

[jcrichlake]

Calling this out as a needed change from the existing ADR. Without having the common_model in the signature we don't have a way to validate an extended class that contains optional fields. @SnowboardTechie for awareness

[SnowboardTechie]

Nice work on round 2. 7 of my 9 earlier findings are fully addressed, and the other 2 are documented-as-intended. The HandlerError refactor in particular reads much cleaner than the hardcoded handler=None we started with.

One latent bug worth surfacing before this ships: transform_node silently treats handler keys as literals when a non-handler sibling iterates first. The fact that build_transforms and transform_from_mapping are both public surface now makes it more load-bearing than before. Reproducer in the inline on utils/transformation.py.

Two small follow-ups on the round-2 fix itself, both flagged on individual threads: HandlerError is a behavior change for dump_with_mapping / validate_with_mapping callers (worth deciding whether to make it ValueError-compatible), and the new attribution path is not covered by an assertion in the exception test.

[SnowboardTechie]

Round-3 looks great! The ADR has some drift against #803 in the merge_extensions / filters area, but IMO we should rebase main onto current HOLD-transforms after this merges.

Approving w/ 2 small suggests: git rm lib/python-sdk/.coverage and add .coverage to either lib/python-sdk/.gitignore or the root. It looks like Pytest-cov SQLite output landed in the tree. And another trimming an extra /.

[SnowboardTechie]

Not sure we want this commited?