757 extend define plugin

[jcrichlake]

Summary

• Fixes [Py SDK] Extend define_plugin() to accept schemas with to_common/from_common #757
• Time to review: 30 minutes

Changes proposed

Core framework (common_grants_sdk/extensions/)

• Added types.py — new plugin framework types per ADR-0022: PluginExtensions, PluginExtensionsMeta, PluginExtensionsSchema, ObjectSchemasInput,
  ObjectSchemas, TransformResult, PluginError, Handler, ObjectMappings, PluginCapability
• Added transforms.py — build_transforms() generates to_common/from_common callables from declarative mapping dicts, with structural validation at call
  time, custom handler registration, and optional Pydantic model validation via common_model
• Updated specs.py — merge_extensions() now accepts PluginExtensions (structured objects) instead of the flat SchemaExtensions TypedDict; added
  _merge_meta() for conflict-aware metadata merging
• Updated plugin.py — define_plugin() now accepts meta, extensions, schemas, get_client, and filters (all optional); Plugin and PluginConfig
  updated to match
• Updated __init__.py — exports build_transforms and all new ADR-0022 types
• Updated generate.py — code generator now handles the new PluginConfig shape, emits ObjectSchemas instances in generated __init__.py, and injects
  transform callables at generation time

Transformation utilities (utils/transformation.py)

• Added numberToString and stringToNumber built-in handlers
• Added HandlerError for attributing transform failures to specific handlers
• Added DEFAULT_HANDLERS registry dict and handlers parameter to transform_from_mapping

Examples

• Added examples/plugins/grants_gov/cg_config.py — grants.gov sample plugin demonstrating all 6 mapping types (field, const, match, numberToString,
  stringToNumber, and custom handlers)
• Added examples/transforms.py — end-to-end demo of bidirectional transforms with roundtrip verification, custom handlers (join/split), and typed model
  validation
• Updated examples/plugins/opportunity_extensions/cg_config.py — migrated from SchemaExtensions dict format to PluginExtensions objects; added module
  docstring
• Updated examples/plugin_custom_fields.py — updated call sites to use schemas.Opportunity.common.model_validate() and
  extensions.schemas["Opportunity"].custom_fields

Tests and docs

• Added tests/extensions/test_plugin.py, test_transforms.py, test_types.py
• Updated test_merge_extensions.py, test_plugin_generator.py, test_transformation.py, test_custom_fields.py, test_plugin_registry.py
• Updated extensions/README.md, ADR-0017, and ADR-0022

Context for reviewers

This PR extends define_plugin() to support the full plugin framework shape described in ADR-0022. The key design decisions:

• build_transforms() validates mapping structure at call time (not generation time) and surfaces failures as PluginError entries in TransformResult.errors
  rather than raising — consumers choose strict vs. lenient handling
• Both transform directions are author-provided; build_transforms() intentionally does not invert one into the other because many-to-one handlers like
  match/switch are not reversible
• Custom handlers are registered per build_transforms() call only and must not shadow DEFAULT_HANDLERS (raises ValueError at call time)
• The code generator injects to_common/from_common callables into generated __init__.py so plugin.schemas.Opportunity.to_common is populated without plugin
  authors needing to do it manually

Testing instructions:

From lib/python-sdk/:

# 1. Install dependencies
poetry install

# 2. Run the full check suite (formatting, linting, type checking)
make checks

# 3. Run all tests
poetry run pytest

# 4. Regenerate plugin schemas from cg_config.py files
make plugins

# 5. Run the bidirectional transforms example (no server required)
poetry run python examples/transforms.py

# 6. Run the plugin custom fields example (no server required)
poetry run python examples/plugin_custom_fields.py

Steps 5 and 6 should produce no errors. The transforms example should end with Roundtrip result (5 mapped fields checked; unmapped fields dropped by design): ALL
PASS.

Additional information

transforms.py output:
============================================================
to_common: grants.gov → CommonGrants
============================================================
Errors: none

Result:
{
  "title": "Research into conservation techniques",
  "status": { "value": "open", "description": "The opportunity is currently accepting applications" },
  "funding": {
    "minAwardAmount": { "amount": 10000, "currency": "USD" },
    "maxAwardAmount": { "amount": 100000, "currency": "USD" }
  },
  "keyDates": { ... },
  "customFields": {
    "legacyIdStr": { "value": "12345" },   ← numberToString: int 12345 → str
    "priorityScore": { "value": 75 }       ← stringToNumber: str "75" → int
  }
}

============================================================
ROUNDTRIP CHECK
============================================================
  [PASS] title: 'Research into conservation techniques' -> 'Research into conservation techniques'
  [PASS] status: 'posted' -> 'posted'
  [PASS] award_floor: 10000 -> 10000
  [PASS] award_ceiling: 100000 -> 100000
  [PASS] priority_score_str: '75' -> '75'   ← numberToString in from_common

Roundtrip result (5 mapped fields checked; unmapped fields dropped by design): ALL PASS

[github-actions]

🚀 Website Preview Deployed!

Preview your changes at: https://cg-pr-838.billy-daly.workers.dev

This preview will be automatically deleted when the PR is closed.

[SnowboardTechie]

Looked through this — really nice work on the layering between the new types and the generator. A few thoughts:

The "bring your own callable" path works cleanly. Anything matching Callable[..., TransformResult] drops into ObjectSchemasInput and build_transforms() is just one way to produce one. Might be worth a README mention that hand-written ones still need to return TransformResult — the type catches it but it's easy to miss when porting existing code.

The bigger thing I'd push on is get_client and filters (inline below) — they're not really stubs, they're plumbed through enough layers that we're committing to their shape today, and waiting to ship them as optional kwargs once the interfaces firm up would be a no-op for adopters.

One thing that didn't fit inline: lib/python-sdk/.coverage (53KB pytest-cov artifact) got committed. Probably just needs a gitignore entry and a git rm.

---

While I was reading the transform surface this builds on (these are in transformation.py and extensions/transforms.py, which already landed in HOLD-transforms via #810 — so not this PR's job to fix, but flagging in case it's useful before that branch merges):

• transform_from_mapping does if hasattr(data, "model_dump"): data = data.model_dump(mode="json") (around line 208). That call doesn't set by_alias=True, but CommonGrantsBaseModel doesn't set by_alias by default either, so dumping a Pydantic CG instance gives you snake_case keys (min_award_amount) while the from_common mappings use the camelCase aliases (funding.minAwardAmount.amount, see examples/plugins/grants_gov/cg_config.py:94). The path lookups silently miss and you get an empty result with no errors raised. The example happens to work because the chain is to_common(dict) → dict → from_common(dict) — both sides are dicts so it never matters. But from_common(opportunity_model) directly will break. Fix is data.model_dump(mode="json", by_alias=True) plus an isinstance(data, BaseModel) check instead of hasattr (the duck-type would match unrelated classes).
• build_transforms's common_model parameter only applies to to_common output — from_common ignores it. Symmetric naming threw me a bit; might read better as to_common_model=.
• The to_common validation block only catches ValidationError. Anything else from model_validate (e.g. TypeError from a misconfigured root validator) bubbles up and breaks the errors-as-values contract the rest of the function maintains.
• DEFAULT_HANDLERS has "switch" aliased to "match" for backward compat, but the tests in test_transforms.py still use switch while the new grants_gov/cg_config.py example uses match. Two names for the same thing across canonical example vs. canonical tests is confusing for someone reading the code to learn the shape.

[SnowboardTechie]

HOLD-transforms synced from main on 2026-05-20: 12 commits — 7 Dependabot / catalog dep bumps (#829, #826, #822, #821, #819, #817, #808), 3 CI / docs chores (#834, #820, #816), 2 release version bumps [skip ci]. Merge was clean (no conflicts). You may want to rebase or merge HOLD-transforms back into your branch.

[SnowboardTechie]

Nice work, this is feeling solid. The by_alias + isinstance(BaseModel) fix, catch-all Exception on both transform directions (with the new test), and dropping get_client / filters all read clean. match as canonical with switch kept as the tested alias is a nice landing.

Two small follow-ups dropped inline — neither needs to land here. Approving.

[SnowboardTechie]

Heads up: this loop covers the mappings-only case, but there's a third bucket that's currently invisible — objects declared in config.schemas (explicit transforms) with no custom_fields and no mappings. They never get added to _Schemas, so _render_plugin_init_py's injected schemas.Opportunity.native = ... blows up at import with AttributeError.

Repro:

config = define_plugin(
    schemas={"Opportunity": ObjectSchemasInput(to_common=_to, from_common=_from)},
)
# generate runs fine; import → AttributeError

Nothing today hits this since the canonical example declares both slots, but it'll bite the first "just transforms, no custom fields" plugin. Probably folds into this loop by also iterating set(config.schemas.keys()) - explicit_cf_objs - mappings_only_objs with common defaulting to the base SDK class, same shape as mappings_only_objs. Fine to defer.

[SnowboardTechie]

The rest of the doc still references getClient / filters throughout, but the merged code has dropped them for this slot. Anyone reading the ADR alongside the code will wonder why the Plugin types don't match. Probably wants a one-line "Implementation status" note near the Decision summary explaining that get_client / filters are deferred to sibling tickets.

[widal001]

@jcrichlake @SnowboardTechie I know I've been a little out of the loop here, but I think something might have gotten lost in translation between our initial discussions around ADR-0022 and this implementation:

Based on this code it looks like we have the following attributes in DefinePluginOptions:

• schemas -- Keyed by object, holds the transformation code logic
• extensions.schemas -- ALSO keyed by object, but only contains the custom fields

I thought the whole point of consolidating transformation logic and custom fields under a single dictionary keyed by object was so that users only had to add one new entry when introducing a new object (e.g. Applications). But this still requires them to add two new entries.

I'd suggest we either consolidate these into a single root-level schemas.<object>.{customFields, native_schema, to_common, from_common} or we keep them separate and rename them:

• transformations with this shape <object>.{native_schema, to_common, from_common}
• customFields with this shape <object>.CustomFieldSpec

[widal001]

@SnowboardTechie does the TypeScript SDK transformation changes also have this same issue?

[jcrichlake]

Thanks for noting this! I went ahead and moved everything under the schemas object. I think this was a hold over from the PoC which was a partial implementation.

[SnowboardTechie]

Good news on the TS side. #825 is still in draft, and current shape has customFields on ObjectSchemasInput, so the per-object entry under transformSchemas carries native + customFields + toCommon + fromCommon together. No second key per object to maintain.

One thing the consolidation surfaces: pulling customFields out of the JSON-safe extensions surface means mergeExtensions over customFields drops out of the consolidated path, and the ADR-0022 text still describes the old dual-keyed shape. Better tracked as ADR cleanup than a separate amendment.

[widal001]

@jcrichlake See my comment above.

I'll continue working through this PR when I have time, but I think we'd need to combine or clarify the naming around schemas and extensions.schemas which will probably trickle down into how the rest of this PR is implemented.

[widal001]

This file looks like it needs to be update to change the Any to a set of generics that actually leverage native and common schemas so that the input and output of to_common() and from_common() have proper type hinting.

[jcrichlake]

I've attempted to address in subsequent commits.