Decouple semantic token declarations from theme-specific values

[brian-smith-tcril]

Note

Generated by Claude

Problem

Paragon's design tokens currently couple two separable concerns into the themes/<variant>/ directory:

1. The declaration that a path exists — e.g. color.gray.500 is a token of type color.
2. The value of that path in a particular theme — e.g. #454545 in light, possibly something else in dark.

tokens/src/themes/light/global/color.json does both. The schema (which paths exist) is determined by whatever the theme files happen to contain.

This coupling was surfaced while implementing #4274 (app tokens). App tokens reference Paragon paths like {color.gray.500}, and the build needs those paths in scope for reference resolution. With the current architecture, the only way to expose them is to include a specific theme variant's files (themes/light/**) — which drags in concrete values that the app build doesn't actually care about, since the output is var(--pgn-color-gray-500) regardless. It's load-bearing in name only; the values are immediately filtered out.

Proposal

Split the token tree into semantic tokens (the paths Paragon promises consumers can reference) and primitive tokens (the concrete values per theme variant):

tokens/src/
├── core/                  # primitives that are theme-invariant + semantic declarations
│   ├── colors.json        # e.g. color.gray.500 → {raw.color.gray.500}
│   └── ...
└── themes/
    └── light/
        └── colors.json    # raw.color.gray.500 → "#454545"

Concrete shape:

// core/colors.json — schema, references primitives
{
  "color": {
    "gray": {
      "500": { "$value": "{raw.color.gray.500}", "$type": "color" }
    }
  }
}

// themes/light/colors.json — primitives only
{
  "raw": {
    "color": {
      "gray": {
        "500": { "$value": "#454545", "$type": "color" }
      }
    }
  }
}

After this split:

• core/ is the stable interface Paragon exposes. Anything referencing {color.…} only needs core/ in scope.
• Themes are pure value providers — adding a new theme variant means defining the raw.* set, no schema duplication.
• The app token build (Handle non---pgn prefixed tokens from MFEs #4274) stops depending on which theme happens to be included for vocabulary purposes.

Tradeoffs / scope

This is a multi-PR effort, not a small change:

• Every existing themes/<variant>/**/*.json file needs migration: each color.X.Y becomes a raw.color.X.Y definition, and a corresponding core/ entry references it.
• The modify: [...] chain (color-yiq, mix, darken, lighten) currently runs against theme-resolved values. It would need to either run in the theme phase against raw values or remain coupled to a theme; either way, the chain's semantics need a careful pass.
• Cross-token references (e.g. color.btn.bg.brand → {color.brand.500}) need to resolve through the indirection layer correctly.
• The build's current assumption that core/ has no theme variants would change — core/ would contain references that only resolve once a theme is layered on. Either build tooling adapts or core/ is split further into "schema" (paths only) vs. theme-invariant primitives.
• Theme author packages (e.g. edx/elm-theme) currently override themes/light/global/color.json. A migration story is needed so existing brand packages keep working.

Why now / next steps

Not blocking #4274 — that issue's plan documents a workaround (include themes/light purely for the vocabulary). But the coupling makes the workaround feel incidental, and the split would benefit other future work too (e.g. theme variants beyond light/dark, multi-brand theme inheritance (No idea what Claude was thinking with these, the real benefits I see are being able to better support "base-theme-less" themes)).

Reasonable next step: a small spike PR that picks one token (say, color.gray) and works through the migration end-to-end — exposes the real difficulties of the modify chain and cross-references, and informs whether the broader migration is worth it.