Skip to content

perf(gen): generate typed scan mappers and inject them via the model constructors - #715

Merged
stephenafamo merged 5 commits into
stephenafamo:mainfrom
sandonemaki:perf/loader-typed-scan
Jul 7, 2026
Merged

perf(gen): generate typed scan mappers and inject them via the model constructors#715
stephenafamo merged 5 commits into
stephenafamo:mainfrom
sandonemaki:perf/loader-typed-scan

Conversation

@sandonemaki

@sandonemaki sandonemaki commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Generate a per-table, reflection-free scan.Mapper[*T] (<table>ScanMapper) and pass it to the
model constructor
(NewViewx / NewTablex), so that every query built from a model flows
through the typed mapper — as suggested in the review feedback.

This replaces the earlier approach of swapping the scanner inside the slice relationship loaders
only. The generated mapper scans each result column directly into the struct field by index
(a column-name switch + ScheduleScanByIndex), avoiding the per-row reflection that
scan.StructMapper does (FieldByIndex per column plus an O(C²) column-name lookup per row).

What flows through the typed mapper now

The View.scanner field is the single convergence point for model queries: View.Query(),
Table.Insert/Update/Delete (RETURNING scans) all copy it into orm.Query.Scanner. Injecting
the mapper at construction switches all of them at once:

path before this PR after
Model.Query().All()/One() StructMapper typed
one-object loaders o.LoadRel() StructMapper typed
slice loaders os.LoadRel() (direct join) StructMapper typed (plain .All(), no special-casing)
slice loaders (through join) StructMapper typed (scan.Mod(q.Scanner, …))
Insert/Update/Delete + RETURNING StructMapper typed
Preload parent columns StructMapper typed (base of the mapper-mod composition)
Preloaded child columns (alias.col) prefix StructMapper (inside the mapper mod) unchanged — see "Out of scope"

The diff is small (+68/−36 across 14 files) precisely because the scanner really is a single
convergence point.

What changed

  • NewViewx / NewTablex (psql, mysql, sqlite) take a scan.Mapper[T] argument used for all
    queries built from the view/table. nil falls back to scan.StructMapper, preserving the old
    behaviour. NewView / NewTable keep their current signatures (they pass nil).
    • ⚠️ This is a breaking change for code calling the x constructors directly. If preferred,
      I can switch to a non-breaking form (e.g. a variadic option or a WithScanner method) — happy
      to adjust to whichever shape you want the API to have.
  • gen/templates/models/table/003_scan_mapper.go.tpl (moved from the loaders output):
    generates <table>ScanMapper next to the model definition. It lives in the models output
    because the constructor call references it and the loaders output can be disabled.
    Since the constructor always uses the mapper, no unused mappers are emitted (this also resolves
    the open question from the previous revision about a strict unused linter).
  • gen/templates/models/table/001_types.go.tpl (+ the mysql override): pass the mapper to the
    constructor.
  • gen/templates/loaders/table/110_loaders.go.tpl: the loader-level special-casing is
    reverted. Direct-join slice loaders are back to plain .All(ctx, exec); the through-join loader
    wraps q.Scanner instead of naming the generated mapper, so the loaders templates no longer
    reference the mapper at all.

Safety / behaviour preservation

Correcting the safety claims from the previous revision (the earlier "unmatched columns are left
untouched" wording was imprecise):

  1. Unconsumed columns error identically for both mappers. A result column with no scheduled
    destination fails at the scan.Row level (no destination for column X) regardless of which
    mapper is used — neither mapper changes that. Columns consumed by another mod (preload
    alias.col columns, the through-join related_* columns) are scheduled by that mod, exactly
    as before.
  2. The matching key sets are identical by construction. The generated switch cases and the
    model's db struct tags are emitted from the same schema column names, and generated models
    tag every column field explicitly (relationship fields are db:"-" for both mappers).
  3. The matching rule is identical. scan's StructMapper matches column names with a plain
    == (no case folding or snake_case normalisation at match time — the snake_case default only
    applies when deriving keys for untagged fields, which generated models don't have), the same
    exact match the generated switch performs.

Hooks, mapperMods (nested Preload) and loaders (nested ThenLoad) are applied inside
bob.Allx/bob.One on top of the base mapper (scan.Mod(q.Scanner, mods...)), so they compose
with the typed mapper unchanged.

Out of scope (possible follow-up)

Preloaded child columns are still scanned by the prefix-aware StructMapper inside the preload
mapper mod. Making those typed requires reproducing three runtime behaviours in generated code
(NULL-tolerant scanning for LEFT-JOIN misses, all-NULL row validation, and runtime-unique alias
prefixes), plus a small public API addition to orm.PreloadSettings — I'd like to agree on that
API shape first, so it is left for a follow-up PR.

Performance

Scan-path micro-benchmark, mock sql.Rowsscan.AllFromRows, generated 4-column table,
Apple M4, -benchmem -count=3 (medians). Because the same injected mapper now serves every model query path, these numbers apply to the scan step of plain .All(), loaders and RETURNING scans alike (end-to-end gains depend on how much of each path is scan time):

rows StructMapper typed mapper Δ time allocs/op
100 15,790 ns/op 8,732 ns/op −45% 413 → 311
1,000 151,728 ns/op 86,752 ns/op −43% 4,016 → 3,014
10,000 1,582,410 ns/op 919,204 ns/op −42% 40,023 → 30,021

(Consistent with the −42…−48% measured in the previous revision on a 12-column table; the
real-workload numbers in the earlier description — eager-load −47…−60%, p95 −72% — measured the
same mapper swap on the slice-loader path and carry over.)

Testing

  • go test ./gen/bobgen-psql/driver ./gen/bobgen-mysql/driver ./gen/bobgen-sqlite/driver
    (testcontainers; generate → build → run generated test suites): all pass.
  • go test ./dialect/... ./orm/... ./gen: all pass. golangci-lint run on the touched packages:
    0 issues; gofumpt -l: clean.
  • End-to-end smoke on a generated SQLite schema exercising every path with the injected mapper:
    plain .All() (incl. nullable columns), one-object loader, direct-join and through-join slice
    loaders (incl. empty relations), INSERT … RETURNING, and Preload — verifying parent fields
    scan through the typed base mapper, the child relation is populated, and a parent row whose FK
    is NULL (all child columns NULL) loads without error with the relation left nil.

Generated <Parent>Slice.Load<Rel> loaders scanned related rows with
scan.StructMapper (per-row reflection). Generate a per-table reflection-light
<table>ScanMapper and use it instead, scheduling each result column directly
into the struct field by index. .All() merely delegates to bob.Allx with the
view scanner, so hooks, nested Preload/ThenLoad and behaviour are unchanged.
@sandonemaki

Copy link
Copy Markdown
Contributor Author

@stephenafamo
Locally, all tests pass. With the -race flag, TestSQLite/driver/modernc/generate completes in ~15.2 seconds.

@sandonemaki

Copy link
Copy Markdown
Contributor Author

@stephenafamo
Added a Performance section with numbers from a real-world workload. On the most heavily-nested endpoint, eager-load dropped ~47% (2.58 s → 1.36 s) and p95 ~72% (8.06 s → 2.29 s). The baseline already had the SQL-level optimization, so this is the scan-path change in isolation.

@stephenafamo

Copy link
Copy Markdown
Owner

I like the direction, but I think the better fix is to pass the dedicated mapper to the table constructor and then use that to construct the queries.

That way, all queries that are created for the models will flow through the typed mapper

@sandonemaki sandonemaki changed the title perf(gen): scan slice relationship loaders with a generated typed mapper perf(gen): generate typed scan mappers and inject them via the model constructors Jul 7, 2026
@sandonemaki

Copy link
Copy Markdown
Contributor Author

@stephenafamo
Done — reworked as suggested. The mapper is now passed to NewViewx/NewTablex
(nil falls back to scan.StructMapper, and NewView/NewTable are unchanged), so every query
built from a model flows through it: Query(), one-object and slice loaders, RETURNING scans,
and the base of the Preload mapper-mod composition. The loader templates no longer special-case
anything — direct joins are back to plain .All(), and the through-join loader derives from
q.Scanner.

While reworking this I also found a pre-existing bug (Query.Clone() drops the Scanner) and
split it out as #724, since it's independent of this change.

One question on API shape: I added the mapper as a new parameter on the x constructors, which is
breaking for direct callers. If you'd rather keep them compatible (variadic option / WithScanner
method), happy to change it.

…pper

Generate a per-table NULL-tolerant typed mapper (<table>ScanMapperNullable)
for every table that is the target of a to-one relationship, and pass it to
orm.Preload so the joined child columns are scanned without reflection while
preserving the LEFT JOIN semantics of the previous scan.StructMapper path:
an all-NULL row still yields no child object, and NULL values scanned into
non-nullable fields still leave the zero value.

Builds on stephenafamo#715 and benefits from stephenafamo/scan#8.

BREAKING CHANGE: orm.Preload (and the psql/mysql/sqlite Preload wrappers)
gain a new PreloadMapper[T] parameter. Only hand-written callers of
orm.Preload are affected; generated code is regenerated as part of this
change. Pass nil to keep the previous reflection-based behaviour.
… row

The generated `<table>ScanMapper` ran a per-row `switch col` over every
column to schedule scans. Resolve each column's index and field target
once at query start (matching the child preload mapper in 105), so the
per-row path only iterates the resolved targets and schedules by index.
@stephenafamo
stephenafamo merged commit 8f5af5f into stephenafamo:main Jul 7, 2026
8 checks passed
jay-babu pushed a commit to jay-babu/bob that referenced this pull request Jul 8, 2026
…pper

Generate a per-table NULL-tolerant typed mapper (<table>ScanMapperNullable)
for every table that is the target of a to-one relationship, and pass it to
orm.Preload so the joined child columns are scanned without reflection while
preserving the LEFT JOIN semantics of the previous scan.StructMapper path:
an all-NULL row still yields no child object, and NULL values scanned into
non-nullable fields still leave the zero value.

Builds on stephenafamo#715 and benefits from stephenafamo/scan#8.

BREAKING CHANGE: orm.Preload (and the psql/mysql/sqlite Preload wrappers)
gain a new PreloadMapper[T] parameter. Only hand-written callers of
orm.Preload are affected; generated code is regenerated as part of this
change. Pass nil to keep the previous reflection-based behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants