Suites are organised by what a failure means, not by which runner executes them. Four layers, each answering a different question:
| Layer | Question it answers | Runner | Where |
|---|---|---|---|
unit |
Does a transformation behave correctly? | vitest | tests/ts/unit/ |
integration |
Does the library handle a whole hand-authored survey? | vitest | tests/ts/integration/ |
contract |
Did output drift from the blessed snapshots (registry entities and whole surveys)? | vitest | tests/ts/contract/ |
validation |
Do the committed artifacts satisfy external standards? | pytest | tests/validation/ |
live |
What does a real survey engine do with them? | pytest + docker | tests/live/ |
TypeScript tests live next to the library they test (src/); Python owns
everything that needs a JVM, an external oracle, or a running engine.
| Transformation | Where the code lives | Where the tests live |
|---|---|---|
| XLSForm → LimeSurvey TSV | in-repo (src/, the TS library) |
vitest — tests/ts/unit/, snapshots in tests/ts/contract/tsvSnapshots.test.ts |
| XLSForm → DDI XML | in-repo (src/ddi/) |
vitest — tests/ts/unit/ddi/, snapshots in tests/ts/contract/ddiSnapshots.test.ts |
| LimeSurvey TSV → DDI / XLSForm (reverse) | in-repo (src/lstsv/, src/pipelines/lstsv2ddi/, src/pipelines/lstsv2xlsform/) |
vitest — those dirs + tests/ts/contract/*Roundtrip.test.ts |
| DDI XSD + Schematron validation | in-repo (workers/schematron-worker/, Java; rules in ddi-validation/) |
pytest — tests/validation/test_ddi_schema.py, test_registry_schematron_conformance.py |
| XLSForm fixture inputs are valid XLSForm | external oracle (pyxform) | pytest — tests/validation/test_xlsform_pyxform.py |
| LimeSurvey accepts the blessed TSV snapshots | live LimeSurvey (docker) | pytest — tests/live/limesurvey/test_registry_entities.py |
| What LimeSurvey stores when a respondent answers | live LimeSurvey (docker) + Playwright | pytest — tests/live/limesurvey/test_response_roundtrip.py |
| Real response exports → DDI data CSV | stored exports: LimeSurvey (tests/live/limesurvey/expected/), Kobo (tests/fixtures/surveys/<name>/kobo/) |
vitest — tests/ts/contract/lstsvRealExports.test.ts, koboRealExports.test.ts |
qwacback converts XLSForm → DDI with this library since
qwacback#3 (a ddi-emitter
sidecar). The qwacback equivalence test that guarded that swap (ported from
survey2ddi in formtransform#14) was retired afterwards: it compared the library
with itself. qwacback keeps its own equivalence check for its wiring (scripts/equivalence-test.mjs, internal/converter/ddi_client_test.go there).
tests/
ts/ # vitest (projects: unit, integration, contract)
contract/ # blessed-snapshot gates (bless, don't edit)
tsvSnapshots.test.ts # registry contract + byte-for-byte tsv.tsv
ddiSnapshots.test.ts # byte-for-byte ddi.xml
lstsv2ddiRoundtrip.test.ts # tsv.tsv → ddi == committed ddi.xml
lstsv2xlsformRoundtrip.test.ts # tsv.tsv → xlsform == authored xlsform.json
fullRoundtrip.test.ts # xlsform → tsv → xlsform
surveySnapshots.test.ts # whole surveys, every direction
integration/ # whole hand-authored surveys, CLI, browser bundle
unit/ # unit tests
ddi/ lstsv/ pipelines/ expressions/ # per subsystem
questionTypes/ *.test.ts # per question type / feature
codegen/ # pytest: registry validators reject broken registries
validation/ # pytest: committed artifacts vs external standards
fixtures.py # registry loader + ddi.xml snapshot loader
test_ddi_schema.py # blessed ddi.xml → XSD + Schematron (Java worker)
test_registry_schematron_conformance.py # schematron covers registry + rejects violations
test_xlsform_pyxform.py # every xlsform.xlsx is valid XLSForm (pyxform oracle)
live/ # pytest, auto-marked `docker`
conftest.py # applies the `docker` marker to this tree
limesurvey/
docker-compose.yml # the repo's single LimeSurvey stack
test_registry_entities.py # each blessed tsv.tsv imports into LimeSurvey
test_response_roundtrip.py # answer each entity, snapshot what gets stored
test_exclusive_choice.py # `exclusive` choices import as exclude_all_others
test_*.py # structure/settings/multilingual scenarios
respondent.py # export + snapshot helpers (citric)
fill_limesurvey.mjs # Playwright: fill + submit a live survey page
answers/<slug>.json # what the respondent enters
expected/<slug>.json # blessed exported response
output/ # generated TSVs (gitignored)
fixtures/surveys/<name>/ # one folder per whole survey, like a registry entity
xlsform.json | xlsform.xlsx # authored source
tsv.tsv ddi.xml # blessed forward snapshots
Stack lifecycle (compose file, base URL, credentials, import+activate) lives in
codegen/limesurvey_stack.py — one stack, shared
by the live suite and codegen --screenshots, so a rendering bug and a data bug
can never disagree about which LimeSurvey they are looking at.
The in-repo emitter's own suite: unit tests for variable extraction, note
classification, XML formatting, and per-type/varGrp construction, plus a
byte-for-byte snapshot comparison of buildDdiXml output against every
committed ddi.xml (prodDate scrubbed).
Each blessed ddi.xml is fed to the in-repo Java worker jar in CLI mode, which
runs both DDI 2.5 XSD validation and the CDL custom Schematron rules
(ddi-validation/). Skips when the jar is unbuilt (bash scripts/build-worker.sh,
JDK 21).
- Coverage audit (always runs): every named registry concept
(multipleResp, grid, other, concept/@vocab, intrvl, responseDomainType,
catValu) must be referenced in
ddi_custom_rules.sch— catches contracts the registry declares but Schematron ignores. - Mutation tests (need the jar): mutate a blessed
ddi.xmlto break a contract, assert Schematron rejects it — proves the rules aren't toothless.
Contract assertions against the in-repo converter: type/scale matches
limesurvey.typeCode per LS-supported type, or_other → other="Y", variant
examples produce Q-rows, plus byte-for-byte tsv.tsv snapshot comparison.
Each entity's xlsform.xlsx is fed to pyxform,
the reference XLSForm→XForm compiler used by ODK Collect / Enketo. It asserts a
standards-compliant engine accepts our fixture inputs — catching malformed
type strings, dangling list_names, or missing columns before they poison the
snapshot suite. pyxform validates inputs only; it emits ODK XForm, not TSV/DDI,
so it is not an output oracle. Skips if pyxform is not installed.
The per-entity suites pin one question type at a time, so anything that only
exists between questions is invisible to them: group nesting, page breaks, a
multilingual column set, expressions referencing another question, the
welcome/end-note conversions. Each tests/fixtures/surveys/<name>/ folder holds
the same trio as a registry entity — authored source plus blessed tsv.tsv and
ddi.xml — and is asserted in four directions:
| Direction | Assertion |
|---|---|
| xlsform → tsv | byte-for-byte vs tsv.tsv |
| xlsform → ddi | byte-for-byte vs ddi.xml (prodDate scrubbed) |
| tsv → ddi | same variables + response domains as the forward DDI, names sanitized |
| tsv → xlsform | every recovered question keeps a name and a type |
Byte parity is impossible on the reverse DDI path for any survey whose authored
names carry separators (LimeSurvey names are alphanumeric-only and capped at 20
chars), so that direction compares the variable list rather than the document.
Surveys whose variable list legitimately differs are listed in
REVERSE_DDI_STRUCTURAL_DIFF with a measured reason, and the list is asserted
to name only real fixtures — a stale entry fails the suite.
This suite immediately earned its keep: it caught the reverse path degrading a
minimal (dropdown) select_one into free text, because LS_TO_STD and the
reverse subset validator each hardcoded the appearance overrides they knew
(T) and missed !. Both now derive from APPEARANCES in the registry, so a
new override teaches both sides automatically. No registry entity uses
minimal, which is exactly why single-question fixtures could not see it.
testA is a real survey with a range question (start=0 end=100 step=5).
Its reverse DDI renames three select_multiple variables through LimeSurvey's
5-character answer codes, which REVERSE_DDI_STRUCTURAL_DIFF records.
Every blessed per-entity tsv.tsv is imported into a dockerized LimeSurvey via
the citric API client and asserted to land with ≥1 question. The vitest suite
proves the TSV matches the converter; this proves LimeSurvey itself accepts it.
Import → activate → fill the survey in a real browser (Playwright,
fill_limesurvey.mjs) → submit → export_responses → compare the stored values
against expected/<slug>.json. The only suite that asserts data semantics
rather than structure: it is what catches a time question stored as a datetime,
an answer code truncated by LimeSurvey's varchar(5), or an "other" free text
landing in an unexpected response column.
Inputs live in answers/<slug>.json, keyed by question code:
| Answer shape | Meaning |
|---|---|
"3" |
single value — a choice code, or free text for text/numeric/date |
["sa", "so"] |
multiple choice — tick these subquestion codes |
{"vertrauenpolizei": "5"} |
array/grid — subquestion code → answer code |
{"other": "Zeitung"} |
the built-in "other" option (also valid inside the array form) |
Every entity with a tsv.tsv must have an answers fixture (note has an empty
one — a note stores nothing, and the blessed snapshot records that).
Needs node + Playwright/Chromium on top of the docker stack; skips if Playwright is not resolvable (locally or globally).
npm test # vitest: all three projects
npm run test:unit # one project (also :integration, :contract)
uv run pytest # python: validation suites (live excluded by default)
npm run test:live # docker: brings the stack up, runs tests/live, tears down
npm run test:live -- -k response_roundtrip # args pass through to pytest
npm run test:all # everything
bash scripts/build-worker.sh # needed once for the DDI XSD/Schematron testsLive tests are excluded from a plain uv run pytest by the default
-m "not docker" filter in pyproject.toml; tests/live/conftest.py applies
that marker to the whole tree at collection time, so no test has to remember it.
registry/entities/<slug>/{ddi.xml,tsv.tsv},
tests/fixtures/surveys/<name>/{ddi.xml,tsv.tsv} and
tests/live/limesurvey/expected/*.json are frozen reference snapshots, not
auto-regenerated by codegen. The contract project and the live response
suite run the current code and assert output matches the committed files.
-
Change is correct → bless new output:
npm run bless # examples + tsv.tsv + ddi.xml + whole surveys npm run bless -- ddi # or one target: examples | tsv | ddi | surveys | responses npm run bless -- surveys # tests/fixtures/surveys/<name>/{tsv.tsv,ddi.xml} npm run bless -- responses # live stored-response snapshots (needs docker) git diff registry/entities/ tests/fixtures/surveys/ tests/live/limesurvey/expected/
-
Change is a regression → fix the emitter/converter, leave snapshots alone.
Default uv run codegen only writes meta.json and xlsform.xlsx
(deterministic from JSON-LD); it does not touch ddi.xml / tsv.tsv.
- Transformation behaviour → vitest under
tests/ts/unit/(next to the subsystem). Assert against the registry contract, not literal expected values, where possible. - A new blessed snapshot → put the test in
tests/ts/contract/so a failure reads as "bless or fix", not "logic bug". - A whole survey scenario → add
tests/fixtures/surveys/<name>/xlsform.json, runnpm run bless -- surveys, andsurveySnapshots.test.tspicks it up in all four directions. Behavioural assertions about that survey go intests/ts/integration/. - External standard / oracle →
tests/validation/. - Behaviour of a real engine →
tests/live/<engine>/; thedockermarker is applied for you.