Multi dataset submission instructions and skill - #24
Merged
Conversation
waltsims
requested changes
Jul 3, 2026
waltsims
reviewed
Jul 6, 2026
| - *Acquisition-induced artifacts* (these are *fine* — note as `info` if interesting): real acoustic shadowing, real reverberation behind a strong reflector, real attenuation | ||
| State *what* you see and *where* (e.g., "ring-down banding across the top third"), not just "looks fine". | ||
| 6. **Match to reference (perceptual, not SSIM).** If the contributor provided a reference B-mode, view both images side by side and judge whether they depict the **same structures / anatomy / geometry** — accounting for differences in grid resolution, field of view, dynamic range, and colormap, which are expected and not defects. SSIM and other pixel-registered metrics are unreliable here because reconstruction and reference rarely share a physical grid; do not use them as a gate. Flag only a *clear* structural disagreement (different scene, mirrored/rotated geometry, features present in one and absent in the other) as `major` — it usually means the metadata in the file disagrees with what the contributor actually used. When the difference is plausibly just scale/colormap/FOV, it is not a finding. | ||
| 7. **`reconstruct.py` stays a thin wrapper.** All reconstruction logic — beamformer choice, grid parameters, pre/post-processing — belongs in `pipeline.yaml`; `reconstruct.py` should do little more than load the pipeline and call `reconstruct` (see the `examples/` directory for the expected shape). A `reconstruct.py` containing custom reconstruction functions or branching on acquisition type/geometry is a sign the pipeline logic hasn't been captured in `pipeline.yaml`. This shows up most often in submissions with multiple sub-datasets sharing one root-level `reconstruct.py` that auto-detects which sub-dataset it's reconstructing — recommend splitting into a dedicated `pipeline.yaml` + `reconstruct.py` per sub-dataset folder instead (a shared root-level pair is fine when the pipeline is genuinely identical across sub-datasets). A `reconstruct.py` can either generate the associated `pipeline.yaml` or load one in for reconstruction, we do not have a preference. |
Collaborator
There was a problem hiding this comment.
@swpenninga can we update this wording as well to allow for both zea based reconstruction.py and loading the pipeline.yaml?
waltsims
reviewed
Jul 6, 2026
| - `blocker`: pipeline crashes, output is not an image, output is all-NaN/zero, NaN holes mid-image, sign-flipped intensities | ||
| - `major`: pipeline-induced artifacts (dominant ringing, wraparound, severe banding), or a clear structural disagreement with the contributor's reference image (different scene/geometry, not merely scale/colormap/FOV) | ||
| - `minor`: image quality issues that don't break interpretability (visible artifacts in corners, slight resolution loss) | ||
| - `minor`: image quality issues that don't break interpretability (visible artifacts in corners, slight resolution loss); `reconstruct.py` carries custom reconstruction logic or acquisition-type branching that belongs in `pipeline.yaml` |
Collaborator
There was a problem hiding this comment.
again here the idea is every possible processing step is completed in Zea or ping organizers in the discord for support.
waltsims
reviewed
Jul 6, 2026
|
|
||
| This is exactly the deliverable set the submission guide requires: zea `.hdf5` files, `reconstruct.py`, `pipeline.yaml` (one per track), `README.md`, and `LICENCE`. A separate `DATA_CARD.md` is **not** part of the expected layout — if one exists, treat it as a contributor-side artifact to be merged into `README.md` and flag the duplication. | ||
|
|
||
| **Multiple sub-datasets.** Some submissions bundle several sub-datasets (distinct acquisition types or geometries) as separate folders under one submission root. The recommended layout gives each sub-dataset folder its own `pipeline.yaml` + `reconstruct.py`, rather than sharing one pair at the root — this keeps each `reconstruct.py` a thin wrapper (load the pipeline, call `reconstruct`) instead of growing acquisition-detection and branching logic to serve every sub-dataset from one script. A shared root-level pair is still an acceptable choice when the pipeline is genuinely identical across all sub-datasets. Seeing multiple `reconstruct.py`/`pipeline.yaml` pairs across sub-dataset folders is the *expected* layout for this case — do not flag it as duplication the way a stray `DATA_CARD.md` would be. |
Collaborator
There was a problem hiding this comment.
Suggested change
| **Multiple sub-datasets.** Some submissions bundle several sub-datasets (distinct acquisition types or geometries) as separate folders under one submission root. The recommended layout gives each sub-dataset folder its own `pipeline.yaml` + `reconstruct.py`, rather than sharing one pair at the root — this keeps each `reconstruct.py` a thin wrapper (load the pipeline, call `reconstruct`) instead of growing acquisition-detection and branching logic to serve every sub-dataset from one script. A shared root-level pair is still an acceptable choice when the pipeline is genuinely identical across all sub-datasets. Seeing multiple `reconstruct.py`/`pipeline.yaml` pairs across sub-dataset folders is the *expected* layout for this case — do not flag it as duplication the way a stray `DATA_CARD.md` would be. | |
| **Multiple sub-datasets.** Some submissions bundle several sub-datasets (distinct acquisition types or geometries) as separate folders under one submission root. The recommended layout gives each sub-dataset folder its own `pipeline.yaml` + `reconstruct.py`, rather than sharing one pair at the root — this keeps each `reconstruct.py` a thin script instead of growing acquisition-detection and branching logic to serve every sub-dataset from one script. A shared root-level pair is still an acceptable choice when the pipeline is genuinely identical across all sub-datasets. Seeing multiple `reconstruct.py`/`pipeline.yaml` pairs across sub-dataset folders is the *expected* layout for this case — do not flag it as duplication the way a stray `DATA_CARD.md` would be. |
- Rewrite reconstruction check 7 around "every processing step is a zea operation": reconstruct.py may build the Pipeline in code and generate pipeline.yaml, or load an existing pipeline.yaml — no preference. Non-zea processing steps are the finding; point contributors to the organizers on Discord when zea cannot express a step. - Reframe the minor severity bullet the same way. - Apply suggested "thin script" wording in SKILL.md and match it in the convert skill. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The prior wording ("either build the zea.Pipeline in code and generate the
associated pipeline.yaml, or load an existing pipeline.yaml") technically
covered both shapes but was easy to skim over as one long clause.
Split it into two labelled bullets — Load-and-run and Build-and-save — with
a direct link to the closest existing example of each, and state the
invariant explicitly: pipeline.yaml is present in the submitted folder
either way, only its author (contributor vs. reconstruct.py) differs.
waltsims
reviewed
Jul 7, 2026
| - **Load-and-run.** `pipeline.yaml` is authored directly (hand-written or copied from a template) and shipped as its own artifact; `reconstruct.py` loads it via `Config.from_path` / `Pipeline.from_config` and calls `reconstruct`. See [`examples/templates/verasonics/reconstruct.py`](../../../../examples/templates/verasonics/reconstruct.py). | ||
| - **Build-and-save.** `reconstruct.py` builds the `zea.Pipeline` in code, saves it to `pipeline.yaml` via `pipeline.to_yaml(...)`, then calls `reconstruct`. The YAML is the script-produced artifact that stays in the submission so reviewers and downstream users can see the recipe. See [`examples/nv-raw2insights-us/reconstruct.py`](../../../../examples/nv-raw2insights-us/reconstruct.py). | ||
|
|
||
| Either way, `pipeline.yaml` is present in the submitted folder — that's the invariant, not how it was authored. What you *do* flag: processing steps implemented outside zea (hand-rolled numpy/scipy beamforming, custom envelope detection, ad-hoc filtering), or acquisition-type detection/branching to serve multiple sub-datasets from one script. If a step genuinely cannot be expressed as a zea operation, the recommendation is to contact the organizers on Discord (<https://discord.gg/gNTJeUsH2B>) for support — not to reject the submission. The branching case shows up most often in submissions with multiple sub-datasets sharing one root-level `reconstruct.py` that auto-detects which sub-dataset it's reconstructing — recommend splitting into a dedicated `pipeline.yaml` + `reconstruct.py` per sub-dataset folder instead (a shared root-level pair is fine when the pipeline is genuinely identical across sub-datasets). |
Collaborator
There was a problem hiding this comment.
@tristan-deep I wil merge this in. Please feel free to update this section with best practices for custom processing steps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update regarding instructions for uploading multiple sub-datasets