Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 37 additions & 8 deletions aws-healthomics/steering/migration-guide-for-nextflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,16 @@ You MUST complete the following steps
4. For **workflow-level content** (provenance reports, pipeline DAGs, etc.):
- Write to `/mnt/workflow/output/` (e.g., via `params.outdir = "/mnt/workflow/output/"`).
- HealthOmics exports files from this directory to the `output/` prefix in the run's S3 output location.
5. For **execution reports** (`dag`, `report`, `timeline`, `trace` config scopes):
- Nextflow v26.10+: you MAY set a relative `directory` in each scope (e.g., `report { enabled = true; directory = 'pipeline_info' }`). HealthOmics exports the reports to that subdirectory, next to the `output { }` results. `directory` MUST be relative, and you MUST NOT combine it with an absolute `file` — Nextflow aborts the run in both cases.
- Any version: set an absolute `file` under `/mnt/workflow/output/` (e.g., `report.file = '/mnt/workflow/output/pipeline_info/report.html'`). HealthOmics exports it to the `output/` prefix.
- A relative `file` without `directory` is NOT exported on any version.

**Done WHEN**:
- All legacy `publishDir` paths use `/mnt/workflow/pubdir/` prefix.
- **ALL** Nextflow 25.10+ output block `path` directives use relative paths (no absolute paths).
- Workflow-level content (provenance reports, DAGs) writes to `/mnt/workflow/output/`.
- Execution reports, if enabled, use a relative `directory` (v26.10+) or an absolute `file` under `/mnt/workflow/output/`.
- No references to ${params.outdir} in publishDir directives — all use the literal /mnt/workflow/pubdir/ path or a subdirectory.
- Relative path structure preserved.

Expand Down Expand Up @@ -185,28 +190,40 @@ You MUST complete the following steps
| v24.10 | `nf-schema@2.3.0` |
| v25.10 | `nf-schema@2.6.1`, `nf-core-utils@0.4.0`, `nf-prov@1.7.0`, `nf-fgbio@1.0.1` |
| v26.04 | `nf-schema@2.7.2`, `nf-core-utils@0.4.0`, `nf-prov@1.7.0`, `nf-fgbio@1.0.1` |
| v26.10 | `nf-schema@2.7.2`, `nf-core-utils@0.4.0`, `nf-prov@1.7.0`, `nf-fgbio@1.0.1` |

For Nextflow v24+, `nf-schema` replaces the deprecated `nf-validation`.

**Steps**:
1. Determine the target engine version. Read `manifest.nextflowVersion` if pinned; otherwise ASK the user.
- To run on v26.10, you MUST pin it exactly: `manifest.nextflowVersion = '=26.10.0'` — see [Pinning the Nextflow Version](./running-a-workflow.md#pinning-the-nextflow-version).
2. Verify every plugin the workflow uses appears in the matrix above for the chosen version. Remove or replace plugin references that aren't pre-installed.
3. IF the target version is v26.04, audit the workflow for v26 breaking changes and deprecations:
3. IF the target version is v26.04 or v26.10, audit the workflow for v26 breaking changes and deprecations:
- **Strict (v2) syntax is the default.** v1-only syntax will fail to parse. The user has two options — present BOTH and let them choose:
- Migrate the workflow to v2 syntax (see the [Strict syntax reference](https://docs.seqera.io/nextflow/strict-syntax)).
- Keep v1 syntax and pass `engineSettings.syntaxVersion = "v1"` on `StartRun`.
- Keep v1 syntax and pass `engineSettings.syntaxVersion = "v1"` on `StartRun`. This currently requires the AWS CLI or an SDK — see [Engine Settings](./running-a-workflow.md#engine-settings).
- Replace `listFiles()` with `listDirectory()` (deprecation warning otherwise).
- Remove `nextflow.enable.strict` from config (no longer needed; strict is the default).
- Remove `manifest.defaultBranch` from config (not used; HealthOmics has never supported Git-based pipeline checkout).
4. IF the target version is v25.10 or v26.04, the workflow MAY use the top-level `output { }` block and write workflow-level content (provenance reports, DAGs) to `/mnt/workflow/output/` — see Phase 4.
5. The following Nextflow v26.04 features are NOT supported on HealthOmics. Remove or avoid them:
- **Nextflow Registry module fetching** — HealthOmics workflows run in an isolated network; include modules directly in the workflow zip.
- **Static typing (preview)**.
- **Auto-load collection params from files** — depends on static typing.
4. IF the target version is v26.10, also audit for v26.10 breaking changes and deprecations:
- **The `echo` process directive is removed.** Replace `echo true` with `debug true` — the behavior is identical. You MUST tell the user that `engineSettings.syntaxVersion = "v1"` does NOT help: v1 is the parser that newly rejects `echo` (`Unknown process directive: echo`). The run fails in the engine before any task starts, and CreateWorkflow does not catch it.
- **`storeDir` is deprecated** (lint warning only). Replace it with workflow logic that reuses an existing output.
- **`eval` outputs re-run once on existing run caches.** The hash of `eval(...)` outputs changed, so tasks that declare one miss the cache the first time a 26.10 run uses a run cache created on an earlier version, then cache normally. Recent nf-core modules commonly use `eval` for version reporting. Tell the user to expect this; it is not a failure.
5. IF the target version is v25.10 or later, the workflow MAY use the top-level `output { }` block and write workflow-level content (provenance reports, DAGs) to `/mnt/workflow/output/` — see Phase 4. On v26.10+, an output's `path` closure MAY also return a map of source files to target paths.
6. IF the target version is v26.10, the following features are supported:
- **Static typing** (no longer preview). Typed processes and typed workflows MUST set `nextflow.enable.types = true`. `nextflow lint` reports type errors; `nextflow run` only warns. Typed `params` that have default values are currently treated as required by HealthOmics — the user MUST pass every typed param in the run parameters.
- **Vendored modules.** Install modules locally (`nextflow module install` or nf-core tools) and include the populated `modules/` directory in the workflow zip. You MUST include them by relative path (e.g., `include { FASTQC } from './modules/nf-core/fastqc/main.nf'`). Registry-style includes (`from 'nf-core/fastqc'`) fail on HealthOmics even when the module is vendored.
7. The following features are NOT supported on HealthOmics. Remove or avoid them:
- **Nextflow Registry module fetching** (v26.04+) — HealthOmics workflows run in an isolated network; include modules directly in the workflow zip.
- **Static typing** on v26.04 (preview on that version). Use v26.10 for static typing.
- **Auto-load collection params from files** on v26.04 — depends on static typing.
- **Agents (preview)** (v26.10).
- **Global cache** (v26.10) — use HealthOmics run caches instead.

**Done WHEN**:
- All plugins referenced by the workflow are in the pre-install matrix for the selected engine version.
- For v26.04 targets: deprecated symbols/configs removed, syntax decision documented, unsupported features absent.
- For v26.04 and v26.10 targets: deprecated symbols/configs removed, syntax decision documented, unsupported features absent.
- For v26.10 targets: `manifest.nextflowVersion = '=26.10.0'`, no `echo` directives remain, modules are included by relative path, and the static-typing decision is documented.

## Technical Patterns

Expand Down Expand Up @@ -271,6 +288,18 @@ output {
}
```

### Execution Reports in the Output Directory (Nextflow 26.10+)
```groovy
// nextflow.config — reports are exported to pipeline_info/ next to the output { } results
dag { enabled = true; directory = 'pipeline_info' }
report { enabled = true; directory = 'pipeline_info' }
timeline { enabled = true; directory = 'pipeline_info' }
trace { enabled = true; directory = 'pipeline_info' }

// Any version — use an absolute file under /mnt/workflow/output/ (exported to output/pipeline_info/)
report { enabled = true; file = '/mnt/workflow/output/pipeline_info/report.html' }
```

### S3 Reference (Before/After)
```groovy
// Before
Expand Down
10 changes: 8 additions & 2 deletions aws-healthomics/steering/running-a-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,14 +45,14 @@ Follow this SOP WHEN:

`StartRun` accepts an `engineSettings` map that customizes how HealthOmics invokes the workflow engine. The map is engine-agnostic in concept; today only Nextflow keys are implemented, so pass it only for Nextflow workflows. Pass it only when the user requests the corresponding behavior.

> **Tooling support**: `engineSettings` is part of the HealthOmics REST API, but it is NOT yet exposed by every released AWS CLI/SDK version — older clients reject it with `Unknown parameter: "engineSettings"`. Prefer the HealthOmics MCP server (which sends the field directly) to start runs that use `engineSettings`. If you must use the AWS CLI, first confirm the installed version accepts it (`aws omics start-run help` lists `--engine-settings`); if it does not, upgrade the CLI/SDK. Do not assume the `--engine-settings` flag exists on the user's installed CLI.
> **Tooling support**: `engineSettings` is part of the HealthOmics REST API, but the HealthOmics MCP server's `StartAHORun` tool does NOT currently accept it. To start a run that uses `engineSettings`, use the AWS CLI or an SDK. First confirm the installed CLI accepts it (`aws omics start-run help` lists `--engine-settings`); older clients reject it with `Unknown parameter: "engineSettings"` and must be upgraded. Do not assume the `--engine-settings` flag exists on the user's installed CLI.

Currently supported keys (Nextflow):

| Key | Purpose | Notes |
| --- | --- | --- |
| `profile` | Selects one or more profiles defined in the workflow's `nextflow.config`. | Comma-separated for multiple (e.g. `"test,docker"`). Order matters: v26.04+ applies in command-line order; earlier versions apply in definition order. A nonexistent profile = validation error. Profiles MUST be inside the workflow zip. |
| `syntaxVersion` | Selects the Nextflow parser syntax. | v26.04 defaults to strict (v2) syntax. Set to `"v1"` to run a workflow authored against the legacy parser. Not supported on v25.10 and earlier. |
| `syntaxVersion` | Selects the Nextflow parser syntax. | v26.04+ defaults to strict (v2) syntax. Set to `"v1"` to run a workflow authored against the legacy parser. Not supported on v25.10 and earlier. On v26.10, `"v1"` does NOT restore the removed `echo` directive — replace it with `debug`. |
| `outputFormat` | Format for the workflow output summary printed on completion. | v26.04+ only. |
| `agentMode` | Enables Nextflow agent logging mode. | v26.04+ only. |

Expand All @@ -61,6 +61,12 @@ Behavior to know (Nextflow profiles):
- Explicit run parameters (in `parameters.json`) override profile-defined parameter values.
- Recommend pinning `manifest.nextflowVersion` in the workflow when profiles are in use, so profile application is consistent across runs.

### Pinning the Nextflow Version

- To run a workflow on Nextflow v26.10, you MUST pin it exactly in the workflow's `nextflow.config`: `manifest.nextflowVersion = '=26.10.0'`.
- You SHOULD recommend an exact pin whenever the user needs a specific engine version or reproducible runs.
- IF a run used an unexpected engine version, check the workflow's `manifest.nextflowVersion` first. The engine log (from `DiagnoseAHORunFailure` or the run's CloudWatch logs) states the Nextflow version the run used.

### Handling Failures

IF the workflow run fails:
Expand Down
14 changes: 14 additions & 0 deletions aws-healthomics/steering/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,20 @@ IF a workflow fails to reach `ACTIVE` status, check these causes in order:
4. IF the previous run used a Run Cache you MUST reference that when starting the new run. Otherwise, you MAY create a Run Cache for this run.
5. Start a new run of the workflow/ workflow version using identical or modified inputs and Run Cache as appropriate.

## Nextflow Version-Specific Issues

IF a Nextflow run fails or behaves unexpectedly, match the symptom before deeper diagnosis:

| Symptom | Cause | Fix |
| --- | --- | --- |
| `Unknown process directive: echo` or `Unrecognized process directive echo`, before any task starts | The `echo` directive was removed in v26.10 | Replace `echo true` with `debug true`. `engineSettings.syntaxVersion = "v1"` does NOT help |
| `NoClassDefFoundError: io/seqera/serde/gson/GsonEncodingStrategy` | A registry-style module include (`include { X } from 'nf-core/foo'`) | Vendor the module into the workflow zip and include it by relative path (`'./modules/nf-core/foo/main.nf'`) |
| `Missing workflow parameters: ...` for typed `params` that have defaults | Typed params with defaults are treated as required | Pass every typed param in the run parameters |
| Execution reports (report, timeline, trace, DAG) missing from the run output | A relative `file` without `directory` is not exported | On v26.10+, set a relative `directory`. On any version, use an absolute `file` under `/mnt/workflow/output/` |
| The run used an unexpected engine version (the engine log states the version used) | `manifest.nextflowVersion` is missing or is a range | Pin the version exactly (e.g. `'=26.10.0'`) — see [Pinning the Nextflow Version](./running-a-workflow.md#pinning-the-nextflow-version) |
| Tasks with `eval(...)` outputs re-ran on a run cache after moving to v26.10 | The `eval` output hash changed in v26.10 | Expected, once per run cache. Later v26.10 runs cache normally |
| Every task missed the run cache | A container referenced by a mutable tag (e.g. `:latest`) was republished; the image digest is part of the task hash | Reference containers by digest when using run caches |

## VPC Connected Workflow Run Failures

IF a workflow run using VPC networking fails with connectivity-related errors:
Expand Down
7 changes: 5 additions & 2 deletions aws-healthomics/steering/workflow-development.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,12 @@ This SOP defines how you, the agent, create and deploy genomics workflows for AW
- Workflow-level content (provenance reports, DAGs) MUST be written to `/mnt/workflow/output/`.

### Nextflow Engine Version
- Pin the engine version with `manifest.nextflowVersion` in `nextflow.config` when the workflow depends on version-specific behavior or plugins.
- Pin the engine version with `manifest.nextflowVersion` in `nextflow.config` when the workflow depends on version-specific behavior or plugins. PREFER an exact pin (e.g. `'=26.10.0'`), which is required for v26.10 — see [Pinning the Nextflow Version](./running-a-workflow.md#pinning-the-nextflow-version).
- HealthOmics workflows run in an isolated network and CANNOT fetch plugins or modules at run time. The workflow MUST only depend on plugins pre-installed by HealthOmics for the target engine version. For the per-version plugin matrix and feature support, see [Phase 6: Nextflow Version Compatibility](./migration-guide-for-nextflow.md#phase-6-nextflow-version-compatibility) in the Nextflow migration guide.
- Nextflow v26.04 defaults to the strict (v2) syntax parser. Workflows authored against the legacy (v1) parser must opt in via `engineSettings.syntaxVersion = "v1"` at run time — see [Engine Settings](./running-a-workflow.md#engine-settings) in the Running a Workflow SOP.
- Nextflow v26.04+ defaults to the strict (v2) syntax parser. Workflows authored against the legacy (v1) parser must opt in via `engineSettings.syntaxVersion = "v1"` at run time — see [Engine Settings](./running-a-workflow.md#engine-settings) in the Running a Workflow SOP.
- Nextflow v26.10 supports static typing. Typed processes and typed workflows MUST set `nextflow.enable.types = true`. Use `nextflow lint` to catch type errors — `nextflow run` only warns.
- Use the `debug` directive, not `echo` (removed in v26.10). Avoid `storeDir` (deprecated in v26.10).
- Include vendored modules by relative path (`'./modules/...'`), not registry-style (`'nf-core/fastqc'`).

### Nextflow Profiles
- HealthOmics supports profiles defined in the workflow's `nextflow.config` `profiles { }` block. Profiles MUST be defined inside the workflow zip — HealthOmics does NOT fetch profile definitions from external sources.
Expand Down
Loading