Add optional data provenance tracking using the SSDS_metadata database

[MBARIMike]

MBARI invested quite a bit into the development of observatory system software. A key part of this was the Shore Side Data System. See this IEEE Conference paper. With the revamp of the software MBARI uses for Mooring data processing we will have more modern methods for recording data set provenance, namely authenticated REST API calls. This issue is meant to track the plan and implementation of this feature in auv-python.

[MBARIMike]

Claude Opus 4.6 Plan generated after providing legacy Perl script that used to track provenance via un-authenticated web requests:

Plan: SSDS Provenance Tracking via REST API

TL;DR

Add a provenance.py module that records processing provenance metadata into the SSDS_Metadata database via REST API calls, replacing the legacy Perl submitDStoNetCDFProcessRun. Integrate it into the existing Dorado and LRAUV processing pipelines so provenance is automatically submitted after each successful processing run.

Assumed REST API

Generic RESTful CRUD endpoints at a configurable base URL (e.g., https://mooring-ssds.shore.mbari.org/api/):

Entity	Endpoints
Person	GET /persons?email={email}
Software	POST /software, GET /software/{id}
Resource	POST /resources (created inline, linked to DataProducer)
DataContainer	POST /datacontainers, GET /datacontainers/{id}
DataProducer	POST /dataproducers, GET /dataproducers/{id}, GET /dataproducers?output_uri={uri}, PUT /dataproducers/{id}
Relationships	POST /dataproducers/{id}/inputs, POST /dataproducers/{id}/outputs, POST /dataproducers/{id}/resources, PUT /dataproducers/{id}/software, PUT /dataproducers/{id}/person

All endpoints accept/return JSON. Standard HTTP status codes (201 Created returns Location header or body with id).

---

Steps

Phase 1: Create src/data/provenance.py (new file)

Imports: requests, logging, socket, pathlib, datetime, argparse, sys, git (from gitpython, already a dependency)

Constants:

• SSDS_API_BASE = "https://mooring-ssds.shore.mbari.org/api"
• GIT_WEB_BASE = "https://github.com/mbari-org/auv-python/blob"
• GITHUB_REPO = "https://github.com/mbari-org/auv-python"
• PATH_TO_URL_MAP — dict mapping local path prefixes to OPeNDAP URL prefixes (mirrors Perl %web_lookup)
• REQUEST_TIMEOUT = 30

Helper functions:

1. _get_git_version() -> str — Uses gitpython (already a project dependency) to get the current commit hash. Falls back to "unknown".

2. get_dods_url(nc_file_path: str) -> str — Translates local NetCDF file path to OPeNDAP URL by matching against PATH_TO_URL_MAP prefixes. Mirrors getDODSURL() from ssds_util.pl.

3. get_git_url(script_name: str, version: str) -> str — Returns {GIT_WEB_BASE}/{version}/{script_name}. Replaces old CVS ?rev= pattern.

4. find_person_by_email(session, email, api_base) -> dict | None — GET /persons?email={email}, returns first match or None. Mirrors SSDS::PersonAccess->findByEmail().

5. _create_entity(session, endpoint, payload, api_base) -> dict — Generic POST helper. Used for Software, DataContainer, DataProducer, Resource creation.

6. _find_process_run_by_output(session, dods_url, api_base) -> dict | None — GET /dataproducers?output_uri={uri}. For idempotent upsert behavior.

Core function:

7. submit_process_run(...) — Main function, direct equivalent of submitDStoNetCDFProcessRun. Parameters:

   • nc_file_path: str — local path to output NetCDF file
   • input_uris: list[str] — input file URIs (OPeNDAP URLs or local paths)
   • poc_email: str — point of contact email (default: "mccann@mbari.org")
   • pr_start: str | None — ProcessRun start datetime (ISO 8601)
   • pr_end: str | None — ProcessRun end datetime (ISO 8601)
   • software_name: str — default "auv-python"
   • software_description: str — default "AUV data processing pipeline"
   • software_version: str | None — Git commit hash/tag, auto-detected if None
   • script_name: str — relative path to the processing script
   • cmd_line_args: str — command line that was executed
   • log_file_url: str | None — URL to the processing log
   • additional_resources: list[dict] | None — extra Resource dicts
   • api_base: str — configurable SSDS REST base URL
   • session: requests.Session | None — optional pre-configured session
   • logger: logging.Logger | None

   Internal flow (mirrors Perl):

   • a. Look up Person by email → return early with warning if not found
   • b. Build Resource list (script URL, processing log, commandline, additional)
   • c. Create Software entity via POST
   • d. Create output DataContainer with OPeNDAP URL
   • e. Find existing ProcessRun by output URI → PUT to update, or POST to create new
   • f. Link relationships: inputs, output, software, person, resources
   • g. Log the provenance URL

CLI entry point:

8. process_command_line() + if __name__ == "__main__": block — argparse CLI matching the parameters above, allowing standalone invocation.

Phase 2: Integrate into src/data/process.py

9. Add import (near other imports, ~line 75):

   • from provenance import submit_process_run, get_dods_url

10. Add _submit_provenance() helper method to Processor class (after create_products() method, ~line 710):

    • Parameters: output_nc, input_files, pr_start, pr_end, script_name, log_file
    • Checks output file exists, computes log URL via get_dods_url(), calls submit_process_run()
    • Wraps in try/except — provenance failures are logged as warnings, never block processing

11. Capture start time in process_log_file() (~line 1043, top of method):

    • Add self._pr_start_time = datetime.now(tz=UTC).isoformat() at the start

12. Call provenance at end of process_log_file() (~line 1082, before "Finished processing" log):

    • After create_products() call, invoke self._submit_provenance() with:
      • output_nc: the resampled _1S.nc file path
      • input_files: the original .nc4 log file path
      • pr_start: self._pr_start_time
      • pr_end: datetime.now(tz=UTC).isoformat()
      • script_name: "src/data/process_lrauv.py"
      • log_file: the _processing.log path

13. Capture start time in process_mission() (~line 790, top of method):

    • Add self._pr_start_time = datetime.now(tz=UTC).isoformat()

14. Call provenance at end of process_mission() full-pipeline branch (~line 857):

    • After create_products(mission) in the else branch, invoke self._submit_provenance() with:
      • output_nc: {auv_name}_{mission}_{FREQ}.nc
      • input_files: missionlogs path
      • pr_start: self._pr_start_time
      • pr_end: datetime.now(tz=UTC).isoformat()
      • script_name: "src/data/process_dorado.py"
      • log_file: {auv_name}_{mission}_processing.log

Phase 3: Add dependency

15. Add requests>=2.31.0 to pyproject.toml dependencies list (already used transitively by logs2netcdfs.py, gulper.py, sipper.py — this makes it explicit).

---

Relevant Files

• src/data/provenance.py — NEW: core module with submit_process_run() and helpers
• src/data/process.py — add import, _submit_provenance() method, calls in process_log_file() (~line 1082) and process_mission() (~line 857)
• pyproject.toml — add requests>=2.31.0 to dependencies
• src/data/processLOPC.pl — legacy Perl reference (read-only, for comparison)
• src/data/resample.py — defines AUVCTD_OPENDAP_BASE, LRAUV_OPENDAP_BASE constants (referenced for URL patterns)
• src/data/nc42netcdfs.py — defines BASE_LRAUV_PATH, BASE_LRAUV_WEB constants

---

Verification

1. Unit tests with mocked REST API — Use responses or pytest-httpserver to mock all endpoints. Verify:
   • Correct JSON payloads sent for each entity (Software, DataContainer, DataProducer, Resources)
   • Person lookup failure returns early without making further API calls
   • Existing ProcessRun is updated (PUT) rather than duplicated (POST)
   • Resources are correctly linked to the ProcessRun
   • Git URLs are correctly formed for each software reference
   • get_dods_url() correctly maps local paths for both LRAUV and Dorado data
2. Integration smoke test — Run against a staging SSDS REST API (or local mock server), submit a test ProcessRun, query back via GET to confirm all relationships exist
3. Compare output — For a known deployment, run both the Perl original and the Python script, compare the resulting SSDS metadata records
4. Failure isolation — Verify that a provenance submission failure (network error, API error) does not interrupt or fail the data processing pipeline

---

Decisions

• Module name: provenance.py (not submit_process_run.py) — shorter, clearer intent
• Scope: Only submitDStoNetCDFProcessRun + insertOrUpdateProcessRun logic. Plotting and data currency checks are excluded.
• CVS → Git: All software provenance URLs use Git web URLs via gitpython (already a dependency)
• REST style: Generic RESTful CRUD with raw requests.Session — no generated client or SSDS client library
• Idempotency: Preserves Perl behavior of finding existing ProcessRuns by output DataContainer URI and updating them
• Failure isolation: All provenance calls wrapped in try/except — failures logged as warnings, never block data processing
• PATH_TO_URL_MAP: Python equivalent of Perl %web_lookup hash, configurable dict mapping local prefixes to OPeNDAP prefixes
• requests dependency: Made explicit in pyproject.toml (was already used transitively)