Skip to content

Trigger Bagit Metadata Extraction Job on Project Publication with Downloadable Archive #1512

Description

@NishaSharma14

Is your feature request related to a problem? Please describe.

When a project is made public, we need to automatically trigger the Bagit metadata extraction job to generate a comprehensive archive that includes all metadata, spectra snapshots, and structured data. Currently, users don't have a convenient way to download this complete archive from the public sample page.

Error screenshot

Describe the solution you'd like

Implement an automated workflow that triggers when a project is made public:

  1. Trigger the Bagit Metadata Extraction Job - Automatically dispatch ProcessMetadataExtractionBagitGenerationJob when a project's is_public status changes to true

  2. Job Processing - The job should:

    • Extract metadata from the study
    • Parse NMRium information from the spectra data
    • Generate a Bagit object following BagIt specification
    • Generate Bioschema JSON for structured data representation
    • Generate snapshot images of the spectra
    • Package all data as a ZIP file
  3. Storage - Save the generated ZIP file in Ceph storage:

    • Location: nmrXiv bucket under folder named "archive"
    • Path structure: archive/{study_identifier}/{study_identifier}.zip
  4. Database Updates - Add a new column to the studies table:

    • Column name: archive_link or bagit_archive_url
    • Type: longText or string(2048) (nullable)
    • Purpose: Store the downloadable link to the Bagit ZIP file
  5. UI Integration - Provide a downloadable link on the public Sample page:

    • Display a "Download Archive" or "Download Bagit Package" button
    • Link directly to the ZIP file stored in Ceph
    • Make it accessible to anyone viewing the public sample page

Describe alternatives you've considered

  • Manual triggering of the extraction job via command line
  • Generating archives on-demand when requested (might cause delays)
  • Storing archives in local storage instead of Ceph (less scalable)

Additional context

Relevant Code References:

The Bagit metadata extraction job already exists and handles most of the processing:

Database Schema:
The studies table already has columns for tracking the job status:

Implementation Tasks:

  1. Create a migration to add archive_link column to studies table
  2. Add event listener or observer for study publication events
  3. Modify the job to upload final ZIP to Ceph storage at archive/ folder
  4. Update the job to save the Ceph URL to the new archive_link column
  5. Update public sample page UI to display download link when archive is available
  6. Add appropriate permissions and access controls for the download endpoint

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

featuresomething which is not there

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions