A set of tools for controlling processing workflow with spiders and script running in scrapinghub ScrapyCloud.

Installation

pip install shub-workflow

If you want to support s3 tools:

pip install shub-workflow[with-s3-tools]

For google cloud storage tools support:

pip install shub-workflow[with-gcs-tools]

For Hubstorage Crawl Frontier (HCF) tools support (shub_workflow.utils.hcf):

pip install shub-workflow[with-hcf-backend]

Usage

Check Project Wiki for documentation. You can also see code tests for lots of examples of usage.

Claude Code plugin

shub-workflow ships a Claude Code plugin, shub-workflow-toolkit, that gives Claude working knowledge of shub-workflow tooling. It currently bundles ten skills:

• scanjobs-programs — authoring and running the scanjobs job-scanning + plotting tool and its command-line "programs".
• shub-workflow-scripts — writing or fixing scripts built on the shub_workflow.script base classes (BaseScript / BaseLoopScript / BaseLoopScriptAsyncMixin), i.e. any script that runs on or operates on Scrapy Cloud.
• shub-workflow-crawl-managers — building, updating or understanding crawl managers (CrawlManager / PeriodicCrawlManager / GeneratorCrawlManager / AsyncSchedulerCrawlManagerMixin): the set_parameters_gen() pattern, outcome/retry hooks, and async scheduling.
• shub-workflow-graph-managers — building, updating or understanding graph managers (GraphManager + Task / SpiderTask): declaring a DAG of tasks in configure_workflow(), dependency linking, on_finish/retry routing, resources, and parallelization.
• shub-workflow-monitors — building, updating or understanding monitors (BaseMonitor): cross-job stat aggregation over a time window, ratios, reports, custom checks, and threshold alerts via Slack / Sentry.
• shub-workflow-fshelper — reading/writing/listing files through the cloud-agnostic filesystem layer (shub_workflow.utils.futils / FSHelper): prefix-based s3:///gs:///local dispatch, role-assumed credentials, the listing variants, and op_kwargs/ACLs.
• shub-workflow-issuers — building data-pipeline issuers (IssuerScript / IssuerScriptWithFileSystemInput / IssuerScriptWithSCJobInput): reading job/file input, dedup, batch output slots, and issuer-based delivery (replacing the deprecated BaseDeliverScript).
• shub-workflow-hcf — working with a Hubstorage Crawl Frontier (shub_workflow.utils.hcf / FrontierTool): running and extending the frontier tool script (list/count/dump/add/delete/move slots), the HCF semantics that bite, and reading/writing seeds from crawl managers and consumers.
• shub-workflow-architectures — designing (or redesigning) a project's workflow architecture in an interactive session: interviews the developer about the requirements, maps them to the independent agents + role separation architecture, writes a design document, checks it against known pitfalls, and scaffolds the script skeletons of the chosen roles.
• shub-workflow-diagrams — generating a project's architecture diagrams from an editable model: banded workflow/pipeline diagrams and UML class-hierarchy diagrams, in the house style shared by these projects, with the rendering engine to copy and the Graphviz constraints that keep the output stable.

Install it from this repository's plugin marketplace, from inside Claude Code:

/plugin marketplace add scrapinghub/shub-workflow
/plugin install shub-workflow-toolkit@shub-workflow

To enable it automatically for a project, add it to that project's .claude/settings.json:

{
  "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
}

Updates

The plugin is unversioned (its plugin.json has no version field), so each commit pushed to this repository is a new version. When Claude Code installs the plugin it copies it into a local cache (~/.claude/plugins/cache/) and uses that copy — it does not read your working tree or re-pull from GitHub on every session. You choose how new commits reach you:

• Automatic. Turn on auto-update for this marketplace: run /plugin, open the Marketplaces tab, and enable auto-update for shub-workflow (or set it in settings — see below). With this on, Claude Code re-pulls the marketplace from GitHub and updates installed plugins at startup, so a new session always loads the latest pushed commit. This is the low-friction option for staying current.

  {
    "extraKnownMarketplaces": {
      "shub-workflow": {
        "source": { "source": "github", "repo": "scrapinghub/shub-workflow" },
        "autoUpdate": true
      }
    },
    "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
  }

• Manual. Leave auto-update off (the default for third-party marketplaces). The cached copy stays pinned until you explicitly update — nothing changes under you between sessions. To pull the latest when you want it:

  /plugin marketplace update shub-workflow            # refresh the catalog from GitHub
  /plugin update shub-workflow-toolkit@shub-workflow  # update the installed plugin
  

The plugin lives in plugins/shub-workflow-toolkit/; the marketplace manifest is .claude-plugin/marketplace.json.

Note

The requirements for this library are defined in setup.py as usual. The Pipfile files in the repository don't define dependencies. It is only used for setting up a development environment for shub-workflow library development and testing.

For developers

For installing a development environment for shub-workflow, the package comes with Pipfile and Pipfile.lock files. So, clone or fork the repository and do:

> pipenv install --dev
> cp pre-commit .git/hooks/

for installing the environment, and:

> pipenv shell

for initiating it.

There is a script, lint.sh, that you can run everytime you need from the repo root folder, but it is also executed each time you do git commit (provided you installed the pre-commit hook during the installation step described above). It checks code pep8 and typing integrity, via flake8 and mypy.

> ./lint.sh