Skip to content

Validate checkpoint callback identity#308

Open
3em0 wants to merge 1 commit into
D4Vinci:mainfrom
3em0:fix/checkpoint-callback-identity
Open

Validate checkpoint callback identity#308
3em0 wants to merge 1 commit into
D4Vinci:mainfrom
3em0:fix/checkpoint-callback-identity

Conversation

@3em0
Copy link
Copy Markdown

@3em0 3em0 commented May 31, 2026

Proposed change

Validate checkpoint callback identity before restoring request callbacks. Checkpointed requests now store callback name, qualname, module, owner class, and owner module, and restore rejects missing or mismatched callback identities instead of silently binding a request to another spider's same-named method.

This prevents checkpoint resume from dispatching a preserved request/response to the wrong spider callback when different spiders share callback names.

Type of change:

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests
  • Add or change doctests? -- Note: Please avoid changing both code and tests in a single pull request.
  • Documentation change?

Additional information

Checklist:

  • I have read CONTRIBUTING.md.
  • This pull request is all my own work -- I have not plagiarized.
  • I know that pull requests will not be merged if they fail the automated tests.
  • All new Python files are placed inside an existing directory.
  • All filenames are in all lowercase characters with no spaces or dashes.
  • All functions and variable names follow Python naming conventions.
  • All function parameters and return values are annotated with Python type hints.
  • All functions have doc-strings.

Testing

  • python -m pytest tests/spiders/test_request.py tests/spiders/test_engine.py::TestCheckpointMethods tests/spiders/test_templates.py::TestCrawlSpiderPickle tests/spiders/test_sitemap.py::TestSitemapSpiderPickle -q
  • python -m pytest tests/spiders -q

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Checkpoint callback restoration can bind requests to the wrong spider callback

1 participant