refactor: Enhance package dependency structure with PEP-compliant boundaries

[eugengi]

Description

This PR refactors the package dependencies to align them with PEP 621 and PEP 735. By default, the package includes all local or non-runtime dependencies in the table [project.optional-dependencies]. optional-dependencies is published package metadata for extras.

Note

Extras are for optional package (runtime) functionality that ships with the distribution and becomes part of the public install contract.

PEP 621 also explicitly rejects the idea of recommending a dev extra for development-related dependencies. Therefore, including dev, examples, test, docs, etc. (internal tooling) under optional-dependencies is an anti-pattern. The solution to this is dependency-groups (PEP 735) for dependency sets that are not published in built distributions. This contrasts with extras or optional-dependencies, which are for consumer-facing, installable features that users can request if needed.

Decisions

This section outlines notable considerations for developers/maintainers.

• Granular group sets have been used to create slim environments with only the dependencies needed (i.e., lint, test, examples-s3, etc.)
• Use PEP 735’s include-group mechanism for composability and aggregate group sets (i.e., dev, examples)
• Only otel has been retained under optional-dependencies as a valid extra exposed in the packages install contract: claude-agent-sdk-python[otel]

Additional

Important

This update requires pip minimum version 25.1.0, which supports installing dependency-groups: pip install --group <group-set>

• Several pip install invocations, particularly in GHA workflows and Markdown files, have been updated to match the above

References

• PEP 621 – Storing project metadata in pyproject.toml
• PEP 735 – Dependency Groups in pyproject.toml
• Extras | Dependency Specifiers – PyPA specifications, Overview of Python Packaging
• Dependency Groups – PyPA specifications, Overview of Python Packaging