feat(crawl): make --depth optional; defaults are 100 pages and 100 items

[gonzalo-zenrows]

Summary

zenrows crawl create no longer requires --depth. Without it the CLI sends no depth, and the crawl follows links until --max-pages or --max-items stops it. The documented defaults for both caps go from 10 to 100. Stacked on #31: merge after it.

What the task asked for

The Crawl API changes POST /v1/crawls in three ways:

• depth is optional. Left out or null, the crawl has no depth limit. It follows links until max_pages or max_items stops it, or it runs out of links to open. An explicit depth is still 1 to 100,000. Anything else gets 422 invalid_parameter.
• max_pages and max_items default to 100 instead of 10. The caps stay at 100,000.
• A crawl created without depth comes back from create, get and list with no depth field. A crawl created with one still echoes it.

How the check confirmed it

npm run typecheck is clean and npm test passes 352 of 352. One new test checks that a create without --depth sends only url and prints a crawl with no depth. Another checks that createCrawl leaves depth out when unset. A third counts an unset --max-pages as 100 against max_pages_per_run. The offline half of acceptance check 1 now also creates without --depth, and check 21 expects 100.

How to try it

npm run dev -- crawl --help
npm run dev -- crawl create https://www.scrapingcourse.com/ecommerce/ --max-pages 5 --json

The second command sends no depth, and once the API change is live the crawl comes back with none.

Provenance

• The command sends depth only when set, the way the command already treats --max-items and --max-pages.
• DEFAULT_MAX_PAGES copies the API default for the local policy check, so it follows the API to 100.

Review

Pass	Model	Findings
1, review of the diff before opening	AI assistant	None

Human must verify:

• The API change is live before acceptance check 21 runs: until then it reads 10 and fails.

Record owed

None in this repo. The changelog entry for the new defaults ships with the API.

Deviations

• With a max_pages_per_run policy below 100, a create without --max-pages is now refused locally with POLICY_LIMIT_EXCEEDED, since the API would fetch up to 100 pages.
• README and skill examples keep --depth 1: it is the cheap advice, and the acceptance check that runs the README examples depends on it.

[StefanoGuerrini]

Folded into #31, which now carries this change, so the crawl command ships as one pull request.

---

Generated by Claude Code