Conversation
* chore(gemini): Add openai features into conditional attribute * Gemini json_schema config still refer to openai_common module item which derive PartialEq * Fixes breakage while running openai example Signed-off-by: dodyirawan85 <dev.irawans@gmail.com> * chore(openai): Reflect 343c471 changes into examples * fix run openai call Signed-off-by: dodyirawan85 <dev.irawans@gmail.com> --------- Signed-off-by: dodyirawan85 <dev.irawans@gmail.com>
…v2.47.3 Only classify 598/599 as ConnectError when the page is empty or has an error_status set. If a real server spoofs 598/599 with content, it falls through to ServerError instead. Added 8 tests covering all status branches. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…, bump to v2.47.4 Add AlibabaTMD variant to AntiBotTech enum with body detection (_____tmd_____, x5secdata) and URL detection (_____tmd_____/punish). Smart mode now checks anti-bot body signatures as an additional Chrome upgrade signal alongside DOM_SCRIPT_WATCH_METHODS. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…, bump to v2.47.5 Short-circuit detect_anti_bot_from_body via Rust || lazy eval so it only runs when rerender and script_src are both false. Added unit tests covering every AC_BODY_SCAN (8 patterns) and AC_URL_SCAN (16 patterns) index to prevent mapping regressions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- detect_anti_bot_from_body/detect_anti_bot_tech_response: &Vec<u8> → &[u8] eliminates .to_vec() heap alloc at every call site - is_cacheable_body_empty: replace full-body lowercase Vec alloc with in-place eq_ignore_ascii_case window matching (zero alloc) - crawl loops: drain().collect() into Vec instead of HashSet (skip rehashing) - extract_root_domain: replace Vec<&str> collect with byte counting + match_indices - content-type charset: iterate split() directly, drop intermediate Vec - build() error check: use .as_ref() instead of .to_string() clone - Replace 5x &Default::default() with &[] (avoid Vec::new heap alloc) All 178 tests pass. No mutexes, no deadlocks, no behavior changes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…, bump to v2.47.7 - external_domains_caseless: Box<HashSet> → Arc<HashSet> across Configuration, Page, and all call sites. Every crawl loop cloned this HashSet per-link via set_external(); now just an atomic refcount bump. Affects 8+ call sites in website.rs crawl paths. - Content-type charset parsing: remove .to_lowercase() String alloc per response — use eq_ignore_ascii_case on raw bytes instead. encoding_rs::Encoding::for_label() is already case-insensitive. All 178 tests pass. No mutexes, no deadlocks, no behavior changes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… alloc, bump to v2.47.8 Eliminates unnecessary heap allocations: Box<usize> → usize for redirect_limit, Box<String> → String for cookie_str, Box<Duration> → Duration for request_timeout. Also fixes second charset parsing site to use eq_ignore_ascii_case instead of to_lowercase(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates triple indirection (stack→Box→Vec→heap) down to double (stack→Box→heap). Saves 8 bytes per Page (no Vec capacity field) and one pointer chase on every access. Also makes encode_bytes and get_bytes use idiomatic &[u8] slices. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rror, bump to v2.47.11 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix serde Arc<T> support by adding "rc" feature to serde dependency. This fixes derive(Serialize, Deserialize) on Configuration struct's Arc<hashbrown::HashSet<CaseInsensitiveString>> field. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DNS failures now return status 525 and should_retry=false so none of the 13 retry loops in website.rs waste attempts on domains that will never resolve. Uses Aho-Corasick single-pass O(n) scan with zero-alloc downcast_ref fast path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…oxy support to agent, bump to v2.47.14 Fixes spider-rs#368: with_openai_compatible() now normalizes base URLs by appending /chat/completions when missing, so users can pass either full or base URLs. Fixes spider-rs#367: new with_proxy()/with_proxies() builder methods for HTTP/SOCKS proxy support on the agent's reqwest client. 14 new tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…, bump to v2.47.15 Fixes spider-rs#369: new with_client() to pass a pre-built reqwest::Client with full control over TLS/timeouts/proxies. New with_timeout(None) for infinite timeout. Agent::client() accessor for the underlying client. 5 new tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes spider-rs#370: new with_openai_responses() and with_openai_compatible_responses() builder methods for the stateful Responses API (/v1/responses). System messages map to `instructions`, user/assistant to `input` items, JSON mode uses `text.format`. Parses both `output_text` shorthand and full `output[]` array. Usage extraction handles both `prompt_tokens`/`input_tokens` keys. URL normalization is mode-aware — auto-strips /chat/completions when switching to Responses. Default remains Chat Completions (zero regression). OpenAI-only. 19 new tests, 215 total passing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ompletions, bump to v2.47.17 Default is now OpenAiApiMode::Auto: detects from URL at request time. api.openai.com → Responses API, everything else → Chat Completions. Users can still force either mode via with_completions_api() or with_responses_api(). Zero perf overhead (single starts_with check). 38 openai tests, 226 total passing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
crawl_timeout was only checked between pages in the tokio::select! loop guard via crawl_duration_expired(). This meant: - In-flight page fetches continued past the timeout - Single-page crawls had no effective timeout - Tarpitting targets could hold connections open indefinitely Wrap the work portion of all public crawl methods (crawl, crawl_raw, crawl_smart, crawl_sitemap, crawl_sitemap_chrome, crawl_raw_send, crawl_chrome_send) with tokio::time::timeout using crawl_timeout. When crawl_timeout is None, no timer is created (zero overhead). Cleanup (set_crawl_status, abort join_handle, replace client) always runs regardless of timeout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… mismatch Adds content_truncated field to PageResponse and Page. Both handle_response_bytes and handle_response_bytes_writer now detect stream errors, chunk idle timeouts, and Content-Length vs actual bytes mismatches. Bump all crates to 2.47.19. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace .unwrap()/.expect() with proper error propagation in solvers (EvaluateParams, CallFunctionOnParams builders), use safe .get() slicing in strip_framework_data_attrs, clamp robotparser crawl-delay float conversion, add response body fallbacks in spider_worker. Auto-retry truncated responses in page.rs and utils/mod.rs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… to v2.47.22 Replace tokio-uring (pre-1.0, forces separate runtime) with the raw io-uring 0.7 crate. Adds kernel probe at init — fails gracefully on AWS Amazon Linux, ECS, Lambda, seccomp-filtered containers, and kernels < 5.1. Worker thread runs a synchronous io_uring submit/reap loop with no async runtime, no mutexes, and no deadlock paths. - Swap dep: tokio-uring 0.5 → io-uring 0.7 - Rewrite uring_fs.rs io_uring inner: raw SQE/CQE ops (OpenAt, Write, Read, Close) with short-write/short-read loops and guaranteed fd cleanup - Unify connect.rs init_background_runtime into single impl (both paths use standard tokio runtime; io_uring only for file I/O) - Remove .expect() panic site in send_to_background_runtime - StreamingWriter always uses tokio::fs fallback (io_uring adds no value for sequential streaming) - 18 tests: 13 cross-platform fallback + 5 Linux-only io_uring path Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Auth page cli
|
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
This PR was closed because it has been stalled for 10 days with no activity. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.