Repository navigation
fix: update logging levels and improve job URL handling in daily task - #83
Conversation
There was a problem hiding this comment.
Pull request overview
This PR reduces noisy HTTP client logging during the daily digest run and refactors daily task job URL handling so URLs map directly to the JobListing objects used for DB upserts and per-subscriber evaluation. It also adjusts tests to better isolate external clients (HTTPX + LLM) and ignores pytest temp directories.
Changes:
- Set
httpx/httpcoreloggers toWARNINGindaily_task.pyto reduce log noise. - Refactor daily task URL→job handling to build
url_to_jobduring search aggregation and generate upsert payloads from it. - Update tests to mock
httpx.Clientandcreate_client(); addpytest-of-*to.gitignore.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
daily_task.py |
Lowers HTTPX logging verbosity; builds url_to_job earlier and uses it to generate DB upsert payloads. |
tests/test_bundesagentur.py |
Mocks httpx.Client to avoid constructing real clients during provider search tests. |
tests/test_app_ui.py |
Mocks immermatch.llm.create_client during Streamlit AppTest runs to prevent LLM client creation. |
.gitignore |
Ignores pytest temp directories (pytest-of-*). |
| # ── 5. Search once per unique (query-set, location) ────────────────── | ||
| # Collect all jobs keyed by title|company_name|location to avoid duplication | ||
| all_jobs: dict[str, JobListing] = {} | ||
| # Track which URLs belong to which normalized location | ||
| location_urls: dict[str, set[str]] = defaultdict(set) | ||
| url_to_job: dict[str, JobListing] = {} |
There was a problem hiding this comment.
all_jobs is now only used for the log line + empty check, while the rest of the pipeline (upserts, DB ID lookup, per-subscriber unseen jobs) is driven by url_to_job/location_urls. This can make the "Found %d unique jobs" log misleading (it’s unique by title|company|location, not by URL), and the title/company/location dedupe no longer affects what actually gets processed. Consider either (a) removing all_jobs and basing the count/empty check on url_to_job, or (b) only populating url_to_job/location_urls for jobs that survive the all_jobs dedupe, and updating the log message accordingly.
b539ef0 to
682f0bb
Compare
682f0bb to
050f388
Compare
No description provided.