Skip to content

fix: update logging levels and improve job URL handling in daily task - #83

Merged
TheTrueAI merged 1 commit into
mainfrom
fix/daily-digest-db
Mar 31, 2026
Merged

TheTrueAI merged 1 commit into
mainfrom
fix/daily-digest-db

Conversation

@TheTrueAI

Copy link
Copy Markdown
Owner

No description provided.

@TheTrueAI TheTrueAI self-assigned this Mar 31, 2026
Copilot AI review requested due to automatic review settings March 31, 2026 11:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces noisy HTTP client logging during the daily digest run and refactors daily task job URL handling so URLs map directly to the JobListing objects used for DB upserts and per-subscriber evaluation. It also adjusts tests to better isolate external clients (HTTPX + LLM) and ignores pytest temp directories.

Changes:

  • Set httpx/httpcore loggers to WARNING in daily_task.py to reduce log noise.
  • Refactor daily task URL→job handling to build url_to_job during search aggregation and generate upsert payloads from it.
  • Update tests to mock httpx.Client and create_client(); add pytest-of-* to .gitignore.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.

File Description
daily_task.py Lowers HTTPX logging verbosity; builds url_to_job earlier and uses it to generate DB upsert payloads.
tests/test_bundesagentur.py Mocks httpx.Client to avoid constructing real clients during provider search tests.
tests/test_app_ui.py Mocks immermatch.llm.create_client during Streamlit AppTest runs to prevent LLM client creation.
.gitignore Ignores pytest temp directories (pytest-of-*).

Comment thread daily_task.py
Comment on lines 115 to +120
# ── 5. Search once per unique (query-set, location) ──────────────────
# Collect all jobs keyed by title|company_name|location to avoid duplication
all_jobs: dict[str, JobListing] = {}
# Track which URLs belong to which normalized location
location_urls: dict[str, set[str]] = defaultdict(set)
url_to_job: dict[str, JobListing] = {}

Copilot AI Mar 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all_jobs is now only used for the log line + empty check, while the rest of the pipeline (upserts, DB ID lookup, per-subscriber unseen jobs) is driven by url_to_job/location_urls. This can make the "Found %d unique jobs" log misleading (it’s unique by title|company|location, not by URL), and the title/company/location dedupe no longer affects what actually gets processed. Consider either (a) removing all_jobs and basing the count/empty check on url_to_job, or (b) only populating url_to_job/location_urls for jobs that survive the all_jobs dedupe, and updating the log message accordingly.

Copilot uses AI. Check for mistakes.
@TheTrueAI
TheTrueAI force-pushed the fix/daily-digest-db branch from b539ef0 to 682f0bb Compare March 31, 2026 13:36
@TheTrueAI
TheTrueAI force-pushed the fix/daily-digest-db branch from 682f0bb to 050f388 Compare March 31, 2026 14:23
@TheTrueAI
TheTrueAI merged commit b47a9a9 into main Mar 31, 2026
4 checks passed
@TheTrueAI
TheTrueAI deleted the fix/daily-digest-db branch March 31, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants