feat(actress-sync): add metadata sync system, cache-first resolution, and actress filters - #176
Open
seeyabye wants to merge 120 commits into
Open
feat(actress-sync): add metadata sync system, cache-first resolution, and actress filters#176seeyabye wants to merge 120 commits into
seeyabye wants to merge 120 commits into
Conversation
…tress filters - Actress sync manager with lease-fenced transactional task processing - Built-in actress cache (25k records) with alias-aware build merging - Cache-first DMM ID resolution with post-live fallback for selected sync - Transactional merge with atomic task reassignment and dedupe coalescing - Scraper session-based thumbnail validation with per-redirect SSRF checks - Actress filters: missing/has DMM ID, missing thumbnail, Japanese name only - Fix cross-page selection persistence - Enrich scraped movie actresses from built-in cache with DMM ID guard - Full i18n support (en, en-XA, ja, zh-Hans, zh-Hant)
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #176 +/- ##
==========================================
+ Coverage 91.84% 92.90% +1.06%
==========================================
Files 461 490 +29
Lines 41877 47123 +5246
==========================================
+ Hits 38461 43779 +5318
+ Misses 2224 2174 -50
+ Partials 1192 1170 -22
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
- Regenerate mocks with mockery v3.7.1 to match CI - Skip os.FileMode permission assertions on Windows (no Unix permissions)
Contributor
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 409cead666
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
github-actions
Bot
added
the
🤖 codex: needs work
- Preserve existing ThumbURL for JavDB resolver so it can extract actor ID - Only fetch task list when sync job reaches terminal state, not on every 1.5s poll
github-actions
Bot
removed
the
🤖 codex: needs work
Contributor
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0675393d53
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
github-actions
Bot
added
the
🤖 codex: needs work
Only clear actress thumbnails on deterministic known-invalid placeholders. Remote validation failures (transient network errors, unreachable CDNs) no longer trigger needsPersistence, preventing permanent erasure of valid URLs.
github-actions
Bot
removed
the
🤖 codex: needs work
Contributor
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41937d66aa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
github-actions
Bot
added
the
🤖 codex: needs work
- Test wrapper functions without task context - Test CancelJob with pending task for correct status transition
- Only clear ThumbURL on deterministic IsKnownInvalidDMMActressThumbnail - Remote validation failures no longer erase valid URLs - Remove query string exclusion from thumbnail filter SQL - Update test to use known-invalid DMM thumbnail
github-actions
Bot
removed
the
🤖 codex: needs work
- Add tests for all sync_jobs API endpoints (create, list, get, tasks, cancel) - Test validation errors, not found cases, and helper functions - Fix P1: don't clear thumbnails on transient validation failures - Fix P2: remove query string exclusion from thumbnail filter SQL
Contributor
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1209f8e96f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
github-actions
Bot
added
the
🤖 codex: needs work
- TableName, SplitFullName, NormalizeActressNameKey in models - ValidateRemoteImage, ValidateRemoteImageWithSafeClient in imageutil - RenameNameFields, ReplaceThumbnailForSyncTask, AssignDMMIDIfMissingForSyncTask in database - ListActresses in r18devdump - newScraper, Name, IsEnabled, Config, Close, GetURL, validateScraperSettings, Register in minnanoav - RegisterAll in scraper registration - ValidateActressThumbnail, ResolveActressThumbnail, ResolveActressMetadata, fetchActressMetadataDoc in dmm - ValidateActressThumbnail in javdb and r18dev - listActressSyncCandidates, RegisterRoutes in api/actress - ListActiveJobs on ActressSyncManager - Name() in legacycsv source
… SSRF at connection time - Reuse ListSyncCandidates results to avoid N+1 FindByID queries - Log and retry dispatcher startup on transient database errors - Wrap scraper client transport with SSRF check at connection time
github-actions
Bot
removed
the
🤖 codex: needs work
- actressMetadataVerified: all branches (nil, no match, DMM mismatch, name/thumb/first-last match) - actressNeedsMetadata: nil, empty, complete cases - actressThumbnailNeedsResolution: empty, invalid, valid - actressThumbnailSourcePriority: all scraper names - scraperThumbnailCanRefresh: all host patterns - authoritativeActressScrapers/actressMetadataScrapers: nil interface - actressIdentityNames: parentheses and comma parsing - identityNameMatches: case insensitive, empty - canMergeMissingDMMActress: all edge cases - lookupActressCache: nil inputs - cacheFallbackMatch: with and without existing matches - needsLinkedActressFallback: complete and incomplete
- worker: cancel in-flight sync task contexts on job cancellation - frontend: reconcile sync queue against server truth, advance past pruned jobs - api: make merge preview timestamps optional (backward-compatible unfenced merge) - api: declare MergeWithVersions on ActressRepositoryInterface - actresscache: gate image decode on bounded header dimensions
- worker: cancel wins over benign sync outcomes; timeouts settle as failed; bound cancellation bookkeeping to per-task entries - actresscache: cancelled minnanoav crawl reports error, never marks complete - actresscache: block SSRF to internal hosts on fetch and redirect hops
- actresscache: pin dialed connections to resolved public IPs (DNS rebinding) - actresscache: keep 10-redirect cap under custom CheckRedirect - worker: never persist running state on shutdown-completed tasks - frontend: keep finished queued jobs; pruned jobs drop via 404 on advance - api: swagger annotations for sync endpoints and actress filter param
- actresscache: block CGNAT/IETF special-use ranges, DNS-validate proxied hosts - worker: guard task untracking by run id (retry re-registration race) - database: fence CompleteTask on job cancellation inside its transaction - javdb: reject ambiguous exact-name actor matches
- actresscache: fail closed for unresolvable hosts behind a proxy; dial all resolved IPs instead of only the first - scrape: per-scraper scrape_actress override beats the global flag - database: skip no-op fills (no updated_at churn); attribute fields from in-transaction state, not a pre-tx snapshot - frontend: refresh failures no longer block terminal queue advancement
- worker: persist no outcome on manager shutdown; recovery requeues instead - api: sync-candidates swagger description matches actual dmm_id<=0 rows - frontend: toast and abort on empty-queue refresh failure when advancing
- sync: honor configured scraper priority/orders and actress field priority for resolver order, name blending, and thumbnail picks; capability-gate per-field picks via optional ActressFieldCapable - scrape: blend actress enrichment per field by configured priority instead of first-resolver-wins; keep capability gating and early-exit - api: bound the default sync task list (limit param, default 500, cap 1000) - actresscache: trust the configured proxy's own address at dial time - frontend: hide actress-only resolvers from movie field editors - frontend: reconcile selections against refreshed queries after merges
- scrape: thread progressively discovered values into later resolvers; honor actress field overrides exclusively and suppress on the skip sentinel - worker: deterministic rank-based actress resolution when priorities are configured; same-rank ties still surface conflicts; exclusive actress overrides restrict sync resolvers too - actresscache: probe HTTP and HTTPS proxies separately for the diverged proxy allowlist - priority UI: actress-only resolvers are never valid movie-field sources; enforce in the field editor, its initial list, and add-all
- actresscache: probe the proxy route per request scheme (HTTP vs HTTPS) - worker: keep linked-movie fallback provenance so configured priority resolves cross-source disagreements deterministically - priority UI: never rewrite the stored field list on editor open
- database: fence task mutations on parent job cancellation (lease validity alone allowed post-cancel commits) - scrape: explicit scraper selection is exclusive for actress enrichment - priority UI: actress-only resolvers stay configurable for the actress field, but a solely-actress-only override is rejected at save
- database: fence task-scoped merges (reassignTaskActressTx) on job cancel - scrape: only explicit selections make enrichment resolver set exclusive
- commandutil: refcount the r18dev dump handle so hot-reload closes it only after in-flight lookups drain - scrape: URL-derived PriorityOverride must not make enrichment exclusive; only caller-supplied selections do - minnanoav: honor explicit rate_limit: 0 (no delay) per documented contract
- database: skipped coalescing winners free their dedupe key, preventing unique-index rollbacks during merge migration - scrape: cached path treats only caller selections as exclusive overrides, matching the fresh path - worker: thread progressively discovered actress values into later resolvers - scraperconfig: defaults merge no longer clobbers explicit rate_limit: 0 - commandutil: Close blocks until in-flight dump lookups drain (Windows-safe) - actresscache: document the accepted proxy-side resolution residual
- database: dedupe/winner joins exclude cancel-requested jobs; createActressSync supersedes a cancelling task's key so retries stay runnable - config: YAML + JSON decode track rate_limit presence incl. request_delay; marshalling omits inherited zeros so round trips persist nothing synthesized
- database: free a cancelling task's canonical key before merge migration - worker: clear resolver-reported fields the source does not advertise - database: sync-candidate prefilter matches only real placeholder thumbs
- worker: skip/exclusive actress overrides apply to identity recovery and linked-movie fallback scrapers too - api: reject actress-only-sole metadata.priority.actress overrides on save - config: legacy request_delay/max_retries aliases no longer clobber explicitly configured canonical keys
- database: cancelling running sources no longer displace pending canonical winners during merge migration - core: warn loudly (boot and reload) when the actress priority override exclusively lists actress-only resolvers
- database: skip-winner merge migration requires the canonical key holder, so a deferred winner never collapses a merge with the running holder's key - models: ResolveActressMetadata now returns (ActressInfo, error) so transient scraper failures surface as task warnings instead of silent skips
- worker: linked-movie identity recovery propagates scraper errors; a task fails for retry when no usable match survives an outage - javdb: findActorID surfaces fetch/parse failures instead of "no actor" - minnanoav: search rejects non-2xx HTTP status (Resty does not by default)
- worker: catalog misses and partial outages no longer fail linked-movie identity recovery; only a full outage with nothing usable does
- minnanoav: redirect policy allows only minnano-av.com hosts (SSRF) - worker: dispatch loop rechecks shutdown before each claim; job cancel sweeps every in-flight run of a task - database: actress delete cancels then detaches sync tasks (SQLite FK enforcement is off in production)
- database: actress delete refreshes parent job aggregates - database: merges re-anchor terminal sync tasks to the canonical actress - frontend: reset the poll-failure toast latch when advancing sync jobs
- database: merge re-anchors terminal tasks before deleting the source, working under both FK-on and FK-off SQLite - worker: CreateJob skips merge-deleted selection IDs instead of rejecting - api/core: capability validation covers every metadata priority field, on save (hard reject) and boot/reload (warnings) - frontend: reset the poll-failure latch on direct job assignment paths too
- actresscache: DMM-anchored merge candidates also pass Japanese-name conflict checks when the target group has no DMM anchor - actresscache: block IPv6 transition/site-local/doc ranges (6to4, Teredo, fec0::/10, 2001:db8::/32) - actresscache: cap decoded thumbnails at 20 MP (~80 MB RGBA) instead of 100 MP to bound parallel-builder memory
- worker: romanized identity names join linked-movie recovery matching - database: a deferred merge winner absorbs redundant duplicate tasks instead of queueing a second run - api: tasks endpoint reports the real per-view total, not the page length
- database: deferred coalescing no longer reports "cancelled" for an un-cancelled job's running task - core: compute actress-priority warnings once per reload
github-actions
Bot
removed
the
🤖 codex: needs work
codecov ignore list; also fix some defensive-callarms for easier auditing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a comprehensive actress metadata sync system with a built-in cache, server-side filtering, and several UX improvements to the actresses page.
Key Features
Actress Metadata Sync System
Actress Filters
Cross-Page Selection Fix
Scrape Cache Enrichment
Full i18n Support
Testing