Skip to content

[Feature] Scheduled GitHub source drift-check job (#1176) - #1184

Merged
chronoai-shining merged 9 commits into
developfrom
feature/1176-source-drift-scheduler
Jul 1, 2026
Merged

[Feature] Scheduled GitHub source drift-check job (#1176)#1184
chronoai-shining merged 9 commits into
developfrom
feature/1176-source-drift-scheduler

Conversation

@chronoai-shining

@chronoai-shining chronoai-shining commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Phase 1 of 5 for automatic sync of GitHub-sourced skills (Depends on #1175, now merged). Adds the scheduled inbound drift-check job — the pull-based baseline that discovers when a linked upstream repo has moved, with no manual trigger. It records drift state only; automatic re-publish is #1177.

What this does

  • Multi-pod-safe Agenda scheduler (sourceSyncScheduler.ts) cloned from the mirror scheduler's proven pattern (per-fire row lock on agendaJobs). Cadence from settings.sourceSync.pollSchedule (Asia/Singapore); a 1-minute tick re-registers it cluster-wide on an admin cadence change.
  • Batch job (sourceDriftJob.ts) — enumerates github-sourced skills due for a check, coalesces by (repo, ref) so a shared upstream costs exactly one probe, and persists each skill's verdict against its own lastSyncedCommit. Bounded concurrency + jitter; pinned 40-hex refs skipped.
  • Resilience — a broken source (404/private/deleted) marks the skill broken and notifies its owner (new skill.source_broken notification); a GitHub rate-limit signal (typed GitHubRateLimitError from 429/403+Retry-After) short-circuits the rest of the tick; no single skill's probe/persist/notify failure aborts the run.
  • Repository — findGithubSourcedSkills({ notCheckedSince }) (excludes pinned refs + freshly-checked skills, returns { guid, source, ownerId }), backed by a partial index over github sources by lastCheckedAt.
  • Bootstrap — started next to the mirror scheduler (start-failure logs, never crashes boot); stopped first on shutdown.

Testing

  • bun test src/domains/skills/crud src/domains/notifications src/domains/settings → 649 pass, 0 fail.
  • New unit tests: coalescing (N skills → 1 probe), pinned-SHA skip, broken→notify + loop-continues, rate-limit short-circuit + skipped count, disabled→no-op, transient-error skip, per-skill persist-failure resilience; scheduler register/re-register/cancel/status (mocked Agenda); classifyProbeResult + pickSourceSyncToken; rate-limit classification; and repository query + dot-path setter against real mongodb-memory-server.
  • End-to-end smoke against real MongoDB + real GitHub (run locally, not committed): 2 coalesced groups, Ornn@develop probed exactly once for two skills, stale commit → drifted (real HEAD persisted), real HEAD → in_sync, missing repo → broken + notified once, counts checked=2/drifted=1/broken=1.
  • tsc --noEmit clean; eslint clean.

Commits (dependency-ordered, buildable per commit)

  1. feat(api) typed GitHubRateLimitError
  2. refactor(api) share classifyProbeResult + pickSourceSyncToken
  3. feat(api) repository query + setter + partial index
  4. feat(api) skill.source_broken notification
  5. feat(api) source-drift batch job (coalesce + backoff)
  6. feat(api) source-sync Agenda scheduler
  7. feat(api) wire scheduler into bootstrap
  8. docs changeset

Closes #1176

@chronoai-shining
feat(api): typed GitHubRateLimitError for 403/429 signals (#1176)
3a54e05 
resolveRefHeadSha now classifies a GitHub rate-limit response (429, or 403 with X-RateLimit-Remaining:0 / a Retry-After header) into a typed GitHubRateLimitError carrying the recommended wait (from Retry-After or X-RateLimit-Reset). Other non-OK statuses stay generic 'transient' errors. This lets the batch drift scheduler back off cleanly instead of hammering a limited API.

Part of #1176
@chronoai-shining
refactor(api): share classifyProbeResult + pickSourceSyncToken (#1176)
17c8ff7 
Extract the probe-result-to-verdict mapping (classifyProbeResult) and the settings>env>anonymous token resolution (pickSourceSyncToken) out of the single-skill drift path so the upcoming batch job can reuse them verbatim — the batch probes once per (repo,ref) group and classifies each member against its own lastSyncedCommit. No behavior change: runSourceDriftCheck and SkillService.resolveSourceSyncToken now delegate to the shared helpers.

Part of #1176
@chronoai-shining
feat(api): source-drift repository query, setter, and index (#1176)
e9f6d08 
Add findGithubSourcedSkills({ notCheckedSince }) — enumerates github-sourced skills due for a check (excluding pinned 40-hex refs via a $not-regex, and skills checked more recently than the cutoff), returning light { guid, source, ownerId } rows. Back it with a partial index over github sources ordered by lastCheckedAt. Extract the source-coercion out of mapDoc into a shared coerceSkillSource helper so both read paths stay consistent.

Part of #1176
@chronoai-shining
feat(api): skill.source_broken owner notification (#1176)
441145a 
Add the skill.source_broken category to the canonical NOTIFICATION_CATEGORIES vocabulary (so the boot migration's allow-list can't wipe it) and a notifySourceBroken() method that tells a skill's owner their linked GitHub source could not be resolved, deep-linking to the skill so they can re-link it.

Part of #1176
@chronoai-shining
feat(api): source-drift batch job (coalesce + rate-limit backoff) (#1176
9352cc6 
)

runSourceDriftJob enumerates due github-sourced skills, coalesces them by (repo,ref) so a shared upstream costs exactly one probe, and persists each skill's drift verdict. A broken source marks the skill broken and notifies its owner; a rate-limit signal short-circuits the rest of the tick (the next fire retries); pinned refs are skipped; probes run under a bounded concurrency with optional jitter. No single skill's persist/notify/probe failure is allowed to abort the run. Records state only — no publish (that is #1177).

Part of #1176
@chronoai-shining
feat(api): source-sync Agenda scheduler (#1176)
3f18604 
createSourceSyncScheduler clones the mirror scheduler's multi-pod-safe Agenda pattern (per-fire row lock on agendaJobs) for the inbound drift job: a source-drift-check job whose cron cadence is driven by settings.sourceSync.pollSchedule (Asia/Singapore), plus a 1-minute sync tick that re-registers it cluster-wide when an admin changes the cadence. Exposes getScheduledRunStatus() for parity with the mirror scheduler.

Part of #1176
@chronoai-shining
feat(api): wire source-sync scheduler into bootstrap (#1176)
c23d8ba 
Construct + start the source-sync scheduler next to the mirror scheduler, injecting the drift job with the skill repo, settings, env token fallback, and the notification service. A start failure logs and leaves this pod without the scan rather than crashing boot; shutdown stops it before the mirror scheduler.

Part of #1176
@chronoai-shining
docs: changeset for #1176 (scheduled source drift-check job)
f4d7f09
@chronoai-shining
Merge remote-tracking branch 'origin/develop' into feature/1176-sourc…
415a3bf 
…e-drift-scheduler
@chronoai-shining
chronoai-shining merged commit daba5e9 into develop Jul 1, 2026
17 checks passed
@chronoai-shining
chronoai-shining deleted the feature/1176-source-drift-scheduler branch July 1, 2026 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature] Scheduled GitHub source drift-check job (auto-detect upstream changes)

1 participant

@chronoai-shining