feat: M2 (part 2) — RSS poller, backfill, enrichment, scheduler

- Free per-channel RSS reader for quota-less fresh-video detection
- Recent-first backfill (configurable: 100 videos / 1 year) plus resumable deep
  backfill from the uploads playlist
- Enrichment via videos.list: duration, view/like counts, category, topics,
  language, Shorts heuristic and livestream/premiere classification
- Reusable sync runners + APScheduler jobs (rss / enrich / backfill), all
  quota-aware with a reserve so backfill never starves fresh enrichment
- Manual triggers: POST /api/sync/{rss,backfill,enrich}
- Exact insert counting via RETURNING with in-batch de-duplication
This commit is contained in:
npeter83 2026-06-11 01:36:41 +02:00
parent 24b6e0026d
commit cff1d23071
10 changed files with 569 additions and 4 deletions

View file

@ -31,9 +31,23 @@ class Settings(BaseSettings):
# Daily quota budget (units). Default leaves headroom under the 10,000/day free limit.
quota_daily_budget: int = 9000
# Recent-first backfill: how far back to fetch on the first pass per channel.
backfill_recent_max_videos: int = 200
backfill_recent_max_videos: int = 100
backfill_recent_max_days: int = 365
# Videos at or below this duration (seconds) are treated as Shorts.
shorts_max_seconds: int = 60
# videos.list accepts up to 50 ids per call.
enrich_batch_size: int = 50
# --- Background scheduler ---
scheduler_enabled: bool = True
rss_poll_minutes: int = 20
enrich_interval_minutes: int = 3
backfill_interval_minutes: int = 10
# Keep this many quota units in reserve so scheduled backfill never starves
# interactive syncs / enrichment of fresh videos.
backfill_quota_reserve: int = 2000
@property
def allowed_email_set(self) -> set[str]:
return {e.strip().lower() for e in self.allowed_emails.split(",") if e.strip()}