feat: M2 (part 2) — RSS poller, backfill, enrichment, scheduler
- Free per-channel RSS reader for quota-less fresh-video detection
- Recent-first backfill (configurable: 100 videos / 1 year) plus resumable deep
backfill from the uploads playlist
- Enrichment via videos.list: duration, view/like counts, category, topics,
language, Shorts heuristic and livestream/premiere classification
- Reusable sync runners + APScheduler jobs (rss / enrich / backfill), all
quota-aware with a reserve so backfill never starves fresh enrichment
- Manual triggers: POST /api/sync/{rss,backfill,enrich}
- Exact insert counting via RETURNING with in-batch de-duplication
This commit is contained in:
parent
24b6e0026d
commit
cff1d23071
10 changed files with 569 additions and 4 deletions
|
|
@ -31,9 +31,23 @@ class Settings(BaseSettings):
|
|||
# Daily quota budget (units). Default leaves headroom under the 10,000/day free limit.
|
||||
quota_daily_budget: int = 9000
|
||||
# Recent-first backfill: how far back to fetch on the first pass per channel.
|
||||
backfill_recent_max_videos: int = 200
|
||||
backfill_recent_max_videos: int = 100
|
||||
backfill_recent_max_days: int = 365
|
||||
|
||||
# Videos at or below this duration (seconds) are treated as Shorts.
|
||||
shorts_max_seconds: int = 60
|
||||
# videos.list accepts up to 50 ids per call.
|
||||
enrich_batch_size: int = 50
|
||||
|
||||
# --- Background scheduler ---
|
||||
scheduler_enabled: bool = True
|
||||
rss_poll_minutes: int = 20
|
||||
enrich_interval_minutes: int = 3
|
||||
backfill_interval_minutes: int = 10
|
||||
# Keep this many quota units in reserve so scheduled backfill never starves
|
||||
# interactive syncs / enrichment of fresh videos.
|
||||
backfill_quota_reserve: int = 2000
|
||||
|
||||
@property
|
||||
def allowed_email_set(self) -> set[str]:
|
||||
return {e.strip().lower() for e in self.allowed_emails.split(",") if e.strip()}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue