feat(scheduler): report progress from every job + parallelize RSS polling

Every background job now reports progress, so the dashboard can show a live
bar for any run (not only enrich/backfill/shorts/maintenance): rss_poll and
subscriptions (runner), autotag, and playlist_sync gained progress.report
calls over their loops.

RSS polling now fetches the channel feeds concurrently (16 workers) instead
of one-at-a-time — a slow/unreachable feed no longer blocks the rest, cutting
a full poll of the catalogue from minutes to seconds. Feed fetches are
network-bound and run in the pool; DB inserts stay on the session's thread
(SQLAlchemy sessions aren't thread-safe), applied as each fetch completes.
Split the DB-write half of poll_rss_channel into apply_rss_feed so the fetch
and apply phases compose cleanly.
This commit is contained in:
npeter83 2026-06-19 02:43:37 +02:00
parent b5141ab112
commit e0971a23ec
4 changed files with 46 additions and 13 deletions

View file

@ -67,8 +67,10 @@ def _insert_stubs(db: Session, rows: list[dict]) -> int:
# --- RSS (free) ---
def poll_rss_channel(db: Session, channel: Channel) -> int:
entries = fetch_channel_feed(channel.id)
def apply_rss_feed(db: Session, channel: Channel, entries: list[dict]) -> int:
"""Insert new video stubs from an already-fetched feed and stamp the channel's last RSS
time. DB-only must run on the session's own thread; the network fetch
(fetch_channel_feed) is kept separate so callers like run_rss_poll can parallelise it."""
inserted = _insert_stubs(db, entries)
channel.last_rss_at = _now()
db.add(channel)
@ -76,6 +78,10 @@ def poll_rss_channel(db: Session, channel: Channel) -> int:
return inserted
def poll_rss_channel(db: Session, channel: Channel) -> int:
return apply_rss_feed(db, channel, fetch_channel_feed(channel.id))
# --- Backfill (uploads playlist; costs quota) ---
def _stub_from_playlist_item(item: dict, channel_id: str) -> dict | None:
content = item.get("contentDetails", {})