feat(scheduler): report progress from every job + parallelize RSS polling
Every background job now reports progress, so the dashboard can show a live bar for any run (not only enrich/backfill/shorts/maintenance): rss_poll and subscriptions (runner), autotag, and playlist_sync gained progress.report calls over their loops. RSS polling now fetches the channel feeds concurrently (16 workers) instead of one-at-a-time — a slow/unreachable feed no longer blocks the rest, cutting a full poll of the catalogue from minutes to seconds. Feed fetches are network-bound and run in the pool; DB inserts stay on the session's thread (SQLAlchemy sessions aren't thread-safe), applied as each fetch completes. Split the DB-write half of poll_rss_channel into apply_rss_feed so the fetch and apply phases compose cleanly.
This commit is contained in:
parent
b5141ab112
commit
e0971a23ec
4 changed files with 46 additions and 13 deletions
|
|
@ -67,8 +67,10 @@ def _insert_stubs(db: Session, rows: list[dict]) -> int:
|
|||
|
||||
|
||||
# --- RSS (free) ---
|
||||
def poll_rss_channel(db: Session, channel: Channel) -> int:
|
||||
entries = fetch_channel_feed(channel.id)
|
||||
def apply_rss_feed(db: Session, channel: Channel, entries: list[dict]) -> int:
|
||||
"""Insert new video stubs from an already-fetched feed and stamp the channel's last RSS
|
||||
time. DB-only — must run on the session's own thread; the network fetch
|
||||
(fetch_channel_feed) is kept separate so callers like run_rss_poll can parallelise it."""
|
||||
inserted = _insert_stubs(db, entries)
|
||||
channel.last_rss_at = _now()
|
||||
db.add(channel)
|
||||
|
|
@ -76,6 +78,10 @@ def poll_rss_channel(db: Session, channel: Channel) -> int:
|
|||
return inserted
|
||||
|
||||
|
||||
def poll_rss_channel(db: Session, channel: Channel) -> int:
|
||||
return apply_rss_feed(db, channel, fetch_channel_feed(channel.id))
|
||||
|
||||
|
||||
# --- Backfill (uploads playlist; costs quota) ---
|
||||
def _stub_from_playlist_item(item: dict, channel_id: str) -> dict | None:
|
||||
content = item.get("contentDetails", {})
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue