feat(search): zero-quota InnerTube scrape source + search_source toggle
Live YouTube search can now use YouTube's internal InnerTube endpoint instead of search.list, materialising results through the same enrich/provenance path at zero API quota (search.list costs 100 units/page; scrape costs nothing, only the cheap shared videos.list enrich is charged). - youtube/search_scrape.py: InnerTube search returning the same page shape as YouTubeClient.search_videos (items + continuation cursor); SOCS consent cookie, videoRenderer walk (channels/playlists/Shorts shelves naturally excluded), type:video filter, cached InnerTube key/version with constant fallbacks. - routes/search.py: admin-selectable source (search_source, default scrape). Scrape path skips the 100-unit budget pre-check and logs a zero-cost search event so the per-user daily cap still counts it; api path unchanged. Response carries the active source. - sysconfig/config: new search_source key (scrape|api). - quota.log_action: record a zero-cost action event for per-user rate limits.
This commit is contained in:
parent
17c17c1d3c
commit
396e09189b
5 changed files with 228 additions and 12 deletions
|
|
@ -40,6 +40,8 @@ SPECS: tuple[ConfigSpec, ...] = (
|
|||
ConfigSpec("quota_daily_budget", "int", "quota", "quota_daily_budget", min=1, max=1_000_000),
|
||||
ConfigSpec("backfill_quota_reserve", "int", "quota", "backfill_quota_reserve", min=0, max=1_000_000),
|
||||
ConfigSpec("search_daily_limit_per_user", "int", "quota", "search_daily_limit_per_user", min=0, max=100_000),
|
||||
# Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
|
||||
ConfigSpec("search_source", "str", "quota", "search_source"),
|
||||
# --- Backfill (recent-first first pass per channel) ---
|
||||
ConfigSpec("backfill_recent_max_videos", "int", "backfill", "backfill_recent_max_videos", min=1, max=100_000),
|
||||
ConfigSpec("backfill_recent_max_days", "int", "backfill", "backfill_recent_max_days", min=1, max=36_500),
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue