feat(search): zero-quota InnerTube scrape source + search_source toggle
Live YouTube search can now use YouTube's internal InnerTube endpoint instead of search.list, materialising results through the same enrich/provenance path at zero API quota (search.list costs 100 units/page; scrape costs nothing, only the cheap shared videos.list enrich is charged). - youtube/search_scrape.py: InnerTube search returning the same page shape as YouTubeClient.search_videos (items + continuation cursor); SOCS consent cookie, videoRenderer walk (channels/playlists/Shorts shelves naturally excluded), type:video filter, cached InnerTube key/version with constant fallbacks. - routes/search.py: admin-selectable source (search_source, default scrape). Scrape path skips the 100-unit budget pre-check and logs a zero-cost search event so the per-user daily cap still counts it; api path unchanged. Response carries the active source. - sysconfig/config: new search_source key (scrape|api). - quota.log_action: record a zero-cost action event for per-user rate limits.
This commit is contained in:
parent
17c17c1d3c
commit
396e09189b
5 changed files with 228 additions and 12 deletions
|
|
@ -71,9 +71,13 @@ class Settings(BaseSettings):
|
|||
youtube_api_proxy: str = ""
|
||||
# Daily quota budget (units). Default leaves headroom under the 10,000/day free limit.
|
||||
quota_daily_budget: int = 9000
|
||||
# Per-user cap on live YouTube searches per day. search.list costs 100 units each, so the
|
||||
# shared budget only affords ~80-90/day total — this stops one user draining it. Admin-tunable.
|
||||
# Per-user cap on live YouTube searches per day. With the official API (search_source="api")
|
||||
# each search.list costs 100 units, so the shared budget only affords ~80-90/day total; with
|
||||
# the zero-quota scrape source it's a pure anti-abuse rate limit. Admin-tunable.
|
||||
search_daily_limit_per_user: int = 70
|
||||
# Source for live YouTube search: "scrape" (InnerTube, no API quota) or "api" (search.list,
|
||||
# 100 units/page). Scrape is the default; flip to "api" if scraping is blocked/breaks.
|
||||
search_source: str = "scrape"
|
||||
# Recent-first backfill: how far back to fetch on the first pass per channel.
|
||||
backfill_recent_max_videos: int = 100
|
||||
backfill_recent_max_days: int = 365
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue