Backend: item detail returns audio_streams + subtitle_streams (from Plex Part.Stream
ordinals). The HLS session accepts audio/subtitle stream ordinals — the selected audio
is mapped (transcoded to AAC), a selected subtitle is muxed in as a WebVTT rendition
via a MASTER playlist (-master_pl_name + -var_stream_map sgroup:subs) so ffmpeg keeps
it in sync per-session. Selecting a track forces the HLS path (even for direct files).
Generic /stream/{rk}/hls/{filename} endpoint serves the master/media/vtt/ts artifacts.
Frontend: gear menu lists real audio + subtitle tracks; changing one restarts the
session at the current position (seek-restart mechanism). hls.js subtitle enabled on
SUBTITLE_TRACKS_UPDATED (not just MANIFEST_PARSED, else the VTT is never fetched).
Cues auto-nudged to line 88% (the full-height <video> clips a default bottom-edge cue).
Controls no longer auto-hide while the menu is open. plex.player.* i18n en/hu/de.
Verified in a real browser (2 Broke Girls S1E1, 2 audio + 2 subs): menu shows the real
tracks, enabling English subtitles renders them correctly positioned; backend master
playlist + vtt segments validated over HTTP.
- Player: control tooltips now open ABOVE the button (no downward viewport clip);
keyboard use also reveals the auto-hidden controls (wake() on keydown); added a
Download button that downloads the ORIGINAL physical file (no re-encode/repackage,
Content-Disposition attachment via GET /stream/{rk}/file?download=1).
- Plex cards: hover affordance shows what a click does (Play / Resume / Open show)
+ a quick watched/unwatched toggle (stopPropagation, no playback) — mirrors the
YT-feed card quick-actions. New plex i18n keys en/hu/de.
- Verified in a real browser: card hover Play overlay + watched toggle (marks watched
without opening), player download button present, tooltip-above, keyboard reveals
controls, correct durations (Enola Holmes 3 = 1:48:04).
- NOTE: 'The Three Diablos' showing 13:05 was NOT a bug — it is a 13-min short (file
ffprobe = 785s); the 1h42m film is 'The Last Wish'.
- backend: GET /api/plex/item/{rk} detail (metadata, cast + intro/credit markers
from Plex, episode prev/next, per-user resume+status); POST /item/{rk}/progress
(resume checkpoint, near-end→watched) + /state (new|watched|hidden).
- frontend PlexPlayer.tsx (lazy, pulls hls.js): full-page player over the Plex
module. Direct files stream raw <video>; remux via hls.js on the seek-restart
session — custom controls map the ABSOLUTE timeline over the session offset, and a
seek beyond the loaded region restarts the session at the target. Play/pause/resume
/restart, ±10s seek, prev/next episode (Shift+←/→), volume, windowed/fullscreen,
full keyboard, Skip intro/Skip credits from markers (shaded on the seekbar),
auto-advance + watched-on-end, resume from saved position, 10s progress checkpoints.
- PlexBrowse: card/episode click opens the player as a history subview (browser Back
returns to the grid/show). api client (plexItem/plexSession/plexProgress/plexSetState)
+ types; plex.player.* i18n en/hu/de. hls.js@^1.6.16.
- Verified over HTTP: detail w/ markers+cast+episode-nav, progress persist. Video
playback itself is pending browser UAT. Subtitle/audio track SELECTION deferred.
Validated end-to-end on a real remux file (h264+ac3 mkv). The riskiest part of the
Plex epic — on-the-fly playback from the local file — now proven.
- app/plex/stream.py: seek-restart HLS session model (one ffmpeg per item, restarted
at a seek offset via -ss). Video stream-COPY (I/O-bound, CPU-light — fine on the
GPU-less prod host) + audio→aac only when not already aac. Maps first video+audio,
drops subtitles (no VTT rendition clutter). Session cap + idle reaper.
- routes: POST /stream/{rk}/session?start= (direct→raw url, remux→HLS session,
transcode→501 P3), GET /stream/{rk}/file (206 range, direct), /index.m3u8, /seg_n.ts.
- KEY FINDINGS (hard-won): (1) ffmpeg -hls_playlist_type VOD does NOT write the
playlist mid-run (only at finalize) → use EVENT (append-only, appears immediately),
which suits the seek-restart model; (2) naive per-segment copy + keyframe-indexed
extraction are both unreliable (non-uniform/imprecise segments) — ffmpeg's own HLS
muxer is the only correct segmenter; (3) dev slowness was the Windows /downloads
bind-mount write + SMB read → PLEX_HLS_DIR env points scratch at fast container-local
/var/tmp on dev (prod keeps download_root, fast local disk).
Per user feedback: promote Plex from a hidden feed-Source option to a first-class
left-nav module, with its own filter sidebar and the shared top search box.
- Plex is now a nav-rail module (page='plex', gated on me.plex_enabled) — its own
page → correct browser Back/Forward; removed the Feed Source-dropdown 'plex' hack.
- PlexSidebar.tsx: left filter column (library scope, watch-state for movies:
all/unwatched/in_progress/watched, sort) — per-account persisted (App state).
- Header search box now also serves the Plex page (integrated search); the live
YouTube-search escalation stays feed-only.
- PlexBrowse.tsx reworked: infinite scroll (IntersectionObserver, no 'Load more'),
content-visibility for smoother long-list scroll, show drill-down rides history
(useHistorySubview) so Back returns to the grid; dropped the stray 'YouTube feed'
button on the show page.
- backend /browse gains a per-user watch-state filter (show=, movies only).
- plex i18n (navLabel + filter.*) en/hu/de.
Note: episode-title 'Episode N' on some shows (e.g. Westworld) is Plex-side
(unmatched show) — we mirror what Plex has; Match/Refresh in Plex + re-sync fixes it.
- PlexBrowse.tsx: self-contained Plex mode (library scope tabs, own search +
sort, poster-card grid with watch state + playability tint, paged; show →
seasons/episodes drill-down). Playback announces (P2 wires the real player).
- Feed source dropdown gains a 'Plex' option (gated on me.plex_enabled); App
swaps <Feed> for <PlexBrowse> when librarySource==='plex' so the video-feed
hooks don't run in Plex mode. me exposes plex_enabled.
- api client (plexLibraries/plexBrowse/plexShow/plexImageUrl) + types; librarySource
union + share-URL restore gain 'plex'; new plex i18n namespace en/hu/de.
- Verified over HTTP (authed): libraries, browse+FTS, image proxy 200 image/jpeg.
- app/plex/sync.py: full reconcile of enabled movie/show sections into plex_*
(upsert by rating_key). Reuses Plex's own codec info to classify browser
playability (direct/remux/transcode) — no ffprobe. Movies + episodes become
playable leaves; shows/seasons mirrored for the drill-down.
- routes/plex.py: GET /libraries, /browse (FTS search + sort + paging, movie
leaves or show cards), /show/{rk} (seasons+episodes, per-user state), /image
(proxies Plex art, no duplication); admin POST /sync. All auth'd, demo-allowed.
- scheduler: plex_sync job (interval settings.plex_sync_interval_min).
- Verified against the real server: 2 libs, 3206 movies, 260 shows/961 seasons/
14768 episodes synced in 13s; FTS search + drill-down work. Playability:
~1% direct, 89% remux, 6% transcode (informs P2/P3).
Every download now records the clean source page URL and shows it on the
Downloads page (open in a new tab, or copy to clipboard). The worker stores
yt-dlp's canonical webpage_url on the asset (migration 0043 adds
media_assets.source_webpage_url); the serializer prefers it and falls back to a
URL derived from source_kind+source_ref, so YouTube, external YouTube links and
external URLs (e.g. Facebook reels) all get a correct reference, and queued/older
rows work before the worker fills it. Edit clips return null (a clip's source is
the user's own earlier download, not a web page). i18n en/hu/de.
The logged-out landing probed /api/me, which 401'd, and the browser logs every
non-2xx fetch as a console error regardless of how the app handles it. The
bootstrap probe now returns 200 with {authenticated:false} via a new
optional_current_user dependency; api.me() maps that to null. The render gate
treats no-data as 'signed out' -> the landing, and a thrown error as a real
network/5xx failure. Other protected endpoints still 401, so mid-session expiry
is still caught by the global handler. Lighthouse (dev): Best Practices 96->100.
The 3 landing screenshots were 2.3 MB of PNG (feed.png alone 1.6 MB, decoded
at 2400x1350 for a ~400px slot) — the biggest LCP/transfer cost on the public
page. Re-encoded to WebP capped at 1600px (~284 KB total, feed 91% smaller).
Also set Cache-Control: content-hashed /assets/* are immutable for a year,
other SPA-root static files (welcome images, favicon, robots) get a 7-day TTL;
index.html stays no-cache. Register image/webp+avif mimetypes so FileResponse
serves the right Content-Type. Lighthouse (dev): Perf 80->93, LCP 1.8s->1.2s.
mp4 downloads used yt-dlp's default best video+audio, which on YouTube is
VP9+Opus at 1080p. Chromium plays that, but iOS/Safari WebKit (which every
iOS browser is forced to use) decodes only H.264/H.265+AAC inside mp4 — so a
shared /watch link showed a broken player on iPhone/iPad while the download
still worked. Rank vcodec:h264 + acodec:aac above resolution for mp4 output
(unless a custom profile pins a vcodec), keeping VP9/AV1 as a graceful
fallback. Bump the format signature (v2) so an identical spec gets a fresh
cache identity instead of hitting a stale VP9 asset.
The worker and API start in parallel; the API applies migrations on its own startup, so on a fresh
deploy the worker hit a missing download_jobs table and crash-looped until migrations landed. The
worker now polls for the schema before recovering orphans, and (belt-and-suspenders) the compose
files gate it on the API being healthy. Fixes the observed prod restart-loop on the v0.22.0 deploy.
A shared-with-me item only had a download button. Add two actions (Share is intentionally NOT
offered — no chain re-sharing of someone else's file):
- Edit: the editor now accepts an accessible (owned OR shared) source, so editing a shared video
produces the editor's OWN clip in their library (counts against their quota, fully theirs
including share); the source file is only read. Route uses _accessible_job.
- Remove from my list: DELETE /api/downloads/shared/{job_id} deletes only the recipient's share
grant — the owner's job and physical file are untouched (per-user dismissal).
i18n en/hu/de. Verified in a real browser (edit a shared 70-min video → own 2:31 clip; remove
confirm shows 'won't delete the owner's file').
Share a download by a capability URL (/watch/{token}) that anyone can play on a login-free page —
distinct from the registered-user ACL share. Per-link controls: optional expiry, allow-download
toggle (stream-only vs downloadable), optional argon2 password. Revoke = delete.
- migration 0042 + DownloadLink model; app/downloads/links.py (token, HMAC grant sign/verify)
- owner endpoints (POST/GET/PATCH/DELETE links) + /recipients for the internal user picker
- public router (no auth): watch meta / password unlock→signed grant / range-aware file serve
(inline vs attachment); unlock is rate-limited; meta verifies the file exists on disk
Verified end-to-end: 206 range, inline vs attachment, wrong-password 403, tampered-grant 403.
Editor v2 backend: an edit_spec can carry a 'segments' cut-list that concatenates the kept
ranges into ONE file (accurate=filter_complex trim+concat re-encode; fast=concat-demuxer
stream-copy with per-segment inpoint/outpoint). normalize/edit_sig/clip_duration/needs_reencode
handle segments; worker writes the concat list + runs build_concat_plan. Single-trim path (used by
'separate files' export) unchanged.
YouTube returns flaky per-client responses: the web clients intermittently mislabel normal
videos as 'DRM protected' (like tv did), and android_vr can get bot-flagged under heavy load.
Neither client alone is reliable. So:
- PRIMARY player client = android_vr: high-quality DASH (399+251), no PO token / Deno needed,
reliable for normal use
- on a bot/DRM/format failure the worker retries with the FALLBACK set (web_safari,web,android
+ POT sidecar + Deno) — bypasses bot-detection when android_vr is flagged
- both client sets admin-tunable (DOWNLOAD_PLAYER_CLIENTS[_FALLBACK])
Verified: the previously DRM-failing video now downloads steadily via android_vr, no DRM error.
The web player clients need two things beyond the PO token: (1) a JavaScript runtime for
YouTube's 'n' signature challenge — install Deno (v2.9.1) + yt-dlp[default] (bundles yt-dlp-ejs
solver scripts); auto-detected on PATH. (2) an audio source — the web clients only expose
video-only DASH, so add the 'android' client (also PO-token-backed) for audio/muxed formats.
Player clients: web_safari,web,android (dropped tv — it falsely reports normal videos as DRM).
End-to-end verified in the worker: extraction (POT + Deno n-challenge) + download completes,
no bot/DRM/format errors. Full YouTube stack now: force-ipv4 + bgutil POT sidecar + Deno.
YouTube increasingly demands a Proof-of-Origin token (or shows 'confirm you're not a bot').
Instead of manual cookies, run the bgutil POT provider as a sidecar that mints tokens on demand:
- compose: bgutil-pot service (brainicism/bgutil-ytdlp-pot-provider:1.3.1, port 4416, init)
- requirements: bgutil-ytdlp-pot-provider==1.3.1 plugin (version pinned to the image)
- config: DOWNLOAD_POT_BASE_URL (http://bgutil-pot:4416) + DOWNLOAD_PLAYER_CLIENTS
(web_safari,web,tv — POT-capable; the default android_vr client never requests a token)
- formats.build_ydl_opts wires extractor_args youtube:player_client + youtubepot-bgutilhttp:base_url
Verified: the sidecar mints a valid PO token (BotGuard challenge solved) and yt-dlp fetches it
from the provider. NOTE: end-to-end download couldn't be confirmed here because the test IP got
hard-flagged by YouTube from heavy testing (~16 GB + dozens of extractions) — a hard abuse block
needs cooldown/cookies, which POT doesn't lift; POT bypasses the normal soft bot-check.
Embedding (thumbnail/chapters) rewrote the whole file — a 10 GB video spent minutes making a
10 GB temp copy — while the Plex .nfo + poster sidecars already carry that metadata. So:
- builtin presets no longer embed (migration 0040 re-seeds them; formats._DEFAULTS embed off);
only the tiny FFmpegThumbnailsConvertor (for poster.jpg) remains, so downloads go
video -> audio -> merge -> done with no full-file rewrite
- default preset is now 1080p (Best stays available, second in the list) so a long video
doesn't silently pull ~10 GB and blow the quota
Users who want self-contained files can enable embedding in a custom profile.
ffmpeg post-steps have no byte-progress, so instead of a silent 'Processing' the row now names
the current step (Merging / Extracting audio / Embedding thumbnail / Removing sponsors / Writing
metadata) with an indeterminate pulse. Byte-progress phases (video/audio) keep the % bar.
i18n en/hu/de.
Root cause of the failing downloads: containers usually have no working IPv6 route, but
googlevideo CDN hosts advertise AAAA records, so the downloader tried IPv6 and failed with
'[Errno -5] No address associated with hostname'. Small clips happened to use IPv4; long videos
(more CDN requests / the ffmpeg path) hit IPv6 and died.
- yt-dlp source_address=0.0.0.0 forces IPv4 for every connection (== --force-ipv4). Verified: the
full video+audio download completes cleanly.
- Also: retry (resume) now resets the shared asset from 'error' to 'pending', so it actually
re-downloads — previously the requeued job instantly re-inherited the asset's stale error
because the worker short-circuits a job whose asset already errored.
- Dropped the localdev DNS override (wrong hypothesis; the issue was IPv6, not the resolver).
- Kept yt-dlp retries/fragment_retries/socket_timeout for transient blips.
Verified end-to-end in the worker: two previously-failing long videos now download to done.
_insert_stubs previously stored the raw title, so a brand-new video showed its unnormalized
title until enrichment caught up. Normalize (and stash original_title) at insert as well, so
every video-title write path is covered: stub insert, enrichment, live search, and the download
worker. Every newly imported feed video is normalized from the first insert.
Noisy YouTube titles are cleaned for display + storage, raw kept in videos.original_title:
- app/titles.normalize_title: strip emoji/symbols (keep accents), remove trailing SEO hashtag
clusters (keep numeric #3 episode markers), context-aware de-shout (mostly-ALL-CAPS titles ->
Title Case with an acronym whitelist + function-word lowercasing; otherwise only long all-caps
words), collapse repeated punctuation
- applied at enrichment (sync/videos.py) and in the download worker (ad-hoc yt-dlp titles);
catalog downloads inherit the normalized title automatically
- migration 0039: add original_title, preserve raw, rewrite title (generated search_vector
regenerates); reversible via original_title
Backfill on localdev: 122115/273417 titles normalized in ~2 min. Verified in the feed + on
real messy samples (emoji/de-shout/hashtags), accents + acronyms (PS5/AI/USA/PC) preserved.
The progress bar looked broken (95% -> 5% -> recount) because yt-dlp downloads the video and
audio streams separately (each 0->100%) then merges — with no indication of which step is
running. Now:
- worker labels the phase from the stream codecs (video / audio) and via a postprocessor hook
(merging / processing), so the user sees what's happening
- DownloadCenter shows the phase as the status and renders an indeterminate pulse (no bogus %)
during merge/processing; phase i18n en/hu/de
- yt-dlp retries (3) + fragment_retries (5) so transient network blips self-heal instead of
failing the job (seen once as a DNS 'No address associated with hostname' error)
Verified: phase transitions queued -> video -> processing -> done.
A queued/downloading row previously showed the bare video id and no thumbnail because asset
metadata was only filled when the download completed. Now:
- service.populate_from_catalog fills title/uploader/thumbnail/date/duration at enqueue from
our own Video/Channel catalog (0 network cost) — feed downloads look right instantly
- worker fills the same from yt-dlp's info_dict on the first progress event (covers ad-hoc
URLs / catalog misses), best-effort, never fails a download
Verified: a paused feed download now shows its real title, channel, thumbnail and duration.
Two UAT findings:
1. The device download filename (Content-Disposition) kept emoji/symbols from the video title.
Add storage.display_filename (drops emoji/symbol/control unicode, keeps spaces + accents)
and use it for the download name — readable and clean ("…alapján!.mp4", no emoji).
2. Deleting/canceling a download removed the job but the shared MediaAsset (and its file) lingered
as cache, so 'Ready files' stayed inflated and disk wasn't freed. Rework: _release_asset drops
the hold and, once no job holds the asset, deletes the file + row immediately (the cache only
needs to span overlapping holders). Also fixes cancel never decrementing (it flipped status to
'canceled' before releasing, tripping the holding-state guard).
Verified: filename emoji-stripped; enqueue→delete removes the asset row + file from disk.
A plain <a> file download can't send the X-Siftlode-Account header, so current_user resolved
it to the session-default account — 404 'Unknown download' when the tab acts as a non-default
wallet account that owns the file. resolved_user_id now also honours a ?account= query param
(the same wallet-gated selection the WebSocket already uses), and downloadFileUrl appends the
active account id. Verified: default account -> 404, ?account=<owner> -> 206 with the right
Content-Disposition.
The .gitignore 'downloads/' pattern for the DOWNLOAD_ROOT bind mount also matched the
backend/app/downloads source package, so formats.py/storage.py/service.py never got
committed with M2. Anchor the ignore to '/downloads/' (repo root only) and add the package.
storage.sanitize() now handles arbitrarily messy titles (emoji, ZWJ, fullwidth, clickbait
punctuation): NFKC-normalize, drop emoji/symbol/control unicode categories, collapse
repeated punctuation, underscore-join words -> space-free paths. Accents (HU/DE) preserved,
no ASCII folding. Plex layout uses _-_ separators + Season_{year}.
Two tabs in one browser can now run two different signed-in accounts at once.
- The signed session cookie stays the browser's account WALLET (account_ids). Which account a
given tab acts as is a per-tab choice held in sessionStorage and sent per-request via the
X-Siftlode-Account header; current_user honours it only for an account already in the wallet,
without mutating the cookie's default account. Switching accounts sets the header + reloads
THIS tab only, instead of the old cookie-wide switch that changed every tab.
- WebSocket can't send headers, so the per-tab account rides in the ?account= query param
(validated against the wallet).
- Logout is per-tab aware: it signs the requesting tab's active account out of the wallet
(promoting a new default only if the removed one was the default), and the tab drops its
override. A stale per-tab header account 401s just that tab instead of clearing the session.
- Serve index.html with Cache-Control: no-cache so a deploy's new hashed bundle is picked up
immediately instead of the browser running a heuristically-cached stale index.html.
- Remove owner-specific / legacy deploy files (home/prod/server compose, deploy/).
The home compose stays as a local untracked file for the maintainer's own deploy.
- Genericise infra-specific code comments (egress-proxy examples) to neutral wording.
- Replace the hardcoded contact email on the legal pages with the instance operator's
configured admin email, served via the public /auth/config and shown with a neutral
fallback — so each self-hosted instance shows its own contact.
- Rewrite README for the current app + a copy-paste self-hosting quick start (prebuilt
image + first-run wizard) with a build-from-source alternative; tidy .env.example.
Per-user named snapshots of the feed's FeedFilters. New saved_views table
(migration 0035, unique per user+name, position, is_default) + SavedView
model, and a /api/saved-views router (list/create/patch/delete/reorder)
gated behind require_human so the shared demo account can't pollute it.
Setting is_default clears the flag on the user's other views (one default).
- Live-search view: a results-count selector (20/40/60/100) replaces manual load-more (the free
scrape source pages until that many are gathered); an 'these results are temporary' banner with
a 'Clear now' button that discards them 'as if never added' (api.clearSearch) and returns to
the feed.
- Channel blocklist: a Block/Unblock toggle + 'Blocked' badge on the channel page (blocked
channels don't auto-explore and their videos are hidden), and a 'Blocked channels' section in
the Channel manager with one-click unblock. ChannelDetail.blocked from the backend.
- Admin: a 'Purge discovery' button on the Scheduler page (immediate un-kept search/explore
cleanup). EN/HU/DE throughout.
- Live-search results stay ephemeral: the discovery-cleanup job now also reclaims un-kept
via_search videos (no watch state / not playlisted / channel not subscribed) after a grace
period (search_grace_days), and POST /api/search/clear discards a given result set 'as if
never added' (drops the user's search-finds + deletes the now-orphaned, un-kept videos).
Admin POST /api/admin/purge-discovery runs it on demand (grace 0).
- Count selector: GET /api/search/youtube gains a 'limit' — the free scrape source pages
through continuations until that many results are gathered (no manual load-more); the API
source stays one ≤50 page (cost).
- Per-user channel blocklist (migration 0034 blocked_channels): block/unblock/list endpoints;
blocked channels' videos are dropped from live search before ingest and hidden from the feed
/ Library / explore / channel page; explore refuses a blocked channel.
- New-first ordering: results you already have (subscribed channel, watched/in-progress/saved/
playlisted) sink below genuinely-new discoveries, preserving relevance within each group.
- Channels manager: prominent channel-name search box and the 'Your tags' chips are now
clickable to filter the table by tag (replacing the hidden per-column DataTable popovers).
Both filter client-side over the status-filtered list; a focus-channel intent seeds the
search, the reset intent clears both.
- Feed 'Your tags' sidebar: count user tags in the facet endpoint (the 'other' category) and
make the widget contextual like language/topic — counts reflect the current filter and
zero-count chips hide (e.g. a source=search view with no tagged channels shows 'no matching
tags' instead of the full static list). EN/HU/DE searchPlaceholder.
Adds the backend for a dedicated channel page + ephemeral browsing of un-subscribed
channels:
- migration 0033: channels.{total_view_count,published_at,banner_url,external_links,
from_explore}, videos.via_explore, explored_channels (per-user, grace-clocked).
- enrich channels with part=brandingSettings (banner/links) + statistics.viewCount +
snippet.publishedAt — no extra quota.
- GET /api/channels/{id}: About detail + this user's relationship; lazy-enriches the new
About fields (published_at sentinel).
- POST /api/channels/{id}/explore (require_human, quota-reserve guarded): records the
exploration, flags the channel ephemeral unless followed, ingests one page of uploads
(via_explore) + enriches; returns next_page_token for load-more.
- feed: per-explorer gate — a via_explore video is visible only to users who explored or
subscribe to its channel, so exploration never leaks into everyone's catalog.
- subscribe = keep: clears from_explore + via_explore on the channel's videos.
- scheduler explore_cleanup job + explore_grace_days config: hard-delete explored-but-
unkept channels and their untouched ephemeral videos after a grace period.
Makes the local search behave more like YouTube's — finding videos by the
uploader's own keywords or the query that surfaced them, not only words in the
title. A DB-generated, weighted search_vector (migration 0032) replaces the
title-only FTS index:
- keywords: the creator's snippet.tags (free — already in the snippet we fetch),
stored on enrich.
- search_terms: distinct live-search queries that surfaced the video (across all
users), appended by the search route — folds YouTube's relevance into local
search (a video YT returned for a query becomes findable by it even without a
title match), the user's own idea.
- description (truncated) for broad recall on the existing catalog.
Weighted title(A) > keywords+queries(B) > description(C) so ts_rank keeps title
hits on top. A plain GIN index on the generated column guarantees index use (no
expression/param matching). Verified on localdev: recall 146->213 for one query;
7 'eurovision' hits via the document but not the title; index scan confirmed.
Two related search improvements:
1) Your own live-search results now belong to your Mine feed. A new per-user
search_finds table (migration 0030) records each video you surface via your
YouTube search (the route inserts them idempotently). The Mine feed becomes
'your non-hidden subscriptions OR your search finds', and the Source filter
now applies in Mine too: organic = subscriptions, search = your search finds,
all = both (default stays organic, so the main feed is unchanged). The shared
Library keeps using the global via_search flag.
2) Feed search ranks by relevance instead of a whole-phrase substring. A custom
unaccent_simple text-search config + GIN index (migration 0031) back a
YouTube-like fuzzy match: word-order-independent, multi-word AND, prefix on
the word being typed, accent-insensitive. A new 'relevance' sort orders by
ts_rank; the channel name still matches as a substring. The rank is scaled to
an integer so the keyset cursor pages it exactly (a raw float4 breaks paging).
_filtered_query returns the rank expr so only the feed list uses it.
Live YouTube search can now use YouTube's internal InnerTube endpoint instead
of search.list, materialising results through the same enrich/provenance path
at zero API quota (search.list costs 100 units/page; scrape costs nothing, only
the cheap shared videos.list enrich is charged).
- youtube/search_scrape.py: InnerTube search returning the same page shape as
YouTubeClient.search_videos (items + continuation cursor); SOCS consent cookie,
videoRenderer walk (channels/playlists/Shorts shelves naturally excluded),
type:video filter, cached InnerTube key/version with constant fallbacks.
- routes/search.py: admin-selectable source (search_source, default scrape).
Scrape path skips the 100-unit budget pre-check and logs a zero-cost search
event so the per-user daily cap still counts it; api path unchanged. Response
carries the active source.
- sysconfig/config: new search_source key (scrape|api).
- quota.log_action: record a zero-cost action event for per-user rate limits.
Feed text search used plain ILIKE, which is case- but not diacritic-insensitive, so 'tiesto'
missed the many titles spelled 'Tiësto' — a search that ingested ~45 results showed only ~12.
Enable the postgres unaccent extension (migration 0029) and wrap both sides of the title/
channel match in unaccent(), so 'tiesto' now matches 'Tiësto'. Applies to feed, count and
facets alike.
Replace the binary 'show search-discovered' toggle with a 3-way Source selector in the
Library toolbar, so users can also see ONLY search-discovered videos — not just hide or
mix them. Backend: feed param library_source = organic (default, hides via_search) | all
(both) | search (only via_search), applied in scope=all. Strings in HU/EN/DE.
Add a live YouTube search that materialises results into the shared catalog so
they render with the normal feed cards + in-app player and gain per-user state.
- YouTubeClient.search_videos(): search.list (100 units), embeddable-only, returns
flat stubs + nextPageToken; surfaces liveBroadcastContent for live filtering.
- routes/search.py GET /api/search/youtube: require_human + per-user daily cap
(search_daily_limit_per_user, default 70) + can_spend pre-check (429 on either);
drops live/upcoming, upserts channel stubs (channels.list) + video stubs, enriches
(videos.list), runs the youtube.com/shorts probe, then excludes Shorts/live and
returns feed cards in relevance order with the YouTube pageToken as the cursor.
- Provenance: videos.via_search / channels.from_search (migration 0028) flag
search-discovered rows; the feed hides them from the Library (scope=all) by default
via exclude_search_discovered, leaving the Mine feed untouched.
- quota.actions_today() counts a user's per-action events today for the cap; only the
search.list call is attributed VIDEOS_SEARCH so the counter is exactly 1 per search.
TimestampMixin (created_at) + UpdatedAtMixin (updated_at) replace the repeated
column blocks across 9 + 3 models. Models that index created_at or default it
Python-side keep their own column on purpose. Verified zero schema drift via an
identical alembic-check diff before/after.
