The edit (pencil) action now edits a download's full display metadata — title,
channel name, channel link and any number of extra reference URLs — instead of
just the name. The channel and links render as clickable links on the library
card, and the channel link is auto-filled from the source (yt-dlp channel_url)
when available. Shared watch pages resolve the same per-download overrides, so a
rename/channel/link edit is reflected on the public /watch page too, with every
link clickable.
Adds migration 0049 (media_assets.uploader_url; download_jobs.display_uploader,
display_uploader_url, extra_links) and generalizes the rename endpoint into a
metadata update with URL validation. EN/HU/DE strings included.
Add POST /playlists/{id}/items/bulk and /items/remove-bulk so a whole
season or show can be added/removed in one call (input order preserved,
duplicates skipped). Extend GET /playlists with contains_group=<rk,rk,…>
returning per-playlist group_in + a top-level group_size for the bulk
add-to-playlist dialog. Include the show's rating_key (show_id) on
playlist episode cards so the client can group a playlist by show/season.
Personal ordered lists of Plex items, kept in Siftlode's own DB (works for users
without a Plex account, like watch-state) — the "your own lists" counterpart to
shared collections. Plex-direction sync is a later phase (plex_rating_key
reserved).
Backend: migration 0048 (plex_playlists + plex_playlist_items with position),
PlexPlaylist/PlexPlaylistItem models, and per-user CRUD endpoints under
/api/plex/playlists (list [+?contains for the add dialog], create [seeded],
detail [ordered cards], rename, delete, add/remove item, reorder). _leaf_card
handles the movie/episode mix. Frontend: "Add to playlist" dialog from the movie
info page (all users), a Playlists section in PlexSidebar (list + create),
PlexPlaylistView (reorder up/down, remove, rename, delete, Play all), and
PlexPlayer gained an optional `queue` so play-through follows the list order
(prev/next + auto-advance). i18n en/hu/de. Verified end-to-end on localdev
(backend CRUD + the create→add→view→play-through UI flow).
Admins can curate Plex collections from a movie's info page ("Collections"
button → PlexCollectionEditor dialog): create a collection (seeded with the
movie), add/remove the movie to/from editable collections, delete a collection,
and "take over" an existing plain Plex collection (mark it editable). All writes
go to Plex (POST/PUT/DELETE via new PlexClient methods) and are reflected on
every client; a targeted single-collection re-sync (sync.resync_collection /
delete_collection_local) updates the local mirror without the full ~4-min sync.
Gating (per the design decisions): editing is ADMIN-ONLY (collections are shared
library-wide); only plain manual collections are editable — smart + external
auto-lists (IMDb/TMDb/…) are always read-only (can_edit = editable && !smart &&
source=="collection"). New admin endpoints under /api/plex/collections
(create/items add+remove/rename/delete/editable). Verified end-to-end incl. the
live Plex write API (create/add/rename/remove/delete all 200, self-cleaned) and
the editor UI (create + delete with confirm) on localdev.
Selecting a subtitle restarted the HLS session with `-map 0:s:{ord}`, assuming an
EMBEDDED stream. Films whose subs are external sidecar .srt files (Plex reports
them, but they aren't in the mkv) matched no stream; the master-playlist's
declared subtitle group then made ffmpeg fail → "Playback couldn't start".
Subtitles now go through a new GET /api/plex/subtitle/{rk}/{ord} → text/vtt
(external subs fetched from Plex via the stream key + SRT→VTT; embedded text subs
extracted with ffmpeg; image subs → 415), served as native <video><track> that
the browser overlays. So choosing/switching a subtitle is instant with NO session
restart, and stream.py drops all subtitle muxing (`-sn`, no master playlist).
Image-based subs (PGS/VobSub) are marked text=false and hidden in the picker.
Verified on prod's Nymphomaniac Vol. II: HU sidecar → 1693 WebVTT cues, no crash.
Posters/art were proxied at full resolution (many 1-7MB, some 9.5MB) into ≤176px
grid/info cells — heavy to fetch + decode, so fast scrolling lagged 1-2s even
with a warm cache (the cache removed the Plex round-trip, not the image weight).
Now PlexClient.image_bytes optionally hits Plex's /photo/:/transcode to resize
server-side; the /image endpoint requests thumb=400x600, art=1280x720 and keys
the cache by width (old full-size files orphaned, re-cached small). Measured:
a poster 5.5MB->50KB, art 8.1MB->137KB, at 400x600 (crisp on 2x DPR). Grid
scroll is near-instant and the disk cache shrinks ~8x.
A collections page fires 100+ concurrent poster requests. Both image proxies
(/image and /person-image) authenticated via current_user + Depends(get_db),
holding a pooled DB connection for the whole request — including the slow Plex/
CDN fetch. Under the burst, the 15-connection pool was exhausted → QueuePool
checkout timeouts → 502s → slow, partially-loaded grids on prod.
Fix: authenticate these two high-fan-out endpoints with a signed-session check
(no DB user-load), serve disk-cache hits with zero DB access, and on a cold miss
open a short session only to resolve the image key + Plex config, releasing it
BEFORE the fetch (image_bytes uses no DB). Also raise the pool (20 + 30 overflow)
as headroom above the sync-endpoint threadpool.
Derive a `source` bucket per collection from the already-synced title + smart
flag (no schema change, no re-sync): external list providers (IMDb/TMDb/TVDb/
Trakt) by leading token, Plex smart lists via the smart flag, everything else a
genuine "collection". Returned on each info-page collection strip so the client
can show/hide each type independently. Adds the stripSource labels (en/hu/de).
Backend (migration 0047_plex_collections): a plex_collections table mirrors every Plex
collection (card metadata + smart flag + an editable flag reserved for Phase 2); membership
is stored as GIN-indexed collection_keys on member movies (plex_items) and shows (plex_shows).
The background sync (plex_sync) now fetches each library's collections + their children and
rebuilds membership — so ALL reads are local (Plex's dual-language collection queries are slow;
this trades a ~4-min background sync for instant reads). New /api/plex/collections endpoint;
/browse gains a combinable filter; item_detail returns the movie's collection
'strips' (sibling titles as playable cards, smallest/most-specific collection first) — all pure
local lookups.
Frontend: PlexSidebar gains a searchable Collection picker + an active-collection chip;
PlexInfo renders the collection strips (playable posters + 'Browse collection' → sets the
filter); the collection is part of PlexFilters (persisted). i18n en/hu/de.
Phase 2 (create/edit collections with write-back to Plex) is separate.
Backend (migration 0046_plex_people_search): a new people_text column (cast +
director names, de-duped) is folded into the generated search_vector at weight B,
so the top search box finds titles by an actor/director name and ranks them above
summary-only mentions. New /api/plex/people endpoint returns the cast/crew matching
the term (name prefix or word-start) with a film count + a headshot (pulled from one
representative film's live metadata; image bytes disk-cached).
Frontend: PlexBrowse shows matching people as virtual cards above the grid; clicking
one adds them to the actor/director filter (multi-value, from the C1 work) and clears
the search box so you land on exactly that person's films. Answers the 'why does
"drew" match Rambo?' confusion — it was matching the word in the synopsis; now names
are searchable. i18n en/hu/de.
⚠️ Prod needs a Plex re-sync after deploy to populate people_text (search_vector
regenerates automatically once the column has data).
UAT follow-ups on the Plex filter epic:
- Sort now has an asc/desc toggle (sort_dir), applied to any sort field.
- Genre multi-select gains an Any/All mode (genre_mode: OR vs AND containment).
- Director/actor/studio become multi-value: people AND (titles featuring all selected),
studios OR; clicking them on the info page stacks (unions) instead of replacing, and
the sidebar shows each as a removable 'Active' chip.
- fix(history): clicking a metadata filter on the info page now pushes a fresh grid
entry instead of history.back(), so browser Back returns to the info page rather than
leaving the Plex module.
- fix(player): fully tear down the <video> + hls on unmount and guard late play() calls,
so backing out of a just-started video no longer leaves audio playing in the background.
i18n en/hu/de (match any/all, sort direction).
Backend (migration 0045_plex_filter_meta): plex_items gains rating (audienceRating
~IMDb), content_rating, studio, originally_available_at, and GIN-indexed genres /
directors / cast_names — all mirrored from the cheap section listing (no per-item API
calls; they also seed a future watch-habit recommender). /browse gains genre / content-
rating / year / rating / duration / added-within / director / actor / studio filters
(@> containment, GIN) + sort by year|rating|duration|release; new /facets endpoint
returns available genres+ratings (with counts) and the year/rating/duration bounds. A
thin on-disk image cache (.plex-img-cache) serves posters/art/cast photos from local
disk after first fetch (~7-14x faster repeat loads).
Frontend: PlexSidebar grows the full filter set (facet-driven genre/age chips, rating
steps, year range inputs, duration buckets, added-within, active people/studio chips,
clear-all); filters persist per-account as one JSON blob. PlexInfo metadata (year,
genre, director, cast, studio, IMDb score) is clickable → sets the matching filter and
returns to the filtered grid (page variant only; the in-player overlay stays read-only
so a stray click can't stop playback). i18n en/hu/de.
Backend (no migration): item_detail now also returns IMDb score + id/url (from the
Plex Rating/Guid arrays), content rating, genres, director(s), studio, tagline, and
cast as {name, role, photo}. New host-whitelisted /person-image proxy serves cast
photos from Plex's public metadata CDN (keeps third-party requests off the browser).
Frontend: reusable PlexInfo component in two forms — a full info page opened from a
card's 'i' button (history subview, art backdrop, Play/Resume + watch controls) and a
lean overlay in the player toggled with 'I' (video keeps playing). Two per-user view
prefs (faint backdrop, cast row) persisted in preferences. Mark-unwatched / clear-resume
cover the watched-reset gap. i18n en/hu/de.
Extend admin POST /api/plex/test to resolve one real media file through the
local mount (a mirrored sample, else a single leaf fetched from Plex) and report
whether it is readable. A missing media bind-mount or a wrong plex_path_map is
now caught at Test time rather than only surfacing later as a dead player.
Backend: item detail returns audio_streams + subtitle_streams (from Plex Part.Stream
ordinals). The HLS session accepts audio/subtitle stream ordinals — the selected audio
is mapped (transcoded to AAC), a selected subtitle is muxed in as a WebVTT rendition
via a MASTER playlist (-master_pl_name + -var_stream_map sgroup:subs) so ffmpeg keeps
it in sync per-session. Selecting a track forces the HLS path (even for direct files).
Generic /stream/{rk}/hls/{filename} endpoint serves the master/media/vtt/ts artifacts.
Frontend: gear menu lists real audio + subtitle tracks; changing one restarts the
session at the current position (seek-restart mechanism). hls.js subtitle enabled on
SUBTITLE_TRACKS_UPDATED (not just MANIFEST_PARSED, else the VTT is never fetched).
Cues auto-nudged to line 88% (the full-height <video> clips a default bottom-edge cue).
Controls no longer auto-hide while the menu is open. plex.player.* i18n en/hu/de.
Verified in a real browser (2 Broke Girls S1E1, 2 audio + 2 subs): menu shows the real
tracks, enabling English subtitles renders them correctly positioned; backend master
playlist + vtt segments validated over HTTP.
- Player: control tooltips now open ABOVE the button (no downward viewport clip);
keyboard use also reveals the auto-hidden controls (wake() on keydown); added a
Download button that downloads the ORIGINAL physical file (no re-encode/repackage,
Content-Disposition attachment via GET /stream/{rk}/file?download=1).
- Plex cards: hover affordance shows what a click does (Play / Resume / Open show)
+ a quick watched/unwatched toggle (stopPropagation, no playback) — mirrors the
YT-feed card quick-actions. New plex i18n keys en/hu/de.
- Verified in a real browser: card hover Play overlay + watched toggle (marks watched
without opening), player download button present, tooltip-above, keyboard reveals
controls, correct durations (Enola Holmes 3 = 1:48:04).
- NOTE: 'The Three Diablos' showing 13:05 was NOT a bug — it is a 13-min short (file
ffprobe = 785s); the 1h42m film is 'The Last Wish'.
- backend: GET /api/plex/item/{rk} detail (metadata, cast + intro/credit markers
from Plex, episode prev/next, per-user resume+status); POST /item/{rk}/progress
(resume checkpoint, near-end→watched) + /state (new|watched|hidden).
- frontend PlexPlayer.tsx (lazy, pulls hls.js): full-page player over the Plex
module. Direct files stream raw <video>; remux via hls.js on the seek-restart
session — custom controls map the ABSOLUTE timeline over the session offset, and a
seek beyond the loaded region restarts the session at the target. Play/pause/resume
/restart, ±10s seek, prev/next episode (Shift+←/→), volume, windowed/fullscreen,
full keyboard, Skip intro/Skip credits from markers (shaded on the seekbar),
auto-advance + watched-on-end, resume from saved position, 10s progress checkpoints.
- PlexBrowse: card/episode click opens the player as a history subview (browser Back
returns to the grid/show). api client (plexItem/plexSession/plexProgress/plexSetState)
+ types; plex.player.* i18n en/hu/de. hls.js@^1.6.16.
- Verified over HTTP: detail w/ markers+cast+episode-nav, progress persist. Video
playback itself is pending browser UAT. Subtitle/audio track SELECTION deferred.
Validated end-to-end on a real remux file (h264+ac3 mkv). The riskiest part of the
Plex epic — on-the-fly playback from the local file — now proven.
- app/plex/stream.py: seek-restart HLS session model (one ffmpeg per item, restarted
at a seek offset via -ss). Video stream-COPY (I/O-bound, CPU-light — fine on the
GPU-less prod host) + audio→aac only when not already aac. Maps first video+audio,
drops subtitles (no VTT rendition clutter). Session cap + idle reaper.
- routes: POST /stream/{rk}/session?start= (direct→raw url, remux→HLS session,
transcode→501 P3), GET /stream/{rk}/file (206 range, direct), /index.m3u8, /seg_n.ts.
- KEY FINDINGS (hard-won): (1) ffmpeg -hls_playlist_type VOD does NOT write the
playlist mid-run (only at finalize) → use EVENT (append-only, appears immediately),
which suits the seek-restart model; (2) naive per-segment copy + keyframe-indexed
extraction are both unreliable (non-uniform/imprecise segments) — ffmpeg's own HLS
muxer is the only correct segmenter; (3) dev slowness was the Windows /downloads
bind-mount write + SMB read → PLEX_HLS_DIR env points scratch at fast container-local
/var/tmp on dev (prod keeps download_root, fast local disk).
Per user feedback: promote Plex from a hidden feed-Source option to a first-class
left-nav module, with its own filter sidebar and the shared top search box.
- Plex is now a nav-rail module (page='plex', gated on me.plex_enabled) — its own
page → correct browser Back/Forward; removed the Feed Source-dropdown 'plex' hack.
- PlexSidebar.tsx: left filter column (library scope, watch-state for movies:
all/unwatched/in_progress/watched, sort) — per-account persisted (App state).
- Header search box now also serves the Plex page (integrated search); the live
YouTube-search escalation stays feed-only.
- PlexBrowse.tsx reworked: infinite scroll (IntersectionObserver, no 'Load more'),
content-visibility for smoother long-list scroll, show drill-down rides history
(useHistorySubview) so Back returns to the grid; dropped the stray 'YouTube feed'
button on the show page.
- backend /browse gains a per-user watch-state filter (show=, movies only).
- plex i18n (navLabel + filter.*) en/hu/de.
Note: episode-title 'Episode N' on some shows (e.g. Westworld) is Plex-side
(unmatched show) — we mirror what Plex has; Match/Refresh in Plex + re-sync fixes it.
- PlexBrowse.tsx: self-contained Plex mode (library scope tabs, own search +
sort, poster-card grid with watch state + playability tint, paged; show →
seasons/episodes drill-down). Playback announces (P2 wires the real player).
- Feed source dropdown gains a 'Plex' option (gated on me.plex_enabled); App
swaps <Feed> for <PlexBrowse> when librarySource==='plex' so the video-feed
hooks don't run in Plex mode. me exposes plex_enabled.
- api client (plexLibraries/plexBrowse/plexShow/plexImageUrl) + types; librarySource
union + share-URL restore gain 'plex'; new plex i18n namespace en/hu/de.
- Verified over HTTP (authed): libraries, browse+FTS, image proxy 200 image/jpeg.
- app/plex/sync.py: full reconcile of enabled movie/show sections into plex_*
(upsert by rating_key). Reuses Plex's own codec info to classify browser
playability (direct/remux/transcode) — no ffprobe. Movies + episodes become
playable leaves; shows/seasons mirrored for the drill-down.
- routes/plex.py: GET /libraries, /browse (FTS search + sort + paging, movie
leaves or show cards), /show/{rk} (seasons+episodes, per-user state), /image
(proxies Plex art, no duplication); admin POST /sync. All auth'd, demo-allowed.
- scheduler: plex_sync job (interval settings.plex_sync_interval_min).
- Verified against the real server: 2 libs, 3206 movies, 260 shows/961 seasons/
14768 episodes synced in 13s; FTS search + drill-down work. Playability:
~1% direct, 89% remux, 6% transcode (informs P2/P3).
Every download now records the clean source page URL and shows it on the
Downloads page (open in a new tab, or copy to clipboard). The worker stores
yt-dlp's canonical webpage_url on the asset (migration 0043 adds
media_assets.source_webpage_url); the serializer prefers it and falls back to a
URL derived from source_kind+source_ref, so YouTube, external YouTube links and
external URLs (e.g. Facebook reels) all get a correct reference, and queued/older
rows work before the worker fills it. Edit clips return null (a clip's source is
the user's own earlier download, not a web page). i18n en/hu/de.
The logged-out landing probed /api/me, which 401'd, and the browser logs every
non-2xx fetch as a console error regardless of how the app handles it. The
bootstrap probe now returns 200 with {authenticated:false} via a new
optional_current_user dependency; api.me() maps that to null. The render gate
treats no-data as 'signed out' -> the landing, and a thrown error as a real
network/5xx failure. Other protected endpoints still 401, so mid-session expiry
is still caught by the global handler. Lighthouse (dev): Best Practices 96->100.
A shared-with-me item only had a download button. Add two actions (Share is intentionally NOT
offered — no chain re-sharing of someone else's file):
- Edit: the editor now accepts an accessible (owned OR shared) source, so editing a shared video
produces the editor's OWN clip in their library (counts against their quota, fully theirs
including share); the source file is only read. Route uses _accessible_job.
- Remove from my list: DELETE /api/downloads/shared/{job_id} deletes only the recipient's share
grant — the owner's job and physical file are untouched (per-user dismissal).
i18n en/hu/de. Verified in a real browser (edit a shared 70-min video → own 2:31 clip; remove
confirm shows 'won't delete the owner's file').
Share a download by a capability URL (/watch/{token}) that anyone can play on a login-free page —
distinct from the registered-user ACL share. Per-link controls: optional expiry, allow-download
toggle (stream-only vs downloadable), optional argon2 password. Revoke = delete.
- migration 0042 + DownloadLink model; app/downloads/links.py (token, HMAC grant sign/verify)
- owner endpoints (POST/GET/PATCH/DELETE links) + /recipients for the internal user picker
- public router (no auth): watch meta / password unlock→signed grant / range-aware file serve
(inline vs attachment); unlock is rate-limited; meta verifies the file exists on disk
Verified end-to-end: 206 range, inline vs attachment, wrong-password 403, tampered-grant 403.
Root cause of the failing downloads: containers usually have no working IPv6 route, but
googlevideo CDN hosts advertise AAAA records, so the downloader tried IPv6 and failed with
'[Errno -5] No address associated with hostname'. Small clips happened to use IPv4; long videos
(more CDN requests / the ffmpeg path) hit IPv6 and died.
- yt-dlp source_address=0.0.0.0 forces IPv4 for every connection (== --force-ipv4). Verified: the
full video+audio download completes cleanly.
- Also: retry (resume) now resets the shared asset from 'error' to 'pending', so it actually
re-downloads — previously the requeued job instantly re-inherited the asset's stale error
because the worker short-circuits a job whose asset already errored.
- Dropped the localdev DNS override (wrong hypothesis; the issue was IPv6, not the resolver).
- Kept yt-dlp retries/fragment_retries/socket_timeout for transient blips.
Verified end-to-end in the worker: two previously-failing long videos now download to done.
Two UAT findings:
1. The device download filename (Content-Disposition) kept emoji/symbols from the video title.
Add storage.display_filename (drops emoji/symbol/control unicode, keeps spaces + accents)
and use it for the download name — readable and clean ("…alapján!.mp4", no emoji).
2. Deleting/canceling a download removed the job but the shared MediaAsset (and its file) lingered
as cache, so 'Ready files' stayed inflated and disk wasn't freed. Rework: _release_asset drops
the hold and, once no job holds the asset, deletes the file + row immediately (the cache only
needs to span overlapping holders). Also fixes cancel never decrementing (it flipped status to
'canceled' before releasing, tripping the holding-state guard).
Verified: filename emoji-stripped; enqueue→delete removes the asset row + file from disk.
Two tabs in one browser can now run two different signed-in accounts at once.
- The signed session cookie stays the browser's account WALLET (account_ids). Which account a
given tab acts as is a per-tab choice held in sessionStorage and sent per-request via the
X-Siftlode-Account header; current_user honours it only for an account already in the wallet,
without mutating the cookie's default account. Switching accounts sets the header + reloads
THIS tab only, instead of the old cookie-wide switch that changed every tab.
- WebSocket can't send headers, so the per-tab account rides in the ?account= query param
(validated against the wallet).
- Logout is per-tab aware: it signs the requesting tab's active account out of the wallet
(promoting a new default only if the removed one was the default), and the tab drops its
override. A stale per-tab header account 401s just that tab instead of clearing the session.
- Serve index.html with Cache-Control: no-cache so a deploy's new hashed bundle is picked up
immediately instead of the browser running a heuristically-cached stale index.html.
Per-user named snapshots of the feed's FeedFilters. New saved_views table
(migration 0035, unique per user+name, position, is_default) + SavedView
model, and a /api/saved-views router (list/create/patch/delete/reorder)
gated behind require_human so the shared demo account can't pollute it.
Setting is_default clears the flag on the user's other views (one default).
- Live-search view: a results-count selector (20/40/60/100) replaces manual load-more (the free
scrape source pages until that many are gathered); an 'these results are temporary' banner with
a 'Clear now' button that discards them 'as if never added' (api.clearSearch) and returns to
the feed.
- Channel blocklist: a Block/Unblock toggle + 'Blocked' badge on the channel page (blocked
channels don't auto-explore and their videos are hidden), and a 'Blocked channels' section in
the Channel manager with one-click unblock. ChannelDetail.blocked from the backend.
- Admin: a 'Purge discovery' button on the Scheduler page (immediate un-kept search/explore
cleanup). EN/HU/DE throughout.
- Live-search results stay ephemeral: the discovery-cleanup job now also reclaims un-kept
via_search videos (no watch state / not playlisted / channel not subscribed) after a grace
period (search_grace_days), and POST /api/search/clear discards a given result set 'as if
never added' (drops the user's search-finds + deletes the now-orphaned, un-kept videos).
Admin POST /api/admin/purge-discovery runs it on demand (grace 0).
- Count selector: GET /api/search/youtube gains a 'limit' — the free scrape source pages
through continuations until that many results are gathered (no manual load-more); the API
source stays one ≤50 page (cost).
- Per-user channel blocklist (migration 0034 blocked_channels): block/unblock/list endpoints;
blocked channels' videos are dropped from live search before ingest and hidden from the feed
/ Library / explore / channel page; explore refuses a blocked channel.
- New-first ordering: results you already have (subscribed channel, watched/in-progress/saved/
playlisted) sink below genuinely-new discoveries, preserving relevance within each group.
- Channels manager: prominent channel-name search box and the 'Your tags' chips are now
clickable to filter the table by tag (replacing the hidden per-column DataTable popovers).
Both filter client-side over the status-filtered list; a focus-channel intent seeds the
search, the reset intent clears both.
- Feed 'Your tags' sidebar: count user tags in the facet endpoint (the 'other' category) and
make the widget contextual like language/topic — counts reflect the current filter and
zero-count chips hide (e.g. a source=search view with no tagged channels shows 'no matching
tags' instead of the full static list). EN/HU/DE searchPlaceholder.
Adds the backend for a dedicated channel page + ephemeral browsing of un-subscribed
channels:
- migration 0033: channels.{total_view_count,published_at,banner_url,external_links,
from_explore}, videos.via_explore, explored_channels (per-user, grace-clocked).
- enrich channels with part=brandingSettings (banner/links) + statistics.viewCount +
snippet.publishedAt — no extra quota.
- GET /api/channels/{id}: About detail + this user's relationship; lazy-enriches the new
About fields (published_at sentinel).
- POST /api/channels/{id}/explore (require_human, quota-reserve guarded): records the
exploration, flags the channel ephemeral unless followed, ingests one page of uploads
(via_explore) + enriches; returns next_page_token for load-more.
- feed: per-explorer gate — a via_explore video is visible only to users who explored or
subscribe to its channel, so exploration never leaks into everyone's catalog.
- subscribe = keep: clears from_explore + via_explore on the channel's videos.
- scheduler explore_cleanup job + explore_grace_days config: hard-delete explored-but-
unkept channels and their untouched ephemeral videos after a grace period.
Makes the local search behave more like YouTube's — finding videos by the
uploader's own keywords or the query that surfaced them, not only words in the
title. A DB-generated, weighted search_vector (migration 0032) replaces the
title-only FTS index:
- keywords: the creator's snippet.tags (free — already in the snippet we fetch),
stored on enrich.
- search_terms: distinct live-search queries that surfaced the video (across all
users), appended by the search route — folds YouTube's relevance into local
search (a video YT returned for a query becomes findable by it even without a
title match), the user's own idea.
- description (truncated) for broad recall on the existing catalog.
Weighted title(A) > keywords+queries(B) > description(C) so ts_rank keeps title
hits on top. A plain GIN index on the generated column guarantees index use (no
expression/param matching). Verified on localdev: recall 146->213 for one query;
7 'eurovision' hits via the document but not the title; index scan confirmed.
Two related search improvements:
1) Your own live-search results now belong to your Mine feed. A new per-user
search_finds table (migration 0030) records each video you surface via your
YouTube search (the route inserts them idempotently). The Mine feed becomes
'your non-hidden subscriptions OR your search finds', and the Source filter
now applies in Mine too: organic = subscriptions, search = your search finds,
all = both (default stays organic, so the main feed is unchanged). The shared
Library keeps using the global via_search flag.
2) Feed search ranks by relevance instead of a whole-phrase substring. A custom
unaccent_simple text-search config + GIN index (migration 0031) back a
YouTube-like fuzzy match: word-order-independent, multi-word AND, prefix on
the word being typed, accent-insensitive. A new 'relevance' sort orders by
ts_rank; the channel name still matches as a substring. The rank is scaled to
an integer so the keyset cursor pages it exactly (a raw float4 breaks paging).
_filtered_query returns the rank expr so only the feed list uses it.
Live YouTube search can now use YouTube's internal InnerTube endpoint instead
of search.list, materialising results through the same enrich/provenance path
at zero API quota (search.list costs 100 units/page; scrape costs nothing, only
the cheap shared videos.list enrich is charged).
- youtube/search_scrape.py: InnerTube search returning the same page shape as
YouTubeClient.search_videos (items + continuation cursor); SOCS consent cookie,
videoRenderer walk (channels/playlists/Shorts shelves naturally excluded),
type:video filter, cached InnerTube key/version with constant fallbacks.
- routes/search.py: admin-selectable source (search_source, default scrape).
Scrape path skips the 100-unit budget pre-check and logs a zero-cost search
event so the per-user daily cap still counts it; api path unchanged. Response
carries the active source.
- sysconfig/config: new search_source key (scrape|api).
- quota.log_action: record a zero-cost action event for per-user rate limits.
Feed text search used plain ILIKE, which is case- but not diacritic-insensitive, so 'tiesto'
missed the many titles spelled 'Tiësto' — a search that ingested ~45 results showed only ~12.
Enable the postgres unaccent extension (migration 0029) and wrap both sides of the title/
channel match in unaccent(), so 'tiesto' now matches 'Tiësto'. Applies to feed, count and
facets alike.
Replace the binary 'show search-discovered' toggle with a 3-way Source selector in the
Library toolbar, so users can also see ONLY search-discovered videos — not just hide or
mix them. Backend: feed param library_source = organic (default, hides via_search) | all
(both) | search (only via_search), applied in scope=all. Strings in HU/EN/DE.
Add a live YouTube search that materialises results into the shared catalog so
they render with the normal feed cards + in-app player and gain per-user state.
- YouTubeClient.search_videos(): search.list (100 units), embeddable-only, returns
flat stubs + nextPageToken; surfaces liveBroadcastContent for live filtering.
- routes/search.py GET /api/search/youtube: require_human + per-user daily cap
(search_daily_limit_per_user, default 70) + can_spend pre-check (429 on either);
drops live/upcoming, upserts channel stubs (channels.list) + video stubs, enriches
(videos.list), runs the youtube.com/shorts probe, then excludes Shorts/live and
returns feed cards in relevance order with the YouTube pageToken as the cursor.
- Provenance: videos.via_search / channels.from_search (migration 0028) flag
search-discovered rows; the feed hides them from the Library (scope=all) by default
via exclude_search_discovered, leaving the Mine feed untouched.
- quota.actions_today() counts a user's per-action events today for the cap; only the
search.list call is attributed VIDEOS_SEARCH so the counter is exactly 1 per search.
- feed_columns() is the single feed-row projection, consumed by both /feed and the
playlist-detail query (was copy-pasted, drift risk on every new column).
- _replace_items_from_youtube() dedups the YouTube playlist-mirror block shared by
sync_user_playlists and repull_playlist.
- _next_playlist_position/_next_item_position/_add_item replace the append-position
idiom (x4) and the exists-then-insert blocks (x2).
- quota.measured() context manager folds the before/after units diff that the
manual sync routes each re-spelled (also makes quota_remaining_today consistent).
- sync pause/resume now use Depends(admin_user) instead of inline role checks.
- is_messageable_user() unifies the 'real, active, non-suspended human' rule that
was encoded three ways (WS auth, send recipient, and the SQL _messageable()).
- app/utils.py: shared valid_email() + now_utc() (one email regex, was duplicated
in auth.py and admin.py with a looser "@"-in check elsewhere).
- security.hash_token(): one SHA-256 token hasher (was duplicated in auth.py + state.py).
- auth.admin_user dependency + count_admins() helper, replacing the inline role
checks and the last-admin count query repeated across admin.py/me.py/tags.py.
The live-message push now carries both parties, so an incoming message opens
that conversation's dock window if it's closed, or flashes it once if it's
already open (expanded or minimised, without disturbing the minimised state) —
only for messages from someone else, never your own echo.
Dock state (open windows + minimised state) is persisted per user, so a reload
restores exactly what was open, minimised, or closed.
Private user-to-user messages are end-to-end encrypted: the server only ever
stores ciphertext + iv and acts as a key directory (public keys) plus an opaque
store for each user's private key, wrapped client-side with a passphrase the
server never sees — so not even an admin can read a conversation. A separate
kind=system message (plaintext, no sender) powers a server-authored Siftlode
welcome shown on first open, reusable later for announcements.
- models: rework Message (kind, nullable sender/body, ciphertext+iv) + MessageKey;
migrations 0026 (table) + 0027 (E2EE rework).
- routes/messages.py: key directory/blob endpoints, ciphertext send, conversations
+ threads (system + user), lazy welcome, all gated by require_human.
- realtime.py: in-process WebSocket connection registry; /ws delivers sent
messages to a user's open tabs instantly (sync-callable push, single-process).
Replace deep-OFFSET paging (which scans-and-discards skipped rows on the
233k-row catalog and shifts/duplicates rows as the scheduler ingests mid-
scroll) with stable keyset pagination over an opaque cursor.
A single sort registry drives both the ORDER BY and a generic, NULL-aware
keyset WHERE so they can never drift, with Video.id as the unique final
tiebreaker. Covers every sort: newest/oldest, views, duration, title,
subscribers (joined), priority (coalesced multi-key) and the seeded
shuffle. The response now returns next_cursor instead of offset.
Replace the leftover 'subfeed' name across logger names + log_config,
frontend localStorage keys, Postgres user/db/volume defaults in the
compose files, .env.example, config.py, backup/restore scripts and the
README. Pure rename; no behavioural change. localStorage keys move from
subfeed.* to siftlode.* (one-time UI-state reset is acceptable).
