Replace the binary 'show search-discovered' toggle with a 3-way Source selector in the
Library toolbar, so users can also see ONLY search-discovered videos — not just hide or
mix them. Backend: feed param library_source = organic (default, hides via_search) | all
(both) | search (only via_search), applied in scope=all. Strings in HU/EN/DE.
Add a live YouTube search that materialises results into the shared catalog so
they render with the normal feed cards + in-app player and gain per-user state.
- YouTubeClient.search_videos(): search.list (100 units), embeddable-only, returns
flat stubs + nextPageToken; surfaces liveBroadcastContent for live filtering.
- routes/search.py GET /api/search/youtube: require_human + per-user daily cap
(search_daily_limit_per_user, default 70) + can_spend pre-check (429 on either);
drops live/upcoming, upserts channel stubs (channels.list) + video stubs, enriches
(videos.list), runs the youtube.com/shorts probe, then excludes Shorts/live and
returns feed cards in relevance order with the YouTube pageToken as the cursor.
- Provenance: videos.via_search / channels.from_search (migration 0028) flag
search-discovered rows; the feed hides them from the Library (scope=all) by default
via exclude_search_discovered, leaving the Mine feed untouched.
- quota.actions_today() counts a user's per-action events today for the cap; only the
search.list call is attributed VIDEOS_SEARCH so the counter is exactly 1 per search.
TimestampMixin (created_at) + UpdatedAtMixin (updated_at) replace the repeated
column blocks across 9 + 3 models. Models that index created_at or default it
Python-side keep their own column on purpose. Verified zero schema drift via an
identical alembic-check diff before/after.
- feed_columns() is the single feed-row projection, consumed by both /feed and the
playlist-detail query (was copy-pasted, drift risk on every new column).
- _replace_items_from_youtube() dedups the YouTube playlist-mirror block shared by
sync_user_playlists and repull_playlist.
- _next_playlist_position/_next_item_position/_add_item replace the append-position
idiom (x4) and the exists-then-insert blocks (x2).
- quota.measured() context manager folds the before/after units diff that the
manual sync routes each re-spelled (also makes quota_remaining_today consistent).
- sync pause/resume now use Depends(admin_user) instead of inline role checks.
- is_messageable_user() unifies the 'real, active, non-suspended human' rule that
was encoded three ways (WS auth, send recipient, and the SQL _messageable()).
- app/utils.py: shared valid_email() + now_utc() (one email regex, was duplicated
in auth.py and admin.py with a looser "@"-in check elsewhere).
- security.hash_token(): one SHA-256 token hasher (was duplicated in auth.py + state.py).
- auth.admin_user dependency + count_admins() helper, replacing the inline role
checks and the last-admin count query repeated across admin.py/me.py/tags.py.
Watch later became a built-in playlist, so "saved" is no longer a VideoState
status — the scan matched nothing and under-counted users impacted by a video
removal. The playlist loop already covers Watch later, so impact is now correct.
Route the local _now through the shared utils.now_utc.
New youtube_api_proxy setting (env fallback + admin Config UI, env YOUTUBE_API_PROXY):
when set, the YouTubeClient routes all its httpx traffic through that HTTP(S) proxy.
Lets a dynamic-IP host send API calls through a fixed-IP host (e.g. the server over
private tunnel) so an IP-restricted API key keeps working. Empty = direct.
The live-message push now carries both parties, so an incoming message opens
that conversation's dock window if it's closed, or flashes it once if it's
already open (expanded or minimised, without disturbing the minimised state) —
only for messages from someone else, never your own echo.
Dock state (open windows + minimised state) is persisted per user, so a reload
restores exactly what was open, minimised, or closed.
Private user-to-user messages are end-to-end encrypted: the server only ever
stores ciphertext + iv and acts as a key directory (public keys) plus an opaque
store for each user's private key, wrapped client-side with a passphrase the
server never sees — so not even an admin can read a conversation. A separate
kind=system message (plaintext, no sender) powers a server-authored Siftlode
welcome shown on first open, reusable later for announcements.
- models: rework Message (kind, nullable sender/body, ciphertext+iv) + MessageKey;
migrations 0026 (table) + 0027 (E2EE rework).
- routes/messages.py: key directory/blob endpoints, ciphertext send, conversations
+ threads (system + user), lazy welcome, all gated by require_human.
- realtime.py: in-process WebSocket connection registry; /ws delivers sent
messages to a user's open tabs instantly (sync-callable push, single-process).
Replace deep-OFFSET paging (which scans-and-discards skipped rows on the
233k-row catalog and shifts/duplicates rows as the scheduler ingests mid-
scroll) with stable keyset pagination over an opaque cursor.
A single sort registry drives both the ORDER BY and a generic, NULL-aware
keyset WHERE so they can never drift, with Video.id as the unique final
tiebreaker. Covers every sort: newest/oldest, views, duration, title,
subscribers (joined), priority (coalesced multi-key) and the seeded
shuffle. The response now returns next_cursor instead of offset.
Replace the leftover 'subfeed' name across logger names + log_config,
frontend localStorage keys, Postgres user/db/volume defaults in the
compose files, .env.example, config.py, backup/restore scripts and the
README. Pure rename; no behavioural change. localStorage keys move from
subfeed.* to siftlode.* (one-time UI-state reset is acceptable).
- /api/me exposes instance-wide google_enabled. The onboarding nudge no longer auto-opens, and
Settings hides the YouTube-access section + the 'Connect Google' option, when the instance has
no Google OAuth configured (e.g. an email+password-only self-host). A linked account still shows
its 'Connected' status.
- POST /api/setup/admin (create the first admin: active+verified+admin), /config (persist any
registry-backed settings — Google creds, SMTP, quota…), /test-email, and /finish (mark
configured + invalidate the token). All behind require_setup (valid token AND not configured).
- GET /api/setup/info reports secrets_manageable so the wizard hides the Google/SMTP secret steps
on a box without TOKEN_ENCRYPTION_KEY.
- app_state gains configured + setup_token_hash (migration 0025); existing installs (any users)
are marked configured so they never see the wizard, a fresh DB starts in setup mode.
- On first boot the app mints a one-time setup token and logs the wizard URL (only the hash is
stored); the token gates the setup steps (added in 6c).
- A middleware locks all /api + /auth routes (except /api/setup, /api/version, /healthz, static)
to 503 until configured, with a DB-free cached fast-path once setup completes.
- GET /api/setup/status tells the SPA whether to show the wizard.
- Build the Google OAuth client lazily from sysconfig (DB override, env fallback) and re-register
when the credentials change, so the install wizard / admin can set them without a restart.
- New 'google' config group (google_client_id/secret, encrypted) on the Configuration page; the
token-refresh reads the creds the same way.
- Public GET /auth/config exposes google_enabled; the login page hides 'Continue with Google'
when Google OAuth isn't configured (email+password still works).
- Serve real files at the SPA root (e.g. /welcome/*.png, favicon) from the built app — the
landing screenshots were only ever shown by the Vite dev server, never in production.
- Add the Channel-manager screenshot; the two secondary previews are smaller thumbnails that
open in a custom full-size lightbox (fit-to-viewport, Esc/backdrop/✕ to close).
- Drop a suspended/deleted user to the login page on the next 401 (and poll the session only
while signed in, so the public login page no longer flickers); 'suspended' and 'account
deleted' confirmation banners; strip redirect query params for a clean address bar.
- New Users page tabs (Users & roles / Access requests / Demo) and a tab-ified Configuration
page, both via a reusable Tabs component (persisted active tab).
- Admin can suspend/unsuspend (migration 0023 adds users.is_suspended) and delete accounts
from the Users & roles tab, with guards (demo / self / last admin).
- Email notifications to the affected user: suspended (reactive, on a blocked sign-in),
reinstated, role changed, and account deleted; the approval email now carries a clickable
app link. The scheduled/manual demo reset also seeds sample channel subscriptions.
- Hide the 'unverified email' chip for Google accounts (Google attests the email).
- Link a Google account to a password account, and adopt the Google identity onto a matching
email account instead of 500ing on a duplicate; set or change a password from Settings.
- Expose has_google/has_password on /api/me for the Sign-in methods UI.
- Mark Google logins email-verified (backfill existing rows, migration 0024); stop a routine
login from clobbering an admin-assigned role (env ADMIN_EMAILS stays the bootstrap admin).
- Suspension login-gates (password + Google callback + current_user) with a rate-limited
'suspended' notice; shared purge_user (cascade delete + access-request cleanup + Google-grant
revoke) behind self- and admin-deletion; single app_base source for user-facing email links.
Add DELETE /api/me/account: permanently erases the signed-in account and all its
personal data — OAuth tokens, subscriptions, tags, video states, playlists,
notifications all cascade on the users row (FK ON DELETE CASCADE); quota-audit
events are kept but anonymised (SET NULL); the email's invite/whitelist row is
removed too. Guards: the shared demo account can't be deleted, and the last
remaining admin can't delete itself. Frontend: a Danger zone in Settings -> Account
(non-demo) with a danger-confirm; on success the session clears and the app lands
on the welcome page. EN/HU/DE. Verified via curl: self-delete + session clear +
demo 403.
A security audit of every mutating endpoint found no isolation gaps — all routes
scope by current_user, admin routes are gated, and the demo account is sandboxed
in its own user_id space (can't reach others' data, escalate, or hit admin
routes). The remaining demo concern is communal pollution of its own shared state,
so add an automatic reset: a new 'demo_reset' scheduler job (admin-tunable
interval, default 12h) reuses the manual reset logic, and no-ops without ever
creating a demo account if none exists. Verified: a triggered run wiped the demo's
video states and re-seeded its sample playlists.
Add email+password auth alongside Google: argon2id hashing; users gains
password_hash/email_verified/is_active and google_sub becomes nullable (migration
0022); a single-use, hashed auth_tokens table for email verification + password
reset. Registration creates a pending (inactive, unverified) account + an access
request; sign-in needs verified email AND admin approval. Anti-enumeration: uniform
register/login/reset responses (a correct-password owner still gets a specific
pending reason). allow_registration flag (DB-tunable). Admin users-list + role
endpoint (guards: not self, not demo, not the last admin); approving access (or a
manual whitelist add) now activates the matching pending account. current_user
rejects deactivated accounts. Verified end-to-end via curl + DB.
Register youtube_api_key in the sysconfig registry as a secret and resolve it from
the DB-override-or-env layer in YouTubeClient (self.db is in scope). Adds a YouTube
API group to the Configuration page (EN/HU/DE).
Google OAuth client id/secret stay env-only for now: the authlib client is
registered at module load (auth.py), and the token-refresh reads them too, so a
runtime DB override needs a lazy/re-registerable OAuth client — that belongs to the
install-wizard epic (first-boot OAuth config without restart). Admin-role-via-UI
moves to the auth-overhaul epic (user management lives there).
Register 8 more keys in the sysconfig registry (quota daily budget + backfill
reserve, recent-backfill window, shorts-probe params, enrich/autotag batch sizes)
and route their reads through the DB-override-or-env resolver at every call site
(quota.py, sync/runner.py, sync/videos.py, sync/autotag.py, sync/maintenance.py,
routes/scheduler.py). All read sites already had a db session in scope. Reads are
read-through (no cache), matching app.state; admin can tune them live, no redeploy.
Add a generic admin-editable config layer (epic 4a): system_config KV table
(migration 0021), a sysconfig registry that is the single source of truth for
DB-overridable keys (type/group/default/bounds/secret) + a DB-override-or-env
resolver, and admin endpoints (GET/PATCH/DELETE /api/admin/config + a test-email
probe). Secrets are Fernet-encrypted at rest (TOKEN_ENCRYPTION_KEY); without it
they stay env-only. First group wired end-to-end: Email/SMTP — email.py now
reads host/port/user/from/password via the resolver (own session, since email
is sent from background tasks).
The quota-attribution action keys were inconsistent (mixed verbs, ad-hoc names,
English-only labels). Introduce a QuotaAction constants holder (one source of
truth), rename every attribution call site to it, and add migration 0020 to
rename historical quota_events.action values. The display label now resolves
from i18n (quotaActions.<key>, EN/HU/DE) instead of a hard-coded English map.
Scheduler job ids and progress phase labels are a separate namespace and are
left untouched.
Same fix as the subscribe reconcile, applied to the remaining endpoints that
wrapped a YouTubeError in a 502: video lookup (feed) and playlist refresh,
push-plan, sync, and delete. A 502 is treated by the client as a transient
"connection lost" blip, so a real, explainable YouTube failure showed the wrong
message. 422 carries the clear detail to the error dialog instead.
Subscribing to a channel the user already follows on YouTube (a local/YouTube
desync — e.g. a stale import dropped the local row, so it resurfaced in the
discovery list) made YouTube return 400 "subscription already exists", which
the endpoint reported as a 502 — and the client shows a 502 as a transient
"connection lost" blip, not the real reason. Treat the duplicate as success:
record the subscription locally (the desired end state already holds; the next
resync fills in the resource id). Surface any other YouTube error as 422 with a
clear message, on both subscribe and unsubscribe, so it isn't mistaken for a
connection drop.
The header's sync indicator spun "fetching history" whenever any channel
still lacked full history — i.e. perpetually, even when the scheduler was
idle between its periodic runs. It now spins only while a channel-sync job
(backfill or RSS poll) is actually running; otherwise pending deep-history
work shows as the calm, static "N without full history" link, and recent
work queued for the next run shows a static "N queued". This makes the
header coherent with the Scheduler dashboard's live state.
Backend exposes running_job_ids() from the scheduler activity and a derived
sync_active flag on /api/sync/my-status.
Every background job now reports progress, so the dashboard can show a live
bar for any run (not only enrich/backfill/shorts/maintenance): rss_poll and
subscriptions (runner), autotag, and playlist_sync gained progress.report
calls over their loops.
RSS polling now fetches the channel feeds concurrently (16 workers) instead
of one-at-a-time — a slow/unreachable feed no longer blocks the rest, cutting
a full poll of the catalogue from minutes to seconds. Feed fetches are
network-bound and run in the pool; DB inserts stay on the session's thread
(SQLAlchemy sessions aren't thread-safe), applied as each fetch completes.
Split the DB-write half of poll_rss_channel into apply_rss_feed so the fetch
and apply phases compose cleanly.
Add a "Discover from playlists" tab to the Channel manager that lists
channels appearing in the user's playlists they don't subscribe to, with
a one-click Subscribe.
- GET /api/channels/discovery: local join (playlist_items -> videos ->
channels) minus the user's subscriptions and their own channel. Enriches
stub channels' metadata up front (title/thumbnail/subscriber count via the
API key) so the user can judge a channel before subscribing; videos are
not pulled (the scheduler picks those up).
- POST /api/channels/{id}/subscribe: write-scope gated, subscriptions.insert
+ local Subscription with the returned resource id.
- YouTubeClient.insert_subscription / get_my_channel_id.
- users.yt_channel_id (migration 0019) caches the user's own channel id so
discovery can exclude it.
- Frontend: ChannelDiscovery DataTable, Channels tab toggle (persisted),
api methods, trilingual strings. Subscribe ships a typed notification
payload (ChannelSubscribedMeta) for the inbox to act on.
uvicorn's default --timeout-keep-alive is 5s, shorter than Caddy's pooled
keepalive idle window, so the proxy reused connections uvicorn had just
closed -> broken pipe / connection reset -> 502. The player's progress/state
POSTs fire every 5s (resonant with the 5s default) and, being non-idempotent,
could not be auto-retried by the proxy, so those 502s reached the user.
Set --timeout-keep-alive 75 so the upstream outlives the proxy's reuse.
Expose the maintenance job's rolling re-validation batch (videos re-checked per
run) as an admin control on the Scheduler dashboard, stored in app_state
(migration 0018; NULL = the env/config default). The job reads the effective
value each run via a state helper, clamped to a sane range. Trilingual.
Add per-job "Run now" buttons and a "Start all now" button to the admin Scheduler
dashboard (admin-gated endpoints). Triggers run the job in a background thread
independent of its interval, refusing a concurrent run (409). While running, the
long jobs (maintenance, enrich, backfill, shorts) report live progress through a
decoupled contextvar sink, shown as a progress bar on the job row via the existing
4s poll. A manually-triggered run posts a completion notification to the triggering
admin's inbox (scheduled runs stay silent to avoid spam); the inbox renders the
"scheduler" type trilingually from type+data. While here, give the maintenance job
its missing dashboard label/description in all three languages.
Add a daily maintenance/validation job that detects videos which can't be played
anywhere and retires them safely. Two phases: re-check already-flagged videos
(recover if available again, else hard-delete once the grace period elapses,
cascading to states/playlist items), and a rolling re-validation of the
least-recently-checked currently-available videos that flags newly-unplayable
ones (hidden from the feed immediately via unavailable_since).
Detection is ~free: a video missing from the videos.list response is
deleted-or-private; an `upcoming` premiere >2 days past its scheduled start that
never went live is abandoned. A still-live broadcast is kept (legit 24/7 stream).
Enrichment now also fetches part=status to populate the status columns. Grace is
7 days for removed videos, none for abandoned. Before deleting, affected users
get one batched notification (never per-video). Interval is admin-tunable via the
Scheduler dashboard; batch size and grace are config. Quota-attributed to the
system and bounded by the same backfill reserve as the other jobs.
Add a server-backed notification center that coexists with the client-side
transient bell: a per-user `notifications` table (type/title/body/data JSON/
read/dismissed), a `/api/me/notifications` CRUD API (list, unread_count, read,
read_all, dismiss, clear), and a left-nav inbox module with a live unread badge
polled via useLiveQuery. Known types render trilingual text from type+data
(English stored text is the fallback); read rows are trimmed past a soft cap.
Also adds the schema the maintenance job builds on: videos.list?part=status
columns (embeddable/privacy_status/upload_status) and the validation lifecycle
columns (last_checked_at, unavailable_since, unavailable_reason).
Page-id validation is centralized in one PAGES source of truth (isPage) so the
new page survives reload without a second allowlist to keep in sync.
- Channel-row tags show only what's attached; a '+' opens a per-channel picker. Clicking a tag
filters the feed by it; a 'Your tags' sidebar widget makes that filter visible/clearable.
- Tag manager dialog (add/rename/delete; delete only confirms when the tag is in use), reachable
from the Channel manager and the feed sidebar; hovering a tag's count lists its channels, each a
link that focuses it in the Channel manager (DataTable gains an external filter input).
- Video cards get a reset action that clears all watch state (incl. an in-progress position).
- api.req() now raises the error dialog on 5xx and 400/409/422 with the server's reason.
- Duplicate-tag create/rename hit the uq_tags_user_name constraint and 500'd; now caught and
returned as a clear 409 ('You already have a tag named …').
- New global error dialog (errorDialog store + ErrorDialog modal) for definitive server refusals;
the api layer wiring + mount land with the rest of the round.
- Modal now keeps a stack so ESC dismisses only the top modal — an error over the tag editor
closes the error and leaves the editor open.
- Add Last upload, total Length and a Normal/Shorts/Live breakdown, computed on read in the
existing grouped query (~35ms over the full catalog — measured, so no denormalization).
- Subscribers shown compact (919K), since the YouTube API already rounds them.
- Tag cell now shows only the tags actually attached to a channel; a '+' opens a per-channel
picker to attach/detach, instead of listing every user tag on every row.
POST /api/channels/{id}/reset-backfill (admin-only): clear the channel's backfill
markers, re-opt into deep backfill and re-run its recent pull now — a re-fetch-from-
scratch trigger regardless of current sync state. Idempotent (videos upsert by id).
The in-app player marks a video watched when it reaches the end, firing
POST /videos/{id}/state at the same instant as a progress checkpoint. The
progress endpoint deletes the 'new' video_states row near the end, so the
concurrent state UPDATE matched 0 rows and raised StaleDataError -> 500. The
watched status was never persisted (and Feed swallowed the error), so the
video stayed in the unwatched feed even after a refresh.
- set_video_state: atomic INSERT ... ON CONFLICT (uq_user_video) DO UPDATE for
watched/hidden, immune to a concurrent delete; tolerate StaleDataError on the
'new' branch and in the progress endpoint.
- PlayerModal: skip the redundant near-end progress checkpoint when we just
auto-marked watched, removing the self-inflicted race.
- Feed: drop the optimistic override if the server rejects the change, so a
failed request can't leave a card phantom-hidden.
Add scheduler_settings (per-job interval override, migration 0015) and a
PATCH /api/admin/scheduler/jobs/{id} that persists the override and live-
reschedules the running APScheduler job. Intervals load from the DB (env
defaults as fallback); the snapshot reports the live trigger interval.
Record each scheduler job's run activity in-process (running / last result /
error / timestamps) and expose a snapshot. New admin-only GET
/api/admin/scheduler returns the per-job activity (with APScheduler next-run
times), the DB-derived work still queued (channels/videos pending, deep ETA,
live-refresh backlog) and the shared quota picture.
enrich_pending only touches enriched_at IS NULL, so a video first seen
while live was stamped live + duration-null and never revisited — staying
'live' with no duration forever after the broadcast ended. Add refresh_live
(run after each enrich pass) that re-fetches anything still live/upcoming,
plus just-ended was_live videos that haven't got their duration yet, until
they settle. Cheap: videos.list is 1 unit per 50 ids.
The shared demo account no longer hits YouTube affordances: the onboarding
wizard never opens (or renders) for it, the empty-feed prompt nudges into
the shared library instead of the connect wizard, and the browser-facing
/auth/upgrade redirects the demo home instead of returning a raw 403 JSON.
Admin endpoints to manage the demo email whitelist (DB-backed, no env)
and a manual reset that wipes the demo account's per-user state and
re-seeds a few sample playlists from the shared catalog.
Whitelisted emails enter the shared demo user via /auth/demo (lazily
created, no OAuth token/scope), rate-limited per IP and answering
uniformly so it can't be hammered as an enumeration oracle. Add a
require_human dependency that blocks the demo account from quota-spending
sync endpoints and the OAuth upgrade flow, and surface is_demo on /api/me.
Add the shared demo-account plumbing: users.is_demo marks the single
shared demo user, demo_whitelist holds the admin-curated emails that may
enter it without Google sign-in, and a small in-process RateLimiter
(generic groundwork) for throttling the demo-login endpoint per IP.
1) Move the Show view filter (Unwatched/In progress/All/Watched/Hidden) up into the
toolbar chip row as its own single-select group, divided from the content-type chips;
removed the 'show' sidebar widget (sidebar now = Upload date / Language / Topic).
2) Rework ordering like the Playlists page: a sort-key dropdown (Date, Popular, Duration,
Name, Channel subscribers, Channel priority, Surprise me) + a single asc/desc arrow
toggle, instead of separate directional entries. 'Most viewed' is now 'Popular'.
Backend gains the missing directions (views_asc, title_desc, subscribers_asc,
priority_asc); the frontend maps (key, dir) -> the backend sort string, so FeedFilters
is unchanged. Trilingual (feed.sortKey.*, feed.dirAsc/dirDesc). Feed.tsx normalized to LF.
