Every download now records the clean source page URL and shows it on the
Downloads page (open in a new tab, or copy to clipboard). The worker stores
yt-dlp's canonical webpage_url on the asset (migration 0043 adds
media_assets.source_webpage_url); the serializer prefers it and falls back to a
URL derived from source_kind+source_ref, so YouTube, external YouTube links and
external URLs (e.g. Facebook reels) all get a correct reference, and queued/older
rows work before the worker fills it. Edit clips return null (a clip's source is
the user's own earlier download, not a web page). i18n en/hu/de.
The whole app shipped in one bundle, so the logged-out landing and every page
pulled all module code. Split into lazy chunks:
- main.tsx: lazy App + the public leaves (WatchPage, Privacy, Terms), so a
public /watch share link never downloads the authenticated app.
- App.tsx: each module page (Feed, Channels, Playlists, Stats, Scheduler,
Config, Users, Settings, Notifications, Messages, Downloads, ChannelPage) and
the About/ReleaseNotes/Onboarding modals load on demand behind <Suspense>.
- Heavy modals lazy in their parents: PlayerModal (Feed, Playlists),
DownloadDialog (DownloadButton — kept out of the feed chunk), and the
VideoEditor/ShareDialog/ProfileEditor (DownloadCenter).
- Extracted focusAccessRequestsTab + the tab constants to lib/adminUsersTab so
callers can pre-select the admin tab without statically importing the now
lazy-loaded AdminUsers page (which would defeat the split).
Build now emits ~25 chunks. Landing no longer downloads any module code
(~350 KB deferred); dev landing Perf 93->95. Verified in a real browser: every
page + the video editor load with no console errors.
Give the feed card thumbnail width/height (16:9) so the browser reserves the
aspect box before load (CLS), and decoding=async so image decode never blocks
the main thread. loading=lazy was already set.
Lighthouse a11y across the authenticated module pages flagged: the clickable
Siftlode logo had aria-label='Feed' (accessible name didn't include its visible
text); the shared Switch, theme swatches, sort selects and several number/range
inputs had no accessible name/label. Give Switch role='switch'+aria-checked and
an optional label (passed at every call site), aria-label the swatches, selects
(feed/playlists sort) and the settings/config/scheduler inputs, and drop the
mismatched logo aria-label so its visible text is the name. All 11 module pages
now score 100 accessibility (settings 86->100, playlists 89->100, others 92-95->100).
The logged-out landing probed /api/me, which 401'd, and the browser logs every
non-2xx fetch as a console error regardless of how the app handles it. The
bootstrap probe now returns 200 with {authenticated:false} via a new
optional_current_user dependency; api.me() maps that to null. The render gate
treats no-data as 'signed out' -> the landing, and a thrown error as a real
network/5xx failure. Other protected endpoints still 401, so mid-session expiry
is still caught by the global handler. Lighthouse (dev): Best Practices 96->100.
The public landing had no meta description and no robots.txt (the SPA fallback
returned index.html for /robots.txt, so crawlers saw an invalid file). Add a
description + basic OG tags to index.html and a real robots.txt that allows the
landing but disallows the API/auth/watch surfaces. Lighthouse (dev): SEO 83->100.
The 3 landing screenshots were 2.3 MB of PNG (feed.png alone 1.6 MB, decoded
at 2400x1350 for a ~400px slot) — the biggest LCP/transfer cost on the public
page. Re-encoded to WebP capped at 1600px (~284 KB total, feed 91% smaller).
Also set Cache-Control: content-hashed /assets/* are immutable for a year,
other SPA-root static files (welcome images, favicon, robots) get a 7-day TTL;
index.html stays no-cache. Register image/webp+avif mimetypes so FileResponse
serves the right Content-Type. Lighthouse (dev): Perf 80->93, LCP 1.8s->1.2s.
mp4 downloads used yt-dlp's default best video+audio, which on YouTube is
VP9+Opus at 1080p. Chromium plays that, but iOS/Safari WebKit (which every
iOS browser is forced to use) decodes only H.264/H.265+AAC inside mp4 — so a
shared /watch link showed a broken player on iPhone/iPad while the download
still worked. Rank vcodec:h264 + acodec:aac above resolution for mp4 output
(unless a custom profile pins a vcodec), keeping VP9/AV1 as a graceful
fallback. Bump the format signature (v2) so an identical spec gets a fresh
cache identity instead of hitting a stale VP9 asset.
The worker and API start in parallel; the API applies migrations on its own startup, so on a fresh
deploy the worker hit a missing download_jobs table and crash-looped until migrations landed. The
worker now polls for the schema before recovering orphans, and (belt-and-suspenders) the compose
files gate it on the API being healthy. Fixes the observed prod restart-loop on the v0.22.0 deploy.
Prep the Download Center epic (Phase 1 + editor + share) for prod/self-host:
- Dockerfile: create /downloads owned by appuser so a named-volume mount is writable (prod Linux).
- docker-compose.{home,selfhost,yml}: add the 'worker' (yt-dlp/ffmpeg job loop) + 'bgutil-pot'
(PO-token) services + a downloads mount (DOWNLOAD_ROOT, WORKER_ENABLED). Media defaults to a
named volume; DOWNLOAD_HOST_PATH points it at a host dir (e.g. a Plex-readable folder).
- README / docs/self-hosting.md / .env.example / install.{sh,ps1}: document the Download Center,
the two extra containers, and DOWNLOAD_HOST_PATH.
- VERSION 0.22.0 + releaseNotes entry.
Session-close cosmetics:
1. Favicon — an 'S' monogram (indigo→violet rounded square, path-drawn so it's font-independent)
at public/favicon.svg, linked in index.html (tab had none before).
2. Dynamic document.title — reflects the current module ('Downloads · Siftlode'); Feed = brand
only; an open channel page shows the channel name. Shared pageTitleKey() in lib/pageMeta.ts
keeps the tab title and the top-bar header in lockstep.
3. The 'Siftlode' logo already navigated to Feed but read as static text — added a hover
background + cursor affordance so it's clearly clickable.
4. Module header — new shared PageTitle component (used by every non-feed module via Header) with
a tracked small-caps 'eyebrow' + leading accent dot (user-picked style B), styled in one place.
Verified in a real browser: favicon served (image/svg+xml), title updates per module, logo→feed,
header restyled; no console errors.
The 'Your storage' footprint used a one-shot query, refetched only at enqueue time — when the job
is still queued (0 bytes) and the file lands asynchronously via the worker. So the footprint stayed
stale (0 B) until a manual reload. Switch the UsageBar to useLiveQuery(2s), like the library list,
so the footprint updates in lockstep the moment a clip/download completes (and after a delete).
A shared-with-me item only had a download button. Add two actions (Share is intentionally NOT
offered — no chain re-sharing of someone else's file):
- Edit: the editor now accepts an accessible (owned OR shared) source, so editing a shared video
produces the editor's OWN clip in their library (counts against their quota, fully theirs
including share); the source file is only read. Route uses _accessible_job.
- Remove from my list: DELETE /api/downloads/shared/{job_id} deletes only the recipient's share
grant — the owner's job and physical file are untouched (per-user dismissal).
i18n en/hu/de. Verified in a real browser (edit a shared 70-min video → own 2:31 clip; remove
confirm shows 'won't delete the owner's file').
The admin System tab only listed users with an existing footprint, so a quota could be set only
after someone downloaded something. Add a 'Set a user's quota…' picker (over /api/admin/users,
excl. demo) beside the Per-user footprint heading that opens the quota editor for ANY user — the
quota GET/PUT endpoints already resolve defaults + create a row on demand, so no backend change.
Also show an empty-state line when no one has a footprint yet. i18n en/hu/de.
The download-center GC job showed its raw id 'download_gc' with a raw-key tooltip, unlike every
other job. Add its label ('Download cleanup') + description to scheduler i18n (en/hu/de), and make
the job tooltip fall back to empty for any job lacking a description (so a future/edge job never
shows a raw 'scheduler.jobDesc.*' key — also fixes demo_reset).
The tooltip clamped its centre to a fixed 92px margin, but the caption is up to 240px wide
(120px half), so a left-edge anchor (e.g. the nav-rail counts icon at x~40) pushed the centred box
~28px off the left of the viewport. Clamp with the caption's max half-width so it can't overflow on
first paint, then a useLayoutEffect re-centres on the anchor using its actual width. Verified: the
nav-rail counts tooltip now sits fully on-screen.
In the left-nav rail SyncStatus, the pause button rendered as its own bottom block, so with only
deep-history pending (idle) it dropped to a lonely line under 'N without full history'. Now it sits
right-aligned on the primary status row — the sync-state line when there is one, otherwise the
'N without full history' row — via flex justify-between. No change to the all-synced state.
Rework the share dialog into two clear modes and add the public /watch player page:
- ShareDialog: (A) 'Share with a user' — autocomplete picker over registered users (was a blind
email box that 404'd on non-users); (B) 'Share a link' — create/list/copy/revoke public links
with allow-download toggle, optional expiry (1/7/30d), optional password; per-link view count.
- WatchPage: standalone login-free player at /watch/<token> (routed in main.tsx like /privacy),
self-contained mini-i18n (en/hu/de by browser language); password gate → unlock → play; shows a
Download button only when the link allows it.
- api: ShareLink/ShareRecipient types + link CRUD + recipients; share i18n (en/hu/de).
Verified end-to-end in a real browser: user picker, link create, public playback, stream-only vs
downloadable, password gate + unlock, no console errors.
Share a download by a capability URL (/watch/{token}) that anyone can play on a login-free page —
distinct from the registered-user ACL share. Per-link controls: optional expiry, allow-download
toggle (stream-only vs downloadable), optional argon2 password. Revoke = delete.
- migration 0042 + DownloadLink model; app/downloads/links.py (token, HMAC grant sign/verify)
- owner endpoints (POST/GET/PATCH/DELETE links) + /recipients for the internal user picker
- public router (no auth): watch meta / password unlock→signed grant / range-aware file serve
(inline vs attachment); unlock is rate-limited; meta verifies the file exists on disk
Verified end-to-end: 206 range, inline vs attachment, wrong-password 403, tampered-grant 403.
The hover-scrub thumbnail was centered on the cursor with a fixed -translate-x-1/2, so near the
filmstrip's right (or left) edge it overflowed the modal and triggered a horizontal scrollbar. Now
its left edge is clamped to [0, trackW - popoverW] so it shifts inward at the edges and stays fully
within the viewport. Verified at both edges in a real browser.
Rework VideoEditor around a segment cut-list: N draggable cut markers → segments, per-segment
keep/drop (eye toggle + dropped hatch), output = Separate files (one job per kept segment) or
Join into one (segments cut-list → one concatenated file). Per-segment numeric Start/End inputs.
Filmstrip fixed: aspect-correct tiles (no vertical squish) + YouTube-style hover-scrub thumbnail
from the sprite. EditSpec.segments type; editor v2 i18n (en/hu/de). Verified end-to-end in a real
browser (3-segment split, drop middle, join → 0:12 clip from a 0:19 source).
Editor v2 backend: an edit_spec can carry a 'segments' cut-list that concatenates the kept
ranges into ONE file (accurate=filter_complex trim+concat re-encode; fast=concat-demuxer
stream-copy with per-segment inpoint/outpoint). normalize/edit_sig/clip_duration/needs_reencode
handle segments; worker writes the concat list + runs build_concat_plan. Single-trim path (used by
'separate files' export) unchanged.
VideoEditor modal on a finished Library download: HTML5 <video> scrubber, filmstrip timeline
(lazy server storyboard sprite) with draggable in/out handles, draggable/resizable crop overlay,
per-edit Precise (re-encode) vs Fast (stream-copy) cut toggle, split-into-N fan-out (N trim jobs),
optional clip name. Edited clips show a 'Clip' badge; 'editing' phase gets a % bar. New api
enqueueEdit/downloadStoryboard/storyboardImageUrl + EditSpec type; editor i18n (en/hu/de).
YouTube returns flaky per-client responses: the web clients intermittently mislabel normal
videos as 'DRM protected' (like tv did), and android_vr can get bot-flagged under heavy load.
Neither client alone is reliable. So:
- PRIMARY player client = android_vr: high-quality DASH (399+251), no PO token / Deno needed,
reliable for normal use
- on a bot/DRM/format failure the worker retries with the FALLBACK set (web_safari,web,android
+ POT sidecar + Deno) — bypasses bot-detection when android_vr is flagged
- both client sets admin-tunable (DOWNLOAD_PLAYER_CLIENTS[_FALLBACK])
Verified: the previously DRM-failing video now downloads steadily via android_vr, no DRM error.
The web player clients need two things beyond the PO token: (1) a JavaScript runtime for
YouTube's 'n' signature challenge — install Deno (v2.9.1) + yt-dlp[default] (bundles yt-dlp-ejs
solver scripts); auto-detected on PATH. (2) an audio source — the web clients only expose
video-only DASH, so add the 'android' client (also PO-token-backed) for audio/muxed formats.
Player clients: web_safari,web,android (dropped tv — it falsely reports normal videos as DRM).
End-to-end verified in the worker: extraction (POT + Deno n-challenge) + download completes,
no bot/DRM/format errors. Full YouTube stack now: force-ipv4 + bgutil POT sidecar + Deno.
YouTube increasingly demands a Proof-of-Origin token (or shows 'confirm you're not a bot').
Instead of manual cookies, run the bgutil POT provider as a sidecar that mints tokens on demand:
- compose: bgutil-pot service (brainicism/bgutil-ytdlp-pot-provider:1.3.1, port 4416, init)
- requirements: bgutil-ytdlp-pot-provider==1.3.1 plugin (version pinned to the image)
- config: DOWNLOAD_POT_BASE_URL (http://bgutil-pot:4416) + DOWNLOAD_PLAYER_CLIENTS
(web_safari,web,tv — POT-capable; the default android_vr client never requests a token)
- formats.build_ydl_opts wires extractor_args youtube:player_client + youtubepot-bgutilhttp:base_url
Verified: the sidecar mints a valid PO token (BotGuard challenge solved) and yt-dlp fetches it
from the provider. NOTE: end-to-end download couldn't be confirmed here because the test IP got
hard-flagged by YouTube from heavy testing (~16 GB + dozens of extractions) — a hard abuse block
needs cooldown/cookies, which POT doesn't lift; POT bypasses the normal soft bot-check.
Embedding (thumbnail/chapters) rewrote the whole file — a 10 GB video spent minutes making a
10 GB temp copy — while the Plex .nfo + poster sidecars already carry that metadata. So:
- builtin presets no longer embed (migration 0040 re-seeds them; formats._DEFAULTS embed off);
only the tiny FFmpegThumbnailsConvertor (for poster.jpg) remains, so downloads go
video -> audio -> merge -> done with no full-file rewrite
- default preset is now 1080p (Best stays available, second in the list) so a long video
doesn't silently pull ~10 GB and blow the quota
Users who want self-contained files can enable embedding in a custom profile.
ffmpeg post-steps have no byte-progress, so instead of a silent 'Processing' the row now names
the current step (Merging / Extracting audio / Embedding thumbnail / Removing sponsors / Writing
metadata) with an indeterminate pulse. Byte-progress phases (video/audio) keep the % bar.
i18n en/hu/de.
Root cause of the failing downloads: containers usually have no working IPv6 route, but
googlevideo CDN hosts advertise AAAA records, so the downloader tried IPv6 and failed with
'[Errno -5] No address associated with hostname'. Small clips happened to use IPv4; long videos
(more CDN requests / the ffmpeg path) hit IPv6 and died.
- yt-dlp source_address=0.0.0.0 forces IPv4 for every connection (== --force-ipv4). Verified: the
full video+audio download completes cleanly.
- Also: retry (resume) now resets the shared asset from 'error' to 'pending', so it actually
re-downloads — previously the requeued job instantly re-inherited the asset's stale error
because the worker short-circuits a job whose asset already errored.
- Dropped the localdev DNS override (wrong hypothesis; the issue was IPv6, not the resolver).
- Kept yt-dlp retries/fragment_retries/socket_timeout for transient blips.
Verified end-to-end in the worker: two previously-failing long videos now download to done.
_insert_stubs previously stored the raw title, so a brand-new video showed its unnormalized
title until enrichment caught up. Normalize (and stash original_title) at insert as well, so
every video-title write path is covered: stub insert, enrichment, live search, and the download
worker. Every newly imported feed video is normalized from the first insert.
Noisy YouTube titles are cleaned for display + storage, raw kept in videos.original_title:
- app/titles.normalize_title: strip emoji/symbols (keep accents), remove trailing SEO hashtag
clusters (keep numeric #3 episode markers), context-aware de-shout (mostly-ALL-CAPS titles ->
Title Case with an acronym whitelist + function-word lowercasing; otherwise only long all-caps
words), collapse repeated punctuation
- applied at enrichment (sync/videos.py) and in the download worker (ad-hoc yt-dlp titles);
catalog downloads inherit the normalized title automatically
- migration 0039: add original_title, preserve raw, rewrite title (generated search_vector
regenerates); reversible via original_title
Backfill on localdev: 122115/273417 titles normalized in ~2 min. Verified in the feed + on
real messy samples (emoji/de-shout/hashtags), accents + acronyms (PS5/AI/USA/PC) preserved.
The progress bar looked broken (95% -> 5% -> recount) because yt-dlp downloads the video and
audio streams separately (each 0->100%) then merges — with no indication of which step is
running. Now:
- worker labels the phase from the stream codecs (video / audio) and via a postprocessor hook
(merging / processing), so the user sees what's happening
- DownloadCenter shows the phase as the status and renders an indeterminate pulse (no bogus %)
during merge/processing; phase i18n en/hu/de
- yt-dlp retries (3) + fragment_retries (5) so transient network blips self-heal instead of
failing the job (seen once as a DNS 'No address associated with hostname' error)
Verified: phase transitions queued -> video -> processing -> done.
A queued/downloading row previously showed the bare video id and no thumbnail because asset
metadata was only filled when the download completed. Now:
- service.populate_from_catalog fills title/uploader/thumbnail/date/duration at enqueue from
our own Video/Channel catalog (0 network cost) — feed downloads look right instantly
- worker fills the same from yt-dlp's info_dict on the first progress event (covers ad-hoc
URLs / catalog misses), best-effort, never fails a download
Verified: a paused feed download now shows its real title, channel, thumbnail and duration.
Two UAT findings:
1. The device download filename (Content-Disposition) kept emoji/symbols from the video title.
Add storage.display_filename (drops emoji/symbol/control unicode, keeps spaces + accents)
and use it for the download name — readable and clean ("…alapján!.mp4", no emoji).
2. Deleting/canceling a download removed the job but the shared MediaAsset (and its file) lingered
as cache, so 'Ready files' stayed inflated and disk wasn't freed. Rework: _release_asset drops
the hold and, once no job holds the asset, deletes the file + row immediately (the cache only
needs to span overlapping holders). Also fixes cancel never decrementing (it flipped status to
'canceled' before releasing, tripping the holding-state guard).
Verified: filename emoji-stripped; enqueue→delete removes the asset row + file from disk.
A plain <a> file download can't send the X-Siftlode-Account header, so current_user resolved
it to the session-default account — 404 'Unknown download' when the tab acts as a non-default
wallet account that owns the file. resolved_user_id now also honours a ?account= query param
(the same wallet-gated selection the WebSocket already uses), and downloadFileUrl appends the
active account id. Verified: default account -> 404, ?account=<owner> -> 206 with the right
Content-Disposition.
The .gitignore 'downloads/' pattern for the DOWNLOAD_ROOT bind mount also matched the
backend/app/downloads source package, so formats.py/storage.py/service.py never got
committed with M2. Anchor the ignore to '/downloads/' (repo root only) and add the package.
storage.sanitize() now handles arbitrarily messy titles (emoji, ZWJ, fullwidth, clickbait
punctuation): NFKC-normalize, drop emoji/symbol/control unicode categories, collapse
repeated punctuation, underscore-join words -> space-free paths. Accents (HU/DE) preserved,
no ASCII folding. Plex layout uses _-_ separators + Season_{year}.
