Siftlode

Self-hosted, multi-user web app for browsing your own YouTube subscriptions the way you actually want: precise filtering and sorting (by language, topic, length, age, watch state…), a fast local-first feed, and one-click playback that opens the real youtube.com — so your browser's ad blocking and SponsorBlock keep working exactly as before.

Each user signs in with their own Google account (invite-only) and sees only their own subscriptions. All the expensive data (channels, videos, metadata) is fetched from YouTube once and stored locally, so filtering/searching/sorting are instant and don't burn API quota.

Status: early development.

• M1 (foundation): docker-compose stack, FastAPI backend, Google OAuth login with an email invite-list, encrypted token storage.
• M2 (ingest core): subscription import, free RSS detection, recent-first + deep backfill from the uploads playlist, enrichment (duration/stats/category, Shorts & livestream classification), a shared daily quota guard, and a background scheduler.
• M3 (auto-tagging): system tags for channel language (offline detection) and topic (from YouTube topics + dominant category), regenerated automatically; user tags are never overwritten.
• M4 (reader UI): React + Vite SPA with four color schemes (dark/light) and adjustable text size; grid/list feed scoped to your subscriptions, with faceted tag filters, content-type toggles (Normal/Shorts/Live), date range, search, sort, watch/save/hide state and per-channel filtering; clicking a video opens youtube.com. Accurate Shorts detection via the /shorts probe, a sync-status indicator with admin pause/resume, a filtered video count, and structured timestamped logging.

Requirements

• Docker + Docker Compose
• A Google Cloud project with an OAuth client (see below)

Quick start

1. Copy the env template and generate secrets:

   cp .env.example .env
   python -c "import secrets;print('SECRET_KEY=',secrets.token_urlsafe(48))"
   python -c "import base64,os;print('TOKEN_ENCRYPTION_KEY=',base64.urlsafe_b64encode(os.urandom(32)).decode())"
   

   Paste the generated values into .env.

2. Create a Google OAuth client (Google Cloud Console → APIs & Services):

   • Enable the YouTube Data API v3.
   • OAuth consent screen: External, publishing status Testing, and add every invited Google account as a Test user (up to 100 — no app verification needed at this scale).
   • Create credentials → OAuth client ID → type Web application.
   • Authorized redirect URI: http://localhost:8080/auth/callback (must match OAUTH_REDIRECT_URL in .env).
   • Put the client ID/secret into .env, and list invited emails in ALLOWED_EMAILS.

3. Start it:

   docker compose up --build
   

   Open http://localhost:8080 and sign in. Database migrations run automatically on startup.

Backup & moving to another machine

All data lives in the Postgres database — moving to another host (e.g. a Proxmox Linux server) does not require re-fetching from YouTube. Copy your .env (keep the same TOKEN_ENCRYPTION_KEY and Google client so stored tokens stay valid), then:

./scripts/backup.sh                 # -> backups/siftlode-<timestamp>.dump   (Windows: scripts\backup.ps1)
./scripts/restore.sh backups/<file> # on the new host after `docker compose up`

Central server + local dev (single shared database)

For always-on operation the recommended topology is a single central Postgres running on a 24/7 host (e.g. a Proxmox LXC) that also runs the background scheduler, with every other instance (your local dev machine, a future VPS) pointing at that same database. One source of truth, no sync.

• Server (docker-compose.server.yml): full stack, Postgres published on the LAN, scheduler on, DB files in a host-visible ./pgdata bind mount. In the LXC, from the project root:

  echo 'COMPOSE_FILE=docker-compose.server.yml' >> .env   # so backup/restore target this stack
  docker compose up -d --build
  ./scripts/restore.sh backups/<file>                     # migrate existing data in
  

  Set YOUTUBE_API_KEY in .env so unattended backfill/enrichment use the API key and don't depend on a refresh token (which expires after 7 days while the OAuth screen is in Testing).

• Local dev (docker-compose.localdev.yml): webapp only, no local DB, no scheduler. In .env set DATABASE_URL to the central DB (e.g. …@your-db-host:5432/subfeed), then docker compose -f docker-compose.localdev.yml up --build and browse http://localhost:8080.

Exactly one instance may run the scheduler. The server keeps SCHEDULER_ENABLED=true; every other instance must be false (the compose files enforce this) to avoid double quota burn and write races on the shared DB.

Tech

FastAPI + PostgreSQL backend, React + Vite frontend (added in a later milestone), packaged with Docker Compose.

Note

This project is developed with AI assistance.