siftlode/README.md

# Siftlode

Self-hosted, multi-user web app for browsing **your own YouTube subscriptions** the way you
actually want: precise filtering and sorting (by language, topic, length, age, watch state…),
a fast local-first feed, and one-click playback that opens the real youtube.com — so your
browser's ad blocking and SponsorBlock keep working exactly as before.

Each user signs in with their own Google account (invite-only) and sees only their own
subscriptions. All the expensive data (channels, videos, metadata) is fetched from YouTube
once and stored locally, so filtering/searching/sorting are instant and don't burn API quota.

> Status: early development.
> - **M1** (foundation): docker-compose stack, FastAPI backend, Google OAuth login with an
>   email invite-list, encrypted token storage.
> - **M2** (ingest core): subscription import, free RSS detection, recent-first + deep backfill
>   from the uploads playlist, enrichment (duration/stats/category, Shorts & livestream
>   classification), a shared daily quota guard, and a background scheduler.
> - **M3** (auto-tagging): system tags for channel language (offline detection) and topic
>   (from YouTube topics + dominant category), regenerated automatically; user tags are
>   never overwritten.
> - **M4** (reader UI): React + Vite SPA with four color schemes (dark/light) and adjustable
>   text size; grid/list feed scoped to your subscriptions, with faceted tag filters,
>   content-type toggles (Normal/Shorts/Live), date range, search, sort, watch/save/hide
>   state and per-channel filtering; clicking a video opens youtube.com. Accurate Shorts
>   detection via the /shorts probe, a sync-status indicator with admin pause/resume, a
>   filtered video count, and structured timestamped logging.

## Requirements

- Docker + Docker Compose
- A Google Cloud project with an OAuth client (see below)

## Quick start

1. Copy the env template and generate secrets:

   ```sh
   cp .env.example .env
   python -c "import secrets;print('SECRET_KEY=',secrets.token_urlsafe(48))"
   python -c "import base64,os;print('TOKEN_ENCRYPTION_KEY=',base64.urlsafe_b64encode(os.urandom(32)).decode())"
   ```

   Paste the generated values into `.env`.

2. Create a Google OAuth client (Google Cloud Console → APIs & Services):
   - Enable the **YouTube Data API v3**.
   - OAuth consent screen: **External**, publishing status **Testing**, and add every invited
     Google account as a **Test user** (up to 100 — no app verification needed at this scale).
   - Create credentials → **OAuth client ID** → type **Web application**.
   - Authorized redirect URI: `http://localhost:8080/auth/callback`
     (must match `OAUTH_REDIRECT_URL` in `.env`).
   - Put the client ID/secret into `.env`, and list invited emails in `ALLOWED_EMAILS`.

3. Start it:

   ```sh
   docker compose up --build
   ```

   Open http://localhost:8080 and sign in. Database migrations run automatically on startup.

## Backup & moving to another machine

All data lives in the Postgres database — moving to another host (e.g. a Proxmox Linux
server) does **not** require re-fetching from YouTube. Copy your `.env` (keep the same
`TOKEN_ENCRYPTION_KEY` and Google client so stored tokens stay valid), then:

```sh
./scripts/backup.sh                 # -> backups/siftlode-<timestamp>.dump   (Windows: scripts\backup.ps1)
./scripts/restore.sh backups/<file> # on the new host after `docker compose up`
```

## Central server + local dev (single shared database)

For always-on operation the recommended topology is a single **central Postgres** running on a
24/7 host (e.g. a Proxmox LXC) that also runs the background scheduler, with every other instance
(your local dev machine, a future VPS) pointing at that same database. One source of truth, no
sync.

- **Server** (`docker-compose.server.yml`): full stack, Postgres published on the LAN, scheduler
  on, DB files in a host-visible `./pgdata` bind mount. In the LXC, from the project root:

  ```sh
  echo 'COMPOSE_FILE=docker-compose.server.yml' >> .env   # so backup/restore target this stack
  docker compose up -d --build
  ./scripts/restore.sh backups/<file>                     # migrate existing data in
  ```

  Set `YOUTUBE_API_KEY` in `.env` so unattended backfill/enrichment use the API key and don't
  depend on a refresh token (which expires after 7 days while the OAuth screen is in *Testing*).

- **Local dev** (`docker-compose.localdev.yml`): webapp only, no local DB, no scheduler. In `.env`
  set `DATABASE_URL` to the central DB (e.g. `…@your-db-host:5432/subfeed`), then
  `docker compose -f docker-compose.localdev.yml up --build` and browse http://localhost:8080.

**Exactly one instance may run the scheduler.** The server keeps `SCHEDULER_ENABLED=true`; every
other instance must be `false` (the compose files enforce this) to avoid double quota burn and
write races on the shared DB.

## Tech

FastAPI + PostgreSQL backend, React + Vite frontend (added in a later milestone), packaged with
Docker Compose.

## Note

This project is developed with AI assistance.