merge: shared-database deployment (24/7 backfill)

shared database + scheduler in the server; local dev points at it with the
scheduler off. Adds docker-compose.server.yml / docker-compose.localdev.yml, the
shared-database docs, and the nested Docker apparmor fix.
This commit is contained in:
npeter83 2026-06-11 16:59:09 +02:00
commit 6120e6b0f8
3 changed files with 81 additions and 1 deletions

View file

@ -28,3 +28,23 @@ ADMIN_EMAILS=
# Optional: origin of a separately-served frontend dev server (enables CORS). Leave empty in production. # Optional: origin of a separately-served frontend dev server (enables CORS). Leave empty in production.
FRONTEND_ORIGIN= FRONTEND_ORIGIN=
# Optional YouTube Data API key (Google Cloud Console -> Credentials -> Create API key).
# When set, all public reads (channels/videos/playlist backfill + enrichment) use the key
# instead of a user's OAuth token, so 24/7 backfill never depends on a refresh token that
# would otherwise expire (Google expires refresh tokens after 7 days while the OAuth consent
# screen is in "Testing"). Strongly recommended for the always-on server instance.
YOUTUBE_API_KEY=
# --- Deployment role ---
# DATABASE_URL is set automatically by docker-compose.yml / docker-compose.server.yml to the
# bundled `db` service. Override it only for local dev against the central server DB, e.g.:
# DATABASE_URL=postgresql+psycopg://subfeed:<password>@your-db-host:5432/subfeed
# Exactly one instance may run the background scheduler. The central server keeps it on; every
# other instance (local dev, etc.) must set SCHEDULER_ENABLED=false. The compose files set this
# for you (server=true via docker-compose.server.yml, local=false via docker-compose.localdev.yml).
SCHEDULER_ENABLED=true
# On the central server, set this so backup.sh / restore.sh and plain `docker compose` commands
# target the server stack automatically:
# COMPOSE_FILE=docker-compose.server.yml

View file

@ -61,7 +61,7 @@ once and stored locally, so filtering/searching/sorting are instant and don't bu
## Backup & moving to another machine ## Backup & moving to another machine
All data lives in the `pgdata` Postgres volume — moving to another host (e.g. a Proxmox Linux All data lives in the Postgres database — moving to another host (e.g. a Proxmox Linux
server) does **not** require re-fetching from YouTube. Copy your `.env` (keep the same server) does **not** require re-fetching from YouTube. Copy your `.env` (keep the same
`TOKEN_ENCRYPTION_KEY` and Google client so stored tokens stay valid), then: `TOKEN_ENCRYPTION_KEY` and Google client so stored tokens stay valid), then:
@ -70,6 +70,33 @@ server) does **not** require re-fetching from YouTube. Copy your `.env` (keep th
./scripts/restore.sh backups/<file> # on the new host after `docker compose up` ./scripts/restore.sh backups/<file> # on the new host after `docker compose up`
``` ```
## Central server + local dev (single shared database)
For always-on operation the recommended topology is a single **central Postgres** running on a
24/7 host (e.g. a Proxmox LXC) that also runs the background scheduler, with every other instance
(your local dev machine, a future VPS) pointing at that same database. One source of truth, no
sync.
- **Server** (`docker-compose.server.yml`): full stack, Postgres published on the LAN, scheduler
on, DB files in a host-visible `./pgdata` bind mount. In the LXC, from the project root:
```sh
echo 'COMPOSE_FILE=docker-compose.server.yml' >> .env # so backup/restore target this stack
docker compose up -d --build
./scripts/restore.sh backups/<file> # migrate existing data in
```
Set `YOUTUBE_API_KEY` in `.env` so unattended backfill/enrichment use the API key and don't
depend on a refresh token (which expires after 7 days while the OAuth screen is in *Testing*).
- **Local dev** (`docker-compose.localdev.yml`): webapp only, no local DB, no scheduler. In `.env`
set `DATABASE_URL` to the central DB (e.g. `…@your-db-host:5432/subfeed`), then
`docker compose -f docker-compose.localdev.yml up --build` and browse http://localhost:8080.
**Exactly one instance may run the scheduler.** The server keeps `SCHEDULER_ENABLED=true`; every
other instance must be `false` (the compose files enforce this) to avoid double quota burn and
write races on the shared DB.
## Tech ## Tech
FastAPI + PostgreSQL backend, React + Vite frontend (added in a later milestone), packaged with FastAPI + PostgreSQL backend, React + Vite frontend (added in a later milestone), packaged with

View file

@ -0,0 +1,33 @@
# Local development against the central (Proxmox) Postgres.
#
# Runs only the webapp/API — no local database, no scheduler — pointed at the shared
# DB on the server. You see the same live data the 24/7 backfill is filling, and you
# never run a second scheduler against it.
#
# Setup: in .env set
# DATABASE_URL=postgresql+psycopg://subfeed:<password>@your-db-host:5432/subfeed
# (use the central DB's host/port and the real POSTGRES_PASSWORD), then:
# docker compose -f docker-compose.localdev.yml up --build
#
# Browse at http://localhost:8080 — Google login works here because the OAuth redirect
# is http://localhost:8080/auth/callback.
#
# Note: the API runs `alembic upgrade head` on startup, so launching this with newer
# local migrations will apply them to the shared DB. For risky schema work, point
# DATABASE_URL at a throwaway local Postgres instead.
services:
api:
build:
context: .
dockerfile: Dockerfile
env_file: .env
environment:
# Exactly one scheduler may write to the shared DB; the server owns it.
SCHEDULER_ENABLED: "false"
# Harmless on Docker Desktop; needed if ever run under Docker-in-LXC.
security_opt:
- apparmor:unconfined
ports:
- "${APP_PORT:-8080}:8000"
restart: unless-stopped