merge: shared-database deployment (24/7 backfill)
shared database + scheduler in the server; local dev points at it with the scheduler off. Adds docker-compose.server.yml / docker-compose.localdev.yml, the shared-database docs, and the nested Docker apparmor fix.
This commit is contained in:
commit
6120e6b0f8
3 changed files with 81 additions and 1 deletions
20
.env.example
20
.env.example
|
|
@ -28,3 +28,23 @@ ADMIN_EMAILS=
|
||||||
|
|
||||||
# Optional: origin of a separately-served frontend dev server (enables CORS). Leave empty in production.
|
# Optional: origin of a separately-served frontend dev server (enables CORS). Leave empty in production.
|
||||||
FRONTEND_ORIGIN=
|
FRONTEND_ORIGIN=
|
||||||
|
|
||||||
|
# Optional YouTube Data API key (Google Cloud Console -> Credentials -> Create API key).
|
||||||
|
# When set, all public reads (channels/videos/playlist backfill + enrichment) use the key
|
||||||
|
# instead of a user's OAuth token, so 24/7 backfill never depends on a refresh token that
|
||||||
|
# would otherwise expire (Google expires refresh tokens after 7 days while the OAuth consent
|
||||||
|
# screen is in "Testing"). Strongly recommended for the always-on server instance.
|
||||||
|
YOUTUBE_API_KEY=
|
||||||
|
|
||||||
|
# --- Deployment role ---
|
||||||
|
# DATABASE_URL is set automatically by docker-compose.yml / docker-compose.server.yml to the
|
||||||
|
# bundled `db` service. Override it only for local dev against the central server DB, e.g.:
|
||||||
|
# DATABASE_URL=postgresql+psycopg://subfeed:<password>@your-db-host:5432/subfeed
|
||||||
|
# Exactly one instance may run the background scheduler. The central server keeps it on; every
|
||||||
|
# other instance (local dev, etc.) must set SCHEDULER_ENABLED=false. The compose files set this
|
||||||
|
# for you (server=true via docker-compose.server.yml, local=false via docker-compose.localdev.yml).
|
||||||
|
SCHEDULER_ENABLED=true
|
||||||
|
|
||||||
|
# On the central server, set this so backup.sh / restore.sh and plain `docker compose` commands
|
||||||
|
# target the server stack automatically:
|
||||||
|
# COMPOSE_FILE=docker-compose.server.yml
|
||||||
|
|
|
||||||
29
README.md
29
README.md
|
|
@ -61,7 +61,7 @@ once and stored locally, so filtering/searching/sorting are instant and don't bu
|
||||||
|
|
||||||
## Backup & moving to another machine
|
## Backup & moving to another machine
|
||||||
|
|
||||||
All data lives in the `pgdata` Postgres volume — moving to another host (e.g. a Proxmox Linux
|
All data lives in the Postgres database — moving to another host (e.g. a Proxmox Linux
|
||||||
server) does **not** require re-fetching from YouTube. Copy your `.env` (keep the same
|
server) does **not** require re-fetching from YouTube. Copy your `.env` (keep the same
|
||||||
`TOKEN_ENCRYPTION_KEY` and Google client so stored tokens stay valid), then:
|
`TOKEN_ENCRYPTION_KEY` and Google client so stored tokens stay valid), then:
|
||||||
|
|
||||||
|
|
@ -70,6 +70,33 @@ server) does **not** require re-fetching from YouTube. Copy your `.env` (keep th
|
||||||
./scripts/restore.sh backups/<file> # on the new host after `docker compose up`
|
./scripts/restore.sh backups/<file> # on the new host after `docker compose up`
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Central server + local dev (single shared database)
|
||||||
|
|
||||||
|
For always-on operation the recommended topology is a single **central Postgres** running on a
|
||||||
|
24/7 host (e.g. a Proxmox LXC) that also runs the background scheduler, with every other instance
|
||||||
|
(your local dev machine, a future VPS) pointing at that same database. One source of truth, no
|
||||||
|
sync.
|
||||||
|
|
||||||
|
- **Server** (`docker-compose.server.yml`): full stack, Postgres published on the LAN, scheduler
|
||||||
|
on, DB files in a host-visible `./pgdata` bind mount. In the LXC, from the project root:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
echo 'COMPOSE_FILE=docker-compose.server.yml' >> .env # so backup/restore target this stack
|
||||||
|
docker compose up -d --build
|
||||||
|
./scripts/restore.sh backups/<file> # migrate existing data in
|
||||||
|
```
|
||||||
|
|
||||||
|
Set `YOUTUBE_API_KEY` in `.env` so unattended backfill/enrichment use the API key and don't
|
||||||
|
depend on a refresh token (which expires after 7 days while the OAuth screen is in *Testing*).
|
||||||
|
|
||||||
|
- **Local dev** (`docker-compose.localdev.yml`): webapp only, no local DB, no scheduler. In `.env`
|
||||||
|
set `DATABASE_URL` to the central DB (e.g. `…@your-db-host:5432/subfeed`), then
|
||||||
|
`docker compose -f docker-compose.localdev.yml up --build` and browse http://localhost:8080.
|
||||||
|
|
||||||
|
**Exactly one instance may run the scheduler.** The server keeps `SCHEDULER_ENABLED=true`; every
|
||||||
|
other instance must be `false` (the compose files enforce this) to avoid double quota burn and
|
||||||
|
write races on the shared DB.
|
||||||
|
|
||||||
## Tech
|
## Tech
|
||||||
|
|
||||||
FastAPI + PostgreSQL backend, React + Vite frontend (added in a later milestone), packaged with
|
FastAPI + PostgreSQL backend, React + Vite frontend (added in a later milestone), packaged with
|
||||||
|
|
|
||||||
33
docker-compose.localdev.yml
Normal file
33
docker-compose.localdev.yml
Normal file
|
|
@ -0,0 +1,33 @@
|
||||||
|
# Local development against the central (Proxmox) Postgres.
|
||||||
|
#
|
||||||
|
# Runs only the webapp/API — no local database, no scheduler — pointed at the shared
|
||||||
|
# DB on the server. You see the same live data the 24/7 backfill is filling, and you
|
||||||
|
# never run a second scheduler against it.
|
||||||
|
#
|
||||||
|
# Setup: in .env set
|
||||||
|
# DATABASE_URL=postgresql+psycopg://subfeed:<password>@your-db-host:5432/subfeed
|
||||||
|
# (use the central DB's host/port and the real POSTGRES_PASSWORD), then:
|
||||||
|
# docker compose -f docker-compose.localdev.yml up --build
|
||||||
|
#
|
||||||
|
# Browse at http://localhost:8080 — Google login works here because the OAuth redirect
|
||||||
|
# is http://localhost:8080/auth/callback.
|
||||||
|
#
|
||||||
|
# Note: the API runs `alembic upgrade head` on startup, so launching this with newer
|
||||||
|
# local migrations will apply them to the shared DB. For risky schema work, point
|
||||||
|
# DATABASE_URL at a throwaway local Postgres instead.
|
||||||
|
|
||||||
|
services:
|
||||||
|
api:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
env_file: .env
|
||||||
|
environment:
|
||||||
|
# Exactly one scheduler may write to the shared DB; the server owns it.
|
||||||
|
SCHEDULER_ENABLED: "false"
|
||||||
|
# Harmless on Docker Desktop; needed if ever run under Docker-in-LXC.
|
||||||
|
security_opt:
|
||||||
|
- apparmor:unconfined
|
||||||
|
ports:
|
||||||
|
- "${APP_PORT:-8080}:8000"
|
||||||
|
restart: unless-stopped
|
||||||
Loading…
Add table
Add a link
Reference in a new issue