chore: prepare repo for public release — scrub internal infra, rewrite README
- Remove owner-specific / legacy deploy files (home/prod/server compose, deploy/). The home compose stays as a local untracked file for the maintainer's own deploy. - Genericise infra-specific code comments (egress-proxy examples) to neutral wording. - Replace the hardcoded contact email on the legal pages with the instance operator's configured admin email, served via the public /auth/config and shown with a neutral fallback — so each self-hosted instance shows its own contact. - Rewrite README for the current app + a copy-paste self-hosting quick start (prebuilt image + first-run wizard) with a build-from-source alternative; tidy .env.example.
This commit is contained in:
npeter83
parent
381794d9ae
commit
0d44d3a34a
10 changed files with 149 additions and 119 deletions
165
README.md
165
README.md
|
|
@ -1,106 +1,109 @@
|
|||
# Siftlode
|
||||
|
||||
Self-hosted, multi-user web app for browsing **your own YouTube subscriptions** the way you
|
||||
actually want: precise filtering and sorting (by language, topic, length, age, watch state…),
|
||||
a fast local-first feed, and one-click playback that opens the real youtube.com — so your
|
||||
browser's ad blocking and SponsorBlock keep working exactly as before.
|
||||
**Your YouTube subscriptions, the way a feed should work.** Siftlode pulls every upload from the
|
||||
channels you follow into one clean, filterable feed — no algorithm deciding what you see, and no
|
||||
Shorts or livestream noise unless you want it. Self-hosted, multi-user, and private: your data
|
||||
stays on your own server.
|
||||
|
||||
Each user signs in with their own Google account (invite-only) and sees only their own
|
||||
subscriptions. All the expensive data (channels, videos, metadata) is fetched from YouTube
|
||||
once and stored locally, so filtering/searching/sorting are instant and don't burn API quota.
|
||||
|
||||
|
||||
> Status: early development.
|
||||
> - **M1** (foundation): docker-compose stack, FastAPI backend, Google OAuth login with an
|
||||
> email invite-list, encrypted token storage.
|
||||
> - **M2** (ingest core): subscription import, free RSS detection, recent-first + deep backfill
|
||||
> from the uploads playlist, enrichment (duration/stats/category, Shorts & livestream
|
||||
> classification), a shared daily quota guard, and a background scheduler.
|
||||
> - **M3** (auto-tagging): system tags for channel language (offline detection) and topic
|
||||
> (from YouTube topics + dominant category), regenerated automatically; user tags are
|
||||
> never overwritten.
|
||||
> - **M4** (reader UI): React + Vite SPA with four color schemes (dark/light) and adjustable
|
||||
> text size; grid/list feed scoped to your subscriptions, with faceted tag filters,
|
||||
> content-type toggles (Normal/Shorts/Live), date range, search, sort, watch/save/hide
|
||||
> state and per-channel filtering; clicking a video opens youtube.com. Accurate Shorts
|
||||
> detection via the /shorts probe, a sync-status indicator with admin pause/resume, a
|
||||
> filtered video count, and structured timestamped logging.
|
||||
Everything expensive (channels, videos, metadata) is fetched from YouTube once and stored locally,
|
||||
so filtering, searching and sorting are instant and don't burn API quota. Click a video to watch it
|
||||
in an in-app player that resumes where you left off — or open it on youtube.com so your own ad
|
||||
blocker and SponsorBlock keep working.
|
||||
|
||||
## Requirements
|
||||
## Features
|
||||
|
||||
- Docker + Docker Compose
|
||||
- A Google Cloud project with an OAuth client (see below)
|
||||
- **A readable subscription feed** — sort and filter by channel, tag, language, topic, length,
|
||||
upload date or watch state; hide channels without unsubscribing; save filter setups as named views.
|
||||
- **Search all of YouTube** from the feed — results play, save and add to playlists like any other
|
||||
video, and are materialised into your catalog.
|
||||
- **Channel pages & a channel manager** — per-channel stats and uploads, priorities, and your own
|
||||
tags to slice the feed by.
|
||||
- **Playlists with two-way YouTube sync** — build them locally, keep them in sync in both directions.
|
||||
- **In-app player** with resume, plus keyboard/scroll controls.
|
||||
- **Multi-user** with per-user private state, a shared catalog, and a fair daily API-quota guard.
|
||||
- **Self-hosted & private**, with a first-run web setup wizard and the interface in **English,
|
||||
Hungarian and German**.
|
||||
|
||||
## Quick start
|
||||
## Quick start (self-hosting)
|
||||
|
||||
1. Copy the env template and generate secrets:
|
||||
You don't need to build anything — Siftlode runs from a prebuilt public image, and all
|
||||
configuration (your admin account, Google sign-in, email) happens in a **first-run web wizard**.
|
||||
You need [Docker](https://docs.docker.com/get-docker/) with the Compose plugin.
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
python -c "import secrets;print('SECRET_KEY=',secrets.token_urlsafe(48))"
|
||||
python -c "import base64,os;print('TOKEN_ENCRYPTION_KEY=',base64.urlsafe_b64encode(os.urandom(32)).decode())"
|
||||
```
|
||||
|
||||
Paste the generated values into `.env`.
|
||||
|
||||
2. Create a Google OAuth client (Google Cloud Console → APIs & Services):
|
||||
- Enable the **YouTube Data API v3**.
|
||||
- OAuth consent screen: **External**, publishing status **Testing**, and add every invited
|
||||
Google account as a **Test user** (up to 100 — no app verification needed at this scale).
|
||||
- Create credentials → **OAuth client ID** → type **Web application**.
|
||||
- Authorized redirect URI: `http://localhost:8080/auth/callback`
|
||||
(must match `OAUTH_REDIRECT_URL` in `.env`).
|
||||
- Put the client ID/secret into `.env`, and list invited emails in `ALLOWED_EMAILS`.
|
||||
|
||||
3. Start it:
|
||||
|
||||
```sh
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
Open http://localhost:8080 and sign in. Database migrations run automatically on startup.
|
||||
|
||||
## Backup & moving to another machine
|
||||
|
||||
All data lives in the Postgres database — moving to another host (e.g. a Proxmox Linux
|
||||
server) does **not** require re-fetching from YouTube. Copy your `.env` (keep the same
|
||||
`TOKEN_ENCRYPTION_KEY` and Google client so stored tokens stay valid), then:
|
||||
**1. Get the files and run the installer:**
|
||||
|
||||
```sh
|
||||
./scripts/backup.sh # -> backups/siftlode-<timestamp>.dump (Windows: scripts\backup.ps1)
|
||||
./scripts/restore.sh backups/<file> # on the new host after `docker compose up`
|
||||
git clone https://forge.b1fr0st.eu/peter/siftlode.git
|
||||
cd siftlode
|
||||
./install.sh # Windows (PowerShell): ./install.ps1
|
||||
```
|
||||
|
||||
## Central server + local dev (single shared database)
|
||||
The installer generates a private `.env` (secrets), pulls the image, starts the app + database, and
|
||||
prints a one-time setup URL like `http://localhost:8080/setup?token=…`.
|
||||
|
||||
For always-on operation the recommended topology is a single **central Postgres** running on a
|
||||
24/7 host (e.g. a Proxmox LXC) that also runs the background scheduler, with every other instance
|
||||
(your local dev machine, a future VPS) pointing at that same database. One source of truth, no
|
||||
sync.
|
||||
**2. Finish in your browser.** Open that URL and follow the wizard:
|
||||
|
||||
- **Server** (`docker-compose.server.yml`): full stack, Postgres published on the LAN, scheduler
|
||||
on, DB files in a host-visible `./pgdata` bind mount. In the LXC, from the project root:
|
||||
1. **Admin account** — the email + password you'll sign in with.
|
||||
2. **Google sign-in** *(optional)* — paste a Google OAuth client to enable "Sign in with Google" and
|
||||
pulling your YouTube subscriptions. Skip it to use email + password only.
|
||||
3. **Email / SMTP** *(optional)* — for verification/notification emails. Skip it and you (the admin)
|
||||
simply approve new accounts yourself.
|
||||
|
||||
```sh
|
||||
echo 'COMPOSE_FILE=docker-compose.server.yml' >> .env # so backup/restore target this stack
|
||||
docker compose up -d --build
|
||||
./scripts/restore.sh backups/<file> # migrate existing data in
|
||||
```
|
||||
Then sign in with your admin account. That's it. See **[docs/self-hosting.md](docs/self-hosting.md)**
|
||||
for the full walkthrough.
|
||||
|
||||
Set `YOUTUBE_API_KEY` in `.env` so unattended backfill/enrichment use the API key and don't
|
||||
depend on a refresh token (which expires after 7 days while the OAuth screen is in *Testing*).
|
||||
> Just trying it out? Press Enter at the installer's URL prompt to run on `http://localhost:8080`.
|
||||
|
||||
- **Local dev** (`docker-compose.localdev.yml`): webapp only, no local DB, no scheduler. In `.env`
|
||||
set `DATABASE_URL` to the central DB (e.g. `…@your-db-host:5432/siftlode`), then
|
||||
`docker compose -f docker-compose.localdev.yml up --build` and browse http://localhost:8080.
|
||||
## Build from source (alternative)
|
||||
|
||||
**Exactly one instance may run the scheduler.** The server keeps `SCHEDULER_ENABLED=true`; every
|
||||
other instance must be `false` (the compose files enforce this) to avoid double quota burn and
|
||||
write races on the shared DB.
|
||||
Prefer to build the image yourself instead of pulling it:
|
||||
|
||||
```sh
|
||||
git clone https://forge.b1fr0st.eu/peter/siftlode.git
|
||||
cd siftlode
|
||||
cp .env.example .env
|
||||
# generate the two secrets and paste them into .env:
|
||||
python -c "import secrets;print('SECRET_KEY='+secrets.token_urlsafe(48))"
|
||||
python -c "import base64,os;print('TOKEN_ENCRYPTION_KEY='+base64.urlsafe_b64encode(os.urandom(32)).decode())"
|
||||
docker compose up --build -d # builds from the included Dockerfile
|
||||
```
|
||||
|
||||
Open `http://localhost:8080` and finish in the setup wizard as above. (Set a `POSTGRES_PASSWORD` in
|
||||
`.env` too.)
|
||||
|
||||
## HTTPS / public access
|
||||
|
||||
Port `8080` over plain HTTP is fine for a LAN or a quick trial. For public access put a reverse
|
||||
proxy (Caddy, Nginx, Traefik…) in front to terminate TLS, and set the **public URL** (the installer
|
||||
prompt, or `OAUTH_REDIRECT_URL` in `.env`) to your `https://…` address — this also marks the session
|
||||
cookie secure. Add that same `…/auth/callback` URL to your Google OAuth client's authorized redirect
|
||||
URIs.
|
||||
|
||||
## Updating & backups
|
||||
|
||||
```sh
|
||||
docker compose -f docker-compose.selfhost.yml pull # or: docker compose pull
|
||||
docker compose -f docker-compose.selfhost.yml up -d
|
||||
```
|
||||
|
||||
Database migrations run automatically on startup. Your data (accounts, subscriptions, playlists, the
|
||||
video catalog) lives in a Postgres volume — back it up with `scripts/backup.sh` (or `backup.ps1` on
|
||||
Windows) and restore with `scripts/restore.sh`.
|
||||
|
||||
## How it works
|
||||
|
||||
- **Shared catalog, private state.** Channels and videos are stored once and shared; each user's
|
||||
subscriptions, tags, playlists and watch/save/hide state are private.
|
||||
- **Cheap by design.** Public reads are cached locally; a shared daily quota budget and a background
|
||||
scheduler keep unattended syncing within YouTube's free API limits. An optional API key lets
|
||||
backfill run without depending on a user's OAuth token.
|
||||
|
||||
## Tech
|
||||
|
||||
FastAPI + PostgreSQL backend, React + Vite frontend (added in a later milestone), packaged with
|
||||
Docker Compose.
|
||||
FastAPI + PostgreSQL (SQLAlchemy, Alembic) backend; React + Vite + Tailwind + TanStack Query
|
||||
frontend; packaged as a single Docker image with Docker Compose.
|
||||
|
||||
## Note
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue