chore: prepare repo for public release — scrub internal infra, rewrite README

- Remove owner-specific / legacy deploy files (home/prod/server compose, deploy/).
  The home compose stays as a local untracked file for the maintainer's own deploy.
- Genericise infra-specific code comments (egress-proxy examples) to neutral wording.
- Replace the hardcoded contact email on the legal pages with the instance operator's
  configured admin email, served via the public /auth/config and shown with a neutral
  fallback — so each self-hosted instance shows its own contact.
- Rewrite README for the current app + a copy-paste self-hosting quick start (prebuilt
  image + first-run wizard) with a build-from-source alternative; tidy .env.example.
This commit is contained in:
npeter83 2026-07-01 12:46:50 +02:00
parent 381794d9ae
commit 0d44d3a34a
10 changed files with 149 additions and 119 deletions

View file

@ -48,15 +48,9 @@ SMTP_USER=
SMTP_PASSWORD=
SMTP_FROM=
# --- Deployment role ---
# DATABASE_URL is set automatically by docker-compose.yml / docker-compose.server.yml to the
# bundled `db` service. Override it only for local dev against the central server DB, e.g.:
# DATABASE_URL=postgresql+psycopg://siftlode:<password>@your-db-host:5432/siftlode
# Exactly one instance may run the background scheduler. The central server keeps it on; every
# other instance (local dev, etc.) must set SCHEDULER_ENABLED=false. The compose files set this
# for you (server=true via docker-compose.server.yml, local=false via docker-compose.localdev.yml).
# --- Scheduler ---
# The background scheduler (subscription sync, backfill, enrichment) runs inside the app.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. If you ever run
# more than one instance against the same database, keep SCHEDULER_ENABLED=true on exactly one
# of them and false on the rest, to avoid double quota use and write races.
SCHEDULER_ENABLED=true
# On the central server, set this so backup.sh / restore.sh and plain `docker compose` commands
# target the server stack automatically:
# COMPOSE_FILE=docker-compose.server.yml

161
README.md
View file

@ -1,106 +1,109 @@
# Siftlode
Self-hosted, multi-user web app for browsing **your own YouTube subscriptions** the way you
actually want: precise filtering and sorting (by language, topic, length, age, watch state…),
a fast local-first feed, and one-click playback that opens the real youtube.com — so your
browser's ad blocking and SponsorBlock keep working exactly as before.
**Your YouTube subscriptions, the way a feed should work.** Siftlode pulls every upload from the
channels you follow into one clean, filterable feed — no algorithm deciding what you see, and no
Shorts or livestream noise unless you want it. Self-hosted, multi-user, and private: your data
stays on your own server.
Each user signs in with their own Google account (invite-only) and sees only their own
subscriptions. All the expensive data (channels, videos, metadata) is fetched from YouTube
once and stored locally, so filtering/searching/sorting are instant and don't burn API quota.
![Siftlode feed](frontend/public/welcome/feed.png)
> Status: early development.
> - **M1** (foundation): docker-compose stack, FastAPI backend, Google OAuth login with an
> email invite-list, encrypted token storage.
> - **M2** (ingest core): subscription import, free RSS detection, recent-first + deep backfill
> from the uploads playlist, enrichment (duration/stats/category, Shorts & livestream
> classification), a shared daily quota guard, and a background scheduler.
> - **M3** (auto-tagging): system tags for channel language (offline detection) and topic
> (from YouTube topics + dominant category), regenerated automatically; user tags are
> never overwritten.
> - **M4** (reader UI): React + Vite SPA with four color schemes (dark/light) and adjustable
> text size; grid/list feed scoped to your subscriptions, with faceted tag filters,
> content-type toggles (Normal/Shorts/Live), date range, search, sort, watch/save/hide
> state and per-channel filtering; clicking a video opens youtube.com. Accurate Shorts
> detection via the /shorts probe, a sync-status indicator with admin pause/resume, a
> filtered video count, and structured timestamped logging.
Everything expensive (channels, videos, metadata) is fetched from YouTube once and stored locally,
so filtering, searching and sorting are instant and don't burn API quota. Click a video to watch it
in an in-app player that resumes where you left off — or open it on youtube.com so your own ad
blocker and SponsorBlock keep working.
## Requirements
## Features
- Docker + Docker Compose
- A Google Cloud project with an OAuth client (see below)
- **A readable subscription feed** — sort and filter by channel, tag, language, topic, length,
upload date or watch state; hide channels without unsubscribing; save filter setups as named views.
- **Search all of YouTube** from the feed — results play, save and add to playlists like any other
video, and are materialised into your catalog.
- **Channel pages & a channel manager** — per-channel stats and uploads, priorities, and your own
tags to slice the feed by.
- **Playlists with two-way YouTube sync** — build them locally, keep them in sync in both directions.
- **In-app player** with resume, plus keyboard/scroll controls.
- **Multi-user** with per-user private state, a shared catalog, and a fair daily API-quota guard.
- **Self-hosted & private**, with a first-run web setup wizard and the interface in **English,
Hungarian and German**.
## Quick start
## Quick start (self-hosting)
1. Copy the env template and generate secrets:
You don't need to build anything — Siftlode runs from a prebuilt public image, and all
configuration (your admin account, Google sign-in, email) happens in a **first-run web wizard**.
You need [Docker](https://docs.docker.com/get-docker/) with the Compose plugin.
**1. Get the files and run the installer:**
```sh
git clone https://forge.b1fr0st.eu/peter/siftlode.git
cd siftlode
./install.sh # Windows (PowerShell): ./install.ps1
```
The installer generates a private `.env` (secrets), pulls the image, starts the app + database, and
prints a one-time setup URL like `http://localhost:8080/setup?token=…`.
**2. Finish in your browser.** Open that URL and follow the wizard:
1. **Admin account** — the email + password you'll sign in with.
2. **Google sign-in** *(optional)* — paste a Google OAuth client to enable "Sign in with Google" and
pulling your YouTube subscriptions. Skip it to use email + password only.
3. **Email / SMTP** *(optional)* — for verification/notification emails. Skip it and you (the admin)
simply approve new accounts yourself.
Then sign in with your admin account. That's it. See **[docs/self-hosting.md](docs/self-hosting.md)**
for the full walkthrough.
> Just trying it out? Press Enter at the installer's URL prompt to run on `http://localhost:8080`.
## Build from source (alternative)
Prefer to build the image yourself instead of pulling it:
```sh
git clone https://forge.b1fr0st.eu/peter/siftlode.git
cd siftlode
cp .env.example .env
python -c "import secrets;print('SECRET_KEY=',secrets.token_urlsafe(48))"
python -c "import base64,os;print('TOKEN_ENCRYPTION_KEY=',base64.urlsafe_b64encode(os.urandom(32)).decode())"
# generate the two secrets and paste them into .env:
python -c "import secrets;print('SECRET_KEY='+secrets.token_urlsafe(48))"
python -c "import base64,os;print('TOKEN_ENCRYPTION_KEY='+base64.urlsafe_b64encode(os.urandom(32)).decode())"
docker compose up --build -d # builds from the included Dockerfile
```
Paste the generated values into `.env`.
Open `http://localhost:8080` and finish in the setup wizard as above. (Set a `POSTGRES_PASSWORD` in
`.env` too.)
2. Create a Google OAuth client (Google Cloud Console → APIs & Services):
- Enable the **YouTube Data API v3**.
- OAuth consent screen: **External**, publishing status **Testing**, and add every invited
Google account as a **Test user** (up to 100 — no app verification needed at this scale).
- Create credentials → **OAuth client ID** → type **Web application**.
- Authorized redirect URI: `http://localhost:8080/auth/callback`
(must match `OAUTH_REDIRECT_URL` in `.env`).
- Put the client ID/secret into `.env`, and list invited emails in `ALLOWED_EMAILS`.
## HTTPS / public access
3. Start it:
Port `8080` over plain HTTP is fine for a LAN or a quick trial. For public access put a reverse
proxy (Caddy, Nginx, Traefik…) in front to terminate TLS, and set the **public URL** (the installer
prompt, or `OAUTH_REDIRECT_URL` in `.env`) to your `https://…` address — this also marks the session
cookie secure. Add that same `…/auth/callback` URL to your Google OAuth client's authorized redirect
URIs.
## Updating & backups
```sh
docker compose up --build
docker compose -f docker-compose.selfhost.yml pull # or: docker compose pull
docker compose -f docker-compose.selfhost.yml up -d
```
Open http://localhost:8080 and sign in. Database migrations run automatically on startup.
Database migrations run automatically on startup. Your data (accounts, subscriptions, playlists, the
video catalog) lives in a Postgres volume — back it up with `scripts/backup.sh` (or `backup.ps1` on
Windows) and restore with `scripts/restore.sh`.
## Backup & moving to another machine
## How it works
All data lives in the Postgres database — moving to another host (e.g. a Proxmox Linux
server) does **not** require re-fetching from YouTube. Copy your `.env` (keep the same
`TOKEN_ENCRYPTION_KEY` and Google client so stored tokens stay valid), then:
```sh
./scripts/backup.sh # -> backups/siftlode-<timestamp>.dump (Windows: scripts\backup.ps1)
./scripts/restore.sh backups/<file> # on the new host after `docker compose up`
```
## Central server + local dev (single shared database)
For always-on operation the recommended topology is a single **central Postgres** running on a
24/7 host (e.g. a Proxmox LXC) that also runs the background scheduler, with every other instance
(your local dev machine, a future VPS) pointing at that same database. One source of truth, no
sync.
- **Server** (`docker-compose.server.yml`): full stack, Postgres published on the LAN, scheduler
on, DB files in a host-visible `./pgdata` bind mount. In the LXC, from the project root:
```sh
echo 'COMPOSE_FILE=docker-compose.server.yml' >> .env # so backup/restore target this stack
docker compose up -d --build
./scripts/restore.sh backups/<file> # migrate existing data in
```
Set `YOUTUBE_API_KEY` in `.env` so unattended backfill/enrichment use the API key and don't
depend on a refresh token (which expires after 7 days while the OAuth screen is in *Testing*).
- **Local dev** (`docker-compose.localdev.yml`): webapp only, no local DB, no scheduler. In `.env`
set `DATABASE_URL` to the central DB (e.g. `…@your-db-host:5432/siftlode`), then
`docker compose -f docker-compose.localdev.yml up --build` and browse http://localhost:8080.
**Exactly one instance may run the scheduler.** The server keeps `SCHEDULER_ENABLED=true`; every
other instance must be `false` (the compose files enforce this) to avoid double quota burn and
write races on the shared DB.
- **Shared catalog, private state.** Channels and videos are stored once and shared; each user's
subscriptions, tags, playlists and watch/save/hide state are private.
- **Cheap by design.** Public reads are cached locally; a shared daily quota budget and a background
scheduler keep unattended syncing within YouTube's free API limits. An optional API key lets
backfill run without depending on a user's OAuth token.
## Tech
FastAPI + PostgreSQL backend, React + Vite frontend (added in a later milestone), packaged with
Docker Compose.
FastAPI + PostgreSQL (SQLAlchemy, Alembic) backend; React + Vite + Tailwind + TanStack Query
frontend; packaged as a single Docker image with Docker Compose.
## Note

View file

@ -805,4 +805,6 @@ def auth_config(db: Session = Depends(get_db)) -> dict:
return {
"google_enabled": google_enabled(db),
"allow_registration": sysconfig.get_bool(db, "allow_registration"),
# Contact shown on the public legal pages (the operator's admin email), if configured.
"operator_contact": operator_contact(),
}

View file

@ -66,8 +66,8 @@ class Settings(BaseSettings):
# preferred for shared backfill/enrichment so it doesn't depend on a specific user.
youtube_api_key: str = ""
# Optional HTTP(S) proxy for outbound YouTube Data API calls. Set this to send the
# scheduler's googleapis traffic through a fixed-IP host (e.g. the VPS over WireGuard) so an
# IP-restricted API key keeps working even when this host's public IP is dynamic.
# scheduler's googleapis traffic through a fixed-IP host so an IP-restricted API key
# keeps working even when this host's public IP is dynamic.
youtube_api_proxy: str = ""
# Daily quota budget (units). Default leaves headroom under the 10,000/day free limit.
quota_daily_budget: int = 9000

View file

@ -44,8 +44,8 @@ class YouTubeClient:
def __init__(self, db, user: User):
self.db = db
self.user = user
# Optionally route all YouTube traffic through a fixed-IP egress proxy (e.g. the VPS
# over WireGuard) so an IP-restricted API key keeps working from a dynamic-IP host.
# Optionally route all YouTube traffic through a fixed-IP egress proxy so an
# IP-restricted API key keeps working from a host with a dynamic public IP.
proxy = sysconfig.get_str(db, "youtube_api_proxy") or None
self._http = httpx.Client(timeout=30.0, proxy=proxy)

View file

@ -46,7 +46,7 @@ services:
depends_on:
db:
condition: service_healthy
# Harmless on Docker Desktop; needed if ever run under Docker-in-LXC.
# Harmless on Docker Desktop; needed on some nested/hardened Docker hosts.
security_opt:
- apparmor:unconfined
ports:

View file

@ -1,10 +1,47 @@
// Shared shell for the public, login-free legal pages (/privacy, /terms). These must be
// reachable unauthenticated — Google's OAuth consent screen links to them and reviewers
// fetch them directly — so they render outside the authenticated App tree (see main.tsx).
// fetch them directly — so they render outside the authenticated App tree (see main.tsx),
// with no react-query provider: the operator contact is fetched with a plain fetch below.
import { useEffect, useState } from "react";
export const CONTACT_EMAIL = "b1fr0stmailops@gmail.com";
export const LAST_UPDATED = "June 14, 2026";
// Contact shown on the legal pages: the instance operator's configured admin email, from the
// public /auth/config. One shared fetch across the page; null when the operator set none.
let contactPromise: Promise<string | null> | null = null;
function fetchOperatorContact(): Promise<string | null> {
if (!contactPromise) {
contactPromise = fetch("/auth/config")
.then((r) => (r.ok ? r.json() : null))
.then((c) => (c?.operator_contact as string | undefined) ?? null)
.catch(() => null);
}
return contactPromise;
}
export function useOperatorContact(): string | null {
const [contact, setContact] = useState<string | null>(null);
useEffect(() => {
let alive = true;
fetchOperatorContact().then((c) => alive && setContact(c));
return () => {
alive = false;
};
}, []);
return contact;
}
// The operator's contact rendered as a mailto link, or a neutral fallback phrase when unset.
export function ContactEmail() {
const email = useOperatorContact();
if (!email) return <>your instance&apos;s administrator</>;
return (
<a href={`mailto:${email}`} className="text-accent hover:underline">
{email}
</a>
);
}
export function LegalLink({ href, children }: { href: string; children: React.ReactNode }) {
return (
<a href={href} target="_blank" rel="noreferrer" className="text-accent hover:underline">
@ -20,6 +57,7 @@ export default function LegalLayout({
title: string;
children: React.ReactNode;
}) {
const contact = useOperatorContact();
return (
<div className="min-h-screen bg-bg text-fg">
<div className="mx-auto w-[min(92vw,720px)] py-12">
@ -35,7 +73,9 @@ export default function LegalLayout({
<a href="/" className="hover:text-fg transition">Home</a>
<a href="/privacy" className="hover:text-fg transition">Privacy Policy</a>
<a href="/terms" className="hover:text-fg transition">Terms of Service</a>
<a href={`mailto:${CONTACT_EMAIL}`} className="hover:text-fg transition">Contact</a>
{contact && (
<a href={`mailto:${contact}`} className="hover:text-fg transition">Contact</a>
)}
</footer>
</div>
</div>

View file

@ -1,4 +1,4 @@
import LegalLayout, { CONTACT_EMAIL, H2, LegalLink } from "./LegalLayout";
import LegalLayout, { ContactEmail, H2, LegalLink } from "./LegalLayout";
export default function PrivacyPolicy() {
return (
@ -71,11 +71,7 @@ export default function PrivacyPolicy() {
myaccount.google.com/permissions
</LegalLink>
; once revoked, the stored tokens can no longer be used. To have your account and
associated data deleted, email{" "}
<a href={`mailto:${CONTACT_EMAIL}`} className="text-accent hover:underline">
{CONTACT_EMAIL}
</a>
.
associated data deleted, contact <ContactEmail />.
</p>
<H2>Cookies</H2>
@ -100,11 +96,7 @@ export default function PrivacyPolicy() {
<H2>Contact</H2>
<p>
Questions about this policy or your data:{" "}
<a href={`mailto:${CONTACT_EMAIL}`} className="text-accent hover:underline">
{CONTACT_EMAIL}
</a>
.
Questions about this policy or your data: contact <ContactEmail />.
</p>
</LegalLayout>
);

View file

@ -1,4 +1,4 @@
import LegalLayout, { CONTACT_EMAIL, H2, LegalLink } from "./LegalLayout";
import LegalLayout, { ContactEmail, H2, LegalLink } from "./LegalLayout";
export default function Terms() {
return (
@ -56,11 +56,7 @@ export default function Terms() {
<H2>Contact</H2>
<p>
Questions:{" "}
<a href={`mailto:${CONTACT_EMAIL}`} className="text-accent hover:underline">
{CONTACT_EMAIL}
</a>
.
Questions: contact <ContactEmail />.
</p>
</LegalLayout>
);

View file

@ -773,8 +773,11 @@ export const api = {
}),
// Public: which sign-in options this instance offers (e.g. hide Google when not configured).
authConfig: (): Promise<{ google_enabled: boolean; allow_registration: boolean }> =>
req("/auth/config"),
authConfig: (): Promise<{
google_enabled: boolean;
allow_registration: boolean;
operator_contact: string | null;
}> => req("/auth/config"),
// --- first-run install wizard (token from the setup URL printed to the container logs) ---
setupStatus: (): Promise<{ configured: boolean }> => req("/api/setup/status"),