feat(downloads): M3 — per-user quota + retention GC

- downloads/quota.py: per-user limits (DownloadQuota row or sysconfig defaults; unlimited
  bypass), footprint = distinct ready-asset bytes the user holds, check_enqueue (max_jobs +
  max_bytes hard at enqueue), at_concurrency_limit (worker-side max_concurrent), usage snapshot
- downloads/gc.py: run_download_gc — pre-expiry warning (once, gc_notified flag), TTL deletion
  (files + row; FK SET NULL orphans holders' jobs -> 'expired'), LRU eviction over a total-cache
  cap; notifies holders on each event
- migration 0038: media_assets.gc_notified
- config/sysconfig: download_total_max_bytes (0=unlimited) in the downloads group
- service.enqueue enforces quota; worker claim skips users at their concurrency limit
- scheduler: download_gc job registered (default 360 min, admin-tunable)

Verified on localdev: footprint accounting, max_jobs block, pre-expiry warning (notifies all
holders), TTL deletion (asset+files gone, dirs pruned, jobs orphaned, holders notified).
This commit is contained in:
npeter83 2026-07-03 00:26:40 +02:00
parent 7f9847c2c2
commit 7148335c09
9 changed files with 325 additions and 15 deletions

View file

@ -12,6 +12,7 @@ from sqlalchemy import select
from app import progress, quota
from app.config import settings
from app.db import SessionLocal
from app.downloads.gc import run_download_gc
from app.models import SchedulerSetting
from app.notifications import create_notification
from app.state import is_sync_paused
@ -59,6 +60,7 @@ JOB_INTERVALS: dict[str, int] = {
"maintenance": settings.maintenance_interval_minutes,
"demo_reset": settings.demo_reset_minutes,
"explore_cleanup": settings.explore_cleanup_minutes,
"download_gc": settings.download_gc_minutes,
}
# Sane bounds for an admin-set interval (minutes).
@ -276,6 +278,12 @@ def _explore_cleanup_job() -> None:
_job("explore_cleanup", purge_ephemeral)
def _download_gc_job() -> None:
# Retention GC for the download center: pre-expiry warnings, TTL deletion, LRU eviction.
# Pure disk/DB work (no YouTube quota).
_job("download_gc", run_download_gc)
# job_id -> wrapper. The single source of truth for which jobs exist and how to run one,
# shared by start_scheduler (recurring registration) and trigger_job (manual "run now").
JOB_FUNCS: dict[str, Callable[[], None]] = {
@ -289,6 +297,7 @@ JOB_FUNCS: dict[str, Callable[[], None]] = {
"maintenance": _maintenance_job,
"demo_reset": _demo_reset_job,
"explore_cleanup": _explore_cleanup_job,
"download_gc": _download_gc_job,
}