fix(deferred): close backend correctness/efficiency tails from the hygiene sweep
Verified each deferred per-subsystem item against current source, then fixed the
real ones (tradeoffs left as-is, see siftlode-ops/CODE-HYGIENE.md closeout):
- search BUG-3: don't finalise shorts_probed on un-enriched stubs (enrichment
failure/omitted rows) — restores the scheduler's enriched_at guard so a real
Short can't leak into the feed and never get reclassified.
- downloads GC: 4th pass reaps orphaned errored, fileless, unreferenced assets
(they carry no TTL, so the expiry passes never cleared them).
- downloads B6: quota/edit errors now return a structured {code,reason,limit}
the trilingual client localises, instead of hardcoded English + dot-decimal GB.
- channels BB1: reset-backfill re-arms deep sync on every subscriber (bulk update)
instead of 404ing when the admin isn't personally subscribed.
- channels BB2: enrich the stub on BOTH the normal and "already exists" desync
path (nest the insert try inside the client `with`).
- channels CB3: drop the redundant second _discovery_rows read (identity-mapped
rows are already enriched; re-sort in Python for the title tie-break).
- playlists PC1: read the live playlist once in push() and share it with plan_push
+ push_playlist (halves read-quota, closes the second TOCTOU window).
- playlists PC2/PC5: batch the cover thumbnails in one window query (was N+1);
rename combines count+duration into one aggregate + a single cover lookup.
- messages MB2: get_thread only resolves a messageable partner OR one we already
share a thread with (closes the user-enumeration oracle).
- messages MB3: push a live unread-count to the user's other tabs after mark-read.
- messages MC4: list_conversations uses DISTINCT ON + grouped COUNT instead of
loading the whole message history into memory.
- config AB3: per-spec allow_empty so smtp_user/youtube_api_proxy can be blanked
to disable them rather than snapping back to the env default.
This commit is contained in:
parent
182dddf5ed
commit
4e80e2b39b
8 changed files with 205 additions and 102 deletions
|
|
@ -1,12 +1,14 @@
|
|||
"""Download retention garbage collector (a scheduler job).
|
||||
|
||||
Three passes each run:
|
||||
Four passes each run:
|
||||
1. Pre-expiry warning — for ready assets expiring within the grace window, notify each holder
|
||||
once (gc_notified flag) so a download isn't silently deleted.
|
||||
2. Expiry deletion — delete assets past their TTL (files + row). The FK SET NULL nulls the
|
||||
holders' jobs' asset_id, so those jobs surface as "expired" (re-downloadable) in the UI.
|
||||
3. LRU eviction — if a total cache cap is set and exceeded, delete least-recently-accessed
|
||||
ready assets until back under it.
|
||||
4. Orphan reap — delete errored, fileless, unreferenced asset rows (they carry no TTL, so
|
||||
passes 1-3 skip them; benign row-count hygiene).
|
||||
|
||||
Runs in the API process (which mounts DOWNLOAD_ROOT). Pure disk/DB work — no YouTube quota.
|
||||
"""
|
||||
|
|
@ -58,7 +60,7 @@ def _delete_asset(db: Session, asset: MediaAsset, ntype: str) -> None:
|
|||
def run_download_gc(db: Session) -> dict:
|
||||
now = datetime.now(timezone.utc)
|
||||
grace_days = sysconfig.get_int(db, "download_gc_grace_days")
|
||||
warned = expired = evicted = 0
|
||||
warned = expired = evicted = reaped = 0
|
||||
|
||||
# 1. Pre-expiry warnings.
|
||||
warn_before = now + timedelta(days=grace_days)
|
||||
|
|
@ -114,6 +116,22 @@ def run_download_gc(db: Session) -> dict:
|
|||
evicted += 1
|
||||
db.commit()
|
||||
|
||||
result = {"warned": warned, "expired": expired, "evicted": evicted}
|
||||
# 4. Reap orphaned errored rows (no file, no holder). get_or_create_asset resets a still-
|
||||
# referenced errored row to pending on re-enqueue, so these only accumulate once every
|
||||
# holder is gone; they carry no expires_at, so passes 1-3 never touch them.
|
||||
orphans = db.execute(
|
||||
select(MediaAsset).where(
|
||||
MediaAsset.status == "error",
|
||||
MediaAsset.ref_count == 0,
|
||||
MediaAsset.rel_path.is_(None),
|
||||
)
|
||||
).scalars().all()
|
||||
for asset in orphans:
|
||||
db.delete(asset) # no file to unlink; FK SET NULL clears any stray job.asset_id
|
||||
reaped += 1
|
||||
if orphans:
|
||||
db.commit()
|
||||
|
||||
result = {"warned": warned, "expired": expired, "evicted": evicted, "reaped": reaped}
|
||||
log.info("download_gc %s", result)
|
||||
return result
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue