d55799cbc1
Noisy YouTube titles are cleaned for display + storage, raw kept in videos.original_title: - app/titles.normalize_title: strip emoji/symbols (keep accents), remove trailing SEO hashtag clusters (keep numeric #3 episode markers), context-aware de-shout (mostly-ALL-CAPS titles -> Title Case with an acronym whitelist + function-word lowercasing; otherwise only long all-caps words), collapse repeated punctuation - applied at enrichment (sync/videos.py) and in the download worker (ad-hoc yt-dlp titles); catalog downloads inherit the normalized title automatically - migration 0039: add original_title, preserve raw, rewrite title (generated search_vector regenerates); reversible via original_title Backfill on localdev: 122115/273417 titles normalized in ~2 min. Verified in the feed + on real messy samples (emoji/de-shout/hashtags), accents + acronyms (PS5/AI/USA/PC) preserved.
80 lines
3.6 KiB
Python
80 lines
3.6 KiB
Python
"""Video title normalization — clean the noisy YouTube titles for display + storage.
|
|
|
|
Applied where a video's title is written (enrichment) and where a download's title comes from
|
|
yt-dlp, so the feed, search, channel pages, and the download center all show tidy titles. The
|
|
raw title is preserved in `Video.original_title` so this is reversible / re-derivable.
|
|
|
|
Rules (user-approved "option b"):
|
|
1. Drop emoji / pictographs / symbols / control chars (keep accents — HU/DE/…).
|
|
2. Strip trailing SEO hashtag clusters (word hashtags at the end); a numeric "#3" episode
|
|
marker survives.
|
|
3. De-shout ALL-CAPS words, context-aware:
|
|
- If the title is *mostly shouting* (≥ half the multi-letter words are all-caps), every
|
|
all-caps word is Title-cased (short shouted words like CAR/WAR/ÉN get fixed too), and
|
|
the first letter is capitalized. Known acronyms (PS, AI, USA, PC…) and words with
|
|
digits (PS5, 3D) are kept; function words (to/the/és/az…) are lowercased.
|
|
- Otherwise only long all-caps words (≥4 letters) are Title-cased, so a lone acronym in
|
|
an otherwise normal title is left alone.
|
|
4. Collapse repeated punctuation (!!! → !) and whitespace.
|
|
"""
|
|
import re
|
|
import unicodedata
|
|
|
|
_DROP_CATEGORIES = {"So", "Sk", "Cc", "Cf", "Cs", "Co", "Cn"}
|
|
_LETTER_RUN = re.compile(r"[^\W\d_]+", re.UNICODE)
|
|
_TRAILING_HASHTAGS = re.compile(r"(?:\s+#[^\s#]*[^\W\d\s_][^\s#]*)+\s*$", re.UNICODE)
|
|
_REPEAT_PUNCT = re.compile(r"([!?.,\-])\1{1,}")
|
|
|
|
# Short function words → lowercased when the title is de-shouted (proper title-case style).
|
|
_SHORT_LOWER = {
|
|
"to", "the", "and", "of", "a", "an", "in", "on", "at", "for", "or", "vs", "the",
|
|
"és", "az", "egy", "meg", "nem", "hogy", "de", "ha", "der", "die", "das", "und", "von",
|
|
}
|
|
# Kept as-is even when everything else is de-shouted (common acronyms; compared lowercased).
|
|
_ACRONYMS = {
|
|
"ps", "ai", "pc", "tv", "hd", "4k", "8k", "uk", "eu", "us", "usa", "gta", "rpg", "fps",
|
|
"diy", "vr", "ar", "id", "ok", "faq", "nasa", "fbi", "cia", "ceo", "amd", "pdf", "usb",
|
|
"gps", "api", "dj", "mc", "suv", "gpu", "cpu", "ssd", "hdmi", "led", "ufo", "dna", "nba",
|
|
"nfl", "asmr", "pov", "diy", "wtf", "lol", "rtx", "gtx", "ios", "mmo", "vip", "hp",
|
|
}
|
|
|
|
|
|
def _titlecase(w: str) -> str:
|
|
return w[0].upper() + w[1:].lower()
|
|
|
|
|
|
def normalize_title(raw: str | None) -> str | None:
|
|
"""Return the cleaned title, or the input unchanged for empty/whitespace."""
|
|
if not raw or not raw.strip():
|
|
return raw
|
|
t = unicodedata.normalize("NFKC", raw)
|
|
t = "".join(ch for ch in t if unicodedata.category(ch) not in _DROP_CATEGORIES)
|
|
t = _TRAILING_HASHTAGS.sub("", t)
|
|
|
|
multi = [r for r in _LETTER_RUN.findall(t) if len(r) >= 2]
|
|
caps = [r for r in multi if r.isupper()]
|
|
shouting = len(caps) >= 2 and len(caps) >= 0.5 * len(multi)
|
|
|
|
def repl(m: re.Match) -> str:
|
|
w = m.group(0)
|
|
if len(w) < 2 or not w.isupper():
|
|
return w
|
|
low = w.lower()
|
|
if low in _ACRONYMS:
|
|
return w
|
|
if low in _SHORT_LOWER:
|
|
return low
|
|
if shouting or len(w) >= 4:
|
|
return _titlecase(w)
|
|
return w # short all-caps in a non-shouting title → likely an acronym, keep
|
|
|
|
t = _LETTER_RUN.sub(repl, t)
|
|
t = _REPEAT_PUNCT.sub(r"\1", t)
|
|
t = re.sub(r"\s+", " ", t).strip()
|
|
|
|
if shouting: # ensure the first letter is capitalized (a leading function word got lowered)
|
|
for i, ch in enumerate(t):
|
|
if ch.isalpha():
|
|
t = t[:i] + ch.upper() + t[i + 1:]
|
|
break
|
|
return t or raw
|