6232a66878
Shared links didn't unfurl in Messenger: robots.txt disallowed /watch/ for ALL user-agents, so facebookexternalhit honoured it and got a blocked (403) fetch — the OG tags were correct but never read. Add an allow group for the link-preview bots (facebookexternalhit, Twitterbot, Discordbot, WhatsApp, Slack, Telegram, LinkedIn, …) so they can fetch share pages; the wildcard still keeps /watch/ out of search indexes. Release v0.32.1.
27 lines
927 B
Text
27 lines
927 B
Text
# Siftlode is mostly a private, login-gated app; only the public landing and legal
|
|
# pages are worth indexing. Allow crawling but keep bots out of the API surface.
|
|
# Share links (/watch/) are unguessable capability URLs: kept out of search indexes,
|
|
# but link-preview crawlers are allowed below so shared links unfurl in chat apps.
|
|
User-agent: *
|
|
Allow: /
|
|
Disallow: /api/
|
|
Disallow: /auth/
|
|
Disallow: /watch/
|
|
|
|
# Link-preview / unfurl bots — allowed to fetch /watch/ so shared links render a rich
|
|
# card (title/channel/thumbnail). They don't build a search index, so this doesn't
|
|
# expose share links to search engines.
|
|
User-agent: facebookexternalhit
|
|
User-agent: Facebot
|
|
User-agent: Twitterbot
|
|
User-agent: Slackbot-LinkExpanding
|
|
User-agent: Slackbot
|
|
User-agent: Discordbot
|
|
User-agent: TelegramBot
|
|
User-agent: WhatsApp
|
|
User-agent: LinkedInBot
|
|
User-agent: Pinterest
|
|
User-agent: redditbot
|
|
Allow: /
|
|
Disallow: /api/
|
|
Disallow: /auth/
|