peter/siftlode
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(share): let link-preview crawlers fetch /watch/ pages (robots.txt)

Browse source 
Shared links didn't unfurl in Messenger: robots.txt disallowed /watch/ for ALL
user-agents, so facebookexternalhit honoured it and got a blocked (403) fetch —
the OG tags were correct but never read. Add an allow group for the link-preview
bots (facebookexternalhit, Twitterbot, Discordbot, WhatsApp, Slack, Telegram,
LinkedIn, …) so they can fetch share pages; the wildcard still keeps /watch/ out
of search indexes. Release v0.32.1.
This commit is contained in:
npeter83 2026-07-07 22:56:38 +02:00
parent b63b3334d6
commit 6232a66878
3 changed files with 29 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

20
frontend/public/robots.txt
View file

@ -1,7 +1,27 @@
# Siftlode is mostly a private, login-gated app; only the public landing and legal
# pages are worth indexing. Allow crawling but keep bots out of the API surface.
# Share links (/watch/) are unguessable capability URLs: kept out of search indexes,
# but link-preview crawlers are allowed below so shared links unfurl in chat apps.
User-agent: *
Allow: /
Disallow: /api/
Disallow: /auth/
Disallow: /watch/
# Link-preview / unfurl bots — allowed to fetch /watch/ so shared links render a rich
# card (title/channel/thumbnail). They don't build a search index, so this doesn't
# expose share links to search engines.
User-agent: facebookexternalhit
User-agent: Facebot
User-agent: Twitterbot
User-agent: Slackbot-LinkExpanding
User-agent: Slackbot
User-agent: Discordbot
User-agent: TelegramBot
User-agent: WhatsApp
User-agent: LinkedInBot
User-agent: Pinterest
User-agent: redditbot
Allow: /
Disallow: /api/
Disallow: /auth/