peter/siftlode
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(share): let link-preview crawlers fetch /watch/ pages (robots.txt)

Browse source 
Shared links didn't unfurl in Messenger: robots.txt disallowed /watch/ for ALL
user-agents, so facebookexternalhit honoured it and got a blocked (403) fetch —
the OG tags were correct but never read. Add an allow group for the link-preview
bots (facebookexternalhit, Twitterbot, Discordbot, WhatsApp, Slack, Telegram,
LinkedIn, …) so they can fetch share pages; the wildcard still keeps /watch/ out
of search indexes. Release v0.32.1.
This commit is contained in:
npeter83 2026-07-07 22:56:38 +02:00
parent b63b3334d6
commit 6232a66878
3 changed files with 29 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
VERSION
View file

@ -1 +1 @@
0.32.0
0.32.1

20
frontend/public/robots.txt
View file

@ -1,7 +1,27 @@
# Siftlode is mostly a private, login-gated app; only the public landing and legal
# pages are worth indexing. Allow crawling but keep bots out of the API surface.
# Share links (/watch/) are unguessable capability URLs: kept out of search indexes,
# but link-preview crawlers are allowed below so shared links unfurl in chat apps.
User-agent: *
Allow: /
Disallow: /api/
Disallow: /auth/
Disallow: /watch/
# Link-preview / unfurl bots — allowed to fetch /watch/ so shared links render a rich
# card (title/channel/thumbnail). They don't build a search index, so this doesn't
# expose share links to search engines.
User-agent: facebookexternalhit
User-agent: Facebot
User-agent: Twitterbot
User-agent: Slackbot-LinkExpanding
User-agent: Slackbot
User-agent: Discordbot
User-agent: TelegramBot
User-agent: WhatsApp
User-agent: LinkedInBot
User-agent: Pinterest
User-agent: redditbot
Allow: /
Disallow: /api/
Disallow: /auth/

8
frontend/src/lib/releaseNotes.ts
View file

@ -14,6 +14,14 @@ export interface ReleaseEntry {
}
export const RELEASE_NOTES: ReleaseEntry[] = [
{
version: "0.32.1",
date: "2026-07-07",
summary: "Fix: shared links now actually unfurl in chat apps.",
fixes: [
"Shared links now show a rich preview in Messenger and other chat apps. The site's robots.txt was telling every crawler — including link-preview bots — to skip /watch/ pages, so Facebook got a blocked response and showed just a bare link. Link-preview crawlers are now allowed to fetch share pages (they still stay out of search indexes).",
],
},
{
version: "0.32.0",
date: "2026-07-07",