# TraderPier — crawler policy.
#
# The application is a hash router with no server-side rendering, so every
# in-app route is `traderpier.com/#/…` and a crawler sees exactly one document
# for the whole product. That has not changed.
#
# What HAS changed is that the product no longer ships only the app. The
# marketing pages are static HTML written into the same build output at real,
# extensionless paths (`frontend/scripts/build-marketing-pages.js`), served
# directly by the existing distribution because a path that exists as an S3
# object never reaches the SPA fallback. Those paths are addressable, so they
# are listable — and `sitemap.xml` lists them and the app root and nothing else:
# the approved marketing pages and, once a post is approved, the blog index and
# its posts, every one without a trailing slash.
#
# Short links (`/r/<code>`) are deliberately NOT disallowed: link-preview bots
# honour this file, and a refused fetch would post the link with no card. Each
# one is `noindex` and stays out of the sitemap, so it is fetched, never listed.
#
# Decision D-C4.6 refused a sitemap and was RIGHT about the app: a sitemap of
# `#/` fragments claims indexability a hash router cannot honour, because a
# crawler cannot address a fragment. The rule NARROWED rather than reversed.
# `src/common/seo/link-preview.test.ts` now fails if anything fragment-shaped
# appears in the sitemap, which is the half of D-C4.6 that was always the point.
#
# `/api/` is disallowed rather than left implicit: it answers JSON, and a crawler
# spending budget on it is pure waste.

# AI answer engines and their crawlers are named on purpose (GEO baseline, docs/marketing/geo/): an assistant that
# cannot read the site cannot name TraderPier, so each is allowed in explicitly instead of relying on the `*` group
# alone. A crawler obeys only its most specific group, so each group repeats the `/api/` disallow.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bingbot
Allow: /
Disallow: /api/

User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://traderpier.com/sitemap.xml
# The API renders /i/ and /s/, so it renders their sitemaps (`api/src/preview/preview-sitemaps.ts`): published
# Community indicators, and public shares made by our own staff accounts. Customers' public shares are never listed.
Sitemap: https://traderpier.com/i/sitemap.xml
Sitemap: https://traderpier.com/s/sitemap.xml