AI crawler technical checklist.

1. Verify the public response

• The canonical URL returns 200 OK without a login, consent wall or challenge.
• Redirects are intentional, short and end on the canonical host.
• The response is not a soft 404 or generic shell with no useful content.
• TLS works and the server is consistently available.
• CDN, WAF and rate-limit rules do not challenge legitimate documented crawlers.

curl -I https://example.com/important-page
curl -A "OAI-SearchBot" -I https://example.com/important-page
curl -A "Claude-SearchBot" -I https://example.com/important-page
curl -A "PerplexityBot" -I https://example.com/important-page

A user-agent string alone does not authenticate a bot. Use official IP information where a provider publishes it and examine server logs when access control matters.

2. Review robots.txt

• robots.txt exists at the origin root and returns plain text.
• No broad Disallow rule blocks public pages you want surfaced.
• Specific search bots are allowed when AI-search visibility is the goal.
• The sitemap URL is declared and resolves successfully.
• Staging, account and private paths remain protected through real access controls; robots rules are not security.

User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

Only add training crawlers such as GPTBot or ClaudeBot after making a deliberate content-use decision. Search visibility and model training are separate controls.

3. Check page-level indexing directives

• No unintended <meta name="robots" content="noindex"> is present.
• No unintended X-Robots-Tag: noindex header is returned.
• The canonical points to the preferred, indexable URL.
• Alternate language and pagination signals, if used, are internally consistent.

Do not block a page in robots.txt when you need a crawler to see its noindex directive. OpenAI’s publisher guidance makes the same important distinction.

4. Inspect the delivered HTML

• The title, primary heading and main answer are present in the initial or server-rendered HTML.
• Important links use ordinary crawlable <a href> elements.
• Content does not require a click, scroll or client-only API response to exist.
• Semantic headings, lists, tables and landmarks reflect the information structure.
• Structured data is valid, visible-content-consistent and uses appropriate Schema.org types.

Modern crawlers can render JavaScript to different degrees, but a useful HTML response is faster, more resilient and easier to debug.

5. Validate discovery and canonicalization

• Every indexable page is reachable from another useful page through internal links.
• The XML sitemap contains only canonical, indexable URLs and uses accurate modification dates.
• HTTP, HTTPS, apex, www, trailing-slash and file-extension variants converge consistently.
• Preview and deployment aliases declare or redirect to the production canonical.
• Broken internal links and missing assets are removed or repaired.

6. Add machine-readable aids proportionately

• Use structured data to state accurate facts already visible to users.
• Provide llms.txt as a concise map if it is useful to systems or people who choose to read it.
• Keep an expanded plain-text summary factual and synchronized if you publish one.
• Do not present these files as substitutes for crawlable pages, a sitemap or authority.

Google’s official guidance says llms.txt neither helps nor hurts visibility in Google Search. There is also no special structured-data type required for generative AI features.

7. Test, log and monitor

• Fetch key URLs with documented crawler user agents and confirm the same useful response.
• Monitor server or CDN logs for bot requests, statuses and blocked paths.
• Recheck after firewall, CMS, framework or routing changes.
• Track index discovery separately from answer citations.
• Preserve a dated technical baseline so regressions are obvious.

Official crawler references