Is your website blocking AI assistants?

Check yoursite.com/robots.txt first. If it blocks OAI-SearchBot, Claude-SearchBot or PerplexityBot, or blocks everyone with User-agent: * and Disallow: /, those assistants can’t use your pages when a client asks. Then check two less obvious things: a “block AI” setting at your host or security service, and facts that live in images or PDFs rather than page text. The whole check takes about 10 minutes, and each AI company’s own documentation tells you which names to allow.

What robots.txt is, in plain words

Every website can have a small text file at /robots.txt. It lists rules for the automated programs that visit websites: which ones may read which pages. Each program announces itself with a name, called a user agent, such as “OAI-SearchBot” for ChatGPT search. A rule like this one tells that program to stay out of the whole site:

User-agent: OAI-SearchBot
Disallow: /

The AI companies’ documentation says their search and training programs follow these rules. Many advisor sites never touch the file, but website builders, security plugins and hosting templates sometimes write rules into it, or block AI traffic before it reaches the site at all.

The user agents that matter

Each major AI company now runs separate programs for separate jobs: one that builds a search index, one that fetches a page because a person just asked a question, and one that collects pages for training. You can allow one and block another.

Three things follow from the companies’ documentation:

• Search and training are separate choices. OpenAI says disallowing GPTBot means your content “should not be used in training”, while OAI-SearchBot decides whether you appear in ChatGPT search. Anthropic, Google and Apple draw the same line. Blocking training doesn’t keep you out of answers.
• Google-Extended is the one to read twice. Google says it doesn’t affect Google Search or rankings. But it also controls grounding, which is how the Gemini app pulls in pages from Google’s index while answering. Block it and Gemini answers can’t draw on your site.
• The in-the-moment fetchers may visit anyway. OpenAI and Perplexity both say robots.txt may not apply when a person asked for the page. That’s not a reason to block them in robots.txt; it’s a reason not to rely on robots.txt alone if you truly want something private.

OpenAI also notes that it can take about 24 hours for a robots.txt change to take effect in its systems.

What to allow

If you want to be recommended, allow every search and answer agent above. Training is a separate decision with no effect on whether you’re named, according to each company. A file that does both looks like this:

# AI search and answers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
Allow: /

# AI training: optional. Delete this group to allow training too.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including Google Search and Apple search
User-agent: *
Allow: /

If your file is short or doesn’t exist, you’re probably fine already: with no rule, everything is allowed. What to look for is any Disallow: / under a name in the table, or under *.

The blocks that aren’t in robots.txt

A clean robots.txt doesn’t settle it. These settings block AI assistants without touching the file:

• A “block AI” switch at your host or security service. Some hosting and security services offer one, and it can be switched on without anyone at the firm noticing. Look in your hosting dashboard for settings about AI, scrapers or automated traffic.
• Security plugins. “Block bad traffic” settings on website builders and WordPress plugins sometimes include the AI companies’ user agents. Check after any website change.
• Challenge pages. A “checking your browser” screen stops a person for a second. It stops an automated visitor completely.
• Noindex and snippet tags. A noindex tag keeps a page out of search. Google’s nosnippet and max-snippet controls limit what can be shown from a page in Search, including its AI features. Website builders sometimes leave these on from a staging site.
• Login walls and pop-ups. If the fees page sits behind a form or a full-screen pop-up, an assistant may never see the text under it.

Facts that are on the page but can’t be read

Even with every door open, an assistant can only repeat what it can read as text. Google’s guidance for its AI features says the same thing in plain words: make sure “important content is available in textual form.”

The usual problems on advisor sites:

• Fees in a PDF. A fee schedule that only exists as a downloadable PDF or inside Form ADV Part 2A is often not read. Put the same numbers on a web page.
• Specialties in a banner image. “Planning for physicians” set in a headline graphic is invisible as text.
• Text that only appears through scripts. Tabs, accordions and calculators that load their content after the page opens may show nothing to an automated reader.

This is one reason so many firms can’t be quoted. In our website study, we counted a price only when it appeared in plain text on the firm’s site, and 52% of fee-only firms had one. For how to write that page, see Publish your fees: why price pages get cited.

Structured data: useful, not required

Structured data is a short block of code, invisible to visitors, that states facts about a page in a fixed format. For an advisory firm, the schema.org type is FinancialService, a kind of local business. It has fields for your service area, a price range, your fees and commissions, the topics you know about and your credentials.

Google is clear that it isn’t required: “There’s also no special schema.org structured data that you need to add” to appear in AI Overviews or AI Mode. And it should say exactly what the visible page says. Used that way, it removes guesswork about your name, address, area and fee model. A minimal version:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FinancialService",
  "name": "Example Planning LLC",
  "url": "https://www.example.com/",
  "address": { "@type": "PostalAddress", "addressLocality": "Denver", "addressRegion": "CO" },
  "areaServed": ["Denver, CO", "Colorado"],
  "knowsAbout": ["Equity compensation", "Retirement income planning"],
  "feesAndCommissionsSpecification": "https://www.example.com/fees/"
}
</script>

Point feesAndCommissionsSpecification at your fees page rather than restating numbers in two places that can drift apart.

The 10-minute check

1. Open yoursite.com/robots.txt. Look for Disallow: / under any name in the table above, or under *.
2. Open your hosting and security dashboards. Look for settings that block AI or automated traffic.
3. Open your fees page. Select the prices with your cursor. If you can’t select them as text, neither can an assistant.
4. Do the same for your specialties and service area on the homepage.
5. Ask ChatGPT and Perplexity, with web search on, “What does [your firm] in [city] charge?” If neither cites your site, something above is still blocking it. The full method is in How to check what AI says about your firm.

Where Wellspent fits

Your website is one source. A Wellspent record is another, always readable: your fees, specialties, service area and the clients you serve in plain text in the public graph at find.wellspent.co, verified against SEC data. Membership is $89 a month and includes the monthly AI visibility report, introductions from prospects with no lead fees, and quarterly press placement with media partners.