Associations
Are Association Sites Visible to AI Search? A 184-Site Scan
In our October 2026 scan of 184 association websites, 44 blocked an AI crawler, and for 31 the only block was a firewall setting. How to check yours.
Alden Do Rosario
Founder and CEO, CustomGPT.ai
·
6 min read
A firewall setting that few people on your staff ever see may be keeping ChatGPT and other AI crawlers away from your public pages. My team at CustomGPT.ai scanned 184 association websites on October 8, 2026. 44 blocked at least one AI crawler, and for 31 of those 44 the only block we found was a firewall or bot-challenge setting, not a line in a robots.txt file your staff can read. Most of the sites we could measure showed no block at all.
Three plain definitions help. A crawler is the automated program an AI service uses to fetch web pages. Robots.txt is a text file on your site that tells crawlers which pages to skip. A firewall is security software, usually run by your host, that can turn a crawler away without any file saying so. Open is not the same as visible: we measured whether crawlers can read a site, not whether ChatGPT cites it.
RallyBoard’s June post, Stop Building a Destination. Start Owning the Relationship, argues that the answer increasingly comes to the person, who no longer goes to the destination. If the answer arrives before the visit, whether it can draw on your public pages starts with settings like the ones we scanned.
How many association websites block AI crawlers?
Of the 184 association websites we scanned, 44 blocked at least one AI crawler. Some of those 44 blocked only a training crawler, which does not keep a site out of AI search answers. For 31 of the 44, the only block we found was a firewall or bot-challenge rule, with no robots.txt rule against any of the AI crawlers we checked, so reading robots.txt alone would have shown nothing.
Home pages of 184 association websites, scanned October 8, 2026.
What we measured | Sites |
|---|---|
Association websites scanned | 184 |
Blocked at least one AI crawler | 44 |
Of those 44, blocked only by a firewall or challenge page, with no robots.txt rule | 31 |
A visitor cannot read a firewall rule, and we cannot see who set it or why. Read the 31 as sites with a rule that fires on a crawler’s name, not as proof that ChatGPT is locked out, because a request that only borrows a crawler’s name can be challenged when the real crawler would be let in. The rule is not in a file your staff can open, so it helps to name one person who owns the question.
What is the difference between blocking AI training and blocking AI search?
A training crawler collects pages so a company can train its models, and an AI search crawler fetches pages so an answer can cite them. Blocking one does not block the other.
OpenAI’s bot documentation says GPTBot crawls content that may be used in training its foundation models, that each bot setting is independent, and that a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot. It also says that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic’s crawler page draws the same line for its own crawlers.
Keep your member-only library behind the login, since content there is outside what a public crawler can read. The thing to check is your public pages, the ones a prospective member or a journalist would land on first.
What does this mean for member relationships?
If an answer reaches your member before your site does, crawler settings are one of the things that can affect whether it has your words to work from. Pew Research Center tracked 900 U.S. adults in March 2025, the same study RallyBoard’s post draws on. Users clicked a traditional result link in 8% of visits when Google showed an AI summary, against 15% when it did not. That is a March 2025 figure about Google users in general, not about associations.
RallyBoard’s point is that the advantage was never in owning the place members log into but in owning the relationships that happen there. Our scan covers only the first step, whether a public page can be read at all. What happens once the answer reaches a member, the relationship, is the part RallyBoard writes about.
“In our scan of 184 association sites, from one IP address, 44 blocked an AI crawler, and 31 of those had no robots.txt rule against any. As far as that address can show, the block sat in a firewall or challenge setting, which you cannot read in a browser. Ask whoever runs your hosting which AI crawlers it lets through.”
Alden Do Rosario
What does this scan not show?
We fetched each home page once from one IP address, first as a normal browser and then with each crawler’s name, so a real crawler may be treated differently, and we did not look at inner pages. The 184 sites are our own sample from two lists of organizations we treat as associations, scanned on October 8, 2026, so read the numbers as what we found and not as a count for all associations.
How can I check whether my association’s website blocks AI crawlers?
You can start this quarter with three steps.
Read your robots.txt. Open yourdomain.org/robots.txt in a browser. A pair of lines such as “User-agent: OAI-SearchBot” followed by “Disallow: /“ asks that crawler, the one behind ChatGPT search, to stay out of the whole site. If you find that pair and want ChatGPT search to cite your public pages, ask whoever manages the file to remove it. If you find no such lines, that does not settle it, because a firewall can block a crawler with nothing written in robots.txt. Step 2 covers that.
Ask your host, in these words. Send this to whoever runs your hosting or security: “Does our bot protection challenge or block OAI-SearchBot, PerplexityBot or ChatGPT-User, and where is that setting?” Keep the answer with your site documentation.
Decide training and search separately, and name an owner. The settings are independent. If you want ChatGPT search to be able to cite your public pages, ask your host to allow OAI-SearchBot, and treat GPTBot as its own decision. Record the decision, the reason and the one person who owns it.
Steps 1 and 2 tell you whether your site has a rule that fires on a crawler’s name.
Frequently asked questions
How many membership associations block ChatGPT and other AI crawlers?
In our October 8, 2026 scan of 184 association websites, 44 blocked at least one AI crawler, and for 31 of those 44 the only block was a firewall or challenge rule, with no robots.txt line. That is our own sample, not a count for all associations.
Should an association block AI crawlers with robots.txt?
There is no blanket answer. Decide training and search separately, because OpenAI treats each bot setting as independent. Remember that robots.txt is a request, so anything truly private belongs behind a login. Whatever you choose, name an owner and record the reason.
How can I check whether my association’s website blocks AI crawlers?
Open yourdomain.org/robots.txt, then ask whoever runs your hosting, in writing, whether bot protection challenges AI search crawlers and where that setting is. Keep the reply with your site documentation, and repeat the check after any change of host or security service.
Alden Do Rosario is the founder and CEO of CustomGPT.ai, where his team ran the scan described here.
