Guide
Is my website invisible to AI?
Figures measured across 5,293 scanned sites, window ending 2026-08-20
Buyers now ask assistants questions they used to type into Google, and an assistant can only recommend what it can read. Whether your site is readable is not a mystery or a vibe. It is three mechanical questions, and you can answer two of them yourself in under two minutes.
The numbers say the problem is not where people think
The common fear is being blocked: a robots.txt file that turns AI crawlers away. Across 5,293 sites we scanned in the 30 days ending 2026-08-20, that fear is almost never the reality. Deliberate AI-crawler blocking shows up on 0.9% of sites. Meanwhile 77% of sites serve pages whose text barely exists until scripts run. Most AI crawlers do not run scripts. They fetch your page source, read what is there, and move on. The lock on the door is rare. The empty room behind the door is the norm.
| Finding | Share of sites | What it means |
|---|---|---|
| AI crawlers blocked in robots.txt | 0.9%95% CI 0.7% to 1.2% | The robots file explicitly turns away AI assistants’ crawlers. |
| Content only appears after scripts run | 77%95% CI 75.8% to 78.1% | The page source carries little readable text, so crawlers that do not run scripts see almost nothing. |
| No structured data | 41.5%95% CI 40.2% to 42.8% | Nothing on the page tells machines what the business is in machine terms. |
| No plain summary for machines to lift | 29.7%95% CI 28.5% to 30.9% | Title or description missing, so an assistant has no clean answer to quote. |
| Page regions not marked semantically | 52.4%95% CI 51% to 53.7% | No header, nav, main or footer landmarks, so machines parse a soup of divs. |
Share of 5,293 scanned sites raising each finding. Intervals are 95% Wilson intervals. Full tables: the benchmarks page.
Check one: what does your page source say?
Open your home page in a browser, view the page source, and search for a sentence you know appears on the page. If it is not in the source, a crawler that does not run scripts cannot read it, and to that crawler your site is close to empty. This one check catches the failure that 77% of scanned sites have. Our own 41-day crawler test found the same thing from the other side: AI crawlers never followed a JavaScript-only link.
Check two: does your robots.txt turn assistants away?
Visit your site's /robots.txt address and look for user-agent lines naming AI crawlers with a disallow underneath. Rare, but it happens: a rule copied from a template, or a firewall preset someone enabled years ago. If you block them, nothing they power can mention you.
Check three: can a machine lift an answer?
This is the one you cannot eyeball. An assistant that reaches your readable page still needs a plain statement of what you do, where, and for whom, plus structured data that says it in machine terms. In our population, 41.5% of sites carry no structured data at all and 29.7% give a machine no clean summary to quote. The free scan runs these checks, along with the two above, and shows you exactly which ones your site raises.
What does not fix invisibility
Publishing an llms.txt file, buying a listing on an AI directory, or scoring well on a readiness checklist. Those make you easier to parse once an assistant already wants to mention you. They do not create the want. What creates it is readable pages, a clear entity, and independent evidence, in that order. The longer version of that argument is in why assistants skip a business.
Questions we get
Does blocking AI crawlers protect my content? It keeps compliant crawlers out, which also keeps you out of the answers those systems give. It does not stop non-compliant scrapers. That trade is a business decision, but it should be a decision, not a template accident.
My site is built on a JavaScript framework. Am I stuck? No. Frameworks can render pages on the server so the text exists in the source. Whether yours does is exactly what the view-source check shows.
How do I know if it is working? Ask the assistants your buyers use the question your buyers ask, more than once, over time. One answer is a coin flip, not a measurement.