signal

A 41-day test found AI crawlers never follow a JavaScript-only link

A 41-day crawler study found GPTBot, ClaudeBot and six other AI crawlers never followed a single JavaScript-only link. Googlebot barely did either.

Go Voltic · published

If your site's main navigation runs on JavaScript, some of the AI systems that answer questions using the web may never reach the pages it points to. Not more slowly. Not partially. Never. A crawler that cannot execute a script cannot follow a link that only exists after the script runs, and a new test of ten named crawlers found that eight of them, including OpenAI's and Anthropic's, are exactly that kind of crawler.

Eight of ten crawlers tested never executed a single script

Vinicius Stanula, associate director of SEO at LOCOMOTIVE, ran the test on a business classification directory he owns, publishing the results in Search Engine Land on August 19. The site had 21 top-level sections. Eleven had their child links hard-coded into the HTML, 748 pages in total. Ten had their child links injected by JavaScript after the page loaded, 293 pages in total. Every other way a crawler might have found those pages, an XML sitemap, breadcrumbs, an on-page hierarchy panel, was deliberately switched off. Discovery required running the script or nothing.

For 41 days, middleware logged every bot request by path, user agent, and IP, with Google's requests confirmed by reverse DNS. Eight of the ten crawlers logged never executed the JavaScript, on any day, in any of the 41: GPTBot, ClaudeBot, Bingbot, OAI-SearchBot (ChatGPT's search crawler), Meta-ExternalAgent, Amazonbot, ChatGPT-User, and PerplexityBot.

GPTBot and ClaudeBot each requested every one of the 748 HTML-linked pages and none of the 293 JavaScript-linked ones. Amazonbot reached 735 of the HTML pages and zero JavaScript ones. Meta's crawler reached 537 and zero. Bingbot reached 282 HTML pages and exactly 3 JavaScript ones, close enough to read as noise rather than real rendering. PerplexityBot stayed on the homepage and robots.txt for the entire first phase of the test and reached its first hierarchy page on day 40, after Stanula had already converted the JavaScript sections to HTML.

Google's own search crawler barely benefited from the JavaScript either

Two crawlers in the test did execute JavaScript: Googlebot and GoogleOther. Their results are easy to read as the same finding and are not. Googlebot reached 5% of the HTML-linked pages and 2% of the JavaScript-linked ones. GoogleOther reached 66% and 48%.

The gap matters because the two crawlers do different jobs. Google's own documentation describes GoogleOther as "the generic crawler that may be used by various product teams for fetching publicly accessible content from sites," including "one-off crawls for internal research and development," not a signal that feeds the search index. Googlebot is the one whose crawl decides what can appear in Search. Stanula's own framing of the split: "The row that speaks to search visibility is Googlebot: 5% of the HTML group and 2% of the JS group." A generous-looking GoogleOther number says nothing about whether a page can be found in Google.

Even GoogleOther, the crawler in the test that does run scripts, paid a speed penalty for it. Stanula found its JavaScript-linked pages were "discovered 18% to 27% later than its HTML-linked pages at every depth," and explained why: "Google must fetch the parent, queue it for rendering, execute the script, extract the URLs, and then schedule the children. HTML links skip the middle steps." That sequence matches how Google's own documentation describes the process: Googlebot crawls the raw HTML first, then queues the page for a separate rendering pass where the JavaScript runs, and that render queue "may stay on this queue for a few seconds, but it can take longer than that." A 41-day test catches whatever fraction of that queue clears in 41 days. It does not catch the rest.

After Phase 1, Stanula converted the JavaScript sections to hard-coded HTML links and kept logging. The crawlers did not respond the same way to the same fix.

GPTBot found the change within two days: it "swept every formerly blocked section in a single sweep lasting under three hours and picked up 250 pages it had never crawled before," against zero new pages in the 27 days it had spent failing to find them under JavaScript. ClaudeBot did not. It "kept recrawling the site after the fix, hundreds of hits through day 41, but only on pages it already knew. It never picked up a single newly linked page." Converting the links restored discovery for one crawler and not, within the window measured, for the other. Google was slower than either: "By day 41, Googlebot had visited exactly one page in the converted sections, and GoogleOther had visited none."

Stanula is direct about the limits of his own result: "This is one site, one niche, one 41-day window, and a fresh domain with no external links pointing at the buried pages. Established sites with strong signals may see Googlebot invest more in rendering." A site with an existing crawl history and inbound links is not the same test subject as a brand-new directory with every fallback switched off on purpose. The finding is a mechanism, confirmed directly for the first time for AI crawlers rather than assumed: eight named crawlers do not run JavaScript, full stop. How fast a given site recovers once its links are fixed will vary with what that site already had going for it.

What to check on your own site

  1. Load your key pages with JavaScript disabled in the browser, or view the page source rather than the rendered DOM, and look for your internal links: navigation, pagination, category filters, "load more" buttons. If a link is not sitting in that raw HTML as a real <a href>, a crawler that does not run scripts will not see it, no matter how normal the page looks once your browser finishes rendering it.
  2. Keep a working XML sitemap and submit it. Stanula's test disabled every fallback on purpose to isolate the JavaScript effect; a real site with a current sitemap gives non-rendering crawlers a second way in that this test deliberately removed.
  3. Do not assume a fix is universal once it ships. Different crawlers picked up the same corrected links on different schedules, and one of the ten had not picked up a single new page by day 41. Recheck server logs for the specific bots you care about after a fix, rather than trusting that one crawler's fast response means the rest followed.
  4. Treat "renders JavaScript" and "counts for search visibility" as two separate questions even inside Google's own crawling. A page reached by GoogleOther is not confirmed reachable by Googlebot, and Googlebot is the one that matters for ranking.

Go Voltic's own scans check whether a page's primary content and internal links are present in the HTML a server returns, independent of what a browser does after the fact. This test is a reason that check exists, not a substitute for reading it yourself on your own site.

Sources

  1. JavaScript links can make your pages invisible to AI search. Search Engine Land, read 2026-08-20
  2. Understand the JavaScript SEO basics. Google Search Central, read 2026-08-20
  3. List of Google's common crawlers. Google Search Central, read 2026-08-20
  4. GPTBot. OpenAI, read 2026-08-20

Run a free Instant Scan on your own site · More from the Newsroom