field note

The four lines that decide whether Google can show your page

Four directives decide whether a page can appear in search. Three of them are invisible while you read the page.

Go Voltic · published

Most audits grade what you can see. Titles, headings, alt text, how fast the page paints, whether the structured data parses. All useful. All downstream of a question that usually goes unasked: can this page appear in a search result at all?

Four things decide that. Three are invisible while you read the page, and none of them produce an error. The page loads. It just cannot be shown.

One meta tag takes the page out of search

<meta name="robots" content="noindex">

Google's definition of noindex: "Do not show this page, media, or resource in search results."

Two variants get missed. content="none" is documented as "Equivalent to noindex, nofollow", so searching your templates for the string "noindex" returns nothing. And the tag can name one crawler instead of all of them. <meta name="googlebot" content="noindex"> is not a robots tag, and it governs the crawler most sites care about.

Nobody types this on a live page on purpose. It arrives three ways: a staging site copied to production with its settings intact, a CMS with a "discourage search engines" box still ticked after launch, or a template written for a page that should be hidden and then reused for one that should not.

The same rule arrives as a header you cannot see

X-Robots-Tag: noindex

Same instruction, sent by the server rather than the document. Google: "Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag."

It lives in server config, CDN rules, or response middleware. None of those get reviewed during a redesign. None of them appear in View Source. You can read every byte of the HTML, find nothing wrong, and still be looking at a page barred from search.

Check this one by hand if you check nothing else. It is the only one you cannot see by reading the page.

A canonical pointing elsewhere hands the page to someone else

<link rel="canonical" href="https://someone-elses-site.example/page">

A canonical names the address you want the page indexed under. Most audits check that one exists. Fewer check where it points.

Google calls rel=canonical "a strong signal that the specified URL should become canonical." Not a command. Unlike noindex, it can be ignored. But a strong signal aimed at a domain you do not own is still aimed at a domain you do not own, and it looks identical to a correct tag until you read the target.

The cause is usually mechanical. A template hard-codes one URL and every page inherits it. A migration carries canonicals over from the old domain. A syndication plugin marks the source as canonical on content the site wrote itself.

A page missing from your sitemap is found later and checked less

This is the weakest of the four. A page left off a sitemap can still be discovered through links. But the sitemap is the list you hand crawlers of what you want found, and a page that is not on it gets crawled later and revisited less often.

While you are in there: Google ignores priority and changefreq. It reads lastmod, and only "if it's consistently and verifiably accurate." A sitemap that stamps every page with today's date on every deploy is not sending a freshness signal. It is teaching Google that the field means nothing on your domain.

Blocking a page in robots.txt makes its noindex unreachable

All four checks assume a crawler can reach the page. When it cannot, the most common self-inflicted indexing problem follows.

Google: "these settings can be read and followed only if crawlers are allowed to access the pages that include these settings." And: "If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored."

A page that is both disallowed in robots.txt and carries a noindex gets neither. The crawler never fetches the page, so it never reads the noindex. The URL can still surface in results without a description, because Google knows the address exists from links pointing at it.

If you want a page out of search, one of the two controls has to be able to reach it.

Check yours in five minutes

  1. View source. Search the head for name="robots" and name="googlebot". Read the content value, not just whether the tag is there.
  2. Open developer tools, reload the page, click the document request, and read the response headers for X-Robots-Tag.
  3. Find <link rel="canonical"> and read the URL it contains. It should be this page's own address.
  4. Open your sitemap and search it for this page's URL.

Then run the same four on a page you have never inspected. A category page, a campaign landing page, anything built from a template. The pages people check are rarely the pages that break.

What we changed

Go Voltic's Deep Scan graded pages on content, speed, accessibility and structured data without reading any of the four. It now reads all four. It reports them as indexability rather than index membership, and it never tells anyone to remove a control that may have been set on purpose.

Each check reads bytes the scan already fetched, so the section costs nothing to run.

Sources

  1. Robots meta tag, data-nosnippet, and X-Robots-Tag specifications. Google Search Central, read 2026-08-01
  2. How to specify a canonical URL with rel="canonical" and other methods. Google Search Central, read 2026-08-01
  3. Build and submit a sitemap. Google Search Central, read 2026-08-01

Run a free Instant Scan on your own site · More from the Newsroom