field note

The four lines that decide whether Google can show your page

A page can pass every content and speed check and still be barred from search by one line nobody reads.

Voltic · published

Most site audits grade the things you can see. Titles, headings, image alt text, how fast the page paints, whether the structured data parses. All of it useful, all of it downstream of a question almost nobody checks first: is this page allowed to appear in a search result at all?

Four lines decide that. Three of them live somewhere you would never look while reading the page, and each fails silently — no error, no warning, no visible symptom. The page loads perfectly. It just cannot be shown.

1. The robots meta tag

<meta name="robots" content="noindex">

Google's definition of the noindex rule is nine words long: "Do not show this page, media, or resource in search results."

There is a second spelling that catches people out. content="none" is documented as "Equivalent to noindex, nofollow" — same effect, different word, and invisible to anyone searching their templates for the string "noindex".

A third variant hides in plain sight: the tag can be addressed to a single crawler rather than to everyone. <meta name="googlebot" content="noindex"> is not a robots tag and will not match a search for one, but it governs the crawler that most sites care about most.

This is almost never typed deliberately on a live page. It arrives one of three ways: a staging site copied to production with its settings intact, a CMS with a "discourage search engines" checkbox left ticked after launch, or a page template written for something that genuinely should be hidden and then reused for something that should not.

2. The X-Robots-Tag header

X-Robots-Tag: noindex

Same instruction, sent by the server instead of the document. Google is explicit that the two are interchangeable: "Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag."

This is the one that survives redesigns. It is set in server config, CDN rules, or a framework's response middleware — none of which get reviewed when the design changes, and none of which appear in View Source. You can read every byte of a page's HTML, find nothing wrong, and still be looking at a page that is barred from search.

If you check one thing on this list by hand, check this one, because it is the only one you cannot see by reading the page.

3. Where the canonical points

<link rel="canonical" href="https://someone-elses-site.example/page">

A canonical tag names the address you want a page indexed under. Most audits check that one exists. Far fewer check where it points, and the difference matters, because a canonical aimed at another page — or another domain — asks search engines to credit that other address instead of this one.

Worth being precise here: Google describes rel=canonical as "a strong signal that the specified URL should become canonical", not as a command. Unlike noindex, it is not guaranteed to be obeyed. But a strong signal pointed at somebody else's domain is still a strong signal pointed at somebody else's domain, and it looks identical to a correct tag unless you read the target.

The usual cause is mechanical. A template hard-codes one URL and every page inherits it. A migration copies canonicals from the old domain. A syndication plugin sets the source as canonical on content the site actually wrote itself.

4. Whether the page is in your own sitemap

Not fatal, and the weakest of the four — a page left off a sitemap can still be found through links. But the sitemap is the list you hand crawlers of what you want found, and a page missing from it is discovered later and revisited less often.

While you are in there: Google states plainly that it ignores the priority and changefreq fields. lastmod is read, and only "if it's consistently and verifiably accurate". A sitemap that stamps every page with today's date each time anything deploys is not providing a freshness signal — it is teaching Google that the field means nothing on your domain.

The trap that hides all of this

The four checks above assume a crawler can reach the page. There is a failure mode where it cannot, and it is the most common self-inflicted indexing problem there is.

Google: "these settings can be read and followed only if crawlers are allowed to access the pages that include these settings." And: "If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored."

So a page that is both disallowed in robots.txt and carries a noindex gets the worst of both. The crawler never fetches the page, so it never sees the noindex — and the URL can still surface in results, without a description, because Google knows the address exists from links pointing at it. The two controls do opposite jobs, and using them together makes the second one unreachable.

If you want a page out of search, one of them has to be able to reach it.

How to check yours in five minutes

  1. View source on the page and search the head for name="robots" and name="googlebot". Read the content value — not just whether the tag exists.
  2. Open your browser's developer tools, reload the page, click the document request, and read the response headers for X-Robots-Tag.
  3. Find the <link rel="canonical"> and read the URL it contains. It should be this page's own address.
  4. Open your sitemap and search it for this page's URL.

Then check the same four on a page you have never inspected — a category page, a landing page from a campaign, anything built from a template rather than by hand. The pages people check are rarely the pages that are broken.

What we changed on our side

Voltic's Deep Scan graded pages on content, speed, accessibility and structured data without reading any of the four. As of today it reads all four, and reports them the way this piece frames them: as indexability, never as index membership, and never with instructions to remove a control that may have been set on purpose.

The checks cost nothing extra to run — every one of them reads bytes the scan already fetched.

Sources

  1. Robots meta tag, data-nosnippet, and X-Robots-Tag specifications — Google Search Central, read 2026-08-01
  2. How to specify a canonical URL with rel="canonical" and other methods — Google Search Central, read 2026-08-01
  3. Build and submit a sitemap — Google Search Central, read 2026-08-01

Run a free Instant Scan on your own site · More from the Newsroom