Site AI AuditInternet Solutionsilt

Noindex Findings: Pages Hidden From Google by Mistake

16. november 20268 min lugemistSEO alused
Noindex Findings: Pages Hidden From Google by Mistake

Short answer: A noindex instruction tells search engines to leave a page out of their results. Site AI Audit reads it on every page it crawls, from the robots meta tag and from the X-Robots-Tag header, and reports two findings. If the home page is hidden, the finding is critical. If other pages are hidden, the finding is a warning that asks you to check whether that is intentional. In both cases the report lists the pages, so you can remove the tag from those that should be found.

What noindex does

Search engines work in two stages. First they crawl a page, which means fetching and reading it. Then they decide whether to index it, which means storing it so that it can appear in results. A noindex instruction, placed in a meta tag in the page or in an HTTP header, tells them to skip the second stage. The page can be visited and read, but it will not be shown to people who search.

That is useful for pages that should exist but should not be found through search: thank-you pages after a purchase, internal search results, login pages, printable versions and duplicates. It is a disaster for pages that you want customers to find. Because the tag is invisible to visitors, a page can be hidden for months without anyone noticing, until the traffic to it quietly falls to nothing.

How the audit finds noindex

As the crawler visits each page, it reads two places. The first is the robots meta tag in the head of the page, for example one that says “noindex, follow”. The second is the X-Robots-Tag header that the server sends with the response, which is the only way to put noindex on files that are not web pages, such as PDFs. If either place contains noindex, the page is counted as hidden. The article on the X-Robots-Tag header explains when headers are used.

Note that this is separate from robots.txt. A robots.txt rule stops crawling, whereas noindex needs the crawler to read the page in order to obey it. The audit has its own findings for robots.txt, described in the article on robots.txt and sitemap findings. The noindex findings are about the instruction inside the page.

The two findings

If the home page is hidden, the report shows the first finding and not the second, because the home page is the more urgent case. Fix it, run the check again and the second finding may appear with a list of other pages.

Where accidental noindex comes from

Almost no one adds noindex to the home page on purpose. It comes in through settings and habits:

Pay attention to the pattern in the list of affected pages. If it is all products, look at the plugin setting for products. If it is a handful of pages, look at each page’s SEO box.

When noindex is the right choice

Not every hidden page is a mistake. Good uses of noindex include thank-you and confirmation pages, internal search results, filtered or sorted lists that duplicate a main list, login and account pages, and very thin pages that have no value in search. For tags, categories and archives, the decision depends on whether they have real content; the article on category and tag pages in SEO helps you decide. The audit’s warning says “check that this is intentional”, and that is the right way to read it: go through the list, mark the pages that should be hidden and fix the rest.

How to remove noindex

  1. Find where it comes from. Open the page source and search for “noindex” in the head. If you find nothing, check the response headers in the browser’s developer tools for X-Robots-Tag.
  2. Remove it at the source. That might be the per-page box in the SEO plugin, a site-wide setting, a theme option or a server rule.
  3. Clear caches. Page caches and CDNs can keep serving the old head.
  4. Request a re-crawl. In Search Console, inspect the home page and ask for indexing. For other pages, the sitemap helps.
  5. Run the audit again. The finding should be gone.

The guide on the noindex tag covers the same steps with more examples, and the one on WordPress SEO settings to check on every site shows where WordPress hides the relevant switches.

How long until pages return

Removing noindex does not restore traffic at once. Search engines must crawl the page again, notice the change and decide to index it. For a busy site this can take days, for a small one weeks. A page that has been hidden for a long time may also need a while to regain its position. Be patient, and check the number of indexed pages in Search Console. If pages stay out of the index long after the tag is gone, the issue may be something else, as explained in crawled or discovered but currently not indexed.

Noindex and your sitemap should agree

A common contradiction is a page that is listed in the sitemap but carries a noindex tag. The sitemap says “please index this” and the page says “do not”. Search engines usually obey the page, but the mixed signal is a sign of a configuration problem, and Search Console reports it as a warning. When you review the list of hidden pages in the audit, compare it with your sitemap. Any page that should be in search belongs in the sitemap without noindex, and any page with noindex does not belong in the sitemap at all.

Check again after updates

Noindex has a habit of returning. A plugin update can reset a setting, a theme change can introduce a default and a developer can copy a staging configuration to the live site during a deployment. A quick check after every release is a cheap habit: the critical home page finding is exactly the kind that appears right after a change and costs the most if it goes unnoticed for weeks.

Who should look at the list

The warning for hidden pages is only useful if somebody who knows the site reads it. The person who built the site may remember why a page was hidden, and the person who writes content knows which pages should earn traffic. Send them the list of addresses and ask one question for each: should a stranger be able to find this page through Google? Their answers turn the report into a short, clear task list.

A worked example

A physiotherapy clinic launches a new site in September. In November, the owner notices that the clinic no longer appears when she searches her own name. She runs a Site AI Audit check and gets a critical finding: the home page tells search engines not to index it. The agency had enabled the “discourage search engines” option during development and forgotten to switch it off. She asks them to turn it off, the agency confirms, and a new check clears the critical finding. A warning remains for four pages, which turn out to be old test pages that the owner is happy to keep hidden. Search Console shows the home page indexed again within ten days.

To see whether any of your pages are hidden, check your website with Site AI Audit. The check needs a free account; the plans unlock the full report with the list of affected pages.

Related reading

The bottom line

A noindex tag on the home page is one of the costliest invisible mistakes on a website. The audit reads the tag and the header on every page, reports the home page as critical and lists other hidden pages. Remove the tag where it does not belong and run the check again.

KKK

Why is the home page case critical?

A noindex tag removes the page from Google results. Because the home page usually carries the most weight, hiding it harms the whole site.

Does noindex also stop crawling?

No. The crawler still visits the page and reads the instruction. If you want to stop crawling, that is a robots.txt matter.

Can noindex be in a header instead of the page?

Yes. The X-Robots-Tag header does the same job and works for files such as PDFs. The audit reads both places.

Should my thank-you pages be noindex?

Usually yes. Pages that exist only to confirm an action have no value in search results.

How do I know the problem is fixed?

Run the check again. The finding should disappear, and Search Console should show the pages as indexed after a re-crawl.

#Indexing#Technical SEO#Website audit
Kontrolli oma veebisaiti.Kas teie veebileht on häkkimise eest kaitstud?
Kontrolli saiti

Veel blogist

Kõik artiklid →
Internet Solutions

Veel meie meeskonnalt

Loonud Internet Solutions. Proovige ka meie teisi tooteid — iga üks säästab aega omal moel.

internet-solutions.net ↗