Site AI Auditby Internet Solutions

Pages That Did Not Respond: Timeouts in Your Site AI Audit

21 November 20268 mnt bacaDasar-dasar SEO
Pages That Did Not Respond: Timeouts in Your Site AI Audit

Short answer: “Pages that did not respond” is a warning in the SEO area of a Site AI Audit report. It lists pages for which the crawler got no HTTP answer at all: the connection timed out, was refused or was cut off. It is different from a broken page, which answers with an error code, and from a slow page, which answers late. The usual causes are an overloaded server, a firewall or security plugin that blocks the crawler and pages that crash before they can send anything. The advice in the report is to check server stability and slow pages.

Three kinds of trouble, three findings

Site AI Audit separates pages that go wrong in different ways, because the causes and the fixes differ:

The third kind is the most puzzling for site owners, because the page may open fine in a browser a minute later. The report is honest about it: during the check, something stopped the answer from reaching the crawler.

What counts as no response

The crawler asks for each page and waits. If the server sends nothing for 15 seconds, or the whole transfer takes more than 25 seconds, the request is given up. Other failures are caught as well: the connection is refused, the connection is reset in the middle of the transfer, the name cannot be resolved or the secure handshake fails for that address. In all of these cases, the audit has no status code to record, so the page is listed as one that did not respond.

The finding is a warning, not a critical item, because a few such pages may be a short hiccup. But the report keeps the addresses, and it is worth looking at them, since a page that cannot be fetched cannot be indexed either, and visitors may have had the same experience.

Common causes

Step 1: look at the list and the pattern

Open the finding in the report. Do not stop at the number; read the addresses. A pattern usually appears:

  1. All from one section. For example, only addresses under /shop/ or /search/. The cause is probably in that part of the application.
  2. Only late in the crawl. If the first pages answered and the later ones did not, a rate limit or a block is likely.
  3. Only heavy pages. Reports, exports and big archives suggest timeouts due to the work they do.
  4. Scattered with no pattern. That suggests general load or short interruptions.

Step 2: try the pages yourself

Open several of the listed addresses in a browser. If they load, time how long it takes. A page that needs ten seconds on your computer is a candidate for the crawler’s limit. If they load fast, the problem was temporary or related to the crawler specifically. If they fail for you too, you have found a real fault: look at the error message and the server log.

Step 3: check the server logs

Your hosting panel or your host can show the access log and the error log. Look at the time of the audit, which is shown in the report. Three things to look for are lines for the listed addresses with a long response time, errors about memory, timeouts or process limits, and requests from the crawler that were refused. The crawler introduces itself as SiteAuditBot, a name that you can search for. The article on what SiteAuditBot does on your site describes its behaviour and its limits.

Step 4: make sure the crawler is not blocked

If the log or the pattern suggests blocking, check what protects your site. Security plugins, web application firewalls, hosting rate limits and bot protection services can all drop connections from automated visitors. Allow the crawler by name or by its behaviour, or raise the rate limit a little. Do this carefully: you want to let your own audit in without opening the door to scrapers. If the crawler is blocked by design, for example on a staging site, the audit will not be able to read it. The article on robots.txt and sitemap findings explains the rules that the crawler follows.

Step 5: fix the real cause

Then run the audit again. If the finding stays, compare the list with the first one. Pages that still fail are the real problems, and pages that disappeared from the list were short interruptions.

Prevent it from coming back

Once the cause is fixed, keep an eye on it. Server trouble tends to return when traffic grows, when a plugin is added or when a backup job moves to a busy hour. Three habits help. Run an audit after any larger change to the site or the hosting plan. Ask your host to alert you when the server reaches its process or memory limits. And when you add an interactive feature, such as a calendar, a search or a filter with many combinations, test how a single request behaves under load before you link to it from every page. A feature that needs twenty seconds to build a result is a risk for visitors as well as for the audit.

If the same pages fail again after a fix, do not assume that the crawler is at fault. Compare the time of the check with the server log, and look at what else was running at that time. Scheduled tasks, imports and backups are common hidden causes.

Why it matters beyond the audit

Search engine crawlers fail in the same way. If Googlebot meets timeouts, it slows down and visits less often, and pages it cannot fetch may drop out of the index. Visitors meet the same wall and do not come back. The Search Console reports on server errors, discussed in the article on 5xx errors in Search Console, are the other side of the same problem. A clean result in the audit is a good sign for both.

A worked example

A cleaning company with a booking calendar runs on shared hosting. The audit shows a warning: 23 pages did not respond. All of them are under /calendar/ and have long parameter strings. The owner opens three of them and sees that each takes about 20 seconds, because the calendar plugin builds a month view from the database each time. The host’s log confirms the process limit being reached. The agency adds caching for the calendar views and removes the links that generate endless date combinations. A new audit lists no pages without a response, and the calendar loads in two seconds.

To see whether your own pages answer, run a Site AI Audit with a free account. The plans unlock the full report with the list of affected pages.

Related reading

The bottom line

Pages that did not respond gave the crawler no answer at all. Look for the pattern, try the pages yourself, read the logs, make sure nothing blocks the crawler and fix the real cause, then run the audit again.

FAQ

How long does the crawler wait for a page?

It gives up when nothing arrives for 15 seconds, or when the whole transfer takes longer than 25 seconds.

Is a page that did not respond the same as a 404?

No. A 404 is an answer that says the page does not exist. A page that did not respond gave no answer at all.

Can the problem be on the crawler’s side?

Rarely. Short network problems happen, so run the audit again. If the same pages fail twice, the cause is on your side.

Could my firewall be the cause?

Yes. Firewalls and security plugins sometimes drop connections from automated visitors after a number of requests. The log will show it.

Should I worry about one or two pages?

Check them, but a single failure may be a short hiccup. A pattern or a growing number is the real signal.

#Crawling#Troubleshooting#Web hosting
Audit situs Anda sendiri.Apakah situs web Anda terlindungi dari peretasan?
Audit website

Lainnya dari blog

Semua artikel →
Internet Solutions

Lainnya dari tim kami

Dibuat oleh Internet Solutions. Coba produk kami yang lain — masing-masing menghemat waktu Anda dengan cara berbeda.

internet-solutions.net ↗