Robots.txt Tester: the question this tool answers
Fetch robots.txt and detect accidental sitewide blocking for common search and AI user agents. The live result starts with robots.txt status, then uses User-agent groups and Disallow / to confirm or challenge the first interpretation. A healthy result here is intentionally scoped to crawler accessibility and canonical signals; it does not claim that unrelated parts of the domain are healthy.
- robots.txt status
- User-agent groups
- Disallow /
- sitemap references
- Googlebot
- common AI crawlers
How to read the robots.txt status, User-agent groups and Disallow / evidence
Start with the value that directly controls the failing service, then use the other signals to confirm or challenge that interpretation. A single record or response can look correct while another hostname, IP family, redirect or mail-authentication path still fails. The tool therefore keeps the raw observed value visible and explains how the signals relate.
- Robots.txt status — Record the public status, final URL and relevant response headers. A working DNS answer does not prove that the application is serving a genuine website response.
- User-agent groups — Verify User-agent groups from the public Internet and compare it with the value the responsible provider says should be live.
- Disallow / — Verify Disallow / from the public Internet and compare it with the value the responsible provider says should be live.
- Sitemap references — Fetch the directive exactly as a crawler would and check both file-level rules and page/header directives; one healthy signal does not override a blocking signal elsewhere.
- Googlebot — Verify Googlebot from the public Internet and compare it with the value the responsible provider says should be live.
- Common AI crawlers — Verify common AI crawlers from the public Internet and compare it with the value the responsible provider says should be live.
Common robots.txt tester failure patterns
The most useful warnings are mismatches rather than isolated missing fields. Robots.txt status can be absent or stale; User-agent groups can point somewhere different from the expected provider; and Disallow / can reveal a second path that behaves differently. If public sources disagree, first decide whether the difference comes from authoritative data, caching, a separate hostname, IPv6, or another protocol layer before editing configuration.
What to change — and what not to change
Change the smallest setting that the evidence proves is wrong. Compare the observed robots.txt status with the current value supplied by the responsible provider, save the existing value, make one correction, and retest. Do not replace nameservers, delete unrelated MX/TXT records, disable TLS controls or remove IPv6 simply because Robots.txt Tester shows a warning in another layer. Broad changes can turn one isolated problem into several independent outages.
How HostWithShery runs Robots.txt Tester
The submitted public input is normalized and validated before any server-side request is made. HostWithShery then obtains robots.txt status and User-agent groups using the appropriate public DNS, HTTP, TLS or mail protocol and adds Disallow / plus sitemap references where it materially changes the interpretation. URL-based checks are restricted to public destinations, and redirect targets are revalidated instead of being trusted automatically.
How to interpret conflicting robots.txt tester signals
When robots.txt status looks correct but User-agent groups or Disallow / disagrees, do not treat the whole domain as broken. Isolate the public path that differs, identify whether it belongs to crawler accessibility and canonical signals, and verify that path against the authoritative provider value before making a change. HostWithShery keeps these signals separate so a healthy value in one layer cannot mask a failure in another.
When a robots.txt tester result is inconclusive
This is a public-configuration diagnostic. Network filtering, private control-panel settings, split-horizon DNS, provider-internal state and transient routing conditions can limit what can be confirmed from outside the account. Public diagnostics cannot see private control-panel state, unpublished origin addresses, provider account status or split-horizon/internal DNS. If the remote service times out, blocks automated requests or hides a signal, the result stays inconclusive instead of inventing a provider or configuration. That distinction matters when the next step is a potentially disruptive DNS, SSL or mail change.
How to verify the repair
The crawler-facing resource should return successfully and the blocking robots/noindex/sitemap condition that caused the warning should be absent. Repeat Robots.txt Tester after the change and compare the same robots.txt status, User-agent groups and Disallow / evidence that exposed the problem. A provider dashboard saying “saved” is not enough when the public Internet still returns the old value.
What to check next if Robots.txt Tester is healthy
If robots.txt status, User-agent groups and Disallow / now agree but the user-facing problem remains, move to the adjacent layer instead of editing the same setting repeatedly. Use the related diagnostics on this page to test the next plausible cause, and keep the previous result as evidence when escalating to a hosting, DNS or email provider.
Technical references
These primary standards and provider documents are used to verify the behavior described on this page. Provider dashboards can change, so use the current official value for tenant-specific DNS records rather than copying an example from another account.