Does a robots.txt Disallow rule remove a page from Google?

No. Robots.txt manages crawling, not guaranteed indexing. A blocked URL may still be known from links, and sensitive content needs access controls rather than crawler directives.

How does the tester choose between matching rules?

It applies local RFC-style matching for the selected user agent and reports the Allow or Disallow pattern that determined each result. The most specific matching user-agent group is used.

Can I add more than one Sitemap directive?

Yes. Each entry must be an absolute HTTP or HTTPS URL. The optional Website URL only helps suggest a conventional sitemap address.

Does the test confirm what a live search crawler sees?

No. It does not fetch your domain, inspect response headers or account for crawler-specific behavior outside the parsed file. Test the deployed file separately after publishing.

Back to Web & SEO Tools

Robots.txt Generator & Tester

Compose ordered crawler rules, check their structure and simulate which directive allows or blocks each path.

Robots.txt Settings

Build explicit crawler groups without adding block rules automatically.

Website URL (optional)

Used only to suggest an absolute Sitemap URL.

Crawler Groups

The most specific matching User-agent group applies.

Crawler Group 1

User-agent entries

Rules

No rules. This group does not block any path.

Sitemaps

Each Sitemap directive must use an absolute URL.

Generated robots.txt

Plain text updates automatically.

User-agent: *

Validation

An empty Disallow rule has no blocking effect and is preserved exactly.

Add crawler rules to check for potential issues.

How to use Robots.txt Generator & Tester

1
In Robots.txt Generator, add crawler groups and enter one or more User-agent values for each group.
2
Add ordered Allow or Disallow rules, then include any absolute Sitemap URLs needed by the file.
3
Review Validation and the live Generated robots.txt before copying or downloading it.
4
Choose Use Generated Rules in Tester, or open Robots.txt Tester and paste another file.
5
Enter a crawler plus one or more URLs or paths, then click Test URLs to see the deciding rule.

Frequently asked questions

About Robots.txt Generator & Tester

Robots.txt mistakes are often small but consequential: a broad path rule, an empty user agent or an unexpected group can change what a crawler is allowed to request. The generator keeps groups, agents and ordered rules visible, preserves intentionally empty Disallow directives and reports structural problems without inventing restrictions.

The independent tester is designed for explanation rather than a simple pass/fail badge. For every submitted path it identifies the matching group, winning rule and allowed or blocked result. This is a local simulation of the supplied text, not a live crawl or an indexing audit, so deployment and search-engine inspection remain separate steps.