Robots.txt Guide: Control What Search Engines Crawl

Robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they can and cannot access. This robots txt guide covers what the file actually does, the directives that matter most, common configuration mistakes that block important content, and how to test your setup before problems accumulate.

What Robots.txt Does and Does Not Do

Robots.txt controls crawler access, not indexing. These are different things, and confusing them leads to misconfigured files that do not accomplish what you intend.

Disallowing a URL in robots.txt prevents Googlebot from crawling that URL, but it does not prevent that URL from appearing in Google's index. If other pages link to a disallowed URL, Google may still index it, showing the URL in search results with no title or snippet because it cannot read the content. A URL can be indexed without ever being crawled if enough external signals point to it.

For pages you want to prevent from appearing in search results entirely, a noindex tag is more appropriate than robots.txt disallowing. The robots.txt file is the right tool for controlling crawl access when you have content that serves functional purposes for users but provides no value in search results.

Robots.txt is also not a security measure. Malicious bots and scrapers typically ignore robots.txt entirely. For content you want to protect from unauthorized access, use authentication, not robots.txt.

Robots.txt Directives

The core robots.txt syntax involves a small set of directives that work together to define access rules for different crawlers.

User-agent

User-agent specifies which crawler the following rules apply to. User-agent: * applies rules to all crawlers. User-agent: Googlebot applies rules only to Google's crawler. You can have multiple rule groups in a single robots.txt file targeting different crawlers separately.

Disallow

Disallow specifies paths the crawler should not access. Disallow: /admin/ blocks crawling of everything under the /admin/ path. Disallow: / blocks crawling of the entire site. An empty Disallow (Disallow:) allows full crawling.

Allow

Allow creates exceptions within a disallowed path. If you disallow an entire directory but want one subfolder accessible, an Allow directive for that subfolder takes precedence over the parent Disallow rule.

Sitemap

Including a Sitemap directive in robots.txt points crawlers to your XML sitemap location. This is standard practice that helps crawlers discover your sitemap independently from Google Search Console submission.

Crawl-delay

Crawl-delay tells crawlers to wait a specified number of seconds between requests. This is primarily relevant for crawlers other than Googlebot, as Google does not honor the Crawl-delay directive in robots.txt. Google's crawl rate is managed through Google Search Console settings instead.

What to Disallow in Robots.txt

The goal of robots.txt for most content sites is reducing crawl noise, not hiding content. Specific categories worth disallowing include:

Admin and authentication paths. Paths like /admin/, /wp-admin/, /login/, and /cart/ serve functional purposes for users but have no value in search results. Blocking them keeps crawl resources focused on indexable content.

Internal search result pages. If your site has a search function at /search/ or ?s=, disallowing these prevents Googlebot from crawling thin, dynamically generated search result pages that will never rank. The crawl budget optimization guide explains how reducing crawl noise like this compounds into meaningful efficiency gains on large sites.

Parameterized URLs and Staging Environments

Parameterized or duplicate content not handled by canonical tags. If your site generates pagination at /page/2/ and beyond, and those pages have no standalone ranking value, disallowing them reduces the crawl surface area. Where canonicals are the first line of defense for duplicate content, robots.txt disallowing is a secondary option for URL patterns that generate significant crawl waste.

Pre-Launch and Development Paths

Staging environments and development paths should be fully disallowed if they are publicly accessible. A staging subdomain or development folder left open to crawlers creates duplicate content and can expose content before it is ready for indexing.

What Not to Disallow: Common Robots.txt Mistakes

Accidental whole-site blocking is the most damaging robots.txt error. A Disallow: / rule with User-agent: * blocks all crawlers from the entire site. This configuration appears in Google Search Console quickly as a collapse in indexed pages. CMS migrations and developer testing sessions are the most common causes.

Blocking CSS and JavaScript that the site depends on for rendering is another significant mistake. If Googlebot cannot load your stylesheets or JavaScript files, it cannot render your pages correctly. Misconfigured robots.txt rules blocking /css/ or /js/ directories produce rendering failures that affect indexing quality across the whole site.

Disallowing pages you actually want indexed happens when teams copy robots.txt rules from a previous site or template without reviewing them for the current site's structure. Every robots.txt should be reviewed in context of the specific site it governs.

Sitemap Consistency

A consistency issue worth addressing separately: your sitemap should not include URLs that are blocked by robots.txt. The XML sitemap best practices guide covers this in detail. Including disallowed URLs in a sitemap sends contradictory signals and generates Search Console errors that are worth eliminating.

Robots txt Guide: Testing and Monitoring Your Setup

Google Search Console's URL Inspection Tool

The URL Inspection tool in Google Search Console shows whether a specific URL is blocked by robots.txt. If you suspect a page is not being crawled, inspecting it here is the first diagnostic step. A "Blocked by robots.txt" status confirms the issue directly.

Robots.txt Tester in Search Console

Google Search Console provides a robots.txt tester under the Legacy Tools section. You can test any URL on your site against your current robots.txt file to see whether it would be allowed or blocked under the current rules. This is the most direct way to verify configuration before relying on the live file.

Ongoing Coverage Monitoring

Robots.txt errors appear in the Coverage report in Google Search Console. URLs that are "Blocked by robots.txt" while being included in your sitemap trigger errors worth reviewing regularly. These often indicate either a sitemap that needs updating or a robots.txt rule with unintended scope.

The SEMrush robots.txt guide covers the most common configuration patterns for different site types, including ecommerce sites with complex URL structures and WordPress sites where plugin-generated paths need blocking.

Robots.txt and Noindex: When to Use Each

Robots.txt and noindex tags address overlapping problems through different mechanisms.

Robots.txt prevents crawling. A disallowed page cannot be read by Googlebot, but the URL itself can still appear in search results if linked from other pages. This makes robots.txt appropriate for content you do not need read and do not need appearing in search results at all.

Noindex prevents indexing. A page with a noindex tag can be crawled but will not appear in search results. This is the right choice for pages where you want Google to process the tag but not index the content, such as internal utility pages that should remain accessible to users.

For most duplicate content management, canonical tags are the first tool, noindex is the second, and robots.txt disallowing is reserved for content with no search value whatsoever. The canonical tags guide covers how these tools layer together when managing duplicate content across a large site.

Maintaining Robots.txt Over Time

Robots.txt is infrastructure that requires periodic maintenance as your site evolves. CMS version upgrades, platform migrations, and developer customizations can overwrite or alter the robots.txt file without an explicit decision being made about the changes. Adding robots.txt review to your quarterly technical audit catches configuration drift before it affects indexing.

The most common maintenance gap is a robots.txt that was configured for an older version of the site and now blocks or allows paths that no longer reflect the current structure. Running a URL inspection on your most important pages after any major site change takes minutes and catches blocking errors before they affect rankings.

The Search Engine Land robots.txt resource documents how major CMS platforms handle robots.txt generation automatically and what to check during routine technical reviews when the file is platform-managed rather than manually maintained.

The technical SEO checklist includes robots.txt validation alongside crawl health checks, canonical tag verification, and sitemap auditing as part of the standard quarterly audit sequence.