robots.txt vs noindex: Controlling Crawling and Indexing

The difference between robots.txt vs noindex comes down to crawling versus indexing. A robots.txt file tells crawlers which URLs they may request; it controls crawling. A noindex rule, set in a meta tag or an HTTP header, tells search engines not to show a page in results; it controls indexing.

The two are not interchangeable, and they can work against each other. If you block a page in robots.txt, Google cannot fetch it, so it never sees a noindex tag on that page, and the URL can still appear in results if other pages link to it. To keep a page out of Google, let it be crawled and mark it noindex. To save crawl effort on URLs that do not matter, block them in robots.txt.

This guide explains both mechanisms with Google's documentation as the reference, then covers the patterns developers most often get wrong.

What robots.txt does

A robots.txt file sits at the root of a host, such as https://example.com/robots.txt, and gives crawlers rules about which paths they may request. Google's introduction to robots.txt describes it as a way to manage crawler traffic, mainly to avoid overloading a site with requests, and explicitly not as a way to keep pages out of Google.

User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /*?sort=
Allow: /admin/public-help/

Sitemap: https://example.com/sitemap.xml

• User-agent names the crawler the group applies to; * means all crawlers.
• Disallow lists paths the crawler should not request.
• Allow makes exceptions inside a disallowed path.
• Sitemap points crawlers to your sitemap, using a full URL.

Each host and protocol has its own file. Rules on example.com do not apply to shop.example.com, which needs its own robots.txt.

Patterns support two wildcards in Google's implementation: * matches any sequence of characters and $ marks the end of a URL. They are powerful and easy to get wrong, so keep rules as simple as possible and comment anything unusual. A rule such as Disallow: /*? blocks every URL with a query string, which may include pages you want crawled.

What noindex does

A noindex rule tells search engines not to include a page in their results. Google's article on blocking indexing with noindex describes two ways to set it.

<!-- In the page head -->
<meta name="robots" content="noindex">

# Or as an HTTP response header
X-Robots-Tag: noindex

When Googlebot next crawls the page and sees the rule, Google drops the page from its results, even if other sites link to it. The page remains available to visitors who follow a link or type the address. That makes noindex the right tool for pages that should exist but not appear in search: thank-you pages, internal search results, account areas and thin filter combinations.

Removal is not instant. Google has to recrawl the page to see the rule, which can take days or weeks for pages it rarely visits. The removals tool in Search Console can hide a URL temporarily while that happens.

robots.txt vs noindex: choosing the right tool

Duplicate URLs are a separate problem with a separate solution. Our guide to canonical URLs and duplicate content explains redirects and rel=canonical.

The mistake: blocking and noindexing together

The most common error is adding a Disallow rule for a section and putting noindex on the same pages, expecting double protection. The opposite happens. Because crawling is blocked, Google never fetches the pages, never sees the noindex, and may keep the URLs in its index based on links from elsewhere, shown without a description.

1. Remove the robots.txt rule for the pages you want removed.
2. Make sure they return a noindex rule, in the head or as a header.
3. Wait for Google to recrawl them, or request indexing for the most important ones.
4. Once they are gone from results, you can block crawling again if you need to save crawl budget.

Even then, keep in mind that blocking later means Google cannot see if the noindex is ever removed by mistake. For most small and medium sites, leaving noindexed pages crawlable is simpler and safer, because the crawl cost is negligible.

The same logic applies to pages you want removed after a redesign. If old URLs should disappear, let them return a 404 or 410, or redirect them to the best replacement, and keep them crawlable so Google can see the change.

Staging and development sites

Staging sites are a classic source of both problems. Developers often add a site-wide Disallow: / to staging, which stops crawling but does not guarantee that staging URLs never appear in results if they are linked publicly. The reverse problem is worse: deploying the staging robots.txt or a site-wide noindex to production, hiding the entire live website.

• Protect staging with authentication such as HTTP basic auth or an IP allow-list. That is the only reliable barrier.
• Generate robots.txt per environment in code, rather than committing a static file that can be deployed anywhere.
• Add a production check to your deployment pipeline that fails if robots.txt contains Disallow: / or if the home page returns noindex.

// Laravel route: robots.txt generated per environment.
Route::get('/robots.txt', function () {
    $lines = app()->isProduction()
        ? ['User-agent: *', 'Disallow: /admin/', '', 'Sitemap: ' . url('/sitemap.xml')]
        : ['User-agent: *', 'Disallow: /'];
    return response(implode("\n", $lines) . "\n")->header('Content-Type', 'text/plain');
});

Our article "Is my website on Google?" covers how a forgotten development block shows up in Search Console.

Large sites and crawl efficiency

robots.txt earns its keep on large sites where crawlers could request near-infinite URL combinations: faceted filters, sort orders, calendars and session parameters. Blocking these patterns lets Google spend its crawl effort on pages that matter.

• Block parameter patterns that never produce useful landing pages, such as sort, view and session options.
• Keep filter combinations that do deserve to rank crawlable and linked.
• Do not block CSS, JavaScript or image files that pages need to render, because Google needs them to understand the page.
• Review the crawl stats report in Search Console to see where Googlebot spends its time, and adjust rules when it keeps visiting low-value URLs.

Our guide to faceted navigation SEO explains how to decide which filter pages should be indexable in a large catalogue.

How Google reads robots.txt

A few details of how Google fetches and interprets robots.txt explain behaviour that otherwise looks strange.

• Caching: Google generally caches robots.txt for up to a day, so changes are not picked up instantly.
• Missing file: if robots.txt returns a 404 or other client error, Google treats the site as having no crawl restrictions.
• Server errors: if robots.txt returns a server error, Google may temporarily stop crawling the site, because it cannot tell what is allowed. Keep the file fast and reliable.
• Size: Google reads only the first 500 kibibytes of the file; anything beyond is ignored.
• Specificity: when Allow and Disallow rules conflict, Google applies the most specific matching rule, meaning the longest path.

Because a server error on robots.txt can pause crawling, it deserves the same monitoring as your home page. A misconfigured firewall or CDN rule that blocks the file can have site-wide effects.

Other robots directives worth knowing

The robots meta tag and X-Robots-Tag accept more than noindex. A few are useful in specific situations.

• nofollow: asks crawlers not to follow links on the page. It is rarely needed on your own pages.
• nosnippet: prevents a text snippet or video preview from being shown in results.
• max-snippet, max-image-preview: control the length of snippets and the size of image previews.
• unavailable_after: asks Google to stop showing a page after a date, useful for time-limited offers.

Rules can target a specific crawler, for example <meta name="googlebot" content="noindex">, though most sites should apply the same rules to all search engines.

Testing robots rules safely

A single wrong character in robots.txt can block an entire site, so treat changes like code changes.

1. Keep robots.txt in version control or generate it in code, with reviews before deployment.
2. Write automated tests that assert key pages are allowed and private paths are disallowed.
3. After deployment, fetch /robots.txt from production and compare it with the expected output.
4. Use the URL Inspection tool to confirm that important pages are crawlable and indexable.
5. Check the page indexing report weekly after major changes for pages "blocked by robots.txt" or "excluded by noindex".

Our technical SEO checklist includes these checks among the routine tasks for online stores.

Frequently asked questions

Pages stuck in Google, or missing from it? Our SEO services include crawl and indexing audits, and our developers can fix the rules for you.