Skip to main content
Technical SEO

Why Is a URL Indexed Despite Being Blocked by robots.txt?

Abdul Aouwal

Abdul Aouwal

September 9, 2026 ยท 4 min read

Question Answered

The Problem

During a technical SEO audit, a URL was found to be indexed by Google even though the website's robots.txt file contained a directive intended to block the URL.

The relevant robots.txt rule was:

User-agent: *
Disallow: /private-url/

The client had added this rule with the intention of preventing the specified URL from being crawled by search engine bots. However, the URL was still appearing as indexed in Google Search Console.

This created confusion for the client, who expected the Disallow directive to prevent the URL from appearing in Google's index.

The main questions raised during the audit were:

  • Why is the URL still indexed despite the robots.txt restriction?
  • Does a Disallow directive in robots.txt remove an already indexed URL?
  • What is the correct way to prevent the URL from appearing in Google's search results?

The Solution

A Disallow rule in robots.txt does not remove a URL from Google's index. It controls crawling, not indexing. If a URL stays indexed even after you block it with robots.txt, here is why it is happening and how to fix it.

1. Crawling vs. Indexing: The Core Distinction

  • robots.txt controls crawling: A Disallow directive tells search engine bots not to visit or fetch a page.
  • It does not control indexing: It does not stop Google from listing a URL in its search index.

When you block a URL that is already indexed, Google Search Console usually flags it as "Indexed, though blocked by robots.txt." Because Googlebot cannot crawl the page, the search snippet displays "No information available" since it cannot read the page content.

2. Why the URL Stays Indexed: Discovery Beats the Block

Even if you block a page, discovery beats the block.

If Google finds a link to your URL from anywhere, such as internal site links, an XML sitemap, external backlinks, or a manual submission, it can still index that address based on those external signals alone. It does not need to read the actual page content to know the URL exists.

3. The Dilemma: Can Disallow Remove an Indexed URL?

No, it cannot. In fact, it creates a trap.

To tell Google not to index a page, you use a noindex meta tag directly on the page:

<meta name="robots" content="noindex">

However, Google must be able to crawl a page to see and process that tag. If your robots.txt file is actively blocking Googlebot from visiting the page, it will never read your noindex tag, trapping the URL in Google's index indefinitely.

4. How to Correctly Remove the URL from Google's Index

To remove the URL from search results, follow this sequence:

  1. Remove discovery sources: Clean up or remove internal links, sitemap entries, and external backlinks pointing to the URL where possible.
  2. Temporarily allow crawling: Remove the Disallow rule for this specific URL from your robots.txt file so Googlebot can access the page again.
  3. Add the noindex tag: Insert the <meta name="robots" content="noindex"> tag into the HTML head of the page. You can also use the Google Search Console Removals tool to speed up the process.
  4. Wait for deindexing: Give Google time to recrawl the page, read the noindex directive, and drop it from the search index.
  5. Apply the block again (optional): Once Google has dropped the page, you can apply the Disallow rule again if you want to keep bots from crawling it in the future.
Key takeaway: robots.txt does not remove a page from the index. If a blocked URL shows up with "no information available" in search results, it was indexed via external signals. Restore crawl access, add a noindex tag, and let Google drop it naturally.
#Technical SEO
Abdul Aouwal

Abdul Aouwal

Technical SEO Consultant

Abdul Aouwal is a Technical SEO Consultant who analyzes how search engines and AI systems interpret websites, diagnosing why visibility fails and designing corrective strategies for long-term discoverability.