Technical SEO
Sep 09, 2026
|
4 min read
Why Is a URL Indexed Despite Being Blocked by robots.txt?
Problem:
During a technical SEO audit, a URL was found to be indexed by Google even though the website's robots.txt file contained a directive intended to block the URL. The relevant …
Solution:
A Disallow rule in robots.txt does not remove a URL from Google's index. It controls crawling, not indexing. If a URL stays indexed even after you block it with robots.txt, …