Check robots.txt & sitemap
Find out in seconds whether Google may crawl your website, whether your sitemap is found and whether the URLs in it respond cleanly.
Check domain
Domain with or without https:// — we find robots.txt and the sitemap automatically.
What does the check test?
robots.txt and the XML sitemap are the signposts for search engines. Small mistakes in them can make entire websites disappear from Google.
robots.txt reachable
Does /robots.txt return a valid text file — or a server error that stops crawling?
Home page blocked?
Detects “Disallow: /” and other rules that lock Google out — using Google’s “longest match wins” logic.
CSS, JavaScript, images
Are files blocked that Google needs to render the page?
Find the sitemap
From the robots.txt reference or at /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml.
Sitemap contents
Number of URLs, child sitemaps, modification dates (lastmod) and the 50,000-URL limit.
Sample
Three URLs from the sitemap are fetched: do they return 200, redirect or fail?
How it works
- Enter your domain and click “Check”.
- The tool reads robots.txt, finds the sitemap and fetches three URLs from it.
- You get a score, a traffic light per check, the robots.txt in plain text and concrete steps.
- Optional: have us review your result.
Why it matters
One character can block everything
“Disallow: /” is often left over from a staging site after a relaunch — and Google gradually drops the pages from its index.
The sitemap speeds up indexing
A clean sitemap with modification dates shows Google new and changed pages faster.
Sitemap errors waste crawl budget
Redirects and 404 pages in the sitemap waste Google’s time and weaken trust in the file.
Frequently asked questions
What happens if there is no robots.txt?
If /robots.txt returns a 404, Google may crawl the whole site. That is allowed. Only the sitemap reference is missing, which we still recommend.
Why is a 5xx error on robots.txt so dangerous?
If robots.txt returns a server error, Google treats the whole site as blocked to be safe and pauses crawling until the file is reachable again.
Does “Disallow” remove a page from Google?
Not reliably. robots.txt only forbids crawling. To remove a page from the index it needs “noindex” on the page itself — and for that it must not be blocked.
How many URLs can a sitemap contain?
At most 50,000 URLs or 50 MB uncompressed per file. Larger sites split the sitemap and link the parts via a sitemap index.
Do I have to submit the sitemap to Google?
The robots.txt reference is enough for Google to find it. Submitting it in Search Console also shows how many URLs Google has read and indexed.
Google can’t find your pages?
We check indexing, crawling and sitemaps in Search Console and make sure your important pages reach Google.
