Crawling only the sitemap on google webmaster tools

by Shan Xue   Last Updated January 26, 2018 17:04 PM

So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites.

So we tried playing with robots.txt i.e.:

User-agent: *

Disallow:

Sitemap: http://www.website.com/sitemap.xml

But when I enter the bad URL in robots.txt tester, it still allow the URL that we don't want.

Is there any way that google only crawls the sitemap on robots.txt without manually entering all the bad links on the "Disallow"?



Answers 1


Google has never limited itself to crawling and indexing just URLs that are in the sitemap. Such functionality does not exist, and I doubt that it ever will.

Stephen Ostermiller
Stephen Ostermiller
January 26, 2018 17:02 PM

Related Questions


Updated February 22, 2018 18:04 PM

Updated August 02, 2019 09:04 AM

Updated September 17, 2019 11:04 AM

Updated May 21, 2019 06:04 AM