Google introduced a brand new robots.txt report inside Google Search Console and on the identical time mentioned it is going to sundown the outdated robots.txt tester device. The brand new device exhibits which robots.txt recordsdata Google discovered for the highest 20 hosts in your website, the final time they had been crawled, and any warnings or errors encountered.
Google additionally mentioned on X, “We’re additionally making related data obtainable from the Web page indexing report. As a part of this replace, we’re sunsetting the robots.txt tester.”
To entry this, go to the settings hyperlink on your Search Console profile or click on right here. That is what it seems to be like:
Here’s what the report seems to be like for this website, sure, I’ve a vital error. Listed below are these particulars and observe, you’ll be able to request recrawl of your robots.txt file:
This assist doc goes via much more about this report.
Some are usually not comfortable that the robots.txt tester device is gone however Bing nonetheless has one:
Good alternate options to Google’s depreciated device to validate your robots.txt:
• Merkle: https://t.co/33yMW9IlPB
• Ryte: https://t.co/jYAGuHN2q5
• Screaming Frog (in-crawl): https://t.co/zsr2KMiNmC— Cyrus search engine optimization (@CyrusShepard) November 15, 2023
I really like the brand new robots.txt reporting (it is in settings btw), however eradicating the robots.txt Tester? I really like that device and it may be super-helpful. There are different instruments that do related issues, together with one from Bing, however I might like to see that keep in GSC. 🙂 https://t.co/hMIJQjfpQV
— Glenn Gabe (@glenngabe) November 15, 2023
Discussion board dialogue at X.
