Gary Illyes shared a pleasant little tidbit on LinkedIn about robots.txt information. He mentioned that solely a tiny variety of robots.txt information are over 500 kilobytes. I imply, most robots.txt information have a couple of strains of textual content, so this is sensible however nonetheless, it’s a good tidbit of information.
Gary checked out over a billion robots.txt information that Google Search is aware of about and mentioned solely 7,188 of them have been over 500 KiB. That’s lower than 0.000719%.
He wrote, “One would assume that out of the billions (sure, with a 🐝) of robots.txt information Google is aware of of greater than 7188 could be bigger in byte dimension than the 500kiB processing restrict. Alas. No.”
Yea, the search engine optimization level right here is that Google can course of as much as 500KB of your robots.txt file however most of these information do not even come near that file dimension.
Discussion board dialogue at LinkedIn.
