Gary Illyes from Google posted a brand new PSA on LinkedIn saying that the most typical purpose a website unexpectedly blocks Googlebot from crawling is because of a misconfiguration of a firewall or CDN.
Gary wrote, “examine what site visitors your firewalls and CDN are blocking.” “By far the most typical difficulty in my inbox is expounded to firewalls or CDNs blocking googlebot site visitors. If I attain out to the blocking website, within the overwhelming majority of the circumstances the blockage is unintended.”
So what are you able to do? Gary stated, “I’ve stated this earlier than, however need to emphasize it once more: make a behavior of checking your block guidelines. We publish our IP ranges so it needs to be very simple to run an automation that checks the block guidelines towards the googlebot subnets.”
Gary linked to this assist doc for extra particulars.
In brief, do what you’ll be able to to check to see in case your website is accessible to Googlebot. You should utilize the URL inspection software in Google Search Console, as one technique. Additionally, affirm together with your CDN or firewall firm that they’re permitting Googlebot and ask them to show it.
Discussion board dialogue at on LinkedIn.
