Googlebot picks up over 200 signals to feedback to its mainframe. Each of these 200 bits of information is then given a rating which is tweaked constantly. Googlebot’s job is to spider your website and collect the relevant keyword information. Then Google’s Algorithm team will work out what ratings and rankings will provide the most relevant results for each keyword.
I read somewhere that In each year, on average, Google conducts over 700 search experiments every day. Its goal? To continuously fine-tune the results of Googlebot’s discoveries. For this reason, if you can’t get Googlebot to fetch your site, you aren’t going to be in any search engine results.
So what is the obsession with Googlebot testing your web site? Humans view websites mainly visually. However, Google sees websites by means of the Googlebot spider. When Googlebot is sent to spider your website, it checks it from a robotic viewpoint. Literally, it slurps up combinations of keywords, what is in your titles, heading tags, image alt tags and a load of other information. Goes back to Google and dumps the results. Like a dog running after a stick and returning it for you to throw again. Constantly, relentlessly Google crawls and fetches. Then With Google Algorithms fine-tuned, your database results will filter into the results pages for certain keywords.
Testing your website can sound a boring pastime. Why would anyone want to check if your website has been spidered? It’s surprising how obsessive it can become when you start manicuring your website. It starts off as a tidy up, a spring clean turns into a manicure, pedicure and plucking of nostril hairs to make your eyes water!
For several years it want easy finding out how google viewed your site. There were several useful tools for google SEO checking and website analysis. But it was always difficult to know the accuracy of these. Some myths said that if you used these tools Google would know, as if they were keeping it all a big secret. Then Matt Cutts a Google frontman of the web spam team, started playing the good guy. He blogged regularly revealing there is no secret, content is king. 20 years on and the web has grown, spam websites are disappearing. Google now supply a wealth of free useful tools for testing if Google bot has visited your site and which pages it discovered.
Google used to provide very geeky tools for nerds. Their tools were aimed at developers, site designers and other more technical users. Recently they revealed the Googlebot Test My Site Tool at https://www.thinkwithgoogle.com/feature/testmysite
This new testing tool is targeting the business owners themselves, who may not have known about these tools’ existence, or didn’t understand their output.
With those tools setup, you are then in a great position to optimise your website and work with Google in a way conducive to all.
Test your robots.txt with the robots.txt Tester
You can submit a URL to the robots.txt Tester tool. The tool operates as Googlebot would to check your robots.txt file and verifies that your URL has been blocked properly.
https://www.google.com/webmasters/tools/robots-testing-tool
Googlebot respects your crawling requirements with robot text you can allow and disallow particular areas of your website.. It tries to figure out where it should go and how often . Googlebot has a crawl budget too. So this is why XML sitemaps have a priority setting. In effect your sitemap is like a map for Googlebot. How often each page is updated, the last date it changed and its importance compared to the main homepage. If it comes across errors it will inform you. So this proves really useful for finding broken links and 404 issues.