Googlebot Test My Site

Ultimately Google want to have the most relevant & reliable fast loading sites at the top. The Googlebot test my site tool is designed to check for any simple issues.

Googlebot picks up over 200 signals to feedback to its mainframe. Each of these 200 bits of information are then given a rating which is tweaked constantly. Googlebot’s job is to spider your website and collect the relevant keyword information. Then Google’s Algorithm team will work out what ratings and rankings will provide the most relevant results for each keyword.

I read somewhere that In each year, on average, Google conducts over 700 search experiments every day. Its goal? To continuously fine tune the results of Googlebots discoveries. For this reason, if you can’t get Googlebot to fetch your site, you aren’t going to be in any search engine results.

Why get Googlebot to crawl your site?

So what is the obsession with Googlebot testing your web site? Humans view websites mainly visually. However Google sees websites by means of the Googlebot spider. When Googlebot is sent to spider your website, it checks it from a robotic viewpoint. Literally it slurps up combinations of keywords, what is in your titles, heading tags, image alt tags and a load of other information. Goes back to Google and dumps the results. Like dog running after a stick and returning it for you to throw again. Constantly, relentlessly Google crawls and fetches. Then With Google Algorithms fine-tuned, your database results will filter into the results pages for certain keywords.

Googlebot obsession

Testing your website can sound a boring pastime. Why would anyone want to check if your website has been spidered? It’s surprising how obsessive it can become when you start manicuring your website. It starts off as a tidy up, a spring clean turns into a manicure, pedicure and a plucking of nostril hairs to make your eyes water!

Get Googlebot to crawl your site

For several years it want easy finding out how google viewed your site. There were several useful tools for google SEO checking and website analysis. But it was always difficult to know the accuracy of these. Some myths said that if you used these tools Google would know, as if they were keeping it all a big secret. Then Matt Cutts a Google frontman of the web spam team, started playing the good guy. He blogged regularly revealing there is no secret, content is king. 20 years on and the web has grown, spam websites are disappearing. Google now supply a wealth of free useful tools for testing if Google bot has visited your site and which pages it discovered.

My advice would be the following

Google used to provide very geeky tools for nerds. Their tools were aimed at developers, site designers and other more technical users. Recently they revealed the Googlebot Test My Site Tool at https://www.thinkwithgoogle.com/feature/testmysite

This new testing tool is targeting the business owners themselves, who may not have known about these tools’ existence, or didn’t understand their output.

Additional Tools I recommend

• Set up a Google Search Console account
• Submit an XML sitemap
• Setup a Google analytics account and connect it to the Search Console
• Check you website using the Google Mobile-Friendly Test. This tools is focused on the site’s design and ability to load quickly.  Additionally the page speed tools are great for improving ranking for fast loading websites.

With those tools setup, you are then in a great position to optimise your website and work with Google in a way conducive to all.

Googlebot test my site with a robots.txt

Test your robots.txt with the robots.txt Tester

You can submit a URL to the robots.txt Tester tool. The tool operates as Googlebot would to check your robots.txt file and verifies that your URL has been blocked properly.

https://www.google.com/webmasters/tools/robots-testing-tool

Googlebot indexing with XML Sitemap

Googlebot respects your crawling requirements with robot text you can allow and disallow particular areas of your website.. It tries to figure out where it should go and how often . Googlebot has a crawl budget too. So this is why XML sitemaps have a priority setting. In effect your sitemap is like a map for Googlebot. How often each page is updated, the last date it changed and its importance compared to the main homepage.  If it comes across errors it will inform you. So this proves really useful for finding broken links and 404 issues.