basicblog

A great thing about Google is, it gives webmasters all the help they need to get their websites into Google's index. There's a nice tool available in Google Webmaster Tools called 'Fetch as GoogleBot'. This tool, as we discussed in our SEO Tips for start-ups, can be a great help diagnosing errors and getting a website in Google's index faster. A robots.txt file is used for crawling efficiency, and preventing certain pages from being crawled etc. Sometimes though, GoogleBot might have difficulty fetching your robots.txt file. Here's a solution from Google to this problem.