Any tips for making your website friendly to Internet bots? Is this different from SEO?
Printable View
Any tips for making your website friendly to Internet bots? Is this different from SEO?
Webmaster Guidelines - Webmaster Tools HelpQuote:
Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it's current for your site so that you don't accidentally block the Googlebot crawler. Visit Frequently asked questions - Controlling Crawling and Indexing - Google Code to learn how to instruct robots when they visit your site. You can test your robots.txt file to make sure you're using it correctly with the robots.txt analysis tool available in Google Webmaster Tools.