crawlers

[PotsNPans]

Any tips for making your site more friendly to web crawlers?

[suwunk]

Check for crawl errors. The Crawl errors page in Webmaster Tools provides details about the URLs in your site that we tried to crawl and couldn't access. Review these errors, and fix any you can. The next time Googlebot crawls your site, it will note the changes and use them to update the Google index.
Review your robots.txt file. The Test robots.txt tool lets you analyze your robots.txt file to see if you're blocking Googlebot from any URLs or directories on your site.
Make sure that the URLs haven't been blocked with meta tags.
Review your site structure and make sure that it's easily accessible. Most search engines are text-based. If you use JavaScript, DHTML, images, or rich media such as Silverlight to create navigation and links, Googlebot and other spiders may have trouble crawling your site.
If you have recently restructured your site or moved to a new domain, pages that previously performed well may now rank poorly. To avoid this, use 301 redirects ("RedirectPermanent") in your .htaccess file to smartly redirect users, Googlebot, and other spiders. (In Apache, you can do this with an .htaccess file; in IIS, you can do this through the administrative console.) For more information about 301 HTTP redirects, please see http://www.ietf.org/rfc/rfc2616.txt.
Consider creating and submitting a Sitemap. Even if your site is already indexed, Sitemaps are a way to give Google information about your site and the URLs you consider most important. Sitemaps are particularly helpful if your site has dynamic content or other content not easily discoverable by Googlebot, or if your site is new or does not have many links to it.


My site isn't doing well in search - Webmaster Tools Help (http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=34444&ctx=cb&src=cb&cbid=-pps8f2sldehg&cbrank=3#2)

[kabonm]

Review your robots.txt file. The Test robots.txt tool lets you analyze your robots.txt file to see if you're blocking Googlebot from any URLs or directories on your site.