Announcement Announcement Module
Collapse

myhosting.com Forum Guidelines

Welcome to the myhosting.com Forums! The purpose of this forum is to allow you to discuss topics relating to hosting and topics of interest with other myhosting.com Customers. The primary focus should relate to issues not generally included in our Wiki or Control Panel, or information not provided by our Customer Support Team. Some examples include programming and coding support, web site design ideas and SEO practices.

The forum is not meant as a replacement to our Wiki or Customer Support. We highly recommend searching our Wiki before posting on the forum. Also, if you have a question for our Customer Support team, please contact them as you would normally.

We ask that if you have any complaints or questions relating specifically to your account, that you address them to our Customer Support department. Do not post personal account information such as passwords or other sensitive and private information the forum, as this type of information is best kept secret. Also, please be respectful of other users and refrain from defamatory comments or use of coarse language.

We will visit the forum from time to time, and contribute to various topics. We reserve the right to delete any posts which are deemed offensive or against the spirit of the forum. Any spamming or illegal activity will not be tolerated. We also reserve the right to block any user who violates these guidelines.

To get started, we welcome you to sign up at our User Management site. Any account created there will have access to these forums, our Wiki, our Ideas site and commenting permission on our Status Blog.
See more
See less
robots.txt Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • robots.txt

    Neophyte here ....

    There is a file on the root of my site ... 'robots.txt' ...

    What does it do?

    According to the access log files, it gets quite a few hits. It's a simple enough looking file.
    One thread on this board mentions it ... and talks about alterations users might have made to the file prior to backing up.

    Thanks for any info.

  • #2
    Re: robots.txt

    Good question The idea is to tell robots or "search engine spiders" where they should not go. Search engines use robots/spiders to index the internet and produce the search results. If you have an area of your site you want them to stay away from, you list it in the robots.txt.

    There's more info here: http://www.robotstxt.org/

    Comment


    • #3
      Re: robots.txt

      Hi,
      I am new here and to my website. I am working on SEO and getting indexed.

      I checked in Google Webmaster tools and have found that my robot.txt file is preventing Googlebot from indexing my site.

      Does anyone know:

      How I change it
      What do I change it to

      I don't know if this matters but I built my site in Front Page and am on the old servers.

      Like i said, I am new and not as computer literate as I would like to be but I am learning.

      Thank you in advance for your help
      ColleenB

      Comment


      • #4
        Re: robots.txt

        Hi Colleen,

        The robots.txt is a simple text file which basically tells bots, such as googlebot, where to go or not go. If you connect to your site using FTP or our File Manager, you should see a file called 'robots.txt' in your main directory. You can simply download that file, and open it up with 'notepad' or any other text editor, and make changes to it.

        One solution would be to simply remove the robots.txt file altogether. If there are no 'instructions' for googlebot, it will simply go through and spider your whole site. And this should be fine, assuming you don't have anything you want to hide from any search engine. You can also check out the site I posted previously, it should give you all sorts of information for what to do next if you want to keep your robots.txt.

        Also, if you post the contents of your robots.txt file here, I'm sure we can try to help you further!

        Comment


        • #5
          Re: robots.txt

          Thanks Tim! I want Googlebot to see EVERYTHING..LOL I even want my images to be in Google Images. I want as much exposure as possible.
          this is the contents of my robot.txt file:

          User-agent: *

          Disallow: /_fpclass
          Disallow: /_private
          Disallow: /_themes
          Disallow: /_vti_cnf
          Disallow: /_vti_log
          Disallow: /_vti_pvt
          Disallow: /_vti_script
          Disallow: /_vti_txt

          Disallow: /cgi-bin
          Disallow: /email
          Disallow: /fpdb
          Disallow: /image

          #Disallow: /w3svc? #Change ? with the instance number please.


          I don't know what most of these files are so I didn't know if I should delete the entire file. Like I said, I am so new at all this.

          My website address is: http://www.ooak-fairy-artdoll-sculptures.com in case that will help

          I did go to that website as well as study on Google Webmaster's section but I am not confident enough in my knowledge to really know if what I am doing is correct.

          Thanks again SO MUCH for your help

          Comment


          • #6
            Re: robots.txt

            This part means the rules should apply to all robots:

            User-agent: *
            And these entries simply say, "do not go here"

            Disallow: /_fpclass
            Disallow: /_private
            Disallow: /_themes
            Disallow: /_vti_cnf
            Disallow: /_vti_log
            Disallow: /_vti_pvt
            Disallow: /_vti_script
            Disallow: /_vti_txt

            Disallow: /cgi-bin
            Disallow: /email
            Disallow: /fpdb
            Disallow: /image
            All of the "/_" directories are related to FrontPage Server Extensions, which you generally don't want indexed. Similarly /cgi-bin and /fpdb are where you would normally store scripts or databases, which you would also want to exclude.

            The /email directory doesn't matter, since its empty. You could remove the directory from your site and the robots.txt file if you want. And finally, I noticed you don't even have a directory called /image on your site, so you could also remove that entry. But these 2 don't really matter since there is no content being disallowed here.

            Any other file or directory not listed above as a "disallow" (i.e. everything else) would be allowed for indexing.

            I don't see why googlebot would be blocked from indexing your site by this robots.txt file. What was the message you got from google webmaster tools?

            Comment


            • #7
              Re: robots.txt

              Thanks for your reply Tim.

              Google said: (which referred to my main website address)

              We had problems crawling the pages listed here, and as a result they won't be added to our index and will not appear in search results

              Error icon: robots.txt unreachable
              ----------------------------------
              I have deleted the file but they have not tried to crawl the site again. It's been since 3/2. I have no clue how long it will take them to try again.

              I wrote my entire site in Front Page so every page has the same extention.
              Thanks again
              Colleen

              Comment


              • #8
                Re: robots.txt

                I'm not sure why they couldn't find your robots.txt file, unless it was moved or renamed. I found the following tool which could be usefull in finding errors in a robots.txt:

                If you still have your robots.txt located in the root folder of your site you can do the following to check it for errors:

                * Sign into Google Webmaster Tools with your Google Account.
                * On the Dashboard, click the URL for the site you want.
                * Click Tools, and then click Analyze robots.txt.

                We had problems crawling the pages listed here, and as a result they won't be added to our index and will not appear in search results
                This error could also be related to problems crawling your site. Make sure your pages are all linked together through internal hyperlinks. You could also try adding a sitemaps.xml file, which you can submit to search engines to help them crawl your site and to suggest to them how often to come back for updates.

                Comment


                • #9
                  Re: robots.txt

                  My internal links are buttons. I had read in the Google Webmaster Tools that they should be text. So I think I am going to add the text links at the bottom of each page to see if that helps.

                  I also saw the information about the site maps but haven't been able to figure out how to do them, even with the demonstrations in Webmaster Tools. Like I said, I am such a beginner in all of this and am doggone proud that I have got this far. LOL I will try again to understand how to make a site map and go that route and see what happens.

                  Thanks again, I appreciate your time and efforts with me.
                  ((HUGS))
                  Colleen

                  Comment


                  • #10
                    Re: robots.txt

                    Originally posted by ColleenB
                    My internal links are buttons. I had read in the Google Webmaster Tools that they should be text. So I think I am going to add the text links at the bottom of each page to see if that helps.

                    I also saw the information about the site maps but haven't been able to figure out how to do them, even with the demonstrations in Webmaster Tools. Like I said, I am such a beginner in all of this and am doggone proud that I have got this far. LOL I will try again to understand how to make a site map and go that route and see what happens.

                    Thanks again, I appreciate your time and efforts with me.
                    ((HUGS))
                    Colleen
                    You can use graphic images as buttons. What you also need to do is use the alt tag and title tags. These can say the name of the image or the button name/action or anything you want "found" for SEO purposes.
                    The alt tag is incorrectly used as a float over option when that should really be handled by the title tag. It's widespread and even accepted in strict validation.

                    Comment

                    Working...
                    X