+  Forum Owners
|-+  The Growing Forum Owner: Maintaining and Promoting a Forum» Forum SEO» Where and how to get Robot.txt?
Username:
Password:
Home Help Search Login Register
Pages: [1]
« previous next »
  Print  
Author Topic: Where and how to get Robot.txt?  (Read 269 times)
qaswer
Newbie
*
Offline Offline

Posts: 41


View Profile
« on: September 08, 2007, 03:06:54 PM »
I have just registered an account at Google sitemaps but there was an error that I haven't Robot.txt. I have tried this file on google but there is no place to find it. There are a lot of instructions to include or exclude some urls. Can someone help me on this?
Logged
shreyas
Newbie
*
Offline Offline

Posts: 12


View Profile
« Reply #1 on: September 26, 2007, 07:53:39 AM »
I think you have to create a file called robots.txt and add the required codes there for the search engine robots.

You can see more info here. It contains a cool description of robots.txt
Logged
ForumOwner
Administrator
Full Member
*****
Offline Offline

Posts: 138


View Profile WWW
« Reply #2 on: September 26, 2007, 08:56:47 AM »
The error is not a showstopper, but you should be making good use of the robots.txt file in order to keep bots out of area they should not go.  You can also specify the location of your sitemap in your robots.txt as pointed out elsewhere on this forum;)

In addition to Shreyas recommendation, this site offers good information:  http://www.robotstxt.org/wc/robots.html
Logged
zoyla
Newbie
*
Offline Offline

Posts: 20


View Profile
« Reply #3 on: October 31, 2007, 04:22:10 AM »
I think the only purpose of Robot.txt is to give information to Bots. It tells bots about pages which should not be crawled. It also helps Bots to crawl the site quickly so I think this is a big help for their websites. Bots can generate errors if someone will has authenticated pages.
Logged
Pages: [1]
  Print  
« previous next »
 
Jump to:  

Theme by m3talc0re. Powered by SMF 1.1.5 | SMF © 2006-2008, Simple Machines LLC Privacy