Add A Robots.txt To Your BANS Store

The brief (and by no means complete) definition of a robots.txt is that it is a document that tells search engine spiders what pages in your site to index and which to ignore. Many people choose to prevent their BANS folders and files from being indexed for several reasons. if indexed, these folders and files could be included in search engine results, and they are not pages you want your customers going to. Because of this it is best to tell spiders to not index your BANS folders and files.

To prevent pages from being indexed you need to do the following:

• Open Notepad or equivelant (do NOT use a word document program as these add hidden characters that can do funny things)
• Copy the following:

  User-agent: *
  Disallow: /cgi-bin/
  Disallow: /admin/
  Disallow: /cont/
  Disallow: /themes/
  Disallow: /scripts/

• Paste it in Notepad
• Save the document as robots.txt
• Upload the robots.txt to your store's root file (in most cases this will be public_html)

That's it. You can add any other page you don't want indexed by following the above format: Disallow: /name of folder to disallow/ or Disallow: /name of page to disallow.htm. Just make sure you don't leave a Disallow action blank (i.e. Disallow: ) as this will prevent your entire site from being indexed.