internet marketing ezine
search engine optimization
  website promotion logo

internet marketing ezine  >   ethical search engine optimization  >   robots.txt file generator

Software to help you create a robots.txt file

By Sumantra Roy

Robogen is a software for Windows 95, Windows 98 and Windows NT based computers which can help you create robots.txt files without having to worry about the correct syntax of a robots.txt file. It comes in two editions: the free Limited edition and the Standard edition which costs $12.95. You can get Robogen from

In this article, I'll tell how you can use Robogen to simplify the process of creating a robots.txt file. This article assumes that you have read my article on what a robots.txt file is and how you can manually create it. If you missed the article, it is available here.

I would recommend that you download the trial version of the Standard Edition. The Limited edition does not support as many search engines as the Standard Edition.

Once you have downloaded Robogen and installed it in your hard drive, start the program. Here are the steps that you should follow to create a robots.txt file:

Step 1: Click on the Logon FTP Server button. In the dialog box that appears, type in the address of the FTP server for your web site, make sure that the "This server requires me to log in" option is checked, and then type in the username and the password which you use to access the server in the appropriate boxes.

Now, depending on how your web host has set up the server, the files for your web site may be present in the root directory of the server itself. Or, it may be present in a sub-directory of the root directory. In the latter case, click on the "Use specified root" option and type the relevant sub-directory in the text box besides the option. If you don't know in which sub-directory the files for your web site are present, ask your web host to give it to you.

Done? Now click on OK. After a while, Robogen will display the directories and files which are present in your site in the right pane of the main Robogen program window. If you find that Robogen has not displayed the files in your site, it means that you did not enter the correct sub-directory in which the files of your web site are present. In this case, ask you web host to give the correct sub-directory to you.

Step 2: In the left pane of the main Robogen window, click on the small button present to the right of the drop-down combo box at the top. In the menu that appears, select "Popular Search Engines".

Step 3: Now, click on the drop-down combo box and choose the search engine that you are targeting (say, AltaVista). In the right pane, select a page which is NOT meant for AltaVista. Now click on the "<<<" button. You will find that the file which you had selected in the right pane is now present in the left pane. In this way, click on each of the pages which are not meant for AltaVista and press the "<<<" button.

Repeat step 3 for each of the search engines that you are targeting. Just select the search engine that you are targeting from the drop-down combo box, select the pages which are not meant for the search engine one by one from the right pane, and click on the "<<<" button after clicking on each age. Repeat this process for all the search engines that you are targeting.

Step 4: Upload the file to your server using the "Upload Robots.Txt" button. If you want, you can also use the Save menu item from the File menu to save a copy of the robots.txt file in your hard drive.

Article by Sumantra Roy. Sumantra is one of the most respected search engine positioning specialists on the Internet. To have Sumantra's company place your site at the top of the search engines, go to For more advice on how you can take your web site to the top of the search engines, subscribe to his FREE newsletter by going to

Discover the 3 major problems of pay per performance web traffic from the ppc search engines and how to solve them.

An introduction to the most popular pay per click search engines, their features, advantages, disadvantages and the differences between them. To get up to 60$ in free traffic, visit the pay per performance search engine review.

Paying for clicks doesn´t guarantee you make any sales / profits. Instead You could pay only when someone buys your products. Find out how you can have an affiliate program set-up within hours and get others to sell your products for You.
Risk-free pay per performance advertsing.

Advertising Financial Services Business General Business Opportunity Computers Hardware Computers Software Education Ezines Home Based Business Opportunities Internet Marketing Reciprocal Linking Search Engine Marketing and Optimization Shopping Web Design and Development Web Hosting Affiliate Resources Web Site Marketing Webmaster Resources

Internet Marketing - Home
Internet Marketing Blog
Contact Me
Newsletter Past Issues
More Marketing Articles
Reviews Index

i-cop member seal

Free SEO Tips and News

Subscribe to my Blog´s RSS Feed to receive online marketing and website promotion tips. RSS Feed

Category: Search Engine Optimization
SEO Tips and News via RSS

Important SEO Tools

  Keyword Research Tools

  Overture´s keyword tool

  Link Popularity Building

  SEO Elite
  Web Directories

  SEO site analysis

  Internet Business Promotor

Useful Resources:

Internet Marketing Tutorials

WordPress Blogging Tutorial

Learn HTML

Google Cash Review

Website Directory


Disclaimer, Terms and Privacy Policy

© COPYRIGHT 2003 all rights reserved