User-agent: *

Disallow:

The User-agent field is for specifying robot name for which the access policy follows in the Disallow field. Disallow field specifies URLs which the specified robots have no access to. An example:

User-agent: *

Disallow: /

Here “*” means all robots and “/ ” means all URLs. This is read as, “No access for any search engine to any URL” Since all URLs are preceded by “/ ” so it bans access to all URLs when nothing follows after “/ “. If partial access has to be given, only the banned URL is specified in the Disallow field. Lets consider this example:

# Research access for Googlebot.

User-agent: Googlebot

Disallow:

User-agent: *

Disallow: /concepts/new/

Here we see that both the fields have been repeated. Multiple commands can be given for different user agents in different lines. The above commands mean that all user agents are banned access to /concepts/new/ except Googlebot which has full access. Characters following # are ignored up to the line termination as they are considered to be comments.

Working with the robots.txt file

1. The robots.txt file is always named in all lowercase (e.g. Robots.txt or robots.Txt is incorrect)

2. Wildcards are not supported in both the fields. Only * can be used in the User-agent field’s command syntax because it is a special character denoting “all”. Googlebot is the only robot that now supports some wildcard file extensions.
Ref: http://www.google.com/support/webmasters/bin/topic.py?topic=8475

3. The robots.txt file is an exclusion file meant for search engine robot reference and not obligatory for a website to function. An empty or absent file simply means that all robots are welcome to index any part of the website.

4. Only one robots.txt file can be maintained per domain.

5. Website owners who do not have administrative rights cannot sometimes make a robots.txt file. In such situations, the Robots Meta Tag can be configured which will solve the same purpose. Here we must keep in mind that lately, questions have been raised about robot behavior regarding the Robots Meta Tag. Some robots might skip it altogether. Protocol makes it obligatory for all robots to start with the robots.txt thereby making it the default starting point for all robots.

6. Separate lines are required for specifying access to different user agents and Disallow field should not carry more than one command in a line in the robots.txt file. There is no limit to the number of lines though i.e. both the User-agent and Disallow fields can be repeated with different commands any number of times. Blank lines will also not work within a single record set of both the commands.

7. Use lower-case for all robots.txt file content. Please also note that filenames on Unix systems are case sensitive. Be careful about case sensitivity when defining directory or files for Unix hosted domains.

You can use the robots.txt Validator to check your robots.txt from www.searchengineworld.com.

Advantages of the robots.txt file
Protocol demands that all search engine robots start with the robots.txt file. This is the default entry point for robots if the file is present. Specific instructions can be placed on this file to help index your site on the web. Major search engines will never violate the Standard for Robots Exclusion.

1. The robots.txt file can be used to keep out unwanted robots like emailing retrievers, imaging strippers etc.

2. The robots.txt file can be used to specify the directories on your server that you don’t want robots to access and/or index e.g. temporary, cgi, and private/back-end directories.

3. An absent robots.txt file could generate a 404 error and redirect the robot to your default 404 error page. Here it was noticed after careful research that sites that do not have a robots.txt file present and had a customized 404-error page, would serve the same to the robots. The robot is bound to treat it as the robots.txt file, which can confuse it’s indexing.

4. The robots.txt file is used to direct select robots to relevant pages to be indexed. This specially comes in handy where the site has multilingual content or where the robot is searching for only specific content.

5. The need for the robots.txt file was also felt to stop robots from deluging servers with rapid-fire requests or re-indexing the same files repeatedly. If you have duplicate content on your site for any reason, the same can be controlled from getting indexed. This will help you avoid any duplicate content penalties.

Disadvantages of the robots.txt file
Careless handling of directory and filenames can lead hackers to snoop around your site by studying the robots.txt file, as you sometimes may also list filenames and directories that have classified content. This is not a serious issue as deploying some effective security checks to the content in question can take care of it. For example if you have your traffic log on your site on a URL such as www.domain.com/stats which you do not want robots to index, then you would have to add a command to your robots.txt file. As an example:

User-agent: *

Disallow: /stats/

However, it is easy for a snooper to guess what you are trying to hide and simply typing the URL www.domain.com/stats in his browser would enable access to the same. This calls for one of the following remedies -

1. Change file names:
Change the stats filename from index.php to something different, such as stats- new.php so that your stats URL now becomes www.domain.com/stats/stats-new.php

Place a simple text file containing the text, “Sorry you are not authorized to view this page”, and save it as index.php in your /stats/directory.

This way the snooper cannot guess your actual filename and get to your banned content.

2. Use login passwords:
Password-protect the sensitive content listed in your robots.txt file.

Optimization of the robots.txt file
The Right Commands in robots.txt :
Use correct commands. Most common errors include – putting the command meant for “User-agent” field in the “Disallow field” and vice-versa.

Please also note that there is no “Allow” command in the standard robots.txt protocol. Content not blocked in the “Disallow” field is considered allowed. Currently, only two fields are recognized: “The User-agent field” and the “Disallow field”. Experts are considering the addition of more robot recognizable commands to make the robots.txt file more Webmaster and robot friendly.

Note     Note: Google is the only search engine, which is experimenting with certain new robots.txt commands.
It recognizes the “allow” command. Please read more details on the google site for robots.txt usage.

Bad Syntax:
Do not put multiple file URLs in one Disallow line in the robots.txt file. Use a new Disallow line for every directory that you want to block access to. Incorrect Robots.txt

Example:

User-agent: *

Disallow: /concepts/ /links/ /images/

Correct robots.txt example:

User-agent: *

Disallow: /concepts/

Disallow: /links/

Disallow: /images/

Files and Directories:
If a specific file has to be disallowed, end it with the file extension and without a forward slash in the end. Study the following robots.txt example:

For file:

User-agent: *

Disallow: /hilltop.phpl

For Directory:

User-agent: *

Disallow: /concepts/

Remember if you have to block access to all files in the directory, you don’t have to specify each and every file in robots.txt. You can simply block the directory as shown above. Another common error is leaving out the slashes altogether. This would leave a very different message than intended.

The Right Location for the robots.txt file:
No robot will access a badly placed robots.txt file. Make sure that the location is www.domain.com/robots.txt.

Capitalization in robots.txt
Never capitalize your syntax commands. Directory and filenames are case sensitive in Unix platforms. The only capitals used per standard are: “User-agent ” and “Disallow”

Correct Order for robots.txt :
If you want to block access to all but one or more than one robot, then the specific ones should be mentioned first. Lets study this robots.txt example:

User-agent: *

Disallow: /

User-agent: MSNbot

Disallow:

In the above case, MSNbot would simply leave the site without indexing after reading the first command. Correct syntax is:

User-agent: MSNbot

Disallow:

User-agent: *

Disallow: /

The robots.txt file :
Not having a robots.txt file at all could generate a 404 error for search engine robots, which could redirect the robot to the default 404-error page or your customized 404-error page. If this happens seamlessly, it is up to the robot to decide if the target file is a robots.txt file or an html file. Typically it would not cause many problems but you may not want to risk it. It’s always a better idea to put the standard robots.txt file in the root directory, than not having it at all.

The standard robots.txt file for allowing all robots to index all pages is:

User-agent: *

Disallow:

Using # Carefully in the robots.txt file:
Adding comments after the syntax commands is not a good idea using “#”. Some robots might misinterpret the line although it is acceptable as per the robots exclusion standard. New lines are always preferred for comments.

Using the robots.txt file

* Robots are configured to read text. Too much graphic content could render your pages invisible to the search engine. Use the robots.txt file to block irrelevant and graphic-only content.

* Indiscriminate access to all files, it is believed, can dilute relevance to your site content after being indexed by robots. This could seriously affect your site’s ranking with search engines. Use the robots.txt file to direct robots to content relevant to your site’s theme by blocking the irrelevant files or directories.

* The robots.txt file can be used for multilingual websites to direct robots to relevant content for relevant topics for different languages. It ultimately helps the search engines to present relevant results for specific languages. It also helps the search engine in its advanced search options where language is a variable.

* Some robots could cause severe server loading problems by rapid firing too many requests at peak hours. This could affect your business. By excluding some robots that might be irrelevant to your site, in the robots.txt file, this problem can be taken care of. It is really not a good idea to let malevolent robots use up precious bandwidth to harvest your emails, images etc.

* Use the robots.txt file to block out folders with sensitive information, text content, demo areas or content yet to be approved by your editors before it goes live.

The robots.txt file is an effective tool to address certain issues regarding website ranking. Used in conjunction with other SEO strategies, it can significantly enhance a website’s presence on the net.

DM

However, before you get excited about dynamic pages and race off to learn PHP, there is a downside to these web pages: Search engine bots cannot see them. And if they can’t be seen by the bots, they can’t be indexed by the engines.

The purpose of this article is to show you how you can index your dynamic pages with the search engines, and some of the pitfalls that you can avoid when submitting these to Google et al.

What is a Dynamic Page?

As mentioned above, a dynamic page is a static webpage that dynamically pulls it’s content from an external source – usually a web database such as mySQL.

This has many benefits, not least of all for the designer, whom only needs to create one webpage design. If he wishes to update his site, he need only change the one page and all pages presented to the end user will be changed too (as they all use the same template and change content dynamically, of course).

So you can use dynamic pages any time that you have information that is categorized by date or if you use an ecommerce shopping sites. Dynamic pages are also compatible with multiple service providers; so no need to worry about cross-browser support or migrating your site to another host (migrating the database is another issue).

You can spot dynamic pages as you surf around the web by looking at their URLs. Dynamic pages contain the character “?” or one of “#&*!%” within their link, such as www.yourdomain.com/?page=10

So What is the Issue?

The issue is the characters used in the URL. These instruct the webpage to send a command to the web server to fetch the content from the web database. So, in essence, the content isn’t retrieved unless the web page asks for it.

Googlebots, and other bots for that matter, cannot send commands to the web server, and so cannot retrieve the content. Remember: if they cannot see it, they cannot index it.

Additionally, some black hat SEO people and spammers use the same characters to trap search engine spiders. When you consider both of these reasons, it’s not hard to see why the spiders are both unable and unwilling (if you can say such a thing about a program) to index your dynamic pages.

So I’m Stuck Then?

Not at all. As webmasters, we have to be a little more imaginative in dealing with the issue of non-indexing and there are plenty of white hat SEO options available to you.

Make regular static pages that link to your dynamic pages
This one is really simple. On areas of your site that are static (and therefore can be indexed), add a link to your dynamic page.

Google and the others will be able to access the page via this link, and index the result. Also, please remember to optimise your anchor text too. You can read how to optimise anchor text in another article.

Make blogs or blog posts that link to your dynamic pages
Similar to the previous suggestion, but involves linking to one dynamic page from another. The difference is that, as well as linking from your blog/post, your blog post would also be submitted to various websites or feeds, such as Digg, DropJack etc.

As these sites are heavily crawled by Google, they will also index your blog articles. Once these are indexed, their “child” links will also be picked up too.

Optimize any dynamic pages that need to be indexed
Just because your page is dynamic, this is no reason to ignore the importance of on-the-page SEO.

Look at your metatags, including the title, and optimise wherever possible. Optimise the content itself. Good content attracts both search engines and visitors. Check your keywords and their densities. In short, do everything that you would do if the page were static.

Articles and content pages
If your dynamic pages are articles, such as a blog post, submit the article to webfeeds and sites, such as Digg.

As mentioned earlier, these sites are heavily crawled and indexed, and, even with a link to a dynamic page, Google will index your page with ease.

Link to your dynamic pages with a table of contents
Create a single page sitemap of your dynamic pages. Do this for your web visitors – not for the search engines – and optimise the anchor text.

The benefit of this is that the static sitemap page will be indexed quickly, depending on the directory depth of the page, and, likewise, the pages it links to will also be indexed.

Rewrite Your Page URLs with .htaccess
This is an extremely powerful, but potentially complex, means of dealing with the issue of dynamic page indexing.

Basically, a rewrite can change your url from yourdomain.com/?page=10 to yourdomain.com/dir/page10/. This works with Apache servers, and is well worth considering, if not implementing.

This rewrite rule will convert pages like this yourdomain/posts.php?page=1 to yourdomain/posts/page1

Options +FollowSymlinks
RewriteEngine on
RewriteRule ^files/([^/]+)/([^/]+).zip /posts.php?page=$1&file=$2 [NC]

Of course, this assumes that you are using PHP as your dynamic page language.

If you are interested in this option, the Apache documentation covers mod_rewrite in much more detail.

So if you really need dynamic pages, remember to set them up so that the Googlebots can see and record all the information on your site. As illustrated this is not an impossible task. It is just a question of working within the Google rules to help those bots to read all the information in your site.

DM