Microsoft Content Management Server : Influencing Search Engines with the ROBOTS META Tag

To allow MCMS authors to benefit from this feature, channels and postings have two additional properties:

• IsRobotIndexable

• IsRobotFollowable

These two properties can be used to generate the ROBOTS META tag. As already mentioned, be aware that some search engines choose to ignore the ROBOTS META tag.

The table below summarizes the behavior of the different property settings:

For more information on the ROBOTS META tag, check out this resource:

http://www.robotstxt.org/wc/exclusion.html#meta

The RobotMetaTag Control and Channel Rendering Scripts

MCMS ships with a control named RobotMetaTag, which is automatically added to all template files created from the Visual Studio .NET “MCMS Template File” template. The RobotMetaTag control is used to generate the ROBOTS META tag based on the IsRobotFollowable and IsRobotIndexable properties of the page.

Unfortunately this control does not work on channel rendering scripts. If you add the control to a channel rendering script, despite setting the IsRobotFollowable and IsRobotIndexable properties of the channel, the ROBOTS META tag does not get generated. To get around this problem let’s implement an enhanced version of this control that works with both channel rendering scripts and templates.

First, open the Tropical Green solution. Add a new class file to the TropicalGreenControlLib project with the name EnhancedRobotMetaTag.cs.

The control requires the use of methods from the namespaces highlighted below, so add them to the class file.

using System;

using System.ComponentModel;
						using System.Web.UI;
						using Microsoft.ContentManagement.Common;
						using Microsoft.ContentManagement.Publishing;
						using Microsoft.ContentManagement.WebControls;

namespace TropicalGreenControlLib
{
  . . . code continues . . .
}

Also, as we are creating a custom server control, the class will inherit from the RobotMetaTag class (in the Microsoft.ContentManagement.WebControls namespace).

namespace TropicalGreenControlLib
{
  /// <summary>
  /// Summary description for EnhancedRobotMetaTag.
  /// </summary>
  public class EnhancedRobotMetaTag : RobotMetaTag
  {
    public EnhancedRobotMetaTag()
    {
    }
  }
}

Above the EnhancedRobotMetaTag constructor, declare four constants, Index, Follow, noIndex and noFollow, which store the strings we will use later for generating the ROBOTS META tag. We will also remove the constructor, EnhancedRobotMetaTag(), as it is not required.

public class EnhancedRobotMetaTag : RobotMetaTag
{
  private const string Index = "INDEX";
						private const string Follow = "FOLLOW";
						private const string noIndex = "NOINDEX";
						private const string noFollow = "NOFOLLOW";
}

In addition to rendering the ROBOTS tag, the original RobotMetaTag control has a different feature: it provides an option to render a <base> tag, which ensures that relative links added to the template files work correctly. Here’s an example of a <base> tag:

<base
href="http://localhost/TropicalGreen/Templates/GardenPublic.aspx?NRMODE=Publis
hed&NRORIGINALURL=%2fTropicalGreen%2fGardens%2fMembers%2ehtm&NRNODEGUID=%7bB95
DD11B-2954-4F84-AB60-AC0A3D39FC49%7d&NRCACHEHINT=NoModifyGuest">

					  

As a single template file can be used by multiple postings, the URL varies from posting to posting. Links relative to the template file will not work in this situation as the browser does not know that postings and channels do not exist as complete entities on the file system. The base tag tells the browser what should be treated as the current location and allows the browser to calculate the final link based on the base tag and the information in the relative link. For example, should the template file contain a relative link, say, <a href="MembersOnly/SomePage.htm">A Relative Link</a>, the browser will treat the URL as if "MembersOnly/SomePage.htm" was pre-pended with the URL stored in the href attribute of the <base> tag.

As we are going to implement a control that extends the RobotMetaTag control, we need to ensure that this feature is also included in our new control.

We will write the ROBOTS META tag and <base> tag by overriding the Render() method of the control. Notice we look at the properties of a ChannelItem object. This ensures that the code works for both postings and channel rendering scripts.

protected override void Render( HtmlTextWriter writer )
{
  ChannelItem currentItem = CmsHttpContext.Current.ChannelItem;

  if( currentItem != null )
  {
    string followKey = (currentItem.IsRobotFollowable)?Follow:noFollow;
    string indexKey = (currentItem.IsRobotIndexable)?Index:noIndex;

    // write the robot META tag
    writer.Write("<meta name=\"ROBOTS\" "
               + "content=\""+followKey+","+indexKey+"\">\n");

    if( this.RenderBaseHref )
    {
      // write the Base tag
      writer.Write("<base href=\""+Page.Request.Url.AbsoluteUri+"\">\n");
    }
  }
}

Save and build the solution. Add the control to the toolbar as you would other custom controls and it’s ready to be dropped into channel rendering scripts or template files between the <head> tags.