Sharepoint 2010 : Setting Up the Crawler - Federated Sources

Federated sources can be either SharePoint sites or sites that conform to the OpenSearch 1.0 or 1.1 standards. These standards define how search queries should be passed and how the data is structured and returned.

New federated sources can be defined or an existing template can be downloaded and imported. To create a new federated source, click New Location and fill out the fields with the appropriate settings. Every source requires a name that will also be used in the search center.

1. Creating a New Federated Source

When creating a new federated location, a name and a display name should be defined. The display name will be shown on the federated search Web Part. A description of the source is also required (see Figure 1).

Figure 1. Defining a new federated location

Triggers can be set on all federated locations to dictate what queries are sent to the federated locations and how they are sent.

There are three basic triggers:

• Always: Send all queries.

• Prefix: Use a trigger prefix to send queries.

• Pattern: Use a pattern trigger to send both the trigger and the query in a new pattern that may be adjusted with regular expressions.

If Always is chosen, any search query or phrase will also be passed to the federated source; if Prefix is set, it will look for a particular term and then pass the following terms as the query. In this way, the first term (the trigger term) can be used to indicate which federated source should be queried to supply this information (Figure 2). So if the Prefix trigger is "pics", it might be advisable to federate in Flickr or some other image search engine. If the Prefix trigger is "weather", it might be advisable to send the query to weather.com. This will require a certain level of training for end users so they understand which triggers will action which information to appear. Federated search results appear in a separate Web Part on the results page. 

Figure 2. Setting triggers in a new federated location

Location information must be provided. If an OpenSearch-compliant site is being used, OpenSearch should be set here. Additionally, a query template should be specified. This query should be the URL to the OpenSearch-compliant RSS feed where the query term is replaced by the {searchTerms} value. This is called a query template in the standard. This offers a certain amount of flexibility, and the {searchTerms} value can be mixed with other accepted parameters to provide more exact results. For example, on Google's blog search feed, we can filter into the result set by clicking some of the options on the page. It is possible to sort the result set by date, and the parameter scoring=d is added to the query URI. This list of parameters can be captured from the address bar of the browser and modified with the {searchTerms} value to create a query template that returns results sorted by date. Not all parameters are supported by RSS feeds as they are generally separate applications that are not a main concern for the search providers. However, often RSS feeds and the OpenSearch definitions will support some added functionality.

For the OpenSearch-compliant RSS feed for blog posts, search from Google, http://blogsearch.google.com/blogsearch_feeds?hl=en-us&q=sharepoint&lr=&ie=utf-8&num=10; if we replace the query term "sharepoint" with the {searchTerms} value and add the parameter scoring=d, we will get the query template http://blogsearch.google.com/blogsearch_feeds?hl=en-us&q=sharepoint&lr=&ie=utf-8&num=10&scoring=d, which can be used to create a federated source that retrieves blog posts sorted by date.

The properties displayed can also be modified by modifying XSLT in the Display Information section. Depending on which provider is federated, it may be necessary to provide a custom XSLT. Providers do provide an OpenSearch declaration that can be used as a starting point, and the example XSLT definitions from the locations available for import will help.

2. Importing a Federated Location from Microsoft's Federated Search Connector Gallery

To import a location, a preconfigured federated location must be downloaded and imported. The available federated location downloads are available for download from Microsoft's Federated Search Connector Gallery for Enterprise Search: http://technet.microsoft.com/en-us/enterprisesearch/ff727944.aspx. Two file types are given, an FLD file and an OSDX file. SharePoint 2010 requires the OSDX file. Earlier SharePoint search products require the FLD files. They are basically just XML files with the settings for the federated locations. They can be edited manually, but entering a new location will be easier.

After downloading the federated location definition file, it can be imported with the import command and eventually edited to match the required settings.

It is also possible to define a federated location as a search index on this server. This will provide search results in the Federated Results Web Parts from a locally stored index. This is not the setting for federating results from a remote SharePoint site. To get federated content from another SharePoint server, use the OpenSearch specification and the RSS feed of the search results from the remote SharePoint site.