SharePoint 2010 Search : Search Extensions - Commercial Solutions

1. User Interface Enhancements

Search user interface enhancements may range from simple formatting and branding with custom XSLT, to completely new navigational functionality that provides for more fluid navigation. Users may complain that they are returning too many results in a search result set, and that the SharePoint 2010 search refiner Web Part is not allowing them to refine on their necessary properties. This may be the result of poor metadata since the search refiner Web Part is simply allowing for pre-defined property restrictions to be executed on the search result set. Users may recognize that although they know a particular suggestion should be appearing in a refinement category, it isn't. This may be because the items with the property they want to use do not appear until late in the result set, and SharePoint 2010's search refiner Web Part considers only the first 50 items. They may want to see how many results are included in each refinement option. Users may notice that while they can navigate into the site refinement category, they cannot backtrack without starting over. The list of requests for improvement could continue, and, depending on the user's needs, they are valid considering the limitations of just the search refinement Web Part included with the platform. This does not even consider the additional features users may request that are not even within the scope of the Web Parts included with SharePoint.

Since the way users work with SharePoint is many times quite different than how site designers originally expected, anyone planning a SharePoint deployment should expect demands for improvements. The user interface is generally the first point of complaint to be brought to the design team's attention because it is the only layer users directly touch. Unlike the components that drive the user interface, when users notice something wrong or a feature they would like to have, this is the only layer where they may be able to easily express their needs. If metadata or taxonomy is poor, users unfamiliar with the technology know there is an issue, but can rarely express the exact problem. By contrast, if a user knows he or she needs the site to include a tag cloud like the blog he or she regularly visits, that need is more easily verbalized. SharePoint designers and administrators should be prepared to handle these user interface needs during initial testing and well after a rollout.

1.1. Individual Commercial Web Parts

To meet the broad and evolving range of search user interface needs, many vendors have brought purchasable solutions to market. The individual Web Parts available from companies such as Bamboo Solutions and SharePoint Boost provide a starting point for meeting specific user enhancement needs. Bamboo Solutions, for example, offers a List Search Web Part. The Web Part allows users to search multiple lists at once, and return the results back into a single consolidated list. In SharePoint 2010, prior to this customization, searches can be directed at either all lists, or one list at a time, but cannot combine results from several specific lists. The Web Part can accept the standard SharePoint 2010 BOOLEAN operators in a query and then return results with customized columns and sorts. After search results are returned from the lists, the user can rearrange column orders and print the search results. This feature has a number of real-world applications such as searching for client information scattered among several lists or searching for a product across several catalog lists. The List Search Web Part is available in simple and advanced versions that cater to different customization needs. The features of each option can be found in Table 1.

Unlike open source solutions, such as those found on CodePlex, these solutions are tested and supported. While caution should always be used whether testing new features from open source, Microsoft, or other commercial vendors, a significantly greater trust can be put into well-reviewed commercial solutions. Although commercial solutions may require an initial financial investment, when compared with the costs of fixing broken solutions or repairing a crashed SharePoint portal, the investment is easily justified in most cases. For most environments, the cost of lost productivity due to a portal crash dwarfs the cost of software licensing. Individual Web Parts are generally available at a fairly low cost and can be obtained through an online purchase system. Although all vendors in this space have different price structures, straightforward Web Parts can be expected to cost less than US$1,000.

For environments that require more complex enhancements to the search user interface, there are many other solution packages on the market. These solutions usually include a broad range of search enhancements including various features for refining search results, suggesting alternative queries, taking action on results, entering more advanced queries, or integrating search into other business processes. These solution sets include a package of Web Parts to directly enhance the user experience, but may also add additional functions such as databases for analytics or workflows for search-based processes. They can also be categorized into two distinctly different paradigms: those that extend SharePoint search, and those that replace it.

2. Comprehensive Commercial Search Extensions

Extensions that directly expand the search functionality of SharePoint 2010 are offered in solutions such as SurfRay's Ontolica Search and BA Insight's Longitude. Search extensions of this kind can sit on top of SharePoint 2010 and leverage the search architecture that is already in place. This provides for some benefits and downsides when compared with the more well-known "enterprise" search solutions such as FAST, Google, and Autonomy. By utilizing SharePoint 2010's search pipeline, administrators need to manage only one crawler and index. This keeps management in one consolidated interface, instead of spreading jobs out between several different search solutions. For organizations without extra expendable employee hours each week, the time saved not having to manage two sets of physical and software architecture is invaluable. In addition, by using only enhancements that directly build on top of SharePoint, security and permissions are preserved. For organizations with strict security and privacy concerns, the avoidance of managing additional permission sets is critical to controlling availability of restricted information.

2.1. Ontolica Search

Ontolica Search, the SharePoint search extension offered by Microsoft partner SurfRay, can add a broad range of additional search features to a SharePoint environment. Unlike the individual Web Parts offered by vendors such as Bamboo Solutions, which meet one specific need, Ontolica Search addresses a broader need for search improvements. All of the features in Ontolica rely on the SharePoint search engine. They pass queries to SharePoint, retrieve information, display that information to users, and allow manipulation of the returned data. While there are additional functionalities in the solution, search features can be categorized as focusing on the four main areas.

• Faster access to information

• Refinements to filter results to the most relevant items

• Actionable results to facilitate business processes

• Experience populated suggestions for query adjustments

2.2. Accessing Information Quickly

Ontolica Search connects users with information faster from any page with a query box. While SharePoint 2010 includes the ability to enable query boxes to present thesaurus-generated suggestions, this solution can provide a list of the top search results for the entered query. As a user enters a query, Ontolica Search returns a list of the most relevant results, which can then be actioned (Figure 1). This allows users to check out the more relevant items directly from the page they are on without ever navigating to the search center.

Figure 1. Ontolica Search As You Type feature

2.3. Enhanced Refinements

When it comes tothe search center, users find a broad range of additional refinement options beyond those available with SharePoint 2010. The search refiner Web Part has been upgraded to present the number of results in each refinement suggestion (Figure 2). The site refiner category allows users to drill in and back out of taxonomies using a breadcrumb navigator. Also, possibly most important, the refinements can analyze up to 2,000 items in a result set instead of just the first 500.

Figure 2. Ontolica search refiner Web Part

In addition to the search refiner Web Part, Ontolica Search also adds the ability to rearrange search results and group them by managed properties. Unlike the search refiner Web Part, which allows users to select one property from a result set, this option allows users to see multiple properties but skip past those that are irrelevant. Finally, the solution provides the ability for administrators to set up quick filters. These are radio buttons or check boxes below the query field that allow users to select pre-defined combinations of property restrictions. SharePoint administrators can customize these to offer users an alternative to building common property restrictions on the advanced search page. These features can be seen in the following figure of an uncustomized Ontolica search center on SharePoint 2010 (Figure 3).

Figure 3. Ontolica search center on SharePoint 2010

2.4. Actionable Results

Actionable items and documents are a common feature throughout SharePoint, but the ability is lacking in search. Ontolica Search provides a drop-down on search results so that users can execute actions on them (Figure 4). The default action taken when clicking different search result types in SharePoint 2010 . This drop-down provides a broader range of default actions, such as the ability to check out Microsoft Office documents directly from the search results page, setting alerts on individual results instead of entire result sets, and executing an e-mail containing the item's link through Microsoft Outlook. In addition to these default actions, custom actions, processes, and workflows can be added.

Figure 4. Ontolica search result actions

2.5. Query Suggestions

The final focus of Ontolica Search is to provide experience-driven query suggestions. On the back end, Ontolica Search contains a business intelligence database cube that gathers a wide range of statistical data from the SharePoint farm. This data is gathered from IIS logs, Central Admin logs, and logs from the use of the tool itself. Ontolica then processes this data and makes it available through a wide range of statistic reports, such as the most popular search terms, queries that returned no results or did not result in a user clicking a result, and storage trends. This information can be used for a wide range of business intelligence, marketing, and search engine optimization needs. It is also used to present query suggestions to end users.

Ontolica Search presents suggestions in three formats, which include autosuggestions in the query field, related queries, and a tag cloud Web Part. When a user is executing a search query in the search center, the autosuggest feature provides a drop-down of query suggestions similar to the SharePoint 2010 Search suggestions. The related query Web Part and tag cloud Web Parts guide users to additional queries that may be useful. Just like SharePoint 2010, these suggestions are based on the aggregated experiences of users. Where these suggestion features differ from SharePoint 2010 is in the scope of suggestions. SharePoint 2010 can provide suggestions based only on the experiences of all users within the farm. Ontolica's suggestion features, however, can be isolated to a search center, the user's department, or the entire company. This means that users can receive search suggestions based off of the experiences of users in their own department, and do not have to receive suggestions based off of the entire company. This is quite practical in that the types of queries done by people in the sales department are most likely very different from those executed by the HR department. Suggestions are not cluttered with those helpful only to users in other departments.

2.6. Customize Search without Programming

In addition to adding a broad range of additional refinement, action, and suggestion features to SharePoint 2010, Ontolica helps overcome the complications of customizing the search experience. Ontolica's features are easily customizable and as stated previously, can all be set to cater to specific user groups, even with only one search center. For example, users from the Human Resources, Marketing, and Engineering departments require very different experiences from search. To give each of these departments a different search experience in SharePoint, separate search pages would need to be created and maintained for each department. With Ontolica however, users can all access the same search page but received customized refinements, search suggestions, and relevancy.

Ontolica also provides administrators with simple tools to leverage metadata and expose it in the search UI without any custom development. For example, customizations such as adding new search refiners for custom properties or even a new search field on the advanced search page can be made by non-technical administrators. The following table summarizes the major differences between SharePoint 2010 and Ontolica Search.

3. Vendor Notes

Many vendors, including Bamboo Solutions and SurfRay, offer free trials of their solutions. This allows for testing prior to a purchase. Make sure to allocate appropriate time for setup of the solutions during testing. Ask vendors about the installation, setup, and customization time needed so that realistic time resources can be dedicated. It is also recommended to do research on the experiences of others who have tested products as vendor estimates may be for best-case scenarios, and not the norm. While first-hand experiences are the best judge, investigating the experiences of others may uncover problems with solutions that may not be immediately apparent.

Take note of system resource requirements and other secondary costs. Many solutions, require large amounts of setup time, regular maintenance, professional services, and additional hardware that can rapidly increase costs and time demands. Asking the right questions and doing sufficient research will help to paint a full picture of the real costs associated with the software and the amount of time needed to maintain it after rollout.

Ask users what they would like to see improved. While this should seem obvious, many times technical teams run off of the assumptions of what needs to be improved without directly asking those affected. While users may not be able to verbalize a solution, they can express their problems. If users complain that they get too many results in their search result set, look into improvements in scopes, filtering capability, and duplicates. If users complain that they can't find items they know exist, look into improvements in tagging, iFilters, permissions, and content sources. Here are a few questions that are essential to determining the improvements that will resolve real findability problems.

• Are users seeing too many irrelevant results high in their search result set? If this is the case, consider adding a custom sorting option so that users can shuffle results based on the properties that are important to them. Set up additional managed properties, and make the fields required to complete when users upload new items to SharePoint. Add additional refiners based on the managed properties that are most widely used. Leverage third-party Web Parts that provide results based on user context or add additional refinement capabilities. Create more specific document libraries, and create processes that make users upload their documents into their relevant libraries. Create scopes for users to search against specific content libraries.

• Are users unable to find items in search that they know exist? It is first important to check that the location of the content is accessible to SharePoint search. It is then necessary to make sure the content of the desired item is searchable. The content of certain file types, such as PDFs, is not accessible to SharePoint's index without the appropriate iFilter. Make sure that any necessary iFilters are installed and working. If SharePoint is able to correctly index items and they are still not appearing to users, check the items' managed properties. SharePoint places a large weight on managed properties when determining relevancy. Adding more defined properties and clarifying ambiguity greatly increases SharePoint search relevancy. This can be done manually or with third-party classification solutions. If users are using keywords that are not within the content or properties of a document, create synonyms to tie their query terms to keywords associated with the document. For frequently required items, set up a best bet for query terms that promote the item to appear at the top of a search result set.

• Are users finding items they shouldn't be able to view? Check that user permissions have been correctly established. Make sure only those users who should have clearance to certain sites, content types, and content sources are set to access restricted content. This may be a problem with LDAP or Active Directory security. If permissions are set correctly, make sure that sensitive documents are correctly tagged as restricted content types.
  

Finally, it is always important for real users to go hands-on with solutions prior to production rollouts. Improvements are much harder to make once a solution has gone live. Set up a development environment that mirrors production, and have real users work on it for a week. Ask them for feedback on improvements and use the advice. If improvements are requested with a small test group, the complaints can be expected to be compounded on a production rollout.