Creating a custom Refinement Filter Generator


A few months ago I added a post on configuring the Refinement web part that ships with SharePoint 2010. The post is mainly focussed on the Filter Category Definition property and the XML that you can add in there to configure the Refinement panel to do what you need. Since then the good folks as MS have added more in-depth documentation to MSDN covering this topic.

In this post I want to move on from that and look at how you can develop your own custom filter generators and call them using the Filter Category Definition property. One situation where this will be necessary is if you’re try to use a multi-value managed property as a refinement. The default ManagedPropertyFilterGenerator doesn’t support this properly. So if you have a column with a Choice data type and you then create a managed property from that column, while you can create a refinement using the managed property, the refinement selections will include combinations of values. For example, if your Choice column has options Red, Green and Blue. Your refinement won’t contain just these options, it’ll contain the combinations that are used on items as shown:

image

While there is an argument for using Managed Metadata in place of Choice columns and then making use of the TaxonomyFilterGenerator, I’d imagine there are still some situations where a Choice is the way to go and for those folks, here’s how you can create a custom Refinement Filter Generator.

How refinements work

As described in my previous post, the RefinementManager object is a singleton that is shared among all components using refinement services on a page. The RefinementManager maintains a reference to a collection of FilterCategory objects where each FilterCategory represents a refinement type. In this image a FilterCategory is represented by the headings Result Type and Site.

image

Each Filter Category is configured with an associated FilterGenerator implementation. There are three possible implementations available out of the box: ManagedPropertyFilterGenerator (this is the most widely used), TaxonomyFilterGenerator and RankingModelFilterGenerator. The configuration for these categories and filter generators can be seen in the xml value of the Filter Category Definitions property and will look like this:

<Category    Title="Modified Date"    Description="When the item was last updated"    Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"    MetadataThreshold="5"    NumberOfFiltersToDisplay="6"    MaxNumberOfFilters="0"    SortBy="Custom"    ShowMoreLink="True"    MappedProperty="Write"    MoreLinkText="show more"    LessLinkText="show fewer" >
<CustomFilters MappingType="RangeMapping" DataType="Date" ValueReference="Relative" ShowAllInMore="False">
  <CustomFilter CustomValue="Past 24 Hours">
    <OriginalValue>-1..</OriginalValue>
  </CustomFilter>
  <CustomFilter CustomValue="Past Week">
    <OriginalValue>-7..</OriginalValue>
  </CustomFilter>
  <CustomFilter CustomValue="Past Month">
    <OriginalValue>-30..</OriginalValue>
  </CustomFilter>
  <CustomFilter CustomValue="Past Six Months">
    <OriginalValue>-183..</OriginalValue>
  </CustomFilter>
  <CustomFilter CustomValue="Past Year">
    <OriginalValue>-365..</OriginalValue>
  </CustomFilter>
  <CustomFilter CustomValue="Earlier">
    <OriginalValue>..-365</OriginalValue>
  </CustomFilter>
</CustomFilters>
</Category>

We can see that the Type attribute contains the class name for the FilterGenerator implementation.

Using this configuration, the RefinementManager then iterates through the configured categories calling the appropriate FilterGenerator instance. This is done using the virtual GetRefinement method that’s defined on the abstract RefinementFilterGenerator class. The signature of the GetRefinement method is:

        public override List<System.Xml.XmlElement> GetRefinement(Dictionary<string, Dictionary<string, RefinementDataElement>> refinedData,
                                                System.Xml.XmlDocument filterXml,
                                                int maxFilterCats)

From this you can see that the RefinementManager populates and passes in a reference to a dictionary of RefinementDataElement objects. This is where the problem comes in in with regard to supporting multi-value managed properties and ultimately with the Choice data-type. The refinedData dictionary is keyed using mapped property names (i.e. the managed property name that you configured in the Search service application). Each RefinementDataElement is an instance of a particular mapped property within the result set. This means that it contains the value of the mapped property for a particular search result. So if a search result has multiple values for a mapped property, there is still only one corresponding RefinementDataElement and it’s value contains a ‘;’ concatenated list. Since the implementation of the ManagedPropertyFilterGenerator uses RefinementDataElements as the basis for generating filters, we end up with the concatenated lists being output as filters as we saw in the image above.

Parsing RefinementDataElements

To get round this limitation of the ManagedPropertyFilterGenerator we need to create a custom RefinementFilterGenerator that performs further parsing of the RefinementDataElement to extract individual values. We can use code similar to:

foreach (RefinementDataElement item in filterResults.Values)
{
    string[] actual = item.FilterDisplayValue.Split(';');
    foreach (string x in actual)
    {
        RefinementDataElement element;
        if (newResults.ContainsKey(x))
        {
            element = newResults[x];
        }
        else
        {
            element = new RefinementDataElement(x, 0, 0);
            newResults.Add(x, element);
        }
        element.FilterValueCount = element.FilterValueCount + item.FilterValueCount;
        resultSum = resultSum + item.FilterValueCount;
        itemCount = (long)(itemCount + item.FilterValueCount);
    }
}

Step-by-step guide

1. Using Visual Studio 2010, create a new Empty SharePoint Project names MultiValudFilterGenerator.

2. Add a new class file named MultiValueFilterGenerator.cs and add the following code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Office.Server.Search.WebControls;
using Microsoft.SharePoint.Utilities;
using System.Xml;
using System.Collections;
using System.Globalization;
using System.Collections.Specialized;
using System.Text.RegularExpressions;
using Microsoft.SharePoint;
using System.Web;

namespace MultiValueFilterGenerator
{
    class MultiValueFilterGenerator : RefinementFilterGenerator
    {

        public MultiValueFilterGenerator()
        {
        }

        public override List<System.Xml.XmlElement> GetRefinement(Dictionary<string, Dictionary<string, RefinementDataElement>> refinedData,
                                                                    System.Xml.XmlDocument filterXml,
                                                                    int maxFilterCats)
        {

            List<XmlElement> result = new List<XmlElement>();
    
            foreach (FilterCategory category in base._Categories)
            {

                long itemCount = 0L;

                Dictionary<string, RefinementDataElement> filterResults = refinedData.ContainsKey(category.MappedProperty) ? refinedData[category.MappedProperty] : new Dictionary<string, RefinementDataElement>();

                Dictionary<string, RefinementDataElement> newResults = new Dictionary<string, RefinementDataElement>();

                //rebuild results
                long resultSum = 0;
                foreach (RefinementDataElement item in filterResults.Values)
                {
                    string[] actual = item.FilterDisplayValue.Split(';');
                    foreach (string x in actual)
                    {
                        RefinementDataElement element;
                        if (newResults.ContainsKey(x))
                        {
                            element = newResults[x];
                        }
                        else
                        {
                            element = new RefinementDataElement(x, 0, 0);
                            newResults.Add(x, element);
                        }
                        element.FilterValueCount = element.FilterValueCount + item.FilterValueCount;
                        resultSum = resultSum + item.FilterValueCount;
                        itemCount = (long)(itemCount + item.FilterValueCount);
                    }
                }


                if (itemCount >= category.MetadataThreshold && itemCount != 0)
                {

                    //Calculate precentages
                    foreach (RefinementDataElement item in newResults.Values)
                    {
                        item.FilterValuePercentage = item.FilterValueCount / resultSum;
                    }

                    //Order by percentage
                    IEnumerable<RefinementDataElement> topResults = newResults.Values.OrderBy(i => i.FilterValuePercentage);

                    XmlElement element = filterXml.CreateElement("FilterCategory");
                    element.SetAttribute("Id", category.Id);
                    element.SetAttribute("ConfigId", category.Id);
                    element.SetAttribute("Type", category.FilterType);
                    element.SetAttribute("DisplayName", RefinementFilterGenerator.TruncatedString(category.Title, base.NumberOfCharsToDisplay));
                    element.SetAttribute("ManagedProperty", category.MappedProperty);
                    element.SetAttribute("ShowMoreLink", category.ShowMoreLink);
                    element.SetAttribute("FreeFormFilterHint", category.FreeFormFilterHint);
                    element.SetAttribute("MoreLinkText", category.MoreLinkText);
                    element.SetAttribute("LessLinkText", category.LessLinkText);
                    element.SetAttribute("ShowCounts", category.ShowCounts);

                    XmlElement containerElmt = filterXml.CreateElement("Filters");

                    string url = string.Empty;

                    bool selectable = BuildRefinementUrl(category, string.Empty, out url);

                    containerElmt.AppendChild(GenerateFilterElement(filterXml,
                        RefinementFilterGenerator.TruncatedString("Any " + category.Title, base.NumberOfCharsToDisplay),
                        url, selectable, "",
                        "",
                        "",
                        ""));

                    foreach (RefinementDataElement item in topResults.Take(category.NumberOfFiltersToDisplay))
                    {

                        selectable = BuildRefinementUrl(category, item.FilterDisplayValue, out url);

                        containerElmt.AppendChild(GenerateFilterElement(filterXml,
                            RefinementFilterGenerator.TruncatedString(item.FilterDisplayValue, base.NumberOfCharsToDisplay),
                            url, selectable, item.FilterDisplayValue,
                            item.FilterValueCount.ToString(),
                            item.FilterValuePercentage.ToString(),
                            ""));
                    }

                    element.AppendChild(containerElmt);
                    result.Add(element);
                }
            }
            return result;

        }
        private static XmlElement GenerateFilterElement(XmlDocument filterXml,
            string truncatedFilterDisplayValue,
            string url,
            bool selectable,
            string filterTooltip,
            string count,
            string percentage,
            string filterIndentation)
        {
            XmlElement element2 = filterXml.CreateElement("Filter");
            XmlElement newChild = filterXml.CreateElement("Value");
            newChild.InnerText = truncatedFilterDisplayValue;
            element2.AppendChild(newChild);
            newChild = filterXml.CreateElement("Tooltip");
            newChild.InnerText = filterTooltip;
            element2.AppendChild(newChild);
            newChild = filterXml.CreateElement("Url");
            newChild.InnerText = url;
            element2.AppendChild(newChild);
            newChild = filterXml.CreateElement("Selection");
            if (selectable)
            {
                newChild.InnerText = "Deselected";
            }
            else
            {
                newChild.InnerText = "Selected";
            }
            element2.AppendChild(newChild);
            newChild = filterXml.CreateElement("Count");
            newChild.InnerText = count;
            element2.AppendChild(newChild);
            newChild = filterXml.CreateElement("Percentage");
            newChild.InnerText = percentage;
            element2.AppendChild(newChild);
            if (!string.IsNullOrEmpty(filterIndentation))
            {
                newChild = filterXml.CreateElement("Indentation");
                newChild.InnerText = filterIndentation.ToString();
                element2.AppendChild(newChild);
            }
            return element2;
        }

        protected bool BuildRefinementUrl(FilterCategory fc, string value, out string url)
        {
            string filteringProperty = fc.MappedProperty;
            bool notSelected = false;
            string origFilter = string.Empty;
            bool hasRefinement = false;

            hasRefinement = (HttpContext.Current.Request.QueryString["r"] != null);

            if (!hasRefinement)
            {
                origFilter = string.Empty;
            }
            else
            {
                origFilter = HttpContext.Current.Request.QueryString["r"];
            }

            string filterString = origFilter;

            if (string.IsNullOrEmpty(value))
            {
                //Remove any filters for this category
                int num = filterString.Length;
                filterString = this.RemoveCategoryFromUrl(filterString, fc);
                //if teh value changed then all values were not selected
                notSelected = (bool)(filterString.Length != num);
            }
            else
            {
                notSelected = false;
                string propertyValue = filteringProperty.ToLower() + ":" + value;

                if (!filterString.Contains(propertyValue))
                {
                    notSelected = true;
                }

                if (notSelected)
                {
                    Regex regex = new Regex(string.Format("(((({0})(:|>|<|<=|>=|=)\"([^\"]|\"\")*\"))|((({0})(:|>|<|<=|>=|=)([^\\s]*))))((\\s+AND\\s+)(((({0})(:|>|<|<=|>=|=)\"([^\"]|\"\")*\"))|((({0})(:|>|<|<=|>=|=)([^\\s]*)))))*", filteringProperty), RegexOptions.IgnoreCase);
                    MatchCollection matchs2 = regex.Matches(filterString);
                    StringBuilder builder2 = new StringBuilder();
                    builder2.Append(" " + propertyValue);
                    foreach (Match match2 in matchs2)
                    {
                        builder2.Append(" AND ");
                        builder2.Append(match2.Value);
                    }
                    filterString = regex.Replace(filterString, string.Empty) + builder2.ToString();
                }
                else
                {
                    StringBuilder builder = new StringBuilder();
                    Regex regex2 = new Regex(string.Format("({0}(?<Operator>:|>|<|<=|>=|=)\"(?<FilterValue>([^\"]|\"\")*)\"(\\s|$))|({0}(?<Operator>:|>|<|<=|>=|=)(?<FilterValue>[^\\s]*)(\\s|$))", filteringProperty), RegexOptions.IgnoreCase);
                    //get a list of values for this category
                    foreach (Match match in regex2.Matches(filterString))
                    {
                        if ((match != null) && !string.IsNullOrEmpty(match.Value))
                        {
                            string trimmedValue = match.Value.Trim();
                            if (trimmedValue!=propertyValue)
                            {
                                if (builder.Length > 0)
                                {
                                    builder.Append(" AND ");
                                }
                                builder.Append(trimmedValue);
                            }
                        }
                    }
                    filterString = this.RemoveCategoryFromUrl(filterString, fc);
                    if (builder.Length > 0)
                    {
                        filterString = filterString + " " + builder.ToString();
                    }
                }
            }

            filterString = HttpUtility.UrlEncode(filterString.Trim());
            string originalUrl = HttpContext.Current.Request.Url.OriginalString;

            Uri request = HttpContext.Current.Request.Url;
            NameValueCollection queryString = HttpContext.Current.Request.QueryString;

            originalUrl = request.OriginalString.Replace(request.Query, string.Empty);
            string qs = string.Empty;

            foreach (string key in queryString.AllKeys)
            {
                if (key != "r")
                {
                    qs = qs + "&" + key + "=" + queryString[key].ToString();
                }
            }

            qs = qs + "&r=" + filterString;

            url = originalUrl + "?" + qs.Substring(1);

            return notSelected;
        }

        private string RemoveCategoryFromUrl(string currentUrl, FilterCategory fc)
        {
            string filteringProperty = fc.MappedProperty;
            string expression = string.Empty;

            if (fc.CustomFiltersConfiguration != null)
            {
                expression = string.Format(CultureInfo.InvariantCulture, "({0}(?<Operator>:|>|<|<=|>=|=)\"(?<FilterValue>([^\"]|\"\")*)\"(\\s|$))|({0}(?<Operator>:|>|<|<=|>=|=)(?<FilterValue>[^\\s]*)(\\s|$))", new object[] { filteringProperty });
            }
            else
            {
                expression = string.Format(CultureInfo.InvariantCulture, "(((({0})(:|>|<|<=|>=|=)\"([^\"]|\"\")*\"))|((({0})(:|>|<|<=|>=|=)([^\\s]*))))((\\s+AND\\s+)(((({0})(:|>|<|<=|>=|=)\"([^\"]|\"\")*\"))|((({0})(:|>|<|<=|>=|=)([^\\s]*)))))*", new object[] { filteringProperty });
            }
            Regex regex = new Regex(expression, RegexOptions.IgnoreCase);
            return regex.Replace(currentUrl, string.Empty);
        }

    }
}

3. Select Deploy Solution from the Build menu to compile the project and copy it to the GAC.

4. Navigate to C:\Windows\Assembly and find the entry for MultiValueFilterGenerator. Right-click to view the properties and copy the value for the PublicKeyToken.

5. On a page containing a Refinement web part, edit the XML in the  Filter Category Definition to include an entry similar to:

  <Category    Title="Color"    
               Description="Use this filter to restrict results to a specific color"    
               Type="MultiValueFilterGenerator.MultiValueFilterGenerator, MultiValueFilterGenerator, Version=1.0.0.0, Culture=neutral, PublicKeyToken=<Add Your Public Key Token Here>"    
               MetadataThreshold="0"    
               NumberOfFiltersToDisplay="4"    
               MaxNumberOfFilters="20"    
               SortBy="Frequency"    
               SortByForMoreFilters="Name"    
               SortDirection="Descending"    
               SortDirectionForMoreFilters="Ascending"    
               ShowMoreLink="True"    
               MappedProperty="ItemColor"    
               MoreLinkText="show more"    
               LessLinkText="show fewer"    />

Note: Make sure that the Use Default Configuration checkbox is Unchecked

image

6. Click on OK to commit the changes and then switch out of edit mode.

If all is well in the world, you’ll now have refinements for each value in our multi-value Managed Property as shown in this image (I’ve left the default implementation in the config to show the difference).

image

Summary

This code is a long way from production ready and there are a few features that are missing such as a “More Items” link. Hopefully it’s enough to give an idea of how the RefinementManager does it’s magic. AS always – any comments, criticisms are questions are more than welcome.

Hope this saves somebody some time!

author: Charlie Holland | posted @ Friday, August 20, 2010 3:41 PM | Feedback (0)

Using the HOSTS file for SharePoint Development


When doing development on a SharePoint dev server, there may be the need to create multiple web applications. To prevent conflicts there are two possibilities: either use host headers or use a different port. To support host headers we can either add additional DNS entries or use the HOSTS file (C:\Windows\System32\drivers\etc).

Since my dev servers generally run as isolated VM’s, creating DNS entries isn’t an option. As a result I’ll commonly make modifications to the HOSTS file such as:

 

# Copyright (c) 1993-2006 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

127.0.0.1       localhost
::1             localhost
127.0.0.1    SP2010DEV
127.0.0.1    SP2010DEV2
 
The problem with this approach is that when attempting to a web application that’s been configured in this manner from the dev server, Internet Explorer will be unable to connect.

This problem isn’t specific to SharePoint, it’s actually a security feature that’s been built into Windows 2003 and later. It just so happens that when we’re building SharePoint applications, we generally do development on the server and therefore the problem raises it’s head.

Solution

Thankfully this problem has an easy solution:

  1. Click Start, click Run, type regedit, and then click OK.
  2. In Registry Editor, locate and then click the following registry key:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa

  3. Right-click Lsa, point to New, and then click DWORD Value.
  4. Type DisableLoopbackCheck, and then press ENTER.
  5. Right-click DisableLoopbackCheck, and then click Modify.
  6. In the Value data box, type 1, and then click OK.

For further details of this problem and the implications of making this change see: http://support.microsoft.com/kb/896861

author: Charlie Holland | posted @ Friday, June 04, 2010 2:21 PM | Feedback (0)

Recycling SharePoint 2010 app pool from Visual Studio


One of the things I liked about WSPBuilder was the ability to easily recycle the application pool from Visual Studio, much quicker then messing around with IISReset or IIS manager. With the new deployment framework in Visual Studio 2010, we don’t have that functionality anymore. It’s all very much about deployment. While there are benefits to this approach, sometimes when you’re messing around with stuff it’s good to be able to copy a file to the SPRoot folder, recycle the app pool and see what happens.

This is just a quick post, more for my memory than anything else, on how to add a command to Visual Studio to recycle a particular app pool.

  1. In Visual Studio 2010, from the Tools menu, select External ToolsExternalTools
  2. Click on Add to add a new tool (A tool is effectively a menu option that points to an external tool)
  3. Set the Title to something descriptive (maybe Recycle SharePoint App Pool?)
  4. Set the Command to: C:\Windows\System32\inetsrv\appcmd.exe
  5. Set the Arguments to: recycle apppool /apppool.name:"Sharepoint - 80"
  6. Check the Use Output window option.

Job done.

author: Charlie Holland | posted @ Wednesday, April 07, 2010 11:58 AM | Feedback (0)