Why isn’t there a generic ReadOnlyDictionary in .net?

It’s trivial to implement but it’s one of those things that really should be in there as well as a generic ObervableDictionary. Anyways, here’s a basic implementation that’ll do the trick:

    public sealed class ReadOnlyDictionary<TKey, TValue> : IEnumerable<KeyValuePair<TKey, TValue>>
    {
        private readonly Dictionary<TKey, TValue> _dictionary;

        public ReadOnlyDictionary(Dictionary<TKey, TValue> dictionary)
        {
            if (dictionary == null) throw new ArgumentNullException("dictionary");
            _dictionary = dictionary;
        }
        
        public int Count
        {
            get { return _dictionary.Count; }
        }

        public bool IsReadOnly
        {
            get { return true; }
        }

        public TValue this[TKey key]
        {
            get { return _dictionary[key]; }
        }

        public ICollection<TKey> Keys
        {
            get { return new ReadOnlyCollection<TKey>(new List<TKey>(_dictionary.Keys)); }
        }

        public ICollection<TValue> Values
        {
            get { return new ReadOnlyCollection<TValue>(new List<TValue>(_dictionary.Values)); }
        }

        #region IEnumerable<KeyValuePair<TKey,TValue>> Members

        public IEnumerator<KeyValuePair<TKey, TValue>> GetEnumerator()
        {
            return _dictionary.GetEnumerator();
        }

        IEnumerator IEnumerable.GetEnumerator()
        {
            return _dictionary.GetEnumerator();
        }

        #endregion

        public bool ContainsKey(TKey key)
        {
            return _dictionary.ContainsKey(key);
        }

        public bool TryGetValue(TKey key, out TValue value)
        {
            return _dictionary.TryGetValue(key, out value);
        }
    }

Formatting dates in SharePoint when using the server object model

This is one of those things that you probably do a lot of if you’re building web parts and such like. Since each SPWeb has it’s own regional settings, determining the correct formatting for dates can be a bit of a pain. As is the case for many things in SharePoint, there is a utility to fix this problem. If you’re anything like me, you’ll read this and think ‘that’s handy’ and then forget about it until some time from now you find yourself trying to remember where you’d seen it!

You can find it at http://msdn.microsoft.com/en-us/library/microsoft.sharepoint.utilities.sputility.formatdate.aspx

and use it as follows:

  LastSynchronized.Text = SPUtility.FormatDate(SPContext.Current.Web, model.LastSynchronized,
                                                                 SPDateFormat.DateTime);
 

Building a custom refinement filter generator for SharePoint 2010

Update 30 September 2010 – I’ve added the source for this project here

Update 29 March 2011 – I’ve done a bit more work on the code for this article and packaged it up as a replacement webpart. It’s now available on CodePlex (http://sprefiner.codeplex.com/)

A few months ago I added a post on configuring the Refinement web part that ships with SharePoint 2010. The post is mainly focussed on the Filter Category Definition property and the XML that you can add in there to configure the Refinement panel to do what you need. Since then the good folks as MS have added more in-depth documentation to MSDN covering this topic.

In this post I want to move on from that and look at how you can develop your own custom filter generators and call them using the Filter Category Definition property. One situation where this will be necessary is if you’re try to use a multi-value managed property as a refinement. The default ManagedPropertyFilterGenerator doesn’t support this properly. So if you have a column with a Choice data type and you then create a managed property from that column, while you can create a refinement using the managed property, the refinement selections will include combinations of values. For example, if your Choice column has options Red, Green and Blue. Your refinement won’t contain just these options, it’ll contain the combinations that are used on items as shown:

image_thumb1

While there is an argument for using Managed Metadata in place of Choice columns and then making use of the TaxonomyFilterGenerator, I’d imagine there are still some situations where a Choice is the way to go and for those folks, here’s how you can create a custom Refinement Filter Generator.

How refinements work

As described in my previous post, the RefinementManager object is a singleton that is shared among all components using refinement services on a page. The RefinementManager maintains a reference to a collection of FilterCategory objects where each FilterCategory represents a refinement type. In this image a FilterCategory is represented by the headings Result Type and Site.

image_thumb

Each Filter Category is configured with an associated FilterGenerator implementation. There are three possible implementations available out of the box: ManagedPropertyFilterGenerator (this is the most widely used), TaxonomyFilterGenerator and RankingModelFilterGenerator. The configuration for these categories and filter generators can be seen in the xml value of the Filter Category Definitions property and will look like this:

<Category    Title="Modified Date"    Description="When the item was last updated"    Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"    MetadataThreshold="5"    NumberOfFiltersToDisplay="6"    MaxNumberOfFilters="0"    SortBy="Custom"    ShowMoreLink="True"    MappedProperty="Write"    MoreLinkText="show more"    LessLinkText="show fewer" >
<CustomFilters MappingType="RangeMapping" DataType="Date" ValueReference="Relative" ShowAllInMore="False">
  <CustomFilter CustomValue="Past 24 Hours">
    <OriginalValue>-1..</OriginalValue>
  </CustomFilter>
  <CustomFilter CustomValue="Past Week">
    <OriginalValue>-7..</OriginalValue>
  </CustomFilter>
  <CustomFilter CustomValue="Past Month">
    <OriginalValue>-30..</OriginalValue>
  </CustomFilter>
  <CustomFilter CustomValue="Past Six Months">
    <OriginalValue>-183..</OriginalValue>
  </CustomFilter>
  <CustomFilter CustomValue="Past Year">
    <OriginalValue>-365..</OriginalValue>
  </CustomFilter>
  <CustomFilter CustomValue="Earlier">
    <OriginalValue>..-365</OriginalValue>
  </CustomFilter>
</CustomFilters>
</Category>

We can see that the Type attribute contains the class name for the FilterGenerator implementation.

Using this configuration, the RefinementManager then iterates through the configured categories calling the appropriate FilterGenerator instance. This is done using the virtual GetRefinement method that’s defined on the abstract RefinementFilterGenerator class. The signature of the GetRefinement method is:

        public override List<System.Xml.XmlElement> GetRefinement(Dictionary<string, Dictionary<string, RefinementDataElement>> refinedData,
                                                System.Xml.XmlDocument filterXml,
                                                int maxFilterCats)

From this you can see that the RefinementManager populates and passes in a reference to a dictionary of RefinementDataElement objects. This is where the problem comes in in with regard to supporting multi-value managed properties and ultimately with the Choice data-type. The refinedData dictionary is keyed using mapped property names (i.e. the managed property name that you configured in the Search service application). Each RefinementDataElement is an instance of a particular mapped property within the result set. This means that it contains the value of the mapped property for a particular search result. So if a search result has multiple values for a mapped property, there is still only one corresponding RefinementDataElement and it’s value contains a ‘;’ concatenated list. Since the implementation of the ManagedPropertyFilterGenerator uses RefinementDataElements as the basis for generating filters, we end up with the concatenated lists being output as filters as we saw in the image above.

Parsing RefinementDataElements

To get round this limitation of the ManagedPropertyFilterGenerator we need to create a custom RefinementFilterGenerator that performs further parsing of the RefinementDataElement to extract individual values. We can use code similar to:

foreach (RefinementDataElement item in filterResults.Values)
{
    string[] actual = item.FilterDisplayValue.Split(';');
    foreach (string x in actual)
    {
        RefinementDataElement element;
        if (newResults.ContainsKey(x))
        {
            element = newResults[x];
        }
        else
        {
            element = new RefinementDataElement(x, 0, 0);
            newResults.Add(x, element);
        }
        element.FilterValueCount = element.FilterValueCount + item.FilterValueCount;
        resultSum = resultSum + item.FilterValueCount;
        itemCount = (long)(itemCount + item.FilterValueCount);
    }
}

Step-by-step guide

1. Using Visual Studio 2010, create a new Empty SharePoint Project names MultiValudFilterGenerator.

2. Add a new class file named MultiValueFilterGenerator.cs and add the following code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Office.Server.Search.WebControls;
using Microsoft.SharePoint.Utilities;
using System.Xml;
using System.Collections;
using System.Globalization;
using System.Collections.Specialized;
using System.Text.RegularExpressions;
using Microsoft.SharePoint;
using System.Web;

namespace MultiValueFilterGenerator
{
    class MultiValueFilterGenerator : RefinementFilterGenerator
    {

        public MultiValueFilterGenerator()
        {
        }

        public override List<System.Xml.XmlElement> GetRefinement(Dictionary<string, Dictionary<string, RefinementDataElement>> refinedData,
                                                                    System.Xml.XmlDocument filterXml,
                                                                    int maxFilterCats)
        {

            List<XmlElement> result = new List<XmlElement>();
    
            foreach (FilterCategory category in base._Categories)
            {

                long itemCount = 0L;

                Dictionary<string, RefinementDataElement> filterResults = refinedData.ContainsKey(category.MappedProperty) ? refinedData[category.MappedProperty] : new Dictionary<string, RefinementDataElement>();

                Dictionary<string, RefinementDataElement> newResults = new Dictionary<string, RefinementDataElement>();

                //rebuild results
                long resultSum = 0;
                foreach (RefinementDataElement item in filterResults.Values)
                {
                    string[] actual = item.FilterDisplayValue.Split(';');
                    foreach (string x in actual)
                    {
                        RefinementDataElement element;
                        if (newResults.ContainsKey(x))
                        {
                            element = newResults[x];
                        }
                        else
                        {
                            element = new RefinementDataElement(x, 0, 0);
                            newResults.Add(x, element);
                        }
                        element.FilterValueCount = element.FilterValueCount + item.FilterValueCount;
                        resultSum = resultSum + item.FilterValueCount;
                        itemCount = (long)(itemCount + item.FilterValueCount);
                    }
                }


                if (itemCount >= category.MetadataThreshold && itemCount != 0)
                {

                    //Calculate precentages
                    foreach (RefinementDataElement item in newResults.Values)
                    {
                        item.FilterValuePercentage = item.FilterValueCount / resultSum;
                    }

                    //Order by percentage
                    IEnumerable<RefinementDataElement> topResults = newResults.Values.OrderBy(i => i.FilterValuePercentage);

                    XmlElement element = filterXml.CreateElement("FilterCategory");
                    element.SetAttribute("Id", category.Id);
                    element.SetAttribute("ConfigId", category.Id);
                    element.SetAttribute("Type", category.FilterType);
                    element.SetAttribute("DisplayName", RefinementFilterGenerator.TruncatedString(category.Title, base.NumberOfCharsToDisplay));
                    element.SetAttribute("ManagedProperty", category.MappedProperty);
                    element.SetAttribute("ShowMoreLink", category.ShowMoreLink);
                    element.SetAttribute("FreeFormFilterHint", category.FreeFormFilterHint);
                    element.SetAttribute("MoreLinkText", category.MoreLinkText);
                    element.SetAttribute("LessLinkText", category.LessLinkText);
                    element.SetAttribute("ShowCounts", category.ShowCounts);

                    XmlElement containerElmt = filterXml.CreateElement("Filters");

                    string url = string.Empty;

                    bool selectable = BuildRefinementUrl(category, string.Empty, out url);

                    containerElmt.AppendChild(GenerateFilterElement(filterXml,
                        RefinementFilterGenerator.TruncatedString("Any " + category.Title, base.NumberOfCharsToDisplay),
                        url, selectable, "",
                        "",
                        "",
                        ""));

                    foreach (RefinementDataElement item in topResults.Take(category.NumberOfFiltersToDisplay))
                    {

                        selectable = BuildRefinementUrl(category, item.FilterDisplayValue, out url);

                        containerElmt.AppendChild(GenerateFilterElement(filterXml,
                            RefinementFilterGenerator.TruncatedString(item.FilterDisplayValue, base.NumberOfCharsToDisplay),
                            url, selectable, item.FilterDisplayValue,
                            item.FilterValueCount.ToString(),
                            item.FilterValuePercentage.ToString(),
                            ""));
                    }

                    element.AppendChild(containerElmt);
                    result.Add(element);
                }
            }
            return result;

        }
        private static XmlElement GenerateFilterElement(XmlDocument filterXml,
            string truncatedFilterDisplayValue,
            string url,
            bool selectable,
            string filterTooltip,
            string count,
            string percentage,
            string filterIndentation)
        {
            XmlElement element2 = filterXml.CreateElement("Filter");
            XmlElement newChild = filterXml.CreateElement("Value");
            newChild.InnerText = truncatedFilterDisplayValue;
            element2.AppendChild(newChild);
            newChild = filterXml.CreateElement("Tooltip");
            newChild.InnerText = filterTooltip;
            element2.AppendChild(newChild);
            newChild = filterXml.CreateElement("Url");
            newChild.InnerText = url;
            element2.AppendChild(newChild);
            newChild = filterXml.CreateElement("Selection");
            if (selectable)
            {
                newChild.InnerText = "Deselected";
            }
            else
            {
                newChild.InnerText = "Selected";
            }
            element2.AppendChild(newChild);
            newChild = filterXml.CreateElement("Count");
            newChild.InnerText = count;
            element2.AppendChild(newChild);
            newChild = filterXml.CreateElement("Percentage");
            newChild.InnerText = percentage;
            element2.AppendChild(newChild);
            if (!string.IsNullOrEmpty(filterIndentation))
            {
                newChild = filterXml.CreateElement("Indentation");
                newChild.InnerText = filterIndentation.ToString();
                element2.AppendChild(newChild);
            }
            return element2;
        }

        protected bool BuildRefinementUrl(FilterCategory fc, string value, out string url)
        {
            string filteringProperty = fc.MappedProperty;
            bool notSelected = false;
            string origFilter = string.Empty;
            bool hasRefinement = false;

            hasRefinement = (HttpContext.Current.Request.QueryString["r"] != null);

            if (!hasRefinement)
            {
                origFilter = string.Empty;
            }
            else
            {
                origFilter = HttpContext.Current.Request.QueryString["r"];
            }

            string filterString = origFilter;

            if (string.IsNullOrEmpty(value))
            {
                //Remove any filters for this category
                int num = filterString.Length;
                filterString = this.RemoveCategoryFromUrl(filterString, fc);
                //if teh value changed then all values were not selected
                notSelected = (bool)(filterString.Length != num);
            }
            else
            {
                notSelected = false;
                string propertyValue = filteringProperty.ToLower() + ":" + value;

                if (!filterString.Contains(propertyValue))
                {
                    notSelected = true;
                }

                if (notSelected)
                {
                    Regex regex = new Regex(string.Format("(((({0})(:|>|<|<=|>=|=)"([^"]|"")*"))|((({0})(:|>|<|<=|>=|=)([^\s]*))))((\s+AND\s+)(((({0})(:|>|<|<=|>=|=)"([^"]|"")*"))|((({0})(:|>|<|<=|>=|=)([^\s]*)))))*", filteringProperty), RegexOptions.IgnoreCase);
                    MatchCollection matchs2 = regex.Matches(filterString);
                    StringBuilder builder2 = new StringBuilder();
                    builder2.Append(" " + propertyValue);
                    foreach (Match match2 in matchs2)
                    {
                        builder2.Append(" AND ");
                        builder2.Append(match2.Value);
                    }
                    filterString = regex.Replace(filterString, string.Empty) + builder2.ToString();
                }
                else
                {
                    StringBuilder builder = new StringBuilder();
                    Regex regex2 = new Regex(string.Format("({0}(?<Operator>:|>|<|<=|>=|=)"(?<FilterValue>([^"]|"")*)"(\s|$))|({0}(?<Operator>:|>|<|<=|>=|=)(?<FilterValue>[^\s]*)(\s|$))", filteringProperty), RegexOptions.IgnoreCase);
                    //get a list of values for this category
                    foreach (Match match in regex2.Matches(filterString))
                    {
                        if ((match != null) && !string.IsNullOrEmpty(match.Value))
                        {
                            string trimmedValue = match.Value.Trim();
                            if (trimmedValue!=propertyValue)
                            {
                                if (builder.Length > 0)
                                {
                                    builder.Append(" AND ");
                                }
                                builder.Append(trimmedValue);
                            }
                        }
                    }
                    filterString = this.RemoveCategoryFromUrl(filterString, fc);
                    if (builder.Length > 0)
                    {
                        filterString = filterString + " " + builder.ToString();
                    }
                }
            }

            filterString = HttpUtility.UrlEncode(filterString.Trim());
            string originalUrl = HttpContext.Current.Request.Url.OriginalString;

            Uri request = HttpContext.Current.Request.Url;
            NameValueCollection queryString = HttpContext.Current.Request.QueryString;

            originalUrl = request.OriginalString.Replace(request.Query, string.Empty);
            string qs = string.Empty;

            foreach (string key in queryString.AllKeys)
            {
                if (key != "r")
                {
                    qs = qs + "&" + key + "=" + queryString[key].ToString();
                }
            }

            qs = qs + "&r=" + filterString;

            url = originalUrl + "?" + qs.Substring(1);

            return notSelected;
        }

        private string RemoveCategoryFromUrl(string currentUrl, FilterCategory fc)
        {
            string filteringProperty = fc.MappedProperty;
            string expression = string.Empty;

            if (fc.CustomFiltersConfiguration != null)
            {
                expression = string.Format(CultureInfo.InvariantCulture, "({0}(?<Operator>:|>|<|<=|>=|=)"(?<FilterValue>([^"]|"")*)"(\s|$))|({0}(?<Operator>:|>|<|<=|>=|=)(?<FilterValue>[^\s]*)(\s|$))", new object[] { filteringProperty });
            }
            else
            {
                expression = string.Format(CultureInfo.InvariantCulture, "(((({0})(:|>|<|<=|>=|=)"([^"]|"")*"))|((({0})(:|>|<|<=|>=|=)([^\s]*))))((\s+AND\s+)(((({0})(:|>|<|<=|>=|=)"([^"]|"")*"))|((({0})(:|>|<|<=|>=|=)([^\s]*)))))*", new object[] { filteringProperty });
            }
            Regex regex = new Regex(expression, RegexOptions.IgnoreCase);
            return regex.Replace(currentUrl, string.Empty);
        }

    }
}

3. Select Deploy Solution from the Build menu to compile the project and copy it to the GAC.

4. Navigate to C:WindowsAssembly and find the entry for MultiValueFilterGenerator. Right-click to view the properties and copy the value for the PublicKeyToken.

5. On a page containing a Refinement web part, edit the XML in the  Filter Category Definition to include an entry similar to:

  <Category    Title="Color"    
               Description="Use this filter to restrict results to a specific color"    
               Type="MultiValueFilterGenerator.MultiValueFilterGenerator, MultiValueFilterGenerator, Version=1.0.0.0, Culture=neutral, PublicKeyToken=<Add Your Public Key Token Here>"    
               MetadataThreshold="0"    
               NumberOfFiltersToDisplay="4"    
               MaxNumberOfFilters="20"    
               SortBy="Frequency"    
               SortByForMoreFilters="Name"    
               SortDirection="Descending"    
               SortDirectionForMoreFilters="Ascending"    
               ShowMoreLink="True"    
               MappedProperty="ItemColor"    
               MoreLinkText="show more"    
               LessLinkText="show fewer"    />

Note: Make sure that the Use Default Configuration checkbox is Unchecked

image_thumb2

6. Click on OK to commit the changes and then switch out of edit mode.

If all is well in the world, you’ll now have refinements for each value in our multi-value Managed Property as shown in this image (I’ve left the default implementation in the config to show the difference).

image_thumb3

Summary

This code is a long way from production ready and there are a few features that are missing such as a “More Items” link. Hopefully it’s enough to give an idea of how the RefinementManager does it’s magic. AS always – any comments, criticisms are questions are more than welcome.

Hope this saves somebody some time!

SharePoint Search and the SocialTagId Managed Property

I’ve been doing some work recently with social tagging in SharePoint 2010 and one of the things I’ve found is that on some installations the following problems exist:

  • On a tag profile page (MySites/tagprofile.aspx?termid=whatever), there are no documents listed under Tagged Items when All is selected.
  • When selecting Popular within last 60 days, the expected documents appear
  • When clicking on the see all results for this tag in search link, no results are found

These problems all boil down to the same thing – The search application has no data for the social tags in it’s property database. I’ve seen reports that an index reset and full crawl resolves this issue but I haven’t had any success with that approach.

So far, I haven’t been able to determine why the problem occurs. Maybe it’s a permissions issue between the Search Service Application and the User Profile Service Application? (The social tags are not ‘crawled’ in the usual sense, they’re extracted from the User Profile database and mapped directly to the hidden SocialTagId  managed property in the Search property database).

Anyways, I’ve found that deleting and recreating the Search Service Application resolves the problem.

Hope this saves some head scratching!

Get the TagProfile Url using Code in SharePoint 2010

If there’s one thing that drives me nuts about writing code for SharePoint, it’s the number of critical values that are not accessible via the public API. One example that I came up against today is the url of the tag profile page. Chances are, since you’re reading this post, you already know that deriving this url isn’t all that straightforward so having the ability to get it from the object model is important of you’re writing any kind of code that requires links to a tag’s problem page.

Here’s my solution, it’s pretty self-explanatory

private Uri GetTagProfileUrl(SPServiceContext ctx)
{
    var socialDataAssembly = typeof(SocialData).Assembly;
    var sdg = socialDataAssembly.GetType("Microsoft.Office.Server.SocialData.SocialDataGlobal");
    var mi = sdg.GetMethod("GetTagProfileUrl", BindingFlags.Static | BindingFlags.NonPublic, null, new[] { typeof(SPServiceContext) },null);
    return mi.Invoke(sdg,new[] { ctx }) as Uri;
}

internal string CreateLinkUrl(SocialTerm term)
{
    Uri tagProfileUrl = GetTagProfileUrl(SPServiceContext.Current);
    if (null == tagProfileUrl)
    {
        return null;
    }
    return SPHttpUtility.NoEncode(ResolveClientUrl(tagProfileUrl.AbsoluteUri + "?" + SPHttpUtility.UrlKeyValueEncode("termid", term.Term.Id.ToString())));
}

Another half-hour of development time saved!

Programmatically configure Service Application permissions in SharePoint 2010

Ever tried to use the UserProfileManager class from PowerShell or a console application? Did you get the ‘No User Profile Application available to service the request. Contact your farm administrator.’ error?

You’ve probably already stumbled upon this post from Steve Peschka that explains what’s going on. It’s basically a permissions issue – in order to access a Service Application you need to have the appropriate access permissions and these can be configured in Central Admin > Manage service Applications > Permissions

 image

During normal operation we don’t see this problem because most of the out-of-the-box service applications are configured to allow access to the W3WP process and the OWSTIMER process without further credentials being required. However, as soon as we try to access from a different process, the sharing permissions come into play hence the issue with PowerShell or some other application.

Fixing the problem manually is pretty straightforward, simply add the appropriate account to the list and you’re done. Although It’s a bit of  a pain for users of your PowerShell cmdlet users when you throw an error with a message telling them what to do. Wouldn’t it be better to programmatically make the configuration change? Given that I’m a developer, it’ll come as no surprise that I believe that it would, so here’s how to do it.

private static void GrantPermissions()
{
    var upServiceproxy = SPFarm.Local.Services.Where(s => s.GetType().Name.Contains("UserProfileService")).FirstOrDefault();

    if (upServiceproxy != null)
    {
        var upServiceApp = upServiceproxy.Applications.OfType<SPIisWebServiceApplication>().FirstOrDefault();
        if (upServiceApp != null)
        {
            var security = upServiceApp.GetAccessControl();
            var mgr = SPClaimProviderManager.Local;
            var identity = WindowsIdentity.GetCurrent();
            if (identity != null)
            {
                var claim = mgr.ConvertIdentifierToClaim(identity.Name, SPIdentifierTypes.WindowsSamAccountName);
                security.AddAccessRule(new SPAclAccessRule<SPIisWebServiceApplicationRights>(claim, SPIisWebServiceApplicationRights.FullControl));
            }
            upServiceApp.SetAccessControl(security);
            upServiceApp.Uncache(); 
            upServiceproxy.Uncache();
        }
    }
}

 

There are a few interesting things to note about this code:

  • The classes that implement the UserProfile service application are internal so we can’t pick up a reference using the type. Hence the  match on type name. Not pretty, but it works! For other service applications you could use:
upServiceApp=SPServiceContext.Current.GetProxies(typeof(SomeNonInternalType));

or if you know the guid for the service application proxy, you could get it directly from the HOS:

upServiceApp = SPFarm.Local.GetObject(someGuid) as SPIisWebServiceApplication;
  • Each SpIisWebServiceApplication derived object has two sets of permissions, accessible via  upServiceApp.GetAccessControl() and upServiceApp.GetAdministrationAccessControl(). These map to the Permissions menu option and the Adminstrators menu options on the ribbon shown above.
  • When adding access rules, it is possible to add non-claims based rules using the SPAclAccessRule class. In the case of the UserProfile service, these will appear in the list as expected but will have no effect. Access rules must use an SPClaim rather than an IdentityReference.
  • All of this good stuff is cached. In order for these changes to take effect within the current session, the cache must be cleared using the appropriate Uncache methods.

SharePoint 2010 bug: The primary lookup field cannot be deleted. There are dependent lookups created on this primary lookup field that should be deleted first.

One of the additions is SharePoint 2010 is the dependent lookup. For us as developers, these are very useful since they allow us to include multiple values from a lookup list while using a single lookup column. To see dependent lookups in action, create a new column of type Lookup and notice the checkboxes. When we select any of these options a dependent lookup is created that is linked to our main lookup. In effect dependent lookups are just like every other lookup field except they are read only and their value is controlled by the field referenced by their PrimaryFieldId.

image

However, there is a problem if we create a custom field type that makes use of dependent lookups. Although MSDN provides a great code sample explaining how to programmatically create dependent lookups, what it doesn’t mention is that if we’re creating a custom field type, there’s a bug in the platform that means we can’t delete them again!

Steps to reproduce

  1. Create a custom field control. Override the OnAdded method to create a dependent lookup.
  2. Create a custom Field Editor and hook it up to the field control.
  3. Deploy the solution and add a new field based on the custom field to a list or library.
  4. Delete the new field.

An SPException is thrown with the description:

The primary lookup field cannot be deleted. There are dependent lookups created on this primary lookup field that should be deleted first.

Why does this problem occur?

Our custom field editor is hosted on the FldEditEx.aspx page and it’s this page that provides the Delete button and handles it’s click event. For most fields, performing a delete is simply a case of calling the Delete method on the referenced field. The code for the Delete method then calls the OnDeleting method on the underlying field and allows for any clean up code. However, as you’ve gathered, for Dependent lookups things work differently. Before the Delete method can be called the dependent lookups must be deleted. So we have a catch 22. We can’t clean up the dependent lookups in the usual manner (ie. using OnDeleting) because we can’t call OnDeleting until we’ve cleaned up the dependent lookups!

To get round this problem, the developers have added some code to FldEditEx (Microsoft.SharePoint.ApplicationPages.FieldEditPage). When the Delete button is clicked, dependent lookups are removed. However, there’s a problem with this. They’re only removed if our field editor is derived from LookupFieldEditor. (Although this MSDN article mentions that deriving an editor from LookupFieldEditor is evil).

So what can we do?

We have three options:

  1. Don’t use dependent lookups – In most cases there are no viable alternatives that don’t involve introducing disproportionate complexity. Plus that would be quitting!
  2. Derive our field editor from LookupFieldEditor – This is unlikely to work. The LookupFieldEditor does all kinds of good stuff that will allow the user to mess about with our custom field in ways that we probably don’t want. Plus Uncle Bill’s boys (and girls!) have specifically frowned upon this idea.
  3. Devise a mechanism to execute some custom code before the call to Delete – now you’re talking!

The solution

In a custom field editor add the following code:

private Delegate _baseClickHandler;
private SPFieldLookup _targetField;

protected override void OnInit(EventArgs e)
{
    if (IsPostBack)
    {
        var fep = Page as FieldEditPage;

        if (fep != null)
        {
            RegisterHandlers(fep);
        }
    }
    base.OnInit(e);
}

void RegisterHandlers(FieldEditPage fep)
{
    var fld = fep.GetType().GetField("BtnDelete", BindingFlags.Instance | BindingFlags.NonPublic);
    if (fld != null)
    {
        var deleteButton = fld.GetValue(fep) as Button;

        if (deleteButton != null)
        {
            var events = ((EventHandlerList)(deleteButton.GetType().GetProperty("Events", BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.FlattenHierarchy | BindingFlags.Public).GetValue(deleteButton, null)));
            var fi = deleteButton.GetType().GetField("EventClick", BindingFlags.Static | BindingFlags.NonPublic);
            if (fi != null)
            {
                var key = fi.GetValue(deleteButton);

                _baseClickHandler = events[key];

                events.RemoveHandler(key, _baseClickHandler);

                deleteButton.Click += delete_onClick;
            }
        }
    }
}

private void delete_onClick(object sender, EventArgs e)
{
    if (_targetField != null && !_targetField.IsDependentLookup)
    {
        if (_targetField.ParentList != null)
        {
            var dependencies = _targetField.GetDependentLookupInternalNames();

            foreach (var dependency in dependencies)
            {
                //add any other cleanup here
                _targetField.ParentList.Fields.GetFieldByInternalName(dependency).Delete();
            }
        }
    }

    if (_baseClickHandler != null)
    {
        _baseClickHandler.DynamicInvoke(sender, e);
    }

}


public void InitializeWithField(SPField field)
{
    _targetField = field as SPFieldLookup;

    //the rest of your code here
}

Quick Tip: Error occurred in deployment step ‘Add Solution’

This is one of those things that I commonly forget because it doesn’t crop up too often. Sometimes when deploying a solution to SharePoint 2010 using Visual Studio, you’ll get an error like this:

Error occurred in deployment step ‘Add Solution’: Error: Cannot add the specified assembly to the global assembly cache: MyCompany.Namespace.widget.dll.

There are a few reasons for this. For example, OWSTimer.exe may be using the dll in question or Visual Studio may have it locked. However, rather than stopping and starting services indiscriminately and restarting Visual Studio, we can find out exactly which process is causing us grief by using the TaskList application.

Open up a command prompt and type:

tasklist /M MyCompany.Namespace.widget.dll

You’ll see results such as:

Image Name                PID      Modules
========================= ======== ============================================
WebAnalyticsService.exe   2040     MyCompany.Namespace.widget.dll
w3wp.exe                  4716     MyCompany.Namespace.widget.dll
w3wp.exe                  1164     MyCompany.Namespace.widget.dll

Armed with this info we know what to shut down to resolve the issue. Another SharePoint mystery unravelled!

The skinny on SPPersistedObject and the Hierarchical Object Store in SharePoint 2010

If you’re doing  any kind of serious development with SharePoint, eventually you’ll find yourself looking for a solution to these problems:

  • You need to offload some processing to an asynchronous process.
  • You need to persist some global state data.

Although there are many solutions to these problems, one of the simplest is to use the Hierarchical Object Store. For the uninitiated, the Hierarchical Object Store is a feature of every SharePoint farm that allows user-defined objects to be persisted in a hierarchical nature within the configuration database. Or in plain English, as developers, we can create our own custom objects and save them to the database without having to think about the database schema or the actual mechanism for saving and retrieving the data. From our code, all we need to deal with is our custom object. For example, we can create and object of type Foo and give it properties a, b and c. By using the Hierarchical Object Store, we can persist the values of these properties by simply saving the object. Handy eh?

So how does it work?

The key to using the Hierarchical Object Store (I’ll call it HOS from now on, RSI is a nasty business), is the SPPersistedObject class. By deriving custom classes from this base type, we have everything that we need to persist data to and read data from the HOS. Simple. Why the look of cynicism?

Ok, so maybe, like many things in SharePoint development, it’s not quite as simple as that. Lets take a look at a few examples and I’ll highlight some gotchas and interesting features along the way.

Why is it hierarchical?

We’ve looked at how to create custom objects and how the SPPersistedObject base class provides the functionality to persist them to the store but one thing we haven’t talked about is the Hierarchical part of the HOS. Where does that come into it? If we look at the constructors to SPPersistedObject we can find out:

protected SPPersistedObject(string name, SPPersistedObject parent);

protected SPPersistedObject(string name, SPPersistedObject parent, Guid id);

In order to create a persisted object we need to specify a parent object which must also be derived from SPPersistedObject. Consequently the HOS assumes a hierarchical nature with each object being defined as a child of another SPPersistedObject. By adopting this approach, the HOS can automatically clean up child objects when their parents are removed from the store without us having to write specific code to do it. This is significant because we can arbitrarily create child objects, if the store was not hierarchical cleaning up this related objects would be challenging. This talk of hierarchy begs the question, If all objects need to have a parent, what’s the root object? Since the HOS is implemented in the farm configuration database, I’m sure it’ll come as no surprise to learn that the root object is Microsoft.SharePoint.Administration.SPFarm. (Which is of course derived from SPPersistedObject)

Enough talk, show me some code

This short sample shows how to create a custom object for persisting global state data:

using System.Runtime.InteropServices;
using Microsoft.SharePoint.Administration;

namespace PersistedObjectDemo
{
    [Guid("F5BC7474-8F88-4C6E-A71C-C56CF8FA3FD8")]
    internal class MyPersistedObject : SPPersistedObject
    {
        [Persisted] private int _bar;
        [Persisted] private string _foo;

        public MyPersistedObject()
        {
        }

        public MyPersistedObject(string valForFoo, string name, SPPersistedObject parent)
            : base(name, parent)
        {
            _foo = valForFoo;
        }

        public string Foo
        {
            get { return _foo; }
            set { _foo = value; }
        }

        public int Bar
        {
            get { return _bar; }
            set { _bar = value; }
        }
    }
}

There are a few things to note in this sample. Firstly, the decoration of the class with GuidAttribute. A proper explanation of why we need this requires some understanding of how the persistence mechanism works. SPPersistedObject is derived from SPAutoSerializingObject and it’s within SPAutoSerializingObject that much of the serialization magic happens. To cut a long story short, reflection is used to build a list of fields that should be persisted, the values of these fields are then stored as XML and the resultant XML is saved to the configuration database by code within the SPPersistedObject base class. In order to deserialize an object from the HOS, the process needs to know the type of the object to populate with the XML-based fields that were stored previously. As we’ll see later, the system knows which type to use because we must pass in the type when attempting to get the object. However, when it comes to retrieving the XML-based field data from the database, the classid of the type is used. For those of us unfortunate enough to remember classid’s from the days of VB6 and COM, yep these are the same beasties! In the .Net framework, every type has a unique classid. Normally we don’t need to know about these unless we’re doing COM Interop since they’re automatically generated by the framework when our code is compiled. However, since they’re auto-generated, they’re not fixed and that can cause a problem with deserialization since the classid may have changed between the time the object was serialized and when the time comes to deserialize it. We’ll see this problem a lot when developing since every time we compile the classid’s can change. By decorating a class with GuidAttribute, we can specify a fixed classId and all will be well in the world. Another SharePoint mystery unravelled!

The next thing to note in the code sample is the use of PersistedAttribute on our fields. Only fields decorated with PersistedAttribute will be serialized by the underlying SPAutoSerializingObject. Note that the PersistedAttribute can only be applied to fields. It won’t work on properties. This means that we can’t really use auto-implemented properties such as:

public string Foo { get; set; }

Since the backing field is added at compile time and therefore can’t be decorated with the PersistedAttribute.

The final thing to note is the constructors that we’ve provided. We have a public constructor that takes no arguments. This is a requirement to support serialization. Note that unlike Xml serialization, our class does not need to be public. Our main constructor overrides one of the base constructors on SPPersistedObject while also accepting a default property value. As we saw earlier, SPPersistedObject has two constructors that we can override. The difference between them is in how we assign an Id for our object.  We can either pass in a specific guid or allow the base SPPersistedObject class to generate one for us.

How can I use this object?

We can use a console application to play around with HOS. These few lines of code will create a new object and store it in the HOS as a child of the SPFarm object.

static void Main(string[] args)
{
    var farm = SPFarm.Local;
    var obj = new MyPersistedObject("this is a test", "Test Obj", farm);
    obj.Update();
}

If we try to run this code again we’ll get:

An object of the type PersistedObjectDemo.MyPersistedObject named “Test Obj” already exists under the parent Microsoft.SharePoint.Administration.SPFarm named “SharePoint_Config”.  Rename your object or delete the existing object.

image

Welcome to the first gotcha when dealing with HOS. Objects must have unique names within the context of their parent. For example, we can’t have two objects named ‘My Object’ as children of the SPFarm object. Note that the type of object and the object ID don’t make any difference, we still can’t have two with the same name.

So what’s the best way to deal with this? One possible approach is to treat persisted objects as singletons and you’ll see this approach being taken with many of the SharePoint server object model classes such as SPFarm. To demonstrate this we can modify our code as follows:

using System.Runtime.InteropServices;
using Microsoft.SharePoint.Administration;

namespace PersistedObjectDemo
{
    [Guid("F5BC7474-8F88-4C6E-A71C-C56CF8FA3FD8")]
    internal class MyPersistedObject : SPPersistedObject
    {
        private const string MY_OBJECT_NAME = "My Test Object";
        [Persisted] private int _bar;
        [Persisted] private string _foo;

        public MyPersistedObject()
        {
        }

        private MyPersistedObject(SPPersistedObject parent)
            : base(MY_OBJECT_NAME, parent)
        {
        }

        public static MyPersistedObject Local
        {
            get
            {
                SPPersistedObject parent = SPFarm.Local;
                var obj = parent.GetChild<MyPersistedObject>(MY_OBJECT_NAME);
                if (obj == null)
                {
                    obj = new MyPersistedObject(parent);
                    obj.Update();
                }

                return obj;
            }
        }

        public string Foo
        {
            get { return _foo; }
            set { _foo = value; }
        }

        public int Bar
        {
            get { return _bar; }
            set { _bar = value; }
        }
    }
}

We now have a static Local property that will return a single instance of our object. We can use it with code similar to this:

static void Main(string[] args)
{
    var obj = MyPersistedObject.Local;
    obj.Foo = "Some value";
    obj.Update();
}

Since we’ve adopted a singleton pattern we can call this code as often as we like in the knowledge that we’re only dealing with a single instance of our object.

Are there any limitations on the types of fields that can be persisted?

Just like other types of serialization, there are limitations on what can be serialized. All primitives can be serialized as can the following:

System.Uri System.Guid System.TimeSpan System.Security.SecureString
System.DateTime System.Version System.Type  

As well as these types, which are serialized directly, it’s also possible to serialize fields derived from SPAutoSerializingObject or SPPersistedObject. Arrays of these objects and IList<T> and IDictionary<T> where T is one of the listed types can also be serialized.

We can see this in action by adding the following property to our object:

  internal class MyPersistedObject : SPPersistedObject
    {
.....
        [Persisted] private List<string> _myStrings=new List<string>();
        
        public List<string> MyStrings
        {
            get { return _myStrings; }
            set { _myStrings = value; }
        }
.....
    }

We can then use this new list as follows:

static void Main(string[] args)
{
    var obj = MyPersistedObject.Local;
    obj.Foo = "Some value";
    obj.MyStrings.Add("A string");
    obj.Update();
}

So far so good. Everything works as expected. Let’s take a look at what happens if we use collections of more complex objects.

Real World Scenario – Commonly, if we’re planning to offload processing to an asynchronous process we’ll create a timer job by deriving from the SPJobDefinition class. SPJobDefinition is based on SPPersistedObject and as a result the techniques discussed here have particular relevance. Rather than creating a timer job for every task that we need to offload, a better approach is to create a timer job for each type of task and have the job process individual tasks whenever it runs. This is a typical scenario where you’d use HOS and complex child objects. The custom job definition is persisted as a child of the Farm  and our tasks are saved as children of the job definition.

We can add a second custom persisted object to demonstrate what happens if we make this a child of our first object using the following code:

In AnotherPersistedObject.cs, add:

using System.Runtime.InteropServices;
using Microsoft.SharePoint.Administration;

namespace PersistedObjectDemo
{
    [Guid("F5BC7474-8F88-4C6E-AAAA-C56CF8FA3FD8")]
    internal class AnotherPersistedObject : SPPersistedObject
    {
        [Persisted]
        private string _someValue;

        public AnotherPersistedObject()
        {

        }

        public AnotherPersistedObject(string name, SPPersistedObject parent)
            : base(name, parent)
        {

        }

        public string SomeValue
        {
            get { return _someValue; }
            set { _someValue = value; }
        }
    }
}

Add the following additional property to MyPersistedObject.cs:

[Persisted]
private List<AnotherPersistedObject> _childObjects = new List<AnotherPersistedObject>();
public List<AnotherPersistedObject> ChildObjects
{
    get { return _childObjects; }
    set { _childObjects = value; }
}

We can make use of this new property by changing Program.cs as follows:

static void Main(string[] args)
{
    var obj = MyPersistedObject.Local;    
    obj.Foo = "Some value";
   
    obj.ChildObjects.Add(new AnotherPersistedObject("My first object", obj) { SomeValue = "First value" });
    obj.ChildObjects.Add(new AnotherPersistedObject("My second object", obj) { SomeValue = "Second value" });
    obj.ChildObjects.Add(new AnotherPersistedObject("My third object", obj) { SomeValue = "Third value" });
    obj.Update();
 
    obj.Uncache();
    obj = MyPersistedObject.Local;

    int i = 1;
    foreach (var s in obj.ChildObjects)
    {
        Console.WriteLine("item {0} value - {1}", i, s.SomeValue);
        i++;
    }

    Console.WriteLine("Press return to exit");
    Console.ReadLine();
}

The above code is pretty straightforward: we’re creating three child objects and adding them to our ChildObjects collection before saving the parent. We’re then iterating through the collection of child objects to confirm that they’ve been saved properly.

However, if we run this program we’ll get:

NullReferenceException was unhandled: Object reference not set to an instance of an object.

image

If we attach a debugger we can see that the error occurs on:

Console.WriteLine("item {0} value - {1}", i, s.SomeValue); 

Welcome to the second gotcha when working with SPPersistedObject:

Although we can create lists, dictionaries and arrays of other SPPersistedObjects, behind the scenes all that is persisted is a reference to each child object. Our child collection reports that it contains 3 objects but when we iterate them we find that they’re not actually there!

To fix this problem we need to ensure that each child object is persisted before calling Update on our parent object. We can do this at the time of creation or we can override the Update method of our parent object. There are drawbacks to both approaches. Calling update at the time of creation can potentially mean that we have orphaned child objects if we don’t call Update on our parent object. Given the naming limitations highlighted in gotcha #1, this could create a difficult to debug problem. On the other hand, overriding the Update method hides what’s going on and this could lead to some confusion.

On balance, overriding the Update method is probably the safest option. We can do this as follows:

public override void Update()
{
    var success = new List<AnotherPersistedObject>();

    //Remove any referenced objects that no longer exist
    _childObjects.RemoveAll(o => o == null);

    foreach (var childObject in _childObjects)
    {
        try
        {
            childObject.Update();
            success.Add(childObject);
        }
        catch
        {
            //remove previously persisted objects to prevent orphans
            foreach (var o in success)
            {
                o.Delete();
            }
            throw;
        }
    }
    base.Update();
}

If we now run our code, we’ll see the results as expected:

image

SPAutoSerializingObject – The pain free alternative

So as we can see, it’s possible to add complex child objects derived from SPPersistedObject but care has to be taken to prevent orphans and to ensure that everything is persisted properly. Thankfully, it doesn’t have to be so painful though. Allow me to introduce the hitherto cast-aside, SPAutoSerializingObject.

Using collections a class derived from SPAutoSerializingObject, we have practically the same degree of flexibility as we do with SPPersistedObject. However there’s one significant difference – whereas collections of SPPersisedObjects are stored as references, collections of SPAutoSerializingObjects are stored as inline serialized values. So what? I hear you say. Let’s modify our custom persisted object to accept another collection of child objects and I’ll demonstrate the significance.

Add a new class named AutoSerializedObject with the following code:

using Microsoft.SharePoint.Administration;

namespace PersistedObjectDemo
{
    internal class AutoSerializedObject : SPAutoSerializingObject
    {
        [Persisted] private string _someValue;

        public string SomeValue
        {
            get { return _someValue; }
            set { _someValue = value; }
        }
    }
}

Then modify MyPersistedObject to incorporate a dictionary of our new child class. Note that we could use a list or and array, I’m using a dictionary just to illustrate that unlike normal XML serialization where properties derived from IDictionary are not serializable, when using a class derived from SPAutoSerializingObject, we can handle dictionaries effortlessly.

[Persisted]
private Dictionary<string, AutoSerializedObject> _childDictionary=new Dictionary<string, AutoSerializedObject>();
public Dictionary<string, AutoSerializedObject> ChildDictionary
{
    get { return _childDictionary; }
    set { _childDictionary = value; }
}

We can see this new collection in action by modifying our Main function as follows:

static void Main(string[] args)
{
    var obj = MyPersistedObject.Local;
    obj.Foo = "Some value";

    obj.ChildDictionary.Add("First", new AutoSerializedObject(){SomeValue = "First value"});
    obj.ChildDictionary.Add("Second", new AutoSerializedObject() { SomeValue = "Second value" });
    obj.ChildDictionary.Add("Third", new AutoSerializedObject() { SomeValue = "Third value" });

    obj.Update();

    obj.Uncache();
    obj = MyPersistedObject.Local;

    foreach (var s in obj.ChildDictionary)
    {
        Console.WriteLine("item {0} value - {1}", s.Key, s.Value.SomeValue);
    }

    Console.WriteLine("Press return to exit");
    Console.ReadLine();
}

As you can see when running this code, our child objects are serialized and retrieved correctly – without us having to add any additional code. In many situations using classes derived from SPAutoSerializingObject is a better approach than jumping straight in with a custom SPPersistedObject.  Naturally there are many factors to be considered when deciding on an appropriate base class but two of the key deciders are:

  • Will I need to pick up a reference to the class without first having a reference to its parent? If the answer is yes than SPPersistedObject is the only viable choice. By using code such as this, we can get a reference to any object within the HOS:
Guid myObjectGuid;

var myObject = SPFarm.Local.GetObject(myObjectGuid);

 

  • How complex is my child object and how many of them will there be? If the child object is complex, possibly containing other child objects and particularly if there are many of them, SPPersistedObject may be a better choice because the serialized value is split among multiple rows in the database. Bear in mind that SPAutoSerializingObject cannot contain child properties of type SPPersistedObject so all children will also be persisted within the database row of the parent. Depending on which version of SQL server is being used, the maximum size of the serialized data can be up to 2Gb, but just because you can, doesn’t always mean that you should!

Security and the HOS

One final topic that requires some coverage is security. As described earlier, the HOS exists in the farm configuration database. Strictly speaking it exists to support administrative functionality however it has uses beyond this scope. For example, if we’re building a custom timer job and we want to have the ability to submit tasks to our job for completion. It’s most likely that these tasks will originate within a non-administrative scope. Possibly as the result of a user clicking a button on a webpart on some other site collection. With previous versions of SharePoint, this simply wasn’t possible. Only farm administrators could update the HOS. However with SharePoint 2010, an addition has been made to the base SPPersistedObject that opens up the HOS to other users. To see how this works, we can modify our Main function as follows:

using System;
using System.Runtime.InteropServices;
using System.Security.Principal;

namespace PersistedObjectDemo
{
    internal class Program
    {
        private const int LOGON_TYPE_INTERACTIVE = 2;
        private const int LOGON_TYPE_PROVIDER_DEFAULT = 0;

        [DllImport("advapi32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
        public static extern bool LogonUser(string userName, string domain, string password, int logonType,
                                            int logonProvider, ref IntPtr accessToken);

        private static void Main(string[] args)
        {
            try
            {
                IntPtr accessToken = IntPtr.Zero;
                if (LogonUser("non-farm-admin-username", "your-domain-name", "your-password", LOGON_TYPE_INTERACTIVE,
                              LOGON_TYPE_PROVIDER_DEFAULT, ref accessToken))
                {
                    using (var identity = new WindowsIdentity(accessToken))
                    {
                        using (identity.Impersonate())
                        {
                            MyPersistedObject obj = MyPersistedObject.Local;
			       //delete existing object and recreate to keep things simple
                            obj.Delete();
                            obj = MyPersistedObject.Local;
                            obj.Foo = "Some updated value";
                            obj.Update();
                        }
                    }
                }

            }
            catch (Exception ex)
            {
                Console.Write(ex);
            }

            Console.WriteLine("Press return to exit");
            Console.ReadLine();
        }
    }
}

This code simulates an attempt by a non-farm admin user to update the HOS. If we run this we’ll see the following exception which is our gotcha number 3:

System.Security.SecurityException: Access denied. at Microsoft.SharePoint.Administration.SPPersistedObject.BaseUpdate() at PersistedObjectDemo.MyPersistedObject.Update()

Unsurprisingly, trying to update the HOS using a non-farm admin user throws an exception. However, if we add the following code to MyPersistedObject.cs and run our program again, we can see that the HOS is updated appropriately.

protected override bool HasAdditionalUpdateAccess()
{
    return true;
}

Of course, in a real-world application, we’d probably want to add some security checking in this function rather than simply returning true and allowing anybody to update our object but hopefully this illustrates the point sufficiently.

There is one other gotcha with regard to security – when SharePoint is 2010 installed, by default only the farm admin account has read/write access to the config database. In a least-privilege installation, all web applications other then Central Administration, will be running under their own service account and these accounts will not have write access to the config database. Consequently, adding, deleting and updating objects in the HOS will not be possible. Although overriding HasAdditionalUpdateAccess allows us a bit more control over who can update the HOS, if the underlying app pool account doesn’t have the appropriate database permissions we’re going to come unstuck pretty quickly.

In my opinion, denying access to the HOS for web application app pool accounts is a step too far. Given the default configuration of only allowing farm admins to make changes to persisted objects, granting execute permissions on the stored procedure that updates HOS for all application pool accounts does not represent a security risk. Of course, I’m a developer not a security expert and your mileage may vary! The only workarounds for this gotcha are:

  1. Run any web application that’s needs to modify the HOS under the same service account as Central Admin, or,
  2. modify the the WSS_Content_Application_Pools database role in the Config database to allow execute permissions on proc_putObject. (I’m sure that I don’t need to say that such a modification is neither endorsed or supported by the good folks at Microsoft, I merely include it here in the interests of completeness!)

Conclusion

SPPersistedObject, SPAutoSerializingObject and the Hierarchical Object Store are useful tools. Maybe in a future article I’ll look at one of the most common uses of these tools – the creation of custom timer jobs. As we’ve seen, there are limitations and nuances but I believe that every serious SharePoint developer needs to know and understand how these objects work and hopefully this article served as a useful introduction.

Persisting a Stream to the SharePoint configuration database

Ever had the situation where you need to perform some long running administrative process on a file or some other stream? Creating long-running processes with SharePoint is easy enough by creating a new class that’s derived from SPJobDefinition, but how do you pass in your stream?  Simply put, you can’t! The job will run in a separate process so any data must be serialized and de-serialized.

The answer is to persist the stream to the hierarchical object store. There are many articles out there on how to derive from and use SPPersistedObject, such as this one from Maurice Prather. It’s easy to persist most object properties using a custom SPPersistedObject and you’d certainly be forgiven for persisting a stream by converting it to a byte array.

There is  a better approach though- the object store is basically the contents of the Objects table in the SharePoint_Config database (changes not supported disclaimer etc.), the Properties column contains the serialized SPPersistedObjects and some examination will reveal that it’s of type nvarchar(max), meaning we can stick about 2Gb of data in there (depending on the version of SQL Server you’re running). If you look at that table though, you’ll notice that there aren’t a lot of huge objects in there and when you think about it from a performance perspective, that makes sense, the Properties field is serialized as XML and is parsed in memory by SharePoint. If we stick a few huge objects in there, it’ll slow everything down.

You may be thinking, SharePoint 2010 with it’s new-fangled sandbox and the like, must store binary data somewhere, if it’s not in the Objects table and it’s not in the file system where is it? The answer is that it’s in the Binaries table. Effectively the Binaries table allows BLOB data to be attached to records in the Objects table. How does SharePoint read and write to this table? It uses the SPPersistedFile class (which is ultimately derived from SPPersistedObject).

Cut to the Chase

Now we can use the SPPersistedFile class in our own code but it’s public interface only supports persisting physical files and that may not be practical if you want to persist a file that a user uploads, so my PersistedStreamObject wraps the SPPersistedFile class and accepts a Stream in the constructor. This class can be used in the same way as any other SPPersistedObject and makes use of compression internally to minimise the database space required.

[Guid("5C2EDB3B-ED8B-4615-A8F5-25C24D044835")]
public class PersistedStreamObject : SPPersistedObject
{
    private const int INT_BUFFER_SIZE = 32768;
    private const string STR_OBJECT_NAME = "Stream";

    [Persisted]
    private long _length;

    [Persisted]
    private long _uncompressedLength;

    private Stream _fileStream;

    public PersistedStreamObject()
    {
    }

    public long Length
    {
        get { return _length; }
    }

    public long UncompressedLength
    {
        get { return _uncompressedLength; }
    }

    public PersistedStreamObject(string name, SPPersistedObject parent, Stream stream)
        : base(name,parent)
    {
        _fileStream = stream;
    }

    public static void CopyStream(Stream input, Stream output)
    {
        if (input == null) throw new ArgumentNullException("input");
        if (output == null) throw new ArgumentNullException("output");

        var buffer = new byte[INT_BUFFER_SIZE];
        while (true)
        {
            int read = input.Read(buffer, 0, buffer.Length);
            if (read <= 0)
                return;
            output.Write(buffer, 0, read);
        }
    }

    public override void Update()
    {
        string tmp = Path.GetTempFileName();

        if (_fileStream != null)
        {
            base.Update();

            using (var w = new FileStream(tmp, FileMode.OpenOrCreate, FileAccess.Write))
            {
                var cs = new GZipStream(w, CompressionMode.Compress);

                CopyStream(_fileStream, cs);
                cs.Flush();
                _length = w.Length;

                _uncompressedLength = _fileStream.Length;
            }

            _fileStream.Close();
            _fileStream.Dispose();

            base.Update();
            var child = GetChild<SPPersistedFile>(STR_OBJECT_NAME);

            if (child != null) child.Delete();

            var file = new SPPersistedFile(STR_OBJECT_NAME, tmp, this);
            file.Update();
            File.Delete(tmp);
        }
    }

    [SuppressMessage("Microsoft.Usage","CA2202",Justification="Using statement handles Dispose")]
    public Stream Stream
    {
        get
        {
            if (_fileStream != null)
            {
                string tmp = Path.GetTempFileName();
                var file = GetChild<SPPersistedFile>(STR_OBJECT_NAME);

                if (file != null)
                {
                    file.SaveAs(tmp);
                    using (var stream = new MemoryStream())
                    {
                        using (var fs = File.OpenRead(tmp))
                        {
                            var cs = new GZipStream(fs, CompressionMode.Decompress);
                            CopyStream(cs, stream);
                            cs.Flush();
                            fs.Close();
                        }


                        File.Delete(tmp);

                        _fileStream = stream;
                    }
                }
            }
            return _fileStream;
        }
       
    }
}