The skinny on SPPersistedObject and the Hierarchical Object Store in SharePoint 2010

If you’re doing  any kind of serious development with SharePoint, eventually you’ll find yourself looking for a solution to these problems:

  • You need to offload some processing to an asynchronous process.
  • You need to persist some global state data.

Although there are many solutions to these problems, one of the simplest is to use the Hierarchical Object Store. For the uninitiated, the Hierarchical Object Store is a feature of every SharePoint farm that allows user-defined objects to be persisted in a hierarchical nature within the configuration database. Or in plain English, as developers, we can create our own custom objects and save them to the database without having to think about the database schema or the actual mechanism for saving and retrieving the data. From our code, all we need to deal with is our custom object. For example, we can create and object of type Foo and give it properties a, b and c. By using the Hierarchical Object Store, we can persist the values of these properties by simply saving the object. Handy eh?

So how does it work?

The key to using the Hierarchical Object Store (I’ll call it HOS from now on, RSI is a nasty business), is the SPPersistedObject class. By deriving custom classes from this base type, we have everything that we need to persist data to and read data from the HOS. Simple. Why the look of cynicism?

Ok, so maybe, like many things in SharePoint development, it’s not quite as simple as that. Lets take a look at a few examples and I’ll highlight some gotchas and interesting features along the way.

Why is it hierarchical?

We’ve looked at how to create custom objects and how the SPPersistedObject base class provides the functionality to persist them to the store but one thing we haven’t talked about is the Hierarchical part of the HOS. Where does that come into it? If we look at the constructors to SPPersistedObject we can find out:

protected SPPersistedObject(string name, SPPersistedObject parent);

protected SPPersistedObject(string name, SPPersistedObject parent, Guid id);

In order to create a persisted object we need to specify a parent object which must also be derived from SPPersistedObject. Consequently the HOS assumes a hierarchical nature with each object being defined as a child of another SPPersistedObject. By adopting this approach, the HOS can automatically clean up child objects when their parents are removed from the store without us having to write specific code to do it. This is significant because we can arbitrarily create child objects, if the store was not hierarchical cleaning up this related objects would be challenging. This talk of hierarchy begs the question, If all objects need to have a parent, what’s the root object? Since the HOS is implemented in the farm configuration database, I’m sure it’ll come as no surprise to learn that the root object is Microsoft.SharePoint.Administration.SPFarm. (Which is of course derived from SPPersistedObject)

Enough talk, show me some code

This short sample shows how to create a custom object for persisting global state data:

using System.Runtime.InteropServices;
using Microsoft.SharePoint.Administration;

namespace PersistedObjectDemo
{
    [Guid("F5BC7474-8F88-4C6E-A71C-C56CF8FA3FD8")]
    internal class MyPersistedObject : SPPersistedObject
    {
        [Persisted] private int _bar;
        [Persisted] private string _foo;

        public MyPersistedObject()
        {
        }

        public MyPersistedObject(string valForFoo, string name, SPPersistedObject parent)
            : base(name, parent)
        {
            _foo = valForFoo;
        }

        public string Foo
        {
            get { return _foo; }
            set { _foo = value; }
        }

        public int Bar
        {
            get { return _bar; }
            set { _bar = value; }
        }
    }
}

There are a few things to note in this sample. Firstly, the decoration of the class with GuidAttribute. A proper explanation of why we need this requires some understanding of how the persistence mechanism works. SPPersistedObject is derived from SPAutoSerializingObject and it’s within SPAutoSerializingObject that much of the serialization magic happens. To cut a long story short, reflection is used to build a list of fields that should be persisted, the values of these fields are then stored as XML and the resultant XML is saved to the configuration database by code within the SPPersistedObject base class. In order to deserialize an object from the HOS, the process needs to know the type of the object to populate with the XML-based fields that were stored previously. As we’ll see later, the system knows which type to use because we must pass in the type when attempting to get the object. However, when it comes to retrieving the XML-based field data from the database, the classid of the type is used. For those of us unfortunate enough to remember classid’s from the days of VB6 and COM, yep these are the same beasties! In the .Net framework, every type has a unique classid. Normally we don’t need to know about these unless we’re doing COM Interop since they’re automatically generated by the framework when our code is compiled. However, since they’re auto-generated, they’re not fixed and that can cause a problem with deserialization since the classid may have changed between the time the object was serialized and when the time comes to deserialize it. We’ll see this problem a lot when developing since every time we compile the classid’s can change. By decorating a class with GuidAttribute, we can specify a fixed classId and all will be well in the world. Another SharePoint mystery unravelled!

The next thing to note in the code sample is the use of PersistedAttribute on our fields. Only fields decorated with PersistedAttribute will be serialized by the underlying SPAutoSerializingObject. Note that the PersistedAttribute can only be applied to fields. It won’t work on properties. This means that we can’t really use auto-implemented properties such as:

public string Foo { get; set; }

Since the backing field is added at compile time and therefore can’t be decorated with the PersistedAttribute.

The final thing to note is the constructors that we’ve provided. We have a public constructor that takes no arguments. This is a requirement to support serialization. Note that unlike Xml serialization, our class does not need to be public. Our main constructor overrides one of the base constructors on SPPersistedObject while also accepting a default property value. As we saw earlier, SPPersistedObject has two constructors that we can override. The difference between them is in how we assign an Id for our object.  We can either pass in a specific guid or allow the base SPPersistedObject class to generate one for us.

How can I use this object?

We can use a console application to play around with HOS. These few lines of code will create a new object and store it in the HOS as a child of the SPFarm object.

static void Main(string[] args)
{
    var farm = SPFarm.Local;
    var obj = new MyPersistedObject("this is a test", "Test Obj", farm);
    obj.Update();
}

If we try to run this code again we’ll get:

An object of the type PersistedObjectDemo.MyPersistedObject named “Test Obj” already exists under the parent Microsoft.SharePoint.Administration.SPFarm named “SharePoint_Config”.  Rename your object or delete the existing object.

image

Welcome to the first gotcha when dealing with HOS. Objects must have unique names within the context of their parent. For example, we can’t have two objects named ‘My Object’ as children of the SPFarm object. Note that the type of object and the object ID don’t make any difference, we still can’t have two with the same name.

So what’s the best way to deal with this? One possible approach is to treat persisted objects as singletons and you’ll see this approach being taken with many of the SharePoint server object model classes such as SPFarm. To demonstrate this we can modify our code as follows:

using System.Runtime.InteropServices;
using Microsoft.SharePoint.Administration;

namespace PersistedObjectDemo
{
    [Guid("F5BC7474-8F88-4C6E-A71C-C56CF8FA3FD8")]
    internal class MyPersistedObject : SPPersistedObject
    {
        private const string MY_OBJECT_NAME = "My Test Object";
        [Persisted] private int _bar;
        [Persisted] private string _foo;

        public MyPersistedObject()
        {
        }

        private MyPersistedObject(SPPersistedObject parent)
            : base(MY_OBJECT_NAME, parent)
        {
        }

        public static MyPersistedObject Local
        {
            get
            {
                SPPersistedObject parent = SPFarm.Local;
                var obj = parent.GetChild<MyPersistedObject>(MY_OBJECT_NAME);
                if (obj == null)
                {
                    obj = new MyPersistedObject(parent);
                    obj.Update();
                }

                return obj;
            }
        }

        public string Foo
        {
            get { return _foo; }
            set { _foo = value; }
        }

        public int Bar
        {
            get { return _bar; }
            set { _bar = value; }
        }
    }
}

We now have a static Local property that will return a single instance of our object. We can use it with code similar to this:

static void Main(string[] args)
{
    var obj = MyPersistedObject.Local;
    obj.Foo = "Some value";
    obj.Update();
}

Since we’ve adopted a singleton pattern we can call this code as often as we like in the knowledge that we’re only dealing with a single instance of our object.

Are there any limitations on the types of fields that can be persisted?

Just like other types of serialization, there are limitations on what can be serialized. All primitives can be serialized as can the following:

System.Uri System.Guid System.TimeSpan System.Security.SecureString
System.DateTime System.Version System.Type  

As well as these types, which are serialized directly, it’s also possible to serialize fields derived from SPAutoSerializingObject or SPPersistedObject. Arrays of these objects and IList<T> and IDictionary<T> where T is one of the listed types can also be serialized.

We can see this in action by adding the following property to our object:

  internal class MyPersistedObject : SPPersistedObject
    {
.....
        [Persisted] private List<string> _myStrings=new List<string>();

        public List<string> MyStrings
        {
            get { return _myStrings; }
            set { _myStrings = value; }
        }
.....
    }

We can then use this new list as follows:

static void Main(string[] args)
{
    var obj = MyPersistedObject.Local;
    obj.Foo = "Some value";
    obj.MyStrings.Add("A string");
    obj.Update();
}

So far so good. Everything works as expected. Let’s take a look at what happens if we use collections of more complex objects.

Real World Scenario – Commonly, if we’re planning to offload processing to an asynchronous process we’ll create a timer job by deriving from the SPJobDefinition class. SPJobDefinition is based on SPPersistedObject and as a result the techniques discussed here have particular relevance. Rather than creating a timer job for every task that we need to offload, a better approach is to create a timer job for each type of task and have the job process individual tasks whenever it runs. This is a typical scenario where you’d use HOS and complex child objects. The custom job definition is persisted as a child of the Farm  and our tasks are saved as children of the job definition.

We can add a second custom persisted object to demonstrate what happens if we make this a child of our first object using the following code:

In AnotherPersistedObject.cs, add:

using System.Runtime.InteropServices;
using Microsoft.SharePoint.Administration;

namespace PersistedObjectDemo
{
    [Guid("F5BC7474-8F88-4C6E-AAAA-C56CF8FA3FD8")]
    internal class AnotherPersistedObject : SPPersistedObject
    {
        [Persisted]
        private string _someValue;

        public AnotherPersistedObject()
        {

        }

        public AnotherPersistedObject(string name, SPPersistedObject parent)
            : base(name, parent)
        {

        }

        public string SomeValue
        {
            get { return _someValue; }
            set { _someValue = value; }
        }
    }
}

Add the following additional property to MyPersistedObject.cs:

[Persisted]
private List<AnotherPersistedObject> _childObjects = new List<AnotherPersistedObject>();
public List<AnotherPersistedObject> ChildObjects
{
    get { return _childObjects; }
    set { _childObjects = value; }
}

We can make use of this new property by changing Program.cs as follows:

static void Main(string[] args)
{
    var obj = MyPersistedObject.Local;
    obj.Foo = "Some value";

    obj.ChildObjects.Add(new AnotherPersistedObject("My first object", obj) { SomeValue = "First value" });
    obj.ChildObjects.Add(new AnotherPersistedObject("My second object", obj) { SomeValue = "Second value" });
    obj.ChildObjects.Add(new AnotherPersistedObject("My third object", obj) { SomeValue = "Third value" });
    obj.Update();

    obj.Uncache();
    obj = MyPersistedObject.Local;

    int i = 1;
    foreach (var s in obj.ChildObjects)
    {
        Console.WriteLine("item {0} value - {1}", i, s.SomeValue);
        i++;
    }

    Console.WriteLine("Press return to exit");
    Console.ReadLine();
}

The above code is pretty straightforward: we’re creating three child objects and adding them to our ChildObjects collection before saving the parent. We’re then iterating through the collection of child objects to confirm that they’ve been saved properly.

However, if we run this program we’ll get:

NullReferenceException was unhandled: Object reference not set to an instance of an object.

image

If we attach a debugger we can see that the error occurs on:

Console.WriteLine("item {0} value - {1}", i, s.SomeValue); 

Welcome to the second gotcha when working with SPPersistedObject:

Although we can create lists, dictionaries and arrays of other SPPersistedObjects, behind the scenes all that is persisted is a reference to each child object. Our child collection reports that it contains 3 objects but when we iterate them we find that they’re not actually there!

To fix this problem we need to ensure that each child object is persisted before calling Update on our parent object. We can do this at the time of creation or we can override the Update method of our parent object. There are drawbacks to both approaches. Calling update at the time of creation can potentially mean that we have orphaned child objects if we don’t call Update on our parent object. Given the naming limitations highlighted in gotcha #1, this could create a difficult to debug problem. On the other hand, overriding the Update method hides what’s going on and this could lead to some confusion.

On balance, overriding the Update method is probably the safest option. We can do this as follows:

public override void Update()
{
    var success = new List<AnotherPersistedObject>();

    //Remove any referenced objects that no longer exist
    _childObjects.RemoveAll(o => o == null);

    foreach (var childObject in _childObjects)
    {
        try
        {
            childObject.Update();
            success.Add(childObject);
        }
        catch
        {
            //remove previously persisted objects to prevent orphans
            foreach (var o in success)
            {
                o.Delete();
            }
            throw;
        }
    }
    base.Update();
}

If we now run our code, we’ll see the results as expected:

image

SPAutoSerializingObject – The pain free alternative

So as we can see, it’s possible to add complex child objects derived from SPPersistedObject but care has to be taken to prevent orphans and to ensure that everything is persisted properly. Thankfully, it doesn’t have to be so painful though. Allow me to introduce the hitherto cast-aside, SPAutoSerializingObject.

Using collections a class derived from SPAutoSerializingObject, we have practically the same degree of flexibility as we do with SPPersistedObject. However there’s one significant difference – whereas collections of SPPersisedObjects are stored as references, collections of SPAutoSerializingObjects are stored as inline serialized values. So what? I hear you say. Let’s modify our custom persisted object to accept another collection of child objects and I’ll demonstrate the significance.

Add a new class named AutoSerializedObject with the following code:

using Microsoft.SharePoint.Administration;

namespace PersistedObjectDemo
{
    internal class AutoSerializedObject : SPAutoSerializingObject
    {
        [Persisted] private string _someValue;

        public string SomeValue
        {
            get { return _someValue; }
            set { _someValue = value; }
        }
    }
}

Then modify MyPersistedObject to incorporate a dictionary of our new child class. Note that we could use a list or and array, I’m using a dictionary just to illustrate that unlike normal XML serialization where properties derived from IDictionary are not serializable, when using a class derived from SPAutoSerializingObject, we can handle dictionaries effortlessly.

[Persisted]
private Dictionary<string, AutoSerializedObject> _childDictionary=new Dictionary<string, AutoSerializedObject>();
public Dictionary<string, AutoSerializedObject> ChildDictionary
{
    get { return _childDictionary; }
    set { _childDictionary = value; }
}

We can see this new collection in action by modifying our Main function as follows:

static void Main(string[] args)
{
    var obj = MyPersistedObject.Local;
    obj.Foo = "Some value";

    obj.ChildDictionary.Add("First", new AutoSerializedObject(){SomeValue = "First value"});
    obj.ChildDictionary.Add("Second", new AutoSerializedObject() { SomeValue = "Second value" });
    obj.ChildDictionary.Add("Third", new AutoSerializedObject() { SomeValue = "Third value" });

    obj.Update();

    obj.Uncache();
    obj = MyPersistedObject.Local;

    foreach (var s in obj.ChildDictionary)
    {
        Console.WriteLine("item {0} value - {1}", s.Key, s.Value.SomeValue);
    }

    Console.WriteLine("Press return to exit");
    Console.ReadLine();
}

As you can see when running this code, our child objects are serialized and retrieved correctly – without us having to add any additional code. In many situations using classes derived from SPAutoSerializingObject is a better approach than jumping straight in with a custom SPPersistedObject.  Naturally there are many factors to be considered when deciding on an appropriate base class but two of the key deciders are:

  • Will I need to pick up a reference to the class without first having a reference to its parent? If the answer is yes than SPPersistedObject is the only viable choice. By using code such as this, we can get a reference to any object within the HOS:
Guid myObjectGuid;

var myObject = SPFarm.Local.GetObject(myObjectGuid);

 

  • How complex is my child object and how many of them will there be? If the child object is complex, possibly containing other child objects and particularly if there are many of them, SPPersistedObject may be a better choice because the serialized value is split among multiple rows in the database. Bear in mind that SPAutoSerializingObject cannot contain child properties of type SPPersistedObject so all children will also be persisted within the database row of the parent. Depending on which version of SQL server is being used, the maximum size of the serialized data can be up to 2Gb, but just because you can, doesn’t always mean that you should!

Security and the HOS

One final topic that requires some coverage is security. As described earlier, the HOS exists in the farm configuration database. Strictly speaking it exists to support administrative functionality however it has uses beyond this scope. For example, if we’re building a custom timer job and we want to have the ability to submit tasks to our job for completion. It’s most likely that these tasks will originate within a non-administrative scope. Possibly as the result of a user clicking a button on a webpart on some other site collection. With previous versions of SharePoint, this simply wasn’t possible. Only farm administrators could update the HOS. However with SharePoint 2010, an addition has been made to the base SPPersistedObject that opens up the HOS to other users. To see how this works, we can modify our Main function as follows:

using System;
using System.Runtime.InteropServices;
using System.Security.Principal;

namespace PersistedObjectDemo
{
    internal class Program
    {
        private const int LOGON_TYPE_INTERACTIVE = 2;
        private const int LOGON_TYPE_PROVIDER_DEFAULT = 0;

        [DllImport("advapi32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
        public static extern bool LogonUser(string userName, string domain, string password, int logonType,
                                            int logonProvider, ref IntPtr accessToken);

        private static void Main(string[] args)
        {
            try
            {
                IntPtr accessToken = IntPtr.Zero;
                if (LogonUser("non-farm-admin-username", "your-domain-name", "your-password", LOGON_TYPE_INTERACTIVE,
                              LOGON_TYPE_PROVIDER_DEFAULT, ref accessToken))
                {
                    using (var identity = new WindowsIdentity(accessToken))
                    {
                        using (identity.Impersonate())
                        {
                            MyPersistedObject obj = MyPersistedObject.Local;
			       //delete existing object and recreate to keep things simple
                            obj.Delete();
                            obj = MyPersistedObject.Local;
                            obj.Foo = "Some updated value";
                            obj.Update();
                        }
                    }
                }

            }
            catch (Exception ex)
            {
                Console.Write(ex);
            }

            Console.WriteLine("Press return to exit");
            Console.ReadLine();
        }
    }
}

This code simulates an attempt by a non-farm admin user to update the HOS. If we run this we’ll see the following exception which is our gotcha number 3:

System.Security.SecurityException: Access denied. at Microsoft.SharePoint.Administration.SPPersistedObject.BaseUpdate() at PersistedObjectDemo.MyPersistedObject.Update()

Unsurprisingly, trying to update the HOS using a non-farm admin user throws an exception. However, if we add the following code to MyPersistedObject.cs and run our program again, we can see that the HOS is updated appropriately.

protected override bool HasAdditionalUpdateAccess()
{
    return true;
}

Of course, in a real-world application, we’d probably want to add some security checking in this function rather than simply returning true and allowing anybody to update our object but hopefully this illustrates the point sufficiently.

There is one other gotcha with regard to security – when SharePoint is 2010 installed, by default only the farm admin account has read/write access to the config database. In a least-privilege installation, all web applications other then Central Administration, will be running under their own service account and these accounts will not have write access to the config database. Consequently, adding, deleting and updating objects in the HOS will not be possible. Although overriding HasAdditionalUpdateAccess allows us a bit more control over who can update the HOS, if the underlying app pool account doesn’t have the appropriate database permissions we’re going to come unstuck pretty quickly.

In my opinion, denying access to the HOS for web application app pool accounts is a step too far. Given the default configuration of only allowing farm admins to make changes to persisted objects, granting execute permissions on the stored procedure that updates HOS for all application pool accounts does not represent a security risk. Of course, I’m a developer not a security expert and your mileage may vary! The only workarounds for this gotcha are:

  1. Run any web application that’s needs to modify the HOS under the same service account as Central Admin, or,
  2. modify the the WSS_Content_Application_Pools database role in the Config database to allow execute permissions on proc_putObject. (I’m sure that I don’t need to say that such a modification is neither endorsed or supported by the good folks at Microsoft, I merely include it here in the interests of completeness!)

Conclusion

SPPersistedObject, SPAutoSerializingObject and the Hierarchical Object Store are useful tools. Maybe in a future article I’ll look at one of the most common uses of these tools – the creation of custom timer jobs. As we’ve seen, there are limitations and nuances but I believe that every serious SharePoint developer needs to know and understand how these objects work and hopefully this article served as a useful introduction.

This entry was posted in Deployment, Hierarchical Object Store, HowTo, InDepth, Security, Serialization, SharePoint 2010, Tip, XML. Bookmark the permalink.
  • http://www.novolocus.com/ Andy Burns

    I’m glad you got around to Least Privileges config – I was beginning to worry that I’d missed something obvious when looking at this.

    I agree about the restrictions of least privileges, but the main problem is, I think, that a lot of folks will read and follow microsoft’s advice on Least Privs being best without analysis – which isn’t unreasonable. If you’re told by the manufacturer that ‘this is the most secure’ and you’re interested in security, well, that’s what you’re going to do.

    But it does basically make the hierarchical object store useless for actual users.

  • http://gl0.com/ fahad
  • Vitaly

    Thank you for post!

  • Jim

    Query: When I change my class to inherit from SPAutoSerializingObject I can no longer use the GetChild method from the Local property you implemented in the singleton pattern. How do I retrieve the SPAutoSerializingObject???

  • http://www.chaholl.com/Default.aspx chaholl

    Generally the SPAuto… object would be a property of an SPPersistedObject otherwise it won’t get stored anywhere.

  • Janak

    I’m inheriting from SPAutoSerializingObject, but still gives me Object reference error. Any ideas why? Thanks!

    This is when I try to reference the property of the SPPersistedObject: [Persisted()] private List<ClaimsProviderMapping> m_providerMappings = new List<ClaimsProviderMapping>();

    Here’s my SPAuto… class:
    public class ClaimsProviderMapping : SPAutoSerializingObject
    {

    [Persisted] private string role;
    [Persisted] private string roleType;
    [Persisted] private string authprovidername;

    /// <summary>
    /// Role, i.e., Security Group
    /// </summary>
    public string Role
    {
    get
    {
    return role;
    }
    set
    {
    role = value;
    }
    }
    /// <summary>
    /// Claim Type of Role/Security Group
    /// </summary>
    /// <example>SPOriginalIssuerType.Forms, SPOriginalIssuerType.Windows, etc.</example>
    public string RoleType
    {
    get
    {
    return roleType;
    }
    set
    {
    if (Enum.Parse(typeof(SPOriginalIssuerType), value, true) == null)
    {
    throw new ArgumentOutOfRangeException("RoleType", value, "The value does not correspond to the a valid SPOriginalIssuerType");
    }
    else
    {
    roleType = value;
    }
    }
    }
    /// <summary>
    /// Name of Claims Authentication Provider
    /// </summary>
    /// <example>Windows Authentication, Forms Authentication, etc.</example>
    public string AuthorizationProviderName
    {
    get
    {
    return authprovidername;
    }
    set
    {
    authprovidername = value;
    }
    }
    }

  • http://www.facebook.com/simtex Simon J.K. Pedersen

    Good article, very informative. 

  • Pingback: SharePoint SPPersistedObject and Hierarchical Object Store ‹ Simon J.K. Pedersen – computer technology and the meaning of life

  • Only_arc

    Hi,
    Regarding security, just wondering if the app pool account has atleast read access to the persisted object? Or would I need to give permissions for that?
    Thanks.

  • Pingback: Curia Damiano blog | SharePoint: SPPersistedObject and hierarchical object store

  • Arun

    Nice Article. Thank you for the post

    -Arun

  • Shikja

    Can any user (including anonymous) read the stored values in SPPersistedObject/SPFarm.Local?