Charlie Holland Freelance SharePoint Consultant, Author and Trainer

Ever had the situation where you need to perform some long running administrative process on a file or some other stream? Creating long-running processes with SharePoint is easy enough by creating a new class that’s derived from SPJobDefinition, but how do you pass in your stream?  Simply put, you can’t! The job will run in a separate process so any data must be serialized and de-serialized.

The answer is to persist the stream to the hierarchical object store. There are many articles out there on how to derive from and use SPPersistedObject, such as this one from Maurice Prather. It’s easy to persist most object properties using a custom SPPersistedObject and you’d certainly be forgiven for persisting a stream by converting it to a byte array.

There is  a better approach though- the object store is basically the contents of the Objects table in the SharePoint_Config database (changes not supported disclaimer etc.), the Properties column contains the serialized SPPersistedObjects and some examination will reveal that it’s of type nvarchar(max), meaning we can stick about 2Gb of data in there (depending on the version of SQL Server you’re running). If you look at that table though, you’ll notice that there aren’t a lot of huge objects in there and when you think about it from a performance perspective, that makes sense, the Properties field is serialized as XML and is parsed in memory by SharePoint. If we stick a few huge objects in there, it’ll slow everything down.

You may be thinking, SharePoint 2010 with it’s new-fangled sandbox and the like, must store binary data somewhere, if it’s not in the Objects table and it’s not in the file system where is it? The answer is that it’s in the Binaries table. Effectively the Binaries table allows BLOB data to be attached to records in the Objects table. How does SharePoint read and write to this table? It uses the SPPersistedFile class (which is ultimately derived from SPPersistedObject).

Cut to the Chase

Now we can use the SPPersistedFile class in our own code but it’s public interface only supports persisting physical files and that may not be practical if you want to persist a file that a user uploads, so my PersistedStreamObject wraps the SPPersistedFile class and accepts a Stream in the constructor. This class can be used in the same way as any other SPPersistedObject and makes use of compression internally to minimise the database space required.

[Guid("5C2EDB3B-ED8B-4615-A8F5-25C24D044835")]
public class PersistedStreamObject : SPPersistedObject
{
    private const int INT_BUFFER_SIZE = 32768;
    private const string STR_OBJECT_NAME = "Stream";

    [Persisted]
    private long _length;

    [Persisted]
    private long _uncompressedLength;

    private Stream _fileStream;

    public PersistedStreamObject()
    {
    }

    public long Length
    {
        get { return _length; }
    }

    public long UncompressedLength
    {
        get { return _uncompressedLength; }
    }

    public PersistedStreamObject(string name, SPPersistedObject parent, Stream stream)
        : base(name,parent)
    {
        _fileStream = stream;
    }

    public static void CopyStream(Stream input, Stream output)
    {
        if (input == null) throw new ArgumentNullException("input");
        if (output == null) throw new ArgumentNullException("output");

        var buffer = new byte[INT_BUFFER_SIZE];
        while (true)
        {
            int read = input.Read(buffer, 0, buffer.Length);
            if (read <= 0)
                return;
            output.Write(buffer, 0, read);
        }
    }

    public override void Update()
    {
        string tmp = Path.GetTempFileName();

        if (_fileStream != null)
        {
            base.Update();

            using (var w = new FileStream(tmp, FileMode.OpenOrCreate, FileAccess.Write))
            {
                var cs = new GZipStream(w, CompressionMode.Compress);

                CopyStream(_fileStream, cs);
                cs.Flush();
                _length = w.Length;

                _uncompressedLength = _fileStream.Length;
            }

            _fileStream.Close();
            _fileStream.Dispose();

            base.Update();
            var child = GetChild<SPPersistedFile>(STR_OBJECT_NAME);

            if (child != null) child.Delete();

            var file = new SPPersistedFile(STR_OBJECT_NAME, tmp, this);
            file.Update();
            File.Delete(tmp);
        }
    }

    [SuppressMessage("Microsoft.Usage","CA2202",Justification="Using statement handles Dispose")]
    public Stream Stream
    {
        get
        {
            if (_fileStream != null)
            {
                string tmp = Path.GetTempFileName();
                var file = GetChild<SPPersistedFile>(STR_OBJECT_NAME);

                if (file != null)
                {
                    file.SaveAs(tmp);
                    using (var stream = new MemoryStream())
                    {
                        using (var fs = File.OpenRead(tmp))
                        {
                            var cs = new GZipStream(fs, CompressionMode.Decompress);
                            CopyStream(cs, stream);
                            cs.Flush();
                            fs.Close();
                        }

                        File.Delete(tmp);

                        _fileStream = stream;
                    }
                }
            }
            return _fileStream;
        }

    }
}