External BLOB/Binary Store for Windows SharePoint Services 3.0 in C#/.NET 2.0 - Part II

Part II – The COM Component

Overview

In the previous post in this series we discussed the exposure of an external storage API from Window SharePoint Services, Microsoft’s implementation documents, what I have been able to figure out as it relates to the implementation under the covers, the architectural decisions that you must make, and the architectural decisions that I’ve made for this blog series.  If you have not read it, please make sure that you do before you continue.

We are going to focus on the COM Component in this blog entry.  This is easily the most important and most difficult piece of this whole solution which makes this one the longest blog in the series…sorry.  This is also the most technical and detailed entry so I’m going to try my best to hold back on the sarcasm, but as a result this one will be very dry (not like the previous one was much better). 

For those who aren’t going to read the previous blog, all I’m going to tell you is that we are doing this as a C# .NET 2.0 solution.

Now, back to my attempted cure for insomnia and excitement…

The EBS Provider

To implement the COM component for the EBS provider you will need to create a C# class project, prepare the class project for GAC installation, create the interface files from the IDL, prepare the provider class for COM Interop, implement the interface methods, create the ILockByte support methods, and create the memory dereferencing method.

Create the C# Class Project
I’m assuming that you know how to create a C# Class project.  If not, then you can read the Microsoft docs here: http://msdn2.microsoft.com/en-us/library/ms173077(VS.80).aspx.

GAC Settings
For this you will need to add your key file to the project and then in the project properties on the “Signing” tab check the “Sign the assembly” checkbox and select the key file in the “Choose a strong name key file:” dropdown.  For further information on this see Global Assembly Cache concepts at http://msdn2.microsoft.com/en-us/library/yf1d93sz(VS.80).aspx

To help with deployment and development you should consider setting these values on the “Build Events” tab:
Pre-build event command line:
"$(DevEnvDir)..\..\SDK\v2.0\bin\gacutil" /u "$(TargetName)"

Post-build event command line:
"$(DevEnvDir)..\..\SDK\v2.0\bin\gacutil" /i "$(TargetPath)"

Run the post-build event:
On successful build

This will remove the project from the GAC before the build and add the project to the GAC after a successful build.

Interface Implementation
There are two interfaces that must be implemented for this component.  They are the ISPExternalBinaryProvider and the ILockBytes interfaces.

Here is the IDL for the ISPExternalBinaryProvider as provided in the Microsoft implementation documentation (http://msdn2.microsoft.com/en-us/library/bb802811.aspx):

/*************************************************
    File: extstore.idl
    Copyright (c): 2006 Microsoft Corp.
*************************************************/
import "objidl.idl";

[
    uuid(39082a0c-af6e-4ac2-b6f0-1a1ff6abbae1)
]

library SharePointBinaryStore
{
    [
        local,
        object,
        uuid(48036587-c8bc-4aa0-8694-5a7000b4ba4f),
        helpstring("ISPExternalBinaryProvider interface")
    ]
    interface ISPExternalBinaryProvider : IUnknown
    {
        HRESULT StoreBinary(
            [in] unsigned long cbPartitionId,
            [in, size_is(cbPartitionId)] const byte* pbPartitionId,
            [in] ILockBytes* pilb,
            [out] unsigned long* pcbBinaryId,
            [out, size_is(, *pcbBinaryId)] byte** ppbBinaryId,
            [out,optional] VARIANT_BOOL* pfAccepted);

        HRESULT RetrieveBinary(
            [in] unsigned long cbPartitionId,
            [in, size_is(cbPartitionId)] const byte* pbPartitionId,
            [in] unsigned long cbBinaryId,
            [in, size_is(cbBinaryId)] const byte* pbBinaryId,
            [out] ILockBytes** ppilb);
    }
}

 

For my implementation I took this IDL and ran it through the MIDL compiler (midl.exe) to get a type library and then through the Type Library Importer (tlbimp.exe) to get an assembly.  Using the IDL file that way created a bunch of gross looking code that was a pain to work with. I took some time through trial and error and came up with the following interface representations for both the ISPExternalBinaryProvider and the ILockBytes that work in a .NET implementation.  I think these are much cleaner and easier to work with.  By the way, each of these where in their own .cs file without any namespace information.

[ComImport, ComConversionLoss, Guid("48036587-C8BC-4AA0-8694-5A7000B4BA4F"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface ISPExternalBinaryProvider
{
[MethodImpl(MethodImplOptions.InternalCall, 
MethodCodeType=MethodCodeType.Runtime)]
    void StoreBinary([In] uint cbPartitionId, 
                     [In] ref byte pbPartitionId, 
                     [In, MarshalAs(UnmanagedType.Interface)] ILockBytes pilb, 
                     out uint pcbBinaryId, 
                     out IntPtr ppbBinaryId, 
                     [Optional] out bool pfAccepted);

    [MethodImpl(MethodImplOptions.InternalCall, 
MethodCodeType = MethodCodeType.Runtime)]
    void RetrieveBinary([In] uint cbPartitionId, 
                        [In] ref byte pbPartitionId, 
                        [In] uint cbBinaryId, 
                        [In] ref byte pbBinaryId, 
                        [MarshalAs(UnmanagedType.Interface)] out ILockBytes ppilb);
}

[ComImport, Guid("0000000A-0000-0000-C000-000000000046"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface ILockBytes
{
    [MethodImpl(MethodImplOptions.InternalCall, 
        MethodCodeType=MethodCodeType.Runtime)]
    void ReadAt([In] UInt64 ulOffset, 
                [In] IntPtr pv, 
                [In] uint cb, 
                [Out] out uint pcbRead);

    [MethodImpl(MethodImplOptions.InternalCall, 
MethodCodeType = MethodCodeType.Runtime)]
    void WriteAt([In] UInt64 ulOffset, 
                 [In] IntPtr pv, 
                 [In] uint cb, 
                 [Out] out uint pcbWritten);

    [MethodImpl(MethodImplOptions.InternalCall, 
MethodCodeType = MethodCodeType.Runtime)]
    void Flush();

    [MethodImpl(MethodImplOptions.InternalCall, 
        MethodCodeType=MethodCodeType.Runtime)]
    void SetSize([In] UInt64 cb);

    [MethodImpl(MethodImplOptions.InternalCall, 
        MethodCodeType=MethodCodeType.Runtime)]
    void LockRegion([In] UInt64 libOffset, 
                    [In] UInt64 cb, 
                    [In] uint dwLockType);

    [MethodImpl(MethodImplOptions.InternalCall, 
        MethodCodeType=MethodCodeType.Runtime)]
    void UnlockRegion([In] UInt64 libOffset, 
                      [In] UInt64 cb, 
                      [In] uint dwLockType);

    [MethodImpl(MethodImplOptions.InternalCall, 
        MethodCodeType=MethodCodeType.Runtime)]
    void Stat([Out, MarshalAs(UnmanagedType.Struct)] out STATSTG pstatstg, 
              [In] int grfStatFlag);
}

 

One thing that I want to point out now is the difference in method declaration for the StoreBinary and RetrieveBinary in the interfaces above versus the IDL and Microsoft’s documentation.  The documentation says that the methods must return an HRESULT of S_OK or E_FAIL, but the methods in the interface are declared as void.  The reason for this is that when I tried to implement them to return the HRESULT it caused the process to fail miserably and causes SharePoint to hang.  When I changed them to void and stopped returning values, then everything worked well.

COM Interop Preparation
In the project properties you will need to check the “Register for COM Interop” checkbox on the “Build” tab.

The provider class must inherit the ISPExternalBinaryProvider interface and have a public default constructor even if it is empty.  Also, you will need to set these class attributes.

•    [ProgId("[Object Name].[Class Name]")]
This is the ProgID for the COM class.  Details on the ProgIdAttribute class can be found at http://msdn2.microsoft.com/en-us/library/system.runtime.interopservices.progidattribute(VS.80).aspx.

•    [Guid("00000000-0000-0000-0000-000000000000")]
This is a new GUID generated using the GuidGen.exe tool shipped with Visual Studio. Detail on the GuidAttribute class can be found at http://msdn2.microsoft.com/en-us/library/system.runtime.interopservices.guidattribute(VS.80).aspx.

•    [ClassInterface(ClassInterfaceType.None)]
Defines the class interface type and for this project I used the above values.  Details on the ClassInterfaceAttribute call can be found at http://msdn2.microsoft.com/en-us/library/system.runtime.interopservices.classinterfaceattribute(VS.80).aspx.

 

Implement the Interface Methods
You must explicitly implement the interface members.  The easiest way to do this is let Visual Studio do this for you.  If you hover your mouse over the ISPExternalBinaryProvider after the : in the class definition you will notice a little blue rectangle/line under the “I”.  If you click on that or press “Shift+Alt+F10” you will get a couple of options including “Explicitly implement interface ‘ISPExternalBinaryProvider’”.  When it has finished it should look like this: 

void ISPExternalBinaryProvider.RetrieveBinary(… 

 

Notice that there is no explicit scope (public, private, internal, etc) on the method definitions.  That’s how they should be so don’t change it.

In the RetrieveBinary method you will basically need to dereference the partition Id, the binary Id, retrieve the byte[] from the EBS file manager, create an ILockBytes objects, and set the ppilb output parameter to the ILockBytes object.  I recommend doing all of this in a try/catch block and because of this you will need to explicitly set the ppilb equal to null at the top of the method to get the component to compile.

In the StoreBinary method you will need to dereference the partition Id, read the byte[] out of the ILockBytes, write the byte[] to a file associated with the partition Id which should create a new binary Id for you, dereference the binary Id into the pcbBinaryId for the size and the ppbBinaryId for the first byte of the binary Id, and finally you need to set the pfAccepted to true if you were able to write the file or false if SharePoint should take care of writing the file.  Again, this should be done in a try/catch block and the three output parameters should be set to default values.  Something to remember is that to dereference the binary Id going back to SharePoint that you should first convert it to a byte[].

In the following two sections I’ll discuss how to read the byte[] from and write the byte[] to an ILockBytes object and how to dereference the memory for the Id pointers.

ILockByte Support
The ILockBytes interface supports 3 methods that we use in this process: Stat, ReadAt, and WriteAt.  We don’t need to use any of the other exposed methods.

To read a byte[] from an ILockBytes interface you need to create an memory buffer using Marshal.AllocHGlobal([buffer size]) (I used 8192 for the buffer size), call the Stat method to get the number of bytes in the ILockBytes, create the a byte[] equal to the resulting size, and loop through reading the bytes from the ILockBytes using ReadAt and write them into the resulting byte[] (shown below).  Don’t forget to free the memory using Marshal.FreeHGlobal([buffer variable]).  So...I was wanting to give you the method for reading the bytes from an ILockBytes but that request was denied by my employer (I know, I don't know why either).  Anyway I found a way to do this in the Memory generation of Excel files.

do
{
    lockBytes.ReadAt(offset, buf, (UInt32)8192, out bytesRead);
    if (bytesRead > 0)
    {
        Marshal.Copy(buf, bytes, (Int32)offset, (Int32)bytesRead);
        offset += bytesRead;
    }
} while (bytesRead > 0);

 

To write a byte[] to an ILockBytes interface you will need to reference an external method in the OLE32.dll called CreateILockBytesOnHGlobal.  Here is the code for that declaration:

[DllImport("ole32.dll")] static extern int CreateILockBytesOnHGlobal(IntPtr hGlobal,
                                                                     bool fDeleteOnRelease,

                                                                     out ILockBytes ppLockbytes); 
 

Now that you have the external definition you can continue with writing a byte[] to an ILockBytes. To write the byte[] you need to create the resulting ILockBytes object, get the size of the byte[], call the CreateILockBytesOnHGlobal, allocate a buffer, and loop through writing the bytes into the ILockBytes using the WriteAt method (shown below).  Again, I was hoping to be able to give you the entire method, but that request was denied.  For this one, basically take the opposite of what you've done for reading from the ILockBytes.

while (byteSize > 0)
{
     bytesRead = (byteSize > 8192 ? 8192 : (Int32)byteSize);
     Marshal.Copy(bytes, (Int32)offset, buf, bytesRead);
     lockBytes.WriteAt(offset, buf, (UInt32)bytesRead, out bytesWritten);
     if (bytesWritten == 0)
     {
         throw new ApplicationException("Unable to write to contents");
     }
     offset += bytesWritten;
     byteSize -= bytesRead; 
} 

 

That pretty much covers reading a byte[] from and writing a byte[] to an ILockBytes interface.

Memory Dereferencing
The last thing that you need to do in this component is to have some way to dereference the pointers coming in from SharePoint and going out to SharePoint.  The only way to achieve this that I could find is to use unsafe code and in order to do that you will need to edit the project properties and select the “Allow unsafe code” in the “General” section on the “Build” tab.

To dereference the pointer coming in from SharePoint you will need to create a byte[] buffer to the size indicated, create a byte* equal to the incoming byte from SharePoint using the fixed keyword, and then perform a memory copy (shown below).  I chose to make this a method since it is needed multiple times and since it is a method using the fixed keyword it has to be flagged as unsafe. 

fixed (byte* refBytes = &bytes)
{
     Marshal.Copy(new IntPtr(refBytes), buffer, 0, (Int32)size); 
} 

 

To dereference the newly created binary Id going back to SharePoint you need to convert the Id to a byte[],  set the pcbBinaryId to the number of bytes in the Id, allocate the memory on the heap, and then copy the bytes into the allocated memory using the Marshal.Copy method.  Since this is needed only once I left this code in the StoreBinary method.  Here is the snippet:

pcbBinaryId = (UInt32)binaryIdBytes.Length;
ppbBinaryId = Marshal.AllocHGlobal((Int32)pcbBinaryId); 
Marshal.Copy(binaryIdBytes, 0, ppbBinaryId, (Int32)pcbBinaryId); 

 

You will notice that we are allocating memory on the heap without releasing it.  If we released it then SharePoint wouldn’t get our id back out.  This is pretty much undocumented in its entirety so I’m hoping the SharePoint is freeing this memory when it is finished with it, otherwise we will have a memory leak here.  Likewise, it isn’t clear who’s supposed to free the memory for the incoming partition Id, so we may have a memory leak there. (Believe me, I’ll rant about this and many other things in the Final Thoughts section of Part IV).

Summary

In this entry we covered all of the technical details for creating the COM Component.  You now know how to setup the COM Interop project, implement the required interface, get information in to and out of the ILockBytes interface, and dereference the memory for the values being passed back and forth with SharePoint.

In the next blog I’ll cover the details for implementing a file manager and an orphan file cleanup process.

Comments (18) -

  • this is great information that i know a lot of people are interested in, especially me and the future kyle. looking forward to the next entries....
  • i have some specific questions, any chance I can get your contact information? i'm at kstiever@hotmail.com
  • Hi Kyle!

    I'm also planning an EBS implementation for WSS, and right now is time for the decisions you have mentioned in the first post of this series.
    I like very much your posts and would like to congratulate you for the technical deep-dive on this series. Sure the best references in this subject i've found so far!
    I think we will have more to talk about in the next few months.

    Cheers!
  • The "Installing and Configuring Your BLOB Provider" page on MSDN (msdn2.microsoft.com/en-us/library/bb802799.aspx) shows the PowerShell script for hooking a custom EBS component into Sharepoint. According to that script (see the 2nd and last lines), there should also be an "Active" boolean property of the component, which presumeably turns the functionality on/off. I don't see such a property in your example, so does that mean that it's not necessary?

    I've completed a draft of a custom EBS component (thanks largely to your BLOG -- thank you!) and simply omitted the PowerShell lines that reference the Active property when I deploy it. I get the error "Unable to get the configured ISPExternalBinaryProvider" in the Sharepoint logs when I try to upload a document, so I'm wondering if the absent Active property is somehow to blame. Can you comment?

    Thanks,
    Brian
  • I hit the same point you did and noticed that property.  I initially assumed that maybe that was something handled by SharePoint, but found out like you that it is something that I was supposed to implement (I’m glad that Microsoft pointed that out in earlier documents in their series).  I simply omitted that from my install script as well and everything went fine.

    I would suspect  from what you are telling me that it is an issue relating to SharePoint resolving the GUID to the COM Component. Reread the COM Interop Preparation section and make sure that you followed those steps correctly.  If you are still having problems, let me know.
  • Thanks for your reply.

    I double-checked the COM settings and they appear to be correct. I also searched the registry on the Sharepoint server for the GUID that I was specifying in my install script to make sure that it resolved to the right ProgId -- which it did. After that, I re-compiled another version of the DLL with some event logging in the constructor and in the store/retrieve methods. After deploying the new version, I attempted another document upload and got the same error in the Sharepoint log, but the event log showed my constructor message. So it does appear that Sharepoint can instatiate the object, however my event message at the beginning of StoreBinary didn't log. Frustrating!

    Can you make anything out of these errors (both appear on every failed upload attempt)?

    ---------------------------------------
    05/05/2008 17:45:49.04   w3wp.exe (0x1130)                         0x1F3C  Windows SharePoint Services     General                         72l8  High      VcomCache: Failed to QI for interface IID = {48036587-C8BC-4AA0-8694-5A7000B4BA4F}, PUNK = 0x01eef88c, HR=0x80004002  
    05/05/2008 17:45:49.04   w3wp.exe (0x1130)                         0x1F3C  Windows SharePoint Services     General                         0  High      Unable to get the configured ISPExternalBinaryProvider. Error 0x80004002.
    ---------------------------------------

    I noticed that the GUID for the interface IID is not the same as the GUID for my component. But I verified that the correct GUID shows in PowerShell (after running my install script, I ran $farm and looked at the ExternalBinaryStoreClassId.

    I greatly appreciate any help.
  • I have the same problem as Brian i marked my custom provider with the attribute     [ProgId("CustomDiskEBSProvider.MyEBSProvider")]
    where "CustomDiskEBSProvider" is my namespace and "MyEBSProvider" is my custom blob provider but it seems that sharepoint cant reach my class
  • i have a problem in my retrieve binary function , how could you get the site id and the blob id from the ref byte parameters pbBinaryId and pbPartitionId if you can explain this it would help a lot
  • Hi,
    A problem occurs when trying to upload massively large files (500 mg +) that SharePoint  expects an Ilockbytes object  locking the uploaded file bytes in memory which causes memory leakage and out of memory exceptions.
    Best Regards,
  • can we overcome this problem by implementing a file-based Ilockbytes not a memory based one (i.e by calling StgCreateDocfile() function instead of CreateILockBytesOnHGlobal() ) to hold our loads on a temp file on disk ???
    is that possible ?
    Zaher
  • Hi,
    Can anyone explain me how to Install and Configure BLOB Provider.
    I have registered EBS provider. Now how to configure that on sharepoint web server
  • Hi,

    When you will cover the details for implementing a file manager and an orphan file cleanup process.

    Or if there please provide me an url.

    Warm Regards
    Dharmesh
  • Hi Brian,
    I faced the same problem, as you've described. Could you please let me know if you've found the solution of that problem?
    Also if anyone else has any thoughts why COM objects constructor is called from SharePoint functionality, but methods are not reachable or visible for SharePoint please let me know?

    Thanks beforehand.
  • Hi Brian,
    I faced the same problem, as you've described. Could you please let me know if you've found the solution of that problem?
    Also if anyone else has any thoughts why COM objects constructor is called from SharePoint functionality, but methods are not reachable or visible for SharePoint please let me know?

    Thanks beforehand.
    Taras
  • Hi Brian,
    I faced the same problem, as you've described. Could you please let me know if you've found the solution of that problem?
    Also if anyone else has any thoughts why COM objects constructor is called from SharePoint functionality, but methods are not reachable or visible for SharePoint please let me know?

    Thanks beforehand.
  • Hi,
    Will you provide all source codes?
    Thanks!

    hooke
  • Hi,

    Have anyone resolved an issue Brian described? I'm getting the same erros in SharePoint log when trying to upload a document, but EBS constructor is invoked fine.

    Thanks


  • Was there a solution found to the "Unable to get the configured ISPExternalBinaryProvider" issue?  I am having the exact same problem.

    I have the dll registered in the GAC and have tried using regasm.exe and am still getting an error code 0x80040154 returned (Class not registered)

    Any help would be greatly appreciated.
Comments are closed