Creating SDL Tridion/Web ECL Provider for AWS S3 bucket

Introduction to ECL

First of all, ECL stands for “External Content Library”. It is a module for exposing multimedia contained in an external system in SDL Web/Tridion so that you can use the media in SDL Web-driven Web sites.

Secondly, Amazon Simple Storage Service (Amazon S3) is object storage with a simple web service interface to store and retrieve any amount of data from anywhere on the web. It is designed to deliver 99.999999999% durability, and scale past trillions of objects worldwide.

When this all started

Recently I was asked to create a solution, an ECL provider, for Tridion with Amazon S3 bucket. It sounded fascinating as this was a bit new to me, not that I didn’t know or heard about ECL but I had never implemented one myself.

Soon I started my quest to learn and see how this works and what it needs. There have been many events where I have gone through a lot of blog posts, Trex talks about ECL. I had great help from the posts and comments Lars Møllebjerg and Bart Koopman has given in many places. I loved the way Lars has explained ECL basics in SDL Docs.

After a bit of R&D, It was time for me to start implementinng it and this blog post aims to provide an overview of my implementation. I  have will put, not fully, but a bit of my code snippet here which will help you walk through the implementation and explain how things are happening.

Perquisites

Please note that AWS S3 buckets are the paid services of amazon and hence prerequisites will cost you/your company for any account/services used.

 

Initial Setup

Before you start implementing it, Let’s first setup the environment.

In Content Manager Explorer (CME)

Multimedia type creation in CME

1. Create a new Multimedia type “S3 External Content Library” by clicking on Administrator > Multimedia type > New Multimedia Type.

 

Multimedia type icons

2. Add icons of size 48×48 px and 16×16 px of your choice in the new Multimedia type.

 

StubFolder

3. Create a folder of your choice in Parent Publication, we call it “StubFoder”

 

Multimedia schema for ECl in CME StubFolder

4. In your Stub Folder, create a Multimedia Schema (say: ExternalContentLibraryStubSchema-s3) and allow your newly created Multimedia Type “S3 External Content Library”

 

In Content Manager Server (CMS)

  1. On your Tridion Content Manager Server (CMS), Copy ExternalContentLibrary.xml from <Tridion_Home>/config/
  2. Add below XML under <MountPoints> tag
<MountPoints> 
<!-- type="S3ECLProvider" : Matchs [AddIn("S3ECLProvider", Version = "1.0.0.0")] in your Provder Code -->
<!-- id="s3" : MountPointId, Hover mouse on the Mountpoint in CME - ecl:2-s3-root-mp-mountpoint -->
<!-- rootItemName="AWS S3 Bucket" : Display name of MountPoint in CME -->
<MountPoint type="S3ECLProvider" version="*" id="s3" rootItemName="AWS S3 Bucket">
<StubFolders>
<!-- StubFolder: Folder your choice in CME in Parent Publication -->
<StubFolder id="tcm:2-84-2" /> <!-- Folder tcmId from your CME -->
</StubFolders>
<!-- PrivilegedUserName: This is often system users (MTSUser or Administrator) -->
<PrivilegedUserName>WIN-PXXXXXXR7\Administrator</PrivilegedUserName> 
<EnableUpload administrators="true"/>
 
<!-- FullBucketUrl: Absolute url of your S3, hit it in browser and you should see your bucket if its public -->
<FullBucketUrl xmlns="http://www.sdltridion.com/S3EclProvider/Configuration">https://s3-us-west-2.amazonaws.com/com-dev-tridion-s3ecl-vikas/</FullBucketUrl> 
 
<S3BucketName xmlns="http://www.sdltridion.com/S3EclProvider/Configuration">com-dev-tridion-s3ecl-vikas</S3BucketName>
<S3AccessId xmlns="http://www.sdltridion.com/S3EclProvider/Configuration">XXXXXXXXXXXXXXXXXXX</S3AccessId>
<S3SecretKey xmlns="http://www.sdltridion.com/S3EclProvider/Configuration">xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</S3SecretKey>
 
<!-- ECLCoreServiceUrl; To be used if you incorporate Core Service in ECL provider -->
<ECLCoreServiceUrl xmlns="http://www.sdltridion.com/S3EclProvider/Configuration">net.tcp://localhost:2660/CoreService/201603/netTcp</ECLCoreServiceUrl> 
</MountPoint>
</MountPoints>

 

In Visual Studio: Dlls

You need – The External Content Library “Tridion.ExternalContentLibrary.V2”,  It is an API for developing Providers, Template Building Blocks and Event Handlers that communicate with an external system and expose the multimedia contained in the system in SDL Web. You can get it from

<Tridion_Home>\bin\client\Tridion.ExternalContentLibrary.V2.dll

You will also need an AWS S3 bucket API which will help you interacting with S3 bucket within the code. The API is called – AWSSDK.S3

 

In Visual Studio: project output

In SDL Tridion/Web, ECL has a default location as below, which is the place where all the ECL Providers need to stay.

..\..\ProgramData\SDL\SDL Tridion\External Content Library\AddInPipeline\AddIns\

 

Hence our motive here is to place the compiled code (i.e Provider dll) to above location. To do this, Go to Visual Studio > Solution Explorer > Click on Project > Properties > in Build Section > in Output Path set below path –

..\..\ProgramData\SDL\SDL Tridion\External Content Library\AddInPipeline\AddIns\S3ECLProvider\

This setting will dump your compiled code to the above mentioned location everytime you build your code.

Note: This will only happen if you are using Visual Studio in your CM Server, in case you are using your local system, you need to copy the same folder and place it to the above mentioned location.

 

In Visual Studio: Debug your Code

To debug your code on the CMS, restart your “Tridion Service Host”, Go to your Visual Studio, Build your code then click on Debug menu > Attach to a Process > scroll down and choose “Tridion Service Host”. Put a break point in the code. When you will go to your CME and open the ECL mount point your service process will hit the break point in your code and you can continue debugging.

Note: Whenever a Tridion Session starts, It locks the ECL provides dlls by the processes using it and hence you will be getting errors while building it again in Visual studio. To avoid this you can do one of these:

  1. Stop the Tridion service (s), mainly  “Tridion Service Host” and then build your code.
  2. Go to the ECL Default location as below, you will see a lock file created called – “AddIns.store“, Delete this and stop Tridion services and then start building it.
  3. ..\..\ProgramData\SDL\SDL Tridion\External Content Library\AddInPipeline\AddIns\

 

With the above setup in place (mainly the ECL.xml), you should be able to see a MountPoint in your CME under you parent publication.

At this point, you are all set for an ECL Provider implementation.

 

Implementation

I won’t say its easy but for sure it’s not hard either.

To start implementing your Provider you need to implement members of below  5 Interfaces of Tridion.ExternalContentLibrary.V2 and that’s it.

  • IContentLibrary 
    • Initialize your provider
    • Call to create IContentLibraryContext
  • IContentLibraryContext
    • Create a context of your Provider
    • Call to implement IContentLibraryItem & IContentLibraryListItem
  • IContentLibraryItem
    • Gets/Creates the ECl Item
  •  IContentLibraryListItem
    • Gets/creates the ECL list Items
  • IContentLibraryMultimediaItem
    • Get Items details to CME for Templating Basically

 

When you start your CME the first call goes to your Provider (create a class S3Provider.cs). This provider will use IContentLibrary interface and implement below methods:

S3Provider.cs : IContentLibrary

using System;
using System.AddIn;
using System.Collections.Generic;
using System.IO;
using System.Reflection;
using System.Xml.Linq;
using S3ECLProvider.api;
using Tridion.ExternalContentLibrary.V2;

namespace S3ECLProvider
{
 [AddIn("S3ECLProvider", Version = "1.0.0.0")]
 public class S3Provider : IContentLibrary
 {
  internal static IHostServices HostServices { get; private set; }

 internal static byte[] GetIconImage(string iconIdentifier, int iconSize)
 {
    int actualSize;
    // get icon directly from default theme folder
    return HostServices.GetIcon(IconBasePath, "_Default", iconIdentifier, iconSize, out actualSize);
 }

 public void Initialize(string mountPointId, string configurationXmlElement, IHostServices hostServices)
 {
       MountPointId = mountPointId;
       HostServices = hostServices;
       XElement config = XElement.Parse(configurationXmlElement);

       // Read configuration form ExternalContentLibrary.xml
       S3 = new S3(
       config.Element(S3Ns + "S3BucketName").Value,
       config.Element(S3Ns + "S3SecretKey").Value,
       config.Element(S3Ns + "S3AccessId").Value,
       config.Element(S3Ns + "FullBucketUrl").Value
       );
 }

 public IContentLibraryContext CreateContext(IEclSession eclSession)
 {
     return new S3Context(eclSession);
 }

 public IList<IDisplayType> DisplayTypes
 {
    get
    { return new List<IDisplayType>
       {
       HostServices.CreateDisplayType("fld", "Folder", EclItemTypes.Folder),
       HostServices.CreateDisplayType("fls", "File", EclItemTypes.File) 
       };
    }
 }

 public byte[] GetIconImage(string theme, string iconIdentifier, int iconSize)
 {
     return GetIconImage(iconIdentifier, iconSize);
 }

 public void Dispose()
 {
 }
 }
}

 

Now you need your context to be implemented. Create a class called “S3Context.cs” and use IContentLibraryContext.

S3Context.cs : IContentLibraryContext

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using S3ECLProvider.api;
using Tridion.ExternalContentLibrary.V2;

namespace S3ECLProvider
{
 class S3Context : IContentLibraryContext
 { 

 public bool CanGetUploadMultimediaItemsUrl(int publicationId)
 {
  // Set to true, if want to upload multimedia to external system from within CME .
 }

 public bool CanSearch(int publicationId)
 {
  // Set to true to enable search and implement "public IFolderContent Search(IEclUri contextUri, string searchTerm, int pageIndex, int numberOfItems)" method below.
 }

 public IList<IContentLibraryListItem> FindItem(IEclUri eclUri)
 {
     // return null to call GetItem(IEclUri)
 }

// GetFolderContent() is the methods which get the request when you click on your S3 MountPint (AWS S3 Bucket, as mentioned in ECL.xml) in CME.
 public IFolderContent GetFolderContent(IEclUri parentFolderUri, int pageIndex, EclItemTypes itemTypes)
 {
     bool canSearch = false; 
     if (parentFolderUri.ItemId == "root")
     {  canSearch = true; }
     List<IContentLibraryListItem> items = new List<IContentLibraryListItem>();
     InfoList = S3Provider.S3.GetItemFromS3(parentFolderUri, itemTypes);
     foreach (S3Info info in InfoList)
     {
         Info = info;
         items.Add(new ListItem(parentFolderUri, info));
     }
 return S3Provider.HostServices.CreateFolderContent(parentFolderUri, items, CanGetUploadMultimediaItemsUrl(parentFolderUri.PublicationId), canSearch);
 }


 public IContentLibraryItem GetItem(IEclUri eclUri)
 { 
     if (eclUri.ItemType == EclItemTypes.File && eclUri.SubType == "fls")
     {
        // retuns implemented memeber resulst from IContentLibraryMultimediaItem
     }

     if (eclUri.ItemType == EclItemTypes.Folder && eclUri.SubType == "fld")
     {
        // retuns implemented memeber resulst from IContentLibraryMultimediaItem
     }
    throw new NotSupportedException();
 }

 public IList<IContentLibraryItem> GetItems(IList<IEclUri> eclUris)
 {
     List<IContentLibraryItem> items = new List<IContentLibraryItem>();
       //Implement to return unique item if the ecl item is read/accessed across publication
     return items;
 }

 public byte[] GetThumbnailImage(IEclUri eclUri, int maxWidth, int maxHeight)
 { 
     // if IsThumbnailAvailable of IContentLibraryListItem is set to True 
     return S3Provider.HostServices.CreateThumbnailImage(maxWidth, maxHeight, ms, null);
     //or
     return null;
 }

 public string GetUploadMultimediaItemsUrl(IEclUri parentFolderUri)
 {
     if (parentFolderUri.ItemType == EclItemTypes.MountPoint)
     {
         return Info.MediaUrl;
     }

     if (parentFolderUri.ItemType == EclItemTypes.Folder && parentFolderUri.SubType == "fld")
     {
         return Info.MediaUrl; 
     }
    throw new NotSupportedException();
 }

 public string GetViewItemUrl(IEclUri eclUri)
 {
     return S3Provider.S3.GetMediaUrl(eclUri.ItemId); 
     throw new NotSupportedException();
 }

 public string IconIdentifier
 {
    get { return "S3"; } // This is the Identifier which will be matched icone name.
 }

 public IFolderContent Search(IEclUri contextUri, string searchTerm, int pageIndex, int numberOfItems)
 { 
     throw new NotSupportedException();
     //throw new InvalidOperationException("Enter search term..");
 }

 public string Dispatch(string command, string payloadVersion, string payload, out string responseVersion)
 {
      throw new NotSupportedException();
 }

 public void StubComponentCreated(IEclUri eclUri, string tcmUri)
 {
 }

 public void StubComponentDeleted(IEclUri eclUri, string tcmUri)
 {
 }

 public void Dispose()
 {
 }
 }
}

 

At this moment, you have your S3Provider creating your S3Context in your ECL Session, You now need to have your items retrieved using IContentLibraryItem (S3Item.cs – This will control how your item should behave in CME) and IContentLibraryListItem (S3ListItem.cs – This will control how your list item should behave in CME) so that your context can use them to return to IContentLibraryMultimediaItem (S3Media.cs).

Also, you will need a class which can get the get the data from your AWS S3 using AWSSDK.S3 API, let’s call it – S3Content.cs

Let’s first focus on S3Content.cs, which will actually use S3 bucket API to retrieve the data from S3 bucket. This class implements the method “GetItemFromS3()” used in IFolderContent GetFolderContent() of S3Context.

GetFolderContent(IEclUri parentFolderUri, int pageIndex, EclItemTypes itemTypes) actually decides the content retrieval (i.e. Whether to fetch Folder or Files in a given context) based on the itemTypes parameter, here we are passing this parameter to our “GetItemFromS3(EclUri, itemTypes)” 

S3Content.cs : uses AWS API to gets data from S3

 public List<S3Info> GetItemFromS3(IEclUri parentFolderUri, EclItemTypes itemTypes)
 {
    //S3Info: A model class which sets properties retrived from S3   
    List<S3Info> myList = new List<S3Info>();
    S3DirectoryInfo s3Root = null;
    GetObjectRequest getDirObjectRequest = null;
    if (parentFolderUri.ItemId == "root")
    {
       s3Root = new S3DirectoryInfo(s3Client, BucketName);
    }
    else
    { 
      s3Root = new S3DirectoryInfo(s3Client, BucketName, parentFolderUri.ItemId.Replace('/', '\\').TrimEnd('/'));
    }

// Below methods retrives files and folders within a given context based on the itemtype.

//Only returns Folders for Left Tree
 if (parentFolderUri.ItemType == EclItemTypes.MountPoint && itemTypes.HasFlag(EclItemTypes.Folder) && !itemTypes.HasFlag(EclItemTypes.File))
 {
   foreach (var subdirectories in s3Root.GetDirectories())
   {
     var item = subdirectories.FullName.Split(':')[1].Replace('\\', '/').TrimStart('/');
     var itemUrl = FullBucketUrl + item;
     myList.Add(new S3Info(subdirectories, itemUrl, "Folder"));
   }
 }

//returns both Files & Folders
 else if (itemTypes.HasFlag(EclItemTypes.File) && itemTypes.HasFlag(EclItemTypes.Folder))
 {
   foreach (var subdirectories in s3Root.GetDirectories())
   {
    var item = subdirectories.FullName.Split(':')[1].Replace('\\', '/').TrimStart('/');
    var itemUrl = FullBucketUrl + item;
    myList.Add(new S3Info(subdirectories, itemUrl, "Folder"));
   }
   foreach (var file in s3Root.GetFiles())
   {
    var item = file.FullName.Split(':')[1].Replace('\\', '/').TrimStart('/');
    var itemUrl = FullBucketUrl + item;
    myList.Add(new S3Info(file, itemUrl, "File"));
  }
 }


//Only returns Folders
 else if (itemTypes.HasFlag(EclItemTypes.Folder))
 {
    foreach (var subdirectories in s3Root.GetDirectories())
    {
       var item = subdirectories.FullName.Split(':')[1].Replace('\\', '/').TrimStart('/');
       var itemUrl = FullBucketUrl + item;
       myList.Add(new S3Info(subdirectories, itemUrl, "Folder"));
    }
 }

//Only returns Files
 else if (itemTypes.HasFlag(EclItemTypes.File))
 {
    foreach (var file in s3Root.GetFiles())
    {
       var item = file.FullName.Split(':')[1].Replace('\\', '/').TrimStart('/');
       var itemUrl = FullBucketUrl + item;
       myList.Add(new S3Info(file, itemUrl, "File"));
    }
 }
 
//In case some magic brings you here
 else
 {
 throw new NotSupportedException();
 } 
 return myList;
 }

With above implementation, your context has your data from S3. Now you need to create items/listItems for Tridion. This will be done by S3Item.cd and S3ListItem.cs

 

S3Item.cs : IContentLibraryItem

 public class S3Item : IContentLibraryItem
 {
 public S3MediaSet(IEclUri ecluri, S3Info info) : base(ecluri, info)
 {
      // if info needs to be fully loaded, do so here
 }

 public bool CanGetViewItemUrl
 {
     get { return true; }
 }

 public bool CanUpdateMetadataXml
 {
     get { return false; }
 }

 public bool CanUpdateTitle
 {
     get { return false; }
 }

 public DateTime? Created
 {
     get { return Info.Created; }
 }

 public string CreatedBy
 {
    get { return S3Provider.S3.AccessId; }
 }

 public string MetadataXml
 {
     get { return null; }
     set { throw new NotSupportedException(); }
 }

 public ISchemaDefinition MetadataXmlSchema
 {
    get { return null; }
 }

 public string ModifiedBy
 {
    get { return CreatedBy; }
 }

 public IEclUri ParentId
 {
   get
   {
     // return mountpoint uri (we only have folders in the top level)
     return S3Provider.HostServices.CreateEclUri(Id.PublicationId, Id.MountPointId);
   }
 }

 public IContentLibraryItem Save(bool readback)
 {
   // as saving isn't supported, the result of saving is always the item itself
   return readback ? this : null;
 }
 }
}

 

S3ListItem.cs : IContentLibraryListItem

public class S3ListItem : IContentLibraryListItem
 {
     internal readonly S3Info Info;
     private readonly IEclUri _id;

 public ListItem(IEclUri ecluri, S3Info info)
 {     
    Info = info;
    if (Info.ContentType == "Folder")
    {
      string itemId = Info.Name;
      String.Format(itemId);
      _id = S3Provider.HostServices.CreateEclUri(ecluri.PublicationId, S3Provider.MountPointId, itemId, DisplayTypeId, EclItemTypes.Folder);
    } 
    else
    {
      string itemId = Info.Name;
      _id = S3Provider.HostServices.CreateEclUri(ecluri.PublicationId, S3Provider.MountPointId, itemId, DisplayTypeId, EclItemTypes.File);
    }
 }

 // for folders only
 public bool CanGetUploadMultimediaItemsUrl
 {
    get { return true; }
 }
 
 public bool CanSearch
 {
    get { return true; }
 }

 public string DisplayTypeId
 {
    get
    {
      if(Info.ContentType == "Folder")
      { return "fld"; }
      else
      { return "fls"; }
    }
 }

 public string IconIdentifier
 {
     get { return null; }
 }

 public IEclUri Id
 {
    get { return _id; }
 }

 public bool IsThumbnailAvailable
 {
    get { return true; }
 }

 public DateTime? Modified
 {
    get { return Info.LastModified; }
 }

 public string ThumbnailETag
 {
    get { return Modified != null ? Modified.Value.ETag() : Info.Created.Value.ETag(); }
 }

 public bool CanUpdateTitle
 {
    //default false, if set to True you can set your Item display title in CME with "public string Title" member
    get { return true; } 
 }

 //below Property allowed me to set the name as I wanted in Tridion CME
 public string Title
 {
    get
    {     
     if (Info.ContentType == "Folder")
     {
        var nameArray = Info.Name.Split('/');
        int count = nameArray.Length;
        return nameArray[count - 2].TrimEnd('/');
     }
    else
    {
       var nameArray = Info.Name.Split('/');
       int count = nameArray.Length;
       return nameArray[count - 1];
    }
 }
 set { throw new NotSupportedException(); }
 }

 // allow override of dispatch
 public virtual string Dispatch(string command, string payloadVersion, string payload, out string responseVersion)
 {
     throw new NotSupportedException();
 }
 }

 

Now you need to implement IContentLibraryMultimediaItem, which will internally implement IContentLibraryItem & IContentLibraryListItem. But mainly it implements below methods. Let’s see –

S3Media.cs : IContentLibraryMultimediaItem

IContentResult GetContent(IList<ITemplateAttribute> attributes)
{
    //returns component Item if you want to publoish your image with your component.
}
string GetDirectLinkToPublished(IList<ITemplateAttribute> attributes)
{
   //returns an absolute Url of the media for template
}
string GetTemplateFragment(IList<ITemplateAttribute> attributes)
{
   //Template takes this as a replacement and use directly Component Presentatin so that publishing does not render it to resolve.
   //This only happens if you GetTemplateFragment() does not return null
}

 

Once you have implemented above, you can start using your Provider with S3. If everything goes right, you would see something like this:

 

S3 Bucket ECL Provider POC - Demo v1.2.0.2

Here 102 is images in my Collection folders, in your case you may have more/less. Please note: ECL would perform better if you have a limited number of items per folder.

Note: In few scenarios, you might want to change the code a bit in a way you want.

Hope this helps you. Enjoy implementing ECL Provider.

 

If you still need help, Please post your question on Tridion Stack Exchange (Trex)

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s