Effective BizTalk

I've had a chance to get to work with the Sharepoint Adapter quite a bit recently and have come away from the experience relatively pleased. That said there were some lessons that could serve as a guide for others on this journey. First the BizTalk Adapter does work great, but it does not work like most of Sharepoint does. I spent a lot of time with Sharepoint and BizTalk separately so bringing them together presented some challenges for me. My ideas of how Sharepoint likes to work and is most often used were not the same as those of the BizTalk Adapter and this did end up having an influence on the shape of my solution structure.

Receiving with the Sharepoint Adapter

Versioning vs Archiving

Sharepoint has the concept of versioning deeply ingrained in the product. From the user perspective versioning is relatively seamless and the user never really sees it happening. The Sharepoint Adapter does not use versioning and will not work if a document library has versioning turned on. Also it's important to remember that the Sharepoint Adapter is only meant to work with documents not lists (despite the two pretty much being the same within Sharepoint).

The Sharepoint Adapter takes an all or nothing approach meaning that when it polls its target library (or view as we'll see later on) it removes all the documents it finds. More precisely it copies them to the archive location that you specify in the Adapter configuration. If you do not specify an archive location the documents are simply deleted. This at first may take a little time to get used to, but it makes sense. The documents must be removed because the adapter will pick them up multiple times if they are not and this is almost certainly not what you want.

The way I got around this (and read to) was to use a folder to copy the documents into that got them out of my pickup location, but kept them in Sharepoint for reference. The nice thing about the Adapter is that it will lock the document and try to make the archived copy first, before deleting the original file. If it fails to do so for some reason (such as a file name conflict) the document will stay in the pickup location (and the Adapter will write an error to the event log).

The configuration window below shows all the settings for the receive adapter, they are all pretty self explanatory and other than the archiving part do exactly what you expect them to do.

The only piece of warning I would give is that unlike many Adapters this one lets you set times in seconds. There is a trade off to be made between a longer polling interval and server traffic (i.e. latency vs. throughput). The Adapter is designed to pull multiple messages at ones from the view and this will make it scale extremely well, but lower that interval too much and your Sharepoint server will start experiencing undue load that could adversely affect its performance.

Views

As mentioned earlier the Adapter can use Views in Sharepoint to filter the pickup location. This can be very useful if you are building a solution where both users and Sharepoint will be performing updates on the documents. I added a Has Chanes (Yes/No) column to my list that allowed the solution to filter which documents required action from the Adapter. Anytime a change was made by a user the flag was set to "Yes" (true), the Adapter picked up the file, and an Orchestration (the child introduced below) immediately sent the document back updated and with the Has Changes column set to "No" (or false). This allowed me bidirectional updates on documents in a library. A child Orchestration (having been created earlier by a parent) listened for updates on an initialized correlation token (an ID in the document) and passed these documents to the parent Orchestration which was then free to send documents back to the library certain that it always had the latest version. By using a self-correlated port that the parent sent into the child it simply used a receive to get the latest document version without having to pull it from Sharepoint directly.

Sending with the Sharepoint Adapter

Much of sending with the Adapter is, as you would expect, similar to receiving. The most important settings to note here are the Namespace Aliases and Overwrite settings as these will impact both your xpath field mappings as well as your writing to document libraries. These same settings also exist in the receive configuration and for a good explanation visit the MSDN Documentation.

InfoPath

InfoPath is the final piece in the Sharepoint Adapter puzzle and can be used to great effect with Sharepoint and BizTalk as the primary User Interface. This is how I was able to control the setting of the Has Changes column that was vital to my bidirectional update solution. When using InfoPath with Sharepoint; templates can be setup to post to a document library and fields in that template can be mapped to columns in the document library. The Adapter works in the same way and can also be used to set any column in the list and even use content from within the message to set the fields without using InfoPath at all. If you're using InfoPath you don't have to do anything special to set these fields, but if you're not the option still exists for setting as many as sixteen fields in the document library.

Conclusion

Ultimately the Adapter makes working with Sharepoint easy, reliable, and scalable. These last two points, reliable and scalable, are something to not be overlooked because there is one thing I have seen a lot of experienced Sharepoint developers tend to do (and I used to as well): build their own Sharepoint Features for interacting with the outside world. The really important aspect of how this Adapter works with Sharepoint is that it is all completely asynchronous and isolated; a failure or outage at any one part will not result in lost documents. They will simply be queued for processing when the outage passes. On top of this it is configurable and centrally managed. It also avoids creating a tangled mess of point to point integrations with Sharepoint.

A while back I ran into Marty Wasznicky and Scott Zimmerman excellent MSDN Article 8 Tips And Tricks For Better BizTalk Programming which is really worth a bookmarking. I really never got into their #1 Always Use Multi-Part Message Types (though the others I had picked up most of the hard way). I normally work with more messaging centric solutions and as a result do less with Orchestration, but the last year and a half has had me doing a lot more Orchestration and now I really understand this tip a lot better.

Although most Messages used in BizTalk Orchestrations are not multipart, using them in an Orchestration provides the benefit of isolating schema changes from impacting Orchestrations. Changing a schema without using multipart messages will require disconnecting all the Port, Send, and Receive shapes within the Orchestration Designer. This is a tedious an error prone process and it is not always easy to track down all of these affected shapes and there is no automated way to do it ahead of time. As the authors correctly point out; binding Ports and Messages within an Orchestration directly to a schema can lead to problems down the road. Usually these problems appear the first time you try to change something, which is often before even getting to production.

Granted there is one extra step here, defining the new part of the multipart message, but this is a small amount of work to do for a large amount of benefit. If you've ever worked with Web Services in an Orchestration (as a consumer) you've seen the Web Message Types that are generated for you; this is basically the same thing as using the multipart message types.

Overview

BizTalk Server is a great tool that can aid in the development in integration and SOA solutions. It is a force multiplier that allows one developer to perform the work of many. It is reliable and scalable. It is also a little difficult to get your head around at first. One of the greatest difficulties developers have starting out with BizTalk is to understand how types work in BizTalk Server. This is a pretty straightforward concept in .NET so the answer can throw off some people.

Types of Types

BizTalk primarily has two major types of types: Message Types and .NET Types. Everything in BizTalk is one of these and some things are both. The .NET types are simple enough, but we'll discuss some of their implications shortly. The Message Types are less so. Being a Messaging Platform everything that flows into or out of BizTalk Server is a message. Binary, XML, Text they are all messages. BizTalk identifies XML Message Types based on the namespace and root node of the XML instance itself. Since XML legally only allows one root node in a document this is a pretty standard identification. An example would be "http://novaenterprisesystems.com/schemas/sql#SqlRequest" where "SqlRequest" is the root element of the message and the schema is "http://novaenterprisesystems.com/schemas/sql". This works fine for XML messages, but other messages don't really work the same.

Types in Contexts

All messages in BizTalk's Messaging Engine are represented as an instance of Microsoft.BizTalk.Message.Interop.IBaseMessage and in a Pipeline the same holds true (as Pipelines are messaging components). Once in an Orchestration a message is represented as an instance of Microsoft.XLANGs.BaseTypes.XLANGMessage and can be sent to or from a user component as such. Interestingly enough these types cannot be converted between them so you'll generally use IBaseMessage in Pipeline Components and XLANGMessage exclusively in Orchestrations.

To make things a little more complicated any message in an Orchestration it is implicitly understood to be an instance of System.Xml.XmlDocument. It is not so much that these classes are all related, but they are functionally and have both practical and historic reasons for this arrangement. One is that it allows you a simple way to deal with non XML messages in orchestrations, you can specify their type as XmlDocument and handle just about any type of message, even binary. That said if you have a binary message and try to use any of the members of XmlDocument on it you will receive an exception from the Orchestration Engine.

These different interfaces allow different types of interactions with messages in their own contexts' which can sometimes be challenging until you understand the lay of the land so to speak.

To sum it up in these contexts messages are these types:

Messaging Engine: IBaseMessage

Orchestration: XLANGMessage or XmlDocument

Later I will post on how to create messages in the different parts of BizTalk Server.

Earlier I wrote about how to use one SQL Port for multiple operations within a BizTalk Application / Orchestration. Now I would like to address another issue I often see handled in a manner I think is not the most appropriate. The SQL Adapter is a great tool, but may be a little too Black Box for many developers to understand the proper uses of. Perhaps the most common is how to handle multiple calls to a SQL Service (be it stored procedure or updategram).

Often the scenario looks like this: A message arrives with a repeating element in it and each element must be inserted into a database. Like many things in BizTalk there are actually several ways to tackle this problem and at least one doesn't even involve Orchestration, but usually the solution I see looks something like this.

Usually the loop iterates over an xpath expression extracting each message and assigning it to another that will then be sent to the SQL Port in question. This does work and does sort of do what the author intended. There are, however, a few potentially serious issues with this. For one there is not transactional integrity between the sends to the SQL Adapter. The last one could fail and the others would already have been sent and committed to the database. You could add compensation logic, but that sort of just piles on more mess on an existing mess.

What many do not realize is that a request can be made much simpler. The schema for the SQL send will probably look something like this:

-

With a instance like:

<ns0:GetDataRequest xmlns:ns0="http://nova/biztalk/sql">

<ns0:ReadSecurityIntervalData LastTime="1143003000" SecurityId="1" TimeFrameId="2" />

</ns0:GetDataRequest>

By default the generated schema does not have the Min Occurs or Max Occurs properties set for an request message (it does however set them to zero and unbounded for response messages). If you simply set these as the response elements are set (zero and unbounded) you can send multiple requests to the SQL Adapter at a single time.

<ns0:GetDataRequest xmlns:ns0="http://nova/biztalk/sql">

<ns0:ReadSecurityIntervalData LastTime="1143003000" SecurityId="1" TimeFrameId="2" />

<ns0:ReadSecurityIntervalData LastTime="1143006000" SecurityId="1" TimeFrameId="2" />

<ns0:ReadSecurityIntervalData LastTime="1143009000" SecurityId="1" TimeFrameId="2" />

<ns0:ReadSecurityIntervalData LastTime="1143012000" SecurityId="1" TimeFrameId="2" />

</ns0:GetDataRequest>

This not only saves creating an indeterminate number of messages, it ensures that they are all sent to the SQL Server in one atomic transaction (regardless of the transaction type in the Orchestration, which can't use an Atomic Scope with Send-Receive ports in any case) and as a single batch which reduces the number of trips to the message box and increases the throughput of a solution. The SQL statements will all be performed in the same DTC transaction in an all or nothing fashion which may be more in line with the business requirement in the first place.

This can be extended even further and combined with my original posting about SQL Server Adapter Port (Ab)Use. Not only can you call multiple operations through a single port, you can combine them through a single request and all in one operation. Suppose you had a series of stored procedures that all needed to be called and succeed or all rollback in a single transaction. Most people don't know that this is even possible with the SQL Adapter. Not only is it possible it's also pretty easy. This time you're request element (the root node you used for all your SQL Wizard-ry) will be made to use types defined for the different operations. You'll want to use a map to create your new request message and fill in all the elements that make up the operations you wish to call. This will result in a message like the following.

<ns0:SqlRequest xmlns:ns0="http://nova/biztalk/sql">

<ns0:ReadSecurityIntervalData LastTime="1143003000" SecurityId="1" TimeFrameId="2" />

<ns0:GetLastTrade TimeFrameId="2" />

</ns0:SqlRequest>

As can be seen here the request message contains multiple SQL Operations in a single message. I have gotten this to work in calling multiple stored procedures of different types, but so far I'm only getting the result of the first query. I will post on that solution after I find it.

One of the features that make BizTalk a great tool to work with is Adapters. Although they can take a while to learn the Adapter in BizTalk Server is the gateway to other systems you integrates in a BizTalk solution and is thus vital to BizTalk solutions.

Like many features in BizTalk there is great beginning documentation for the SQL Adapter, but shortly after that it sort of drops off and it can be difficult to find more advanced information.

One of the common complaints I see on the USENET groups I take part it (Microsoft.public.biztalk.*) is having to create multiple SQL Ports to call different stored procedures or updategrams on the same database. The Add Generated Items wizard for the SQL Adapter certainly contributes to this and it also does a few things I don't particularly care for.

For one it creates a new orchestration containing the multipart message types and the port that you can use for calling the Adapter. This in itself is not that bad (see my post about multipart messages in Orchestrations to see why). But most of us either call more than one SQL Service or are adding it to an existing orchestration (or not using orchestration at all). I don't like this new orchestration just to hold two multipart message types and a port that take very little time to create manually and in the location and manner you prefer. Worse still the orchestration is in the same Visual Studio Project as the schema which violates one of my guidelines for solution structure in BizTalk (see post)

Secondly it gives some terrible names to the Port, Orchestration, and schema file/ type. You should always change those names because they are meaningless by default (doing so, however, will break the orchestration that the wizard created for you since it will no longer find the types it wants).

This is the downside to wizards. To be more useful in the majority of cases they are less so in the minority of cases.

After generating the artifacts for your SQL use you will be left with a schema that looks like the following.

As you can see the schema has two root elements in it, one for your request and the other for your response.

The real issue comes when it is time to bind your orchestration ports. If you used the wizard and call four different stored procedures on the same database by default you will have four different send ports because each requires a namespace and root element of the returned message. This is actually where we can make a few changes to avoid having to create a separate port for each SQL call (even across orchestrations within the application). By this I mean we can share a port that connects to one database to call whatever operations on it we like. And it's not nearly as much work as I thought it would be.

Steps

Step 1

First when creating your schemas using the wizard be sure to use the same namespace every time. This really makes sense since you're doing the same thing, calling a database, in the same application. This is important because BizTalk identifies message types as a combination of namespace and root node name (for more info see: How BizTalk Identifies Types).

Next also in the wizard name your root response element the same such as SqlResponse. The Adapter actually puts whatever SQL Server returns under this node so it will not interfere with your returned data (you can skip this step if you like, but it might save you some headache down the road).

Step 2

Now comes the tricky part, by naming all of your response elements the same you're about to break BizTalk, so you need to combine them all into one schema. Either create a new schema for them (using the same namespace again and same root node name) or pick one of your existing ones to build off of. I've never found a friendly way to do this moving in Visual Studio so I just fire up TextPad and do it manually. Put each of these child elements under the root node you've designated as your Response Document Root Element (from the wizard in the previous step). After you put them there specify that they are Nilable and set their Min Occurs to zero. Don't worry this is the worst part and it's still not that bad. Also note if you were already doing things with these messages before consolidation (as I was when I wrote this article) you may see your orchestrations no longer able to identify the message types you just moved. If this happens it generally is resolved by closing Visual Studio and reloading the project.

Now in your Orchestration (if you're using one) just add operations to the one SQL Port you have. This can be done in the Orchestration View under Types -> Port Types. Technically I think you can get away without doing this, but it's still a good idea. It conveys intent and it allows for better message routing.

That's it, now you have one SQL Port that can be used for an unlimited number of operations in your BizTalk Application. If you want to be even bolder you can make this port public and allow other applications to use it, but I generally shy away from this for dependency reasons.