Effective BizTalk

Earlier I wrote about how to use one SQL Port for multiple operations within a BizTalk Application / Orchestration. Now I would like to address another issue I often see handled in a manner I think is not the most appropriate. The SQL Adapter is a great tool, but may be a little too Black Box for many developers to understand the proper uses of. Perhaps the most common is how to handle multiple calls to a SQL Service (be it stored procedure or updategram).

Often the scenario looks like this: A message arrives with a repeating element in it and each element must be inserted into a database. Like many things in BizTalk there are actually several ways to tackle this problem and at least one doesn't even involve Orchestration, but usually the solution I see looks something like this.

Usually the loop iterates over an xpath expression extracting each message and assigning it to another that will then be sent to the SQL Port in question. This does work and does sort of do what the author intended. There are, however, a few potentially serious issues with this. For one there is not transactional integrity between the sends to the SQL Adapter. The last one could fail and the others would already have been sent and committed to the database. You could add compensation logic, but that sort of just piles on more mess on an existing mess.

What many do not realize is that a request can be made much simpler. The schema for the SQL send will probably look something like this:

-

With a instance like:

<ns0:GetDataRequest xmlns:ns0="http://nova/biztalk/sql">

<ns0:ReadSecurityIntervalData LastTime="1143003000" SecurityId="1" TimeFrameId="2" />

</ns0:GetDataRequest>

By default the generated schema does not have the Min Occurs or Max Occurs properties set for an request message (it does however set them to zero and unbounded for response messages). If you simply set these as the response elements are set (zero and unbounded) you can send multiple requests to the SQL Adapter at a single time.

<ns0:GetDataRequest xmlns:ns0="http://nova/biztalk/sql">

<ns0:ReadSecurityIntervalData LastTime="1143003000" SecurityId="1" TimeFrameId="2" />

<ns0:ReadSecurityIntervalData LastTime="1143006000" SecurityId="1" TimeFrameId="2" />

<ns0:ReadSecurityIntervalData LastTime="1143009000" SecurityId="1" TimeFrameId="2" />

<ns0:ReadSecurityIntervalData LastTime="1143012000" SecurityId="1" TimeFrameId="2" />

</ns0:GetDataRequest>

This not only saves creating an indeterminate number of messages, it ensures that they are all sent to the SQL Server in one atomic transaction (regardless of the transaction type in the Orchestration, which can't use an Atomic Scope with Send-Receive ports in any case) and as a single batch which reduces the number of trips to the message box and increases the throughput of a solution. The SQL statements will all be performed in the same DTC transaction in an all or nothing fashion which may be more in line with the business requirement in the first place.

This can be extended even further and combined with my original posting about SQL Server Adapter Port (Ab)Use. Not only can you call multiple operations through a single port, you can combine them through a single request and all in one operation. Suppose you had a series of stored procedures that all needed to be called and succeed or all rollback in a single transaction. Most people don't know that this is even possible with the SQL Adapter. Not only is it possible it's also pretty easy. This time you're request element (the root node you used for all your SQL Wizard-ry) will be made to use types defined for the different operations. You'll want to use a map to create your new request message and fill in all the elements that make up the operations you wish to call. This will result in a message like the following.

<ns0:SqlRequest xmlns:ns0="http://nova/biztalk/sql">

<ns0:ReadSecurityIntervalData LastTime="1143003000" SecurityId="1" TimeFrameId="2" />

<ns0:GetLastTrade TimeFrameId="2" />

</ns0:SqlRequest>

As can be seen here the request message contains multiple SQL Operations in a single message. I have gotten this to work in calling multiple stored procedures of different types, but so far I'm only getting the result of the first query. I will post on that solution after I find it.

One of the features that make BizTalk a great tool to work with is Adapters. Although they can take a while to learn the Adapter in BizTalk Server is the gateway to other systems you integrates in a BizTalk solution and is thus vital to BizTalk solutions.

Like many features in BizTalk there is great beginning documentation for the SQL Adapter, but shortly after that it sort of drops off and it can be difficult to find more advanced information.

One of the common complaints I see on the USENET groups I take part it (Microsoft.public.biztalk.*) is having to create multiple SQL Ports to call different stored procedures or updategrams on the same database. The Add Generated Items wizard for the SQL Adapter certainly contributes to this and it also does a few things I don't particularly care for.

For one it creates a new orchestration containing the multipart message types and the port that you can use for calling the Adapter. This in itself is not that bad (see my post about multipart messages in Orchestrations to see why). But most of us either call more than one SQL Service or are adding it to an existing orchestration (or not using orchestration at all). I don't like this new orchestration just to hold two multipart message types and a port that take very little time to create manually and in the location and manner you prefer. Worse still the orchestration is in the same Visual Studio Project as the schema which violates one of my guidelines for solution structure in BizTalk (see post)

Secondly it gives some terrible names to the Port, Orchestration, and schema file/ type. You should always change those names because they are meaningless by default (doing so, however, will break the orchestration that the wizard created for you since it will no longer find the types it wants).

This is the downside to wizards. To be more useful in the majority of cases they are less so in the minority of cases.

After generating the artifacts for your SQL use you will be left with a schema that looks like the following.

As you can see the schema has two root elements in it, one for your request and the other for your response.

The real issue comes when it is time to bind your orchestration ports. If you used the wizard and call four different stored procedures on the same database by default you will have four different send ports because each requires a namespace and root element of the returned message. This is actually where we can make a few changes to avoid having to create a separate port for each SQL call (even across orchestrations within the application). By this I mean we can share a port that connects to one database to call whatever operations on it we like. And it's not nearly as much work as I thought it would be.

Steps

Step 1

First when creating your schemas using the wizard be sure to use the same namespace every time. This really makes sense since you're doing the same thing, calling a database, in the same application. This is important because BizTalk identifies message types as a combination of namespace and root node name (for more info see: How BizTalk Identifies Types).

Next also in the wizard name your root response element the same such as SqlResponse. The Adapter actually puts whatever SQL Server returns under this node so it will not interfere with your returned data (you can skip this step if you like, but it might save you some headache down the road).

Step 2

Now comes the tricky part, by naming all of your response elements the same you're about to break BizTalk, so you need to combine them all into one schema. Either create a new schema for them (using the same namespace again and same root node name) or pick one of your existing ones to build off of. I've never found a friendly way to do this moving in Visual Studio so I just fire up TextPad and do it manually. Put each of these child elements under the root node you've designated as your Response Document Root Element (from the wizard in the previous step). After you put them there specify that they are Nilable and set their Min Occurs to zero. Don't worry this is the worst part and it's still not that bad. Also note if you were already doing things with these messages before consolidation (as I was when I wrote this article) you may see your orchestrations no longer able to identify the message types you just moved. If this happens it generally is resolved by closing Visual Studio and reloading the project.

Now in your Orchestration (if you're using one) just add operations to the one SQL Port you have. This can be done in the Orchestration View under Types -> Port Types. Technically I think you can get away without doing this, but it's still a good idea. It conveys intent and it allows for better message routing.

That's it, now you have one SQL Port that can be used for an unlimited number of operations in your BizTalk Application. If you want to be even bolder you can make this port public and allow other applications to use it, but I generally shy away from this for dependency reasons.