Effective BizTalk

I have been writing a lot for my upcoming BizTalk book and after locking myself away in my office I often turn to The Modern Japanese Garden to relax. Besides being a refreshing reminder why digital books will never have the place in my life that physical books do (the images are amazing), the book helps me relax my mind and focus my thoughts.

I've always appreciated Japanese design aesthetic, but it was not until learning more about Japanese gardening in particular I began to appreciate why. A quote from the book above reads: "As in sumi painting, we should remove, not add, and so stimulate the imagination". I find myself today reviewing many BizTalk designs both of my own and by others and one thing strikes me that made that quote stand out in my mind all day. Odd as it may seem the orchestration canvas can almost be likened to a garden. You run into orchestrations that are like Victorian gardens: highly organized, regimented, packed with components and shapes. You also run into some that are like Japanese gardens: profound in their simplicity. I certainly started in what I would now call the Victorian School of Orchestration Design, but have found my way to the school which now I subscribe; the Japanese school.

In fact the most successful BizTalk solutions I see are remarkable for their simplicity. The same holds for non-BizTalk solutions. I guess what I'm driving at is that no matter what you prefer in garden design, software should be made as simple as possible and large busy orchestrations are generally not that simple despite the visual nature of the design canvas.

For a long time I never worked very much with Parties and Role Links in BizTalk and I've been teaching it some lately (and had a question posted on a board) so here goes. I really just plan to talk about them and tell you where to get more info on using them. I guess it started when I was just trying to touch all the features I never had and found Role Links in the Orchestration designer. I checked MSDN which does give the answers, but not as directly as I was looking for (in retrospect the documentation is great, but I just needed a good overview to understand what I was looking at).

As I looked more I came to Richard Seroter's blog posting here which also points to a good post by Todd Uhl. This all started to make more and more sense over time. Then I had the requirement for a solution to send messages to a large variety of partners depending on some information that is looked up based on the input of an incoming message. There are a lot of ways to solve this in BizTalk and Dynamic Ports is certainly one, but I really don't like dynamic ports because they obscure important solution details from the administrator / operations team. To make matters worse the number of possible partners was likely to be in the hundreds for this solution.

After yet more research and playing around I stumbled onto the Party Resolution Sample (it's in the older versions too despite this link now pointing to 2009) and this is where I had the epiphany. Parties and Role Links could solve my problem with almost no work on my part. Here was my chance to look like a superhero without needing to do much more than figure out how to tie the pieces together (something that I find is common in BizTalk).

Basically it works like this: you use Role Links in your Orchestration like Richard suggests and if you have the defining piece of information in your inbound message (like in Richard's case) you're set. All I did was have a simple external service lookup that provided me that data from an existing repository. The best part of this solution is that the Maps go on the Ports (like they always should) and the Orchestration simply uses the internal (canonical) schemas. The other than assigning/resolving the party in the expression shape the Orchestration knows nothing about the ports and their details.

There are a few caveats such as: the Parties must exist in the BizTalk Group (surprise) and that they must have ports tied to them which will become a requirement of application (or more precisely Orchestration) binding before everything can be turned on. Also I'm pretty sure the ports need the same basic communication pattern (One Way or Two Way).

I've spent a lot of time lately designing stateless services and the schemas to enable them to be both stateless and extensible (which will be another post), but in doing so I began to realize that very often people embrace BizTalk Orchestration as the closest analogy to Business Process and tend to over leverage it in an attempt to unify and track their business processes. There certainly are times when an Orchestration is a business process, but often business processes will span multiple Orchestrations. Fortunately BAM and the TPE allow for this.

So what's wrong with just making really large, long running Orchestrations anyway? I strongly believe that the less in flight instances of Orchestrations you have in your BizTalk servers the better. It's not that the product can't do it (scalability isn't really a big issue here when done right) or that it's that difficult to make the product do it well; side by side deployment of Orchestrations is simple in BizTalk and schema and artifact versioning strategies (yet another post) are widely established / embraced. It is really back to the idea of the business process as a whole and its constituent parts as stages. Again I am reminded of good programming practices: in general simple is good. If a method has too many lines of code it is difficult to understand and maintain, breaking it into smaller chunks is probably a good idea once this happens. This also promotes reuse.

The same is true of Orchestration, only more so because the administrative aspects of Orchestration are more pronounced and visible. The Orchestration designer is a great tool and the HAT view of Orchestration is very useful for both tracking and debugging, but it is easy to get carried away. A good all around architectural principal is that you should not have to change your solution architecture simply to support features that aren't a core part of the solution itself. Build processes and tracking often fall into this category. If your build process causes you to change your solution perhaps it is not your solution that needs changing, but your build process. The same is true of tracking.

This ultimately calls for Business Activity Monitoring, which I having been working with more and more lately. I think BAM really is the analogy that best represents the business process that project sponsors understand and want to see. I personally am a big fan of the Orchestration Designer for Business Analysts for use in creating tracking definitions. This can serve not just as a lunch point for BAM, but for the solution that you are creating. This tool is allows business users to define the data and milestones that are important to them. The output, an XML file, can then be imported into Excel for completing the definition and defining views. Once I show businesses what BAM offers them (especially when they learn no custom database code or structures must be written to support it) they want to use it for everything. This isn't a bad idea, it scales really well and is probably better than most of us would be given the time to develop ourselves. Jesus Rodriguez wrote a great BAM article recently for MSDN here.

Importantly BAM was not as robust in previous versions of BizTalk so the 2006 releases really took this a lot further. The greatest advance was undoubtedly implementing multiple continuations in Tracking Profiles. I think this contributes largely to the use of Orchestration as the primary business process analogy. Allowing multiple continuations really allows the visibility business want in a much more elegant solution than large Orchestrations, which were really the only way to accomplish this in the past.

I recently posted about Handling Binary Messages in Orchestrations and took a little time to reflect upon a key issue I saw recently (after I wrote that post) and that is working with non-XML documents within BizTalk in general. BizTalk is certainly an XML-centric product. This is a natural fit for any sort of integration platform; which as the XML name states is extensible. But often we're forced to cross out of the clean world of XML (yes, I think XML is clean, I've worked with it for almost nine years). This is where developers can get into trouble. Anytime you find yourself dealing with a non-XML artifact in BizTalk you should really convert it into XML as quickly as possible before it has a chance to get into the center of your solution.

An Example

A good example I saw is dealing with email. I ran across a solution that was handling inbound and outbound email and it was clear there had been challenges for the developers during its creation. For one as I already pointed out, BizTalk is really XML based and handling non XML in Orchestrations can be very tricky. This can also lead to unintentional dependencies that can limit the flexibility of the solution and complicate the testing.

On that testing note, I like to be able to substitute all external systems in my solutions during development. This can be a bit trickier for Request-Response ports, but it is still possible even if you're just using simple stubs or stand-ins. Further it is always good to be able to change adapters after solution rollout for operational reasons. Basically this means any One Way port should be replaceable with another. A great example would be SMTP or File; you should be able to swap these at anytime or switch to something else like FTP. This just makes a solution more flexible without requiring code changes and this is one of BizTalk's greatest strengths.

An Example Solution

At first the value of this may not be clear. Let's say for instance that your solution needs to receive emails and do something with them. When the POP3 adapter receives an email it creates a binary multipart message with parts for the subject, body, and attachments (one each). You would probably also want to know other information such as From and To. You could write a solution that uses POP3 properties directly from within an Orchestration like: EmailMessage(POP3.From), but this will be both harder to test and propagate adapter details into your Orchestration. Make no mistake; that is an external dependency.

I think a better way to handle this type of requirement is to define an XML message to represent everything you need from the email. Such a schema may look like this:

From here you would probably want to use a Pipeline to create your message. Since the Pipeline executes in the context of the Adapter, by the time the Message Box is reached this is no longer a binary multipart message, but a single part XML message. In this schema I've defined Attachment as Base64Binary so it can safely store any type of attachment.

Now when I want to build functional tests for my Orchestration using BizUnit (which is really a great tool and even trivial solutions should use it) I can just use a File Receive location instead of the POP3 Adapter (which is both faster than POP3 and doesn't require another external dependency: a POP3 server). Obviously you would want your Deployment Verification Tests to use the POP3 Adapter for true end to end verification and you would also want to test your Pipeline either using pipeline.exe or some of the techniques Michael Stephenson recommends here.

Ultimately this is the same as mapping at the Port, which should always be done to avoid external dependencies. Often doing something right in BizTalk requires me to rethink my initial approach in the context of how the product works. I have always had great results if I take the time to do this.

An Easy Way Out

The last thing I'll add is that creating a Pipeline to do this isn't really an easy task if you're new to BizTalk; so if you need a quicker method that you may be more comfortable with: simply receive the email into an Orchestration that just does the XML message creation for you reading from the message and the adapter properties. Then you just submit your message directly to the Message Box and your processing Orchestration can receive them directly as well (it will not know the difference between these two approaches: pipeline and receiving orchestration).

Being an extensible .NET based product it is certainly possible to make BizTalk do pretty much anything you want (even deal with non-XML documents ad nauseam), but you may find yourself fighting the product rather than leveraging it.

Introduction

In my post How BizTalk Identifies Types I explained that any message can be represented within an Orchestration as an instance of XmlDocument even if it is not a valid XML document (such as a binary file). In this post I will explain how you can handle binary documents in Orchestrations and walk through how I handled receiving reports from SQL Server Reporting Services (SSRS) and sending them to Sharepoint via the Sharepoint Services Adapter.

Often there are times in an integration solution where non XML documents must be operated on by a solution. One such example would be receiving PDF or image files and sending them to another location. In simple cases you may be able to do this in a messaging only scenario, in more complicated cases you may need to bring in Orchestration.

Sending and Receiving Binary Messages in Orchestration

Although any message can indeed be represented as an XmlDocument in an Orchestration performing any operation on the XmlDocument (i.e. the message or variable that is the message) will result in a runtime exception. If you're solution requires some sort of action to be performed on these documents that cannot be done via direct subscription you are limited to setting promoted properties on the message (i.e. Adapter / BizTalk properties). This may be all you need to do and if that's the case you may not need an Orchestration in the first place, but maybe a Pipeline.

Creating Binary Messages in Orchestration

At times you may find yourself required to do something a bit more complex, like retrieve the binary result of a Web Service call (like I did with SSRS) and doing something with the message payload. In my case I wanted to put the report in a Sharepoint site dynamically, which certainly could have been done with the SSRS Sharepoint integration, but I don't like that solution as it is a point to point rather than hub and spoke remedy.

There are several ways to create messages in Orchestrations and some are not very well documented, but the one recommended way (documented in the Sdk\Samples\Pipelines\ComposedMessageProcessor example) to create new Binary Messages is to use a send Pipeline called by a Message Assignment shape. Creating the Pipeline can be made much easier by using the BizTalk Server Pipeline Component Wizard which is a free Visual Studio plug-in that makes this a very simple process (we'll look at this later).

The main caveat here is something I pointed out in How BizTalk Identifies Types and that is that Pipelines and Orchestrations use different interfaces to represent messages. As a result, you must the use the supplied wrapper classes for executing this functionality. For one you will need an Orchestration variable of type Microsoft.XLANGs.Pipeline.SendPipelineInputMessages (which requires a reference to Microsoft.XLANGs.Pipeline.dll, in your BizTalk install location).

Calling the Pipeline

The basic steps are quite simple: add a message to an instance of this class via the Add method which takes one parameter: the message to be added; then call the Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteSendPipeline method from within a Message Assignment shape or an Expression shape before the Message Assignment.

Every example I've ever seen including Composed Message Processor always first sets the output message (the XLANGMessage that is the last parameter) to null before calling this method so it's probably a good idea to do so.

Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteSendPipeline(

    Type, SendPipelineInputMessages, XLANGMessage);

Be sure the Type is the that of your .btp pipeline that uses your component, not the Pipeline component created by the Wizard that we discuss below.

After this your binary message is in the message you supplied for the XLANGMessage parameter which I generally make as type XmlDocument. From here you are free to do what you like with the message, but remember it is not really an XmlDocument so don't call any of its methods.

More about that Pipeline

After you've run the Pipeline Component Wizard you have a class that implements IComponent, IBaseComponent, IPersistPropertyBag, and IComponentUI. The single method Execute of IComponent is the one we're interested in, this is what will be called when ExecuteSendPipeline is invoked.

If your situation is like mine this is all slightly complicated by the fact that the binary data you want is in a Base64String typed element within a message you have received. This is where the Pipeline Component Wizard can really help by allowing you to specify properties that can be set for the Pipeline. I declared an xpath used to read out the payload element from the body, this way I can reuse my Pipeline or change message formats. You can also create a class that represents your original message (containing the binary element) and use the XmlSerializer to create an instance of this message and access the byte[] property containing your raw byes (the XmlSerializer handles the translation to and from Base64String for you). End result, don't forget to convert from Base64String to byte[] or your file will be corrupted.

Below is the code from my Execute method. The properties are created for you in the Wizard and you don't need to worry about the other parts of the class.

//This is required: create the message to return

XmlTextReader reader = new XmlTextReader(inmsg.BodyPart.Data);

IBaseMessageFactory factory = pc.GetMessageFactory();

IBaseMessage message = factory.CreateMessage();

IBaseMessagePart body = factory.CreateMessagePart();

//This uses more properties and reflection to process the message

System.Runtime.Remoting.ObjectHandle objectHandle = Activator.CreateInstance(_assemblyName, _typeName);

object underlyingObject = objectHandle.Unwrap();

XmlSerializer serializer = new XmlSerializer(underlyingObject.GetType());

underlyingObject = serializer.Deserialize(reader);

object result = underlyingObject.GetType().InvokeMember(PropertyName, System.Reflection.BindingFlags.Public |

System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.GetProperty, null, underlyingObject, null);

byte[] resultBytes = (byte[])result;

Importantly you use either the segment above or the one below, but not both.

//This avoids the relfection, but makes the pipeline only useful to one message type

XmlSerializer serializer = new XmlSerializer(typeof(RenderResponse));

RenderResponse response = (serializer.Deserialize(reader)) as RenderResponse;

byte[] resultBytes = response.Result;

//This also is common to both approaches

MemoryStream memStream = new MemoryStream();

memStream.Write(resultBytes, 0, resultBytes.Length);

memStream.Position = 0;

body.Data = memStream;

pc.ResourceTracker.AddResource(memStream);

StreamReader sr = new StreamReader(body.Data);

message.AddPart(inmsg.BodyPartName, body, true);

return message;

At some point you'll probably want to populate some properties for the Adapter you're using either in the Message Assignment shape in your Orchestration (which I did) or in the Pipeline itself.

Once you have your pipeline component you add it to your \Microsoft BizTalk Server 2006\Pipeline Components directory and you are free to use it when creating .btp files. At this point you simply add the Pipeline to your BizTalk Toolbox (by picking Choose Items) and then drag it into the encode stage.

If you've used properties to make your Pipeline configurable you would set them here as well. This is the Type that you will specify in the call to ExecuteSendPipeline in the Message Assignment shape.

Improvements

I know there must be a way to stream this and it would be a good idea. Loading this message into an XmlDocument or using the XmlSerializer will load the entire message into memory; this can cause problems with very large messages. The entire BizTalk engine is designed to use streaming and it would be nice to incorporate it here as well. If you can think of a way to do this, please let me know.

A while back I ran into Marty Wasznicky and Scott Zimmerman excellent MSDN Article 8 Tips And Tricks For Better BizTalk Programming which is really worth a bookmarking. I really never got into their #1 Always Use Multi-Part Message Types (though the others I had picked up most of the hard way). I normally work with more messaging centric solutions and as a result do less with Orchestration, but the last year and a half has had me doing a lot more Orchestration and now I really understand this tip a lot better.

Although most Messages used in BizTalk Orchestrations are not multipart, using them in an Orchestration provides the benefit of isolating schema changes from impacting Orchestrations. Changing a schema without using multipart messages will require disconnecting all the Port, Send, and Receive shapes within the Orchestration Designer. This is a tedious an error prone process and it is not always easy to track down all of these affected shapes and there is no automated way to do it ahead of time. As the authors correctly point out; binding Ports and Messages within an Orchestration directly to a schema can lead to problems down the road. Usually these problems appear the first time you try to change something, which is often before even getting to production.

Granted there is one extra step here, defining the new part of the multipart message, but this is a small amount of work to do for a large amount of benefit. If you've ever worked with Web Services in an Orchestration (as a consumer) you've seen the Web Message Types that are generated for you; this is basically the same thing as using the multipart message types.