Effective BizTalk

This post presents some useful suggestions for the Visual Studio Solution structure to use in BizTalk development. It is based on both my experiences as well as that of others within the BizTalk community.

Suggestion

Generally I like to break my solutions up into the seven projects show below. I find that this meets the needs of most solutions, but it is not a hard fast rule. Some solutions will have more, others less, but each project is explained below.

External Schemas (.xsd files)

This project contains all of the schemas that are sent or received by the BizTalk solution. Port level mapping ensures no dependencies are leaked into or out of the BizTalk implementation. This would include schemas generated by adapter wizards, SOAP references, etc. It is vital that these be treated as what they really are: external artifacts (i.e. external dependencies).

Schemas (.xsd files)

The Schemas project contains all the schemas used internally by a BizTalk solution. These are schemas that are never exposed to any other systems and would be used to define entities within the actual solution. Every external schema should have a corresponding internal schema or translate to part of a composite schema.

Maps (.btm files)

The Maps project contains all maps within the solution. It references the Schemas and External Schemas projects and nothing else (with the exception of custom functoid assemblies). This is the guardian that prevents external dependencies from permeating a BizTalk solution.

Pipelines (.btp files)

All pipeline components (assemblers and disassemblers) are grouped in this project to make testing and maintenance easier. This project should reference the External Schemas project and /or the Schemas project if needed. Alternatively if you want to provide slightly more robust isolation you could create internal and external pipeline projects, but I really only recommend this if you need to call a Pipeline from within an Orchestration to avoid leaking a dependency into your Orchestrations layer.

Orchestrations (.odx files)

This project contains all Orchestrations used in the solution. It references the Schemas project and possibly the Pipelines project (if you're using pipelines from within an Orchestration). Workflows can also exist in this assembly in BizTalk 2009.

Library (C#, resources, etc)

Any artifacts that are used by your solution should also be broken into their own projects. It is important to not let these bleed dependencies into your solution. If you have some custom components to do processing in an orchestration and some for custom functiods or pipeline components these should be broken into their own distinct projects.

Testing (.xml, .dtd, .cs files)

This project hosts all Unit and Functional tests and their supporting data. The only direct references should be to the utilities assembly and to the testing frameworks (e.g. NUnit and BizUnit). This also ensures a clean separation of tests and artifacts from the other parts of a solution.

Non Project Artifacts

There are also several Solution level folders I like to use to organize my solutions. These generally are:

• 3rdParty Assemblies – for storing any external assemblies and components the solution may need to utilize.
• Bindings – to store deployment bindings used in each of the environments you will have in your solution. I generally have Local Development, Integration, UAT, and Production.
• Build – used to hold build scripts.

Relative Path Locations

Always use relative path locations for all artifacts including:

• Keys used to sign assemblies
• References to other assemblies (i.e. project references not absolute references)
• Paths of test files

This will make it easier to build the solution on multiple machines and also make it easier to use a CI server to perform automated building of the solution.

Remember not all solutions will have these seven; some will have less others more, but keep in mind the separation of concerns brought up in the discussion about layers in a BizTalk solution. Later on I'll cover some specific examples of when to combine which parts of this guidance.

Reasoning

It is natural and healthy to ask why go through all this trouble (ignore that creating projects in a solution is very simple; even more so with a template). Earlier I mentioned the need to control dependencies and this is a certain way to do that, but there are other reasons as well. You may find yourself needing to update in-flight Orchestrations or Maps in a solution with such Orchestrations. Perhaps it's a simple fix and you haven't quite worked out you versioning strategy. With separate assemblies you are free to deploy just the changes necessary rather than all assemblies and artifacts at once.

Perhaps more importantly you are also free to create packages that are specific to the type of server you are deploying them to. If you are working in a large BizTalk environment you may have a dozen or more servers partitioned by their functional role. You may want to create deployment packages for each role. This goes back to the previous discussion about Layers vs. Tiers. Building solutions in a separable manner is a good practice that allows administrators and operators more control over the solution once it is out of development. This is critical to the success of any enterprise software system and even more so to BizTalk solutions.

This is going to be one of a series of posts on the topic I plan to release over the next several weeks. Part of this is driven by what I've seen as a weakness in the community: a lack of direct application of established design principles. Most .NET projects these days follow a now well established set of guidelines and principles that help ensure elegant easily maintainable solutions. These guidelines were established in the context of .NET development; borrowing from and building upon the experiences of older platforms such as Smalltalk, C++, and Java. In an effort to lead the community back to these guidelines I'd like to recover some of these concepts in the context of BizTalk solutions.

Separation of Concerns

BizTalk even more than most platforms is fundamentally involved with separation of concerns (being an integration solution), but often this idea is lost in the implementation of a BizTalk Solution. I think that developers can get so focused on being the separation for external applications that they neglect to keep this principle in mind in their own solutions and often inadvertently let dependencies and tight coupling permeate their solutions.

Layers of a BizTalk Solution

The proximity diagram below depicts how the layers in a BizTalk solution should interact. Only layers that actually touch each other should be interacting in any way. The intention from this should be quite clear; it is to isolate the layers in the solution and minimize their interaction points so as to create a more robust solution that does not allow changes to propagate throughout the system.

This layout will give us the ability to change the solution at its edges or in its core drastically without impacting other parts of the solution. A new external partner schema should have no impact on our Orchestration layer nor on our Internal Schemas.

This can also be expressed in a more detailed fashion in a UML diagram. The following diagram expresses the basic layers (or dependencies) in a BizTalk Solution that follows these guidelines.

Again the goal is flexibility by tightly controlling contact points and Visual Studio can actually help to enforce this architecture in your solution.

So why do this? Some would argue that you could just do a logical separation using solution folders and not need separate Visual Studio projects for each of the packages shown above. While there certainly is some merit to this argument, it overlooks some very important capabilities inherent in this architecture.

For one there is no chance of a dependency creeping throughout your solution, the Visual Studio references between projects will ensure that. Further you can also choose now to deploy you solution in a variety of arrangements of Tiers derived from the Layers presented above.

I recently posted about Handling Binary Messages in Orchestrations and took a little time to reflect upon a key issue I saw recently (after I wrote that post) and that is working with non-XML documents within BizTalk in general. BizTalk is certainly an XML-centric product. This is a natural fit for any sort of integration platform; which as the XML name states is extensible. But often we're forced to cross out of the clean world of XML (yes, I think XML is clean, I've worked with it for almost nine years). This is where developers can get into trouble. Anytime you find yourself dealing with a non-XML artifact in BizTalk you should really convert it into XML as quickly as possible before it has a chance to get into the center of your solution.

An Example

A good example I saw is dealing with email. I ran across a solution that was handling inbound and outbound email and it was clear there had been challenges for the developers during its creation. For one as I already pointed out, BizTalk is really XML based and handling non XML in Orchestrations can be very tricky. This can also lead to unintentional dependencies that can limit the flexibility of the solution and complicate the testing.

On that testing note, I like to be able to substitute all external systems in my solutions during development. This can be a bit trickier for Request-Response ports, but it is still possible even if you're just using simple stubs or stand-ins. Further it is always good to be able to change adapters after solution rollout for operational reasons. Basically this means any One Way port should be replaceable with another. A great example would be SMTP or File; you should be able to swap these at anytime or switch to something else like FTP. This just makes a solution more flexible without requiring code changes and this is one of BizTalk's greatest strengths.

An Example Solution

At first the value of this may not be clear. Let's say for instance that your solution needs to receive emails and do something with them. When the POP3 adapter receives an email it creates a binary multipart message with parts for the subject, body, and attachments (one each). You would probably also want to know other information such as From and To. You could write a solution that uses POP3 properties directly from within an Orchestration like: EmailMessage(POP3.From), but this will be both harder to test and propagate adapter details into your Orchestration. Make no mistake; that is an external dependency.

I think a better way to handle this type of requirement is to define an XML message to represent everything you need from the email. Such a schema may look like this:

From here you would probably want to use a Pipeline to create your message. Since the Pipeline executes in the context of the Adapter, by the time the Message Box is reached this is no longer a binary multipart message, but a single part XML message. In this schema I've defined Attachment as Base64Binary so it can safely store any type of attachment.

Now when I want to build functional tests for my Orchestration using BizUnit (which is really a great tool and even trivial solutions should use it) I can just use a File Receive location instead of the POP3 Adapter (which is both faster than POP3 and doesn't require another external dependency: a POP3 server). Obviously you would want your Deployment Verification Tests to use the POP3 Adapter for true end to end verification and you would also want to test your Pipeline either using pipeline.exe or some of the techniques Michael Stephenson recommends here.

Ultimately this is the same as mapping at the Port, which should always be done to avoid external dependencies. Often doing something right in BizTalk requires me to rethink my initial approach in the context of how the product works. I have always had great results if I take the time to do this.

An Easy Way Out

The last thing I'll add is that creating a Pipeline to do this isn't really an easy task if you're new to BizTalk; so if you need a quicker method that you may be more comfortable with: simply receive the email into an Orchestration that just does the XML message creation for you reading from the message and the adapter properties. Then you just submit your message directly to the Message Box and your processing Orchestration can receive them directly as well (it will not know the difference between these two approaches: pipeline and receiving orchestration).

Being an extensible .NET based product it is certainly possible to make BizTalk do pretty much anything you want (even deal with non-XML documents ad nauseam), but you may find yourself fighting the product rather than leveraging it.

Introduction

In my post How BizTalk Identifies Types I explained that any message can be represented within an Orchestration as an instance of XmlDocument even if it is not a valid XML document (such as a binary file). In this post I will explain how you can handle binary documents in Orchestrations and walk through how I handled receiving reports from SQL Server Reporting Services (SSRS) and sending them to Sharepoint via the Sharepoint Services Adapter.

Often there are times in an integration solution where non XML documents must be operated on by a solution. One such example would be receiving PDF or image files and sending them to another location. In simple cases you may be able to do this in a messaging only scenario, in more complicated cases you may need to bring in Orchestration.

Sending and Receiving Binary Messages in Orchestration

Although any message can indeed be represented as an XmlDocument in an Orchestration performing any operation on the XmlDocument (i.e. the message or variable that is the message) will result in a runtime exception. If you're solution requires some sort of action to be performed on these documents that cannot be done via direct subscription you are limited to setting promoted properties on the message (i.e. Adapter / BizTalk properties). This may be all you need to do and if that's the case you may not need an Orchestration in the first place, but maybe a Pipeline.

Creating Binary Messages in Orchestration

At times you may find yourself required to do something a bit more complex, like retrieve the binary result of a Web Service call (like I did with SSRS) and doing something with the message payload. In my case I wanted to put the report in a Sharepoint site dynamically, which certainly could have been done with the SSRS Sharepoint integration, but I don't like that solution as it is a point to point rather than hub and spoke remedy.

There are several ways to create messages in Orchestrations and some are not very well documented, but the one recommended way (documented in the Sdk\Samples\Pipelines\ComposedMessageProcessor example) to create new Binary Messages is to use a send Pipeline called by a Message Assignment shape. Creating the Pipeline can be made much easier by using the BizTalk Server Pipeline Component Wizard which is a free Visual Studio plug-in that makes this a very simple process (we'll look at this later).

The main caveat here is something I pointed out in How BizTalk Identifies Types and that is that Pipelines and Orchestrations use different interfaces to represent messages. As a result, you must the use the supplied wrapper classes for executing this functionality. For one you will need an Orchestration variable of type Microsoft.XLANGs.Pipeline.SendPipelineInputMessages (which requires a reference to Microsoft.XLANGs.Pipeline.dll, in your BizTalk install location).

Calling the Pipeline

The basic steps are quite simple: add a message to an instance of this class via the Add method which takes one parameter: the message to be added; then call the Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteSendPipeline method from within a Message Assignment shape or an Expression shape before the Message Assignment.

Every example I've ever seen including Composed Message Processor always first sets the output message (the XLANGMessage that is the last parameter) to null before calling this method so it's probably a good idea to do so.

Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteSendPipeline(

    Type, SendPipelineInputMessages, XLANGMessage);

Be sure the Type is the that of your .btp pipeline that uses your component, not the Pipeline component created by the Wizard that we discuss below.

After this your binary message is in the message you supplied for the XLANGMessage parameter which I generally make as type XmlDocument. From here you are free to do what you like with the message, but remember it is not really an XmlDocument so don't call any of its methods.

More about that Pipeline

After you've run the Pipeline Component Wizard you have a class that implements IComponent, IBaseComponent, IPersistPropertyBag, and IComponentUI. The single method Execute of IComponent is the one we're interested in, this is what will be called when ExecuteSendPipeline is invoked.

If your situation is like mine this is all slightly complicated by the fact that the binary data you want is in a Base64String typed element within a message you have received. This is where the Pipeline Component Wizard can really help by allowing you to specify properties that can be set for the Pipeline. I declared an xpath used to read out the payload element from the body, this way I can reuse my Pipeline or change message formats. You can also create a class that represents your original message (containing the binary element) and use the XmlSerializer to create an instance of this message and access the byte[] property containing your raw byes (the XmlSerializer handles the translation to and from Base64String for you). End result, don't forget to convert from Base64String to byte[] or your file will be corrupted.

Below is the code from my Execute method. The properties are created for you in the Wizard and you don't need to worry about the other parts of the class.

//This is required: create the message to return

XmlTextReader reader = new XmlTextReader(inmsg.BodyPart.Data);

IBaseMessageFactory factory = pc.GetMessageFactory();

IBaseMessage message = factory.CreateMessage();

IBaseMessagePart body = factory.CreateMessagePart();

//This uses more properties and reflection to process the message

System.Runtime.Remoting.ObjectHandle objectHandle = Activator.CreateInstance(_assemblyName, _typeName);

object underlyingObject = objectHandle.Unwrap();

XmlSerializer serializer = new XmlSerializer(underlyingObject.GetType());

underlyingObject = serializer.Deserialize(reader);

object result = underlyingObject.GetType().InvokeMember(PropertyName, System.Reflection.BindingFlags.Public |

System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.GetProperty, null, underlyingObject, null);

byte[] resultBytes = (byte[])result;

Importantly you use either the segment above or the one below, but not both.

//This avoids the relfection, but makes the pipeline only useful to one message type

XmlSerializer serializer = new XmlSerializer(typeof(RenderResponse));

RenderResponse response = (serializer.Deserialize(reader)) as RenderResponse;

byte[] resultBytes = response.Result;

//This also is common to both approaches

MemoryStream memStream = new MemoryStream();

memStream.Write(resultBytes, 0, resultBytes.Length);

memStream.Position = 0;

body.Data = memStream;

pc.ResourceTracker.AddResource(memStream);

StreamReader sr = new StreamReader(body.Data);

message.AddPart(inmsg.BodyPartName, body, true);

return message;

At some point you'll probably want to populate some properties for the Adapter you're using either in the Message Assignment shape in your Orchestration (which I did) or in the Pipeline itself.

Once you have your pipeline component you add it to your \Microsoft BizTalk Server 2006\Pipeline Components directory and you are free to use it when creating .btp files. At this point you simply add the Pipeline to your BizTalk Toolbox (by picking Choose Items) and then drag it into the encode stage.

If you've used properties to make your Pipeline configurable you would set them here as well. This is the Type that you will specify in the call to ExecuteSendPipeline in the Message Assignment shape.

Improvements

I know there must be a way to stream this and it would be a good idea. Loading this message into an XmlDocument or using the XmlSerializer will load the entire message into memory; this can cause problems with very large messages. The entire BizTalk engine is designed to use streaming and it would be nice to incorporate it here as well. If you can think of a way to do this, please let me know.