Effective BizTalk

Although XML has been around for quite some time many developers still don't know too much about it. Not needing to know too much about it is one of XML's strengths, but cursory knowledge is rapidly becoming mandatory for all developers. This post will discuss some of the details of namespaces in XML.

In XML a namespace is just that, a space (or area) identified by a name. When we look at a namespace such as http://novaenterprisesystems.com/schemas/purchaseorder we are not actually identifying a location on the web. Granted it does look like a URL and your browser will probably make you want to click it (don't bother, at least for now it doesn't actually go anywhere). In an XML document this simply means in a namespace that certain nodes of the document exist in this namespace. That is they are identified by their Element / Attribute name and this namespace. Here is an example:

 

In this case this means that the PurchaseOrder element is identified using the specific namespace http://novaenterprisesystems.com/schemas/purchaseorder. I could have just as easily (and legally) made the name space "bob". Although bob would be a legal namespace, it doesn't do much to describe what this schema represents. All this really means is that the PurchaseOrder element has an identity in the definition space of this schema. I could import another schema that defines a different PurchaseOrder element in another schema with no ill-effect (it would clearly need another namespace in order to be unique).

Namespaces ultimately should convey information about the service or document in question. The generally accepted pattern for naming schemas is as follows:

http://[company]/[role]/[businessarea]/[date or version]

Where Company is obviously the entity exposing this schema, Role is either schema (for message types) or interface (for operations such as web services), business area would be the vertical or horizontal segment of the service or document Company is providing (such as purchasing or billing). Date / version should be self-explanatory, but is most often the year and month the schema is effective (more on that in another post). A better example would be: http://novaenterprisesystems.com/schemas/purchasing/2009-11.

You may be asking yourself 'so why all the fuss'? The two main issues are legality in XML (more of a technical concern) and much more importantly readability by consumers. A namespace should clearly express the intent of a schema to which it is attached. If this is for a business document or file that should be clear (as well as what business document this represents). The definition difference between schema (message) and interface (operation) should clear confusion about what the document really means. You wouldn't want the consumers of your services to tightly couple the purchase order document to the operation of submitting a purchase order.

A last note I'd like to stress is that although the domain name http://novaenterprisesystems.com appears in this schema and message, the domain will receive no traffic as a result. Again that name is just an identifier, very similar to a namespace in .NET such as System.Data. Only when an import or include location points to a URL will network traffic occur.

I recently started working with the DB2 Adapter for BizTalk Server and must admit I've been pretty happy with the results. DB2 not one of my specialties, but it is still out there and not going away anytime soon. I always enjoy getting the chance to work with new adapters when I can. The Adapter is reliable and fast and makes working with yet another platform a pretty simple experience for the BizTalk developer.

I did, however, have one unexpected experience with it having to do with Element Form Defaults (which I guess I'll write about more later). After generating schemas for receive locations I eventually figured out that the adapter was returning data with an Element Form Default of qualified even though the schema was set to unqualified. As I tested my maps it was clear from the validation failure messages that the namespace was an issue. This was pretty easy to fix (change the Element Form Default to Qualified). Easy enough.

Later I found myself doing sends to the DB2 Adapter and found some more interesting details. Request messages sent to the DB2 Adapter must Unqualified (that is their Element Form Default cannot be Qualified). Easy enough again. Unfortunately the response to these messages still follow the requirement of being Element Form Default Qualified. This presented a bit of a Chicken and Egg scenario. The solution is actually a good practice in my book: don't have different messages in the same schema. I broke the request and response into their own schemas and set the Element Form Default separately.

After years of working with the product I finally came across my first real bug in BizTalk, or more precisely in BAM. To start off I'd say the vast majority of times I think I've found a bug in BizTalk I am just not using the product correctly. That said, even monkeys fall from trees. I'll preface this by stating that I use BAM a lot, probably more than any other BizTalk developer I know. I love BAM and the visibility it provides into business processes. I particularly like the Tracking Profile Editor as I prefer not to change my solution, even in simple ways, to accommodate tracking. The next thing I'll say is that Microsoft Support was amazing at fixing these issues for me and their team was honestly interested in the bugs and their resolution. I cannot thank them enough for their help.

There were in fact two separate issues that had come up. The first had to do with de-batching and tracking the individual messages. You can read about it here KB 970856. The symptom was that one message in the batch was not tracked. I had my first test hotfix for this so quickly I was amazed. This first scenario was the beginning of a multi-stage business process that could take a very long time to complete. It was quite easy to see and replicate this issue (in fact, my BizUnit tests did so). I did learn than my BizTalk Visual Studio Solution Structure works quite well as MS Support was able to unzip, build, and deploy my application very quickly. I took this as a good test and compliment.

The second issue proved more difficult for me because it involved further stages in the business process and it was hard to tell if the first issue was interfering here or not. It is documented here: KB 967765. At the end of the day it was not, but I did learn a lot about how BAM actually works and how it is meant to be used. I will post on that soon enough. This second issue had to do with multiple continuations and de-batching (I'm just not a big fan of the batch if you can't tell). I would get unpredictable results in my tracking for these and could not for the life of me figure out why. Some records would not be tracked, others would be partially tracked. I was pulling my hair out. Microsoft kindly pointed out to me that the issue could be avoided by changing my tracking solution in two ways: I had to have all my continuations at the beginning of the profile; I also had to track on the external message format (post map) for send ports. Although this did work, I was glad Microsoft provided a fix because the only way to move the continuations was in a text editor and I like to track on internal schemas so my tracking profiles don't change when my trading partners do. Again, more on this in my next BAM post.

Again I really want to thank the fantastic team at Microsoft Support. They were friendly, professional, and fast! I have never had a better support experience with Microsoft or any other vendor before.

I've been on a pretty serious WCF kick for the past few months and more and more I love it. I like even more how well BizTalk works with WCF. BizTalk is the application server you normally need to provide yourself to make WCF very useful. In Handling Binary Messages in Orchestrations I had mentioned that I used SQL Reporting Services via Web Services to run reports and do something with the output. As I had been experimenting with the solution I tried using both the SOAP Adapter and the WCF Adapter and found the WCF Adapter to be much more to my liking.

Using the WCF Http Basic binding you are able to call existing SOAP services but I did run into a slight problem with the SSRS services. These services require impersonation and by default WCF handles impersonation differently than ASMX in IIS did. At first I thought I needed to change the web.config on the reporting server, but eventually I stumbled onto Delegation and Impersonation with WCF on MSDN. More digging informed me what it was I needed to do, change the default client credentials (which I think was delegate) to impersonate. This lead me down a quite difficult path with WCF and configuration files that helped me understand the concepts more, but didn't really help me solve my issue. Fortunately after searching around the WCF Custom Adapter settings for a while I found the solution and it was a lot easier than hacking configuration files.

To make my life easier I used the basicHttpBinding as my Binding Type In the properties for the WCF Custom Adapter. Also in these properties, under the Behavior tab, select EndpointBehavior on the left. Then right click and go to the menu option Add Extension; selecting clientCredentials as the extension. Now go into the client credentials and under the Windows section there is a dropdown for allowedImpersonation. This is not only exactly what I needed, it looks like most of the WCF config options are here and best of all you can import and export to a single config file for the bindings, so if you're sick of doing WCF bindings by hand you can use BizTalk to do them with a nice GUI.

Soon I'll post about some things I like more about the WCF Adapter over the SOAP Adapter.

I recently posted about Handling Binary Messages in Orchestrations and took a little time to reflect upon a key issue I saw recently (after I wrote that post) and that is working with non-XML documents within BizTalk in general. BizTalk is certainly an XML-centric product. This is a natural fit for any sort of integration platform; which as the XML name states is extensible. But often we're forced to cross out of the clean world of XML (yes, I think XML is clean, I've worked with it for almost nine years). This is where developers can get into trouble. Anytime you find yourself dealing with a non-XML artifact in BizTalk you should really convert it into XML as quickly as possible before it has a chance to get into the center of your solution.

An Example

A good example I saw is dealing with email. I ran across a solution that was handling inbound and outbound email and it was clear there had been challenges for the developers during its creation. For one as I already pointed out, BizTalk is really XML based and handling non XML in Orchestrations can be very tricky. This can also lead to unintentional dependencies that can limit the flexibility of the solution and complicate the testing.

On that testing note, I like to be able to substitute all external systems in my solutions during development. This can be a bit trickier for Request-Response ports, but it is still possible even if you're just using simple stubs or stand-ins. Further it is always good to be able to change adapters after solution rollout for operational reasons. Basically this means any One Way port should be replaceable with another. A great example would be SMTP or File; you should be able to swap these at anytime or switch to something else like FTP. This just makes a solution more flexible without requiring code changes and this is one of BizTalk's greatest strengths.

An Example Solution

At first the value of this may not be clear. Let's say for instance that your solution needs to receive emails and do something with them. When the POP3 adapter receives an email it creates a binary multipart message with parts for the subject, body, and attachments (one each). You would probably also want to know other information such as From and To. You could write a solution that uses POP3 properties directly from within an Orchestration like: EmailMessage(POP3.From), but this will be both harder to test and propagate adapter details into your Orchestration. Make no mistake; that is an external dependency.

I think a better way to handle this type of requirement is to define an XML message to represent everything you need from the email. Such a schema may look like this:

From here you would probably want to use a Pipeline to create your message. Since the Pipeline executes in the context of the Adapter, by the time the Message Box is reached this is no longer a binary multipart message, but a single part XML message. In this schema I've defined Attachment as Base64Binary so it can safely store any type of attachment.

Now when I want to build functional tests for my Orchestration using BizUnit (which is really a great tool and even trivial solutions should use it) I can just use a File Receive location instead of the POP3 Adapter (which is both faster than POP3 and doesn't require another external dependency: a POP3 server). Obviously you would want your Deployment Verification Tests to use the POP3 Adapter for true end to end verification and you would also want to test your Pipeline either using pipeline.exe or some of the techniques Michael Stephenson recommends here.

Ultimately this is the same as mapping at the Port, which should always be done to avoid external dependencies. Often doing something right in BizTalk requires me to rethink my initial approach in the context of how the product works. I have always had great results if I take the time to do this.

An Easy Way Out

The last thing I'll add is that creating a Pipeline to do this isn't really an easy task if you're new to BizTalk; so if you need a quicker method that you may be more comfortable with: simply receive the email into an Orchestration that just does the XML message creation for you reading from the message and the adapter properties. Then you just submit your message directly to the Message Box and your processing Orchestration can receive them directly as well (it will not know the difference between these two approaches: pipeline and receiving orchestration).

Being an extensible .NET based product it is certainly possible to make BizTalk do pretty much anything you want (even deal with non-XML documents ad nauseam), but you may find yourself fighting the product rather than leveraging it.

Introduction

In my post How BizTalk Identifies Types I explained that any message can be represented within an Orchestration as an instance of XmlDocument even if it is not a valid XML document (such as a binary file). In this post I will explain how you can handle binary documents in Orchestrations and walk through how I handled receiving reports from SQL Server Reporting Services (SSRS) and sending them to Sharepoint via the Sharepoint Services Adapter.

Often there are times in an integration solution where non XML documents must be operated on by a solution. One such example would be receiving PDF or image files and sending them to another location. In simple cases you may be able to do this in a messaging only scenario, in more complicated cases you may need to bring in Orchestration.

Sending and Receiving Binary Messages in Orchestration

Although any message can indeed be represented as an XmlDocument in an Orchestration performing any operation on the XmlDocument (i.e. the message or variable that is the message) will result in a runtime exception. If you're solution requires some sort of action to be performed on these documents that cannot be done via direct subscription you are limited to setting promoted properties on the message (i.e. Adapter / BizTalk properties). This may be all you need to do and if that's the case you may not need an Orchestration in the first place, but maybe a Pipeline.

Creating Binary Messages in Orchestration

At times you may find yourself required to do something a bit more complex, like retrieve the binary result of a Web Service call (like I did with SSRS) and doing something with the message payload. In my case I wanted to put the report in a Sharepoint site dynamically, which certainly could have been done with the SSRS Sharepoint integration, but I don't like that solution as it is a point to point rather than hub and spoke remedy.

There are several ways to create messages in Orchestrations and some are not very well documented, but the one recommended way (documented in the Sdk\Samples\Pipelines\ComposedMessageProcessor example) to create new Binary Messages is to use a send Pipeline called by a Message Assignment shape. Creating the Pipeline can be made much easier by using the BizTalk Server Pipeline Component Wizard which is a free Visual Studio plug-in that makes this a very simple process (we'll look at this later).

The main caveat here is something I pointed out in How BizTalk Identifies Types and that is that Pipelines and Orchestrations use different interfaces to represent messages. As a result, you must the use the supplied wrapper classes for executing this functionality. For one you will need an Orchestration variable of type Microsoft.XLANGs.Pipeline.SendPipelineInputMessages (which requires a reference to Microsoft.XLANGs.Pipeline.dll, in your BizTalk install location).

Calling the Pipeline

The basic steps are quite simple: add a message to an instance of this class via the Add method which takes one parameter: the message to be added; then call the Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteSendPipeline method from within a Message Assignment shape or an Expression shape before the Message Assignment.

Every example I've ever seen including Composed Message Processor always first sets the output message (the XLANGMessage that is the last parameter) to null before calling this method so it's probably a good idea to do so.

Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteSendPipeline(

    Type, SendPipelineInputMessages, XLANGMessage);

Be sure the Type is the that of your .btp pipeline that uses your component, not the Pipeline component created by the Wizard that we discuss below.

After this your binary message is in the message you supplied for the XLANGMessage parameter which I generally make as type XmlDocument. From here you are free to do what you like with the message, but remember it is not really an XmlDocument so don't call any of its methods.

More about that Pipeline

After you've run the Pipeline Component Wizard you have a class that implements IComponent, IBaseComponent, IPersistPropertyBag, and IComponentUI. The single method Execute of IComponent is the one we're interested in, this is what will be called when ExecuteSendPipeline is invoked.

If your situation is like mine this is all slightly complicated by the fact that the binary data you want is in a Base64String typed element within a message you have received. This is where the Pipeline Component Wizard can really help by allowing you to specify properties that can be set for the Pipeline. I declared an xpath used to read out the payload element from the body, this way I can reuse my Pipeline or change message formats. You can also create a class that represents your original message (containing the binary element) and use the XmlSerializer to create an instance of this message and access the byte[] property containing your raw byes (the XmlSerializer handles the translation to and from Base64String for you). End result, don't forget to convert from Base64String to byte[] or your file will be corrupted.

Below is the code from my Execute method. The properties are created for you in the Wizard and you don't need to worry about the other parts of the class.

//This is required: create the message to return

XmlTextReader reader = new XmlTextReader(inmsg.BodyPart.Data);

IBaseMessageFactory factory = pc.GetMessageFactory();

IBaseMessage message = factory.CreateMessage();

IBaseMessagePart body = factory.CreateMessagePart();

//This uses more properties and reflection to process the message

System.Runtime.Remoting.ObjectHandle objectHandle = Activator.CreateInstance(_assemblyName, _typeName);

object underlyingObject = objectHandle.Unwrap();

XmlSerializer serializer = new XmlSerializer(underlyingObject.GetType());

underlyingObject = serializer.Deserialize(reader);

object result = underlyingObject.GetType().InvokeMember(PropertyName, System.Reflection.BindingFlags.Public |

System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.GetProperty, null, underlyingObject, null);

byte[] resultBytes = (byte[])result;

Importantly you use either the segment above or the one below, but not both.

//This avoids the relfection, but makes the pipeline only useful to one message type

XmlSerializer serializer = new XmlSerializer(typeof(RenderResponse));

RenderResponse response = (serializer.Deserialize(reader)) as RenderResponse;

byte[] resultBytes = response.Result;

//This also is common to both approaches

MemoryStream memStream = new MemoryStream();

memStream.Write(resultBytes, 0, resultBytes.Length);

memStream.Position = 0;

body.Data = memStream;

pc.ResourceTracker.AddResource(memStream);

StreamReader sr = new StreamReader(body.Data);

message.AddPart(inmsg.BodyPartName, body, true);

return message;

At some point you'll probably want to populate some properties for the Adapter you're using either in the Message Assignment shape in your Orchestration (which I did) or in the Pipeline itself.

Once you have your pipeline component you add it to your \Microsoft BizTalk Server 2006\Pipeline Components directory and you are free to use it when creating .btp files. At this point you simply add the Pipeline to your BizTalk Toolbox (by picking Choose Items) and then drag it into the encode stage.

If you've used properties to make your Pipeline configurable you would set them here as well. This is the Type that you will specify in the call to ExecuteSendPipeline in the Message Assignment shape.

Improvements

I know there must be a way to stream this and it would be a good idea. Loading this message into an XmlDocument or using the XmlSerializer will load the entire message into memory; this can cause problems with very large messages. The entire BizTalk engine is designed to use streaming and it would be nice to incorporate it here as well. If you can think of a way to do this, please let me know.

A while back I ran into Marty Wasznicky and Scott Zimmerman excellent MSDN Article 8 Tips And Tricks For Better BizTalk Programming which is really worth a bookmarking. I really never got into their #1 Always Use Multi-Part Message Types (though the others I had picked up most of the hard way). I normally work with more messaging centric solutions and as a result do less with Orchestration, but the last year and a half has had me doing a lot more Orchestration and now I really understand this tip a lot better.

Although most Messages used in BizTalk Orchestrations are not multipart, using them in an Orchestration provides the benefit of isolating schema changes from impacting Orchestrations. Changing a schema without using multipart messages will require disconnecting all the Port, Send, and Receive shapes within the Orchestration Designer. This is a tedious an error prone process and it is not always easy to track down all of these affected shapes and there is no automated way to do it ahead of time. As the authors correctly point out; binding Ports and Messages within an Orchestration directly to a schema can lead to problems down the road. Usually these problems appear the first time you try to change something, which is often before even getting to production.

Granted there is one extra step here, defining the new part of the multipart message, but this is a small amount of work to do for a large amount of benefit. If you've ever worked with Web Services in an Orchestration (as a consumer) you've seen the Web Message Types that are generated for you; this is basically the same thing as using the multipart message types.