Effective BizTalk

I recently started working with the DB2 Adapter for BizTalk Server and must admit I've been pretty happy with the results. DB2 not one of my specialties, but it is still out there and not going away anytime soon. I always enjoy getting the chance to work with new adapters when I can. The Adapter is reliable and fast and makes working with yet another platform a pretty simple experience for the BizTalk developer.

I did, however, have one unexpected experience with it having to do with Element Form Defaults (which I guess I'll write about more later). After generating schemas for receive locations I eventually figured out that the adapter was returning data with an Element Form Default of qualified even though the schema was set to unqualified. As I tested my maps it was clear from the validation failure messages that the namespace was an issue. This was pretty easy to fix (change the Element Form Default to Qualified). Easy enough.

Later I found myself doing sends to the DB2 Adapter and found some more interesting details. Request messages sent to the DB2 Adapter must Unqualified (that is their Element Form Default cannot be Qualified). Easy enough again. Unfortunately the response to these messages still follow the requirement of being Element Form Default Qualified. This presented a bit of a Chicken and Egg scenario. The solution is actually a good practice in my book: don't have different messages in the same schema. I broke the request and response into their own schemas and set the Element Form Default separately.

After years of working with the product I finally came across my first real bug in BizTalk, or more precisely in BAM. To start off I'd say the vast majority of times I think I've found a bug in BizTalk I am just not using the product correctly. That said, even monkeys fall from trees. I'll preface this by stating that I use BAM a lot, probably more than any other BizTalk developer I know. I love BAM and the visibility it provides into business processes. I particularly like the Tracking Profile Editor as I prefer not to change my solution, even in simple ways, to accommodate tracking. The next thing I'll say is that Microsoft Support was amazing at fixing these issues for me and their team was honestly interested in the bugs and their resolution. I cannot thank them enough for their help.

There were in fact two separate issues that had come up. The first had to do with de-batching and tracking the individual messages. You can read about it here KB 970856. The symptom was that one message in the batch was not tracked. I had my first test hotfix for this so quickly I was amazed. This first scenario was the beginning of a multi-stage business process that could take a very long time to complete. It was quite easy to see and replicate this issue (in fact, my BizUnit tests did so). I did learn than my BizTalk Visual Studio Solution Structure works quite well as MS Support was able to unzip, build, and deploy my application very quickly. I took this as a good test and compliment.

The second issue proved more difficult for me because it involved further stages in the business process and it was hard to tell if the first issue was interfering here or not. It is documented here: KB 967765. At the end of the day it was not, but I did learn a lot about how BAM actually works and how it is meant to be used. I will post on that soon enough. This second issue had to do with multiple continuations and de-batching (I'm just not a big fan of the batch if you can't tell). I would get unpredictable results in my tracking for these and could not for the life of me figure out why. Some records would not be tracked, others would be partially tracked. I was pulling my hair out. Microsoft kindly pointed out to me that the issue could be avoided by changing my tracking solution in two ways: I had to have all my continuations at the beginning of the profile; I also had to track on the external message format (post map) for send ports. Although this did work, I was glad Microsoft provided a fix because the only way to move the continuations was in a text editor and I like to track on internal schemas so my tracking profiles don't change when my trading partners do. Again, more on this in my next BAM post.

Again I really want to thank the fantastic team at Microsoft Support. They were friendly, professional, and fast! I have never had a better support experience with Microsoft or any other vendor before.

For a long time I never worked very much with Parties and Role Links in BizTalk and I've been teaching it some lately (and had a question posted on a board) so here goes. I really just plan to talk about them and tell you where to get more info on using them. I guess it started when I was just trying to touch all the features I never had and found Role Links in the Orchestration designer. I checked MSDN which does give the answers, but not as directly as I was looking for (in retrospect the documentation is great, but I just needed a good overview to understand what I was looking at).

As I looked more I came to Richard Seroter's blog posting here which also points to a good post by Todd Uhl. This all started to make more and more sense over time. Then I had the requirement for a solution to send messages to a large variety of partners depending on some information that is looked up based on the input of an incoming message. There are a lot of ways to solve this in BizTalk and Dynamic Ports is certainly one, but I really don't like dynamic ports because they obscure important solution details from the administrator / operations team. To make matters worse the number of possible partners was likely to be in the hundreds for this solution.

After yet more research and playing around I stumbled onto the Party Resolution Sample (it's in the older versions too despite this link now pointing to 2009) and this is where I had the epiphany. Parties and Role Links could solve my problem with almost no work on my part. Here was my chance to look like a superhero without needing to do much more than figure out how to tie the pieces together (something that I find is common in BizTalk).

Basically it works like this: you use Role Links in your Orchestration like Richard suggests and if you have the defining piece of information in your inbound message (like in Richard's case) you're set. All I did was have a simple external service lookup that provided me that data from an existing repository. The best part of this solution is that the Maps go on the Ports (like they always should) and the Orchestration simply uses the internal (canonical) schemas. The other than assigning/resolving the party in the expression shape the Orchestration knows nothing about the ports and their details.

There are a few caveats such as: the Parties must exist in the BizTalk Group (surprise) and that they must have ports tied to them which will become a requirement of application (or more precisely Orchestration) binding before everything can be turned on. Also I'm pretty sure the ports need the same basic communication pattern (One Way or Two Way).

I've spent a lot of time lately designing stateless services and the schemas to enable them to be both stateless and extensible (which will be another post), but in doing so I began to realize that very often people embrace BizTalk Orchestration as the closest analogy to Business Process and tend to over leverage it in an attempt to unify and track their business processes. There certainly are times when an Orchestration is a business process, but often business processes will span multiple Orchestrations. Fortunately BAM and the TPE allow for this.

So what's wrong with just making really large, long running Orchestrations anyway? I strongly believe that the less in flight instances of Orchestrations you have in your BizTalk servers the better. It's not that the product can't do it (scalability isn't really a big issue here when done right) or that it's that difficult to make the product do it well; side by side deployment of Orchestrations is simple in BizTalk and schema and artifact versioning strategies (yet another post) are widely established / embraced. It is really back to the idea of the business process as a whole and its constituent parts as stages. Again I am reminded of good programming practices: in general simple is good. If a method has too many lines of code it is difficult to understand and maintain, breaking it into smaller chunks is probably a good idea once this happens. This also promotes reuse.

The same is true of Orchestration, only more so because the administrative aspects of Orchestration are more pronounced and visible. The Orchestration designer is a great tool and the HAT view of Orchestration is very useful for both tracking and debugging, but it is easy to get carried away. A good all around architectural principal is that you should not have to change your solution architecture simply to support features that aren't a core part of the solution itself. Build processes and tracking often fall into this category. If your build process causes you to change your solution perhaps it is not your solution that needs changing, but your build process. The same is true of tracking.

This ultimately calls for Business Activity Monitoring, which I having been working with more and more lately. I think BAM really is the analogy that best represents the business process that project sponsors understand and want to see. I personally am a big fan of the Orchestration Designer for Business Analysts for use in creating tracking definitions. This can serve not just as a lunch point for BAM, but for the solution that you are creating. This tool is allows business users to define the data and milestones that are important to them. The output, an XML file, can then be imported into Excel for completing the definition and defining views. Once I show businesses what BAM offers them (especially when they learn no custom database code or structures must be written to support it) they want to use it for everything. This isn't a bad idea, it scales really well and is probably better than most of us would be given the time to develop ourselves. Jesus Rodriguez wrote a great BAM article recently for MSDN here.

Importantly BAM was not as robust in previous versions of BizTalk so the 2006 releases really took this a lot further. The greatest advance was undoubtedly implementing multiple continuations in Tracking Profiles. I think this contributes largely to the use of Orchestration as the primary business process analogy. Allowing multiple continuations really allows the visibility business want in a much more elegant solution than large Orchestrations, which were really the only way to accomplish this in the past.

I'm a big fan of MSI installation and I think BizTalk does a better job than any other product in leveraging MSI deployment. There are many reasons why I like MSI installation, but the number one has to be the ability to hand off from Development to Operations (and possibly through Test in-between).

Separating Development from Operations teams is critical for enterprise systems and is something BizTalk does well; if you let it. In an effort to facilitate such clean handoffs I always package bindings in my MSIs to enable the installation onto a variety of environments. Generally I create Developer, Integration, Test, and Production bindings. This has worked well for me for years.

Recently I did see one downside and that was that an Administrator may want to know more about those bindings without installing the MSI anywhere. To enable this I created an XSLT to view bindings files and added a step to my build process to create the transformed bindings view. I then either packaged these into the MSIs as file or left them in the same directory as the MSI itself. This was a bit clunky, but the Operations teams from my customers liked it.

Recently I was reading this BizTalk post and saw Randal van Splunteren's post about extracting MSI contents. I had been thinking about creating a BizTalk MSI Viewer for some time and this showed me how easy it was to do. Randal had figured out the undocumented API which made this really quite simple. This first version unpacks the MSI and displays the bindings files, then transforms them via an XSLT in the install directory of the application (if you would like to change it). I've already got a long list of features to add any may end up opening the project up in the future.

I've posted the first beta of BizTalk Msi Viewer for feedback here.

I recently let my mind drift a bit while working on my book and a few training lessons and began to contemplate this question: what are the core guidelines for a BizTalk solution? In many ways these tenets are the same for non-BizTalk solutions as well and can serve as general guidelines for all distributed application (i.e. Enterprise System) development.

An Effective BizTalk Solution is loosely coupled and operationally flexible. It has clear boundaries, carefully controlled dependencies, and is as simple as possible without sacrificing the previous tenets. The nonfunctional requirements are well understood upfront and documented. Automated tests exist at the Functional and Stress Testing level. Building and MSI construction can be performed with very few simple commands and MSI is the method for installation (that is to say deployment is a clean, self contained, and easily repeatable process).

I've been on a pretty serious WCF kick for the past few months and more and more I love it. I like even more how well BizTalk works with WCF. BizTalk is the application server you normally need to provide yourself to make WCF very useful. In Handling Binary Messages in Orchestrations I had mentioned that I used SQL Reporting Services via Web Services to run reports and do something with the output. As I had been experimenting with the solution I tried using both the SOAP Adapter and the WCF Adapter and found the WCF Adapter to be much more to my liking.

Using the WCF Http Basic binding you are able to call existing SOAP services but I did run into a slight problem with the SSRS services. These services require impersonation and by default WCF handles impersonation differently than ASMX in IIS did. At first I thought I needed to change the web.config on the reporting server, but eventually I stumbled onto Delegation and Impersonation with WCF on MSDN. More digging informed me what it was I needed to do, change the default client credentials (which I think was delegate) to impersonate. This lead me down a quite difficult path with WCF and configuration files that helped me understand the concepts more, but didn't really help me solve my issue. Fortunately after searching around the WCF Custom Adapter settings for a while I found the solution and it was a lot easier than hacking configuration files.

To make my life easier I used the basicHttpBinding as my Binding Type In the properties for the WCF Custom Adapter. Also in these properties, under the Behavior tab, select EndpointBehavior on the left. Then right click and go to the menu option Add Extension; selecting clientCredentials as the extension. Now go into the client credentials and under the Windows section there is a dropdown for allowedImpersonation. This is not only exactly what I needed, it looks like most of the WCF config options are here and best of all you can import and export to a single config file for the bindings, so if you're sick of doing WCF bindings by hand you can use BizTalk to do them with a nice GUI.

Soon I'll post about some things I like more about the WCF Adapter over the SOAP Adapter.

It never ceases to amaze me how many people can start off with bad experiences with BizTalk (which is why I do this blog and share my experiences). It really is an amazing product, but most people don't (or often can't) give it the time or respect it really deserves / demands to make it work correctly. I suppose because it does have a nice installation and configuration UI most people just think it does what they want. I know people make this mistake with SQL Server as well. The following are what I consider some of the minimum best practices for configuring a BizTalk Server.

Separated Hosts

I see this one a lot. If you just configure BizTalk out of the box you will have one Host and Host Instance: BizTalk Server Application (isolated hosts don't really count for this). I don't even really like the name of this host, but it is what you end up with (although you can change it in the configuration tool). At the very least I recommend the following host configuration:

Receive

Send

Orchestration (you can leave as BizTalk Server Application if you like)

Tracking

I even make these hosts on my development machine. There are quite a few good reasons for this so I'll start with the most basic: resources. In .NET a process has only so many threads in its thread pool. Sure you can always increase this number via a global setting, but there is a point at which this will generally stop helping you and start hurting. Part of this has to do with context switching overhead part with pure resource allocation. BizTalk is a massively parallel product and as a result requires many threads to run. If you leave all of the above features configured in a single process (as it is by default) you will run into thread starvation. I've run into this just on a development machine during testing, but you will really see it pronounced in a production environment. BizTalk will run slowly, though like expected still run reliably.

Further in reality you probably want to have different user accounts for each of these hosts, this can give you tight security controls between parts of your application. You may not need the security aspect on your development machine, but any servers (Integration, UAT, Production) will definitely benefit from this. Better still isolating processing, sending, and receiving will allow you to monitor and profile your solution more closely. You can attach perfmon logs to each service as it makes sense; no need to track messages received per second in the Orchestration host.

Basic Database Configurations

It is truly shocking how often this one happens. Most people installing BizTalk Server are completely unaware that installing the BizTalk databases on a SQL Server changes the server's MaxDOP (Maximum Degree of Parallelism) to one. If you're not exactly sure what this means it does what it sounds like: limits how many processors will be used in a parallel query. The setting is per query so its impact may not be that big, but it affects the Server, not just the Database. This may not really matter too much for you, but if this database server is used by other applications it probably will have an effect. This option is required for proper functioning of the Message Box database only. Other databases installed with BizTalk actually benefit from being on a server with a higher degree of parallelism. This is a great argument for spreading your databases across multiple SQL Servers (or clusters).

The other one I see a lot is to put the databases on an Analysis Services server. Analysis and Transaction are two diametrically opposed types of processing. They each require a different set of configuration and resources and they do not live well together. If your BizTalk solution is going to face significant volume I would recommend against host its databases on a shared database server (i.e. a database server that hosts other applications). Most people totally underestimate how hard BizTalk will be on SQL Server. Since everything goes through the Message Box, which is in SQL, it will be under tremendous load. Usually when a BizTalk application is running slow, SQL Server is a big contributor.

Microsoft has published some excellent information here that can help you really optimize your databases for BizTalk. It is really worth taking the time to go through this and the other optimization resources Microsoft has made available for BizTalk. Soon I will post on my own test results from this optimization process.

This post presents some useful suggestions for the Visual Studio Solution structure to use in BizTalk development. It is based on both my experiences as well as that of others within the BizTalk community.

Suggestion

Generally I like to break my solutions up into the seven projects show below. I find that this meets the needs of most solutions, but it is not a hard fast rule. Some solutions will have more, others less, but each project is explained below.

External Schemas (.xsd files)

This project contains all of the schemas that are sent or received by the BizTalk solution. Port level mapping ensures no dependencies are leaked into or out of the BizTalk implementation. This would include schemas generated by adapter wizards, SOAP references, etc. It is vital that these be treated as what they really are: external artifacts (i.e. external dependencies).

Schemas (.xsd files)

The Schemas project contains all the schemas used internally by a BizTalk solution. These are schemas that are never exposed to any other systems and would be used to define entities within the actual solution. Every external schema should have a corresponding internal schema or translate to part of a composite schema.

Maps (.btm files)

The Maps project contains all maps within the solution. It references the Schemas and External Schemas projects and nothing else (with the exception of custom functoid assemblies). This is the guardian that prevents external dependencies from permeating a BizTalk solution.

Pipelines (.btp files)

All pipeline components (assemblers and disassemblers) are grouped in this project to make testing and maintenance easier. This project should reference the External Schemas project and /or the Schemas project if needed. Alternatively if you want to provide slightly more robust isolation you could create internal and external pipeline projects, but I really only recommend this if you need to call a Pipeline from within an Orchestration to avoid leaking a dependency into your Orchestrations layer.

Orchestrations (.odx files)

This project contains all Orchestrations used in the solution. It references the Schemas project and possibly the Pipelines project (if you're using pipelines from within an Orchestration). Workflows can also exist in this assembly in BizTalk 2009.

Library (C#, resources, etc)

Any artifacts that are used by your solution should also be broken into their own projects. It is important to not let these bleed dependencies into your solution. If you have some custom components to do processing in an orchestration and some for custom functiods or pipeline components these should be broken into their own distinct projects.

Testing (.xml, .dtd, .cs files)

This project hosts all Unit and Functional tests and their supporting data. The only direct references should be to the utilities assembly and to the testing frameworks (e.g. NUnit and BizUnit). This also ensures a clean separation of tests and artifacts from the other parts of a solution.

Non Project Artifacts

There are also several Solution level folders I like to use to organize my solutions. These generally are:

• 3rdParty Assemblies – for storing any external assemblies and components the solution may need to utilize.
• Bindings – to store deployment bindings used in each of the environments you will have in your solution. I generally have Local Development, Integration, UAT, and Production.
• Build – used to hold build scripts.

Relative Path Locations

Always use relative path locations for all artifacts including:

• Keys used to sign assemblies
• References to other assemblies (i.e. project references not absolute references)
• Paths of test files

This will make it easier to build the solution on multiple machines and also make it easier to use a CI server to perform automated building of the solution.

Remember not all solutions will have these seven; some will have less others more, but keep in mind the separation of concerns brought up in the discussion about layers in a BizTalk solution. Later on I'll cover some specific examples of when to combine which parts of this guidance.

Reasoning

It is natural and healthy to ask why go through all this trouble (ignore that creating projects in a solution is very simple; even more so with a template). Earlier I mentioned the need to control dependencies and this is a certain way to do that, but there are other reasons as well. You may find yourself needing to update in-flight Orchestrations or Maps in a solution with such Orchestrations. Perhaps it's a simple fix and you haven't quite worked out you versioning strategy. With separate assemblies you are free to deploy just the changes necessary rather than all assemblies and artifacts at once.

Perhaps more importantly you are also free to create packages that are specific to the type of server you are deploying them to. If you are working in a large BizTalk environment you may have a dozen or more servers partitioned by their functional role. You may want to create deployment packages for each role. This goes back to the previous discussion about Layers vs. Tiers. Building solutions in a separable manner is a good practice that allows administrators and operators more control over the solution once it is out of development. This is critical to the success of any enterprise software system and even more so to BizTalk solutions.

This is going to be one of a series of posts on the topic I plan to release over the next several weeks. Part of this is driven by what I've seen as a weakness in the community: a lack of direct application of established design principles. Most .NET projects these days follow a now well established set of guidelines and principles that help ensure elegant easily maintainable solutions. These guidelines were established in the context of .NET development; borrowing from and building upon the experiences of older platforms such as Smalltalk, C++, and Java. In an effort to lead the community back to these guidelines I'd like to recover some of these concepts in the context of BizTalk solutions.

Separation of Concerns

BizTalk even more than most platforms is fundamentally involved with separation of concerns (being an integration solution), but often this idea is lost in the implementation of a BizTalk Solution. I think that developers can get so focused on being the separation for external applications that they neglect to keep this principle in mind in their own solutions and often inadvertently let dependencies and tight coupling permeate their solutions.

Layers of a BizTalk Solution

The proximity diagram below depicts how the layers in a BizTalk solution should interact. Only layers that actually touch each other should be interacting in any way. The intention from this should be quite clear; it is to isolate the layers in the solution and minimize their interaction points so as to create a more robust solution that does not allow changes to propagate throughout the system.

This layout will give us the ability to change the solution at its edges or in its core drastically without impacting other parts of the solution. A new external partner schema should have no impact on our Orchestration layer nor on our Internal Schemas.

This can also be expressed in a more detailed fashion in a UML diagram. The following diagram expresses the basic layers (or dependencies) in a BizTalk Solution that follows these guidelines.

Again the goal is flexibility by tightly controlling contact points and Visual Studio can actually help to enforce this architecture in your solution.

So why do this? Some would argue that you could just do a logical separation using solution folders and not need separate Visual Studio projects for each of the packages shown above. While there certainly is some merit to this argument, it overlooks some very important capabilities inherent in this architecture.

For one there is no chance of a dependency creeping throughout your solution, the Visual Studio references between projects will ensure that. Further you can also choose now to deploy you solution in a variety of arrangements of Tiers derived from the Layers presented above.