Effective BizTalk

I was recently approached by a client about what options they had for configuring BizTalk across their global enterprise. They really wanted to avoid idle servers and spread their processing load over their global enterprise as much as possible. They also wanted to have failover capabilities in case they lost their primary datacenter.

I thought a lot about this and had some ideas, but it was immediately clear to me that ultimately SQL Server is the critical point of failure in a BizTalk deployment. I've used a few approaches in the past to address this, such as the typical Log Shipping method that is outlined in the BizTalk documentation, but I wanted to push further.

As I thought about the options for this particular situation I reach out to Tim Wieman, one of the extremely knowledgeable BizTalk guys at Microsoft, for advice. Tim outlined four major approaches; three of which I was aware of and one that was new and quite innovative. Further he outlined them much more elegantly than I had been able to.

Log Shipping to Warm Standby

This approach is the most traditional for BizTalk and the one I have used most often. The spare servers are certainly a cost and the downside is that the lag time between the log shipping can result in lost data. The plus side is that it is quite simple, very widely used and understood, and can have you up and running again in a very short time (certainly within the uptime requirements of the vast majority of enterprises).

Independent BizTalk Groups

This approach involves having two separate BizTalk Groups, each doing some processing, for separate applications. They don't run the same sets of applications (but they can if you want to do load balancing through another mechanism) which means that they are both slightly underutilized, but this is probably the best option for a lot of shops. If you include load balancing from a hardware option you get the added benefit of getting full utilization of your infrastructure. There are some downsides, however. For one your tracking data may need to be aggregated from the two groups. If your tracking doesn't have as stringent of uptime requirements I think you might be able to apply the same tracking profile in each group and be able to avoid the aggregation step.

Geographically separated servers in the same group

This is one potential approach and I have heard of it being done. The real downside is the latency between the BizTalk servers and the SQL Server running the databases. This can be solved by high speed dedicated links (the only time I saw this used the two datacenters were actually in the same city). The weak link here though is the SQL Server. Lose your SQL Server and you've lost BizTalk for the most part. If your connection is fast enough, however, you can log ship across it and get some pretty good redundancy.

BizTalk Database Stretch Failover Cluster

Also known as a Geo-Cluster this approach addresses the core issue of BizTalk geographic redundancy directly, the SQL Server. This is the option I had not really heard of before. The key to this is SAN mirroring, which must be synchronous and support atomic synchronous writes. This is a very expensive approach as it requires both fast networks (fiber speeds) and vendor specific mirroring features in your SAN. I was extremely interested in this approach and looked into it more to find that iSCSI is one way in which this is performed by some SANs.

The remote SAN would have its own database server in the remote datacenter you would also put some BizTalk servers in your Group in the redundant location. This remote SQL Server would be a passive node in the cluster. The BizTalk servers need not be running (as in Host Instances started) before the cutover and you could use either SCOM or some other monitoring solution to enable those servers when the cluster fails over (or even windows cluster). This does represent the ultimate in failover capability with a minimum of down time, but as previously mentioned it would be very expensive. There would also be some time required to determine the right failover settings for the cluster so as to avoid false failovers.

I definitely have more work to do in this area and hope to write more about it in the future, but there is at least one more option than I hadn't thought too much about. The SAN failover also raises the possibility of redundant SAN in a single location (or a nearby datacenter) and this raises some great possibilities for customized solutions.

Although XML has been around for quite some time many developers still don't know too much about it. Not needing to know too much about it is one of XML's strengths, but cursory knowledge is rapidly becoming mandatory for all developers. This post will discuss some of the details of namespaces in XML.

In XML a namespace is just that, a space (or area) identified by a name. When we look at a namespace such as http://novaenterprisesystems.com/schemas/purchaseorder we are not actually identifying a location on the web. Granted it does look like a URL and your browser will probably make you want to click it (don't bother, at least for now it doesn't actually go anywhere). In an XML document this simply means in a namespace that certain nodes of the document exist in this namespace. That is they are identified by their Element / Attribute name and this namespace. Here is an example:

In this case this means that the PurchaseOrder element is identified using the specific namespace http://novaenterprisesystems.com/schemas/purchaseorder. I could have just as easily (and legally) made the name space "bob". Although bob would be a legal namespace, it doesn't do much to describe what this schema represents. All this really means is that the PurchaseOrder element has an identity in the definition space of this schema. I could import another schema that defines a different PurchaseOrder element in another schema with no ill-effect (it would clearly need another namespace in order to be unique).

Namespaces ultimately should convey information about the service or document in question. The generally accepted pattern for naming schemas is as follows:

http://[company]/[role]/[businessarea]/[date or version]

Where Company is obviously the entity exposing this schema, Role is either schema (for message types) or interface (for operations such as web services), business area would be the vertical or horizontal segment of the service or document Company is providing (such as purchasing or billing). Date / version should be self-explanatory, but is most often the year and month the schema is effective (more on that in another post). A better example would be: http://novaenterprisesystems.com/schemas/purchasing/2009-11.

You may be asking yourself 'so why all the fuss'? The two main issues are legality in XML (more of a technical concern) and much more importantly readability by consumers. A namespace should clearly express the intent of a schema to which it is attached. If this is for a business document or file that should be clear (as well as what business document this represents). The definition difference between schema (message) and interface (operation) should clear confusion about what the document really means. You wouldn't want the consumers of your services to tightly couple the purchase order document to the operation of submitting a purchase order.

A last note I'd like to stress is that although the domain name http://novaenterprisesystems.com appears in this schema and message, the domain will receive no traffic as a result. Again that name is just an identifier, very similar to a namespace in .NET such as System.Data. Only when an import or include location points to a URL will network traffic occur.

I've spent a lot of time lately designing stateless services and the schemas to enable them to be both stateless and extensible (which will be another post), but in doing so I began to realize that very often people embrace BizTalk Orchestration as the closest analogy to Business Process and tend to over leverage it in an attempt to unify and track their business processes. There certainly are times when an Orchestration is a business process, but often business processes will span multiple Orchestrations. Fortunately BAM and the TPE allow for this.

So what's wrong with just making really large, long running Orchestrations anyway? I strongly believe that the less in flight instances of Orchestrations you have in your BizTalk servers the better. It's not that the product can't do it (scalability isn't really a big issue here when done right) or that it's that difficult to make the product do it well; side by side deployment of Orchestrations is simple in BizTalk and schema and artifact versioning strategies (yet another post) are widely established / embraced. It is really back to the idea of the business process as a whole and its constituent parts as stages. Again I am reminded of good programming practices: in general simple is good. If a method has too many lines of code it is difficult to understand and maintain, breaking it into smaller chunks is probably a good idea once this happens. This also promotes reuse.

The same is true of Orchestration, only more so because the administrative aspects of Orchestration are more pronounced and visible. The Orchestration designer is a great tool and the HAT view of Orchestration is very useful for both tracking and debugging, but it is easy to get carried away. A good all around architectural principal is that you should not have to change your solution architecture simply to support features that aren't a core part of the solution itself. Build processes and tracking often fall into this category. If your build process causes you to change your solution perhaps it is not your solution that needs changing, but your build process. The same is true of tracking.

This ultimately calls for Business Activity Monitoring, which I having been working with more and more lately. I think BAM really is the analogy that best represents the business process that project sponsors understand and want to see. I personally am a big fan of the Orchestration Designer for Business Analysts for use in creating tracking definitions. This can serve not just as a lunch point for BAM, but for the solution that you are creating. This tool is allows business users to define the data and milestones that are important to them. The output, an XML file, can then be imported into Excel for completing the definition and defining views. Once I show businesses what BAM offers them (especially when they learn no custom database code or structures must be written to support it) they want to use it for everything. This isn't a bad idea, it scales really well and is probably better than most of us would be given the time to develop ourselves. Jesus Rodriguez wrote a great BAM article recently for MSDN here.

Importantly BAM was not as robust in previous versions of BizTalk so the 2006 releases really took this a lot further. The greatest advance was undoubtedly implementing multiple continuations in Tracking Profiles. I think this contributes largely to the use of Orchestration as the primary business process analogy. Allowing multiple continuations really allows the visibility business want in a much more elegant solution than large Orchestrations, which were really the only way to accomplish this in the past.

I recently let my mind drift a bit while working on my book and a few training lessons and began to contemplate this question: what are the core guidelines for a BizTalk solution? In many ways these tenets are the same for non-BizTalk solutions as well and can serve as general guidelines for all distributed application (i.e. Enterprise System) development.

An Effective BizTalk Solution is loosely coupled and operationally flexible. It has clear boundaries, carefully controlled dependencies, and is as simple as possible without sacrificing the previous tenets. The nonfunctional requirements are well understood upfront and documented. Automated tests exist at the Functional and Stress Testing level. Building and MSI construction can be performed with very few simple commands and MSI is the method for installation (that is to say deployment is a clean, self contained, and easily repeatable process).

This post presents some useful suggestions for the Visual Studio Solution structure to use in BizTalk development. It is based on both my experiences as well as that of others within the BizTalk community.

Suggestion

Generally I like to break my solutions up into the seven projects show below. I find that this meets the needs of most solutions, but it is not a hard fast rule. Some solutions will have more, others less, but each project is explained below.

External Schemas (.xsd files)

This project contains all of the schemas that are sent or received by the BizTalk solution. Port level mapping ensures no dependencies are leaked into or out of the BizTalk implementation. This would include schemas generated by adapter wizards, SOAP references, etc. It is vital that these be treated as what they really are: external artifacts (i.e. external dependencies).

Schemas (.xsd files)

The Schemas project contains all the schemas used internally by a BizTalk solution. These are schemas that are never exposed to any other systems and would be used to define entities within the actual solution. Every external schema should have a corresponding internal schema or translate to part of a composite schema.

Maps (.btm files)

The Maps project contains all maps within the solution. It references the Schemas and External Schemas projects and nothing else (with the exception of custom functoid assemblies). This is the guardian that prevents external dependencies from permeating a BizTalk solution.

Pipelines (.btp files)

All pipeline components (assemblers and disassemblers) are grouped in this project to make testing and maintenance easier. This project should reference the External Schemas project and /or the Schemas project if needed. Alternatively if you want to provide slightly more robust isolation you could create internal and external pipeline projects, but I really only recommend this if you need to call a Pipeline from within an Orchestration to avoid leaking a dependency into your Orchestrations layer.

Orchestrations (.odx files)

This project contains all Orchestrations used in the solution. It references the Schemas project and possibly the Pipelines project (if you're using pipelines from within an Orchestration). Workflows can also exist in this assembly in BizTalk 2009.

Library (C#, resources, etc)

Any artifacts that are used by your solution should also be broken into their own projects. It is important to not let these bleed dependencies into your solution. If you have some custom components to do processing in an orchestration and some for custom functiods or pipeline components these should be broken into their own distinct projects.

Testing (.xml, .dtd, .cs files)

This project hosts all Unit and Functional tests and their supporting data. The only direct references should be to the utilities assembly and to the testing frameworks (e.g. NUnit and BizUnit). This also ensures a clean separation of tests and artifacts from the other parts of a solution.

Non Project Artifacts

There are also several Solution level folders I like to use to organize my solutions. These generally are:

• 3rdParty Assemblies – for storing any external assemblies and components the solution may need to utilize.
• Bindings – to store deployment bindings used in each of the environments you will have in your solution. I generally have Local Development, Integration, UAT, and Production.
• Build – used to hold build scripts.

Relative Path Locations

Always use relative path locations for all artifacts including:

• Keys used to sign assemblies
• References to other assemblies (i.e. project references not absolute references)
• Paths of test files

This will make it easier to build the solution on multiple machines and also make it easier to use a CI server to perform automated building of the solution.

Remember not all solutions will have these seven; some will have less others more, but keep in mind the separation of concerns brought up in the discussion about layers in a BizTalk solution. Later on I'll cover some specific examples of when to combine which parts of this guidance.

Reasoning

It is natural and healthy to ask why go through all this trouble (ignore that creating projects in a solution is very simple; even more so with a template). Earlier I mentioned the need to control dependencies and this is a certain way to do that, but there are other reasons as well. You may find yourself needing to update in-flight Orchestrations or Maps in a solution with such Orchestrations. Perhaps it's a simple fix and you haven't quite worked out you versioning strategy. With separate assemblies you are free to deploy just the changes necessary rather than all assemblies and artifacts at once.

Perhaps more importantly you are also free to create packages that are specific to the type of server you are deploying them to. If you are working in a large BizTalk environment you may have a dozen or more servers partitioned by their functional role. You may want to create deployment packages for each role. This goes back to the previous discussion about Layers vs. Tiers. Building solutions in a separable manner is a good practice that allows administrators and operators more control over the solution once it is out of development. This is critical to the success of any enterprise software system and even more so to BizTalk solutions.

This is going to be one of a series of posts on the topic I plan to release over the next several weeks. Part of this is driven by what I've seen as a weakness in the community: a lack of direct application of established design principles. Most .NET projects these days follow a now well established set of guidelines and principles that help ensure elegant easily maintainable solutions. These guidelines were established in the context of .NET development; borrowing from and building upon the experiences of older platforms such as Smalltalk, C++, and Java. In an effort to lead the community back to these guidelines I'd like to recover some of these concepts in the context of BizTalk solutions.

Separation of Concerns

BizTalk even more than most platforms is fundamentally involved with separation of concerns (being an integration solution), but often this idea is lost in the implementation of a BizTalk Solution. I think that developers can get so focused on being the separation for external applications that they neglect to keep this principle in mind in their own solutions and often inadvertently let dependencies and tight coupling permeate their solutions.

Layers of a BizTalk Solution

The proximity diagram below depicts how the layers in a BizTalk solution should interact. Only layers that actually touch each other should be interacting in any way. The intention from this should be quite clear; it is to isolate the layers in the solution and minimize their interaction points so as to create a more robust solution that does not allow changes to propagate throughout the system.

This layout will give us the ability to change the solution at its edges or in its core drastically without impacting other parts of the solution. A new external partner schema should have no impact on our Orchestration layer nor on our Internal Schemas.

This can also be expressed in a more detailed fashion in a UML diagram. The following diagram expresses the basic layers (or dependencies) in a BizTalk Solution that follows these guidelines.

Again the goal is flexibility by tightly controlling contact points and Visual Studio can actually help to enforce this architecture in your solution.

So why do this? Some would argue that you could just do a logical separation using solution folders and not need separate Visual Studio projects for each of the packages shown above. While there certainly is some merit to this argument, it overlooks some very important capabilities inherent in this architecture.

For one there is no chance of a dependency creeping throughout your solution, the Visual Studio references between projects will ensure that. Further you can also choose now to deploy you solution in a variety of arrangements of Tiers derived from the Layers presented above.

Overview

BizTalk Server is a great tool that can aid in the development in integration and SOA solutions. It is a force multiplier that allows one developer to perform the work of many. It is reliable and scalable. It is also a little difficult to get your head around at first. One of the greatest difficulties developers have starting out with BizTalk is to understand how types work in BizTalk Server. This is a pretty straightforward concept in .NET so the answer can throw off some people.

Types of Types

BizTalk primarily has two major types of types: Message Types and .NET Types. Everything in BizTalk is one of these and some things are both. The .NET types are simple enough, but we'll discuss some of their implications shortly. The Message Types are less so. Being a Messaging Platform everything that flows into or out of BizTalk Server is a message. Binary, XML, Text they are all messages. BizTalk identifies XML Message Types based on the namespace and root node of the XML instance itself. Since XML legally only allows one root node in a document this is a pretty standard identification. An example would be "http://novaenterprisesystems.com/schemas/sql#SqlRequest" where "SqlRequest" is the root element of the message and the schema is "http://novaenterprisesystems.com/schemas/sql". This works fine for XML messages, but other messages don't really work the same.

Types in Contexts

All messages in BizTalk's Messaging Engine are represented as an instance of Microsoft.BizTalk.Message.Interop.IBaseMessage and in a Pipeline the same holds true (as Pipelines are messaging components). Once in an Orchestration a message is represented as an instance of Microsoft.XLANGs.BaseTypes.XLANGMessage and can be sent to or from a user component as such. Interestingly enough these types cannot be converted between them so you'll generally use IBaseMessage in Pipeline Components and XLANGMessage exclusively in Orchestrations.

To make things a little more complicated any message in an Orchestration it is implicitly understood to be an instance of System.Xml.XmlDocument. It is not so much that these classes are all related, but they are functionally and have both practical and historic reasons for this arrangement. One is that it allows you a simple way to deal with non XML messages in orchestrations, you can specify their type as XmlDocument and handle just about any type of message, even binary. That said if you have a binary message and try to use any of the members of XmlDocument on it you will receive an exception from the Orchestration Engine.

These different interfaces allow different types of interactions with messages in their own contexts' which can sometimes be challenging until you understand the lay of the land so to speak.

To sum it up in these contexts messages are these types:

Messaging Engine: IBaseMessage

Orchestration: XLANGMessage or XmlDocument

Later I will post on how to create messages in the different parts of BizTalk Server.