Effective BizTalk

I was recently approached by a client about what options they had for configuring BizTalk across their global enterprise. They really wanted to avoid idle servers and spread their processing load over their global enterprise as much as possible. They also wanted to have failover capabilities in case they lost their primary datacenter.

I thought a lot about this and had some ideas, but it was immediately clear to me that ultimately SQL Server is the critical point of failure in a BizTalk deployment. I've used a few approaches in the past to address this, such as the typical Log Shipping method that is outlined in the BizTalk documentation, but I wanted to push further.

As I thought about the options for this particular situation I reach out to Tim Wieman, one of the extremely knowledgeable BizTalk guys at Microsoft, for advice. Tim outlined four major approaches; three of which I was aware of and one that was new and quite innovative. Further he outlined them much more elegantly than I had been able to.

Log Shipping to Warm Standby

This approach is the most traditional for BizTalk and the one I have used most often. The spare servers are certainly a cost and the downside is that the lag time between the log shipping can result in lost data. The plus side is that it is quite simple, very widely used and understood, and can have you up and running again in a very short time (certainly within the uptime requirements of the vast majority of enterprises).

Independent BizTalk Groups

This approach involves having two separate BizTalk Groups, each doing some processing, for separate applications. They don't run the same sets of applications (but they can if you want to do load balancing through another mechanism) which means that they are both slightly underutilized, but this is probably the best option for a lot of shops. If you include load balancing from a hardware option you get the added benefit of getting full utilization of your infrastructure. There are some downsides, however. For one your tracking data may need to be aggregated from the two groups. If your tracking doesn't have as stringent of uptime requirements I think you might be able to apply the same tracking profile in each group and be able to avoid the aggregation step.

Geographically separated servers in the same group

This is one potential approach and I have heard of it being done. The real downside is the latency between the BizTalk servers and the SQL Server running the databases. This can be solved by high speed dedicated links (the only time I saw this used the two datacenters were actually in the same city). The weak link here though is the SQL Server. Lose your SQL Server and you've lost BizTalk for the most part. If your connection is fast enough, however, you can log ship across it and get some pretty good redundancy.

BizTalk Database Stretch Failover Cluster

Also known as a Geo-Cluster this approach addresses the core issue of BizTalk geographic redundancy directly, the SQL Server. This is the option I had not really heard of before. The key to this is SAN mirroring, which must be synchronous and support atomic synchronous writes. This is a very expensive approach as it requires both fast networks (fiber speeds) and vendor specific mirroring features in your SAN. I was extremely interested in this approach and looked into it more to find that iSCSI is one way in which this is performed by some SANs.

The remote SAN would have its own database server in the remote datacenter you would also put some BizTalk servers in your Group in the redundant location. This remote SQL Server would be a passive node in the cluster. The BizTalk servers need not be running (as in Host Instances started) before the cutover and you could use either SCOM or some other monitoring solution to enable those servers when the cluster fails over (or even windows cluster). This does represent the ultimate in failover capability with a minimum of down time, but as previously mentioned it would be very expensive. There would also be some time required to determine the right failover settings for the cluster so as to avoid false failovers.

I definitely have more work to do in this area and hope to write more about it in the future, but there is at least one more option than I hadn't thought too much about. The SAN failover also raises the possibility of redundant SAN in a single location (or a nearby datacenter) and this raises some great possibilities for customized solutions.

I've got a new article in the Principles of Service Design Series posted on cio.com.  It can be found here: http://advice.cio.com/dan_rosanova/10416/statelessness_part_1

I was reading today a great blog entry by the immensely knowledgeable Richard Seroter about debatching inbound messages from biztalk wcf-sql adapter and was inspired to write some of my own musings on this great new adapter. Today will be transaction scopes.

The WCF SQL adapter allows you to choose to use the "Ambient" transaction or a different one when communicating with the Message Box. This means the adapter can enlist its communication with the Message Box (i.e. submission to BizTalk) in the same DTC as the read from the SQL database where the receive is happening.

The great feature with that is if you go "ambient" and you have no subscriptions set up in BizTalk the records stay in the source database instead of being pulled and simply suspended in BizTalk (the other option). This can be a life saver because a source system may think the records have gone to BizTalk and all is well, but they're getting lost or suspended. Like many great BizTalk adapter features this is runtime configurable; so your operators can decide how they want this to work.

With the new Adapter this is a choice, not a requirement. This is even better. I really like this because it now gives us the option, based on our solution needs, to decide how we handle error situations regarding routing failure exceptions.

I have added a new post to the Principles of Servies Oriented Design series on cio.com.  This one is focused on Loose Coupling and introducts the concept, some background and a brief definition.

I’m overlapping my series on UDDI with one on Principles of Service Design that should be interesting.  Mostly me ranting and raving about how a focus on the basics leads to effective services and the trade-offs and considerations this implies.  The first entry can be found here: Principles of Service Design.  The series should last about a month (maybe more since the UDDI series continues). 

I am writing for cio.com again and thought I'd go back down the SOA track. This will be in parallel to my BizTalk specific posts here, but there will be a lot of overlap. The UDDI series I am just starting will mostly be based on the UDDI v3 that is part of BizTalk Server 2009. The first of the cio.com articles can be found here: http://advice.cio.com/dan_rosanova/why_uddi_is_important

Future articles will elaborate on the points presented in the article above.

Over the last two years I have really pushed the envelope with BAM. I use it in every solution I implement, even if just for providing support information. One of the best features I've found with BAM is related activities. Unfortunately this feature is not very well documented and it took me several weeks of playing with it to understand how to make it work correctly.

Ultimately what must be done is that you create two Activities then create one View that spans the data in the activities (it's OK to grab everything into this view if you want to) then you deploy the definition. Next you create two Tracking Profiles in the Tracking Profile Editor (one for each Activity). The profiles use the Related Activity menu option to relate to each other. The really cool thing is that you can easily make bidirectional relationships that allow tracking from / to parent child structures (something BAM "can't do" out of the box). These let you click back and forth between the activities to see the details.

I'll try to post up some images and more info about this later because it is a little hard to explain, but it does work and it works great.

Although XML has been around for quite some time many developers still don't know too much about it. Not needing to know too much about it is one of XML's strengths, but cursory knowledge is rapidly becoming mandatory for all developers. This post will discuss some of the details of namespaces in XML.

In XML a namespace is just that, a space (or area) identified by a name. When we look at a namespace such as http://novaenterprisesystems.com/schemas/purchaseorder we are not actually identifying a location on the web. Granted it does look like a URL and your browser will probably make you want to click it (don't bother, at least for now it doesn't actually go anywhere). In an XML document this simply means in a namespace that certain nodes of the document exist in this namespace. That is they are identified by their Element / Attribute name and this namespace. Here is an example:

In this case this means that the PurchaseOrder element is identified using the specific namespace http://novaenterprisesystems.com/schemas/purchaseorder. I could have just as easily (and legally) made the name space "bob". Although bob would be a legal namespace, it doesn't do much to describe what this schema represents. All this really means is that the PurchaseOrder element has an identity in the definition space of this schema. I could import another schema that defines a different PurchaseOrder element in another schema with no ill-effect (it would clearly need another namespace in order to be unique).

Namespaces ultimately should convey information about the service or document in question. The generally accepted pattern for naming schemas is as follows:

http://[company]/[role]/[businessarea]/[date or version]

Where Company is obviously the entity exposing this schema, Role is either schema (for message types) or interface (for operations such as web services), business area would be the vertical or horizontal segment of the service or document Company is providing (such as purchasing or billing). Date / version should be self-explanatory, but is most often the year and month the schema is effective (more on that in another post). A better example would be: http://novaenterprisesystems.com/schemas/purchasing/2009-11.

You may be asking yourself 'so why all the fuss'? The two main issues are legality in XML (more of a technical concern) and much more importantly readability by consumers. A namespace should clearly express the intent of a schema to which it is attached. If this is for a business document or file that should be clear (as well as what business document this represents). The definition difference between schema (message) and interface (operation) should clear confusion about what the document really means. You wouldn't want the consumers of your services to tightly couple the purchase order document to the operation of submitting a purchase order.

A last note I'd like to stress is that although the domain name http://novaenterprisesystems.com appears in this schema and message, the domain will receive no traffic as a result. Again that name is just an identifier, very similar to a namespace in .NET such as System.Data. Only when an import or include location points to a URL will network traffic occur.

I'm a really big fan of BAM and I use it in basically every solution I create. Even if it is just for support use the ability to see into running business processes is vital. As much as I love BAM I would say that there is still a lack of good information out there about it. Here is one of my favorite details. Which schema you tie your BAM to depends on if you're binding to a Send or Receive port.

I always build my solutions with a consistent Visual Studio Solution Structure that has always worked well for me. Binding BAM to my internal (or canonical) schemas always made sense and I did it on Receive ports religiously. As I got further into BAM and pushed into more complex Continuations and features that were simply were not available in 2004 I discovered something interesting. I could never make my Send ports track (i.e. correctly) to my canonical schema.

It turns out that this is by design. BAM tracks at different points in the Pipeline / Port processing depending upon if a port is a send or receive port (or at least this is what I have gathered on my own). I was told of this during support research on My First BizTalk Bug. I really didn't like this, but some of it does make sense. Ultimately what this means is that if you want to track receipt and sending of business data in BAM your Tracking Profile must bind to the external schema (i.e. the post map result) of the send port.

If this weren't enough, however, you must also track the external schema on the response of a Solicit-Response send port (a two-way send port). I found this one just purely through trial and error. I suspect that in the name of consistency the Response from a Request-Response port (two-way receive) probably also needs to track on the internal schema.

I recently started working with the DB2 Adapter for BizTalk Server and must admit I've been pretty happy with the results. DB2 not one of my specialties, but it is still out there and not going away anytime soon. I always enjoy getting the chance to work with new adapters when I can. The Adapter is reliable and fast and makes working with yet another platform a pretty simple experience for the BizTalk developer.

I did, however, have one unexpected experience with it having to do with Element Form Defaults (which I guess I'll write about more later). After generating schemas for receive locations I eventually figured out that the adapter was returning data with an Element Form Default of qualified even though the schema was set to unqualified. As I tested my maps it was clear from the validation failure messages that the namespace was an issue. This was pretty easy to fix (change the Element Form Default to Qualified). Easy enough.

Later I found myself doing sends to the DB2 Adapter and found some more interesting details. Request messages sent to the DB2 Adapter must Unqualified (that is their Element Form Default cannot be Qualified). Easy enough again. Unfortunately the response to these messages still follow the requirement of being Element Form Default Qualified. This presented a bit of a Chicken and Egg scenario. The solution is actually a good practice in my book: don't have different messages in the same schema. I broke the request and response into their own schemas and set the Element Form Default separately.