Effective BizTalk

Over the last two years I have really pushed the envelope with BAM. I use it in every solution I implement, even if just for providing support information. One of the best features I've found with BAM is related activities. Unfortunately this feature is not very well documented and it took me several weeks of playing with it to understand how to make it work correctly.

Ultimately what must be done is that you create two Activities then create one View that spans the data in the activities (it's OK to grab everything into this view if you want to) then you deploy the definition. Next you create two Tracking Profiles in the Tracking Profile Editor (one for each Activity). The profiles use the Related Activity menu option to relate to each other. The really cool thing is that you can easily make bidirectional relationships that allow tracking from / to parent child structures (something BAM "can't do" out of the box). These let you click back and forth between the activities to see the details.

I'll try to post up some images and more info about this later because it is a little hard to explain, but it does work and it works great.

I'm a really big fan of BAM and I use it in basically every solution I create. Even if it is just for support use the ability to see into running business processes is vital. As much as I love BAM I would say that there is still a lack of good information out there about it. Here is one of my favorite details. Which schema you tie your BAM to depends on if you're binding to a Send or Receive port.

I always build my solutions with a consistent Visual Studio Solution Structure that has always worked well for me. Binding BAM to my internal (or canonical) schemas always made sense and I did it on Receive ports religiously. As I got further into BAM and pushed into more complex Continuations and features that were simply were not available in 2004 I discovered something interesting. I could never make my Send ports track (i.e. correctly) to my canonical schema.

It turns out that this is by design. BAM tracks at different points in the Pipeline / Port processing depending upon if a port is a send or receive port (or at least this is what I have gathered on my own). I was told of this during support research on My First BizTalk Bug. I really didn't like this, but some of it does make sense. Ultimately what this means is that if you want to track receipt and sending of business data in BAM your Tracking Profile must bind to the external schema (i.e. the post map result) of the send port.

If this weren't enough, however, you must also track the external schema on the response of a Solicit-Response send port (a two-way send port). I found this one just purely through trial and error. I suspect that in the name of consistency the Response from a Request-Response port (two-way receive) probably also needs to track on the internal schema.

After years of working with the product I finally came across my first real bug in BizTalk, or more precisely in BAM. To start off I'd say the vast majority of times I think I've found a bug in BizTalk I am just not using the product correctly. That said, even monkeys fall from trees. I'll preface this by stating that I use BAM a lot, probably more than any other BizTalk developer I know. I love BAM and the visibility it provides into business processes. I particularly like the Tracking Profile Editor as I prefer not to change my solution, even in simple ways, to accommodate tracking. The next thing I'll say is that Microsoft Support was amazing at fixing these issues for me and their team was honestly interested in the bugs and their resolution. I cannot thank them enough for their help.

There were in fact two separate issues that had come up. The first had to do with de-batching and tracking the individual messages. You can read about it here KB 970856. The symptom was that one message in the batch was not tracked. I had my first test hotfix for this so quickly I was amazed. This first scenario was the beginning of a multi-stage business process that could take a very long time to complete. It was quite easy to see and replicate this issue (in fact, my BizUnit tests did so). I did learn than my BizTalk Visual Studio Solution Structure works quite well as MS Support was able to unzip, build, and deploy my application very quickly. I took this as a good test and compliment.

The second issue proved more difficult for me because it involved further stages in the business process and it was hard to tell if the first issue was interfering here or not. It is documented here: KB 967765. At the end of the day it was not, but I did learn a lot about how BAM actually works and how it is meant to be used. I will post on that soon enough. This second issue had to do with multiple continuations and de-batching (I'm just not a big fan of the batch if you can't tell). I would get unpredictable results in my tracking for these and could not for the life of me figure out why. Some records would not be tracked, others would be partially tracked. I was pulling my hair out. Microsoft kindly pointed out to me that the issue could be avoided by changing my tracking solution in two ways: I had to have all my continuations at the beginning of the profile; I also had to track on the external message format (post map) for send ports. Although this did work, I was glad Microsoft provided a fix because the only way to move the continuations was in a text editor and I like to track on internal schemas so my tracking profiles don't change when my trading partners do. Again, more on this in my next BAM post.

Again I really want to thank the fantastic team at Microsoft Support. They were friendly, professional, and fast! I have never had a better support experience with Microsoft or any other vendor before.

I've spent a lot of time lately designing stateless services and the schemas to enable them to be both stateless and extensible (which will be another post), but in doing so I began to realize that very often people embrace BizTalk Orchestration as the closest analogy to Business Process and tend to over leverage it in an attempt to unify and track their business processes. There certainly are times when an Orchestration is a business process, but often business processes will span multiple Orchestrations. Fortunately BAM and the TPE allow for this.

So what's wrong with just making really large, long running Orchestrations anyway? I strongly believe that the less in flight instances of Orchestrations you have in your BizTalk servers the better. It's not that the product can't do it (scalability isn't really a big issue here when done right) or that it's that difficult to make the product do it well; side by side deployment of Orchestrations is simple in BizTalk and schema and artifact versioning strategies (yet another post) are widely established / embraced. It is really back to the idea of the business process as a whole and its constituent parts as stages. Again I am reminded of good programming practices: in general simple is good. If a method has too many lines of code it is difficult to understand and maintain, breaking it into smaller chunks is probably a good idea once this happens. This also promotes reuse.

The same is true of Orchestration, only more so because the administrative aspects of Orchestration are more pronounced and visible. The Orchestration designer is a great tool and the HAT view of Orchestration is very useful for both tracking and debugging, but it is easy to get carried away. A good all around architectural principal is that you should not have to change your solution architecture simply to support features that aren't a core part of the solution itself. Build processes and tracking often fall into this category. If your build process causes you to change your solution perhaps it is not your solution that needs changing, but your build process. The same is true of tracking.

This ultimately calls for Business Activity Monitoring, which I having been working with more and more lately. I think BAM really is the analogy that best represents the business process that project sponsors understand and want to see. I personally am a big fan of the Orchestration Designer for Business Analysts for use in creating tracking definitions. This can serve not just as a lunch point for BAM, but for the solution that you are creating. This tool is allows business users to define the data and milestones that are important to them. The output, an XML file, can then be imported into Excel for completing the definition and defining views. Once I show businesses what BAM offers them (especially when they learn no custom database code or structures must be written to support it) they want to use it for everything. This isn't a bad idea, it scales really well and is probably better than most of us would be given the time to develop ourselves. Jesus Rodriguez wrote a great BAM article recently for MSDN here.

Importantly BAM was not as robust in previous versions of BizTalk so the 2006 releases really took this a lot further. The greatest advance was undoubtedly implementing multiple continuations in Tracking Profiles. I think this contributes largely to the use of Orchestration as the primary business process analogy. Allowing multiple continuations really allows the visibility business want in a much more elegant solution than large Orchestrations, which were really the only way to accomplish this in the past.