Effective BizTalk

I’m overlapping my series on UDDI with one on Principles of Service Design that should be interesting.  Mostly me ranting and raving about how a focus on the basics leads to effective services and the trade-offs and considerations this implies.  The first entry can be found here: Principles of Service Design.  The series should last about a month (maybe more since the UDDI series continues). 

I am writing for cio.com again and thought I'd go back down the SOA track. This will be in parallel to my BizTalk specific posts here, but there will be a lot of overlap. The UDDI series I am just starting will mostly be based on the UDDI v3 that is part of BizTalk Server 2009. The first of the cio.com articles can be found here: http://advice.cio.com/dan_rosanova/why_uddi_is_important

Future articles will elaborate on the points presented in the article above.

Over the last two years I have really pushed the envelope with BAM. I use it in every solution I implement, even if just for providing support information. One of the best features I've found with BAM is related activities. Unfortunately this feature is not very well documented and it took me several weeks of playing with it to understand how to make it work correctly.

Ultimately what must be done is that you create two Activities then create one View that spans the data in the activities (it's OK to grab everything into this view if you want to) then you deploy the definition. Next you create two Tracking Profiles in the Tracking Profile Editor (one for each Activity). The profiles use the Related Activity menu option to relate to each other. The really cool thing is that you can easily make bidirectional relationships that allow tracking from / to parent child structures (something BAM "can't do" out of the box). These let you click back and forth between the activities to see the details.

I'll try to post up some images and more info about this later because it is a little hard to explain, but it does work and it works great.

Although XML has been around for quite some time many developers still don't know too much about it. Not needing to know too much about it is one of XML's strengths, but cursory knowledge is rapidly becoming mandatory for all developers. This post will discuss some of the details of namespaces in XML.

In XML a namespace is just that, a space (or area) identified by a name. When we look at a namespace such as http://novaenterprisesystems.com/schemas/purchaseorder we are not actually identifying a location on the web. Granted it does look like a URL and your browser will probably make you want to click it (don't bother, at least for now it doesn't actually go anywhere). In an XML document this simply means in a namespace that certain nodes of the document exist in this namespace. That is they are identified by their Element / Attribute name and this namespace. Here is an example:

In this case this means that the PurchaseOrder element is identified using the specific namespace http://novaenterprisesystems.com/schemas/purchaseorder. I could have just as easily (and legally) made the name space "bob". Although bob would be a legal namespace, it doesn't do much to describe what this schema represents. All this really means is that the PurchaseOrder element has an identity in the definition space of this schema. I could import another schema that defines a different PurchaseOrder element in another schema with no ill-effect (it would clearly need another namespace in order to be unique).

Namespaces ultimately should convey information about the service or document in question. The generally accepted pattern for naming schemas is as follows:

http://[company]/[role]/[businessarea]/[date or version]

Where Company is obviously the entity exposing this schema, Role is either schema (for message types) or interface (for operations such as web services), business area would be the vertical or horizontal segment of the service or document Company is providing (such as purchasing or billing). Date / version should be self-explanatory, but is most often the year and month the schema is effective (more on that in another post). A better example would be: http://novaenterprisesystems.com/schemas/purchasing/2009-11.

You may be asking yourself 'so why all the fuss'? The two main issues are legality in XML (more of a technical concern) and much more importantly readability by consumers. A namespace should clearly express the intent of a schema to which it is attached. If this is for a business document or file that should be clear (as well as what business document this represents). The definition difference between schema (message) and interface (operation) should clear confusion about what the document really means. You wouldn't want the consumers of your services to tightly couple the purchase order document to the operation of submitting a purchase order.

A last note I'd like to stress is that although the domain name http://novaenterprisesystems.com appears in this schema and message, the domain will receive no traffic as a result. Again that name is just an identifier, very similar to a namespace in .NET such as System.Data. Only when an import or include location points to a URL will network traffic occur.

I'm a really big fan of BAM and I use it in basically every solution I create. Even if it is just for support use the ability to see into running business processes is vital. As much as I love BAM I would say that there is still a lack of good information out there about it. Here is one of my favorite details. Which schema you tie your BAM to depends on if you're binding to a Send or Receive port.

I always build my solutions with a consistent Visual Studio Solution Structure that has always worked well for me. Binding BAM to my internal (or canonical) schemas always made sense and I did it on Receive ports religiously. As I got further into BAM and pushed into more complex Continuations and features that were simply were not available in 2004 I discovered something interesting. I could never make my Send ports track (i.e. correctly) to my canonical schema.

It turns out that this is by design. BAM tracks at different points in the Pipeline / Port processing depending upon if a port is a send or receive port (or at least this is what I have gathered on my own). I was told of this during support research on My First BizTalk Bug. I really didn't like this, but some of it does make sense. Ultimately what this means is that if you want to track receipt and sending of business data in BAM your Tracking Profile must bind to the external schema (i.e. the post map result) of the send port.

If this weren't enough, however, you must also track the external schema on the response of a Solicit-Response send port (a two-way send port). I found this one just purely through trial and error. I suspect that in the name of consistency the Response from a Request-Response port (two-way receive) probably also needs to track on the internal schema.

I recently started working with the DB2 Adapter for BizTalk Server and must admit I've been pretty happy with the results. DB2 not one of my specialties, but it is still out there and not going away anytime soon. I always enjoy getting the chance to work with new adapters when I can. The Adapter is reliable and fast and makes working with yet another platform a pretty simple experience for the BizTalk developer.

I did, however, have one unexpected experience with it having to do with Element Form Defaults (which I guess I'll write about more later). After generating schemas for receive locations I eventually figured out that the adapter was returning data with an Element Form Default of qualified even though the schema was set to unqualified. As I tested my maps it was clear from the validation failure messages that the namespace was an issue. This was pretty easy to fix (change the Element Form Default to Qualified). Easy enough.

Later I found myself doing sends to the DB2 Adapter and found some more interesting details. Request messages sent to the DB2 Adapter must Unqualified (that is their Element Form Default cannot be Qualified). Easy enough again. Unfortunately the response to these messages still follow the requirement of being Element Form Default Qualified. This presented a bit of a Chicken and Egg scenario. The solution is actually a good practice in my book: don't have different messages in the same schema. I broke the request and response into their own schemas and set the Element Form Default separately.

After years of working with the product I finally came across my first real bug in BizTalk, or more precisely in BAM. To start off I'd say the vast majority of times I think I've found a bug in BizTalk I am just not using the product correctly. That said, even monkeys fall from trees. I'll preface this by stating that I use BAM a lot, probably more than any other BizTalk developer I know. I love BAM and the visibility it provides into business processes. I particularly like the Tracking Profile Editor as I prefer not to change my solution, even in simple ways, to accommodate tracking. The next thing I'll say is that Microsoft Support was amazing at fixing these issues for me and their team was honestly interested in the bugs and their resolution. I cannot thank them enough for their help.

There were in fact two separate issues that had come up. The first had to do with de-batching and tracking the individual messages. You can read about it here KB 970856. The symptom was that one message in the batch was not tracked. I had my first test hotfix for this so quickly I was amazed. This first scenario was the beginning of a multi-stage business process that could take a very long time to complete. It was quite easy to see and replicate this issue (in fact, my BizUnit tests did so). I did learn than my BizTalk Visual Studio Solution Structure works quite well as MS Support was able to unzip, build, and deploy my application very quickly. I took this as a good test and compliment.

The second issue proved more difficult for me because it involved further stages in the business process and it was hard to tell if the first issue was interfering here or not. It is documented here: KB 967765. At the end of the day it was not, but I did learn a lot about how BAM actually works and how it is meant to be used. I will post on that soon enough. This second issue had to do with multiple continuations and de-batching (I'm just not a big fan of the batch if you can't tell). I would get unpredictable results in my tracking for these and could not for the life of me figure out why. Some records would not be tracked, others would be partially tracked. I was pulling my hair out. Microsoft kindly pointed out to me that the issue could be avoided by changing my tracking solution in two ways: I had to have all my continuations at the beginning of the profile; I also had to track on the external message format (post map) for send ports. Although this did work, I was glad Microsoft provided a fix because the only way to move the continuations was in a text editor and I like to track on internal schemas so my tracking profiles don't change when my trading partners do. Again, more on this in my next BAM post.

Again I really want to thank the fantastic team at Microsoft Support. They were friendly, professional, and fast! I have never had a better support experience with Microsoft or any other vendor before.

For a long time I never worked very much with Parties and Role Links in BizTalk and I've been teaching it some lately (and had a question posted on a board) so here goes. I really just plan to talk about them and tell you where to get more info on using them. I guess it started when I was just trying to touch all the features I never had and found Role Links in the Orchestration designer. I checked MSDN which does give the answers, but not as directly as I was looking for (in retrospect the documentation is great, but I just needed a good overview to understand what I was looking at).

As I looked more I came to Richard Seroter's blog posting here which also points to a good post by Todd Uhl. This all started to make more and more sense over time. Then I had the requirement for a solution to send messages to a large variety of partners depending on some information that is looked up based on the input of an incoming message. There are a lot of ways to solve this in BizTalk and Dynamic Ports is certainly one, but I really don't like dynamic ports because they obscure important solution details from the administrator / operations team. To make matters worse the number of possible partners was likely to be in the hundreds for this solution.

After yet more research and playing around I stumbled onto the Party Resolution Sample (it's in the older versions too despite this link now pointing to 2009) and this is where I had the epiphany. Parties and Role Links could solve my problem with almost no work on my part. Here was my chance to look like a superhero without needing to do much more than figure out how to tie the pieces together (something that I find is common in BizTalk).

Basically it works like this: you use Role Links in your Orchestration like Richard suggests and if you have the defining piece of information in your inbound message (like in Richard's case) you're set. All I did was have a simple external service lookup that provided me that data from an existing repository. The best part of this solution is that the Maps go on the Ports (like they always should) and the Orchestration simply uses the internal (canonical) schemas. The other than assigning/resolving the party in the expression shape the Orchestration knows nothing about the ports and their details.

There are a few caveats such as: the Parties must exist in the BizTalk Group (surprise) and that they must have ports tied to them which will become a requirement of application (or more precisely Orchestration) binding before everything can be turned on. Also I'm pretty sure the ports need the same basic communication pattern (One Way or Two Way).

I've spent a lot of time lately designing stateless services and the schemas to enable them to be both stateless and extensible (which will be another post), but in doing so I began to realize that very often people embrace BizTalk Orchestration as the closest analogy to Business Process and tend to over leverage it in an attempt to unify and track their business processes. There certainly are times when an Orchestration is a business process, but often business processes will span multiple Orchestrations. Fortunately BAM and the TPE allow for this.

So what's wrong with just making really large, long running Orchestrations anyway? I strongly believe that the less in flight instances of Orchestrations you have in your BizTalk servers the better. It's not that the product can't do it (scalability isn't really a big issue here when done right) or that it's that difficult to make the product do it well; side by side deployment of Orchestrations is simple in BizTalk and schema and artifact versioning strategies (yet another post) are widely established / embraced. It is really back to the idea of the business process as a whole and its constituent parts as stages. Again I am reminded of good programming practices: in general simple is good. If a method has too many lines of code it is difficult to understand and maintain, breaking it into smaller chunks is probably a good idea once this happens. This also promotes reuse.

The same is true of Orchestration, only more so because the administrative aspects of Orchestration are more pronounced and visible. The Orchestration designer is a great tool and the HAT view of Orchestration is very useful for both tracking and debugging, but it is easy to get carried away. A good all around architectural principal is that you should not have to change your solution architecture simply to support features that aren't a core part of the solution itself. Build processes and tracking often fall into this category. If your build process causes you to change your solution perhaps it is not your solution that needs changing, but your build process. The same is true of tracking.

This ultimately calls for Business Activity Monitoring, which I having been working with more and more lately. I think BAM really is the analogy that best represents the business process that project sponsors understand and want to see. I personally am a big fan of the Orchestration Designer for Business Analysts for use in creating tracking definitions. This can serve not just as a lunch point for BAM, but for the solution that you are creating. This tool is allows business users to define the data and milestones that are important to them. The output, an XML file, can then be imported into Excel for completing the definition and defining views. Once I show businesses what BAM offers them (especially when they learn no custom database code or structures must be written to support it) they want to use it for everything. This isn't a bad idea, it scales really well and is probably better than most of us would be given the time to develop ourselves. Jesus Rodriguez wrote a great BAM article recently for MSDN here.

Importantly BAM was not as robust in previous versions of BizTalk so the 2006 releases really took this a lot further. The greatest advance was undoubtedly implementing multiple continuations in Tracking Profiles. I think this contributes largely to the use of Orchestration as the primary business process analogy. Allowing multiple continuations really allows the visibility business want in a much more elegant solution than large Orchestrations, which were really the only way to accomplish this in the past.

I'm a big fan of MSI installation and I think BizTalk does a better job than any other product in leveraging MSI deployment. There are many reasons why I like MSI installation, but the number one has to be the ability to hand off from Development to Operations (and possibly through Test in-between).

Separating Development from Operations teams is critical for enterprise systems and is something BizTalk does well; if you let it. In an effort to facilitate such clean handoffs I always package bindings in my MSIs to enable the installation onto a variety of environments. Generally I create Developer, Integration, Test, and Production bindings. This has worked well for me for years.

Recently I did see one downside and that was that an Administrator may want to know more about those bindings without installing the MSI anywhere. To enable this I created an XSLT to view bindings files and added a step to my build process to create the transformed bindings view. I then either packaged these into the MSIs as file or left them in the same directory as the MSI itself. This was a bit clunky, but the Operations teams from my customers liked it.

Recently I was reading this BizTalk post and saw Randal van Splunteren's post about extracting MSI contents. I had been thinking about creating a BizTalk MSI Viewer for some time and this showed me how easy it was to do. Randal had figured out the undocumented API which made this really quite simple. This first version unpacks the MSI and displays the bindings files, then transforms them via an XSLT in the install directory of the application (if you would like to change it). I've already got a long list of features to add any may end up opening the project up in the future.

I've posted the first beta of BizTalk Msi Viewer for feedback here.