Effective BizTalk

Madhav Abhyankar has created a Visual Studio solution template that implements the exact recommendations from my book Microsoft BizTalk Server 2010 Patterns.  This is a great way to save a lot of time on a solution and follow what I believe is a great architecture for BizTalk applications.

I’ve been working a lot with performance and performance tuning over the last few months.  This is an area I have a pretty heavy background in and I always enjoy the chance to work with.  Although somewhat of a mysterious art there are tools out there that make it far easier; the first among these being Performance Analysis of Logs (PAL) by Clint Huffman.  It is the most valuable tool I know of when working with performance analysis on the Windows platform.  PAL continues to be updated regularly and even had a release earlier this month (thanks Clint).  PAL was the first tool that ever got me really thinking about PowerShell and I have to say the more I get into PowerShell the more amazed I am by it.  It’s like everything I liked from UNIX and everything I still love about Windows (and more). 

Today I had a request to time box the performance logs I’ve been running (templates from PAL) to working hours for a client.  This would require filtering out segments of time from the perf logs (PAL will let you pick start and end dates/times, but only one of each).  While looking for solutions I thought about ways to edit the log files.  Working with binary perfmon files has never seemed easy to me and everyone I ask seems to not really know how to do it.  Some offer up relog.exe as a solution (and it is a good solution) but I wanted to automate this process and I thought it’d be more work.  This would work with PAL though because as of version 2.0 it can take log files as a list and process them together (as one logical log). 

As I thought about how to do this I turned to Bing hoping there would be a PowerShell solution to my predicament.  I was NOT to be disappointed by the power that is PowerShell.  First I found the import-counter cmdlet and saw I could iterate over each PerformanceCounterSampleSet and even check the Timestamp property (kudos to Allen White for this BTW).  After playing around with this I realized I could actually pipe this to the Where-Object cmdlet to filter my selection.  A little deeper digging and I quickly realized I could ditch the iteration altogether and pipe these cmdlets to the export-counter cmdlet.  The result was this:

$logWindowStart = '09:00:00'

$logWindowEnd = '17:00:00'

$sunday = (Get-Date 2011/11/20).DayOfWeek

$saturday= (Get-Date 2011/11/19).DayOfWeek

import-counter $logfile | where {$_.Timestamp.TimeOfDay -gt $logWindowStart -and $_.Timestamp.TimeOfDay -lt $logWindowEnd -and $_.Timestamp.DayOfWeek -gt $sunday -and $_.Timestamp.DayOfWeek -lt $saturday} | export-counter -path $outputFile

The input and output filenames are parameters to the script and the hard coded dates don’t really matter because they’re just used to get DayOfWeek enums.  Hopefully this can be useful to someone else as well. 

I wrote a new post to the MSDN Wiki about some new experiences with the WCF-SQL Adapter.  I really like the MSDN Wiki concept. 

Using FOR XML queries with the WCF-SQL Adapter.

Feeback welcome. 

The BRE is one of the least understood parts of BizTalk server and after teaching Effective BizTalk again last week I was again confronted by why this is. When we first look at BRE one of the things that is not clear is what is meant by the forward chaining nature of a policy. This can be a tough one to get around. The policy is really sort of a graph and that graph has its own memory stack. When the rules that will fire are selected they are placed on the agenda and if they change the memory then all the rules in the policy will evaluate again. This is why there isn't looping the way we think of in BRE, but there is actually looping.

This rather obscure MSDN article gives us some of the answer: How to Iterate ArrayList in Business Rules

To make it work you will need add mscorlib (4.0 if you're on 2010) and your custom class to the class explorer of the business rules editor.

You can see here that rule A always fires (once) and asserts the array list into memory, this causes the policy to be reevaluated.

Rule B fires next and calls MoveNext (in the condition), get current, and update, which cause the policy to evaluate again.

At this point there is not a MyClass instance in the memory and rule C fires.

The upcoming book has a good example of using this, but I found this was the best way to explain the forward chaining nature of the BRE and how rules actually work.

I have been writing a lot for my upcoming BizTalk book and after locking myself away in my office I often turn to The Modern Japanese Garden to relax. Besides being a refreshing reminder why digital books will never have the place in my life that physical books do (the images are amazing), the book helps me relax my mind and focus my thoughts.

I've always appreciated Japanese design aesthetic, but it was not until learning more about Japanese gardening in particular I began to appreciate why. A quote from the book above reads: "As in sumi painting, we should remove, not add, and so stimulate the imagination". I find myself today reviewing many BizTalk designs both of my own and by others and one thing strikes me that made that quote stand out in my mind all day. Odd as it may seem the orchestration canvas can almost be likened to a garden. You run into orchestrations that are like Victorian gardens: highly organized, regimented, packed with components and shapes. You also run into some that are like Japanese gardens: profound in their simplicity. I certainly started in what I would now call the Victorian School of Orchestration Design, but have found my way to the school which now I subscribe; the Japanese school.

In fact the most successful BizTalk solutions I see are remarkable for their simplicity. The same holds for non-BizTalk solutions. I guess what I'm driving at is that no matter what you prefer in garden design, software should be made as simple as possible and large busy orchestrations are generally not that simple despite the visual nature of the design canvas.

Continuing with my HPC adventures I thought I'd give a quick overview of what HPC is and how it works.

HPC is Windows High Performance Computing platform, part of the Technical Computing initiative out of Microsoft. The goal is to create a grid computing platform that is simpler to setup, run, and use. Microsoft has a long and successful track record of making complex computing tasks much easier and it is refreshing to see them continue to bring these resources to bear on Technical Computing.

First things first, what is HPC? HPC is a technology built on top of Windows to handle massive parallel processing. Massive parallel processing is the modern day replacement of supercomputing. HPC is the Microsoft approach to large scale parallel processing and is even compatible with of MPI. I've worked with MPI in the past and used to be quite into Linux, but I never thought grid computing could be made this simple. That said it's still distributed computing, it's still difficult, but now you can focus on the real problems and get the infrastructure down more easily.

In HPC 2008 R2 the cluster is made up of nodes that play different roles.

Head Node

This node is the brain of a cluster. It provides management and scheduling for the cluster.

Broker Node

This node is the node that dishes out request across the cluster and aggregates their responses. This node now runs WCF services that make integrating with HPC much easier.

Compute Node

This is where the rubber meets the road so to speak. These nodes actually carry out processing. You can even use Windows 7 Professional or Enterprise workstations as compute nodes, which I recently blogged about.

As I cover more features of HPC I will show how you can use the product to create node templates and then use iSCSI to boot nodes directly into templates that enlist them in the cluster about as effortlessly as one can imagine (well, better than I could have imagined). The best part is in my labs I was even able to use Windows Storage Server to host these drives.

I've been working on Windows HPC Server 2008 R2 for nearly six months now and I've decided it is time to start posting some of my thoughts and discoveries. Today I feel inspired to write about the Windows 7 capabilities of the platform.

Motivation

I recently modified a BizTalk solution that had been largely functioning as a grid computing platform to start using HPC 2008 R2. The hardware this system ran on is dated at this point, but the solution is CPU intensive (extremely) and even older hardware can still crunch numbers quite well. Moreover part of the move to HPC was because BizTalk is not really made for massively parallel computations (and by massive I've mean tens if not hundreds of processing nodes).

Some of this hardware was still 32bit and I found myself unable to install HPC 2008 R2 on it. I thought I had hit a snag. Then I learned a little more about HPC's new ability to enlist Windows 7 workstations into an HPC cluster as workstation nodes. To me this is one of the most exciting features of a platform that I think is full of exciting features. Basically here's how it works.

An HPC agent is installed on Windows 7 machines that can be either 32bit or 64 bit. The head node (I'll blog about HPC architecture later, but for now it's basically the management node) is then configured with a policy listing the workstations and when they can be used. If this weren't cool already the workstation node agent can also detect if the user is actually using the node and decrease its participation in the cluster automatically if a user begins using the workstation.

So where does this leave us?

If you have large processing jobs that can be parallelized (another blog post) and you have workstations in your enterprise running Windows 7 Professional or Enterprise then you can have your own super computer.

So who all can benefit from this?

Many insurance companies I've worked with process actuarial models or financial statements that require immense calculations. They also tend to have lots of workstation used by the insurance business. They can have their own supercomputer. Trading firms doing technical analysis can also use these features. Universities can provide supercomputing capability to their students with hardware they already have. This last one interests me a lot because computer science students can work with students and researchers from other parts of the university to solve problems that previously were simply not feasible. Large universities have hundreds or thousands of workstations available AND the need for massive computing power.

Introduction

Administrators who haven't worked with BizTalk much rarely seem to appreciate its power and flexibility and almost never respect its demands. Yes I used the words "respect" and "demands", BizTalk requires both. Allow me to demonstrate with a recent example. A client of mine had a rather sophisticated four node SQL cluster with three active nodes and one spare. The nodes were configured to keep failing over in the event of a major catastrophe (that's the good part later). They ran BizTalk's databases (all of them, also against my advice) on one node; along with SSIS for the entire enterprise (again against my advice). Their business had a very much peak and trough pattern with floodgate scenarios common.

At one point this customer also wanted to create a database to hold all incoming messages. It functioned almost like the message box, but without the extensive R&D of Microsoft behind it. I was unable to persuade them against this. That said we did a pretty good job with it. BizTalk itself would log messages to this database and different orchestrations/messaging solutions would process them in progression until they were sent out to other systems when their enrichment was complete. The solution was modular, easily extensible, and worked well even with moderate stress testing.

The Issue

When it was deployed to production everything was running fine until the first major floodgate. This particular incident involved months of backlogged transactions being queued up awaiting processing. Due to strict, appropriate, and well enforced security procedures none of the development team was aware of the sort of volume awaiting our solution. This wouldn't be a minor flood; we were talking Noah and the Ark. Worse still for some other reason, unlike in all other environments, this new database was created on the same SQL instance as the BizTalk databases. This was the real epicenter of the issue and one we never quite got to the bottom of. As mentioned in the beginning of this post, some organizations just don't give BizTalk the proper respect it deserves.

All the servers in the group eagerly began processing tens of thousands of transactions – we think even far more. As BizTalk and the adapters (all spread across hosts and instances on multiple servers) began hammering this new database they were hammering the message box as well; on the same SQL instance. We didn't know it yet, but we had hit the iceberg. The server began blocking connections and many of the database counters that throttle BizTalk also were not getting the access they needed. Soon SQL Server pulled a Cartman: "Screw you guys I'm going home" and shutdown; that is it existed the SQL process. I had never really seen this before, but when the deadlock queue was past 1000 it happened. The failover worked flawlessly and in minutes the second node followed the first Cartman. The third node followed suit and an administrator decided to not let the fourth go on and took the offending node offline. And the band played on. It took a while to clear up the backlog and get everything to process well. This was done mostly by not turning on all the BizTalk servers at once and putting some more aggressive throttling in place.

Conclusion

As we attempted to figure out where our brilliant ideas went astray we tried replicating the issue in other, much less powerful environments which did not have this new database installed on the same SQL instance as the message box. No matter what load we used, we could not replicate it. The solution would slow, it would throttle, but SQL would never pull a Cartman. The moral of the story is despite what your teacher told you: sharing isn't always caring. When it comes to BizTalk databases, especially the message box, do not share them with other databases on the same SQL instance; especially if that database is an endpoint of a BizTalk solution.

I'm working on a new set of training materials for an Administrator's version of Effective BizTalk Server and was recently faced with the choice of what to do about SANs and Shared Storage for the labs. At my office I have a vast range of Dell Precision workstations I use for my labs and testing. I considered some lower end hardware based units, but even they were $1500+ and I probably would have needed several. Worse still because I often teach classes on site for clients, I would now have to lug these iSCSI based SANs all over the place. I also considered open source or lower cost iSCSI software. There is a bit out there it turns out, but I wasn't sold.

Fortunately I found out about Windows Storage Server 2008 R2. Ok, that link goes to the old version, but you can see the Beta on there. The beta, I believe, has already been released to OEMs. Here is where things get interesting; this is not a version of windows you will be able to buy. When you buy certain NAS and SAN products, however, they will be running this version of windows. It's basically a streamlined edition made for storage appliances. If you have an MSDN account you can actually download this edition.

The primary part of interest to me was the iSCSI Target software that is part of this edition. It gets more interesting from there… inside the ISO is an MSIs for installing this iSCSI Target software on an existing Windows Server 2008 R2 installation. REMEMBER THIS IS NOT FOR PRODUCTION NOR IS IT SUPPORTED!!! BEWARE!!! That said it does install extremely easily and it works great. You can create LUNs, Disks, and Targets all from inside the Server Manager console. The disks are VHDs, which is also sort of cool. I installed this on my Hyper-V Host and now all the machines for the lab can share storage. It should be pretty obvious I am aiming for single powerful laptop capable of running an entire BizTalk installation. I will be writing more on this as I do more with it.

On a closing thought I had never really paid much attention to the older Storage Server editions and most of my work with SANs has been… frustrating at best involving the proprietary offerings of several vendors. I think this is the early stages of Microsoft doing some of what it does best: take great ideas that are hard to work with and make them more accessible to the broader technical community. Performance wise Windows Storage Server is able to handle loads like any SAN offerings if sufficient money is spent on the hardware. The features of Storage Server I have not covered are actually really amazing, like data deduplication and management, but those are best left to another MVP.