Gobbledygooks

How to document an integration architecture

2024-03-08T09:00:00+00:00

One could argue that creating good documentation for an integration solution is harder than with other types of software projects. The reason is that integration solutions almost always span over many systems and services and that we often talk about hundreds of integrations, touching almost every part of your architecture.

Keeping track of such a complex system is literally impossible without the right documentation. 🤯

An example of bad documentation ...

Basic principles of good documentation for an integration architecture

1. Visualizations and diagrams are more important than textual documentation.

Textual documentation has its place and is important. But when it comes to trying to describe complex scenarios, visualizations in the form of architectural diagrams are easier to quickly understand and therefore arguably the most important part of your documentation.

2. Focus on the high-level diagrams.

As developers, we’re good at documenting all the little technical details in our solutions. But we often document things we end up looking up in the live code and configurations. Focus on high-level documentation, the overall picture, and how the different parts of your solutions relate to each other.

3. Make it readable for everyone.

UML and complex notations are great - if you understand them fully. The C4 model formalized the idea of a much simpler notation, based on boxes and arrows - easier to understand and work with.

Use a dead simple notation that everybody can write and read.

4. Make it useful for everyone.

Besides a simple notation, the C4 model describes a few different diagrams with increasing levels of detail. Using different levels makes it possible to drill down from an overview into more detailed diagrams. Similar to zooming in and out of a digital map.

Utilizing the levels and the simple notation in the C4 model, diagrams can have information useful for non-technical business users and describe large parts of architecture without cluttering them down with too many details.

The C4 levels in an integration context

Let’s look at the three levels that make up good integration documentation.

C4 model diagrams levels with increasing detail.

Level 1 (Context) - The Overall picture

For quickly understanding an integration architecture the most important part of documentation is the high-level documentation. It shows all the systems and the data-flows between them. Not the nitty-gritty details, just what systems are involved and what data is transferred between them.

In smaller architectures, it’s possible to describe all systems and data-flows in a single level 1 diagram. In large ones, we need to choose a context to avoid too large and hard-to-read diagrams. In this example, the context is the e-Commerce part of the architecture.

Level 1, Overall picture example - https://my.revision.app/diagram/6idZKQTeD8C9.

It’s important to keep these diagrams as simple as possible. It’s often tempting to include more details than necessary. Try to skip everything besides concrete systems and what type of data that’s transferred. The reason for this is that these diagrams otherwise get too complicated, hard to work with and hard to understand.

This part of the documentation is important. This is information that often is hard to understand by looking at the code and configuration.

This is also the part of the documentation that business users will understand. As we all know, getting the business users involved is critical for making the right decisions and in the end creating solutions that actually help the business in the right way.

Even though most people would agree that the overall picture is important to have documented, I’ve found that this is the part most developers tend to skip.

Level 2 (Container) - The Integration Processes

It’s hard to define an integration. In the overall picture product data are moved between several systems. Is the integration the part that moves the products between InRiver PIM and Dynamics 365 PO? Or is the integration the overall process that makes sure the products end up in all the destination systems?

Leaving that discussion aside we can agree that the purpose of an integration is to support some sort of business process. In this case, the handling of product master data.

Documenting how we support a business process is the second level of diagrams.

The difference between this and the level 1 diagram is that we’ll zoom in to a specific area and that we’ll here include more details - things that we didn’t want to include in the overall picture to keep it easy to understand. In the example, we’re showing that product master data actually is distributed using a product service. We’re also including other details we didn’t show in the level 1 diagram.

Level 2, Integration Process example - https://my.revision.app/diagram/bwoG8AXEhhxD. Shows how product master data flows between systems.

Again this is information that is hard to get from reading code and configuration and that is important for getting the business users to understand what we’re actually building.

It’s important to try and keep purely technical details out of this diagram and try and focus on more business rules like information. Information that is useful for a business user.

Example of information we might include on level 2;

what’s triggering the transfer
duplication checks
conversions
transformations of data, and so on

Level 3 - The Integration Details

This is probably the diagram that needs the least explanation. This is the part I often find well-documented and maintained. Here we document all the technical components and implementation details.

This is however also the part that doesn’t always require that much documentation. Much of the details at this level can easily be found in code and configuration and it’s easy to spend time documenting things that in the end developers will look up code and configuration instead.

Level 3, Integration Details example - https://my.revision.app/diagram/RmXsxDtuAEHy. Shows a detailed view of how product master data in moved between one system to another.

Build and generate environment specific binding files for BizTalk Server using Team Foundation Build Services

2015-03-31T11:00:00+00:00

As most know a BizTalk Solution has two major parts – its resources in form of dlls, and its configuration in form of bindings. In an previous post I described how to build and pack resources into an MSI using Team Foundation Server (TFS) Build Services. Managing the configuration in a similar way is equally important. So let’s see how we can build environment specific binding files using TFS Build Services and some config transformations syntax goodness!

Creating a simple binding example for development and test environment

Let’s start with a simple example of a simple binding with a receive port, receive location and a send port for two different environments - one called “Test” and one “Production”.

Port type	Name	Destination path in Test	Destination path in Production
Receive Port	BtsSample_ReceivePort_A	N/A	N/A
Receive Location	BtsSample_ReceivePort_A_Location (File)	C:\Temp\In\*.xml	C:\Temp\In\*.xml
Send Port	BtsSample_SendPort_A (File)	C:\Temp\TEST\Out\%MessageID%.xml	C:\Temp\PROD\Out\%MessageID%.xml

As one can see there’s a small difference between the send ports destinations paths in Test and Production.

Exporting a binding template

Next we’ll create a binding template. The binding template will hold all information that is shared between the different environments. This is achieved this by an ordinary export of the application binding from the BizTalk Administration Console - as you’ve probably done many times before.

Creating environment specific bindings using web.config Transformation Syntax

The Web.config Transformation Syntax is feature that showed up in Visual Studio 2010 and is often used to transform app.config and web.config files between different versions and environments – but it will of course work on any type of configuration file. Including BizTalk binding files!

So for each environment we’ll then create an environment specific config file that only contains the values that differs between the template and the values for that environment. We use the Web.config Transformation Syntax to match the nodes and values that we like to update in the template.

Below is the Test environment specific file matching the send port and replacing the value with the value specific for Test.

The Production specific file also matches the send port but with a different value for the Destination path.

Using MSBuild to execute the transformation

As part of the Visual Studio installation an MSBuild target is installed for executing the transform. The target is installed into the standard MSBuild Extensions Path which usally mean something lilke C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.targets depending a Visual Studio version etc.

Finally we’ll add a small .proj file to pass some parameters to the MSBuild process. We need to tell the process what file to use as template and what different environment specific files we like to use.

Next we can kick of MSBuild and point it to the cretated proj file. C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild build.proj.

Voila! MSBuild has performed our transformation and created two complete environment specific binding file by combining the template with the specific environment files - one of test and one for production.

Generating the file using TFS Build Services

Start by setting up a TFS build definition as described in previous post on Generating BizTalk MSI using TFS Build Services.

We then need to add the $(DestinationPath) property to our Destination path to make sure the outputted binding files are written to the same path as the rest of the resources.

Once we have our build template all that’s needed is to add the build.proj file to the files to compile as shown below.

When finally running the build our two complete binding files are written to the deployment area and ready for installation!

Build and generate MSI for BizTalk Server using Team Foundation Build Services

2015-02-26T11:00:00+00:00

Using a build server and leveraging continuous integration is good practice in any software development project. The idea behind automated build and continuous integrations is to have a server that monitors one’s source code repository and builds the solution as changes occur. This separate build activity alone will ensure that all artifacts are checked in and that a successful build doesn’t depend on any artifacts or settings on the development machines.

Today build servers do a lot more as part of the build - the build process usually involves execution of tests, labeling the source as well as packing the solution into a deployable artifact.

In this post we’ll see how a build process can be achieved using Team Foundation (TFS) Build Services, building a BizTalk project that results in a deployable MSI artifact.

TFS Build Services

TFS Build services is a component that is part of the standard TFS install media. Each TFS build controller controls a number of “Build Agents” that will perform the actual build process. For each solution to build one has to define its process. These processes are described in a “Build Template” that tells the agent what steps to go through and in what order.

“Build Templates” in TFS Build Services are defined using Visual Studio. The image below shows a build template accessed through Visual Studio Team Explorer.

Major steps in a build template

As one creates a new build template for a solution one has to go through the following major steps:

1. Define a trigger

Decides what should trigger the build. Should it be triggered manually, should it be a scheduled build or should it be triggered at a check-in of new code?

2. Source Setting

This will tell the build process what part of the source tree the build template is relevant for. When queueing a new build this is the part of the source tree that will be downloaded to the staging area. It also tells the build services where on disk the source should be downloaded to.

3. Process

This is where all the steps and activities that the build service should perform are defined. Team Foundation Build Services comes with a number of standard templates and custom ones can be added. In this post we’ll however stick with the default one.

Build your first BizTalk solution

Building BizTalk Server solution using TFS Build Services is straight forward.

In this post I will use this sample BizTalk solution. After checking it into Team Foundation Source Control (I’ll use TFS Source control in this post but it’ll work similarly using Git) I’ll create a new build template for the solution. All that’s needed to change is the MsBuild platform setting property, so we’re using x86 when executing MsBuild as shown below.

After queuing a build we can in the TFS Build Explorer see a successful build!

We can also download the output from the build where we can see all our build artifacts!

Using BtsMsiTask to create a MSI as part of the build

So far so good, but we started the article by saying that what we wanted was a deployable artifact. In the case of BizTalk this means a BizTalk MSI. Let’s see what we need to change to also have the build process create a MSI.

1. Install BtsMsiTask

Download and install BtsMsiTask. This will install a MsBuild task for generating the MSI.

2. Add a MsBuild project file

Add a MsBuild project file (build.proj) to the solution

The project file will tell the BtsMsiTask process what artifacts to include. Add the created project file to the solution and check it in as part of the solution.

3. Add the MsBuild project file to the TFS build template

Add the created MsBuild project file to the TFS build template by adding it to the list of projects to build.

After another successful build we can see that we also created a MSI as part of the build!

Adding build information to the MSI

File name

As we can see the MSI we just created ended up with the default BtsMsiFile name that is a combination of the BizTalk application name property and the current date and time. Wouldn’t it be nice of we instead could the build number as part of the name? BtsMsiTask has an optional property called FileName that we for example can set to <FileName>$(TF_BUILD_BUILDNUMBER).msi</FileName>

Source location

When installing the artifact to BizTalk Server we can see that the source location property in the BizTalk Administration Console is set to where the artifact was built on the staging area.

It’d be nice to also have information about what build that produced these artifacts. This will give the required information to know exactly what builds that are used for all the installed artifacts.

We can change what is set in the source location by using the SourceLocation property of BtsMsiTask <SourceLocation>c:\$(TF_BUILD_BUILDNUMBER)</SourceLocation>

So after setting the property as below, queue another build, reinstall using the MSI and we’ll get the following result with the build number in the source location property.

And finally this is the MsBuild project file we ended up with in our example.

Calling an on premise Microsoft Dynamics CRM using BizTalk and Active Directory Federated Security (ADFS)

2014-10-17T11:00:00+00:00

Federated security is a great way of accomplishing single-sign-on (SSO) security for your applications. It’s also a technique that is becoming increasingly relevant as things move to the cloud and we’re starting to get more hybrid situation with some applications on premise and some in the cloud.

Active Directory Federated Security (ADFS) is an implementation of federated security and is used by a number of Microsoft Applications, Microsoft Dynamics CRM being one of them.

Windows Communication Foundation (WCF) has a few techniques to simplify federated security communication and this post will show an example of using Microsoft BizTalk Server and WCF to communicate with an ADFS secured CRM installation.

What is federated security?

Federated security at its core is pretty simple. In our scenario BizTalk Server (client) wants to login in and authenticate itself with the CRM system.

Traditionally the CRM system then had to manage the credentials for the client and verify these as login happened. There a number of drawback to this, the main one being that it doesn’t scale that well when we’re getting many separated systems that need access to each other as login information and login logic is spreads out across a number of systems.

When using federated security each part instead chooses to trust a common part (in this case the ADFS and AD), and as long as someone provide a token that can be validated with the trusted part, the CRM system will trust that it already has been authenticated and that everything is ok.

Authentication and requesting token
Authentication against AD
Login authenticated
ADFS token response
ADFS token based authentication
Response from CRM

So basically a federated security model allows for separating all authentication and authorization out to a separate system.

As mentioned ADFS is just an implementation of federated security were Active Directory acts as the main repository with a Security Token Service implementation on top of it.

BizTalk and ADFS

As BizTalk has great WCF support we can use the WCF stack to handle all of communication with ADFS and CRM. But it does involve a fair bit of configuration. BizTalk and Visual Studio will help in most mainstream WCF scenario where one can point Visual Studio at the WSDL location and a basic binding file is generated for us. However, in the case of and ADFS based WSDL this will just result in an empty binding file that doesn’t help much.

Lots of projects I’ve seen makes the decision at this point to use custom code and create a facade to solve authentication. As Microsoft Dynamics CRM comes with a nice SDK including a C# library to handle authentication is easy to understand how one would end up using that. The problem is however that this creates another code base that again needs to be maintained over time. One could also argue that using custom code that is called by BizTalk further complicates the overall solution and makes it harder to maintain.

So let’s configure the authentication from scratch using Windows Communication Foundation.

Choosing the right WCF binding

First thing to do is to choose the right WCF binding. Let’s create a WCF-Custom Static Solicit-Response send port and choose the ws2007FederationHttpBinding.

Adding the Issuer

First thing we need to add is information on how to connect to the Issuer. The Issuer is the one issuing the ticket, in our case that’s the ADFS.

First we need to add information about the address of the Issuer. The WSDL tells us that the mex endpoint for the ADFS server is located at https://adfs20.xxx.yy/adfs/trust/mex.

Browsing the WSDL for the ADFS server shows a number of different endpoints. Which one to use depends on what kind of authentication being used when requesting the token. In our case we’re using a simple username and password so we’re using the usernamemixed endpoint (https://adfs20.xxx.yy/adfs/services/trust/2005/usernamemixed).

Secondly we need to add information about the binding and the binding configuration for communication with the ADFS service.

What this basically means is that we need to add information to a second, or an inner, binding configuration. The BizTalk Server WCF configuration GUI doesn’t provide a way to set this so the only way is to configure this is to use one of the relevant configuration files (“machine.config” or the BizTalk config) and ad a binding manually.

<bindings>
 <ws2007HttpBinding>
  <clear/>
  <binding name="stsBinding">
   <security mode="TransportWithMessageCredential">
    <transport clientCredentialType="None"/>
    <message clientCredentialType="UserName" establishSecurityContext="false"/>
   </security>
  </binding>
 </ws2007HttpBinding>
</bindings>

Once this is setup we can point our BizTalk Server WCF configuration to the correct URL and reference the WCF inner binding we just configured.

Finally we need to provide the username and password to authenticate ourselves to the ADFS server.

We now have communication setup to the ADFS service and should be able to get a valid ticket that we then can use to authenticate ourselves to the CRM system!

We now however also need to provide information on how to connect to the actual CRM system.

Configure communication to CRM

The rest is easy. Let’s start with adding the URL to the end service we want to call. As with any other service call we’ll also add the SOAP Action Header that in this case is the Update service (http://schemas.microsoft.com/xrm/2011/Contracts/Services/IOrganizationService/Update) of the OrganizationService service.

As out service also uses SSL for encryption we need to tell the binding to use TransportWithMessageCredentials.

Establishing a Security Context – or not

Finally there is a little tweak that is needed. WCF supports establishing a Security Context. This will cache the token and avoid asking the STS for a new token for each call to the CRM system. BizTalk Server however doesn’t seem to support this so we need to turn it off.

Conclusion

Understanding federated security is important and will become increasingly important as we move over to systems hosted in the cloud – federated security is the de facto authentication standard used by hosted systems in the cloud. Avoiding custom code and custom facades is a key factor in building maintainable large scale BizTalk based systems over time. BizTalk has great WCF support and taking full advantage of it is important to be able to build solutions that easy to oversee and possible to maintain not just by those familiar and comfortable in custom code.

Getting digital certificates right in BizTalk using a third party root certificate in combination with client certificate security and WCF BasicHttp

2014-08-28T22:00:00+00:00

Digital certificates and asymmetric security is notoriously hard to get right in a Windows environment. Getting it right in a BizTalk context isn’t exactly easier.

In this scenario a BizTalk Server act as a client and communicates with a service over https. The service also uses a client certificate for client authentication.

Long story short

Third party root certificates always needs to be places under “Third-Party Root Certification Authorities” or directly under the “Trusted Root Certification Authorities” folder on Local Machine level in Windows. When however also configuring the “WCF-BasicHttp” adapter to also use client certificate authorization the BizTalk Administration console requires the thumbprint id of a specific server certificate (in addition to the client certificate thumbprint). This makes the runtime also look for the for the public certificates under “Trusted People” folder and causes an error if we don’t also place it that folder.

In the end this requires us to add the public root certificate in two different places.

Server certificate

Let’s start by getting the server certificate right.

After configuring everything in BizTalk using a standard WCF-BasicHttp port and selecting Transport security I encountered the following error message.

A message sent to adapter “WCF-BasicHttp” on send port “SP1” with URI “https://skattjakt.cloudapp.net/Service1.svc” is suspended.

Error details: System.ServiceModel.Security.SecurityNegotiationException: Could not establish trust relationship for the SSL/TLS secure channel with authority ‘skattjakt.cloudapp.net’. —> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

The error message is pretty straightforward: Could not establish trust relationship for the SSL/TLS secure channel with authority.

The first thing that happens when trying to establish SSL channel is that a public server certificate is sent down the client for the client to use when encrypting further messages to the server. This certificate is validated so it hasn’t been revoked, that it’s Valid to date hasn’t passed, that the Issued to name actually matches the services domain and so on.

But to be able to trust the information in the certificate it needs to be issued by someone we trust, a certificate authority (CA).

If we take an example of a request to Google we actually don’t trust the information in the Google server certificate, neither do we trust the intermediate certificate they use to sign their public server certificate. The root certificate that issued the intermediary Google Certificate is however one of the preinstalled trusted certificate authorities in Windows.

What authorities and certificates to trust is in Windows based on what certificates exists in the Certificate Store under the Trusted Root Certificate Authorities folder.

In our case the service didn’t use a certificate from one of the trusted authorities but had based their certificate on a root certificate they created themselves.

Further the Certificate Manager in Windows has three different levels: “Local Machine”, “Service” and “Current User”. The top level is the “Local Machine” and certificates added on this level are available for all users. “Service” and “Current User” are more specific and only available for specific services and users. From a BizTalk perspective it’s important to place the certificate so it’s accessible for the user running the BizTalk host instance.

So after requesting the used root certificate and placing it in the trusted authorities folder for the Local Machine we’re able to successfully establish an SSL session!

Client certificate

As the server however required a client certificate for authorization I reconfigured the send port to use Certificate as client credential type.

The BizTalk Administration Console then requires one to enter the thumbprint of the private client certificate to use. When browsing for picking the client certificate the console will look for certificates to choose from in the “Personal” folder on the “Current User” level. So for the certificate to show up one has to add the client certificate to the “Personal” folder running as the user that eventually will hit the browse button in the console. Adding it only to the “Personal” folder of “Local Machine” will not make it show up in the console. As the “Current User” level also is separate for each user it’s very important to add it to the “Personal” folder for the user that eventually will run the BizTalk process as this user otherwise won’t find the certificate at runtime. In this case just pasting the thumbprint id from the certificate will work fine.

When selecting Certificate client credential type the BizTalk Administration console also requires one to pick what public server certificate to use - even though we still just want to use the same root certificate as just added to the trusted store on machine level ..? When locating server certificates to display the console will look in the “Other People” folder on “Local Computer” level. So for making our root certificate show up in the console we also have to this to this folder. It turns however out that when having a pinpointed specific server certificate the BizTalk runtime will throw an error if the server certificate is not placed in the “Other People” folder. Likewise will an error we be thrown if the certificate is placed only in one of the trusted authorities folders.

A message sent to adapter “WCF-BasicHttp” on send port “SP1” with URI “https://skattjakt.cloudapp.net:444/Service42.svc” is suspended. Error details: System.InvalidOperationException: Cannot find the X.509 certificate using the following search criteria: StoreName ‘AddressBook’, StoreLocation ‘LocalMachine’, FindType ‘FindByThumbprint’, FindValue ‘70A9899E6CF89B014E6195ADE6E1BA12BEA58728’.

So in this case we need to add the public CA certificate in two different places for the communication to work.

Frankly I don’t see the point of having to point out a server certificate at all in this case - all I want is to configure what client certificate to use for authorization and the runtime to validate the server certificate against all CA I have in the trusted folders.

Why full NuGet support for BizTalk projects is important!

2014-06-03T09:33:09+00:00

Let’s start with a summary for those who don’t feel like reading the full post.

Using NuGet to handle BizTalk dependencies for shared schemas, pipeline components and so on works fine today.

As .btproj files however aren’t supported by NuGet (as shown in this pull request) and are not in the current white list of allowed project types Package Restore will not work (issue closed as by design here).

Not having Package Restore of course is a problem as one now is forced to check in all packages as part of the solutions, something that in the end leads to bloated and messy solutions.

So please reach out to your Microsoft contacts and let’s get this fixed!

NuGet

As most people know NuGet is the package management solution from Microsoft for .NET. It started off as an initiative to further boost open source within the .NET community and NuGet packages uploaded to the NuGet.org platform are open and available directly within Visual Studio through the NuGet add-in. Currently there are well over 20 000 open packages for everyone to download and use.

Lately there has however been lots of discussions within the community to use NuGet as a package manager for internal shared resources as well (by Hanselman and others). Solutions like MyGet allows for private NuGet feeds – only available to those within your organization but still levering all the ease and control offered by NuGet.

Using NuGet for references has a number of of advantages:

Communication
All available resources are directly visible in Visual Studio and when updates to a used library is a available a notification is shown. No more spam mails about changes and never read list of available libraries.
Versioning
A NuGet package has it’s own versioning. This is useful as it isn’t always optimal to change the dll version, but by using the NuGet package version one can still indicate that something has changed.
As you also reference a specific version of a NuGet package from your solution you always have full control of exactly what version you’re targeting and where to find the built and ready bits.
Efficiency
When starting to work on a project with many references one first have to get the source code from source control for the references, build these (hopefully in the right version … hopefully you have your tags and labels in order …) until all the broken references are fix.
With NuGet references this just works straight away and you can be sure you get the right version as the resource isn’t the latest from source control but the actual built dlls that’s part of the referenced NuGet package.

NuGet Feeds

As mentioned NuGet feeds can be public or private. A NuGet feed is basically a RSS feed with available resources and versions of these. A feed and a NuGet server can be a hosted web based solution or something as simple as a folder where you write your NuGet packages to. The NuGet documentation covers these options in depth.
The point being that creating your own private NuGet feed is very simple!

So if you haven’t already realized it by now - NuGet is not only a great tool to manage all public external dependencies but can add a a lot a value for internal references as well.

Couple of relevant NuGet features

NuGet Package Restore
NuGet Package Restore enables NuGet to download the used referenced package from the package area. The goal is to avoid having to check in the actual references in source control as this will bloat the version control system and in the end the create a messy solution.
NuGet Specification (nuspec) metadata token replacements
All packages are based on a nuspec file that dictates the version, package description and other meta information.
NuGet has the capability to by using replacement tokens (such as $version$ for example) read some of this information from the AssemblyInfo files.
This is far from a critical feature but nice to have and avoid having to repeat oneself and have the same information in a number of places.**

BizTalk and NuGet?

A typical BizTalk solution has a number of shared resources such as shared schemas, shared libraries and pipeline components. As the resources usually are shared between a number of project they often live in a separate development cycle. So when opening a BizTalk project with such resources it’s not only a lot of work getting the referenced code and building the references, there’s also this nagging feeling that it _might not be in the right version _and that the source might have changed since the first time the reference was added.

Another reference issue occurs when using a build server for a solution with references. As the solution has a dependency to the referenced project one has to make sure not only that the main solution is fetched to the build workarea by the build server, but also that all the referenced project are fetched from version control – and again, hopefully the latest version in the attended version …
This kind of works using TFS Build Service and common TFS Source Control. If however one’s using Git and have resources in separate repositories this becomes impossible as TFS Build Service currently only supports fetching a single repository per build definition to the build workarea …
(This issue does not apply for TeamCity that has a number of different options for dependency management)

All the these issues are actually solved when using NuGet references instead of traditional references as we can be sure we’re getting the packaged dlls as part of the NuGet package in the version that one referenced and not the latest checked in version.
A NuGet reference also makes things a bit easier when it comes to managing the workarea for the TFS Build Service as one only have to sure the NuGet package is available (either as checked in as part of the solution or by using Package Restore).

But …

NuGet doesn’t support BizTalk projects!

As disused here NuGet currently doesn’t support .btproj files. As BizTalk project files are are basically a standard .NET project file with some extra information a two line change in the NuGet whitelist is needed as in this pull request.

So the main issue it that by not having full support of .btproj files Package Restore won’t work and we’re for now force to check in all the NuGet packages as part of our solutions.
An other minor issue is that the token replacement feature also doesn’t work. I also think that if we could actually get full BizTalk support we’d see more BizTalk specific open shared packages for things like useful libraries and pipeline components.

Call for action: Reach out to the NuGet Team or other Microsoft connections and let’s get those two lines added to the white list!

Talking Application Lifecycle Management and BizTalk in Gothenburg

2014-02-22T14:13:29+00:00

Back in February I and my college Robin did an presentation on ALM and BizTalk for the BizTalk User Group in Stockholm. In the two hour presentation we talked about things like:

Pros and cons on distributed version control handling systems and BizTalk (more specially Git).
Identifying the right level of automated test – what are the differences between integration and unit tests and how and what should one use when in a BizTalk context.
Using NuGet and BizTalk to handle dependencies and packages. How can we use the NuGet infrastructure to handle all BizTalk dependencies and also distribute artifacts like pipeline components within our company and teams?
How, in our opinion, are Continuous Integration and Delivery best handled when developing with BizTalk. In the presentation we’re looking at everything from build servers to how to handle BizTalk automated builds and packing with minimal amount of work.

So, now we doing the presentation all over again for the newly started BizTalk User Group in Gothenburg the 24 of Mars. The presentation will be in Swedish and packed with demos.

Hope to see you there!

Export BizTalk Server MSI packages directly from Visual Studio using BtsMsiTask

2014-01-14T10:42:25+00:00

Getting a full Continuous Integration (CI) process working with BizTalk Server is hard!

One of the big advantages in a working CI process is to always have tested and verified artifacts from the build server to deploy into test and production. Packaging these build resources into a deployable unit is however notorious hard in BizTalk Server as a Visual Studio build will not provide a deployable artifact (only raw dlls). The only way to get a deployable MSI package for BizTalk Server is to first install everything into the server and then export – until now.

Why Continuous Integration?

Continuous Integration is a concept first described by Martin Fowler back in 2006. At its core its about team communication and fast feedback but also often leads to better quality software and more efficient processes.

A CI process usually works something like the above picture.

A developer checks in code to the source control server.
The build server detects that a check in has occurred, gets all the new code and initiates a new build while also running all the relevant unit tests.
The result from the build and the tests are sent back to the team of developers and provides them with a up to date view of the “health” in the project.
If the build and all the test are successful the built and tested resources are written to a deploy area.

As one can see the CI build server acts as another developer on the team but always builds everything on a fresh machine and bases everything on what is actually checked in to source control – guaranteeing that nothing is build using artifacts that for some reasons is not in source control or that some special setting etc is required to achieve a successful build.

In step 4 above the CI server also writes everything to a deploy area. A golden rule for a CI workflow is to use artifacts and packages from this area for further deployment to test and production environments – and never directly build and move artifacts from developer machines!
As all resources from each successful build is stored safely and labeled one automatically achieves versioning and the possibility to roll back to previous versions and packages if needed.

What is the problem with CI and BizTalk?

It is important to have the build and feedback process as efficient as possible to enable frequent checkins and to catch possible errors and mistake directly. As mentioned it is equally as important that the resources are written to the deploy area are the ones used to deploy to test and production so one gets all the advantages with versioning and roll back possibilities etc.

The problem with BizTalk Server is however that only building a project in Visual Studio does not gives us a deployable package (only raw dlls)!

There are a number of different ways to get around this. One popular option is to automate the whole installation of the dlls generated in the build. This not only requires a whole lot of scripting and work, it also requires a full BizTalk Server installation on the build server. The automated process of installation also takes time and slows down the feedback loop back to development team. There are however great frameworks as for example the BizTalk Deployment Framework to help with this (this solution of course also enables integration testing using BizUnit and other framework).

Some people would also argue that the whole script package and the raw dlls could be moved onto test and production and viewed on as a deployment package. But MSI is a powerful packaging tool and BizTalk Server has a number of specialized features around MSI. As MSI also is so simple and flexible it usually the preferred solution by IT operations.

A final possibility is of course to directly add the resources one by one using BizTalk Server Administration console. In more complex solutions this however takes time and requires deeper knowledge into the solution as one manually has to know in what order the different resources should be added.

Another option in BtsMsiTask

Another option is then to use BtsMsiTask to directly generate a BizTalk Server MSI from the Visual Studio build and MsBuild.

The BtsMsiTask uses same approach and tools as the MSI export process implemented into BizTalk Server but extracts it into a MSBuild task that can be directly executed as part of the build process.

BtsMsiTask enables the CI server to generate a deployable MSI package directly from the Visual Studio based build without having to first install into BizTalk Server!

Presenting on “Efficient system integration documentation” at BizTalk User Group Sweden

2012-10-14T20:02:45+00:00

I’ve talked a lot about efficient documentation previously on this blog, both when in comes to splitting the documentation up in different layers as here, but also in relation to tools BizTalk Documenter and BizTalk Web Documenter for automatically generation technical documentation as here.

Last week I however had the pleasure to present on the BizTalk User Group Sweden meeting and got to talk all documentation for a whole hour! You can find the PPT from the meeting here.

I ended the presentation by presenting the ten commandments for efficient system integration documentation:

I. Thou shall not manually document anything that can be automatized!

II. Thou shall keep it simple & make it look nice

III. Thou shall use a wiki based platform

IV. Thou shall use pictures whenever appropriate

V. Thou shall have well defined guidelines for your documentation

VI. Thou shall have a well defined target audience for your documentation

VII. Thou shall document continuously in your project

VIII. Thou shall have a common vocabulary and common icons defined

IX.Thou shall test your documentation with target audience

X. Thou as the developer of an integration should document it

Reborn as BizTalk Web Documenter!

2012-09-30T20:27:40+00:00

About a year ago I published a project that helps automating documentation for BizTalk Server – much like BizTalk Documenter but with the difference that this tools instead generates a dynamic web site. I’ve called in Config Explorer, a name I knew was sh*t from the start but the best I could come up with at the time.

After a while however, when enough people told me that they also thought the name was crap I decided to change it to BizTalk Web Documenter. I’ve also moved the whole project to GitHub.

I’ll try and spend a bit more time on the project than I have the last year and will update the documentation, wiki and roadmap as well as implement a few new features.

I hope you’ll help me – fork the project today!

Using BizTalk Web Documenter as a new way of documenting your BizTalk solutions

2011-09-02T11:01:22+00:00

2012-09-30: I’ve renamed the whole project and moved it to GitHub. Hope to see you there!

As most BizTalk developers/architects you have probably been in a situation were documentation of your integration solution has been discussed. And you probably also been just as happy as me when you first found BizTalk Documenter. BizTalk Documenter is a great tool for extracting all that configuration data that is so vital to BizTalk and create a nice technical documentation of it.

There are of course several levels to documentation in BizTalk Server based solution were the technical information is at the lowest level. Above that you would then have you integration process descriptions and then possible another level above that where you fit the integration processes into bigger business processes, and so on – but let’s stick to the technical documentation for now.

BizTalk Documenter does a nice job of automating the creation of technical documentation, instead of basically having to all that configuration job a second time when documenting. Doing technical documentation by hand usually also breaks down after a while as solution setup and configuration changes a lot and the work is really time consuming. And as documentation that isn’t up to date frankly is completely worthless people soon looses interest in both reading and maintaining the technical documentation.

Using BizTalk Documenter however has a few problems …

The tool generates either a CHM file or a word file with documentation. As more and more organizations however move their documentation online new output formats are needed.
The latest 3.4 versions of the tools throws OutOfMemory Exceptions on bigger solutions.

BizTalk Config Explorer Web Documenter

As we started trying to find ways to solve the issues above we had a few ideas we wanted the new solution to handle better.

Web based documentation
A web based documentation is both much easier to distribute and more accessible to read – sending a link and reading a web page rather then sending around a CHM file.
Easy way of see changes in versions
Sometimes one wants to see how things have changed and how things used to be configured. The tools should support “jumping back in time” and see previous versions.

Check out an an example of a Config Explorer yourself.

Getting started

Config Explorer is a open source tool published on ~~CodePlex GitHub~~ - a good place to start to understanding the workflow of creating documentation and to download the tool itself.

Contribute!

This release is a 1.0 release and there will are many features we haven’t had time for, but a few of them we’ve planned for future releases. Unfortunately there will however also be bugs and obvious things we’ve missed. Please use the ~~CodePlex GutHub~~ Issue Tracker to let us know what you like us to add and what possible issues you’ve found.

I’ll of course also check for any comments here! Winking smile

Update to BAM Service Generator

2011-06-03T05:58:29+00:00

I have previously written about how to generate a types set of WCF services to achieve end-to-end BAM based tracking – making BAM logging possible from outside potential firewalls and/or from non .NET based clients.

In that post I reference a open source tool to generate the services based on the BAM definition files to be able to make types BAM API calls. I called the tool BAM Service Generator and it’s published on CodePlex.

The first version of the tool only made it possible to make simple activity tracking calls, even missing the option to defining a custom activity id. All advanced scenarios like continuation, activity and data reference or even updating a activity wasn’t possible. That has now been fixed and the second version of the tool enables the following operations.

LogCompleteActivity
Creates a simple logging line and is the most efficient option if one only want’s to begin, update and and end an activity logging as soon as possible.
BeginActivity
Start a new activity with a custom activity id
UpdateActivity
Updates a already started activity and accepts a types object as a parameter. The typed object is of course based on what’s has been defined in the BAM definition file. Multiple updates can be made until EndActivity is called.
EnableContinuation
Enables continuation. This is used if you have multiple activity loggings that belong together but you can not use the same activity id. This enables you to tie those related activities together using some other set of id (could for example be the invoice id).
AddActivivityReference
Add a reference to an another activity and enables you to for example display a link or read that whole activity when reading the main one.
AddReference
Adds a custom reference to something that isn’t necessarily an activity. Could for example be a link to a document of some sort. Limited to 1024 characters.
AddLongReference
Similar to AddReference but enables to also store a blog of data using the LongRefernceData parameter – limited to 512 KB.
EndActivity
Ends a started activity.

Let me know if you think that something is missing, if something isn’t working as would expect etc.

End-to-end tracking using BAM services and the BAM Service Generator tool

2011-01-18T14:48:36+00:00

An integration between systems can often be view upon as a chain of system and services working together to move a message to its final destination.

One problem with this loosely-coupled way of dealing with message transfers is that its hard to see if and where something has gone wrong. Usually all we know is that the sending system has send the message, but the final system never received it. The problem could then be within any of the components in between. Tracking messages within BizTalk is one thing but achieving a _end-to-end tracking _within the whole “chain” is much harder.

One way of solving the end-to-end tracking problem is to use BAM. BAM is optimized for these kind of scenarios were we might have to deal with huge amount of messages and data – several of the potential problems around these issues are solved out-of-the-box (write optimized tables, partitioning of tables, aggregated views, archiving jobs and so on).

And even though BAM is a product that in theory isn’t tied to BizTalk its still a product that is easier to use in the context of BizTalk than outside of it due tools like the BizTalk Tracking Profile Editor and the built in BAM-interceptor patterns within BizTalk. It is however fully possible to track BAM data and take advantage of the BAM infrastructure even outside of BizTalk using the BAM API.

The BAM API is a .NET based API used for writing tracking data to the BAM infrastructure from any .NET based application. There are however a few issues with this approach.

The application sending the tracking data has to be .NET based and use the loose string based API. So for example writing to the “ApprovedInvoice” activity below would look something like this – lots of untyped strings.

public void Log(ApprovedInvoicesServiceType value) 
{ 
    string approvedInvoicesServiceActivityID = Guid.NewGuid().ToString();
    
    var es = new DirectEventStream("Integrated Security=SSPI;Data Source=.;Initial Catalog=BAMPrimaryImportw", 1);

    es.BeginActivity("ApprovedInvoices", approvedInvoicesServiceActivityID);

    es.UpdateActivity("ApprovedInvoices", approvedInvoicesServiceActivityID, 
        "ApprovedDate", "2011-01-18 12:02", 
        "ApprovedBy", "Richard", 
        "Amount", 122.34, 
        "InvoiceId", "Invoic123");
 
    es.EndActivity("ApprovedInvoices", approvedInvoicesServiceActivityID); 
}

One way of getting around problem with the usage of strings is to use the “Generate Typed BAM API tool”. The tool reads the BAM definition file and generates a dll containing strong types that corresponds to the fields in the definition. By referencing the dll in the sending application we can get a strong typed .NET call. The fact that this still however requires a .NET based application remains.

The obvious way to solve the limitation of a .NET based client - and to get one step closer to a end-to-end tracking scenario - is of course to wrap the call to the API in a service.

BAM Service Generator

BAM Service Generator is very similar to the Generate Typed BAM API tool mentioned above with the difference that instead of generation a .NET dll it generates a WCF service for each activity.

c:\Tools\bmsrvgen.exe /help 
	
BAM Service Generator Version 1.0 

Generates a WCF service based on a BizTalk BAM definition file.

 -defintionfile: Sets path to BAM definition file.
 -output: Sets path to output folder.
 -namespace: Sets namespace to use.

Example: bmsrvgen.exe -defintionfile:c:\MyFiles\MyActivityDef.xml 
-output:c:\tempservices -nampespace:MyCompanyNamespace

The BAM Service Generator is a command line tool that will read the BAM definition file and generate a compiled .NET 4.0 WCF service. The service is configured with a default basicHttp endpoint and is ready to go straight into AppFabric or similar hosting.

This service-approach makes it possible to take advantage of the BAM infrastructure from all different types of system, and even in cases when they aren’t behind the same firewall! As shown in figure below this could take us one step closer to the end-to-end tracking scenario.

“BAM Service Generator” is open-source and can be found here.

Promote properties in a EDI schema using the EDI Disassembler

2010-04-23T07:08:46+00:00

I’ve doing a lot of EDI related work in BizTalk lately and I have to say that I’ve really enjoyed it! EDI takes a while to get used to (see example below), but once one started to understand it I’ve found it to be a real nice, strict standard - with some cool features built into BizTalk!

UNB+IATB:1+6XPPC+LHPPC+940101:0950+1'
UNH+1+PAORES:93:1:IA'
MSG+1:45'
IFT+3+XYZCOMPANY AVAILABILITY'
ERC+A7V:1:AMD'
IFT+3+NO MORE FLIGHTS'
ODI'
TVL+240493:1000::1220+FRA+JFK+DL+400+C'
...

There are however some things that _doesn’t _work as expected …

Promoting values

According to the MSDN documentation the EDI Disassembler by default promotes the following EDI fields: UNB2.1, UNB2.3, UNB3.1, UNB11; UNG1, UNG2.1, UNG3.1; UNH2.1, UNH2.2, UNH2.3.

There are however situation where one would like other values promoted.

I my case I wanted the C002/1001 value in the BGM segment. This is a value identifying the purpose of the document and I needed to route the incoming message based on the value.

The short version is that creating a property schema, promoting the field in the schema and having the EDI Disassembler promoting the value will not work (as with the XML Disassembler). To do this you’ll need to use a custom pipeline component to promote the value. Rikard Alard seem to have come to the same conclusion here.

Promote pipeline component to use

If you don’t want to spend time on writing your own pipeline component to do this yourself you can find a nice “promote component” on CodePlex here by Jan Eliasen.

If you however expect to receive lots and lots of big messages you might want to look into changing the component to use XPathReader and custom stream implementations in the Microsoft.BizTalk.Streaming.dll. You can find more detailed information on how to do that in this MSDN article.

Use code blocks to extend your BizTalk custom XSLT maps

2010-04-12T07:38:47+00:00

Update 2010-04-13: Grant Samuels commented and made me aware of the fact that inline scripts might in some cases cause memory leaks. He has some further information here and you’ll find a kb-article here.

I’ve posted a few times before on how powerful I think it is in complex mapping to be able to replace the BizTalk Mapper with a custom XSLT script (here’s how to). The BizTalk Mapper is nice and productive in simpler scenarios but in my experience it break down in more complex ones and maintaining a good overview is hard. I’m however looking forward to the new version of the tool in BizTalk 2010 – but until then I’m using custom XSLT when things gets complicated.

Custom XSLT however lacks a few things once has gotten used to have - such as scripting blocks, clever functoids etc. In some previously post (here and here) I’ve talked about using EXSLT as a way to extend the capabilities of custom XSLT when used in BizTalk.

Bye, bye external libraries – heeeello inline scripts ;)

Another way to achieve much of the same functionality even easier is to use embedded scripting that’s supported by the XslTransform class. Using a script block in XSLT is easy and is also the way the BizTalk Mapper makes it possible to include C# snippets right into your maps.

Have a look at the following XSLT sample:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet
    version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:msxsl="urn:schemas-microsoft-com:xslt"
    xmlns:code="http://richardhallgren.com/Sample/XsltCode"
    exclude-result-prefixes="msxsl code"
    >
    <xsl:output method="xml" indent="yes"/>

    <xsl:template match="@* | node()">
        <Test>
            <UniqueNumber>
                <xsl:value-of select="code:GetUniqueId()" />
            </UniqueNumber>
            <SpecialDateFormat>
                <xsl:value-of select="code:GetInternationalDateFormat('11/16/2003')" />
            </SpecialDateFormat>
            <IncludesBizTalk>
                <xsl:value-of select="code:IncludesSpecialWord('This is a text with BizTalk in it', 'BizTalk')" />
            </IncludesBizTalk>
        </Test>
    </xsl:template>

    <msxsl:script language="CSharp" implements-prefix="code">
        //Gets a unique id based on a guid
        public string GetUniqueId()
        {
            return Guid.NewGuid().ToString();
        }

        //Formats US based dates to standard international
        public string GetInternationalDateFormat(String date)
        {
            return DateTime.Parse(date, new System.Globalization.CultureInfo("en-US")).ToString("yyyy-MM-dd");
        }

        //Use regular expression to look for a pattern in a string
        public bool IncludesSpecialWord(String s, String pattern)
        {
            Regex rx = new Regex(pattern);
            return rx.Match(s).Success;
        }
    </msxsl:script>
</xsl:stylesheet>

All one has to do is to define a code block, reference the xml-namespace used and start coding! Say goodbye to all those external library dlls!

It’s possible to use a few core namespaces without the full .NET namespace path but all namespaces are available as long as they are fully qualified. MSDN has a great page with all the details here.

Is there a bug in BizTalk 2006 R2 SP1?

2010-03-17T11:14:25+00:00

Update 2012-03-28: This bug was fixed in the CU2 update for BizTalk 2006 R2 SP 1. Further details can be found here and here.

Update 2010-04-12: Seems like there is a patch coming that should fix all the bugs in SP1 … I’ve been told it should be public within a week or two. I’ll make sure to update the post as I know more. Our problem is still unsolved.

Late Thursday night last week we decided to upgrade one of our largest BizTalk 2006 R2 environment to recently released Service Pack 1. The installation went fine and everything looked good.

… But after a while we started see loads of error messages looking like below.

Unable to cast COM object of type 'System.__ComObject' to interface type
'Microsoft.BizTalk.PipelineOM.IInterceptor'. This operation failed because the
QueryInterface call on the COM component for the interface with IID
'{24394515-91A3-4CF7-96A6-0891C6FB1360}' failed due to the following error: Interface not
registered (Exception from HRESULT: 0x80040155).

After lots of investigation we found out that we got the errors on ports with the follow criteria:

Send port
Uses the SQL Server adapter
Has a mapping on the port
Has a BAM tracking profile associated with the port

In our environment the tracking on the port is on “SendDateTime” from the “Messaging Property Schema”. We haven’t looked further into if just any BAM tracking associated with port causes the error or if only has to do with some specific properties.

Reproduce it to prove it!

I’ve setup a really simple sample solution to reproduce the problem. Download it here.

The sample receives a XML file, maps it on the send port to schema made to match the store procedure. It also uses a dead simple tracking definition and profile to track a milestone on the send port.

Sample solution installation instructions

Create a database called “Test”
Run the two SQL scripts (“TBL_CreateIds.sql” and “SP_CreateAddID.sql”) in the solution to create the necessary table and store procedure
Deploy the BizTalk solution just using simple deploy from Visual Studio
Apply the binding file (“Binding.xml”) found in the solution
Run the BM.exe tool to deploy the BAM tracking defintion.
Should look something like:

bm.exe deploy-all -definitionfile:\BAM\SimpleTestTrackingDefinition.xml
Start the tracking Profile editor and open the “SimpleTestTrackingProfile.btt“ that you’ll find in the solution and apply the profile
Drop the test file in the receive folder (“InSchema_output.xml”)

The sample solution fails on a environment with SP 1 but works just fine on a “clean” BizTalk 2006 R2 environment.

What about you?

I haven’t had time to test this on a BizTalk 2009 environment but I’ll update the post as soon as I get around to it.

We also currently have a support case with Microsoft on this and I’ll make sure to let you as soon as something comes out of that. But until then I’d be _really grateful _to hear from you if any of you have the same behavior in your BizTalk 2006 R2 SP1 environment.

Using XSLT 1.0 to summarize a node-set with comma separated values

2010-02-26T09:31:50+00:00

Pure XSLT is very powerful but it definitely has its weaknesses (I’ve written about how to extend XSLT using mapping and BizTalk previously here) … One of those are handling numbers that uses a different decimal-separator than a point (“.”).

Take for example the XML below

<Prices>
  <Price>10,1</Price>
  <Price>10,2</Price>
  <Price>10,3</Price>
</Prices>

Just using the XSLT sum-function on these values will give us a “NaN” values. To solve it we’ll have to use recursion and something like in the sample below.

The sample will select the node-set to summarize and send it to the “SummarizePrice” template. It will then add the value for the first Price tag of the by transforming the comma to a point. It will then check if it’s the last value and if not use recursion to call into itself again with the next value. It will keep adding to the total amount until it reaches the last value of the node set.

<xsl:template match="Prices">
  <xsl:call-template name="SummurizePrice">
    <xsl:with-param name="nodes" select="Price" />
  </xsl:call-template>
</xsl:template>

<xsl:template name="SummurizePrice">
  <xsl:param name="index" select="1" />
  <xsl:param name="nodes" />
  <xsl:param name="totalPrice" select="0" />

  <xsl:variable name="currentPrice" select="translate($nodes[$index], ',', '.')"/>

  <xsl:choose>
    <xsl:when test="$index=count($nodes)">
      <xsl:value-of select="$totalPrice + $currentPrice"/>
    </xsl:when>
    <xsl:otherwise>
      <xsl:call-template name="SummurizePrice">
        <xsl:with-param name="index" select="$index + 1" />
        <xsl:with-param name="totalPrice" select="$totalPrice + $currentPrice" />
        <xsl:with-param name="nodes" select="$nodes" />
      </xsl:call-template>
    </xsl:otherwise>
  </xsl:choose>

</xsl:template>

Simple but a bit messy and nice to have for future cut and paste ;)

Streaming pipeline and using context ResourceTracker to avoid disposed streams

2010-02-22T08:50:16+00:00

Recently there’s been a few really good resources on streaming pipeline handling published. You can find some of the here and here.

The Optimizing Pipeline Performance MSDN article has two great examples of how to use some of the Microsoft.BizTalk.Streaming.dl classes. The execute method of first example looks something like below.

public IBaseMessage Execute(IPipelineContext context, IBaseMessage message)
{
    try
    {
        ...
        IBaseMessageContext messageContext = message.Context;
        if (string.IsNullOrEmpty(xPath) && string.IsNullOrEmpty(propertyValue))
        {
            throw new ArgumentException(...);
        }
        IBaseMessagePart bodyPart = message.BodyPart;
        Stream inboundStream = bodyPart.GetOriginalDataStream();
        VirtualStream virtualStream = new VirtualStream(bufferSize, thresholdSize);
        ReadOnlySeekableStream readOnlySeekableStream = new ReadOnlySeekableStream(inboundStream, virtualStream, bufferSize);
        XmlTextReader xmlTextReader = new XmlTextReader(readOnlySeekableStream);
        XPathCollection xPathCollection = new XPathCollection();
        XPathReader xPathReader = new XPathReader(xmlTextReader, xPathCollection);
        xPathCollection.Add(xPath);
        bool ok = false;
        while (xPathReader.ReadUntilMatch())
        {
            if (xPathReader.Match(0) && !ok)
            {
                propertyValue = xPathReader.ReadString();
                messageContext.Promote(propertyName, propertyNamespace, propertyValue);
                ok = true;
            }
        }
        readOnlySeekableStream.Position = 0;
        bodyPart.Data = readOnlySeekableStream;
    }
    catch (Exception ex)
    {
        if (message != null)
        {
            message.SetErrorInfo(ex);
        }
        ...
        throw ex;
    }
    return message;
}

We used this example as a base when developing something very similar in a recent project. At first every thing worked fine but after a while we stared getting an error saying:

Cannot access a disposed object. Object name: DataReader

It took us a while to figure out the real problem here, everything worked fine when sending in simple messages but as soon as we used to code in a pipeline were we also debatched messages we got the “disposed object” problem.

It turns out that when we debatched messages the execute method of the custom pipeline ran multiple times, one time for each sub-messages. This forced the .NET Garbage Collector to run.

The GC found the XmlTextReader that we used to read the stream as unreferenced and decided to destoy it.

The problem is that will also dispose the readOnlySeekable-Stream stream that we connected to our message data object!

It’s then the BizTalk End Point Manager (EPM) that throws the error as it hits a disposed stream object when trying to read the message body and save it to the BizTalkMsgBox!

ResourceTracker to the rescue!

Turns out that the BizTalk message context object has a nice little class connected to it called the ResourceTracker. This object has a “AddResouce”-method that makes it possible to add an object and the context will the hold a reference to this object, this will tell the GC not to dispose it!

So when adding the below before ending the method everything works fine - even when debatching messages!

context.ResourceTracker.AddResource(xmlTextReader);

Checking if BizTalk binding file is up-to date during deployment

2009-12-11T10:19:52+00:00

As all of you know the number one time consuming task in BizTalk is deployment. How many times have you worked your way through the steps below (and even more interesting - how much time have you spent on them …)

Build
Create application
Deploy schemas
Deploy transformations
Deploy orchestration
Deploy components
Deploy pipelines
Deploy web services
Create the external databases
Change config settings
GAC libraries
Apply bindings on applications
Bounce the host instances
Send test messages
Etc, etc …

Not only is this time consuming it’s also drop dead boring and therefore also very prone - small mistakes that takes ages to find and fix.

The good news is however that the steps are quite easy to script. We use a combination of a couple of different open-source MsBuild libraries (like this and this) and have created our own little build framework. There is however the BizTalk Deployment Framework by Scott Colescott and Thomas F. Abraham that looks great and is very similar to what we have (ok, ok, it’s a bit more polished …).

Binding files problem

Keeping the binding files in a source control system is of course a super-important part of the whole build concept. If something goes wrong and you need to roll back, or even rebuild the whole solution, having the right version of the binding file is critical.

A problem is however that if someone has done changes to the configuration via the administration console and missed to export these binding to source control we’ll deploy an old version of the binding when redeploying. This can be a huge problem when for example addresses etc have changed on ports and we redeploy old configurations.

What!? If fixed that configuration issue in production last week and now it back …

So how can we reassure that the binding file is up-to-date when deploying?

One solution is to do and export of the current binding file and compare that to one we’re about to deploy in a “pre-deploy”-step using a custom MsBuild target.

Custom build task

Custom build task in MsBuild are easy, a good explanation of how to write one can be found here. The custom task below does the following.

Require a path to the binding file being deployed.
Require a path to the old deployed binding file to compare against.
Using a regular expression to strip out the time stamp in the files as this is the time the file was exported and that will otherwise differ between the files.

Compare the content of the files and return a boolean saying if they are equal or not.

 public class CompareBindingFiles : Task
 {
     string _bindingFileToInstallPath;
     string _bindingFileDeployedPath;
     bool _value = false; 

     [Required]
     public string BindingFileToInstallPath
     {
         get { return _bindingFileToInstallPath; }
         set { _bindingFileToInstallPath = value; }
     } 

     [Required]
     public string BindingFileDeployedPath
     {
         get { return _bindingFileDeployedPath; }
         set { _bindingFileDeployedPath = value; }
     } 

     [Output]
     public bool Value
     {
         get { return _value; }
         set { _value = value; } 

     } 

     public override bool Execute()
     {
         _value = GetStrippedXmlContent(_bindingFileDeployedPath).Equals(GetStrippedXmlContent(_bindingFileToInstallPath));
         return true; //successful
     } 

     private string GetStrippedXmlContent(string path)
     {
         StreamReader reader = new StreamReader(path);
         string content = reader.ReadToEnd();
         Regex pattern = new Regex("<Timestamp>.*</Timestamp>");
         return pattern.Replace(content, string.Empty);
     } 

 }

Using the build task in MsBuild

After compiling the task above when have to reference the dll in element like below.

<UsingTask AssemblyFile="My.Shared.MSBuildTasks.dll" TaskName="My.Shared.MSBuildTasks.CompareBindingFiles"/>

We can then do the following in out build script!

<!--
    This target will export the current binding file, save as a temporary biding file and use a custom target to compare the exported file against the one we’re about to deploy.
    A boolean value will be returned as IsValidBindingFile telling us if they are equal of not.
-->
<Target Name="IsValidBindingFile" Condition="$(ApplicationExists)=='True'">
    <Message Text="Comparing binding file to the one deployed"/> 

    <Exec Command='BTSTask ExportBindings /ApplicationName:$(ApplicationName) "/Destination:Temp_$(BindingFile)"'/> 

    <CompareBindingFiles BindingFileToInstallPath="$(BindingFile)"
                         BindingFileDeployedPath="Temp_$(BindingFile)">
        <Output TaskParameter="Value" PropertyName="IsValidBindingFile" />
    </CompareBindingFiles> 

    <Message Text="Binding files is equal: $(IsValidBindingFile)" /> 

</Target>

<!--
    This pre-build step runs only if the application exists from before. If so it will check if the binding file we try to deploy is equal to one deployed. If not this step will break the build.
-->
<Target Name="PreBuild" Condition="$(ApplicationExists)=='True'" DependsOnTargets="ApplicationExists;IsValidBindingFile">
    <!--We'll break the build if the deployed binding files doesn't match the one being deployed-->
    <Error Condition="$(IsValidBindingFile) == 'False'" Text="Binding files is not equal to deployed" /> 

<!--All other pre-build steps goes here-->

</Target>

So we now break the build if the binding file being deployed aren’t up-to-date!

This is far from rocket science but can potentially save you from making some stupid mistakes.

BAM tracking data not moved to BAM Archive database

2009-11-13T15:31:20+00:00

There are a few really good blog post that explains BAM – like this from Saravana Kumar and this by Andy Morrison. They both do a great job explaining the complete BAM process in detail.

This post will however focus on some details in the last step of the process that has to do with archiving the data. Let’s start with a quick walk-through of the whole process.

BAM Tracking Data lifecycle

The tracked data is intercepted in the BizTalk process and written to the “BizTalk MsgBox” database.
The TDDS service reads the messages and moves them to the correct table in the “BAM Primary Import” database.
The SSIS package for the current BAM activity has to be triggered (this is manual job or something that one has to schedule). When executing the job will do couple of things.
1. Create a new partitioned table with a name that is a combination of the active table name and a GUID.

Move data from the active table to this new table. The whole point is of course to keep the active table as small and efficient as possible for writing new data to. 
   
Add the newly created table to a database view definition. It is this view we can then use to read all tracked data (including data from the active and partitioned tables). 
   
Read from the “BAM Metadata Activities” table to find out the configured time to keep data in the BAM Primary Import database. This value is called the “online window”. 
   
Move data that is older than the online window to the “BAM Archive” database (or delete it if you have that [option](http://support.microsoft.com/kb/971984)).

Sound simple doesn’t it? I was however surprised to see that my data was not moved to the BAM Archive database, even if it was clearly outside of the configured online window.

So, what data is moved to the BAM Archive database then?

Below there is a deployed tracking activity called “SimpleTracking” with a online window of 7 days. Ergo, all data that is older than 7 days should be moved to the BAM Archive database when we run the SSIS job for the activity.

If we then look at the “BAM Completed” table for this activity we see that all the data is much older than 7 days as today’s date is “13-11-2009”.

So if we run the SSIS job these rows should be moved to the archive database. lets run the SSIS job. Right?

But when we execute the SSIS job the BAM Archive database is still empty! All we see are the partitioned tables that were created as part of the first steps of the SSIS job. All data from the active table is however moved to the new partitioned table but not moved to the Archive database.

It turns out that the SSIS job does not at all look at the the “Last Modified” values of each row but on the “Creation Time” of the partitioned table in the “BAM MetaData Partitions” table that is shown below.

The idea behind this is of course to not have to read from tables that potentially are huge and find those rows that should be moved. But it also means that it will take another 7 days before the data in the partitioned view is actually move to the archive database.

This might actually be a problem if you haven not scheduled the SSIS job to run from day one and you BAM Primary Import database is starting to get to big and you quickly have to move data over to archiving. All you then have to is of course to change that “Creation Time” value in the BAM Metadata Partitions table so it is outside of the online window value for the activity.

What kind of integration patters do you implement in BizTalk?

2009-11-10T19:41:58+00:00

I like to divide BizTalk based integrations into three different patterns.

Point-to-point integration
Broker-based integration
ESB-based integration

I am sure someone could come up with fancier names but I will in this post try and dig into each of them, explain what I mean by them. I will also try and highlight some issues with each of the patterns.

Point-to-point integration

This is where a lot of us started. The idea is that the sending system has information and knowledge about the receiving system.

This usually means that the messages are exported in a message format that is tailored to the format of the particular receiving system’s needs. It also means that we usually get one integration process for each receiving system there is.

The example in the figure on the right shows a sending system that sends information about invoices to two different receiving systems (“System A” and “System B”). Using a point-to-point pattern in this scenario we end up with two different types of messages that are exported. Each of the exported messages that are suited and tailored to the format of the receiving system.

Issues

The main problem with this kind of point-to-point integration, where the sending system has detailed knowledge about the receiving systems, is that this knowledge translates into a coupling between the systems. When we tie the export message format from the sending system to the format of the receiving, all changes in the receiving system will also cause changes in the sending system.

Suppose there is a sudden need to change the format of the receiving system - as we use that format as our export format we now also have to change it there.

Another problem is the lack of agility. In this invoice scenario all other system that also has the need of invoice based information has to get this by going all the way back to the sending system and develop a whole new specific integration - separate to the existing ones.

Broker-based integration

In the broker-based scenario the integration platform is used as a broker of messages.

This means that only **one canonical **format is exported from the sending system. The broker then handles the routing to the different systems and the transformation to the message formats that the receiving systems expects.

The main advantage between this approach - where the sending system do not know anything about the receiving systems – and the point-to-point pattern is agility.

If there now is a need for invoice information in a third, new system, we do not have to change the export from the sending system (as long as we have all the information need that is) or develop a whole new integration. All we have to do is to route the invoices so that they also are sent to the third system and transformed into the format that the system expects. A new BizTalk map and port and we are done!

Issues

In my point of view this approach to integration has a lot of advantages over the point-to-point integrations previously discussed. And in a lot of simpler, stabile scenarios it works just fine and is the way to go.

But in some scenarios it kind of breaks down and becomes hard to work with. The problems is related to configuration and how we define the routing information in BizTalk. In BizTalk we can either create an orchestration and “hardcode” the process that defines which systems the messages should be sent to. We can also create a “messaging” scenario where we configure this routing information in the different BizTalk port artifacts by setting up filters.

Regardless if we choose a “messaging” or orchestration based solution the routing information becomes hard to update as the solution grow in size. We either get very complicated orchestrations or loads of ports to maintain.

Furthermore the whole process is very coupled to the one canonical schema which makes versioning and updates hard. If the canonical schema needs to be updated and we still need to be able to send information using the old schema (so we have a “version 1” and “version 2” of the schema side-by-side) all artifacts needs to be more or less duplicated and the complexity of the solutions grows fast.

This makes business agility hard and any changes takes long time to develop and deploy correctly.

I guess these issues has more to do with how BizTalk works than the integration pattern itself - but this is a BizTalk blog!

ESB-based integration

ESB-based integration in BizTalk is today achieved using ESB Toolkit from Microsoft.

One of the ideas of ESB-based integration is that the integration platform should not have hardcoded routing information. These routing rules should either be looked up at run time or travel with the actual payload of the message (called “dynamic routing”).

This in combination with having generic on- and off-ramps instead of loads of separate ports to maintain _promises to create more agile and configurable solutions _(called “Loosely coupled service composition” and “Endpoint run-time discovery and virtualization”).

The figure again shows the invoice scenario used previously. In this case we export a “Invoice 1.0”-format from the sending system. The format is not directly bound to a schema in the port (as we are used to in BizTalk) but it is basically possible to send any message format to the on-ramp.

The on-ramp then identifies what kind of message it received (for example using the XML-namespace) and looks up the routing rules for that specific message (the message “itinerary”) in a repository.

In this scenario there could be rules to route the message to a orchestration or directly to a send port. _All the configuration for this port is however configured as part of the itinerary and applied at run-time on the generic off-ramp port. And as itineraries are simple XML documents they are super light to update and deploy in the repository! _

So if we now wanted to update the “Invoice”-format to “version 2.0” all we would have to do is to create a new XML itinerary for the “Invoice 2.0” message type and possibly create new orchestration to handle the new message type. No new ports, no new bindings etc, etc. The configuration and deployment would be a lot simpler than before! We would end up with a lot fewer artifacts to maintain.

**And **the itinerary for “Invoice 1.0” would still be applied to all incoming Invoice 1.0 messages. Thus we have achieved what used to be so hard with a lot less development power!

Issues

With agility and dynamic capabilities comes complexity in debugging when something bad happens … I also feel that we give up a lot of validating and control capabilities that we had in the more schema-coupled patterns previously discussed.

I am new to this ESB-based model and I would love to hear your experiences, problems, stories etc!

I am however sure that this is the way to go and I would not be surprised if ESB like patterns will play an ever bigger role in future BizTalk versions!

Third party BizTalk monitoring tools

2009-08-11T15:18:11+00:00

Update 2012-03-28
When looking into BizTalk monitoring and overall governance tools for BizTalk, don’t miss BizTalk360. BizTalk 360 wasn’t available when this post was written and has some really compelling features.

Update 2010-09-30: BizMon is now owned and developed by Communicate Norway. They have renamed, and further developed the product, to IPM (Integration Monitoring Platform) – check it out here.

I’ll start this post by clarifying two important things

I am involved in the development and marketing of “BizMon”. Therefore I am biased and you have to decide for yourself if that affects the content of the post. As always it is best to try it for yourself and see if it is useful for you.
I have talked about BizTalk monitoring tools in a previous post and my goal then was then to start a an open source project. That did not happened and you can read why in the update to that post.

Why “monitoring” for BizTalk?

I have worked as a BizTalk developer for many years but it was not until I really got in to maintaining a large integration solution that I realized that the tools I really needed was not there. I found myself using the following “tools” and techniques over and over again.

Open the BizTalk Administration Console and query for suspended messages, running instances, routing errors etc, etc.

But as I had to pull for this information it took time and discipline (two things I’m short of) to quickly find out when errors occurred.

I used the HAT to try and find out when the last messages was sent and received on the different applications. This gave me a “guarantee” that things worked as I accepted and that the solution had a “pulse” – messages at least moved back and forward.

The problem is that the HAT tool is bad and it is hard to find what one is looking for (It is a bit better in BizTalk 2009 but it is still tricky to get useful information out of it.)

Some of the integrations in our environment used BAM to track messages and their state.

The problem was that all solutions was developed by either myself or different consultants. This made it hard to get everyone to use the same tracking. It was also hard to convince management to go back and try and “instrument” old working integrations with BAM tracking.

At the same time as we had the “tools” and techniques mentioned above available, management had the following requirements for us.

Start working on fixing an error within 10 minutes after it occurred 24/7 all 365 days …
Be able to delegate simple monitoring task to support personnel (a help desk).
Not have to actively “pull” for information but be quickly altered of errors and get the information “pushed” to us.

The idea was that this would would save time as people don’t have to look for errors when everything is working fine. Time that people can use for other tasks …

Enable reporting so we can provide systems owners and other interested people with information on how much data has been sent received to the systems and parties they care about.

All the above lead up to the realization that we needed some sort of tool.

What are the existing options for BizTalk monitoring tooling?

At the time we started looking for options all we could find was System Center Operations Manager (SCOM). We looked at SCOM BizTalk Management Pack and decided that for us this was not the right solution. It was too big, too complicated and it would be to hard to get it to the what we wanted to do.

> > The decision to not use SCOM I think was right for us. We wanted something leaner and more specialized. I am however **_not saying _**that it is the right decision for you. > > > > If you are successfully suing SCOM to monitor BizTalk I would love to [hear about it](mailto:richard.hallgren@gmail.com)! > >

What we ended up with

We ended up building BizMon. It does what we need and our help desk can now basically monitor about 100 different BizTalk application themselves. At the same time they do all the other support task they have to do. When something happens (and it does …) they are the first to know. Some easy tasks they can solve themselves, otherwise they make sure to notify the users and quickly call the developer that knows more and can help them.

Support personnel can now also setup custom reports that users can subscribe to, all based on BAM that they now easily can interject tracking points in existing solutions – both new and old ones.

As I said. This worked out out good and helped us. If you think that it could work for you as well – give it a try.

I am also really interested to how you have solved similar requirements as we had with your own tool or other solutions.

What else is there?

Recently FRENDS released a beta version of their FRENDS Helium product that looks promising could potentially solve a lot of the same issues that BizMon does and that I have discussed in this post.

Check it out and let us know what you think.

Tips and tricks on BizTalk generated SOAP Web Services

2009-06-16T08:30:32+00:00

Traditional SOAP Web Services might feel kind of old as more and more people move over to WCF. But a lot of integration projects still relay heavily on old fashion SOAP Web Services.

Using BizTalk generated Web Services however has a few issues and one needs to add a few extra steps and procedures to make them effective and easy to work with. This post aims to collect, link and discuss all those issues and solutions.

1. Building and deploying

BizTalk Server includes the “BizTalk Web Services Publishing Wizard” tool that integrates with Visual Studio. This is basically a tool to generate a DSL based script for generating web services.

The wizard collects information about what schema or a orchestration to expose, namespaces, names of service and method, where on IIS to publish the service etc, etc.

The output of the tool is then a xml file (a “WebServiceDescription” file) that has all the collected values in it.

As a final step Visual Studio uses the newly created description file as input to a class called WebServiceBuilder in the .NET Framework. It is this class that is responsible for interpreting the description, configure and generate the service.

A common procedure is to use the wizard and click thru it and input values for every single deployment. This is of course slow, error prone and stupid.

What is much better is to take a copy of the generated “WebServiceDescription” file, save it to your source control system and then programmatically pass the file to the WebServiceBuilder class as part of your deployment process. Possible changes to the details of the service can then be done directly in the description file.

I have seen this approach save lots of time and problems related to deployment.

Paul Petrov has a great post on how to call the “WebServiceBuilder” class and pass the description file using C#.
http://geekswithblogs.net/paulp/archive/2006/05/22/79282.aspx
Michael Stephenson has a good post on how to used the “WebServiceBuilder” class via MSBuild.
http://geekswithblogs.net/michaelstephenson/archive/2006/09/16/91369.aspx

2. Fixing namespace

Another annoying problem (I’d would actually go so far as calling it a bug) is the problem with the bodyTypeAssemblyQualifiedName value in the generated Web Service class.

This causes BizTalk to skip looking up the actual message type for the incoming message. As no message type exists for the message is in BizTalk mapping and routing on message types etc will fail. It is a know problem and there are solutions to it. I would also recommend take the extra time need to make this small “post process step” be part of your deployment process (see how here).

Saravana Kumar discusses the problem and solutions in this great post.
http://www.digitaldeposit.net/saravana/post/2007/08/17/SOAP-Adapter-and-BizTalk-Web-Publishing-Wizard-things-you-need-to-know.aspx
Richard Seroter has a short and well written post published.
[http://blogs.msdn.com/richardbpi/archive/2006/09/15/Fixing-_2200SOAP-_2F00-Envelope-Schema_2200-Error-In-BizTalk.aspx](http://blogs.msdn.com/richardbpi/archive/2006/09/15/Fixing-_2200_SOAP-_2F00-Envelope-Schema_2200_-Error-In-BizTalk.aspx)
I have a post on how to solve the issue using MSBuild.
http://www.richardhallgren.com/handle-the-bodytypeassemblyqualifiedname-soap-adapter-bug-in-msbuild-as-a-regex-ninja/

3. Pre-compiling

By default the “WebServiceBuilder” class generates a web service without pre-compiling it. Usually this is not a problem. But in some cases were one really on the web service being online and give a quick response-message the performance problems in this approach can be a huge problem.

When generating the web service without pre-compiling it IIS has to compile the service and then keep the compiled service in memory. That means that when IIS releases the service from memory there is a latency before IIS re-compiled the service, loaded it into memory and executed it. This is a known problem and I have seen this “slow first hit” issue been a frequent question the different forums.

The solution is to use the aspnet_compiler.exe tool and pre-compile the service and the use those pre-compiled dlls as the service. IIS then never has to recompile it and will serve initial hits much faster.

Here is an example of how we defined a target to do this as part of our deployment process using MSBuild.

Pre-compile the service into a new folder
Clean out the “old” not compile service folder.
Copy the pre-complied service into the service folder

<Target Name="CompileWeb">
    <Message Text="Uses aspnet compiler to compile the service into a new folder. Then copies the compiled content back into its original place" />
    <AspNetCompiler
        PhysicalPath="$(WebSiteServicePath)InitiateProjectService\"
        VirtualPath="/WebServiceName"
        TargetPath="$(WebSiteServicePath)$(WebServiceName)Compiled\"
        Force="true" />

    <Folder.CleanFolder Path="$(WebSiteServicePath)$(WebServiceName)\"/>

    <Folder.CopyFolder
                Source="$(WebSiteServicePath)$(WebServiceName)Compiled\"
                Destination="$(WebSiteServicePath)$(WebServiceName)\" />

</Target>

Paul Petrov has two different articles here describing the process and also a different way that above on how to include the pre-compilation in you build process.

http://geekswithblogs.net/paulp/archive/2006/03/30/73900.aspx, http://geekswithblogs.net/paulp/archive/2006/04/19/75633.aspx

BAM ate my XML declaration!

2009-05-28T12:14:49+00:00

There are integrations which only purpose is to move a file just as it is. No advanced routing. No orchestration processing. No transformation. Just a simple port-to-port messaging scenario.

It is however still a good idea to monitor these just as one would monitor a more complicated integration. We use BAM to monitor all our integrations and to measure how many messages that has been processed in a integration. Using BAM monitoring in a simple solution as the above however has its issues …

Setting up a simple test solution

The solution will move a XML file between two port called “SimpleTrackingReceivePort” and “SimpleTrackingSendPort”.
Both port have PassThru pipelines configured.
The XML file does not have a installed schema. Remember we are just moving the file not actually doing anything with it.
A BAM tracking definition with one milestone called “StartPort” will be used. This will be mapped to the “PortStartTime” property on both the receiving and sending port .

Our tracking profile configuration will like below. Dead simple.

So – what’s the problem?

Let us drop a XML message looking some like this.

<?xml version="1.0" encoding="UTF-16"?>
<SimpleTest>
    <SimpleContent Text="This is a test" />
</SimpleTest>

Remember that there is not a schema installed so we do not really have to worry about the structure of the file. It should just be “a file” to BizTalk and everything should be transferred between the ports. Even if we drop a executable or whatever - it should just be transferred. Nothing should read or examine the file as it’s just a pass thru!

As soon as BAM tracking is configured on a port that is however not the case. Lets take a look at the file we receive on the other end of our integration.

<SimpleTest>
    <SimpleContent Text="This is a test" />
</SimpleTest>

BizTalk now removed our XML declaration! Basically it treated the message as a XML message and parsed the message as such while tracking it. It’ will also add the dreaded Byte-Order-Mark and fail any non-valid XML messages. The problem is that this is not the behavior what one expects and causes receiving systems that rely on the XML declaration to fail!

As we also don’t have a installed schema it is not possible to use a XMLTransmit pipeline to add the XML declaration and remove the BOM.

What to do?

If you’d like to track a XML based message using BAM make sure you have the schema installed … Even if you are just using PassThru.

Is it a bug or just something one should expect? In my opinion it is at least very annoying!

Receiving scheduled MsgBoxViewer-reports via e-mail

2009-03-19T16:37:01+00:00

I attended a session the other day at TechDays here in Sweden with Microsoft Escalation Engineer Niklas Engfelt. The session was about troubleshooting BizTalk and Niklas of course showed the wonderful MsgBoxViewer (MBV) tool by Jean-Pierre Auconie. If you haven’t tested and looked deeper into this tool you need to do so. It’s great!

I worked with the tool before but now I wanted to schedule the tool and to have MBV-reports e-mailed to relevant persons within the company on a weekly basis. This is quite easy to accomplish as MBV comes in two version. One GUI-based (shown below) version and one command-line based.

The command-line version is of course perfect for scheduling using the Windows Task Scheduler.

If you feel uncomfortable running all the queries (there is a lot of them) on a schedule you can pick some you find important and configure the tool to only run those. Jean-Pierre has a post on how to do just that here.

After MBV has completed all its queries and done its magic it will produce a html-report in the working folder (that’s the folder in the “Start in” field in the scheduled task example above).

We then use a tool called AutoMailer NT (cost €20 - there is a 30 days trial) to:

Poll the working folder for a *.html report file.
Compress the file (using zip).
Send the report to a configured list of recipients.
Delete the report file.

The AutoMailer NT installation is a bit rough (don’t miss to the ****separate download (!) of the trial certificate). But once you have everything working it’s great to have a fresh MBV report in you inbox every Monday telling you how your BizTalk environment is doing and possible issues to attend to.

Better performance in batch imports to SQL Server using BizTalk

2009-02-17T21:36:09+00:00

During my years of BizTalk development I’ve been warned of a couple of scenarios that the product wouldn’t handle very well. Yesterday another of those scenarios turned out to kind of false and, if done right, not really a problem at all.

The scenario I’m talking about is a batch import of data to SQL Server using the SQL adapter. In my case the data is received as a flat text file containing a large number of rows. These rows should the be places inside a database table as one table-row per row in the flat file.

The common way of dealing with batch incoming data like this is to split (aka disassemble) it in the receive port using the Flat File Disassembler pipeline component (for a good example - look here). Disassembling the data when receiving it is usually good practice to avoid running into OutOfMemoryException when dealing with big messages.

Sometimes the requirements also forces one into reading each row to a separate message to be able to route and handle each messages in a unique way depending of it’s content. If that so - this is a not a good post for you. In this post I’ll discuss the scenario were all the data just needs to go as fast as possible from the text file into a single database table. No orchestration or anything, just a simple batch import.

So, what’s The problem with the batch import scenario?

When I implemented similar batch import scenarios in the past I tried to practice good practice and split the data into separate files that I then filtered to the SQL adapter send port, one by one.

The received flat file files has been split into thousands of small little message that one by one are sent to the SQL adapter send port.
The SQL adapter then parses each message into a SQL script that executes a store procedure and the message is finally inserted to the target database.

“So what’s the problem?” you then ask? It’s slow! It’s very slow! Each message gets stored a couple of times in the BizTalk database and each message is sent inside it’s own DTC transaction against the target database. And all this adds up …

And after reading this this interview by Richard Seroter with Alan Smith I also felt I was the only one having the problem either …

There are quite a few people asking about using BizTalk for heavy database integration, taking flat files, inserting the data in databases and processing it. SQL Server Integration Services (SSIS) does a much better job than BizTalk at this, and is worth looking at in those scenarios. BizTalk still has its uses for that type of work, but is limited be performance. The worst case I saw was a client who had a daily batch that took 36 hours to process using BizTalk, and about 15 minutes using SSIS. On another project I worked on they had used BizTalk for all the message based integration, and SSIS for the data batching, and it worked really well. > >

Note: As I’ll described later in this post my import scenario went from something like 3-4 hours to 2 minutes (importing 10 MB). Alan talks about a 36 hours (!) import. I don’t know anything more about the scenario he mentions and it might not even be solved using the technique discussed below. Hopefully Alan might comment on the post and give us more details. ;)

How can we get better performing imports using BizTalk?

As the import scenario we described doesn’t involve any orchestration but is a pure messaging scenario and we do all the transformation on the ports we don’t really have to worry about _OutOfMemoeyExceptions _even though the message is quite big.

**Large message transformation.** In previous versions of BizTalk Server, mapping of documents always occurred in-memory. While in-memory mapping provides the best performance, it can quickly consume resources when large documents are mapped. In BizTalk Server 2006, large messages will be mapped by the new large message transformation engine, which buffers message data to the file system, keeping the memory consumption flat. ([Source](http://www.microsoft.com/technet/prodtechnol/biztalk/2006/evaluate/overview/default.mspx)) > >

Another reason for splitting the message was for it to work with the SQL adapter. When setting up the SQL adapter to work with a store procedure the adapter expects a message that looks something like the below.

<ns0:ImportDataSP_Request xmlns:ns0="http://FastSqlServerBatchImport.Schemas.SQL_ImportDataSP">
    <ImportData Name="Name 1" Value="1"></ImportData>
</ns0:ImportDataSP_Request>

This tells us that the store procedure called is “ImportData” with “Name 1” as the value for the “Name” parameter and “1” as the value for the parameter called “Value” in the stored procedure. So each little separate message would get mapped on the send port into something like this.

What I however didn’t know until I read this post was that the message I send to the SQL adapter port just as well could look like this!

<ns0:ImportDataSP_Request xmlns:ns0="http://FastSqlServerBatchImport.Schemas.SQL_ImportDataSP">
    <!-- TWO rows!!! -->
    <ImportData Name="Name 1" Value="1"></ImportData>
    <ImportData Name="Name 2" Value="2"></ImportData>
</ns0:ImportDataSP_Request>

So basically we can have as many store procedure calls as we want in one single file that then can send to the SQL adapter send port!

Eureka! **__This means that we don’t have to split the incoming file! We can keep it as one big single file and just transform it to a huge file containing separate nodes that we send to the SQL Adapter send port! The SQL adapter will then parse this into separate store procedure calls for us.**

Is it really any faster?

As the technique above drastically reduced the amount of database communication needed I knew it’d be much faster. Some initial testing shows that an import of a file containing somewhere around 55 000 rows (about 10 MB) into our article database went from 3-4 hours to under two minutes!

See for yourself!

In this sample solution I have a text file containing 2 600 rows. I’ve then created two separate projects in a solutions. One that splits the messages into separate messages (called “SlowImport”) and one that just transforms it and send it as one message to the send port (called “FastImport”). One takes 1:50 minutes and 2 seconds on my development machine … I won’t tell you which one is the faster one …

Test it for yourself and let me know what you think.

Aggregated monitoring of BizTalk solutions using 'BizMon'

2009-01-06T15:25:20+00:00

> > **Update 2010-09-30:** BizMon is now owned and developed by [Communicate Norway](http://www.communicate.no). They have renamed, and further developed the product, to IPM (Integration Monitoring Platform) – check it out [here](http://ipm.communicate.no/exciting-news). > >

> > _**Update 2009-08-11: **This project turned out to be far more complicated and bigger than I first expected (ever heard that before?). Due to that and the fact that we wanted to have a company behind that could offer full-time support and stability “BizMon” has been released as a commercial product that you can find [here](http://bizmontool.com/). _ > > > > _I love to get some [help from you](http://bizmontool.com/we-have-a-rtm-version)**** to test it and make it as good as possible. Even if it is commercial and cost money we have a free alternative for small environments and we work hard to keep the license price as low as possible._ > >

****

> > _**Update 2009-02-25:** In the original post I said I'd post more on the architecture and the code during February 09. I'm however current struggling getting the needed legal rights etc, etc to be able to talk further about the "BizMon"-solution. It was harder than I thought ... I'll get back to posing on the subject as soon as I have that sorted._ > >

Integration of enterprise processes often ends up being very business critical. If a integration fails delivering the messages it was supposed to it usually means the business will be affected in a very negative way (for example losing money or delivering bad service). That of course means that monitoring the status of the integrations soon becomes very important (if you’re not into getting yelled at or potentially loosing your job).

Strangely enough BizTalk Server 2006 R2 _in my humble opinion_ doesn’t come with the right tool to efficiently monitoring big enterprise integration solutions!

What do I mean by monitoring?

Before I get myself deeper into trouble I’d like to define what I mean by monitoring. I think monitoring a BizTalk integration solution could be divided into four categories.

Infrastructure **(traditional)
**This is the easy one and one that IT-pros and alike are used to monitor. Hardware status, network traffic, disk space, event logs etc all fall under this category. If the storing area for the databases start running low on memory we can be pretty sure it’ll eventually effect the integration somehow.
**BizTalk infrastructure
**This is where it starts getting a bit trickier. This category includes the status of receive locations, orchestrations, host instances and send ports. If a receive location is down no messages will be picked up (but we can also be sure of not getting any suspended messages).
**Suspended messages
**As most reader of this blog probably know suspended message is due to some sort of failure in BizTalk. It can be an actually exception in code or something that went wrong while trying to send messages. It’s however and important category to monitor.
**Heartbeat (monitoring actual successful traffic)
**While the points 1-3 above focuses on errors and that things being inactive this category actually monitors that the integration runs as expected.

To me this final point is almost the most important one. What I mean is that if everything runs as expected and we’re sending the expected amount of messages in the right pace everything else must be ok - right? It’s however the one that in my experience almost always overlooked!

“What do you mean ‘Not the right tools to monitor’? We have loads of tools in BizTalk 2006 R2!”

OK. So let’s see what tools we have available actually monitor the categories above.

Infrastructure **(traditional)**
I won’t discuss this kind of monitoring in this post. There are loads of tools (all from the huge expensive enterprise ones to plenty of good open-source alternatives) for this and you’re probably already using one or several of them already.
**BizTalk infrastructure
**There are a couple of way of achieving this. One of the is to use the Microsoft BizTalk Server Management Pack for Operation Manager. It does however of course require that you have invested in System Center Operation Manager already …

Another way is to either use the ExplorerOM classes or connecting directly to the BizTalk configuration database and code your own report of some sort.

The final (and most common way in my experience) is to try and document the correct configuration and settings and then have someone check these manually (if you’re that person I feel for you …).

**Suspended messages
**Suspended messages are of course very important to monitor and it’s for some reason also the first thing developers think of monitoring when developing BizTalk integration (maybe because of the fact that they’re similar to traditional exceptions in software). There are also here a couple of different ways to solve the problem.

Microsoft BizTalk Server Management Pack for Operation Manager mentioned above has the functionality to monitor and altering on suspended messages.

BizTalk Server fires the MSBTS_ServiceInstanceSuspendedEvent WMI event every time a service instance gets suspended. It’s fully possible to write a service that watches for this event and then for example sends some sort of alert. Darren Jefford has an example on how do something like that in this post.

In BizTalk 2006 Failed Message Routing was introduced. This gives the developer the possibility to subscribe to suspended messages. These can then for example be sent out to file system or written to a database. Microsoft ESB Guidance for BizTalk Server 2006 R2 Exception management component uses this approach. The problem with this approach is however that the message is moved out of BizTalk and one loses all the built in possibilities of resending them etc.

**Heartbeat (monitoring actual successful traffic)

As I said before I think this is a very important metric. If you can see that messages travel through BizTalk in a normal rate things much be pretty ok – right? Without doing to much coding and developing you own pipeline components for tracking etc there are two options.

**

The first one is of course using the Health and Activity Tracking tool (HAT). This shows a simple view of receives, processed and sent messages. I hate to say it but the HAT tool is bad. It’s slow, it’s hard to use, it’s hard to filter information, it times out, it doesn’t aggregate information, it’s basically almost useless … (Just to make one thing clear: I make my living working with BizTalk and I really enjoy the product but tracking and monitoring is really one of it’s ugly sides. I hate to say it.)

The other option is to develop a simple BAM tracking profile to monitoring the send and receive port ports of the different processes.

So to repeat what I said earlier: no I don’t think BizTalk comes with the right tool to monitor integration solutions. I do however think that the platform has the capabilities to create something that could close that gap in the product.

What I need!

Much of what’s discussed in this post can be solved using the BizTalk Administrations Console (to manually monitor BizTalk infrastructure status) or in the Health and Activity Tracking tool (to manually monitor traffic). The aim of this post is however to discuss the possibilities to use this information, aggregate it and give the persons responsible for monitoring integration a dashboard that shows the current status of all integrations within the enterprise.

The dashboard monitor application need the following main features.

In one single screen give an overview of the overall status of all the integrations. By status I mean if there are ports, orchestration or host instances that aren’t running that should be running or if there is any suspended traffic on that particular integration.
The possibility to show detailed information for a specific integration on what artifacts (ports, host instances etc) that are/aren’t running. How much traffic that’s been sent/received via the integration. When traffic was sent/received and if there’s any suspended messages on the integration.
The option to filter exclude specific artifacts from monitoring (for example receive locations that’s usually turned off etc).
Setting up monitoring by for example email and also define what integrations to be included in one specific monitoring (different persons are usually responsible for monitoring different integrations).

Introducing “BizMon”

Based on the needs and “requirements” above I’ve started developing a application. The idea is to release it as open-source as soon as I get to a first stable version (I’d be very interested in help on practical details on how to do so). For now I’ll demonstrate it by showing some screenshots. The application is a web application based on ASP.NET MVC.

Screenshot: “Applications” page

The above image shows a screenshot from the start page of the BizMon-application that shows the aggregated status of the entire BizTalk group it’s connected to. The applications is build to monitor one BizTalk group and the shown page displays all applications within that BizTalk group.

In the example image the two first rows have an OK status. That means that all of the monitored artifacts (receive locations, send ports, orchestrations and host instances) within that application are in a running and OK status.
The yellow line on the YIT.NO.Project-application indicates a warning. That means that all the artifacts are in a OK status but there’re suspended messages within that application. The red line indicates that one or more of the monitored artifacts are in a inactive status.

Each row and application show when the last message on that application was received and/or sent. It also show how many suspended messages exists and when the last message got suspended.

Screenshot: “Application-detail” page

When clicking on a application on the main page previously shown the application-detail page is displayed for that application. This page shows detailed information on each of the artifacts within that application. I also shows suspended messages and the date and time of the last suspended.

It also displays a graph showing how many messages that has been processed by each of the ports. Currently the graph can view data from the last 7 days. In the screenshot above data from the 6th of January is shown and as it’s set to display data for a specific day the data is grouped in hours of that day. It’s also possible to view the aggregated data from all the traced days as show below. When viewing data from all days the graphs is grouped by days.

(The graph only shows data from the 6th of January as this is from test and there was no traffic of the previous days but I’m sure you get the idea …)

Screenshot: “Application-detail” page with inactive artifacts

This final page show details of an application with some inactive artifacts. The small cross highlighted by the arrow in the image show the possibility to filter out a single artifact from monitoring. If an excluded artifacts is failing the overall status of the application will still be OK and no alerts will be sent.

Help!

I’d love to get some input and feedback on all this. What do you think could be useful, what do you think won’t? Do you know of something similar, how do you solve this kind of monitoring?

I’d also like to know any suitable placed to publish the code as an open-source project or is the best thing to just release it here on the blog? What do you think? Use the comments or send me a mail.

What’s next?

I have a few thing on the alerts part of the application left and then I’ll release a first version. I’m hoping that could happened at the ~~end of February 09~~ (look at the update at the top of the post) . Make sure to let me know what you think!

I’ll publish a follow-up post discussing the technical details and the architecture more in detail shortly.

Handle the 'bodyTypeAssemblyQualifiedName' SOAP Adapter bug in MSBuild as a RegEx ninja

2008-11-12T12:53:32+00:00

This is a very specific problem but I’m sure some of you stumbled over it. When disassembling a XML message in a SOAP port BizTalk can’t read the message type. This causes problems when for example trying to handle an envelope message and split it to smaller independent messages in the port. It’s a known problem discussed here and here (you also find information about it in the BizTalk Developer’s Troubleshooting Guide) and the solution is to make a small change in the generated web service class. Below is a small part of he generated class.

//[cut for clarity] ...
            Microsoft.BizTalk.WebServices.ServerProxy.ParamInfo[] outParamInfos = null;
            string bodyTypeAssemblyQualifiedName = "XXX.NO.XI.CustomerPayment.Schemas.r1.CustomerPayments_v01, XXX.NO.XI.CustomerPaym" +
                "ent.Schemas.r1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ac564f277cd4488" +
                "e";
            // BizTalk invocation
            this.Invoke("SaveCustomerPayment", invokeParams, inParamInfos, outParamInfos, 0, bodyTypeAssemblyQualifiedName, inHeaders, inoutHeaders, out inoutHeaderResponses, out outHeaderResponses, null, null, null, out unknownHeaderResponses, true, false);
        }
    }
}

Basically the problem is that the generated code puts the wrong _DocumentSpecName _property in the message context. I’ll not dicusses the problem in detail here but Saravana Kumar does thorough dissection of the problem in his post on it.

The solution is to update the bodyTypeAssemblyQualifiedName to set a null value. That will cause the XmlDiassasemler to work as we’re used to and expect.

> > If the value _null_ is passed instead of _bodyTypeAssemblyQualifiedName_, SOAP adapter won't add the _DocumentSpecName_ property to the context. Now, when we configure our auto-generated SOAP_ReceiveLocation_ to use _XmlReceive_ pipeline, the _XmlDisassembler_ component inside _XmlReceive_ will go through the process of automatic dynamic schema resolution mechanism, pick up the correct schema and promotes all the required properties (distinguished and promoted) defined in the schema and it also promotes the _MessageType_ property. > > > > **From:** [http://www.digitaldeposit.net/saravana/post/2007/08/17/SOAP-Adapter-and-BizTalk-Web-Publishing-Wizard-things-you-need-to-know.aspx](http://www.digitaldeposit.net/saravana/post/2007/08/17/SOAP-Adapter-and-BizTalk-Web-Publishing-Wizard-things-you-need-to-know.aspx) > >

//[cut for clarity] ...
            Microsoft.BizTalk.WebServices.ServerProxy.ParamInfo[] outParamInfos = null;
            string bodyTypeAssemblyQualifiedName = null;
            // BizTalk invocation
            this.Invoke("SaveCustomerPayment", invokeParams, inParamInfos, outParamInfos, 0, bodyTypeAssemblyQualifiedName, inHeaders, inoutHeaders, out inoutHeaderResponses, out outHeaderResponses, null, null, null, out unknownHeaderResponses, true, false);
        }
    }
}

But if you have an automated deployment process you probably use MSBuild to generate your Web Services. Then is soon becomes _very _annoying to remember to update the .cs-file again and again for every deployment. So how can we script that update?

First we need to find a regular expression to find the right values. With some help from StackOverflow (let’s face it, there are some crazy regular expressions skills out there …) I ended up on the following.

(?<=string\sbodyTypeAssemblyQualifiedName\s=\s)(?s:[^;]*)(?=;)

If you’re not a RegEx ninja the line above does something like this:

After the string “string bodyTypeAssemblyQualifiedName = “
turn on single line (treat “\r\n” as any other character) ( this is what “(?s: )” does)
match every character that is not a semicolon
until a single semicolon is reached.

Then I used a task from the SDC Task library (you probably already use this if you’re using MSBuild and BizTalk). More specially we use the File.Replace

<Target Name="FixSOAPServiceCode">
    <File.Replace
            Path="$(WebSiteServicePath)CustomerPaymentService\App_Code\CustomerPaymentService.asmx.cs"
            Force="true"
            NewValue="null"
            RegularExpression="(?&lt;=string\sbodyTypeAssemblyQualifiedName\s=\s)(?s:[^;]*)(?=;)">
    </File.Replace>
</Target>

Now this task is part of the build script and called right after the tasks that generates the web service. This saves me a lot of manual work and potential errors!

Using BizUnitExtensions to poke around in some XML

2008-10-30T14:59:17+00:00

I’ll start by saying that I really (like in “really, really!”) like BizUnit! BizUnit in combination with MSBuild, NUnit and CruiseControl.NET has really changed the way work and how I feel about work in general and BizTalk development in particular.

If you haven’t started looking into what for example MSBuild can do you for you and your BizTalk build process you’re missing out on something great. The time spent on setting up a automatic build process is time well spent - take my word for it!

But build processes isn’t was this post is supposed to be about. This post is about one of the few limitations of BizUnit and the possibilities to work around one of those in particular.

BizUnit is a test framework that is intended to test Biztalk solutions. BizUnit was created by Kevin.B.Smith and can be found on [this CodePlex space](http://www.codeplex.com/bizunit) . BizUnit has quite a significant number of steps that have nothing to do with Biztalk per se and can be used for any integration project testing. > >

[![note](../assets/2008/10/windowslivewriterusingbizunitextensionstopokesomexml-d286note-thumb.gif)](../assets/2008/10/windowslivewriterusingbizunitextensionstopokesomexml-d286note-2.gif)If you have used BizUnit before and need an introduction before reading further, start [here](http://www.codeproject.com/KB/biztalk/BizUnit2006.aspx), [here](http://www.codeplex.com/bizunit) or [here](http://biztalkia.blogspot.com/2007/03/getting-started-with-nunit-and-bizunit.html) - they're all excellent articles. > >

As stated BizTalk has quite a significant number of steps but to my knowledge it’s missing a step to change and update file from within the test script. This step and a couple of other are added in separate fork-project to BizUnit called BizUnitExtensions.

This project [_BizUnitExtension_] aims to provide some more test step libraries, tools and utilities to enhance the reach of BizUnit. Here you can find some enhancements/extensions to the steps in the base libraries , new steps, support applications, tutorials and other documentation to help you understand and use it....This project is currently owned and contributed to by Santosh Benjamin and Gar Mac Críostaand. Our colleagues have also contributed steps and suggestions. We welcome more participation and contributions. > >

Amongst other steps (some for Oracle DBs etc) BizUnitExtensions adds a XmlPokeStep!

**XmlPokeStep:** This step is modelled on the lines of the NAnt XmlPoke task The XmlPokeStep is used to update data in an XML file with values from the context This will enable the user to write tests which can use the output of one step to modify the input of another step. > >

A cool thing about BizUnitExtensions is that it really just extends BizUnit. You’ll continue to run on the BizUnit dll:s when you use steps form BizUnit and just use the BizUnitExtensions code when you actually use some of steps from that library.

The example below shows how we first use an ordinary BizUnit task to validate and read a value from a file. We then use BizUnitExtension to gain some new powers and update the file with that value we just read.

<!–Ordinary BizUnit step to validate a file –>
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileValidateStep">
    <Timeout>5000</Timeout>
    <Directory>..\..\ReceiveRequest</Directory>
    <SearchPattern>*Request.xml</SearchPattern>
    <DeleteFile>false</DeleteFile>

    <ContextLoaderStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.XmlContextLoader">
        <XPath contextKey="messageID">/*[local-name()=’SystemRequest’]/ID</XPath>
    </ContextLoaderStep>

</TestStep>

<!–Use BizUnitExtensions to poke the value and change it –>
<TestStep assemblyPath="BizUnitExtensions.dll" typeName="BizUnit.Extensions.XmlPokeStep">
    <InputFileName>..\..\SystemResponse.xml</InputFileName>
    <XPathExpressions>
        <Expression>
            <XPath>/*[local-name()=’SystemResponse’]/ID</XPath>
            <NewValue takeFromCtx="messageID"></NewValue>
        </Expression>
    </XPathExpressions>
</TestStep>

This of course means that all you need to extend you current test steps and gain some cool new abilities is to add another assembly to you test project

Using BizUnitExtensions in a “real” scenario

An scenario when this can be useful is the following example were we need to test some message correlation.

A message request is received via a web service.
The message is sent to a queue via an orchestration in BizTalk. To be able to correlate the response a message id is added to the message request sent to the back-end system.
A message response is sent from the back-end system using a second queue. The response message contains the same message id as the incoming request contained.
BizTalk correlates the message back to the web service using the message id.

So how can we now test this? The steps should be something like the below.

> > [![note](../assets/2008/10/windowslivewriterusingbizunitextensionstopokesomexml-d286note-thumb-1.gif)](../assets/2008/10/windowslivewriterusingbizunitextensionstopokesomexml-d286note-4.gif) Notice that we read and write to the file system in the example. Once deployed to test these send ports and receive location will be reconfigured to use the queuing adapter. But for testing the scenario the file system works just fine a simplifies things IMHO. > >

Send a request message using BizUnit and the HttpRequestResponseStep. Make sure it runs concurrently with the other steps and then wait for a response (using the _runConcurrently-_attribute on the step).
Configure the send port so the orchestration that added the generated message id writes the message to a folder. Use the FileValidateStep and a nested XmlContextLoader to read the message from the folder and write the message id the context.
Use the context and the XmlPokeStep from BizUnitExtensions to update a response message template with the message id from the request message (this is of course needed so we can correlate the response message back to the right orchestration).
Copy the update response message template using the FileCreateStep to the folder that is monitored by the the receive location used for reading responses.

<TestCase testName="TestCorrelationTest">
    <TestSetup>
        <!–Clean up!–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
            <Directory>..\..\ReceiveRequest</Directory>
            <SearchPattern>*.*</SearchPattern>
        </TestStep>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
            <Directory>..\..\SendResponse</Directory>
            <SearchPattern>*.*</SearchPattern>
        </TestStep>
    </TestSetup>

    <TestExecution>
        <!–Post a request message on the SOAP port. Run it Concurrently–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.HttpRequestResponseStep" runConcurrently="true">
            <SourcePath>..\..\WebRequest.xml</SourcePath>
            <DestinationUrl>http://localhost:8090/BizTalkWebService/WebService1.asmx?op=WebMethod1</DestinationUrl>
            <RequestTimeout>15000</RequestTimeout>
        </TestStep>

        <!–Read the system request message and read the generaed id to the context–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileValidateStep">
            <Timeout>5000</Timeout>
            <Directory>..\..\ReceiveRequest</Directory>
            <SearchPattern>*Request.xml</SearchPattern>
            <DeleteFile>false</DeleteFile>
            <ContextLoaderStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.XmlContextLoader">
                <XPath contextKey="messageID">/*[local-name()=’SystemRequest’]/ID</XPath>
            </ContextLoaderStep>
        </TestStep>

        <!–If we have the file in source control it might be read-only -> remove that attribute–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.ExecuteCommandStep">
            <ProcessName>attrib</ProcessName>
            <ProcessParams>SystemResponse.xml -r</ProcessParams>
            <WorkingDirectory>..\..\</WorkingDirectory>
        </TestStep>

        <!–Update our response template (using BizUnitExtensions) and add the message id that we read into the the context–>
        <TestStep assemblyPath="BizUnitExtensions.dll" typeName="BizUnit.Extensions.XmlPokeStep">
            <InputFileName>..\..\SystemResponse.xml</InputFileName>
            <XPathExpressions>
                <Expression>
                    <XPath>/*[local-name()=’SystemResponse’]/ID</XPath>
                    <NewValue takeFromCtx="messageID"></NewValue>
                </Expression>
            </XPathExpressions>
        </TestStep>

        <!–Wait a moment so we don’t copy the file until we’re done updating it–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.DelayStep">
            <Delay>1000</Delay>
        </TestStep>

        <!–Copy the file to the folder that monitored by the receive location for opicking up system responses–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileCreateStep">
            <SourcePath>..\..\SystemResponse.xml</SourcePath>
            <CreationPath>..\..\SendResponse\SystemResponse.xml</CreationPath>
        </TestStep>
    </TestExecution>
</TestCase>

This might look messy at first but I think it’s really cool I also think it worth thinking about on how you should run this at during development otherwise? You would then have to build some small stub to return a response message with the right id … I prefer this method!

BizUnitExtension makes me like BizUnit even more! Thanks to Kevin. B. Smith, Santosh Benjamin and Gar Mac Críostaand for spending so much time on this and sharing it with us mere mortals!

Update: Gar’s blog can be found here.

How the extend a custom Xslt in BizTalk using EXSLT and the Mvp.Xml project

2008-09-05T20:15:55+00:00

Lately I’ve been using custom Xslt more and more instead of the BizTalk mapping tool. I still use the mapping tool in easy scenarios when I just need to do some straight mapping or maybe even when I need to concatenate some fields, but as soon as I need to to some looping, grouping, calculations etc I’ve made a promise to myself to use custom Xslt!

I find custom Xslt so much easier in more complex scenarios and once one get past the template matching and understands how and when to use recursion (No you can’t reassign a variable in Xslt and you’re not supposed to!) I find it to be a dream compared to the mapping tool. I also find the code so much easier to maintain compared to the result from the mapping tool. I mean someone would have to pay me good money to even start figuring out what this map is doing. And the scary thing is that if you worked with BizTalk for a while you probably know that maps like this isn’t that rare! I’ve even seen worse!

Don’t get me wrong, Xslt definitely has some major limitations.

Some of the acute limitations of XSLT 1.0 I can think of off the top of my head are: > > * The lack of real string comparison > * No support for dates > * No simple mechanism for grouping > * No querying into RTF's >

And it doesn’t take long before one runs up against one of these and suddenly you wish you were back in mapping tool were we just could add scripting functoid and some code or a external assembly. But then you remember … (Sorry, I know it’s painful just to watch it).

**There has to be a better way of doing this and combining the best out of the two worlds! **

I started looking into to how BizTalk actually solves combining Xslt and the possibility to use external assemblies. After a couple of searches I found Yossi’s nice article that explained it to me (from 2005! I’m behind on this one!) and it even turns out that there an example in the BizTalk SDK.

Ok, so now I had what I need. I started a new class library project and began writing some date parsing methods, some padding methods and so on.

It somehow however felt wrong from the start and I got this grinding feeling that I must be reinventing the wheel (I mean these are well know limitations of Xslt and must have been solved before). Even worse I also felt that I was creating a stupid single point of failure as I started using the component from all different maps in my projects and I have actually seen how much pain a bug in similar shared dll:s could cause. Basically a small bug in the component could halt all the process using the library! Finally I realized that this kind of library would be under constant development as we ran into more and more areas of limitations in the our Xslt:s and that would just increase the risk of errors and mistakes.

After some further investigation I found EXSLT which looked like a solution to my problems! A stable, tested library of Xslt extensions that we could take dependency on as it’s unlikely to have any bugs and that should include the functionality we’re missing in standard Xslt!

How I used EXSLT in BizTalk

These days it’s the Xml Mvp crowd over at the Mvp.Xml project who develops and maintains the .NET implementation of EXSLT. So I downloaded the latest binaries (version 2.3). Put the the Mvp.Xml.dll in the GAC. Wrote a short custom extension Xml snippet that looked like this (using what I’ve learnt from Yossi’s article).

<?xml version="1.0" encoding="utf-8"?>
<ExtensionObjects>
    <ExtensionObject
     Namespace="http://exslt.org/dates-and-times"
     AssemblyName="Mvp.Xml, 
     Version=2.3.0.0, Culture=neutral, 
     PublicKeyToken=6ead800d778c9b9f"
     ClassName="Mvp.Xml.Exslt.ExsltDatesAndTimes"/>
</ExtensionObjects>

All you define is the Xml namespace _you like to use in your Xslt to reference the dll, the _full assembly name and finally the name of the class in Mvp.Xml.Exslt you want to use (make sure you also download the source to Xml.Mvp, it helps when looking up in what classes and namespaces different methods are placed).

That means you need one ExtensionObjects block for each class you want you use which really isn’t a problem as the methods are nicely structured based on there functionality.

Then we can use this in a Xslt like this:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:S1="http://ExtendedMapping.Schema1"
                xmlns:S2="http://ExtendedMapping.Schema2"
                xmlns:exslt="http://exslt.org/dates-and-times"
                version="1.0"> 

    <xsl:template match="/">
        <S2:Root>
            <Field>
                <xsl:value-of select="exslt:dateTime()"/>
            </Field>
        </S2:Root>
    </xsl:template>
</xsl:stylesheet>

Which gives us the below output. Notice the current time and date! Cool!

<S2:Root xmlns:S1="http://ExtendedMapping.Schema1" xmlns:S2="http://ExtendedMapping.Schema2" xmlns:exslt="http://exslt.org/dates-and-times">
  <Field>2008-09-05T20:45:13+02:00</Field> 
</S2:Root>

All you then have to do in you map is to reference the Xslt and the extension Xml.

Just as final teaser I’ll paste a few methods from the EXSLT documentation

Some string methods:

str:align()
str:concat()
str:decode-uri()
str:encode-uri()
str:padding()
str:replace()
str:split()
str:tokenize()

Some date and time methods:

date:add()
date:add-duration()
date:date()
date:date-time()
date:day-abbreviation()
date:day-in-month()
date:day-in-week()
date:day-in-year()
date:day-name()
date:day-of-week-in-month()
date:difference()
date:duration()
date:format-date()
date:hour-in-day()
date:leap-year()
date:minute-in-hour()
date:month-abbreviation()
date:month-in-year()
date:month-name()
date:parse-date()
date:second-in-minute()
date:seconds()
date:sum()
date:time()
date:week-in-month()
date:week-in-year()
date:year()

As if this was enough (!) the Mvp Xml project added a couple of there own methods! What about string lowercase and _string uppercase - _all in Xslt! And about 30 new date-time related methods extra to the standard ones already in EXSLT!

Check out the full documentation here!

Let me know how it works out for you.

Efficient grouping and debatching of big files using BizTalk 2006

2008-07-16T10:06:06+00:00

I’ve seen people struggle both on the forums and while doing consulting when in it comes to finding an good way of grouping and transforming content in file before debatching it. Say for example we have a text file like the example below.

0001;Test row, id 0001, category 10;10 
0002;Test row, id 0002, category 10;10
0003;Test row, id 0003, category 10;10
0004;Test row, id 0004, category 20;20
0005;Test row, id 0005, category 20;20
0006;Test row, id 0006, category 20;20
0007;Test row, id 0007, category 20;20
0008;Test row, id 0008, category 10;10
0009;Test row, id 0009, category 10;10
0010;Test row, id 0010, category 30;30

Notice how the the ten rows belong to three different categories (10,20 and 30). These kind of export are in my experience quite common batch export from legacy systems and they usually aren’t ten rows (in my last project the sizes ranged from 5 MB to 25 MB) …

The problem

The problem is that the receiving system expects the data to be in separate groups, grouped by the categories the rows belong to. The expected message might look something like the below for category 10 (notice how all rows within the group are from category 10)

<ns1:Group numberOfRows="5" xmlns:ns1="http://Blah/Group">
  <Row>
    <Id>0001</Id>
    <Text>Test row, id 0001, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0002</Id>
    <Text>Test row, id 0002, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0003</Id>
    <Text>Test row, id 0003, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0008</Id>
    <Text>Test row, id 0008, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0009</Id>
    <Text>Test row, id 0009, category 10</Text>
    <Category>10</Category>
  </Row>
</ns1:Group>

The problem is now that we need to find a efficient way of first grouping the incoming flat file based message and then to _debatch it using those groups. Our ultimate goal is to have separate messages that groups all rows that belongs to the same category and then send these messages to the receiving system. _How would you solve this?

I’ve seen loads of different solution involving orchestrations, databases etc, but the main problem they all had in common is that they’ve loaded up to much of the message in memory and finally hit an OutOfMemoryException.

The solution

The way to solve this is to use pure messaging as one of the new features in BizTalk 2006 is the new large messages transformation engine.

> > **Large message transformation.** In previous versions of BizTalk Server, mapping of documents always occurred in-memory. While in-memory mapping provides the best performance, it can quickly consume resources when large documents are mapped. In BizTalk Server 2006, large messages will be mapped by the new large message transformation engine, which buffers message data to the file system, keeping the memory consumption flat. > >

So the idea is the to read the incoming flat file, use the Flat File Disassembler to transform the message to it’s XML representation (step 1,2 and in the figure below) and the to use XSLT to transform in to groups (step 4 and 5). We will then use the XML Disassembler to split those groups into separate messages containing all the rows within a category (step 6 and 7).

Step 1, 2 and 3 are straight forward and pure configuration. Step 4 and 5 will require some custom XSLT and I’ll describe that in more detail in the section below. Step 6 and 7 will be discussed in the last section of the post.

Grouping

Let’s start by looking at a way to group the message. I will use some custom XSLT and a technique called the Muenchian method. A segment from the XML representation of the flat file message could look something like this.

<Rows xmlns="http://Blah/Incoming_FF">
    <Row xmlns="">
        <ID>0001</ID>
        <Text>Test row, id 0001, category 10</Text>
        <Category>10</Category>
    </Row>
    <Row xmlns="">
        <ID>0002</ID>
        <Text>Test row, id 0002, category 10</Text>
        <Category>10</Category>
    </Row>
...
[message cut for readability]

The XSLT will use could look something like the below. It’s kind of straight forward and I’ve tried commenting the important parts of in the actual script. Basically it will use keys to fins the unique categories and then (again using keys) selecting those rows within the category to loop and write to a group.

<?xml version="1.0" encoding="utf-8"?>

<xsl:stylesheet version="1.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:ns1="http://GroupAndDebatch.Schemas.Incoming_FF"
                xmlns:ns2="http://GroupAndDebatch.Schemas.Grouped"
                >
    <!--Defining the key we're gonna use-->
    <xsl:key name="rows-by-category" match="Row" use="Category" />

    <xsl:template match="/ns1:Rows">
        <ns2:Groups>
        
        <!--Looping the unique categories to get a group for-->
        <xsl:for-each select="Row[count(. | key('rows-by-category', Category)[1]) = 1]">
            
            <!--Creating a new group and set the numberOfRows-->
            <Group numberOfRows="{count(key('rows-by-category', Category))}">
            
            <!--Loop all the rows within the specific category we're on-->
            <xsl:for-each select="key('rows-by-category', Category)">
                <Row>
                    <ID>
                        <xsl:value-of select="ID"/>
                    </ID>
                    <Text>
                        <xsl:value-of select="Text"/>
                    </Text>
                    <Category>
                        <xsl:value-of select="Category"/>
                    </Category>
                </Row>
            </xsl:for-each>
            </Group>
        </xsl:for-each>
        </ns2:Groups>
    </xsl:template>

</xsl:stylesheet>

> > [![note](../assets/2008/07/windowslivewriterefficientgroupingandsplittingofbigfiles-8295note-thumb.gif)](../assets/2008/07/windowslivewriterefficientgroupingandsplittingofbigfiles-8295note-2.gif)You have found all the [XSLT and XML related features](http://msdn.microsoft.com/en-us/library/aa302298.aspx) in Visual Studio - right? > >

Ok, so the above XSLT will give us a XML structure that looks some like this.

<?xml version="1.0" encoding="utf-8"?>
<ns2:Groups xmlns:ns2="http://Blah/Groups" xmlns:ns1="http://Blah/Group">
    <ns1:Group numberOfRows="5">
        <Row>
            <ID>0001</ID>
            <Text>Test row, id 0001, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0002</ID>
            <Text>Test row, id 0002, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0003</ID>
            <Text>Test row, id 0003, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0008</ID>
            <Text>Test row, id 0008, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0009</ID>
            <Text>Test row, id 0009, category 10</Text>
            <Category>10</Category>
        </Row>
    </ns1:Group>
    <ns1:Group numberOfRows="4">
        <Row>
            <ID>0004</ID>
            <Text>Test row, id 0004, category 20</Text>
            <Category>20</Category>
        </Row>
...
[message cut for readability]

Finally! This we can debatch!

Debatching

Debatch the Groups message above is also rather straight forward and I won’t spend much time on in this post. The best way to learn more about it is to have a look ate the EnvelopeProcessing sample in the BizTalk SDK.

And the end result of the debatching are single messages within a unique category, just as the receiving system expects! Problem solved.

Issue #1 - slow transformations

The first time I’ve put a solution like this in test and started testing with some real sized messages (> 1 MB) I really panicked, the mapping took forever. And I really mean forever, I sat there waiting for 2-3 hours (!) for a single file getting transformed. When I had tested the same XML based file in Visual Studio the transformation took about 10 seconds so I knew that wasn’t it. With some digging here I found the TransformThreshold parameter.

TransformThreshold decides how big a message can be in memory before BizTalk start buffering it to disk. The default value is 1 MB and one really has to be careful when changing this. Make sure you thought hard about your solution and situation before changing the value - how much traffic do you receive and how much of that can you afford reading in to memory?

In my case I received a couple of big files spread out over a night so setting parameter with a large amount wasn’t really a problem and that really solved the problem. The mapping finished in below 10 minutes as I now allow a much bigger message to be read into memory and executed in memory before switching over to the large message transformation engine and start buffering to disk (which is always much slower).

Problem #2 - forced to temp storage

Looking at the model of the data flow again you probably see that I’m using the XML Disassembler to split the grouped files (step 5 to step 6).

The only way I’ve found this to work is actually to write the Grouped XML message to file and the to read that file in to BizTalk again and in that receive pipeline debatch the message. Not the most elegant solution, but there really isn’t a another out-of-the-box way of debatching messages (the XML Assembler can’t do it) and I don’t want to use an orchestration to execute the a pipeline as I want to keep the solution pure messaging for simplicity and performance reasons.

Finishing up

Have you solved similar cases differently? I’d be very interested in your experience! I also have a sample solution of this - just send me an email and make sure you’ll get it.

Update

Also don’t miss this issue (pdf) of BizTalk Hotrod magazine. There is an article on “Muenchian Grouping and Sorting using Xslt” describing exactly the problem discussed above.

Running MSBuild scripts from Visual Studio

2008-07-09T09:04:22+00:00

It seems like there more build script one writes, the more often one wants to run them and it’s always a bit annoying (and time consuming) having to leave Visual Studio and start MSBuild from the command line. Brennan Stehling has a cool solution to that problem here were he sets up MSBuild as an external tool and runs it.

One problem for us was the we had our solution files in one place on the file system and our build files in a totally different place. The solution was to add the build file for the current solution as a Solution Folder (as shown in the figure below) and then set MSBuild to use $(ItemDir) as its Initial Directory. That will kick of MSBuild from the directory that the current selected Solution Folder points to and in our case that’s were the XXX.Build.Article.proj file exists.

Looking forward to stackoverflow.com

2008-07-09T06:52:12+00:00

I’m really looking forward to the release of stackoverflow.com - have you heard about it? If not it’s a collaboration between Joel Spolsky and Jeff Atwood. Joel and Jeff are of course the two blogger behind the popular Joel on Software and Coding Horror blogs.

Joel is also one of the founders of the FogCreek Software company. FogCreek has a couple of different products but the most known is probably FogBugz which is a really cool project management/bug tracking/wiki/support tool for software development and management. At first it might not look like a very impressing tool but have a look at this presentation by Joel and I think you’ll start thinking different (I know we did and we’re now using it for both planing and managing big integrations projects).

Anyway, if you missed the story behind stackoverflow have a look here and don’t miss the podcasts that Joel and Jeff record, some of them are really cool. I know I’ve learnt a lot by just listening in to their conversations.

We not have quite a few online BizTalk communities such as the Microsoft forums, Google Groups, BizTalk Gurus, facebook (search for BizTalk related groups) and at LinkedIn (a couple of open BizTalk related groups) - is there room for another one at stackoverflow? Do we need one?

Removing XML namespaces - revisit

2008-06-26T07:25:33+00:00

I have a old post on removing XML namespace from outgoing messages using XSLT in a map on the send port. Removing XML namespace is usually a late requirement that shows up during integration tests with for example legacy systems that has problems reading XML and only finds XML namespaces messy and confusing and wants it removed.

The post recently got commented by Jeff Lynch (one of the codebetter.com bloggers) asking me why I just didn’t create a schema without any XML namespace in it to represent the outgoing schema (see figure below) and then map to that in the send port.

Say for example that we have incoming messages like the one below with namespaces.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">ns0:BlahRoot </span><span style="color: #FF0000; ">xmlns:ns0</span><span style="color: #0000FF; ">="http://Sample.BlahIncomingSchema"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
  </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">BlahNode</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">Test Value</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">BlahNode</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ns0:BlahRoot</span><span style="color: #0000FF; ">></span></div>

We’ve then defined a schema without namespace and map to that and get the following result.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">BlahRoot</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
  </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">BlahNode</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">Test Value</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">BlahNode</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">BlahRoot</span><span style="color: #0000FF; ">></span></div>

This method is of course much cleaner then any previous and it’s also more conceptually correct as the schema actually represents the contact between BizTalk and the receiving system (the contract is a message without namespace in it, not one with that we then remove).

The only problem with this kind of approach is that as BizTalk recognized the message type using a combination between root node and XML namespace we can’t have another schema with the BlahRoot root node without a defined XML namespace. Even if those two schemas would look totally different in structure and be two different message types BizTalk would be able to see the difference (to BizTalk they both be #BlahRoot message types).

Thanks Jeff for pointing this out to me.

Does BizTalk have man-boobs?

2008-06-09T07:14:04+00:00

I’ve just finished watching this webcast from QCon 2008 with Martin Fowler and Jim Webber. It’s basically their view on how integration middleware is used today and how we plan and implement our SOA and ESB project.

Their main point is that we use much to bloated middleware (BizTalk is mentioned as an example here) and that we should have a more agile approach to implement SOA and ESBs. They’ve used all the agile ideas (testing, isolated testable functionality, deliver small and often, continuous builds etc, etc) and applied them to integration - fair enough. I totally agree that trying to convert your complete enterprise architecture into a SOA architecture is a guaranteed failure. This is also something we heard for a while now from others as well.

I do also agree that BizTalk is a huge platform and that it isn’t perfect in all aspects. IMHO it does however give us some important advantages compared to a custom coded message bus and services. I’ll try and list a few of them below.

Fixed architecture
We don’t have invent the wheel every time. BizTalk is a product with an architecture that one have to learn and use. There are times when this is a pain (did I hear low latency and BizTalk persistence points?) but it’s also a huge kick start to all projects once you learnt it. Once you figured out how you use the products you’ll actually have something up and running in no time.

Isn’t an early delivery that we can test something good? I’m sure I can deliver a BizTalk based integration faster that some can using custom code when starting from scratch.

Drag-and-drop
There is a learning curve to BizTalk and all it’s tools but once one gotten over this one can move really fast, even without a deep understanding of .NET and software development (there are of course both pros and cons to this). I’ve seen projects with 50+ integration processes (to me that’s a big, complex project) where we actually used people fresh out of school, spent two weeks to teach them basic BizTalk and had them deliver critical parts of the projects. I’d like to see that happen custom coded ESB project with thousands lines of code …

You probably get a nice design and implementation if you can hire 10 top developers and a couple of architects, but that isn’t always possible.

Tools
**Does a custom code, lean approach, really scale in this scenario? Do you take the time to pause and build that management and configuration tool that you don’t get with a custom code project? I don’t say that we got the perfect view and control of our processes and messages in BizTalk but at least we got **some control. At least I got the BizTalk Administration Console to let me see how my different application are doing, what messages and process etc that got suspended. At least I got the BAM framework where I can configure a tracking and monitoring in no time (usually …) etc, etc.
**It works
**Say your implementing a process that receives purchase orders and that these orders might contain orders for a couple of millions dollars. Do you want to be developer that tells you boss that you think you might have lost a message due to a exception in your custom code? Of course you have 90% test coverage and continuous integration but you never tested for this exception case … I don’t want to be that developer/architect.

I don’t say don’t test. I’m very pro testing and I really feel that agile is the right approach. I’m just saying I need something tested and safe to build this super critical solutions on. Something that I know works and that I can be really productive on and start solving business cases from minute one.

What do you think? Does BizTalk have man-bobs and is that only a bad thing? And does Martin Fowler really have leather pants on?

Speaking at KNUG - Karlstad .NET User Group

2008-05-09T14:05:53+00:00

Yesterday I presented the MasterData Management using BizTalk 2006 R2 talk (I’ll soon have a post out with the presentation in English) I recently held at Developer Summit at the local .NET user group in Karlstad (KNUG).

KNUG is a new .NET user group that I actually helped start a couple of months ago. This meeting was the second meeting for the group. The meeting was attended by about 20 persons and we had two presentations on the agenda. Besides my own Thomas Heder showed the group some LINQ and how he and his colleagues uses LINQPad to develop and test there queries.

We also discussed future subjects, possible speakers and moving information on the group over to a Community Server driven site.

Does anyone have any experience on Community Server and how the feature set matches those need for running a user group (managing users, blogs, email lists, calendar etc)?

Using BAM for latency tracking in a BizTalk request response scenario

2008-04-28T13:35:15+00:00

This post will try and explain how BAM tracking can be used in SOAP based request response scenario in BizTalk 2006. It important to notice that some of the issues discussed in the post are specific to the SOAP adapter and are non-issues if the scenario would for example use the the WCF adapter or similar.

Describing the scenario

In this case we have a SOAP receive port that receives a request and returns a response. The request is routed to a orchestration that calls three different send ports. These ports then sends new requests to back-end systems and returns responses (communication with back-ends systems are also SOAP based). The three responses are used to build up the final response that is then returned to original receive port as a final response.

Our goal is to track the duration between the request and response on each of the ports. The idea is also to find a solution and tracking model that doesn’t have to change if we add or remove ports or add similar processes to track.

Defining and deploying the tracking model

We’ll start by defining our tracking model in Excel. Our activity contains of the following items:

InterchangeId (data item)
As we won’t correlate all the tracking point into one single row (that would break the goal of having one model for all processes, the model would then have to be specific for one process and it’s specific ports) the interchange id will then tell us what different rows belong together and that describes one process.
ReceivePortName (data item)
The name of the receive port.
Request (milestone item)
The time the request was either sent or received (depending on if we track a port that received/sent the request using a receive port or send port).
Response (milestone item)
The time the response was either sent or received (depending on if we track a port that received/sent it’s response on a receive port or send port).
SendPortName (data item)
The name of the send port.

After we described the model it’s time to export it to an XML representation and then to use the BM tool to deploy it and generate the BAM database infrastructure. You’ll find some nice info on this here.

Using the Tracking Profile Editor to bind the model

Next step is to bind the model to the implementation using the Tracking Profile Editor. The figure below shows the different properties that were used. Notice that none of the items was bound to the actual orchestration context. All properties are general properties that we track on the ports.** **This is important as that gives us the possibility to just add and remove ports to change the tracking.

** **

The next figure shows how the tracking of the request milestone event actually happens on either the RP1 port or on any of the three different send ports! If we developed a new process using other ports we could just add it here, no new model required.

What about the continuation then?

Our final problem is that unless we somehow correlate our request tracking point with our receive tracking point the receive we’ll end up with each tracking point spread over several different rows. In the example below I’ve marked the request for the RP01 port and the response event on the same port.

The reason for this is of course that BAM doesn’t have a context for the two tracking points and doesn’t know that actually belongs together. This differs from tracking in a orchestration were we always are in a context (the context of the orchestration), it’s then easy for BAM to understand that we like to view all the tracking point as one row – when tracking on ports it’s different. Continuation helps us tell BAM that we like have a context and correlate these two points.

In our case ServiceID is the prefect candidate for correlating the two points. A request and a response will have the same service id. In an other situation we could just as well have used a value from inside the message (say for example an invoice id).

The result is one single row for the request response for each port. So in our case a complete process (a complete interchange) is shown on four rows (one row for each of the ports). In the example below the first rows shows us the complete duration (and the other tracking data) between the request response to the client. The other rows show the duration for the send ports communication with the back-ends systems.

This model might not be optimal in an other scenario where your process are more fixed and you can then create a tracking model that is more specific to you actual process. But this solution meets our design goal as we’re now able to just add and remove port using the tracking profiler to track new port in completely new processes without having to go back and change the tracking model.

> > [![note](../assets/2008/04/windowslivewriterusingbamforlatencytrackinginabiztalkrequ-9c0bnote-thumb.gif)](../assets/2008/04/windowslivewriterusingbamforlatencytrackinginabiztalkrequ-9c0bnote-2.gif) NOTE: When configuring BAM to track a port the _[MessageType](http://msdn2.microsoft.com/en-us/library/aa561650.aspx)_ is actually promoted. This causes some problems in combination with the SOAP based ports that have been published using the Web Services Publishing Wizard. Saravana writes about this [here](http://www.digitaldeposit.net/blog/2007/08/soap-adapter-and-biztalk-web-publishing.html) and all his articles on this subject is a must read when working with SOAP ports. The problem however comes down to that the Web Services Publishing Wizard generates code that puts the wrong _DocumentSpecName _in the message context and that causes the _XmlDisassembler_ to fail (it tricks the _XmlDisassembler_ to look for a _MessageType_ that doesn't exists). > > > > This usually isn’t a problem (unless you like to use a map on a port) but as BAM will force the port to promote the _MessageType_ based on the _DocumentSpecName _we’ll have to fix this. Saravana has two solutions to the problem and I find the one that replaces the _DocumentSpecName_ with a null value and lets the _XmlDisassembler_ find the _MessageType_ to work well. > >

Speaking at Developer Summit on Masterdata Management using BizTalk 2006

2008-04-24T20:33:39+00:00

I recently spoke at the leading developer conference here in Sweden called Developer Summit. The talk was called Masterdata Management using BizTalk 2006. The slides can be found here.

I can really recommend Developer Summit as a conference. Everything is super well organized and having the opportunity to listen to celebrities like David Chappell, Jim Webber, Dan North and Christian Weyer in Sweden is really great!

This year I also liked the mix of presentation as some where more general presentation as for example Benjamin Ling who’s the director of platform at Facebook and who gave an insight to how the platform is managed and developed. I also really enjoyed (besides the obvious ones as for example Christian Weyer’s WCF talk) Frans Hanel’s talk on the history and architecture behind a major price comparison site called prisjakt.nu - very interesting and a great technical mix in otherwise Microsoft focused conference.

Writing BizTalk context properties to a message from a WCF service using behaviors

2008-04-24T17:43:28+00:00

The new WCF adapter in BizTalk 2006 R2 offers a lot of new possibilities. One of those is to write data to the BizTalk Message context properties directly from an exposed WCF Service. A practical use of this technic could be to write the username from the Windows credentials of the calling client into the context of the BizTalk message. This could be useful as this information is encrypted in messages that are received via the WCF adapter and isn’t possible to read when inside BizTalk. I’ll try and demonstrate the technique in this post.

If you have used the SOAP adapter before you might know that all you had to do was to turn on Windows based security for the exposed SOAP service and the username was automatically promoted to the context of the incoming BizTalk message. That username could then be used for routing, tracking which user called the service or using the value in plain text when communicating further to other connected systems. However using the WCF adapter this is not true anymore - when using the new WCF Message Security model the username and password is encrypted in the message and once the message is received by BizTalk it’s to late to read it. Basically we have to read the username in the actual service and write it into our own context property (that doesn’t get encrypted).

One way of achieving this is to read the username in the service and then to add it to the WCF Message Headers. All WCF message headers will by default be written to a the BizTalk Message context property called InboundHeaders (in the http://schemas.microsoft.com/BizTalk/2006/1/Adapters/WCF-properties namespace). First we’ll create an EndpointBehavior that will use a MessageInspector to add the username to the message header. Finally we create BehaviorExtensionElement so we can use a WCF Custom Binding in BizTalk and configure it to add our new behavior.

Creating the new EndpointBehavior

To create the configurable behavior we’ll need the three classes we mentioned above.

A class that implements the IDispatchMessageInspector interface to handle to reading and writing to the actual message.
A class that implements the IEndpointBehavior interface to define what kind of endpoint we’re creating and what it should do.
A class that implements the BehaviorExtensionElement abstract class to create the behavior and make it configurable.

<div><span style="color: #0000ff">using</span><span style="color: #000000"> System;
</span><span style="color: #0000ff">using</span><span style="color: #000000"> System.Collections.Generic;
</span><span style="color: #0000ff">using</span><span style="color: #000000"> System.Text;
</span><span style="color: #0000ff">using</span><span style="color: #000000"> System.ServiceModel;
</span><span style="color: #0000ff">using</span><span style="color: #000000"> System.ServiceModel.Channels;
</span><span style="color: #0000ff">using</span><span style="color: #000000"> System.ServiceModel.Dispatcher;
</span><span style="color: #0000ff">using</span><span style="color: #000000"> System.ServiceModel.Description;
</span><span style="color: #0000ff">using</span><span style="color: #000000"> System.ServiceModel.Configuration;

</span><span style="color: #0000ff">namespace</span><span style="color: #000000"> CustomWCFProperties.Behavior
{
    </span><span style="color: #808080">///</span><span style="color: #008000"> </span><span style="color: #808080"><summary></span><span style="color: #008000">
    </span><span style="color: #808080">///</span><span style="color: #008000"> PromoteUserNameMessageInspector implements IDispatchMessageInspector and adds the name from the WindowsIdentity to a WCF header called WindowsUserName in the </span><span style="color: #008000; text-decoration: underline">http://CustomWCFProperties.Schema</span><span style="color: #008000"> namespace. BeforeSendReply only returns as we're not interested in handling the response.
    </span><span style="color: #808080">///</span><span style="color: #008000"> </span><span style="color: #808080"></summary></span><span style="color: #808080">
</span><span style="color: #000000">    </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">class</span><span style="color: #000000"> PromoteUserNameMessageInspector : IDispatchMessageInspector
    {
        </span><span style="color: #0000ff">#region</span><span style="color: #000000"> IDispatchMessageInspector Members</span><span style="color: #000000">

        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">object</span><span style="color: #000000"> AfterReceiveRequest(</span><span style="color: #0000ff">ref</span><span style="color: #000000"> System.ServiceModel.Channels.Message request, System.ServiceModel.IClientChannel channel, System.ServiceModel.InstanceContext instanceContext)
        {
            </span><span style="color: #0000ff">string</span><span style="color: #000000"> windowsUserName </span><span style="color: #000000">=</span><span style="color: #000000"> ServiceSecurityContext.Current.WindowsIdentity.Name;
            request.Headers.Add(MessageHeader.CreateHeader(</span><span style="color: #000000">"</span><span style="color: #000000">WindowsUserName</span><span style="color: #000000">"</span><span style="color: #000000">, </span><span style="color: #000000">"</span><span style="color: #000000">http://CustomWCFProperties.Schema</span><span style="color: #000000">"</span><span style="color: #000000">, windowsUserName));
            </span><span style="color: #0000ff">return</span><span style="color: #000000"> </span><span style="color: #0000ff">null</span><span style="color: #000000">;
        }

        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">void</span><span style="color: #000000"> BeforeSendReply(</span><span style="color: #0000ff">ref</span><span style="color: #000000"> Message reply, </span><span style="color: #0000ff">object</span><span style="color: #000000"> correlationState)
        {
            </span><span style="color: #0000ff">return</span><span style="color: #000000">;
        }

        </span><span style="color: #0000ff">#endregion</span><span style="color: #000000">
    }

    </span><span style="color: #808080">///</span><span style="color: #008000"> </span><span style="color: #808080"><summary></span><span style="color: #008000">
    </span><span style="color: #808080">///</span><span style="color: #008000"> PromoteUserNameBehavior implements IEndpointBehavior and adds a message inspector to the dispatch behavior. Doesn't use any binding parameters, doesn't validate any configuration etc and can't be used in a client (only in a service).
    </span><span style="color: #808080">///</span><span style="color: #008000"> </span><span style="color: #808080"></summary></span><span style="color: #808080">
</span><span style="color: #000000">    </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">class</span><span style="color: #000000"> PromoteUserNameBehavior : IEndpointBehavior
    {
        </span><span style="color: #0000ff">#region</span><span style="color: #000000"> IEndpointBehavior Members</span><span style="color: #000000">

        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">void</span><span style="color: #000000"> AddBindingParameters(ServiceEndpoint endpoint, System.ServiceModel.Channels.BindingParameterCollection bindingParameters)
        {
            </span><span style="color: #0000ff">return</span><span style="color: #000000">;
        }

        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">void</span><span style="color: #000000"> ApplyClientBehavior(ServiceEndpoint endpoint, System.ServiceModel.Dispatcher.ClientRuntime clientRuntime)
        {
            </span><span style="color: #0000ff">throw</span><span style="color: #000000"> </span><span style="color: #0000ff">new</span><span style="color: #000000"> Exception(</span><span style="color: #000000">"</span><span style="color: #000000">The method or operation is not implemented.</span><span style="color: #000000">"</span><span style="color: #000000">);
        }

        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">void</span><span style="color: #000000"> ApplyDispatchBehavior(ServiceEndpoint endpoint, System.ServiceModel.Dispatcher.EndpointDispatcher endpointDispatcher)
        {
            endpointDispatcher.DispatchRuntime.MessageInspectors.Add(</span><span style="color: #0000ff">new</span><span style="color: #000000"> PromoteUserNameMessageInspector());
        }

        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">void</span><span style="color: #000000"> Validate(ServiceEndpoint endpoint)
        {
            </span><span style="color: #0000ff">return</span><span style="color: #000000">;
        }

        </span><span style="color: #0000ff">#endregion</span><span style="color: #000000">
    }

    </span><span style="color: #808080">///</span><span style="color: #008000"> </span><span style="color: #808080"><summary></span><span style="color: #008000">
    </span><span style="color: #808080">///</span><span style="color: #008000"> Defines the behavior.
    </span><span style="color: #808080">///</span><span style="color: #008000"> </span><span style="color: #808080"></summary></span><span style="color: #808080">
</span><span style="color: #000000">    </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">class</span><span style="color: #000000"> PromoteUserNameBehaviorElement : BehaviorExtensionElement
    {
        </span><span style="color: #0000ff">protected</span><span style="color: #000000"> </span><span style="color: #0000ff">override</span><span style="color: #000000"> </span><span style="color: #0000ff">object</span><span style="color: #000000"> CreateBehavior()
        {
            </span><span style="color: #0000ff">return</span><span style="color: #000000"> </span><span style="color: #0000ff">new</span><span style="color: #000000"> PromoteUserNameBehavior();
        }

        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">override</span><span style="color: #000000"> Type BehaviorType
        {
            </span><span style="color: #0000ff">get</span><span style="color: #000000"> { </span><span style="color: #0000ff">return</span><span style="color: #000000"> </span><span style="color: #0000ff">typeof</span><span style="color: #000000">(PromoteUserNameBehavior); }
        }
    }
}</span></div>

Finally we have to sign the assembly using a strong key and add it to the GAC.

Configure the machine.config

As we need BizTalk and the WCF adapter to pick up the need behavior and make it possible to configure our receive port we need to to add the behavior element to the machine.config. The easiest way of doing this is to use the new WCF Service Configuration Editor tool and point to the machine.config file.

After the dll been added and the machine.config file has been saved the the line below should have been added to the [ element](http://msdn2.microsoft.com/en-us/library/aa734726.aspx) (that is if you use the same strong name key as in the sample project I've linked here).

<div><span style="color: #0000ff"><</span><span style="color: #800000">add </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="addCustomWCFProperties"</span><span style="color: #ff0000"> type</span><span style="color: #0000ff">="CustomWCFProperties.Behavior.PromoteUserNameBehaviorElement, AddCustomWCFPropertiesBehavior, Version=1.0.0.0, Culture=neutral, PublicKeyToken=705e34637fdffc54"</span><span style="color: #ff0000"> </span><span style="color: #0000ff">/></span></div>

Create the BizTalk Receive Port and Receive Location

Next thing to do is to start the BizTalk WCF Service Publishing Wizard. Choose to publish a service endpoint and make sure you enable metadata and create a receive location. In this example we’ll next choose to “Publish schemas as WCF service” and then define our service by naming service operations and so on.

When you then browse to the URL you choose to publish your service to you’ll see the nice example of how to instance the service you just defined.

If we then send a request message to service (you’ll find a client as part of the attached solution here) and inspect the message and its context properties in BizTalk we’ll see that the username of the calling client is nowhere to be found.

Configure a WCF-Custom binding and adding a Endpoint Behavior

To add the username to the message context we’ll need to add our newly created behavior to our service. We’ll do this by switch the service over to use a WCF-Custom binding to enable configuration. We then need to add the URL in the address field, define the binding type to a wsHttpBinding and to add our addCustomeWCFProperties behavior to the list of endpoint behaviors.

> > [![note](../assets/2008/04/windowslivewriterwritingbiztalkcontextpropertiestoamessag-72f4note-thumb.gif)](../assets/2008/04/windowslivewriterwritingbiztalkcontextpropertiestoamessag-72f4note-2.gif) NOTE: there is a limitation in the BizTalk WCF implementation in that you can't create the WCF-Custom receive location that uses a HTTP in-process based binding (like the wsHttpBinding used in a WCF-Custom endpoint is) first and then use the WCF Publishing Wizard to only publish a metadata endpoint. > > > > [Richard Seroter writes about it here](http://www.topxml.com/code/cod-72_10213_biztalk-and-wcf-part-v-publishing-operations-patterns.aspx) and I found the same thing to be true. > > > > _"This error doesn’t have to do with mixing MEX endpoints and “regular” endpoints in the same IIS web site, but rather, creating MEX endpoints for in-process HTTP bindings seems to trigger this. **Note that an IIS-hosted MEX endpoint CAN be created for IIS-hosted HTTP endpoints, but not for in-process hosted HTTP endpoints."**_ > > > > If you however choose a different binding that Http or (as in this case) publishes the metadata first and then switches over to a custom binding you're ok. > >

If we then post another message to the service and inspect the message we’ll see that the behavior actually added a header and that it’s part of our BizTalk context properties. The adapter is also smart enough to know that this header isn’t part of the original headers and therefore stores in it’s own field within the context properties (you’ll find as part of the InboundHeaders block as well).

One problem remains - the actual value of the user is nested inside a XML node and the property isn’t promoted.

Extract and promote the value

To extract and promote the value we use an old fashion pipeline component using the following code in the execute method (the complete project is part of the downloadable sample project).

<div><span style="color: #000000">        </span><span style="color: #0000ff">public</span><span style="color: #000000"> IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
        {
            StringReader reader </span><span style="color: #000000">=</span><span style="color: #000000"> </span><span style="color: #0000ff">new</span><span style="color: #000000"> StringReader(inmsg.Context.Read(</span><span style="color: #000000">"</span><span style="color: #000000">WindowsUserName</span><span style="color: #000000">"</span><span style="color: #000000">, </span><span style="color: #000000">"</span><span style="color: #000000">http://CustomWCFProperties.Schema</span><span style="color: #000000">"</span><span style="color: #000000">).ToString());
            
            </span><span style="color: #0000ff">if</span><span style="color: #000000"> (reader </span><span style="color: #000000">!=</span><span style="color: #000000"> </span><span style="color: #0000ff">null</span><span style="color: #000000">)
            {
                XPathDocument document </span><span style="color: #000000">=</span><span style="color: #000000"> </span><span style="color: #0000ff">new</span><span style="color: #000000"> XPathDocument(reader);
                XPathNavigator navigator </span><span style="color: #000000">=</span><span style="color: #000000"> document.CreateNavigator();
                </span><span style="color: #0000ff">string</span><span style="color: #000000"> value </span><span style="color: #000000">=</span><span style="color: #000000"> navigator.SelectSingleNode(</span><span style="color: #000000">"</span><span style="color: #000000">/</span><span style="color: #000000">"</span><span style="color: #000000">).Value;
                inmsg.Context.Promote(</span><span style="color: #000000">"</span><span style="color: #000000">WindowsUserName</span><span style="color: #000000">"</span><span style="color: #000000">, </span><span style="color: #000000">"</span><span style="color: #000000">http://CustomWCFProperties.Schema</span><span style="color: #000000">"</span><span style="color: #000000">, value);
            }

            </span><span style="color: #0000ff">return</span><span style="color: #000000"> inmsg;
        }</span></div>

All the component does is reading the XML node the value exists inside and then it reads the actual value. Finally it writes the value back and promotes it. To be able to promote the value we also have to have a Property Schema deployed with a corresponding property name and namespace (WindowsUser and http://CustomeWCFProperties.Schema in this case).

The end results looks something like this.

The username is extracted and promoted and available for example for tracking or to for example use in a routing scenario.

This technique could of course be used for all kinds of scenarios where you like to add information to the context properties and could potentially replace a lot of the classic scenarios for custom pipelines.

All kind of comments are of course more than welcome!

Download the sample solution here.

Random thoughts on SQL Server Adapter and T-SQL Stored Procedures for BizTalk 2006

2008-02-18T22:36:06+00:00

Writing Stored Procedures is an art of its own. As all of you know it’s very different from writing ordinary code and presents its own kind of problems and issues when it comes to performance, builds, version control, testing etc. This post will try and highlight a few points that I find important when it comes to Stored Procedures, and the especially in a BizTalk and integration related scenario.

Consider the simplified procedure below, I’ll use is an example for much of the discussion in the rest of the post. Basically the procedure inserts or updates (based on if the optional EmployeeID exists from before or not) employee information to the Employee table of a database named “TestDB”.

<span class="kwrd"></span>



  




<div><span style="color: #0000ff">Use</span><span style="color: #000000"> </span><span style="color: #ff0000">[</span><span style="color: #ff0000">TestDB</span><span style="color: #ff0000">]</span><span style="color: #000000">
</span><span style="color: #0000ff">Go</span><span style="color: #000000"> 
</span><span style="color: #0000ff">If</span><span style="color: #000000"> </span><span style="color: #808080">Exists</span><span style="color: #000000">(</span><span style="color: #0000ff">select</span><span style="color: #000000"> </span><span style="color: #808080">*</span><span style="color: #000000"> </span><span style="color: #0000ff">From</span><span style="color: #000000"> dbo.sysobjects </span><span style="color: #0000ff">Where</span><span style="color: #000000"> id </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #ff00ff">object_id</span><span style="color: #000000">(N</span><span style="color: #ff0000">'</span><span style="color: #ff0000">AddEmployee</span><span style="color: #ff0000">'</span><span style="color: #000000">) </span><span style="color: #808080">And</span><span style="color: #000000"> </span><span style="color: #ff00ff">OBJECTPROPERTY</span><span style="color: #000000">(id, N</span><span style="color: #ff0000">'</span><span style="color: #ff0000">IsProcedure</span><span style="color: #ff0000">'</span><span style="color: #000000">) </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #800000; font-weight: bold">1</span><span style="color: #000000">) </span><span style="color: #0000ff">Drop</span><span style="color: #000000"> </span><span style="color: #0000ff">Procedure</span><span style="color: #000000"> AddEmployeeGo 
</span><span style="color: #0000ff">Create</span><span style="color: #000000"> </span><span style="color: #0000ff">Procedure</span><span style="color: #000000"> AddEmployee    
    </span><span style="color: #008000">@EmployeeId</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">Int</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #808080">-</span><span style="color: #800000; font-weight: bold">1</span><span style="color: #000000">,    
    </span><span style="color: #008000">@LastName</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">Varchar</span><span style="color: #000000">(</span><span style="color: #800000; font-weight: bold">30</span><span style="color: #000000">),    
    </span><span style="color: #008000">@FirstName</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">Varchar</span><span style="color: #000000">(</span><span style="color: #800000; font-weight: bold">30</span><span style="color: #000000">),    
    </span><span style="color: #008000">@MiddleName</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">Varchar</span><span style="color: #000000">(</span><span style="color: #800000; font-weight: bold">30</span><span style="color: #000000">)
</span><span style="color: #0000ff">As</span><span style="color: #000000"> 
</span><span style="color: #0000ff">Begin</span><span style="color: #000000">    
    </span><span style="color: #0000ff">Declare</span><span style="color: #000000"> </span><span style="color: #008000">@currentEmployeeId</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">int</span><span style="color: #000000">    
    </span><span style="color: #0000ff">Declare</span><span style="color: #000000"> </span><span style="color: #008000">@currentAction</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">tinyint</span><span style="color: #000000">    
    </span><span style="color: #008080">--</span><span style="color: #008080"> Check based on the EmployeeId if the employee exists from before    </span><span style="color: #008080">
</span><span style="color: #000000">    </span><span style="color: #0000ff">If</span><span style="color: #000000"> </span><span style="color: #808080">Not</span><span style="color: #000000"> </span><span style="color: #808080">Exists</span><span style="color: #000000">(</span><span style="color: #0000ff">Select</span><span style="color: #000000"> EmployeeId </span><span style="color: #0000ff">From</span><span style="color: #000000"> Employee </span><span style="color: #0000ff">Where</span><span style="color: #000000"> EmployeeId </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #008000">@EmployeeId</span><span style="color: #000000">)    
        </span><span style="color: #0000ff">Begin</span><span style="color: #000000">        
            </span><span style="color: #008080">--</span><span style="color: #008080"> Inserts new employee        </span><span style="color: #008080">
</span><span style="color: #000000">            </span><span style="color: #0000ff">Insert</span><span style="color: #000000"> </span><span style="color: #0000ff">Into</span><span style="color: #000000"> Employee (LastName,FirstName,MiddleName)         
            </span><span style="color: #0000ff">Values</span><span style="color: #000000"> (</span><span style="color: #008000">@LastName</span><span style="color: #000000">, </span><span style="color: #008000">@FirstName</span><span style="color: #000000">, </span><span style="color: #008000">@MiddleName</span><span style="color: #000000">)        
            </span><span style="color: #0000ff">Set</span><span style="color: #000000"> </span><span style="color: #008000">@currentEmployeeId</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #ff00ff">Scope_Identity</span><span style="color: #000000">()        
            </span><span style="color: #0000ff">Set</span><span style="color: #000000"> </span><span style="color: #008000">@currentAction</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #800000; font-weight: bold">1</span><span style="color: #000000">    
        </span><span style="color: #0000ff">End</span><span style="color: #000000">    
    </span><span style="color: #0000ff">Else</span><span style="color: #000000">    
        </span><span style="color: #0000ff">Begin</span><span style="color: #000000">        
            </span><span style="color: #008080">--</span><span style="color: #008080"> Updates employee        </span><span style="color: #008080">
</span><span style="color: #000000">            </span><span style="color: #0000ff">Update</span><span style="color: #000000"> Employee </span><span style="color: #0000ff">Set</span><span style="color: #000000"> LastName </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #008000">@LastName</span><span style="color: #000000">,        
            FirstName </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #008000">@FirstName</span><span style="color: #000000"> </span><span style="color: #0000ff">WHERE</span><span style="color: #000000"> EmployeeId </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #008000">@EmployeeId</span><span style="color: #000000">        
            </span><span style="color: #0000ff">Set</span><span style="color: #000000"> </span><span style="color: #008000">@currentEmployeeId</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #008000">@EmployeeId</span><span style="color: #000000">        
            </span><span style="color: #0000ff">Set</span><span style="color: #000000"> </span><span style="color: #008000">@currentAction</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #800000; font-weight: bold">2</span><span style="color: #000000">    
        </span><span style="color: #0000ff">End</span><span style="color: #000000">      

    </span><span style="color: #0000ff">Select</span><span style="color: #000000"> </span><span style="color: #008000">@currentEmployeeId</span><span style="color: #000000"> </span><span style="color: #0000ff">As</span><span style="color: #000000"> </span><span style="color: #ff0000">[</span><span style="color: #ff0000">@EmployeeId</span><span style="color: #ff0000">]</span><span style="color: #000000">, </span><span style="color: #008000">@currentAction</span><span style="color: #000000">     
    </span><span style="color: #0000ff">As</span><span style="color: #000000"> </span><span style="color: #ff0000">[</span><span style="color: #ff0000">@ProcedureAction</span><span style="color: #ff0000">]</span><span style="color: #000000"> </span><span style="color: #0000ff">For</span><span style="color: #000000"> Xml Path (</span><span style="color: #ff0000">'</span><span style="color: #ff0000">Response</span><span style="color: #ff0000">'</span><span style="color: #000000">)
</span><span style="color: #0000ff">End</span><span style="color: #000000">
</span><span style="color: #0000ff">Go</span></div>

Builds and version control

I always try to have the complete database version controlled. One solution to this that some people advocate is to have a central development database. I have however always found it to be a lot of hassle with overwriting each others changes and so on and always try to make it possible for each developer to build a local version of the database for development. It should also be possible to always check out the latest version and run it to build all the database objects - no matter what you have from before. This is of course a bit tricky when it comes to the actual tables as you might have data and so on (in my opinion it shouldn’t be the case during development but that’s another post discussing test data for the database etc).

When it however comes to Stored Procedure as in this post it’s easy - drop the old one if it exists and create the new one as in the example.

<div><span style="color: #0000ff">If</span><span style="color: #000000"> </span><span style="color: #808080">Exists</span><span style="color: #000000">(</span><span style="color: #0000ff">select</span><span style="color: #000000"> </span><span style="color: #808080">*</span><span style="color: #000000"> </span><span style="color: #0000ff">From</span><span style="color: #000000"> dbo.sysobjectsWhere id </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #ff00ff">object_id</span><span style="color: #000000">(N</span><span style="color: #ff0000">'</span><span style="color: #ff0000">AddEmployee</span><span style="color: #ff0000">'</span><span style="color: #000000">) </span><span style="color: #808080">And</span><span style="color: #000000"> </span><span style="color: #ff00ff">OBJECTPROPERTY</span><span style="color: #000000">(id, N</span><span style="color: #ff0000">'</span><span style="color: #ff0000">IsProcedure</span><span style="color: #ff0000">'</span><span style="color: #000000">) </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #800000; font-weight: bold">1</span><span style="color: #000000">) </span><span style="color: #0000ff">Drop</span><span style="color: #000000"> </span><span style="color: #0000ff">Procedure</span><span style="color: #000000"> AddEmployee
</span><span style="color: #0000ff">Go</span><span style="color: #000000">
</span><span style="color: #0000ff">Create</span><span style="color: #000000"> </span><span style="color: #0000ff">Procedure</span><span style="color: #000000"> AddEmployee </span><span style="color: #008000">@EmployeeId</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">Int</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #808080">-</span><span style="color: #800000; font-weight: bold">1</span><span style="color: #000000">, ...</span></div>

If found this article by Jeff Atwood over at Coding Horror (and the articles by K. Scott Allen it links to) very good on how to think about version control database objects.

Idempotent procedures

In complex integration solution it’s a common problem with multiple messages - sending systems might send the same message twice for a number of reasons and one will end up with a shaky system if that causes an error in the integration. I’ve also seen solutions to this problem where a check is done against the database (say from a BizTalk orchestration) and based on that result a decision is made to call an update or an insert procedure. In my opinion that just unnecessary complexity and database communication. I believe one should try and handle whether to update or insert inside the procedure as long as that’s possible and effective.

I in the AddEmployee Stored Procedure example might receive a EmployeeID as an input parameter and based on a check inside the procedure if the employee exists from before with that id a post is either updated or inserted.

Response message

I generally always want two things in return from a Store Procedure like the one in the AddEmployee example (In a select based procedure that would of course be very different as we like all the data in return).

The id that been added or changed
What action that’s been performed (insert or update)

What action that was performed is a good idea to return as it’s quite usual to use this in for example BAM tracking or other logging and so on (we might want know how many employees that was added versus updated for example).

Using the Xml Path syntax for generating the Xml response (thanks again Nick for a great post) from the procedure has made it even simpler to actually skip using the Add Generated Item option in BizTalk. All you have think about is to set name of the child node the request root node to the same name as the procedure (in this case AddEmployee). Also remember that the type should be set up as a Multi-Part Message Type in the orchestrations you use it from and that the **Receive Pipeline you choose in the Send Port must use a Xml Disassembler component **(XmlReceive pipeline will do fine unless you have your custom pipeline for some reason).

Error handling

Some people might find it strange that I don’t have any error handling in my Stored Procedures but in most cases an error in the procedure will cause an exception in the port communication with the database and that’s fine. If I’d like to handle that error I’ll handle that either in the orchestration or using other mechanisms in BizTalk like Failed Message Routing etc. If I can’t handle the error in the procedure I don’t see a reason to catch it.

Security

I’ve seen to many cases where developers actually used the user and password login option available on the database adapter and port. Even if the credentials are safe in the SSO database there is a risk you’ll end up in a mess with login data spread all over the BizTalk administration. One should always use try to use Windows NT Integrated Security!

Even more common than the above is to have a user that part of the administrators group or similar to hit the database - don’t! I usually try an set up one Host to run as in all the communication with a specific database (Could also be a Host for that database and that BizTalk Application so that different applications have their own Host). I’ll then give that specific user the **least privileges **needed in the specific database.

In the AddEmployee and TestDB example I’d create a new Group in the AD called for example “BizTalk TestDB Users”. I’d then create a User in the AD called “BizTalk TestDB User”. It’s based on this I’d then create a Host in BizTalk referencing the above created Group.

Finally we’ll create a Host Instance and make a Send Handler of it on the SQL Server Adapter.

Now we can set the least required privileges on the “BizTalk TestDB Users” in Sql Server as that the Group the User belongs that’ll hit the database. In this case this means granting the Group to the standard BizTalk required privileges for a Host besides granting it Execute right on the AddEmployee Stored Procedure.

Something I missed? How do you handle your database communication and security in your BizTalk implementations?

A loosely coupled Scatter-and-Gather implementation In BizTalk 2006

2008-01-15T17:40:05+00:00

Why loose coupling?

There was a question in Microsoft BizTalk forums the other day about how one could implement a Scatter and Gather pattern in a more loosely coupled fashion.

Most examples on the implementation of this pattern in BizTalk use the Call Shape functionality available in BizTalk orchestrations. This however creates a hard coupling between the “Scatter orchestration” and it’s “partner orchestrations”. The downside of that is that when one adds or removes a partner the whole solution has to be recompiled and redeployed.

If one however could use the publish-subscribe architecture in the MessageBox to route messages between the “Gather orchestration” and it’s partners, it’d be possible to add partners without having to worry about the rest of the solution. This post shows and example on how to implement a solution like that.

The BizTalk process in steps

> > [![Aa559774_note(ja-jp,MSDN_10)](../assets/2008/01/aa559774-noteja-jpmsdn-10-thumb.gif)](../assets/2008/01/aa559774-noteja-jpmsdn-10.gif) NOTE: Notice the difference between the **Partners**Request, the **Partners**Response, the **Partner**Request and the **Partner**Response messages. The names are unfortunately very similar. > > > > The **Partners**Request and **Partners**Response messages are used for communication between the Scatter and the Gather orchestrations. It's also a **Partners**Request message that activates the process. > > > > **Partner**Response and **Partner**Request are used for communicating between the Scatter and the Gather orchestration and all the partner orchestrations. > >

Request and scatter
A PartnersRequest message is received. This message is an empty message and is only used for activating the process in this example scenario. The PartnersRequest message is consumed by the Scatter orchestration. The Scatter orchestration creates one PartnerRequest message. The orchestration also generates a unique key called a RequestID and start a correlation combining that that id and the PartnersRequest MessageType. Finally it post the PartnerRequest message to message box, writes the generated RequestID to the request messages context (RequestID is a MessageContextPropertyBase based context property) and dehydrates itself.
Partners
All the enlisted Partner orchestrations pick up the PartnerRequest message from the message box. These orchestration then communicates with their specific data source (could be a service, database, file or whatever), receives a response. Finally these orchestrations transform the response they received and creates a PartnerResponse message that’s posted to the message box. Notice that the RequestID that was generated by the Scatter orchestration is also part of the context of the newly created PartnerResponse message.
Gather
The PartnerResponse messages are routed to the Gather orchestration. This orchestration uses a Singleton pattern based on the RequestID which all PartnerResponse messages carried with them in their context. This means that it’ll receive all the PartnerResponse messages containing that same RequestID into the same orchestration instance (ergo all the Partners that were activated by the request message being sent from one Scatterer). For each message instance it receives it add it’s price to a total price variable. When the Gather orchestration has received all the PartnerResponse messages (the orchestration knows how many Partners responses it should expect from one Scatterer orchestration and we can timeout if we don’t get all expected with a timeframe) the total price we calculated is written to a PartnersResponse message.
Response
This message is routed back the Scatter orchestration by using the correlation it initialized in the start. It’s finally this orchestration that send the final outgoing message (a PartnersResponse message).

Example solution

An example of the implementation can be downloaded from here.

The solution contains five different schemas.

PartnersResponse
Used for initializing the process.
PartnersRequest
Send from the Gather orchestration to the Scatter orchestration. It’s also the final result and outgoing message from the process.
PartnerRequest
Picked up and activates all enlisted Partner orchestrations.
PartnerRequest
Send from the Partner orchestration containing the result from the Partner Service and send to the Gather orchestration.
**LooselyCoupledScatterGatherExampleProperties
**Property schema for storing the RequestID and to correlate all the PartnerResponses as well as the final PartnersResponse back to the Scatter orchestration.

Five orchestrations

**Scatterer orchestration
**The “main orchestration” that receives a request message from outside and “scatters” party requests to all the party orchestrations.
**Gatherer orchestration
**Gatherer orchestration that gathers all the responses from the partners and transforms these to a reply that is being routed back to the Scatterer and back out.
Partner1, Partner2 and **Partner3 orchestration
**Partner orchestrations that communicates to different services and receives price information.

Setting up and testing the example solution - it’s easy!

When the solution is built and deployed one needs to setup and bind two ports; one outgoing port and one incoming port (this could also be a Request-Response port by changing the port type in the Scatterer orchestration). That’s it!

Enlist and start everything by dropping a PartnersRequest test message (you’ll find one among the zipped files) in the incoming folder. A PartnersResposne message should then be published in the outgoing folder containing a calculated price from all the Partner orchestrations.

Test message and a binding file are part of the zipped solution.

What would be different in “real life” solution?

I’ve made some major simplifications in this example to make it easy for setting it up and test the concept. These would very different in a “real” solution.

Partner Services

The Partner orchestrations are very simple. They actually don’t communicate with outside world at all. All the do is setting a hard coded price and post a response. In a real solution these would not be part of the same solution as the Scatter and Gather orchestration (otherwise we would be force to redeploy when adding a Partner orchestration to the dll).

The Partner orchestrations would also communicate with some sort of outbound source like a web service or database for example. This would however complicate the setup therefore I’ve skipped that part in the example.

Managing partners

One of the benefits with a loosely coupled implementation is the possibility to add and remove Partner orchestrations without having redeploy the rest of the solution. Using this implementation the Gatherer orchestration needs to know how many Partner responses it should wait for before timing out. This requires that value being set in a config file or something similar. In this example the number of Partners are hard coded into the Gather orchestration (it’s set to 3 Partners) to simplify the setup.

Final thoughts

Knowing how to create loosely coupled solutions like this is good knowledge to have. It’s my own and others belief that this architecture makes it possible to create more robust and separated solutions that one can update without having to do a lot a work and disturb the current processes. It’s however not the best solution performancewise as it adds a lot of extra hits on the MessageBox database and generates more work for the MessageAgent.

There are also a few things to watch out for:

Eternal loops

It easy to end up in a situation where you’re subscribing to the same message as you posting to the MessageBox. That’ll create and endless loop and cause a lot disturbance before you’ll find it. Think through and document you subscriptions!

Correlations for promoting values

When doing a direct post in BizTalk most properties are not promoted. To force you properties to be promoted you’ll have to initialize a correlation on the property as you send it. I can’t say I like this. There should be a other way of saying that one wants it promoted.

A couple of other useful articles as we on the subject:

Download the example orchestration and let me know how you used it and what your solution looks like!

Speaking at Developer Summit 2008!

2007-12-20T13:26:59+00:00

Whoho! I’ll be speaking at Developer Summit 2008. Developer Summit is a conference hosted by Cornerstone, it’s being held in Stockholm between the 9th and 11th of April 2008.

I’ll be doing a talk on Master Data Management (MDM) for the Enterprise using BizTalk 2006. Basically I’ll present some basic theory behind the MDM concept and how it relates to SOA. Then I’ll relate all that to a customer case I recently worked on solving a MDM requirement. Finally I’ll be showing a short demo where I publish some Master Data and update all subscribers to it. In the same demo I’ll also demonstrate on how to monitor the process using Business Activity Monitoring.

Sound interesting? Something specific you think should be in the presentation? Let me know!

Hope to see you there!

Removing namespace from outgoing messages

2007-12-20T12:38:24+00:00

Way to often I get a request to deliver a message from BizTalk without any XML namespace. Comments like the one below aren’t that rare.

> > Why do you put those "ns0" I front of every tag!? We can't read the XML when it's written like that! All our XPath and XSLT seems to fail. > > > > We don't want _****_ ... We expect the messages from you to be like _****_ ... > > > > Is that some BizTalk specific? > > </blockquote> I always try to explain what XML namespaces are, and why it's a good idea to use them (especially when it comes to versioning of messages). Sometimes it's just impossible to get people to understand the advantages of using it and to persuade them to change their solutions to handle XML namespaces. It's in these cases it'll be up to the implementation in BizTalk to remove the namespace. ### How to remove XML namespace There are a couple of ways of achieving this, we can use [.NET code](http://www.pluralsight.com/blogs/keith/archive/2005/10/19/15714.aspx) that we call in a pipeline or in an orchestration. But we can also handle this using XSL - and that's what I'll show in this post. The XSL stylesheet below will remove all XML namespaces while transforming the message. Basically it just copies the nodes, attributes and it's values using the [local-name()](http://www.webhostingsearch.com/blogs/richard/nevermind-the-xml-namespaces-in-xpath-expressions/) function to ignore the XML namespaces.
xml version="1.0" encoding="utf-8"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:template match="/"> <xsl:copy> <xsl:apply-templates /> </xsl:copy> </xsl:template> <xsl:template match="*"> <xsl:element name="{local-name()}"> <xsl:apply-templates select="@* | node()" /> </xsl:element> </xsl:template> <xsl:template match="@*"> <xsl:attribute name="{local-name()}"> <xsl:value-of select="."/> </xsl:attribute> </xsl:template> <xsl:template match="text() | processing-instruction() | comment()"> <xsl:copy /> </xsl:template> </xsl:stylesheet>
### Apply the XSL transformation We can use this XSL stylesheet in a ordinary BizTalk map on the send port using the _Custom XSL Path_ property in the BizTalk Mapper. The result is that the XSL we usually generate in the mapping tool will be overridden by our own XSL stylesheet. The figure below shows how we use the property windows of the grid in the BizTalk Mapper to set the property and point the Mapper to our XSLT document. [![](../assets/2007/12/windowslivewriterremovingnamespacefromoutgoingmessage-a4edxsltpath-thumb11.jpg)](../assets/2007/12/windowslivewriterremovingnamespacefromoutgoingmessage-a4edxsltpath51.jpg) **But what if we already have a map on the send port and it's that already transformed message we like to remove the namespace from?** One possibility is to use the [_XSLT Transform_ pipeline component](http://msdn2.microsoft.com/en-us/library/aa561389.aspx) that comes with the BizTalk 2006 SDK. It's usually located at _C:\Program Files\Microsoft BizTalk Server 2006\SDK\Samples\Pipelines\XslTransformComponent\XslTransform_ on your development machine. I've written about this sample component before [here](http://www.webhostingsearch.com/blogs/richard/transform-and-split-messages-using-an-xslt-transformation-pipeline-component-and-the-xmldissasembler/) were I used it another scenario. The figure below shows how we use the property windows of the _XSLT Transform Component_ in a pipeline in the Pipeline designer tool to set the path to our XSLT stylesheet. [![](../assets/2007/12/windowslivewriterremovingnamespacefromoutgoingmessage-a4edxsltpath2-thumb1.jpg)](../assets/2007/12/windowslivewriterremovingnamespacefromoutgoingmessage-a4edxsltpath221.jpg) ### Final thoughts The XSLT Transform component is far from perfect and the obvious problem is of course that the component loads the whole message into memory using the [XmlDocument class](http://msdn2.microsoft.com/en-us/library/system.xml.xmldocument.aspx) to read the message. That means that this solution isn't for those scenarios where you'll have huge messages coming in by the thousands. But for those cases where you have normal sized messages and you have a idea of the traffic you receive, it's a quick and easy solution. Any comments on the pro and cons on this solution and how you usually solve this scenario will be appreciated!
> > **UPDATE #1: **Make sure you don't miss [Johan Hedbergs solution](http://blogical.se/blogs/johan/archive/2008/01/07/removing-xml-namespace-in-a-pipeline-component.aspx) to this problem. Basically he solved it using the [Microsoft.BizTalk.Streaming.dll](http://technet.microsoft.com/en-us/library/microsoft.biztalk.streaming.aspx) which will give you better memory management. > >

> > **UPDATE #2: **Based on the comments to this post I've posted an "updated" post [here](http://www.richardhallgren.com/removing-xml-namespaces-revisit/) solving this using ordinary mapping and defining a schema without a XML namespace - easier and more correct. > >

Dealing with cXML based messages in BizTalk

2007-11-29T16:33:30+00:00

cXML (commerce eXtensible Markup Language) is a XML based standard for communication of data related to electronic commerce. The problem from a BizTalk perspective is that they don’t publish any XML schemas (XSD), only Document Type Definition (DTD).

When trying to generate a schema based on a DTD using the functionality in BizTalk (via Add Generated Items) one ends up with a schema split of three files that really doesn’t make any sense (XmlSpy doesn’t do a very good job either …). So after a while I found Nick Heppleston schema repository! After some tweaking I actually had a cXML Order schema in the version I was looking for! Thanks Nick!

The next set of problems was to handle the lack of XML namespace and the DOCTYPE declaration that messages validating against DTD carries on top.

<div><span style="color: #0000FF; "><?</span><span style="color: #FF00FF; ">xml version="1.0" standalone="no"</span><span style="color: #0000FF; ">?></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "><!</span><span style="color: #FF00FF; ">DOCTYPE cXML SYSTEM "http://xml.cxml.org/schemas/cXML/1.2.014/cXML.dtd"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "><</span><span style="color: #800000; ">cXML </span><span style="color: #FF0000; ">xml:lang</span><span style="color: #0000FF; ">="en-US"</span><span style="color: #FF0000; "> payloadID</span><span style="color: #0000FF; ">="2007117.25919@Contempus"</span><span style="color: #FF0000; "> timestamp</span><span style="color: #0000FF; ">="2007-11-07T11:06:16+01:00"</span><span style="color: #0000FF; ">></span></div>

To handel these two issues I set up a receive pipeline that looked like the one below.

Remove the DOCTYPE declaration

First I created a pipeline component to remove the DOCTYPE node. It’s simple code using regular expression to find the DOCTYPE node, replace it with an empty string and return the message.

<div><span style="color: #0000FF; ">public</span><span style="color: #000000; "> IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
{
   </span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> messageString </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> StreamReader(inmsg.BodyPart.Data).ReadToEnd();
   Regex doctypePattern </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> Regex(</span><span style="color: #000000; ">"</span><span style="color: #000000; "><!DOCTYPE.+?></span><span style="color: #000000; ">"</span><span style="color: #000000; ">);
   messageString </span><span style="color: #000000; ">=</span><span style="color: #000000; "> doctypePattern.Replace(messageString, </span><span style="color: #0000FF; ">string</span><span style="color: #000000; ">.Empty);
   MemoryStream memStream </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> MemoryStream();
   </span><span style="color: #0000FF; ">byte</span><span style="color: #000000; ">[] data </span><span style="color: #000000; ">=</span><span style="color: #000000; "> Encoding.UTF8.GetBytes(messageString);
   memStream.Write(data, </span><span style="color: #000000; ">0</span><span style="color: #000000; ">, data.Length);
   memStream.Seek(</span><span style="color: #000000; ">0</span><span style="color: #000000; ">, SeekOrigin.Begin);
   inmsg.BodyPart.Data </span><span style="color: #000000; ">=</span><span style="color: #000000; "> memStream;
   </span><span style="color: #0000FF; ">return</span><span style="color: #000000; "> inmsg;
}</span></div>

Set an XML namespace

Secondly I used Richard Seroter’s post on how to change the SetNSForMsg component to add a XML namespace. That’s the second component showing in the decode stage of the pipeline.

Arrow number 3 shows how the SetMsgNS exposes a property that allows us to set the namespace that we can configure per pipeline. In this case I’ve set it to _http://schemas.modhul.com/cXML/1.2.014/OrderRequest__ _which is the namespace of the cXML schema I’m currently working agains.

In the end we’ll have a message with the following declaration and root node.

<div><span style="color: #0000FF; "><?</span><span style="color: #FF00FF; ">xml version="1.0" encoding="utf-16" standalone="no"</span><span style="color: #0000FF; ">?></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "><</span><span style="color: #800000; ">cXML </span><span style="color: #FF0000; ">xml:lang</span><span style="color: #0000FF; ">="en-US"</span><span style="color: #FF0000; "> payloadID</span><span style="color: #0000FF; ">="2007117.25919@Contempus"</span><span style="color: #FF0000; "> xmlns</span><span style="color: #0000FF; ">="http://schemas.modhul.com/cXML/1.2.014/OrderRequest"</span><span style="color: #FF0000; "> timestamp</span><span style="color: #0000FF; ">="2007-11-07T11:06:16+01:00"</span><span style="color: #0000FF; ">></span></div>

Now we’re ready to start mapping!

A pattern for Ordered Delivery using orchestrations

2007-11-01T13:37:46+00:00

Ordered Delivery is a great feature of BizTalk 2006. The problem is however that as soon as one introduce a orchestration in the process it doesn’t work. The Ordered Delivery option on ports in BizTalk 2006 ensures two things today:

Messages are published to the MessageBox in the same order as they were consumed by the Receive Port.
Messages are consumed from the MessageB0ox and published to the end destination by the send port in the same order as they were persisted in the MessageBox.

What’s the problem?

Fine. But if we introduce an orchestration each message being persisted in the the MessageBox will start it’s own orchestration instance. These instances could (and in many cases will) finish in different order than they started. This means that we loose the correct order of the messages one they are persisted back to the MessageBox from the orchestration instance.

The above “limitation” is well known and the solution to this has always been a Sequential Orchestration pattern (also called Singleton Orchestration). Basically this correlates messages from for example a specific Receive Port of a particular Message Type. This ensures that only one instance of the orchestration will be started and we can keep the order of the messages. We’ve experienced a lot of problems using this this kind of solution for Ordered Delivery - all from Zombies to them using huge amounts of memory and so on.

Microsoft’s new solution

Now it finally looks like this problem is about to be solved! Microsoft released a white paper recently (download it here - found it via Richard Seroter’s blog) to solve these problems in a other way. It basically means that a pipeline will stamp the messages with a sequence number as they enter BizTalk (the messages are then in order as we’re then using Ordered Delivery on the port). Then one can have as many orchestration as one like processing these messages and publish them back to the MessageBox (now possibly out of order but with a sequence number stamped on them). The last business orchestration will set a “destination” property that will route the message to a resequence orchestration. This orchestration resequences the messages and decides by checking the last sequence number it sent out if the current message should be sent out or put back on queue.

Conclusion

My biggest concern in this solution is that is still based on a singleton. We’ve had cases where where the send procedure been extremely slow (for example when we used SOAP and had a slow Web Service in the other end), then the orchestration has built up in memory as it queued messages internally. However it looks like this solution is well thought through with a “flush queue” functionality, stop message, some ideas on how to handle errors (remember that if the singleton fails and you don’t handle this your processes stop) and so on.

Read it and please let me know what you think!

Why BizTalk isn’t an ESB today and is it heading there?

2007-09-25T14:01:57+00:00

I’ve written about the ESB concept before and what I think an ESB architecture is. In the posts comments there were some discussion about if BizTalk is an ESB or not. And if not - why not?

I think this article does a great job in explaining and discussing this subject. Basically it says that the main reason for BizTalk not qualifying as an ESB today (I know about the ESB Guidance - we get there ;)) is because of it’s “all-or-nothing” packaging. What that means is that it’s different functionality can not be separately deployed a cross a bus structure. For example; the scenario of having the transformation functionality in one place and the routing functionality in another just isn’t possible in today’s architecture. Today you install the full product in one place.

I think I’ve personally have learnt to live with this limitation. On the other hand I can see that the possibility of splitting parts up definitely changes things as the possibility of reuse and single point of failure problems etc.

Is BizTalk going the ESB way?

I still haven’t had as much time as I’d like to examine the ESB Guidance but I look forward to see how they worked around the problem described above.

Just looking at this architecture image shows that they’ve split things up in a new way and that each part is accessible trough services - nice! Could this be the future architecture of BizTalk server? What do you think?

I’ll try and install the ESB project as soon as I get some more time on my hands. In the mean time I’d love some tips and comments on articles and other reading on the experiences of the ESB Guidance project.

SOA isn’t response request

2007-08-10T17:48:17+00:00

I’ve often been told that one of the biggest mistakes people make when implementing a service oriented architecture is that they don’t re-architect their current architecture to become service oriented. I’ve never really understood what they meant by that until I read this article. SOA is not about adding a service based call to expose your current procedure calls as a service - it’s so much more. It’s about enabling ease of change and to create a more agile architecture.

Each of these assumptions exist in a Remote Procedure Call. They are forms of coupling, pure and simple. They fly in the face of SOA. > >

What do you think? Does it make sense in your world?

XPathmania and XPath on TV!

2007-08-03T08:08:46+00:00

I’ve blogged about the Visual Studio extension XPathmania before. It’s a very simple little tool that lets you write and test XPath inside of Visual Studio 2005. No big deal if you already have tools like XMLSpy or XML Notepad but still. I like not having to start another application, opening the XML document I’m working with and so on. Doing stuff inside of Visual Studio just feels right and saves some time anyway.

One of the last episodes of dnrTV hosted Dom Demsak (Don XML), the creator of XPathmania. The show is 30% about XPathmania and 70% about XPath and XPath syntax in general. It’s kind of basic XPath but I think it can be useful for someone who feels they haven’t got full control of the language.

During the show they touch on XML namespaces and XML default namespaces. However they don’t really explain the difference between them and how it effects the document. Something that’s sad as I feel that XML namespaces (and especially default namespaces) is something that most people haven’t fully understood.

Anyway, if you got some spare time watch it or forward it someone you think should watch it. ;)

Transform and split messages using an Xslt transformation pipeline component and the XmlDissasembler

2007-08-02T08:44:09+00:00

There was a question on the Microsoft newsgroup the other day where someone had to split a message into parts. But some of the information that was supposed to go in to the different parts were part of the envelope. I thought I’d give my solution to the problem a try - here it is.

Say for example that we receive the following message.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">ns0:EmployeeSalesReport </span><span style="color: #FF0000; ">ReportID</span><span style="color: #0000FF; ">="R100"</span><span style="color: #FF0000; "> EmployeeID</span><span style="color: #0000FF; ">="0012345"</span><span style="color: #FF0000; "> xmlns:ns0</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.EmployeeSalesReport"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Sales</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ns1:Sale </span><span style="color: #FF0000; ">xmlns:ns1</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.Sale"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">100</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">10</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ns1:Sale</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ns1:Sale </span><span style="color: #FF0000; ">xmlns:ns1</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.Sale"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">200</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">20</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ns1:Sale</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Sales</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ns0:EmployeeSalesReport</span><span style="color: #0000FF; ">></span></div>

In the above message we have different sales information from one employee but all the global employee and report information (the ReportID and EmployeeID attributes) exists in the root node (what we’ll call the envelope of the message).

What we like to achieve is to split this envelope message into it’s different sale item so we get separate message looking something like the below where each item contains the global information from the envelope.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">Sale </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.Sale"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ItemID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">200</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Amount </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">20</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">EmployeeID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">0012345</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">EmployeeID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ReportID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">R100</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ReportID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Sale</span><span style="color: #0000FF; ">></span></div>

There might be different needs for doing this in a pipeline but much of it comes down to choice of architecture style in your BizTalk solution. Personally I’d refuse to introduce another orchestration for splitting a message like this. I don’t believe that’s what orchestrations are for (they should deal with possible logic and workflow in my world). I’m sure other people feel different - feel free to use the comments.

Step 1 - Transforming the envelope message using XSLT

The first thing we’ll need to do is to transform the envelope message so that the information from the root node (the ReportID and the EmployeeID) get into every single _Sale-_node. There are a couple of ways of achieving this but I’ll use the XSLT-transformation pipeline component that ships with the BizTalk 2006 SDK.

All this component does is to let you point out a XSLT stylesheet you like to use to transform your message.

The stylesheet I’ve used is part of the solution that you can download here. I will not show the stylesheet full script here but rather the result after the message travel through the component.

<div><span style="color: #0000FF; "><?</span><span style="color: #FF00FF; ">xml version="1.0" encoding="utf-8"</span><span style="color: #0000FF; ">?></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "><</span><span style="color: #800000; ">EmployeeSalesReport </span><span style="color: #FF0000; ">EmployeeID</span><span style="color: #0000FF; ">="0012345"</span><span style="color: #FF0000; "> ReportID</span><span style="color: #0000FF; ">="R100"</span><span style="color: #FF0000; "> xmlns</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.EmployeeSalesReport"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Sales </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Sale </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.Sale"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ItemID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">100</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Amount </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">10</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">EmployeeID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">0012345</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">EmployeeID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ReportID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">R100</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ReportID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Sale</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Sale </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.Sale"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ItemID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">200</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Amount </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">20</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">EmployeeID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">0012345</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">EmployeeID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ReportID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">R100</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ReportID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Sale</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Sales</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">EmployeeSalesReport</span><span style="color: #0000FF; ">></span></div>

Ok, nice. Now we have all the information we like in each of the Sale-nodes! All we have to do now is to split the message!

Step 2 - Splitting the message

We’ll use the standard XmlDisassembler component for splitting the message. All we have to do is to make sure we done the following.

Set the schema envelope property to “true” in the schema editor.
Set the Body Xpath property on the _EmployeeSalesReport-_node to point to the _Sale-_node.
Configure the Document Schemas _and Envelope Schemas (same properties are called _DocumentSpecNames and EnvelopeSpecNames if your doing the configuration after deployment in the administration console_)_ properties of the XmlDisassember component to match the names of your schemas.

This is what the envelope schema looks like in this example.

This is what the XmlDisassembler configuration looks like.

There, we’re done! Now we can drop a test file containing two Sale-nodes and all the report/employee information on the top and have the two following separate files as a result when using a file send port.

Setting up the test solution

Download a zipped version of the solution.
You’ll have to unzip the project to C:\Example.TransformAndSplit. If that isn’t possible you’ll have to change the path to the XSLT stylesheet in the XsltComponent in the pipeline as this is a fixed path.
When you build the XsltComponent the Output path is set to the Pipeline Components folder. This assumes that BizTalk is installed at _C:\Program Files\Microsoft BizTalk Server 2006_. If that isn’t the case make sure to change the output path of the component.
If you’d like to run the pipeline component in debug I’ve set this up using the pipeline.exe tool. There are however some paths in the debug setting of the XsltComponent project also that assumes that your BizTalk solutions is installed at C:\Program Files\Microsoft BizTalk Server 2006. _If this isn’t the case you’ll have to change some values in the _Command lines arguments property, but that only if you like to run debug.
Build and deploy.
Set up a receive port and location using the TransformAndSplitSales pipeline.
Set up a send port with for example a filter on the name of the Receiveport you’ve just set up.
Drop the example message that’s part of the solution!

Final thoughts

This example didn’t really take much more than an hour to put together. I do however realize that this is a simplified incoming message and that in the case of a more complex message we’d get a messy XSLT stylesheet to maintain. It’s also important to remember that using this method with XSLT transformation means that we will load the entire incoming Xml document into memory, so when we’re dealing with bigger Xml documents we’ll have manipulate the incoming message using other techniques.

Hope this is useful for someone!

Developing and debugging orchestrations using DebugView and SOAPTrace tools

2007-07-31T07:38:33+00:00

The main problem I have with developing BizTalk orchestrations is the fact that I’m so blind when it comes to follow the runtime processing. Using the debugger that is part of the HAT tool is slow and clumsy which IMHO makes the tool almost useless in everyday development. But there is hope!

DebugView

Sysinternals (Windows Sysinternal now - Microsoft bought them last year) DebugView is a wonderful little tool and is especially useful when it comes to figure out what’s actually going on inside an orchestration. Basically the tool listens to system wide debug output. From an orchestration it’s possible to write debug information using the .NET System.Diagnostics namespace and the Debug or Trace class.

Decide on how to filter

There are a couple of handy little tricks that makes DebugView a even better in BizTalk development. First one should try and have something in the debug messages that makes it possible to filter and distinct one’s own (as DebugView listens system wide debug output all running applications debug info will show up). Our team decided on “Sogeti” (our company name) for all our development and to have a method in our BaseLibrary component that outputs something like the below (the BaseLibrary is a small little .NET component with a couple of very useful classes we use company wide in our BizTalk related development).

<div><span style="color: #000000; ">System.Diagnostics.Debug.WrtieLine(</span><span style="color: #000000; ">"</span><span style="color: #000000; ">Sogeti, your debug/trace message here</span><span style="color: #000000; ">"</span><span style="color: #000000; ">)</span></div>

This make is possible to have a filter in DebugView and to for example have it look something like this.

Trace full context of messages

Another little useful trick is to trace the full context of messages. This is done be storing the message in a XmlDocument typed variable and get the OuterXml property of that variable. The below code is and example of this.

<div><span style="color: #000000; ">tempXml </span><span style="color: #000000; ">=</span><span style="color: #000000; "> msgFindPartyRequest.parameters;
System.Diagnostics.Trace.Write(System.String.Concat(</span><span style="color: #000000; ">"</span><span style="color: #000000; ">Sogeti, msgFindPartyRequest: </span><span style="color: #000000; ">"</span><span style="color: #000000; ">, tempXml.OuterXml));</span></div>

Example of a full message trace.

To Trace or Debug - that’s the question

As stated earlier both System.Diagnostics.Debug and System.Diagnostics.Trace has methods (Write, WriteLine and so on) for outputting debug information. However there is only one that stays in your compiled code when switching from Development to Deployment compilation mode (guess which one ;)). So make sure you choose the right class for the right information. I like to have some critical messages left using Trace and be able to trace these even on the test and production server.

DebugView on a remote desktop

When running DebugView on an other server (say a test or a production server) using Remote Desktop I’ve found that ones has to use the console user on the server. This kind of makes sense as if we’re connection “normally” we’re creating a virtual session and that’s not were the debug information is written to.

Microsoft SOAP Toolkit 3.0

This is a totally other tool than DebugView but I thought it fit here any way. It’s a handy tool when working with SOAP based messages. Without it’s very hard to actually figure out how the raw request and response message look and why your orchestration web service is acting the way it does.

The trace tool is placed as a reverse proxy between BizTalk and the Internet. It’s setup by telling the tracing tool which localhost port to listen at (for example 9091 as in the example below) then we’ll redirect to that port by changing the setting in the BizTalk send port.

Finally we’ll set up the trace tool to listen to port 9091 and redirect all traffic to our web service URL at port 80 in this case. So basically the trace tool will catch all the traffic hitting the 9091 port and forward it.

That’s it! This is probably basic stuff for most of you but hopefully it’s useful for someone!

I’ve also noticed that the SOAP Toolkit is deprecated by Microsoft and I’d like to hear if anyone used something else (like Fiddler example) for tracing SOAP messages. I’d also love some other tips, tools and methods you use for debugging BizTalk orchestrations.

Reading text value from node using XPath function directly in BizTalk orchestrations

2007-07-24T14:10:02+00:00

The XPath function that’s available directly inside BizTalk orchestration is a powerful little tool. However I’ve seen a couple of project where developers just grown tired of it and started creating their own little libraries instead. I’ll be the first to admit that the XPath function isn’t perfect, and it sure doesn’t work like most of the other XPath engines (which is the biggest problem) but it’s still inside the orchestration and you can use it to both read and assign values to a message which is super useful! Basically I don’t see a valid reason for bringing more complexity into your solution by adding another library - as long as you’re just going to read or set value using XPath.

However there is one trick that you should know of when it comes to reading a text value from a node. Basically you have to use both the string() and
text() XPath functions. Both Charles Young and Yossi Dahan has good post on this subject. Also if your new to writing XPath expressions for complex schemas with loads of namespaces and stuff (like schemas in BizTalk) this post could be useful for you.

Finally a nice tool for writing and testing small XPath expression inside Visual Studio (if you don’t want to spend x minutes waiting for XmlSpy to start up …) is XPathmania. Read about it here - I use it all the time!

Lists, Arrays and collections within BizTalk orchestrations

2007-07-05T12:40:30+00:00

> > **Update 2012-03-28** Unfortunately the images for this post was lost in upgrade of the blog. > >

This post shows how it’s possible to create your own .NET class and then use this within your orchestration (as a variable) to work with typed lists - something that unfortunately isn’t supported out of the box in BizTalk orchestrations. Below is the class used to create a typed list for System.String objects.

<span class="kwrd">using</span> System;
<span class="kwrd">using</span> System.Collections.Generic;
<span class="kwrd">using</span> System.Text;

<span class="kwrd">namespace</span> Sample.BizTalkCollections
{
    [Serializable]
    <span class="kwrd">public</span> <span class="kwrd">class</span> StringList
    {
        <span class="kwrd">private</span> List<<span class="kwrd">string</span>> _list = <span class="kwrd">new</span> List<<span class="kwrd">string</span>>();

        <span class="kwrd">public</span> <span class="kwrd">void</span> Add(<span class="kwrd">string</span> item)
        {
            _list.Add(item);
        }

        <span class="kwrd">public</span> <span class="kwrd">int</span> Count()
        {
            <span class="kwrd">return</span> _list.Count;
        }

        <span class="kwrd">public</span> <span class="kwrd">bool</span> Remove(<span class="kwrd">string</span> item)
        {
            <span class="kwrd">return</span> _list.Remove(item);
        }

        <span class="kwrd">public</span> <span class="kwrd">void</span> RemoveAt(<span class="kwrd">int</span> index)
        {
            _list.RemoveAt(index);
        }

        <span class="kwrd">public</span> <span class="kwrd">override</span> <span class="kwrd">string</span> ToString()
        {
            StringBuilder builder = <span class="kwrd">new</span> StringBuilder();
            <span class="kwrd">foreach</span> (<span class="kwrd">string</span> item <span class="kwrd">in</span> _list)
            {
                builder.AppendLine(item);
            }

            <span class="kwrd">return</span> builder.ToString();
        }
    }
}

Notice that the class is serializable so BizTalk can serialize when dehydrating the orchestration to the database. We’ve also chosen not to extend the IList interface as we don’t want to expose all methods of that interface, but only the once we’ll really going use within our orchestrations.

This StringList class exists within the BizTalkCollections namespace where we have other classes for lists types with different datatypes (for example ObjectList, Int32List and so on). Below is a picture showing the structure of the Visual Studio project containing all the collections we current have implemented. This project and the BizTalk testproject is available for download here.

In the orchestration we then have set up a variable and type this to our StringList class within the Sample.BizTalkCollections namespace.

Also make sure the assembly with your collection class is installed in the GAC, otherwise you’ll end up with a _FileNotFound exception _saying

> > Could not load file or assembly 'Sample.BizTalkCollections, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7109528d5dbf986b' or one of its dependencies. The system cannot find the file specified. > >

Then you can use the following code within your orchestration! Download the sample project and test it.

A really generic class

I’d really like for it be possible to create my own generic classes and then initialize these using something like the below.

genericList = Sample.BizTalkCollections.GenericList<System.String>();

However this isn’t possible as BizTalk orchestrations don’t seem to understand generic classes … If someone found a solution to this please let me/us know via the comments to this post.

TechEd Barcelona here I come!

2007-07-02T11:41:34+00:00

I’ve just found out that I get to go to TechEd in November! I haven’t been there before. I’ve been to a lot of conferences but not **the **Microsoft conference. Do I have to say that I’m looking forward to it?

If you’re going and feel like meeting up for a BizTalk lunch, dinner or whatever don’t hesitate to drop me an e-mail.

Reading the message body and context from the BizTalkDTADb using operations library in BizTalk 2006

2007-06-04T14:18:20+00:00

Update 2007-07-05: The example project used in the post can be downloaded from here.

The operations dll (Micrsoft.BizTalk.Operations) is one of the new bits in BizTalk 2006 that at least I haven’t heard much about (most of you probably find in the C:\Program Files\Microsoft BizTalk Server 2006 folder). However it’s a library with a lot of useful functionality.

In this post I’ll focus on how it’s possible to use the library to get hold of message parts and message context from the BizTalk tracking database (usually called the BizTalkDTADb). If you’re unfamiliar with the architecture of the tracking in BizTalk this article is a good place to start.

What used to be the problem?

In BizTalk 2004 one soon got into problems when trying to read the message parts and the message context from the tracking database. The problem is that the the Xml is compressed (something that makes totally sence - Xml is a perfect candidate for compression). To my knowledge no one has found a good way to decompress the message context. There’s however a way to decompress the message parts (and only the parts) that Rob posted on in the BizTalk newsgroup. But this method doesn’t work on the compressed context of the same message that the parts belong to! One could assume that the same method could be used for both compression and decompression but I haven’t got it to work for the context of the message (read about my frustration here)

What’s wrong with WMI and MSBTS_TrackedMessageInstance?

So the option that was left to us before BizTalk 2006 for reading all the message parts as well as the message context was to use the MSBTS_TrackedMessageInstance WMI scrtipt and save everything down to file. The problem with this approach is that it’s slow and ugly! Say that we like to save a couple of thousand messages from the tracking database. When forced to save the messages to file we’ll end up with a slow and ugly solution where writing to file takes ages, then we have to read the content of the different files before cleaning up and deleting the files.

Micrsoft.BizTalk.Operations to the rescue

A quick view in Lutz Roeder’s Reflector shows us a couple of interesting methods in the operations dll. However I’ll only use the GetTrackedMessage method in this post.

I’ve written a tiny test application that uses the library to read the body part of the message (the meat of the message) and a couple of properties I’m interested in from the context of the message. It looks something like this. If your interested drop me an email and I’ll send it. The first screenshot below shows an example of reading the InterchangeID from the context of a tracked message.

This screenshot shows an example of reading the full body of a tracked message from the tracking database. Try doing this with less then 10 lines of code using the MSBTS_TrackedMessageInstance script!

There is really nothing to the application, reading a tracked message using the MessageID will return an object implementing IBaseMessage and we basically read the BodyPart property of - it really couldn’t be any easier.

<div><span style="color: #0000FF; ">public</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">static</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> GetMessageBodyByMessageID(</span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> dbServer, </span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> dbName, Guid messageID)
{
    TrackingDatabase dta </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> TrackingDatabase(dbServer, dbName);
    BizTalkOperations operations </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> BizTalkOperations();
    IBaseMessage message </span><span style="color: #000000; ">=</span><span style="color: #000000; "> operations.GetTrackedMessage(messageID, dta);
    </span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> body </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">string</span><span style="color: #000000; ">.Empty;
    </span><span style="color: #0000FF; ">using</span><span style="color: #000000; ">(StreamReader streamReader </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> StreamReader(message.BodyPart.Data))
    {
        body </span><span style="color: #000000; ">=</span><span style="color: #000000; "> streamReader.ReadToEnd();
    }

    </span><span style="color: #0000FF; ">return</span><span style="color: #000000; "> body;
} </span></div>

I’d be very interested in how people use the operations library both when it comes to read and use tracked messages, but also in other ways! Use the comments to tell me and other readers how you use it your solutions!

Visual Studio 2005 tries to load the .pdb debug file from the GAC

2007-05-15T06:14:01+00:00

Ever seen this alert when trying to debug a project using Visual Studio 2005?

The following module was build with optimizations enabled or without information: > > C:\Windows\assembly\GAC_MSIL\ ... > > To debug this module, change its build configuration to debug mode. TO suppress this message, disable the 'Warn if no user code on launch' debugger option. > >

Basically this means that Visual Studio can’t find the debug file **(the .pdb file) at the same location as the dll is loaded from. Reading the error message tells us that the **dll been loaded from the GAC and there aren’t any pdb files there! Just make sure to uninstall the dll from the GAC and Visual Studio should be able to load it from your Debug folder and you’ll probably be good to go.

XPathUpdateException when assigning value to constructed messages

2007-05-14T14:11:53+00:00

Ever had a “use of unconstructed message” error message in your orchestrations? Then you know that messages has to be constructed before one can use them within orchestrations. Basically messages in orchestrations are created in ports or when executing a map in a construct shape. However there is also the possibility of creating messages in expression shapes using code. Matt Meleski does a great job explain the different approaches in his Constructing BizTalk 2004 XML Messages (In an Orchestration) - Choices blog post_._

However there are some pitfalls when constructing messages from scratch within orchestration (a part from the fact that there really isn’t a clean way of doing it … Scott Colestock has some ideas in this post).

A common problem is XPathUpdateException. Say one has a mapping that looks like the one below where one node isn’t mapped. This will create a message without the ElementB element. This doesn’t have to be in a mapping, it could also happend when constructing a message using code in a expression shape. A common scenario is that we add code for constructing the bone structure of the XML message, then change the schema without updating the construct code. Ergo a message will be constructed without that element (could be both valid or invalid XML then depending on the Min Occurs value)!

[](../assets/2007/05/windowslivewriterxpa.setdistinguishedfieldwhileassigningv-ceb2complexmapping23.jpg)

When we then try to add a value to the element that hasn’t been created the exception will be thrown!

One way around when using a map is to make sure to have a default value on every element to force them be constructed. Adding the following to map will make it work.

It’s a one time mistake but it’s good to know what to look for when the XPathUpdateException in Microsoft.XLANGs.Core.XSDPart.SetDistinguishedField (if it’s a distinguished field that is) shows up in the event log.

What’s an ESB architecture!?

2007-05-11T06:18:39+00:00

I’ve just finished reading a paper written by David Pallmann (PDF) from Neudesic. Belive it or not (or even better check it out), it’s a short and understandable description of what an ESB is!

Neudesic sells a product called Cyclone which is something they call a ESB architecture and framework software and even if describing this product is the main purpose of the paper the first part deals with describing the basics of an ESB and try’s to “assemble a synthesis of popular ESB definitions”.

I’ve chosen a couple of parts of the paper that really appealed to me but I really recommend the full length version (only 8 pages!).

Defining the ESB > > * An ESB is a backbone for connecting and integrating an enterprise’s applications and services. > * An ESB provides the necessary infrastructure to create a service oriented architecture. > * An ESB is a convergence of EAI, MOM, and SOA concepts. > * An ESB is based on open standards such as XML, SOAP, and WS-*. > * An ESB provides intelligent routing, such as publish-subscribe, message brokering, and failover routing. > * An ESB provides mediation, overcoming data, communication, and security differences between endpoints. > * An ESB integrates with legacy systems using standardsbased adapters. > * An ESB provides logical centralized management but is physically decentralized. > * An ESB is able to apply EAI concepts such as rules and orchestrations. > * An ESB is able to monitor and throttle activity as per a Service Level Agreement (SLA). >

I never thought of an ESB as an convergence of EAI (Enterprise Application Integration), MOM (Message Oriented Middleware) and SOA (Service Oriented Architecture) concepts. But, ok, so what does the writer really mean by these concepts?

SOA makes loosely-coupled, decentralized solutions possible that are enterprise-ready and based on interoperable standards. EAI allows integration of any combination of systems, with sophisticated message brokering, message translation, business process orchestration, and rules engine processing. MOM provides intelligent routing such as publish-subscribe topical messaging and strong managerial controls over routing, auditing, activity monitoring, and throttling. > >

And why do we need a bus architecture on top of SOA, MOM and EAI?

Point-to-point architecture works all right on a small scale, but its problems become apparent when used at the enterprise level. If each system has to know the connection details of every other system, then each new system added increases the problem of configuration and management exponentially. This was the impetus that led us to hub-and-spoke architectures, which most EAI products use. This architecture was a vast improvement over point-to-point architectures, and each system needed to communicate with only the hub. In addition, the hub could provide excellent management features since it was a party to all communication. It only took time to reveal some shortcomings with the hub-and-spoke approach, and today it is often associated with concerns about scalability, single point of failure, and vendor lock-in. > > ... > > Fortunately, there is a sound compromise to be found in the bus architecture, which provides the benefits of logical centralization but is physically decentralized. The bus architecture in earlier days was often used in message bus systems based on proprietary technologies, but an ESB implements this architecture using WS-* standards. > >

So using all three of them will make an ESB? Well, not really …

If combining disciplines was all there was to an ESB, we’d simply call it “consulting”. To properly leverage these disciplines they need to be combined in the right way through an architecture that lets their strengths shine and overcome their inherent weaknesses. Each discipline has some weak areas that the others help to resolve: SOA needs better enterprise manageability; EAI needs to become decentralized; MOM needs to get away from proprietary technologies. Combining these disciplines properly in an ESB overcomes these weaknesses. > >

I’d be very intreseted in comments of other artciles, papers and so on relating to the ESB concept!

Get row identity from SQL Adapter response

2007-04-18T14:11:52+00:00

Recently I had to insert a record using a stored procedure and the SQL Adapter in BizTalk 2006. There are lots of examples on both how to insert records and how to select a number of record using this adapter. However I had problems finding how to **insert a record and receiving the new id of the inserted row in return **(the SCOPE_IDENTITY()). In my scenario I needed the id to insert into another database further down in the orchestration.

I ended up with a stored procedure looking like the below.

<div><span style="color: #0000FF; ">ALTER</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">PROCEDURE</span><span style="color: #000000; "> </span><span style="color: #FF0000; ">[</span><span style="color: #FF0000; ">dbo</span><span style="color: #FF0000; ">]</span><span style="color: #000000; ">.</span><span style="color: #FF0000; ">[</span><span style="color: #FF0000; ">TestInsertParty</span><span style="color: #FF0000; ">]</span><span style="color: #000000; ">
    </span><span style="color: #008000; ">@partyName</span><span style="color: #000000; "> </span><span style="color: #000000; font-weight: bold; ">nchar</span><span style="color: #000000; ">(</span><span style="color: #800000; font-weight: bold; ">30</span><span style="color: #000000; ">) </span><span style="color: #808080; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">NULL</span><span style="color: #000000; ">
</span><span style="color: #0000FF; ">AS</span><span style="color: #000000; ">
</span><span style="color: #0000FF; ">BEGIN</span><span style="color: #000000; ">
    </span><span style="color: #0000FF; ">SET</span><span style="color: #000000; "> NOCOUNT </span><span style="color: #0000FF; ">ON</span><span style="color: #000000; ">;
    </span><span style="color: #0000FF; ">Insert</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">Into</span><span style="color: #000000; "> Party (</span><span style="color: #FF0000; ">[</span><span style="color: #FF0000; ">name</span><span style="color: #FF0000; ">]</span><span style="color: #000000; ">, chain_idx) </span><span style="color: #0000FF; ">Values</span><span style="color: #000000; ">(</span><span style="color: #008000; ">@partyName</span><span style="color: #000000; ">, </span><span style="color: #0000FF; ">NULL</span><span style="color: #000000; ">)
    </span><span style="color: #0000FF; ">Select</span><span style="color: #000000; "> </span><span style="color: #FF00FF; ">Scope_Identity</span><span style="color: #000000; ">() </span><span style="color: #0000FF; ">As</span><span style="color: #000000; "> Id </span><span style="color: #0000FF; ">For</span><span style="color: #000000; "> Xml </span><span style="color: #000000; font-weight: bold; ">Raw</span><span style="color: #000000; "> (</span><span style="color: #FF0000; ">'</span><span style="color: #FF0000; ">Response</span><span style="color: #FF0000; ">'</span><span style="color: #000000; ">)
</span><span style="color: #0000FF; ">END</span></div>

The trick was to use the XML RAW Mode. This mode transforms the result set into a generic identifier as __. It is however possible to provide a element name, as <_Response>_. Basically this will insert the new value and return something like this from the stored procedure.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">Response </span><span style="color: #FF0000; ">Id</span><span style="color: #0000FF; ">="1054"</span><span style="color: #FF0000; "> </span><span style="color: #0000FF; ">/></span></div>

After return via the send port the orchestration will receive something like the below.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">TestInsertResponse </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">="TestInsert"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Response </span><span style="color: #FF0000; ">Id</span><span style="color: #0000FF; ">="1054"</span><span style="color: #FF0000; "> </span><span style="color: #0000FF; ">/></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">TestInsertResponse</span><span style="color: #0000FF; ">></span></div>

The schema that I use to both handling the response and request against the SQL Adapter is shown below. First I set the type of the Id-attribute to xs:int but this gave me some problems when using the promoted value in the orchestration, everything worked fine when switching back to xs:string.

The same technique would be used for receiving a code from the stored procedure (say 1 for success and 0 for failure or whatever) and then to make a logical decision in the orchestration.

Convoying

2007-04-05T07:21:51+00:00

Convoying is one of BizTalk’s real strengths but it has a lot of pitfalls (I’m talking performance, zombies etc). This article is really good at explaining the different patterns used for creating convoys. It also makes some deep dives into how the subscriptions are solved, why zombies are created and how to deal with them.

Even if you feel you understand the convoy patters from before the part about how BizTalk solves the subscriptions for correlations (the part called Basic Convoy Theory) is great.

Using sequential convoy to handle ordered delivery

A big selling point in BizTalk 2006 is it’s ability to handle Ordered Delivery. It’s important however to understand that this setting (on the Receive and Send port) only works in a pure messaging scenario (a scenario without orchestration, just passing messages between ports). To get order delivery in a scenario using orchestrations one has to use the sequential convoy pattern (basically forcing the orchestration to only one instance on one thread). This Webcast explains this in depth and also deals with some of the different problems that are related to the issue. Problems like performance of course, but also the requirement for receive adapter (on the send side all adapters support ordered delivery) for enabling ordered delivery (you’ll have to use MSMQ, MQSeries or specific scenarios of File, SOAP or HTTP).

Both convoying and ordered delivery are important concepts to understand in depth to be able to make the right decisions in a BizTalk 2006 solution.

Weekly sum up

2007-03-02T08:31:13+00:00

I’ll start trying to post a weekly sum up of BizTalk, .NET topics I’ve come across during the week. We’ll see how goes as I’m not the most constant blogger …

However I worked with mainly three things this week.

Error handling and General Exception

I’ve tested some ideas relating to an error handling pattern we like to implement in a one of our major integration projects. We’ve looked at how the new BizTalk 2006 Failed Message Routing fits in with our project. The idea is to have and Orchestration listening to the Error Message, log these and then put the failing message parts to file. We’ll then have a system that picks up the failed messages, launches them in an editor and makes it possible to resend the message after it’s been edited.

Problem arises when we get to handling exceptions in Orchestrations. Orchestrations doesn’t have the possible to use Failed Message Routing as ports does (I guess that ok, we might want handle different errors in different ways and so on.). However that means that we have to catch an error and create our own Error Message to submit to the message data box for further routing to the Orchestration handling errors (the one that logs and puts the message to file.) So far so good as long as we don’t get an General Exception! Usually when getting an General Exception, and we don’t catch it, the message gets suspended and we get a some error details giving us an idea of what went wrong. However as soon as one catches a General Exception we loose all information about the error … We’ve still haven’t found a good way of handling these errors. We just can’t afford loosing that error information!

General Exceptions are still a mystery to me. I guess that are there so BizTalk has a chance in catching exception that aren’t raised from .NET based code. But what kind of errors within an orchestration raises these kind of exceptions? I know a failure in a mapping does, what else? I’d really like this scenario to better documented. The error handling patter is based on a chapter in the Pro BizTalk 2006 book (which is a excellent book in my opinion - buy it!) but the General Exception and how to deal with it isn’t discussed there either. Any ideas, book, articles are highly appreciated!

Validation

One important thing to think of in a BizTalk solution is to not let anything either in or out that doesn’t validate (ok, there are exception to this, but generally speaking)! BizTalk has great built in support for this in the XML Validator Pipeline Component (you might also want to have a look at this code). However when working with this it’s important to understand what one is validating against and it suddenly even more important to understand every details in the schemas (external system owners usually like some technical explanation when your telling them that their messages don’t validate in your pipeline ;)). Things we run into this week is the elementFormDefault attribute. The following reading help me understanding what the attribute does:

Another thing we ran into was white space handling and the xml:space attribute. Apparently if one likes to have a node with a space in it BizTalk removes this if the element doesn’t have the xml:space attribute. so will come out as - sound familiar?

But if the schema doesn’t declare that the node will have a xml:space attribute the validation will fail! To get this working one has to get the schema declaring the the xml:space attribute and then reference it like this example. The schema from w3c (if you don’t write your own like in the article) is located here.

End-to-end tracing in a SOA

I’ll make this short even if it should be the longest part in this post. Basically we’re trying to archive the following:

Our solution send loads of different messages types between five different BizTalk servers. The client likes to be able to have full text search within these messages and to also be able to see all the messages in each interchange within a server.

So say for example that we receive and Order message in a flat file format. This is transformed in to a Xml message that is then routed down to two other different BizTalk servers. It should then be possible for the client to start an application, click on the Order message type, enter for example the order number (an element i in the message) and see the the full content of the different files within the interchange (in this example that would be the flat file and the Xml file) where that Order number is found.

Basically it’s does what the HAT does (with full text search and a custom GUI). In the first phase of the project we’ll have it work per server, but in the final solution this should perform over all servers … I’ll come back to this in a later post. In the mean time listen to this episode of Hanselminutes on end-to-end tracking. This will be a challenge in every service orientated architecture …

Finally

I’ve sold my iPod Nano and ordered the Creative Vision Zen:M. Now I’ll catch up on all those dnrTV Webcasts! I’ll also make sure to watch everyone of the BizTalk 2006 Webcasts during my commute to work.

Let me know if you found some of this information useful and I’ll try harder to post some like this every Friday.

Simple RegEx replace in C#

2007-02-01T07:22:47+00:00

As I don’t use regular expressions that often I always forget the syntax. So I thought I just put a basic replace pattern up here.

This method takes a schema, finds all places where is says schemaLocation=”whatever” in a text and changes this to schemaLocation=”whatever.xsd“ and then returns the schema.

<div><span style="color: #0000FF; ">private</span><span style="color: #000000; "> XmlSchema FixSchemaLocation(XmlSchema schema)
{
  System.Text.RegularExpressions.Regex locationReplacePattern </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> System.Text.RegularExpressions.Regex(</span><span style="color: #000000; ">"</span><span style="color: #000000; ">schemaLocation=\"(?<location>.*?)\"</span><span style="color: #000000; ">"</span><span style="color: #000000; ">);
  </span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> locationReplaceValue </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #000000; ">"</span><span style="color: #000000; ">schemaLocation=\"${location}.xsd\"</span><span style="color: #000000; ">"</span><span style="color: #000000; ">;
  </span><span style="color: #008000; ">//</span><span style="color: #008000; ">Puts .xsd after the schemaLocation. We need this find the imported schemas</span><span style="color: #008000; ">
</span><span style="color: #000000; ">  StringWriter sw </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> StringWriter();
  schema.Write(sw);
  XmlSchema formatedSchema </span><span style="color: #000000; ">=</span><span style="color: #000000; "> XmlSchema.Read(</span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> StringReader(locationReplacePattern.Replace(sw.ToString(), locationReplaceValue)),</span><span style="color: #0000FF; ">null</span><span style="color: #000000; ">);
  </span><span style="color: #0000FF; ">return</span><span style="color: #000000; "> formatedSchema;
}</span></div>

BizTalk assembly version redirection

2007-01-26T13:34:56+00:00

The version redirection in BizTalk do NOT work as one is used to coming from a ordinary .NET developer background (so I was wrong in this post …). Say that we made a reference to a code library in one of our orchestrations. This code library is in version 1.0.0.0 when we build and deploy the orchestration to BizTalk and to the GAC. The setup in VS 2005 looks something like this.

Then we make some minor changes in the code library, we fix them and set the version to 1.0.1.0. Build and deploy it. This means that we now have one 1.0.0.0 and one 1.0.1.0 version side-by-side in the GAC (as shown in the figure below).

Coming from .NET the CLR should now load the 1.0.1.0 version as it identifies assemblies based on the assembly name, major or minor version (NOT the build and revision version), public key token and culture (more about this here).

However, this does NOT work in BizTalk. BizTalk loads assemblies by fullname it has stored in the management database and the build and revision number are part of the fullname … This means we have to build and redelopy EVERY part where we like to use new version of the code library.

There is one way around this for emergency use. Say that one has a code library that is used in loads of BizTalk artifacts (A place where we store all base functionality for the solution). Rebuilding all those parts and redeploying it, just to be able to update the version number, is not going to happened! It’s to much work and to risky. **So it’s possible to make a change in the BTSBTSvc.exe.config file instead. **In my case the change would be something like the below (read more about here).

<div><span style="color: #000000; "><</span><span style="color: #000000; ">dependentAssembly</span><span style="color: #000000; ">></span><span style="color: #000000; ">
  </span><span style="color: #000000; "><</span><span style="color: #000000; ">assemblyIdentity name</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">StaticCodeLibrary</span><span style="color: #000000; ">"</span><span style="color: #000000; "> publicKeyToken</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">32ab4ba45e0a69a1</span><span style="color: #000000; ">"</span><span style="color: #000000; "> culture</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">neutral</span><span style="color: #000000; ">"</span><span style="color: #000000; "> </span><span style="color: #000000; ">/></span><span style="color: #000000; ">
  </span><span style="color: #000000; "><</span><span style="color: #000000; ">bindingRedirect oldVersion</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">1.0.0.0</span><span style="color: #000000; ">"</span><span style="color: #000000; "> newVersion</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">1.0.1.0</span><span style="color: #000000; ">"</span><span style="color: #000000; ">/></span><span style="color: #000000; ">
</span><span style="color: #000000; "></</span><span style="color: #000000; ">dependentAssembly</span><span style="color: #000000; ">></span></div>

Assembly loading policy from the GAC

2007-01-17T17:27:04+00:00

UPDATE: This does not apply to BizTalk … I’ve made an update post here. Sorry.

We’re working with a lot of code libraries that we use in different parts of our BizTalk solutions. As the are used on several servers and by loads of different “BizTalk parts” (both in orchestrations and maps) it’s important that we always keep the version number of the assemblies up to date. That means that every little change should increase the current version number. But as they are used in so many places people have started to skip this step as they thought they had to compile all parts that should use the new code (say it’s a bug fix and you’d like all “using parts” of the assembly to load the updated version). This is where GAC loading policy comes to the resource!

First we have to understand that every .NET assembly is identified using four characteristics:

Assembly name
Major or minor version
Public key token
Culture

Then we need to know that the first version number in for example version 1.1.2.1 is the major version. The second is the minor version and the third and fourth are build, revision version number. So this means that if you have 1.1.2.1 installed and make a minor change the easiest way to use the new assembly is to change the one of the build or revision numbers (the third or fourth number). Then the CLR will load the new assembly without any other changes!

But sometimes we have to change the minor or major version - and we still don’t have to recompile a thing! We can use a publisher policy file. This is an example of such a file defined for version 1.0.0.0 moving to 2.0.0.0.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">configuration</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">runtime</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">assemblyBinding </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">="urn:schemas-microsoft-com:asm.v1"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">dependentAssembly</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
                </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">assemblyIdentity </span><span style="color: #FF0000; ">name</span><span style="color: #0000FF; ">="BaseHelper"</span><span style="color: #FF0000; "> publicKeyToken</span><span style="color: #0000FF; ">="18517ea673f8584b"</span><span style="color: #FF0000; "> culture</span><span style="color: #0000FF; ">="neutral"</span><span style="color: #FF0000; "> </span><span style="color: #0000FF; ">/></span><span style="color: #000000; ">
                    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">bindingRedirect </span><span style="color: #FF0000; ">oldVersion</span><span style="color: #0000FF; ">="1.0.0.0"</span><span style="color: #FF0000; "> newVersion</span><span style="color: #0000FF; ">="2.0.0.0"</span><span style="color: #0000FF; ">/></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">dependentAssembly</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">assemblyBinding</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">runtime</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">configuration</span><span style="color: #0000FF; ">></span></div>

This kb article describes what to do next:

Change the version and recompile. The first step is to create the new version of your component. After you’ve done that, you will need to modify the version number in the AssemblyInfo file for your component.
Create the publisher policy file. **Create the publisher policy file for the assembly using the format shown above.
**Use Assembly Linker (Al.exe) to create the publisher policy assembly. The Assembly Linker is included with the .NET Framework SDK. To create the publisher policy assembly that redirects a binding from version 1.0 of Website.dll to version 2.0 using a publisher policy file called website.config, run the following command:

<div><span style="color: #000000; ">al /link:BaseHelper.config /out:policy.1.0.BaseHelper.dll /keyfile:c:\keyfile.snk</span></div>

This command will create a new assembly called policy.1.0.BaseHelper.dll. This naming convention is important, as indicated in the “What Is a Publisher Policy Assembly?” section.

Install the publisher policy assembly into the Global Assembly Cache. The publisher policy assembly is installed into the GAC. It will be used by the .NET runtime when any application attempts to bind to version 1.0 of the BaseHelper.dll, and it will force the application to bind to the new version automatically.
**Install the new version into the Global Assembly Cache. **Install the new version of the component into the GAC. After the new version has been installed, the old version can be safely removed.

So no more excuses for not updating the version number!

Understanding persistence points

2007-01-08T12:11:25+00:00

I found this post from Richard Seroter’s blog on persistence points interesting. The post deals with the basics in persistence points and how to tweak them to get a resumed orchestration to resume at an expected step. Not understanding persistence point and not thinking about them when planing and building an orchestration might cause some strange bugs and errors.

Testable data access code

2007-01-03T15:26:55+00:00

As one of the most important rules for a test is that it can’t rely on state, I’ve always found automated data access testing hard. As I view it there are three approaches to accomplish reliable testing of such a layer:

A local database that is restored for each and every test.
Using transactions that can rollback the database after the test completed
Using mock objects

I guess all of them has drawbacks and are suitable in different situations.

It would be possible to have ones data access layer connected to a local database that’s fully restored to a know state for every test. But as the database grows that process would soon get extremely slow. And as one of the requirements for testing and test driven development is that is possible to test often (and if we like to test often is has to be fast) I think this is a method that only works when we have a simple tiny database (something we almost never have …).

Another way of do it would be use transactions to rollback each test operation on the database. A problem here is that we might use transactions within the access layer we like to test! We might even have transactions disabled. This excellent article by Roy Osherove discusses the possibilities of using the transactions from Enterprise Services (COM+) for rolling back database operations. This is an interesting approach and with the new System.Transactions namespace in .NET 2.0 things should be even easier. But still, if one is using transactions in the actually access layer that is being tested we still have to use something else. Another problem is that transactions always will cause a minor performance hit that might grow when using it in a large project with loads of tests.

And then we have mock objects. The first thing we have to think about here is what we should test in a unit test? Should we really test the all the way to the database or should we stay within the boundaries of the application? My option is that a unit test should not test all the way through to the database, then it’s a integration test. However it might be very useful to have such a test in our test suite. If we decide to actually test trough to the database mock object will not do us any good. They will only be able to test that the right method in the data access class were called in correct order.

Some links I found useful:
http://weblogs.asp.net/rosherove/archive/2004/12/10/279258.aspx
http://weblogs.asp.net/rosherove/articles/dbunittesting.aspx
http://weblogs.asp.net/rosherove/archive/2004/07/20/187863.aspx
http://msdn.microsoft.com/msdnmag/issues/04/10/NMock/default.aspx
http://msdn.microsoft.com/library…testwithnunit.asp
http://msdn.microsoft.com/msdnmag…default.aspx

Web service times out before the orchestration completes

2006-12-20T10:09:33+00:00

Today we ran into some problems when making a call to a web service that took more than 90 seconds via the SOAP adapter. After 90 seconds we got a “System.Net.WebException: The operation has timed-out” error in return. It turned out that we had to set the SOAP.ClientConnectionTimeOut context property for the request message. We did this in the message construction using the following code.

<div><span style="color: #000000; ">MyRequestMessage(SOAP.ClientConnectionTimeout) = 200000;</span></div>

I found this post by Thomas Restrepo useful for understanding both the problem and solution. Apparently the default value is 90000 (90 seconds) before the adapter times out. This article on MSDN is also an excellent read when working with web services and BizTalk server.

How to set a custom file name using the file adapter in BizTalk

2006-12-15T10:28:26+00:00

Found this article on the code project on how to set a custom name on a output file - something I recently had to do. It also contains a list of all the macros that are available in the file adapter - so does this post from Owen Allen.

Advanced Microsoft BizTalk Server 2004 and Microsoft Operations Manager 2005 Scenarios

2006-12-15T07:27:02+00:00

Scott Colestock has a vary thorough article on TechNet about MOM 2005 and BizTalk. It’s really amazing what it possible to accomplish when one gets these two solutions to play together.

BOM - Byte Order Mark in BizTalk output

2006-12-13T15:38:05+00:00

Today was another day of new BizTalk problems … The task was easy; produce a XML file and send to a receiving system. Fine. Done. … Not.

The receiving system could not read the file because of some strange characters in the beginning of the file! It looked something like the below when opening the file in a fancy text editor (UltraEdit or TextPad for example).

<div><span style="color: #000000; ">i»¿</span><span style="color: #0000FF; "><?</span><span style="color: #FF00FF; ">xml version="1.0" encoding="utf-8"</span><span style="color: #0000FF; ">?></span></div>

After a couple of very interesting (:/) hours of Googling I found this from Ben McFarlin

> > It is because internal BizTalk messages are in UTF8 format and include the byte order mark. When you added the xml declaration for UTF16 it confused the engine; the declaration read UTF16 but the byte order mark indicated UTF8. > >

When I finally found the cause the solution was easy. Just another of those weird properties … This time it’s called Preserve BOM (on the properties of the XML Assembly pipeline component - see figure on the right). BOM of course stands for Byte Order Mark.

Gmail and Opera Mini on the phone!

2006-12-13T11:51:46+00:00

A couple of months ago I found the new Gmail mobile application. It’s a clean small Java application that one downloads to the mobile phone for accessing the Gmail account.

The application has easy access to almost all of the features that the ordinary web interface has. There was (and probably still are) an mobile version of the web interface but it didn’t have the easy access to all the different menus as this application has. I really like it and use it several times each day.

The supported devices are listed here and to get on your phone all you do is to visit this URL http://gmail.com/app with your device.

Another cool little application is the Opera Mini browser. These a full feature list here. Basically it transforms the pages so that they fit the mobile screen, and it’s fast! It also has some special features for navigation on small screens. And it supports RSS bookmarks!

Nevermind the XML namespaces in Xpath expressions

2006-12-11T15:45:59+00:00

Understanding Xpath expressions is definitely a success factor when working with BizTalk development. Xpath is extremely powerful (when one gets it right) but it’s one of the must frustration techniques I’ve ever worked with! Xpath is one of those languages that either gives you to answer you want to a query - or (more often) just doesn’t give you anything in return! In 99 percent of the cases this means that you missed something in your match. In 99 percent of these cases this means that you got some problems with namespaces in the XML document your querying (This is certainly true when working with BizTalk that “namespace-heavy”). One technique to get around this is to use the same approach as BizTalk itself uses - the Xpath local-name() function!

Consider the following XML document.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">Message </span><span style="color: #FF0000; ">xmlns:ns0</span><span style="color: #0000FF; ">="Standard.Envelope/1.1"</span><span style="color: #FF0000; "> xmlns:xsi</span><span style="color: #0000FF; ">="http://www.w3.org/2001/XMLSchema-instance"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ns0:Envelope</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Header</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">MessageTypeInfo</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
                </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">MessageType</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">.</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">MessageType</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">MessageTypeInfo</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Action</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">INSERTED</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Action</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Addressees</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
                </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">SenderCode</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">Mill</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">SenderCode</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Addressees</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Header</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ns0:Envelope</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Message</span><span style="color: #0000FF; ">></span></div>

The Xpath expression below will hit the Envelope _node. But is’s very fragile! As soon as the namespace prefix changes (It might change to _ns1 and namespace ns0 will be used for something else.) you’ll end up with an empty result.

<div><span style="color: #000000; ">/Message/ns0:Envelope</span></div>

However, if one uses the below expressions instead - that makes use of the local-name function - it ignores the namespace and only cares about the name of the node (Envelope that is ;)). And this is of course far less sensitive to change.

<div><span style="color: #000000; ">/Message/*[local-name()='Envelope']</span></div>

BizTalk Deploy Tool

2006-12-11T14:44:13+00:00

I’m currently working in a stabilization phase on a deployment application for BizTalk 2006 projects. We based the solution on the Enterprise Solutions Build Framework and the BizTalk Explorer Object Model. Basically our solution has a GUI that makes it possible to point out which artifacts one likes to deploy (from the developers local BizTalk server).

The application then figures out all the dependencies that the selected artifacts has (In our solution an Orchestration for example might have > 20 dependencies to different schemas, pipelines, C# libraries etc, etc).

All DLL:s of the different artifacts are then extracted from the local GAC (where they exist when deployed to the local BizTalk). These DLL:s are then packed in to one single deployment package also containing a single XML file that keeps track of the dependencies and the order of witch they have to be deployed in BizTalk (the most depending schemas first and so on). The XML file also contains other meta data information such as ports the artifacts use etc.

This package is then loaded into another part of the application were it’s possible to point out the different servers one like to deploy to. This view then shows information about what has to be done on the server to make it possible to deploy without conflicts (One might have running Orchestration or suspended messages for example that has to be stopped or terminated.). At this stage we also check the naming of the different artifacts. These have to comply with the naming conventions that are configured in the config file of the applications (a warning is shown if the artifact doesn’t validate towards these).

When no warnings (its possible to override a warning) or conflicts are shown one can deploy. We then move the deploy scripts to the servers and use the MSBuild tasks in the SBF to deploy everything for us. We really saved some serious time with this approach and even if it took us a while to get everything working it’s been well worth it!

We have a huge feature list for the next phase of the tool and are planing to and support for BizUnit tests in the DLL:s (so that one gets a warning when deploying a Orchestration without etc). We also like to add more meta data about the port and make it possible to configure new ports as a part of the deployment process. Etc, etc …

Feel free to comment or write we line for further information about this approach. It would also be interesting to hear about other approaches for deployment and what kind of features these solutions have.

ARCast - Patterns and Anti-Patterns for SOA

2006-12-11T13:37:10+00:00

I’m a loyal ARcast listener and I think I’ve listened to most of the shows that ever been produced (at least those that are available in the archive). I also listen to a couple of other podcasts that are focusing on .NET and Microsoft techniques (for example .NET Rocks!) and in my opinion ARcast is by far the most rewarding (I guess this is both a matter of taste and depending on what kind of work one does.)! However these two shows on Patterns and anti-patterns for SOA (part 1 and part 2) really stands out!

They really got me thinking of a couple of anti-patterns the project I’m currently working on are stuck in. And Ron Jacob delivers (as always) the message with a lot of humor and enthusiasm. Recommended!

XML comment in BizTalk output message

2006-12-11T13:18:36+00:00

We ran into an issue today when we had to have an XML comment just below the XML declaration in a message we’re producing in our BizTalk 2006 solution. We needed to produce something like the below.

<div><span style="color: #0000FF; "><?</span><span style="color: #FF00FF; ">xml version="1.0"</span><span style="color: #0000FF; ">?></span><span style="color: #000000; ">
</span><span style="color: #008000; "><!--</span><span style="color: #008000; "> Comment goes here </span><span style="color: #008000; ">--></span></div>

Our first approach was to create a map in the send port and to have a custom XSLT template in there that added the comment before the whole transformation process started. We actually got this to work but it meant that we had to have a whole new map with a separate XSLT document in the project!

Eventually we found a property called Xml Asm Processing Instructions (highlighted in the figure - click it to view it in original size) on the XML assembly component (used in the standard XMLTransmit pipeline).

We just put the comment as a value in this property and we were done! No new artifacts or code! Just a weird property - typical BizTalk behavior!

Virtual Desktop Manager in XP PowerToys

2006-09-29T08:57:21+00:00

Today someone complained about Windows XP’s lack of virtual desktops and that Mac had the possibility to have several desktops active with different content. I remembered that had tried something like that a couple of years ago and after a while I found Microsoft PowerToys for Windows XP again.

The PowerToys package contains several small XP add-ins. For example the Virtual Desktop Manager!

As the image above show the manager makes it possible to have four separate desktops running at one - nice! A few configurations tips that I found useful was turn of the shared desktops option and to set the shortcut for showing the desktops to alt-§ (that’s close to the alt-tab option).

An other cool application in the PowerToys package is the Alt-Tab Replacement that shows a small preview of the different windows one is multi tasking between (as shown in the image above).

Visual Studio Database Project

2006-09-28T09:00:54+00:00

I’ve realized that I never understood the full potential of the Visual Studio Database Project. None of the projects I’ve worked on has had a effective process on handling changes and versioning of the database schema. Test data, build scripts, changes to the local development database versus the test and production base has also been a bit bumpy. I’ve haven’t understood that working with the Database Project potentially could solves a lot of these challenges .

A chapter from the Database Access with Visual Basic® .NET book describes some features in the Database Project.

CSS Control Adapter Toolkit

2006-09-28T06:40:09+00:00

A while ago Microsoft released the CSS Control Adapter Toolkit for ASP.NET 2.0. It finally makes it possible to use all the time saving ASP.NET controls in 2.0 without all the horrible HTML they used to produce. The toolkit gives access to the now improved source of the following adapter controls:

Menu
TreeView
GridView
DetailsView
FormsView
DataList
Login
ChangePassword
CreateUser
PasswordRecovery
LoginStatus

Make sure to have the latest update. The update fixed a few things in the old controls and added a couple more.

One can almost think of the this toolkit as a patch for ASP.NET and when creating a public website this really is a must!

BizTalk Server 2006 Troubleshooters Guide

2006-09-27T20:16:10+00:00

I’ve just started reading the new BizTalk Server 2006 Troubleshooters Guide, it’s basically 133 pages packed with best practices and common pitfalls to avoid. It’s seems to be a great resource and I’ll try to use this blog to summarize some of the different parts - just have to get some more time! ;)

FileSystemWatcher Class

2006-09-25T10:02:04+00:00

I just found the FileSystemWatcher class in the System.IO namespace. It provides the functionality to listen on the file system for changes on a file or directory. The class fires a couple of events: Changed, Created, Deleted and Renamed.

I didn’t know of this class until now. I always thought one had to write a service to accomplish something like this! Definitely very useful!

More BizTalk and integration related Webcasts!

2006-09-20T18:40:59+00:00

I found a couple of really good webcasts here!

VS 2005 project on the old IIS

2006-09-13T06:24:29+00:00

As most people know the web project that are created and tested in VS 2005 run on an internal web server (called Cassini).

In some situations this is useful as it enables one to quickly get the project up and running and start testing. But as the sites usually are going to run on an IIS in production I usually like to test it and develop it on one as well (for other reasons, check this out).

Here’s a good article on how to change the setting that makes VS behave as it it used to (put it on an URL and create a virtual directory for it).

The only drawback I can think of is that it becomes a bit more complicated when it comes to deleting a project …

Excel into BizTalk 2006 - the easy way

2006-09-11T20:08:41+00:00

FarPoint just released a pipeline disassembler component for Excel documents (called “FarPoint Spread for Microsoft® BizTalk® Server 2006”). It looks pretty cool! It comes with a “Spreadsheet Schema wizard” that integrates with Visual Studio 2005. It could potentially save a lot of work.

I guess the reason that Microsoft hasn’t released anything like this is that they like to push the use of Infopath. Basically everything would be much easier if one uses Infopath, but there might be situations where this isn’t possible.

Viewstate

2006-09-11T07:10:45+00:00

Before writing a new user or custom control read this article about Viewstate!

BizTalk Webcasts

2006-09-09T10:58:53+00:00

I’m not sure of how many know of these Webcasts that Microsoft keep publishing. I didn’t know of them until recently and that I’ve learnt a lot by viewing them. I find it a excellent way of learning new things and tips from some of the most skilled people working with BizTalk.

I recommend the following:

Design Human Workflow Solutions with BizTalk Server 2004 (medium)
This is a absolute must to watch. The demo is this presentation is a killer! It’s a complete workflow system build in **four **days that really shows the strengths of BizTalk. It’s also emphasizes some important architecture best practices in BizTalk.
BizTalk Web Services and WSE (medium)
Getting up to speed with BizTalk Server 2004 for the Visual Studio .NET Developer (basic)
Biztalk Server 2004 Architecture** (basic)
Biztalk Server 2004 Orchestration for the Visual Studio .NET Developer (basic)

BizTalk exams passed!

2006-09-09T10:44:33+00:00

Yesterday I passed both the 2004 exam (074-135: Developing E-Business Solutions Using Microsoft BizTalk Server 2004) and the new 2006 BizTalk exam (070-235: TS: Developing Business Process and Integration Solutions by Using Microsoft® BizTalk® Server 2006)!

Webcasts

When I first started studying for the exam I hadn’t really worked with BizTalk. These Webcasts provided an easy introduction to the basics of the products:

The Webcasts are help by Scott Woodgate. He used to be the product manager for BizTalk and knows it inside out (besides being a skilled presenter).

Installation

Next thing I did was to install a Virtual PC with a Windows 2003 server. Then I used this list. The list saved me from some major pitfalls while installing BizTalk (especially 2004 is known to be a messy installation)!

The only problem I ran into during the installation was a collation error on the SQL server (BizTalk Server does not support case-sensitive collations, use Latin1_General_CI_AS).

Tutorials

Then I worked my way through these tutorials. They cover a big part of the product. I can also recommend reading the BizTalk Unleashed 2004 book. Make sure to understand everything in the tutorials as most of it actually will show up on the exam. Make sure to pay extra attention to deployment and the development of business rules! Of course one also has to understand the basics of Orchestration and XML mapping.

Exams

To pass the exams you’ll have to understand the whole architecture and when messages are written to the message database (dehydrate and hydrate). Understand pipelines - develop your own pipeline and deploy it to really understand this part (remember all the interfaces etc). While developing pipeline use the different pipeline testing tools that are available (pipeline.exe etc). Read about message patters and particularly the convoy pattern etc. You will need to know most of the mapping functiods and what they are used for. Use and understand the import, include and redefine functionality in the schema editor.

One big miss I did while studying was the security part and what different BizTalk users groups that exists. Use this list and learn the most important ones (both the Windows and the SQL Server Roles). To pass the exam you don’t really have to know that much about the specifics of BAS, BAM ands HWS - you do need to know what the different solutions are used for and what their limits are.

The new exams are a bit different as they involve setting up a lot of action-lists where one drag and drop actions to create a list of actions for completing a described task. No information about how many actions are required in the finished list etc. However the 2006 is “easier” as it deals a lot with actual development tasks and not so much with security things and details as the old ones (like specific user groups and interfaces etc).

A few tips if your planing to take the 2006 exams is that you study the new BTSTask tool, developing business rules using the Business Rule Composer tool and BAM (Business Activity Monitoring) and especially the process of configuring BAM.

Resources …

2006-08-31T05:59:16+00:00

I’ve started reading this list. Jisses! There’s is literally a ton of links to excellent resources here!

Validation

I’ll work my way down the list when I have some more time. However I found this, seems like it could have saved my a lot if I’d known about earlier. It’s a package with validation controls for ASP.NET for every situation. Looks really good.

Date

This seems like another usefull resource. Loads and loads of different date controls. I’ll have to remeber this site the next time I’ll nedd one.

My first NNUG meeting

2006-08-30T05:32:30+00:00

I just came back from my first NNUG meeting (.NET user group) ever! The meeting consisted of two presentations (plus some Microsoft info about Tech Ed etc), one about the new Windows Workflow Foundation and one about the Provider model in .NET 2.0.

Neither of them were really breathtaking. Besides the fact that deleting the namespace in the web.coning in ASP.NET 2.0 actually enables the XML config code hinting!

<div><span style="color: #000000; "><</span><span style="color: #000000; ">configuration
     xmlns</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">http://schemas.microsoft.com/.NetConfiguration/v2.0</span><span style="color: #000000; ">"</span><span style="color: #000000; ">></span></div>

Becomes

<div><span style="color: #000000; "><</span><span style="color: #000000; ">configuration</span><span style="color: #000000; ">></span></div>

Apparently it’s a bug in Visual Studio 2005! It’s always irritated me as I’ve seen it working in different presentations etc. At leats it’s an easy fix.

The new Windows Workflow Foundation is basically Orchestration from Biztalk that Microsoft put into a isolated component. This makes it possible to call orchestrations like processes (called workflows) from all .NET code.

I also got to talk to a guy with heavy Biztalk 2004 experience. It’s interesting as my new job will involve some major Biztalk development. Apparently Biztalk has some issues consering the XML mapper and the XSLT it generates (I’ve heard that before). Sounds like I got some XSLT writing a head of me …

The Regulator!

2006-08-29T14:26:30+00:00

I had to write a few regular expression for a project today. As always I couldn’t remember the syntax. After a bit of searching I found this excellent free tool!

It saved me a lot of time. It even generates .NET code.

Code formater Plugin for Windows Live Writer

2006-08-26T13:11:57+00:00

I use Windows Live Writer for writing and posting to my blog. However I’ve always had problems with code formating. Until now!

This little plug in to Live Writer does the job!

Thanks to Steve Dunn whos excellt blog you’ll find here