A loosely coupled Scatter-and-Gather implementation In BizTalk 2006
Why loose coupling?
There was a question in Microsoft BizTalk forums the other day about how one could implement a Scatter and Gather pattern in a more loosely coupled fashion.
Most examples on the implementation of this pattern in BizTalk use the Call Shape functionality available in BizTalk orchestrations. This however creates a hard coupling between the “Scatter orchestration” and it’s “partner orchestrations”. The downside of that is that when one adds or removes a partner the whole solution has to be recompiled and redeployed.
If one however could use the publish-subscribe architecture in the MessageBox to route messages between the “Gather orchestration” and it’s partners, it’d be possible to add partners without having to worry about the rest of the solution. This post shows and example on how to implement a solution like that.
#
The BizTalk process in steps
> > [](../assets/2008/01/aa559774-noteja-jpmsdn-10.gif) NOTE: Notice the difference between the **Partners**Request, the **Partners**Response, the **Partner**Request and the **Partner**Response messages. The names are unfortunately very similar. > > > > The **Partners**Request and **Partners**Response messages are used for communication between the Scatter and the Gather orchestrations. It's also a **Partners**Request message that activates the process. > > > > **Partner**Response and **Partner**Request are used for communicating between the Scatter and the Gather orchestration and all the partner orchestrations. > >
-
Request and scatter
A PartnersRequest message is received. This message is an empty message and is only used for activating the process in this example scenario. The PartnersRequest message is consumed by the Scatter orchestration. The Scatter orchestration creates one PartnerRequest message. The orchestration also generates a unique key called a RequestID and start a correlation combining that that id and the PartnersRequest MessageType. Finally it post the PartnerRequest message to message box, writes the generated RequestID to the request messages context (RequestID is a MessageContextPropertyBase based context property) and dehydrates itself.
-
Partners
All the enlisted Partner orchestrations pick up the PartnerRequest message from the message box. These orchestration then communicates with their specific data source (could be a service, database, file or whatever), receives a response. Finally these orchestrations transform the response they received and creates a PartnerResponse message that’s posted to the message box. Notice that the RequestID that was generated by the Scatter orchestration is also part of the context of the newly created PartnerResponse message. -
Gather
The PartnerResponse messages are routed to the Gather orchestration. This orchestration uses a Singleton pattern based on the RequestID which all PartnerResponse messages carried with them in their context. This means that it’ll receive all the PartnerResponse messages containing that same RequestID into the same orchestration instance (ergo all the Partners that were activated by the request message being sent from one Scatterer). For each message instance it receives it add it’s price to a total price variable. When the Gather orchestration has received all the PartnerResponse messages (the orchestration knows how many Partners responses it should expect from one Scatterer orchestration and we can timeout if we don’t get all expected with a timeframe) the total price we calculated is written to a PartnersResponse message. -
Response
This message is routed back the Scatter orchestration by using the correlation it initialized in the start. It’s finally this orchestration that send the final outgoing message (a PartnersResponse message).
Example solution
An example of the implementation can be downloaded from here.
The solution contains five different schemas.
-
PartnersResponse
Used for initializing the process. -
PartnersRequest
Send from the Gather orchestration to the Scatter orchestration. It’s also the final result and outgoing message from the process. -
PartnerRequest
Picked up and activates all enlisted Partner orchestrations. -
PartnerRequest
Send from the Partner orchestration containing the result from the Partner Service and send to the Gather orchestration. -
**LooselyCoupledScatterGatherExampleProperties
**Property schema for storing the RequestID and to correlate all the PartnerResponses as well as the final PartnersResponse back to the Scatter orchestration.
Five orchestrations
-
**Scatterer orchestration
**The “main orchestration” that receives a request message from outside and “scatters” party requests to all the party orchestrations. -
**Gatherer orchestration
**Gatherer orchestration that gathers all the responses from the partners and transforms these to a reply that is being routed back to the Scatterer and back out. -
Partner1, Partner2 and **Partner3 orchestration
**Partner orchestrations that communicates to different services and receives price information.
Setting up and testing the example solution - it’s easy!
When the solution is built and deployed one needs to setup and bind two ports; one outgoing port and one incoming port (this could also be a Request-Response port by changing the port type in the Scatterer orchestration). That’s it!
Enlist and start everything by dropping a PartnersRequest test message (you’ll find one among the zipped files) in the incoming folder. A PartnersResposne message should then be published in the outgoing folder containing a calculated price from all the Partner orchestrations.
Test message and a binding file are part of the zipped solution.
What would be different in “real life” solution?
I’ve made some major simplifications in this example to make it easy for setting it up and test the concept. These would very different in a “real” solution.
Partner Services
The Partner orchestrations are very simple. They actually don’t communicate with outside world at all. All the do is setting a hard coded price and post a response. In a real solution these would not be part of the same solution as the Scatter and Gather orchestration (otherwise we would be force to redeploy when adding a Partner orchestration to the dll).
The Partner orchestrations would also communicate with some sort of outbound source like a web service or database for example. This would however complicate the setup therefore I’ve skipped that part in the example.
Managing partners
One of the benefits with a loosely coupled implementation is the possibility to add and remove Partner orchestrations without having redeploy the rest of the solution. Using this implementation the Gatherer orchestration needs to know how many Partner responses it should wait for before timing out. This requires that value being set in a config file or something similar. In this example the number of Partners are hard coded into the Gather orchestration (it’s set to 3 Partners) to simplify the setup.
Final thoughts
Knowing how to create loosely coupled solutions like this is good knowledge to have. It’s my own and others belief that this architecture makes it possible to create more robust and separated solutions that one can update without having to do a lot a work and disturb the current processes. It’s however not the best solution performancewise as it adds a lot of extra hits on the MessageBox database and generates more work for the MessageAgent.
There are also a few things to watch out for:
Eternal loops
It easy to end up in a situation where you’re subscribing to the same message as you posting to the MessageBox. That’ll create and endless loop and cause a lot disturbance before you’ll find it. Think through and document you subscriptions!
Correlations for promoting values
When doing a direct post in BizTalk most properties are not promoted. To force you properties to be promoted you’ll have to initialize a correlation on the property as you send it. I can’t say I like this. There should be a other way of saying that one wants it promoted.
A couple of other useful articles as we on the subject:
Download the example orchestration and let me know how you used it and what your solution looks like!