WABS Integration Patterns – Content Based Routing
Gregor Hohpe’s EAI Book (and site) has detailed information on design and implementation of enterprise integration. In traditional BizTalk Server implementation, a framework has been created on proven orchestration design and is available on codeplex.
This blog is the first in several blog entries in which in we will describe and implement an integration pattern utilizing current on premises tools like BizTalk Server and Windows Azure BizTalk Services (WABS).
The patterns that we will address will be the most common integration patterns falling into message routing groups.
Message routing groups primarily focus on decoupling the source message that can be routed to destination message based on different set of filter conditions.
In our first scenario, we will address one of the most common integration patterns: content based routing. In content based routing, messages are routed to a single or multiple destinations based on the content of the message itself. It is a core feature of most integration platforms and helps provide the framework for publish-subscribe architectures. When using content based routing, it is often ideal to have dynamic subscriptions to allow very loosely coupled systems to be built around the process.
Scenario
Contoso accepts orders for multiple brands of products they sell from many different partners in many different formats. Contoso partners could be sending order data either via FTP in flat file or via web-services. Along the way the message must be enriched by looking up the brand name based on a product id in the message. Their order fulfillment is often based on one of their 2 major brands, ContosoBusiness and ContosoConsumer. In all other cases, the orders are fulfilled regionally, so the message must also be augmented with a region as well. This is done by looking up the region by the ShipToCustomer name. Once the order is enriched, ContosoBusiness orders and ContosoConsumer orders are placed in separate holding areas for further processing by the back end line of business system. All other orders are also routed to a holding area that will be accessed by the regional fulfillment centers. Each regional fulfillment center will then subscribe to these order messages and process them appropriately.
BizTalk Server Approach
This is one of the most common patterns within BizTalk server. To implement this pattern as BizTalk Server Solution we would:
- Create a canonical order schema that promotes certain properties to the message context.
- For the partner that sends flat files, we would create a flat file schema and a map to the canonical order schema.
- We would put this on an FTP port and allow the message to be received, transformed, and published to the message box.
- For the partner sending us orders via a service, we would expose our schema as a WCF service and use SOAP/REST endpoints.
- In this case we would receive a message and pass it straight to the message box as well.
- Our schema has promoted properties, We would then create send ports for ContosoConsumer and ContosoBusiness and use filter expressions to pull these messages out of the message box based on the content value that was promoted.
- We would do the same for the regional fulfillment center. As new regional centers are added, configuration will have to be done to add appropriate subscriptions at a later date.
WABS Integration Services Approach
We will implement the same scenario using Windows Azure BizTalk Services (WABS), creating a hybrid approach leveraging Azure service bus and On-Premises BizTalk Server SB-Messaging adapter bridging connectivity between Cloud and on-premises server.
The figure below represents a BizTalk Service Project. We do very similar processing as in the BizTalk scenario with some different terminology and some extended features. As can be seen, we have 2 Bridges that allow input into our process:
OrderBridge
XML bridge that allows messages to be accepted via a REST or SOAP service bus endpoint. This bridge accepts the message in canonical form but does not do any further enrichment or transformations of message received.
The figure below shows Orderbridge configuration.
Below is the canonical form of message that is received in OrderBridge.
FfOrderBridge
FfOrderBridge is an XML Bridge that is surfaced through an FTP Source. The source allows a pull from an FTP site. The bridge itself does the parsing from flat file to an XML message and runs a transformation to map it to canonical order message.
The figure below shows the FfOrderBridge configuration.
Message received on FfOrderBridge is a flat-file schema as below:
Flat-file schema was created using Flat-file schema wizard for sample below:
00001,Acme,Fabrikam,WA
00001,B00001,1,1.45,2014-01-28
00002,B00002,2,5.55,2014-01-27
Flat-file message received into FfOrderBridge was transformed to canonical using newer set of features and functions (List Operation, For Each and Map Each) available in BizTalk Service Mapper tool.
Once the message is processed by both the bridge, it is in canonical form, at which point it is enriched. Message enrichment is done in another XML Bridge (OrderCanonicalBridge) at which both Bridge are connected to.
OrderCanonicalBridge
OrderCanonicalBridge creates message properties by doing XPATH lookups (STCustomer, BTCustomer, Pid) and SQL Azure Lookup to determine ProductBrand (Bid) and Region (RegionId). Product Brand is determined based on ProductId and Region is determined based on ShipToCustomer.
Following screenshot displays the OrderCanonicalBridge and all the properties that are enriched in Enrich Stage within Bridge.
Xpath properties are created using the standard “Instance Xpath” property from within the Schema.
SQL Azure lookup two tables are created with following definition:
CREATE TABLE [dbo].[ProductCbr](
[Id] [int] NOT NULL,
[ProductId] [char](50) NOT NULL,
[Brand] [char](100) NOT NULL,
CONSTRAINT [PK_ProductCrb] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)
CREATE TABLE [dbo].[RegionCbr](
[Id] [int] NOT NULL,
[ShipToCustomer] [char](150) NOT NULL,
[Region] [char](50) NOT NULL,
CONSTRAINT [PK_RegionCbr] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)
Two SQL Azure table lookups are performed and the properties are set for both as Bid and RegionId. At the time of creating lookup following connection string should be used (Important to specify username as user@servername).
Server=<ServerName>.database.windows.net,1433;
Database=biztalk_db;User ID=<User>@<ServerName>;
Password=<Password>;Trusted_Connection=False;Encrypt=True;
Routing and Filtering
Once the enriched message exits the bridge, it is sent to one of three destinations:
- Messages with BId=”Consumer” are routed to the Consumer queue. Filter conditions are set on the connector between Bridge and Destination as below.
- Messages with BId=”Business” are routed to the Businessqueue.
- All other messages are routed to a topic named Regional for further processing. Topics give us the ability to add subscriptions on the fly and for consumers to get copies of the message for their own processing. Once the process is put together, it can be deployed to the WABS for processing using Visual Studio Deployment. At the time of deployment you will need to enter the secret key and the BizTalk services namespace information.
Once the artifacts are deployed, they can be viewed and verified in BizTalk Services portal as shown below:
Tracking
The XML Bridges support tracking in three different ways:
- Exceptional messaging that occurs during that bridge’s processing.
- Tracking of message flow information.
- Selective tracking of properties.
The picture below shows how tracking is configured at bridge level.
The user can select the properties to be tracked and whether or not to track flow data. The tracking data can be then viewed from BizTalk Services portal for debugging and troubleshooting purpose.
Below is the detail tracking information on all the activity performed when the bridge was executed.
Upon clicking on detail on one of the tracking activity, it will show the detail message information (in case below it shows the all Enriched properties that we tracked and were promoted).
Pre-Requisite :
- Install Windows Azure BizTalk Services SDK. Developer SDK features the support for BizTalk service Project template in Visual Studio
- Provision BizTalk Services
- Required information to Deploy BizTalk Services Project
Testing :
FTP Route:
- Drop FfOrder1.txt (Consumer) or FfOrder2.txt (Business) into FTP Source Location (free FTP hosting is available at DriveHq.com)
- FfOrderBridge will pickup the data from FTP Source
- FfOrder will be transformed to Canonical Order
- It will be sent to OrderCanonicalBridge
- OrderCanonicalBridge will enrich data (Xpath and Lookup) after validating data
- Based on the route filter condition data will be routed to appropriate queue or Topics
Conclusion
This example showed how the WABS are beginning to show some new and exciting features the Windows Azure Cloud can provide. We were able to accomplish a very common scenario that is encountered with enterprise integration. The Integration workload along with the Service Bus will provide a very robust messaging platform moving forward. The queues and topic infrastructure gives us a much more dynamic way to route and subscribe to messages in our integration solutions.
