Simple Hybrid Integration With BizTalk 2016, Azure Service Bus Queues

With any new development or training, we often find ourselves wanting to try something new and need a simple example to get us going. In this post, I’ll demonstrate BizTalk 2016 using a simple hybrid integration scenario with Azure. This simple hybrid demonstration will use BizTalk to pick up a file, securely send it to an inbound Azure Service Bus queue and trigger a Logic App to get Facebook or Twitter data. The data will be sent to an outbound Azure Service Bus queue and BizTalk will securely pick it up and send the messages to a folder location.

What you need:

• Azure subscription
• BizTalk 2016 (I am using developer edition)
• Service bus explorer

Technologies Used:

• BizTalk 2016 Developer Edition
  • File adapter
  • SB-Messaging adapter
• Azure Service Bus Queues
  • SAS Policies
• Logic App
  • Service Bus connector
  • Facebook connector
  • Twitter connector

Create Service Bus Queues

First let’s create a service bus queues we will use in Azure. Login to your Azure subscription.

• Click New (plus sign) -> Enterprise Integration -> Service Bus
• Fill out the Create namespace blade as follows:
  • Name: spr-integration-sbdemo
  • Pricing tier: Standard
  • Subscription: Visual Studio Enterprise (select your subscription here)
  • Resource group: select Create new and name it SprSBDemo
  • Location: Central US (select the region closest to you)

Once it is provisioned, you can search for service bus and select it to get to Service Bus blade.

Create a service bus queue that we can send files to.

• Click plus sign Queue icon
• Fill out the Create Queue blade as follows:
  • Name: InboundFileQueue
  • Max size: 1 GB
  • Message time to live (default): 14 days
  • Lock duration: 30 seconds
  • Uncheck all the options.
    • NOTE:  Enable partitioning is checked by default, so uncheck it.  This is a critical step. If you leave this checked, you will get the following error in BizTalk if you try to send to an Azure service bus queue that has partitioning: This client is not supported for a partitioned entity. The client version should be greater or equal than version ‘2013-10’
  • Click Create

Now let’s create a policy for our InboundFileQueue so that you can securely send files to the queue using a Shared Access Policy.

• Once your InboundFileQueue is created, select it.
• Under Settings, select Shared access policies.
• Click the plus sign Add and populate the Add new shared access policy blade
  • Name: InboundFileWritePolicy
  • Claim: check the Send checkbox
  • Click Create

You will see the following that will get generated for an application to use.

• Primary key (blacked out). We will use this later so you might want to copy it into a notepad.
• Secondary key (blacked out)
• Connection string primary key
• Connection string secondary key

Next we’ll create two Service Bus queues that we can receive files from. The first one we will send our Facebook timeline posts and the second queue we will send our Twitter tweets, both of which we will pull from BizTalk.

First, create the Facebook queue (You will also repeat these steps for the Twitter queue)

• Click plus sign Queue icon.
• Fill out the Create Queue blade.
  • Name: FacebookOutboundQueue
  • Max size: 1 GB
  • Message time to live (default): 14 days
  • Lock duration: 30 seconds
  • Uncheck all the options.
    • NOTE:  Enable partitioning is checked by default, so uncheck it. This is a critical step in that if you leave this checked then you will get the following error in BizTalk if you try to send to an Azure service bus queue that has partitioning: This client is not supported for a partitioned entity. The client version should be greater or equal than version ‘2013-10’
  • Click Create

 Again we will create a policy for our FacebookOutboundQueue. This time our policy will allow you to only securely receive files in the queue using a Shared Access Policy.

• Select FacebookOutboundQueue.
• Under Settings, select Shared access policies.
• Click the plus sign Add icon and populate the Add new shared access policy blade.
  • Name: FacebookReadOnlyPolicy
  • Claim: Read
  • Click Create

Repeat these same steps, and create the Twitter queue.

• Your queue name will be TwitterOutboundQueue.
• Your policy name will be TwitterReadOnlyPolicy.

Here are the two policies once you have them created.

Create Azure Logic App

Now we will create a basic Logic App that will read from the inbound Service Bus queue. It will get either the Facebook or Twitter data, and post to an outbound queue for BizTalk to pick up.

The trigger uses the Service Bus connector. It will trigger this Logic App workflow when there exists a message in the InboundFileQueue. We set it up to check for messages every 10 minutes so we can look at the queue. More on this later when we execute our process.

Next is the condition action. It will check if the xml element named “Type” has a value equal to  ‘Facebook’ using the following expression:

@equals(string(xpath(xml(base64ToString(triggerBody()?[‘ContentData’])), ‘string(/user/type)’)), ‘facebook’)

If Yes, then it will use the Facebook connector and call the Get Page Feed method to get the posts from the Chicago Bears Facebook page. It will then send it to the Facebook Service Bus Queue.

Otherwise, it will get the Twitter tweets from the Chicago Bulls timeline and send it to the Twitter Service Bus Queue.

Configure the BizTalk application

Setup the Folders In Windows Explorer.

Create a parent folder SBDemo:

• Go into the properties for this folder.
• Under the Security tab, click Edit to change the permissions.
• If the BizTalk service account of the host instance is not listed here, you will need to add it and set it to full control.

Under it, create:

• SendToServiceBusQueue (our trigger file drop folder to send to Azure service bus)
• ReceivedMessagesFromFacebookQueue (folder to place Facebook output messages)
• ReceivedMessagesFromTwitterQueue (folder to place Twitter output messages)

Created a facebook.xml and twitter.xml file, which will contain a field called type. This will be used in the logic app to determine whether to get the Facebook or Twitter data.

<user>

<type>facebook</type>

<searchcriteria>ChicagoBears</searchcriteria>

<firstname>carlo</firstname>

<lastname>herrero</lastname>

<userid>myuserid@gmail.com</userid>

</user>

<user>

<type>twitter</type>

<searchcriteria>ChicagoBulls</searchcriteria>

<firstname>carlo</firstname>

<lastname>herrero</lastname>

<userid>myuserid@gmail.com</userid>

</user>

Create the BizTalk Application

1. Open BizTalk Administration console.
2. Create a new application called SPR.Integration.ServiceBusDemo

Sending To Azure Service Bus Queue

1. Create a receive port.
   • Name: PickupFileToSendToAzureServiceBusQueue

2. Create a receive location to pick up our trigger file.
   • Name: PickupFileToSendToAzure.FILE
   • Set Receive Folder location: C:SBDemoSendToServiceBusQueue

1. Create a send port that will subscribe to the RP.PickupFileToSendToAzureServiceBusQueue receive port.
   • Name: SendFileToAzure.SB
   • Setup a filter to subscribe to any messages coming into the RP.PickupFileToSendToAzureServiceBusQueue receive port:
     • Filters: ReceivePortName == RP. PickupFileToSendToAzureServiceBusQueue
   • Select the SB-Messaging adapter as your Type:
   • Click the Configure… button to configure the servicebus connection.
   • In the General Tab, we will need to get the servicebus url for the InboundFileQueue.
     • In the Azure Portal, under servicebus -> InboundFileQueue -> Overview property.
     • Copy the Queue URL.
     • 
     • Replace the “http” with “sb” and this will be your Destination URL.

sb://spr-integration-sbdemo.servicebus.windows.net/inboundfilequeue

1. • • 
   • In the Authentication Tab, we need to configure the Shared Access Signature credentials.
     • In the Azure Portal, under servicebus -> InboundFileQueue -> Shared access policies -> InboundFileWritePolicy
     • Set Shared Access Key Name to Policy Name, InboundFileWritePolicy
     • Set Shared Access Key to Primary key value

Receive from Azure Service Bus Facebook Queue

Create a receive port for the Azure Service Bus Facebook queue.

• Name: GetMessagesFromAzureFacebookQueue

Create a receive location for the Azure Service Bus Facebook queue.

• Name: GetMessagesFromAzureFacebookQueue.SB-Messaging
• Type: SB-Messaging
• Receive Pipeline: PassThruReceive
  • Click the Configure… button to configure the Service Bus connection.
  • In the General Tab, we will need to get the Service Bus url for the FacebookOutboundQueue.
    • In the Azure Portal, under Service Bus -> FacebookOutboundQueue -> Overview property.
    • Copy the Queue URL.
    • 
    • Replace the “http” with “sb” and this will be your Destination URL
    • sb://spr-integration-sbdemo.servicebus.windows.net/facebookoutboundqueue
    • 
    • In the Authentication Tab, we need to configure the Shared Access Signature credentials.
      • In the Azure Portal, under servicebus -> FacebookOutboundQueue -> Shared access policies -> FacebookReadOnlyPolicy
  • • • Set Shared Access Key Name to Policy Name, FacebookReadOnlyPolicy
      • Set Shared Access Key to Primary key value
      • 
    • Create a send port that will subscribe to the receive port receiving the Facebook messages with the following configurations:
      • Name: FacebookQueueFolder.FILE
      • Setup a filter:
      • Filters: ReceivePortName == RP. GetMessagesFromAzureFacebookQueue
      • Type: FILE
      • Send Pipeline: PassThruTransmit
      • In configure
      • Destination folder: C:SBDemoReceivedMessagesFromFacebookQueue
      • File name: %MessageID%.txt

 Receive from Azure Service Bus Twitter Queue

Create a receive port for the Azure Service Bus Twitter queue.

• Name: GetMessagesFromAzureTwitterQueue

Create a receive location for the Azure Service Bus Facebook queue.

• Name: GetMessagesFromAzureTwitterQueue.SB-Messaging
• Type: SB-Messaging
• Receive Pipeline: PassThruReceive
  • Click the Configure… button to configure the Service Bus connection.
  • In the General Tab, we will need to get the Service Bus url for the TwitterOutboundQueue.
    • In the Azure Portal, under Service Bus -> TwitterOutboundQueue -> Overview property.
    • Copy the Queue URL.
    • 
    • Replace the “http” with “sb” and this will be your Destination URL.
    • sb://spr-integration-sbdemo.servicebus.windows.net/twitteroutboundqueue
    • 
    • In the Authentication Tab, we need to configure the Shared Access Signature credentials.
      • In the Azure Portal, under servicebus -> TwitterOutboundQueue -> Shared access policies -> TwitterReadOnlyPolicy
      • Set Shared Access Key Name to Policy Name, TwitterReadOnlyPolicy
      • Set Shared Access Key to Primary key value

Create a send port that will subscribe to the receive port receiving the Facebook messages with the following configurations:

• Name: TwitterQueueFolder.FILE
• Setup a filter:
  • Filters: ReceivePortName == RP. GetMessagesFromAzureTwitterQueue
• Type: FILE
• Send Pipeline: PassThruTransmit
• In configure
  • Destination folder: C:SBDemoReceivedMessagesFromTwitterQueue
  • File name: %MessageID%.txt

 Service Bus Explorer

Open the Service Bus Explorer. In the Service Bus Namespace, select SASConnectionString.  In the Connection Settings fill in the following fields

URI or Server FQDN:  sb://spr-integration-sbdemo.servicebus.windows.net

Namespace:  spr-integration-sbdemo

Shared Access Key Name:  RootManageSharedAccessKey

Shared Access Key:   <put the SAS key for the RootManageSharedAccessKey for your service bus here>

RootManagerSharedAccessKey and SharedAccessKey can be found under your service bus in the Settings -> Shared access policies

Once connected, you will see your queues in the Service Bus Explorer

Start your BizTalk application

1. In the BizTalk Administration console, right click your SPR.Integration.ServiceBusDemo and select Start…
2. Click the Start
3. Under Platform Setting, Host Instances, make sure your host instances are started.

Execute to get Facebook

Let us run and test our process. First, trigger the process to go get the Facebook posts.

Copy the facebook.xml file and drop it into the C:SBDemoSendToServiceBusQueue.

In the Service Bus Explorer, you can see one message in the InboundFileQueue.

Here is the message in the InboundFileQueue

This shows the Logic App fired and succeeded in processing.

We can see each step in the Logic App executing correctly. It received the message, checked the message type, called into Facebook and sent the messages to the service bus queue.

You can see BizTalk then pulled the messages from the Azure Facebook Service Bus queue and sent them to the folder location.

Execute to get Twitter

Now let’s get the Twitter feeds.

Copy the twitter.xml file and drop it into the C:SBDemoSendToServiceBusQueue.

In the Service Bus Explorer, you can see  the message in the InboundFileQueue

This shows the Logic App fired and succeeded in processing.

We can see each step in the Logic App executing correctly. It received the message, checked the message type, called into Facebook and sent the messages to the service bus queue.

You can see BizTalk then pulled the messages from the Azure Twitter Service Bus queue and sent them to the folder location.

That wraps up this simple hybrid integration sample with BizTalk 2016 and Azure Service Bus. Hopefully, this was simple enough to for you to try, get your feet wet and have fun with it.