Demo: Logic Apps Using Claim Check Pattern
Azure Integration Services allows seamless integration of enterprise solutions. These services consist of Logic Apps, Service Bus, Event Grid, and API Management. Of these, we use Logic Apps and Service Bus fairly consistently in our client engagements.
I’ve grown familiar with these services, their advantages, and their shortcomings. One specific shortcoming is when a message is greater than the 256Kb size limit; and, unless you select the premium tier limit of 1MB, you’ll find yourself searching for an alternative solution. In the following article, we will focus on a solution for the 256Kb limit—the claim check design pattern—and break down how to run it step by step.
With the claim check pattern, a large message is stored, and the claim check information is eventually sent as a message via a downstream process.
In this example, I will use Logic Apps to retrieve electronic data interchange (EDI) files from an FTP server. Given that the majority of the files are under 256Kb, the payload will be archived, and then pushed into a Service Bus queue. From there, a processing Logic App will retrieve and process the EDI message. In the case of an EDI document that manages files greater than 256Kb, the message will pass the blob path (or claim check) and then the processing Logic App can retrieve the message contents and process it. Below is a diagram of our solution.
For this solution I use the following Azure resources:
- A new resource group
- Logic App to receive files and send to a Service Bus queue
- Logic App to receive and process files from Service Bus queue
- Service Bus queue
- Storage Account
Create Resource Group
Let’s create the resource group for our solution.
Click Resource groups ->Add -> and name the resource group ClaimCheckPattern -> Review + Create -> Create.
Let’s create the resource group for our solution.
Click Resource groups ->Add -> and name the resource group ClaimCheckPattern -> Review + Create -> Create.
Logic App to Get FTP Files And Set Claim Check
Create the Logic App that will retrieve the EDI files.
From the Azure Portal select Create a Resource -> search “Logic App” -> click the Create button
Create with the following:
- Name: GetFtpFile
- Subscription: <select your Azure subscription>
- Resource Group: Use Existing and select ClaimCheckPattern (the resource group you created earlier)
- Location: Central US
Once the Logic App is provisioned, then select the template for Recurrence.
In the first action, retrieve the list of files from an existing FTP server. In this case, you already have an Azure VM with FTP setup in IIS. Let’s connect to this. Click New Step and search for the FTP. Select the List Files In Folder action. Under Folder, click the folder icon to traverse to the folder that contains your inbound EDI files. This action will output a Json array of all of the files in the folder.
Next, loop through each file in the Jjson array. Find the Control connector and select For Each. In the Select An Output From Previous Step, get the dynamic content for List Files in Folder Body as shown below:
You only want the files from your Json array output list, so ignore any item that is a folder. Again, add the Control connector, and select the Condition action. In the left side of the condition express, find the IsFolder property under List files in Folder action output. Select Is Equal To for the compare operation and False as the value to compare to.
When this expression evaluates to true, you want to get the file contents, archive it, and send it to the Service Bus queue. Under the expression path that evaluates to True, retrieve the file contents by adding another FTP connector and select the Get File Content action. Set the File value to the output of the action List Files In Folder and select the Path property.
In the next step, archive the EDI file to a storage blob. The output of this action will return metadata about the blob, including the filename and path. The path will essentially be our claim check to pass on if the files are too big. Set the Specify Folder Path to Upload to the blob container folder location. Use the Name and File Content properties of the List files in Folder action output for the other two values.
Check the file size by checking the Size property from the output of the Create Block Blob. Use the expression:
“@body(‘Create_block_blob’)?[‘Size’]”
Note, you are checking for file sizes less than 250Kb (250000) in this example, not 256Kb.
When this expression evaluates to true, then build the message without a claim check.
- To simplify this demo, we will build a Json message with all the data we need. You can just as easily pass this as metadata properties of the message and not as part of the message payload.
- Set claim check to false and don’t pass the contents.
{
“ClaimCheck”: false,
“Contents”: “@{body(‘Get_file_content’)}”,
“Path”: “”
}
Send the message to the Service Bus queue.
When this expression evaluates to false, then build message with claim check.
Set claim check to true and don’t pass the contents.
{
“ClaimCheck”: true,
“Contents”: “”,
“Path”: “@{body(‘Create_block_blob’)?[‘Path’]}”
}
Send the message to the Service Bus Queue.
Logic App to Process Service Bus Message
Create the Logic App that will process the Service Bus message.
Create another Logic App with the following:
- Name: ProcessFiles
- Subscription: <select your Azure subscription>
- Resource Group: Use Existing and select ClaimCheckPattern (resource group you created earlier)
- Location: Central US
Once the Logic App is provisioned, select the template for When a Message is Received in a Service Bus Queue.
When you receive the message from the Service Bus queue, check to see if there is a claim check. Add a Control connector, and select the Condition action. Note, the message in the Service Bus queue is Base64 encoded, so use the Logic App function base64ToString to decode it. The left side of the condition expression should be:
json(base64ToString(triggerBody()?[‘ContentData’]))?[‘ClaimCheck’]
To break this expression down:
base64ToString(triggerBody()?[‘ContentData’]) — converts the payload from Base64 encoded to string
json(base64ToString(triggerBody()?[‘ContentData’])) — converts the string to a json object
?[‘ClaimCheck’] — selects the ClaimCheck property and it’s value from the json object
For the rest of the expression, select is equal to for the compare operation and False as the value to compare to.
When the expression evaluates to true, then get the contents of EDI file from storage using the storage blob path (claim check). Add a Blob connector and select the Get Blob Content Using Path. To get the path from the service bus message, set the Blob Path expression to:
json(base64ToString(triggerBody()?[‘ContentData’]))?[‘Path’]
When the expression evaluates to false then you can get the contents of the EDI file from the Service Bus queue using the following expression:
json(base64ToString(triggerBody()?[‘ContentData’]))?[‘Contents’]
Testing
Run the Logic App and test the claim check.
Using WinSCP, we log into the FTP location and see you have three files and one folder.
When you trigger your GetFtpFile Logic App to run, you can see the output of the List Files In Folder. The output is an array list of files (and folders) as shown below:
[
{ “Id”: “L0luYm91bmQvYXJjaGl2ZQ==”, “Name”: “archive”, “DisplayName”: “archive”, “Path”: “/Inbound/archive”, “LastModified”: “2019-07-09T08:59:00Z”, “Size”: 0, “MediaType”: “application/octet-stream”, “IsFolder”: true, “ETag”: “\”MDcvMDkvMjAxOSAwODo1OTowMHww\””, “FileLocator”: null, “LastModifiedBy”: null },
{ “Id”: “L0luYm91bmQvZjEtODUwLmVkaQ==”, “Name”: “f1-850.edi”, “DisplayName”: “f1-850.edi”, “Path”: “/Inbound/f1-850.edi”, “LastModified”: “2018-10-18T10:26:00Z”, “Size”: 1922, “MediaType”: “application/octet-stream”, “IsFolder”: false, “ETag”: “\”MTAvMTgvMjAxOCAxMDoyNjowMHwxOTIy\””, “FileLocator”: “L0luYm91bmQvZjEtODUwLmVkaQ==”, “LastModifiedBy”: null },
{ “Id”: “L0luYm91bmQvZjItODUwLTk5Ny5lZGk=”, “Name”: “f2-850-997.edi”, “DisplayName”: “f2-850-997.edi”, “Path”: “/Inbound/f2-850-997.edi”, “LastModified”: “2018-10-05T12:37:00Z”, “Size”: 452, “MediaType”: “application/octet-stream”, “IsFolder”: false, “ETag”: “\”MTAvMDUvMjAxOCAxMjozNzowMHw0NTI=\””, “FileLocator”: “L0luYm91bmQvZjItODUwLTk5Ny5lZGk=”, “LastModifiedBy”: null },
{ “Id”: “L0luYm91bmQvZjMtODY3LmVkaQ==”, “Name”: “f3-867.edi”, “DisplayName”: “f3-867.edi”, “Path”: “/Inbound/f3-867.edi”, “LastModified”: “2019-07-09T15:05:00Z”, “Size”: 304189, “MediaType”: “application/octet-stream”, “IsFolder”: false, “ETag”: “\”MDcvMDkvMjAxOSAxNTowNTowMHwzMTExODk=\””, “FileLocator”: “L0luYm91bmQvZjMtODY3LmVkaQ==”, “LastModifiedBy”: null }
]
The last entry in the array is the file f3-867.edi. This will trigger your claim check condition. Let’s look at the results for this. Since there are four entries in this array list, in the For Each action, click the Next > link to cycle through the array values until you find the f3-867.edi file.
In the next step, when the file is archived to blob storage, you can see the size of file in the Output of the action. The file has a size of 311,189 (311Kb) and thus is greater than the max service bus message size limit of 256Kb. Also note that WinSCP displays the file as 304Kb in size, but Azure storage has it as 311Kb.
This meets the condition for a claim check, so compose the message to write to the Service Bus queue with the ClaimCheck value set to true.
It will send the message to the queue.
Now, switch to see the run instance for the ProcessFiles Logic App. It will pick up the message with the claim check from the Service Bus queue. The condition will be met to check if the ClaimCheck value is true, and it will retrieve the content from blob storage using the path passed in the message.
The file is pretty big so it won’t display in the output. Hit the Click to Download link and it will open another window with the output. You can look for the $content field and the value is the contents of the f3-867.edi file. This is Base64 encoded, so go to https://www.base64decode.org/ and paste the content to decode and retrieve the EDI contents.
That concludes our example of the claim check design pattern. This is a very handy design pattern, especially when passing messages using a Service Bus queue and taking advantage of its other features including dead lettering, custom policies, and detection of duplicate messages. While nearly 98-99 percent of messages are under the 256Kb limit, the Service Bus queue should not be completely discarded, especially when a claim check pattern workaround is available.
For more information on this topic, please do not hesitate to reach out here.
