When we look at the messaging space on Azure, we have a wide variety of options, each with it’s own strengths and weaknesses. For example, we have Event Hubs, which provides capabilities for processing large amounts of data streams, Service Bus for enterprise messaging, and Event Grid for reactive eventing.
In this blog post, we will look into the the Publish – Subscribe pattern, and how we could implement this using Azure services. For this scenario, we receive messages from one or multiple sources, which we need to route to one or more subscribers, based on properties or content. Of the different messaging offerings, two of these jump out for this scenario, Service Bus and Event Grid. Let’s have a look at when to use which, and the reasons behind these decisions.
When working with Azure Logic Apps, I like to have each Logic App do a single piece of work, as this allows us to mix and match these Logic Apps in various flows. For this demo, we will be using a very simple representation of this, where we have one Logic App which receives the message and send back a response to the original caller, another Logic App which does transformation of the message, and finally a Logic App which calls a backend system. To decouple these Logic Apps we will be using Azure Service Bus topics, providing us with routing capabilities and allowing us to handle downtime more easily.
Now the challenge we were running into, is that we needed to give the response which we received from the backend, back as a response to the client.
Of course, since we have implemented communication between the Logic Apps asynchronously and decoupled by using Service Bus in between, we don’t have a return channel on which we can send the response. In this post, I will show how we can solve this by using Service Bus sessions.
In the previous post, we have seen how we can implement the Message Router pattern when working with Logic Apps. As discussed, Logic Apps are a great fit if you have a limited set of endpoints to which you want to route the message, and if you have a need for various connectors. In this post we will look into another technology to implement this pattern, Azure Service Bus Topics. Topics are a great solution if we want to implement a publish / subscribe mechanism.
- Capability to send our messages to one or more subscriptions in our topic.
- Each subscription represents a virtual queue, from where subscribers can pull their messages, allowing receiving systems to process messages at their own speed.
- Receiver and sender are completely decoupled, so systems can work independently from each other.
- Topics have dead-lettering capabilities built in, so messages are not lost even in case of issues.
- Easily add new subscriptions, so we can quickly on-board new systems.
Azure Service Bus Topics
Last week I attended Integrate 2016 in London, the biggest Microsoft integration event this year, organized by BizTalk360. The outcome was almost 400 attendees, and there were sessions from the Microsoft Product Group, industry leaders and MVP’s.
The major take-away I have from the event, is that Microsoft now has a great vision on the future of integration, which I felt was missing the last couple of years. Now though, they recognize that even though the cloud is a great asset, on premise is not going away for a long time. Also Microsoft has now officially announced that their on-premise integration solution will be BizTalk, which has not been getting a lot of love lately.
In my my previous post, I showed how we can use a WebJob to process a Service Bus queue and store the data in an Azure SQL database. This was pretty simple to set up, but it did require a good understanding of how to connect with these and process the data. Sometimes however we just want to do a quick integration without needing to set up all this plumbing. Recently Microsoft announced a new feature called Azure Functions, with now makes this possible. Azure functions can be used to create a small function which can run stand-alone, or be called from other applications, for example from a logic app, as has been described here by Sandro Pereira. Azure Functions provide out of the box connections for triggers, input and output to a lot of other Azure features, including Event Hubs, Service Bus, Azure Storage and DocumentDB. In this post I will show how we can process our message from the queue we created in this blogpost, and store it in an Azure Storage table. We will start by creating a new Function App in the portal.
This is the fifth post in my series on Integration of Things. In this post I showed how you can send messages from a Raspberry Pi 2 into a Service Bus Queue, and in our previous blogpost we have set up a library for connecting to an Azure SQL database. Today I will explain how we can use a WebJob to retrieve the messages from the queue and send them to our database. The code for this blogpost can be found here.
A WebJob is a simple way to set up a background job, which can process continuously or on a schedule. WebJobs differ from a cloud service (which we discussed in this blogpost) as it gives you get less fine-grained control over your processing environment, making it a more true PaaS service.
We will need a Web App to host our WebJob, so lets create one in the Azure Portal. You can create a new Web App by going to App Services, and selecting New.
This is the third post in my series on Integration of Things. In my previous post I explained how you could send and receive data on a Raspberry Pi 2 to Azure. Today I will explain how you can use an Azure cloud service as a worker role for retrieving the data from Event Hubs using the Event Processor Host library. We will save the retrieved data in an Azure Table Storage, which is a great service for working with large amounts of structured, non-relational data. Azure Table Storage is very fast, and cost efficient especially when working with lots of data, which makes it ideal for our scenario. The code for this blogpost can be found here.
The Event Processor Host library will be used to retrieve the data from our event hub, and load it into Azure Table Storage. This library will distribute Event Hubs partitions accross our instances of the worker role, keeping track of leases and snapshots. This library really makes working with Event Hubs from .NET code a breeze to go through. We will need a blob storage for for the table and for the library to store its data, so let’s start by setting one up via the Azure Portal.
This is the second post in my series on Integration of Things. In my previous post I have explained the scenario and architecture, so today I will explain how you can use a Raspberry Pi 2 to act as an IoT fieldhub. The RPi2 is a low power ARM based single board computer at the size of a creditcard, which can run Windows 10 IoT Core, a windows version created specifically for these kind of boards. This also means we can use the .NET framework to set up our solution. The code for this post can be dwonloaded here.
First we will have to flash your RPi2 with Windows 10. There are some great walkthroughs out there, so I will not explain this here, but instead link you to this site which will explain all the steps to be taken.
After we have flashed our RPi2, it’s time to set up Visual Studio. To develop on Windows IoT Core, we will need to install the project templates. This can be done from Visual Studio by going to Tools, Extensions and Updates, and searching for Windows IoT Core Project Templates. Install these templates, and restart Visual Studio to start creating your own IoT solutions. Don’t forget to enable developer mode on your machine by following these instructions, as this is needed to publish your solution to your device.
Last year I did an IoT session at the Dutch BizTalk User Group, and since then I have had several requests for more information on this topic. After having had a couple of very busy months at my client, I finally decided to make a series of blogposts on this topic. IoT is a major growing industry, and gives a lot of really nice opportunities for us integration specialists. All the code and projects I will be creating will be provided along with the posts, and can be downloaded from here. Coming from a nautical background myself, I see more and more scenarios here where IoT might be a real game changer, so this will be the scenario I will be using throughout these series.
In the upcoming weeks I will be creating my blogposts from the following scenario. In this scenario an engine supplier for ships, like Caterpillar or ABC, might want to get information about the health and status of their engines. In general, a ship has several engines for propulsion (main engine and bow thrusters), generators, pumps on tankers, etc. All these engines already have a lot of data about their health, like temperatures, oil and filter conditions, or issues that might occur during operations. Currently most of this information is displayed to the crew of the ship, and they have to interpret this information, which often means issues are not noticed until it becomes a real problem, and they have to go in for repairs or adjustments, where every day they are not out working can cost a lot of money, especially if the ship is working under contract.
In my previous post I talked about how we are using Service Bus as our queueing mechanic, from which we read messages using BizTalk. When using Paulo Salvatori’s test client we could read these messages in BizTalk, however when we were publishing messages from our own .NET applications, we weren’t able to receive them in BizTalk. These were the errors we got in the event log:
The adapter “WCF-Custom” raised an error message. Details “System.ServiceModel.CommunicationException: Unrecognized message version.
The adapter “WCF-Custom” raised an error message. Details “System.FormatException: Input string was not in a correct format.
After some research, I found out this is due to the way the messages are being deserialized by the BizTalk WCF adapter. To solve this, when placing the messages on Service Bus, we have to publish them as WCF messages, instead of using the Service Bus SDK.