Choosing your pub-sub messaging service; Service Bus and Event Grid

When we look at the messaging space on Azure, we have a wide variety of options, each with it’s own strengths and weaknesses. For example, we have Event Hubs, which provides capabilities for processing large amounts of data streams, Service Bus for enterprise messaging, and Event Grid for reactive eventing.

In this blog post, we will look into the the Publish – Subscribe pattern, and how we could implement this using Azure services. For this scenario, we receive messages from one or multiple sources, which we need to route to one or more subscribers, based on properties or content. Of the different messaging offerings, two of these jump out for this scenario, Service Bus and Event Grid. Let’s have a look at when to use which, and the reasons behind these decisions.

Event Grid Preferred

These days, I use an Event Grid preferred strategy, meaning that I will use Event Grid Topics as my message routing solution, unless we need one of the Service Bus specific capabilities. This means we will often use custom topics in Event Grid, to implement decoupling between systems, especially when implementing these inside of Azure. There are several reasons for this strategy.

  • Follows the serverless model, so we only pay for what we use, while taking advantage of a highly scalable platform. And the costs are low as well, at just €0,506 (yes, that’s just over 50 cents) per million requests. Additionally, if we want to send messages larger messages than 64KB, we don’t need to change anything, as this is supported up to 1MB, where we pay per 64KB chunk. On a personal opinion, once messages are going over this kind of size, we should really be looking at other options for our messaging needs, either by using a different service, or by implementing a Claim Check pattern.
  • Has capabilities for the most important messaging features, like dead-lettering and retry mechanisms , including backing off retry mechanisms and back-offs if the receiver is throttling. This ensures we have guaranteed delivery on our messages.
  • Advanced message routing capabilities, which can use the event type, subject or even on the body payload. And the filtering on body payload works for up to ten levels deep in our payload, without the need to set any headers or anything else.
  • Uses a push – push mechanism, meaning we no longer need to poll for new messages. Consequently, messages placed on my topic are pushed directly to their subscribers, meaning we have minimal latency for message processing. Also, in a world where we pay per use, this means lower costs, because we no longer pay for our polling actions.
  • Provides out of the box integration with various Azure services, both as event sources and event handlers. Moreover, it’s even possible to use events cross cloud thanks to the CloudEvents standard, which is supported by Event Grid.
Event Grid model of sources and handlers

Service Bus

So does this mean we are going to implement everything with Event Grid now? No, like I said in the introduction, we may have scenarios where Service Bus provides additional capabilities, which are mostly focused on advanced enterprise messaging. Therefor, if we need one or more of these capabilities, we will still use Service Bus for our pub – sub needs.

  • Order delivery, where messages need to be transferred in the exact order as they were received. This is something which is not possible with Event Grid, and as such would need to become part of a custom implementation. Service Bus on the other hand has support for sessions, which combined with partitions can help you creating an ordered delivery system. It is important to note, that each piece of your solution needs to be prepared for this kind of pattern, otherwise we will just shift the problem. For example, if a Logic App or Function is used to process the messages, make sure to adjust them to implement a singleton.
  • Transactions are another capability provided exclusively by Service Bus. Here messages can be part of a transaction, and as such also be rolled back inside this transaction. Due to the decoupled and distributed nature of the cloud, personally I hardly come across this requirement anymore, instead we use compensation techniques to roll back in these cases.
  • When the client needs to do a pull instead of receiving a push, for example because it is not up at all times, or because it has low processing capabilities. We find these kind of restrictions mostly in legacy systems. In this case we will implement Service Bus as our decoupling point, and the receiver can process the messages at its own pace.
  • Deduplication of messages, which uses the MessageId of a message to determine if it is a duplicate. This is especially useful when messages are sent over instable networks, and the receiver does not implement idempotency.
  • In case there is a need for temporal actions or forwarding. Service Bus has the option to store messages, and only make them visible after a certain amount of time / at a certain moment. Also, Service Bus has the option to forward messages from one queue to another, which is ideal in migration scenarios, or to implement additional routing rules.
Image result for service bus forwarding

Using together

As is often the case on Azure, it should not be a matter of choosing a single service, but instead using a best of breed landscape. The same can be said in the pub – sub messaging space, where we can use Event Grid in conjunction. Especially when working with external third party applications, Service Bus can be a great decoupling point. Event Grid on the other hand is ideal to use within your internal Azure landscape, integrating with many of the services. in practice we often use Service Bus as a source and destination of Event Grid, while doing internal routing in our solutions all through Event Grid.

So in conclusion, use Event Grid by default, but don’t hesitate to bring in Service Bus for those specific scenario’s.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.