Take a Load Off with Azure Queues

Software today is constantly increasing in scale, performance, and responsiveness requirements. For example, your average website is now expected to provide real-time updates on long-running processes, and the large number of Internet of Things (IoT) products are rapidly increasing the amount of requests to web servers. Old mechanisms for meeting these requirements involved tedious and error-prone programming for developers, which often resulted in unreliable services and frustrated users. Thankfully, developers today no longer have to reinvent the wheel as there are tools readily available to assist with meeting these demands. 

One such tool provided by Microsoft is Azure Queue storage. As should be obvious from the name, this service provides a "queue" for inserting and removing messages. With this seemingly simple service, developers can benefit from some very enticing benefits:

• Doing computation in the background so that web/mobile application requests will not timeout.

• Hosted processing via Azure WebJobs.

• Automatic retry logic when message processing fails for any reason.

• Out-of-the-box clients in most common languages (.NET, Java, Node, C++, PHP, Python, Ruby, and Powershell as of this writing).

• Easy testing/mocking with the Azure Storage Emulator.

Azure Queues are considered to be a Software as a Service (SaaS), where Microsoft provides the full stack to host the service: underlying infrastructure (servers, networks, etc.), operating system, and the application software itself (Azure Queue). From the perspective of a developer, no additional work is necessary beyond signing up with an Azure account, creating a Storage Account resource, and then creating as many Queues as desired. The only gotcha is the monthly bill, but then again nothing in life is free, right? (Actually Microsoft provides a free 1-month trial with $200 credit, for the purposes of assessing whether Azure meets the needs of your application).

It is interesting to note that Azure provides several other similar services for processing queues/messages: Service Bus, Service Fabric, Event Hub, and Event Grid. This article will not directly compare Azure Queues with these services, but they are all useful for various scenarios and could be worth your time to research.

Asynchronous Data Processing

As already mentioned, the basic concept for Azure Queue storage is dead simple: provide a "queue" for processing messages. However, if you dig a little deeper, you will find that there is a bit more nuance to the service it provides. The first thing to note is that the queue is not strict in the order in which messages are handled. In a classic computer science queue, messages are processed in a first-in, first-out fashion (FIFO). However, Azure Queues do not make this guarantee; messages may be removed from the queue in a different order from which they were inserted. What this means is that Azure Queues are slightly less of a queue, and more of a persistent storage mechanism for inserting and removing messages. If your application requires true FIFO operations, then you should look at the comparison of Azure Queues with Service Bus.

Also, Azure Queues do not actually provide a way to host code for the purposes of processing the messages. If you want the messages to actually do anything, you must write code yourself and host it. This is not to say that Microsoft does not provide assistance with this task. In fact, the recommended way to process the queues is to host a WebJob in Azure App Services. The WebJob SDK provides developers with C# classes and attributes for writing their queue processing code. It will handle reading messages from the queue, deleting finished messages, and placing failed messages onto a "poison" queue, among other features. Various settings can be configured (number of retries, number of messages processed in parallel, etc.) to suit the needs of each application.

Note that queue processing code is inherently asynchronous. The code has nothing to do until a message arrives (the WebJobs SDK will handle polling of the queue for messages, with an exponential back-off algorithm to minimize polling), so it is essentially triggered in response to an incoming queue message. The asynchronous nature of queues allows an application to easily throw a message onto the background, rather than processing directly during a request. For any task which may take a long time, Azure Queues are perfect for the job.

A typical Main method for an Azure WebJob is pretty simple and would look like this:

static void Main()

{

    var config = new JobHostConfiguration();

// Gets or sets the number of times to try processing a message before moving it to the poison queue
config.Queues.MaxDequeueCount = 3;
// Gets or sets the number of queue messages to retrieve and process in parallel
config.Queues.BatchSize = 16;

if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}

var host = new JobHost(config);
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}

The developer can then provide static functions for processing queue messages (you can process multiple queues, but there should be one static function per queue). If a matching queue with a suffix of "-poison" is created and configured, then additional static methods for processing "poison" messages can be provided. There are many different inputs which can be provided to these static methods: 

• The contents of the message as a string, byte array, or plain-old-common-object (POCO).

• TextWriter for logging to the console.

• Insertion date.

• Dequeue count.

The following code demonstrates a function for processing a queue with name "queue-name" and its corresponding poison queue called "queue-name-poison".

// This function will get triggered/executed when a new

// message is written on an Azure Queue called queue-name.

public static void ProcessQueueMessage(

    [QueueTrigger("queue-name")] string messageBody,

    TextWriter logger,

    DateTimeOffset insertionTime,

    int dequeueCount)

{

    logger.WriteLine(messageBody);

}

// Handles queue of failed messages.
public static void ProcessQueueMessagePoison(
[QueueTrigger("queue-name-poison")] string messageBody,
TextWriter logger,
DateTimeOffset insertionTime,
int dequeueCount)
{
logger.WriteLine(messageBody);
}

Deployment and Monitoring

Once you have written the code for the Azure WebJob, you can host it within an Azure Web App resource. If you are publishing your project via Visual Studio, then make sure to click "Publish as Azure WebJob" instead of just "Publish" when right-clicking the project.

Azure Web Apps are generally used for hosting web sites, but can also used for hosting WebJobs. You may host as many WebJobs as you like, as long as the web server can handle it. If you are hosting a critical web site in the same Web App, then you should be careful not to host too many WebJobs, or to perform tasks which may cause the machine to run out of memory or crash in some other way. 

If you want to be sure that your Web App and WebJob do not interfere with each other, then no problem. Just create a separate Web App and host the WebJob in there. This will incur additional cost, so the choice boils down to the complexity and budget of your application. A common pattern is to create a single WebJob for performing simple background tasks for you Web App, with additional Web Apps for more intensive queue processing (such as handling millions of IoT messages or long-running tasks).

One more great feature of using Azure WebJobs is the automatically-provided Kudu tools. This is a portal hosted in all Azure Web Apps for the purposes of monitoring and debugging. The main Kudu tools can be accessed at:

[your-web-app-name].scm.azurewebsites.net

The WebJobs section can be accessed by navigating directly to:

[your-web-app-name].scm.azurewebsites.net/azurejobs/#/jobs

This portal provides a log history of all WebJobs running in the App Service, including information such as success/failure, console messages, and date started/finished.

"Fake It Till You Make It" with Azure Storage Emulator

One often overlooked aspect of software is how to handle prototyping and testing of your complete system. Fortunately, Azure Queues have us covered here with the Storage Emulator. This is a free tool which can be run on a local machine and emulates the Azure Queue service which normally runs in the Azure cloud. Developers can build and test against this local setup instead of having to create and publish a real Azure Queue. Only when testing is complete would a real queue be required.

To assist with creating/viewing local queues and messages, Microsoft has created the Storage Explorer. This is a GUI application which allows for managing blobs, queues, and tables in your local Storage Emulator. The tool is jam-packed with goodies, including the ability to manage real Azure Queues using your Azure account.

Go Forth and Queue All The Things

If you are already in the Azure ecosystem, and especially if you are making use of the Azure App Service, then I heartily recommend Azure Queues. I have personally utilized it to great success on several projects. Even if you are not currently using Azure, your applications can still easily utilize it through a REST API or the many provided client libraries.

I hope you enjoyed reading this overview on Azure Queues. If you are hungry for more knowledge, then feel free to roam the Sparkhound blog, where we write about everything from the latest web development trends to business development techniques.

Until next time, have fun and happy coding!

Image Sources:

• Representation of a FIFO (first in, first out) queue

• Storage Explorer

Information and material in our blog posts are provided "as is" with no warranties either expressed or implied. Each post is an individual expression of our Sparkies. Should you identify any such content that is harmful, malicious, sensitive or unnecessary, please contact marketing@sparkhound.com