Azure Function to Cosmos DB message pipeline via service bus
Service Bus
Service Bus is a middleware integration tool hosted on the Microsoft Azure Cloud platform. It acts as a broker between different applications and services. Service bus aims at enabling communication by decoupling applications. The platform is reliable and offers asynchronous transport. Messages are sent and received through queues and topics.
Azure Functions
Azure functions enable serverless coding without worrying about the infrastructure. They allow us to run small chunks of code that get triggered by specific events. The triggers supported by Azure functions include responding to changes in data, responding to messages, running on a schedule or due to some HTTP request
Cosmos DB
Cosmos DB is a fully managed NoSQL database hosted on the Microsoft Azure cloud platform. The main features of Cosmos DB include high response time, automatic and instant scalability options, speed scale, and availability.
What we are doing
Our aim is to develop a communication pipeline between an HTTP trigger and Cosmos DB via service bus topics. We will be sending weather information triggered through an HTTP Function through service bus and the retrieved information will be logged in a Cosmos Db database.
Pre-requisites
· Azure subscription
· Microsoft .Net Framework
· Microsoft Visual Studio (minimum version VS 2019)
Solution
The solution would comprise of the following steps:
Service Bus resource on portal
1. Create a service bus resource on the Azure portal
2. Create a topic. The one shown above Is called “az_training_topic”.
3. Create two subscriptions called “sub1” and “sub2” for the same topic.
Create a Cosmos DB database on Azure Portal
1. Create a Cosmos DB resource in the portal
2. On the cosmos DB dashboard go to Data Explorer and create a new database called “WeatherInfo”.
3. In the database create a new container for storing the data called “CityWeather”
Save the connection strings from both the service bus and cosmos DB resource
Visual Studio Code
Sender Side
1. Initialize by adding the variables:
a. Service Bus Connection string
b. Topic name
2. Add an HTTP trigger that accepts that city name from the user through a post request.
3. The city name received is sent to the weather API and a JSON response is received that has the complete weather information of the specified city/state.
4. The weather information is sent to the service bus topic
5. The System.Net.Http library is used to send the request by adding the city name in the request URL.
6. The secret variable names such as API key and service bus connection string should be passed by using a config file for security reasons and they should never be shared with anyone.
The complete sender side logic is shown below:
Receiver Side
1. On the receiver side we are using a server bus trigger that runs and receives the data stored in the service bus topic.
2. For this purpose, we will pass the name of variables such as the connection string, topic name, and subscription name as input binding variables.
3. For the purpose of logging data in the cosmos DB database we will add the cosmos DB output binding variables as well that include the connection string, database name, and collection name.
4. The output binding sends the incoming data to a collection of a specific database and creates a new document.
5. Finally, we are binding the document to the data we are receiving in the final code logic on line 19.
Local.settings.json
This file will contain the variables that are to be picked in the code like cosmos DB connection string, service bus connection string. The values are stored in the form of string key-value pairs.
Testing
1. Run the Sender side App.
2. You will get an HTTP URL that can be hit with a POST request.
3. Send a post request through Postman by specifying a city/state name in the body of the request.
4. The weather information is stored in the service bus in the form of a string Message.
5. On the receiver side we are receiving through our subscription “sub1”.
6. It is visible that the function executed successfully. The data received through the service bus is visible as well.
7. Since there is no error, we can safely say that the data got inserted to cosmos DB.
8. It can be seen in the data explorer snapshot above that a new document got created.
9. The Id was assigned by using a GUID in code.
10. The description contains the weather information of the city we requested.
Conclusion
The blog aimed at achieving an end to end communication between a weather API and cosmos DB using Azure Functions and service bus as the middleware between them. Using Azure function providers, the developers with a lot of flexibility to play around with their code. We saw how an HTTP request at one end can call an API and the data was logged to the database on the receiver end. This scenario can be further extended by:
· Introducing C# object models and using serialization/deserializations functions at both ends
· Using multiple subscribers for receiving data based on different filters applied to the messages. The filters can be SQL filters or correlation filters.
· Chaining multiple queues and topics
· Insertion in other databases along with Cosmos DB like SQL Server or MySQL
Weather API:
SDKs Required
· Microsoft.Azure.ServiceBus
· Microsoft.Azure.WebJobs.Extensions.CosmosDB
· Microsoft.NET.Sdk.Functions
