Correlation with Activity with Application Insights (1) Overview
Now I’m playing with Activity which enables us the distributed tracing with Application Insights. However, it took some time to understand the behavior and relationship with related classes. I want to share what I learned.
I’m going to share it with a series of blog posts.
Why distributed tracing is important?
I work with a lot of Serverless projects. At the first time, people are just excited about the feature and convenience of the development.
However, once a system go on production state, people start to realize the monitoring is very hard. If you choose serverless architecture, you might write small pieces of code with a lot of functions. They talk to each other with a queue or other messaging protocols asynchronously.
If you find an issue, how to identify the root cause? The root cause might be on the same function or upstream functions. You need to read whole logs with comparing these or need to reproduce with a debugger.
The same thing happened in the Microservices world. The solution is distributed tracing. If we could track the relationship between functions, it helps to reduce the time to find the root cause. This tracing mechanism is called distributed tracing. We can do it with Application Insights.
What is Activity?
The Activity, a class that allows storing and accessing diagnostics context and consuming it with logging system. This class takes a key role for the distributed tracing of Application Insights. For more detail, you can refer to this article.
Goal
Understanding the design and behavior of Activity, Request/DependencyTelemetry, and TelemetryClient which are used for a custom code.
In this blog post, I’d like to explain the overview of the concept. Then I’ll tell the internal behavior on the following blog posts. Currently, some of the distributed tracings are tracked automatically for some resources.
e.g., HttpTrigger, ServiceBus Queue of Azure Functions. ASP.NET also support HttpRequest.
It might expand whole Azure. However, you might need to add a correlation for your system which is not supported. Also, this post helps to understand the under-the-hood.
Let’s get started with the fundamental concept.
Data model
You can find the Application Insights telemetry data model in here.
For the correlation, you need to track two telemetries at least.
Request Telemetry
If you have an application, you need to track incoming request. This is called Request telemetry. For example, if you receive an HttpRequest or receive Queue or other messages, you need to track this.
Dependency Telemetry
When your application call the other REST API or send message to other application, you need to track Dependency Telemetry.
Also you can track Exception, Event, Metric, and Custom Trace.
Telemetry Client
Telemetry Client is a client of the Application Insights. You can send the telemetry using this client. For more details, you can refer this documentation.
Correlate the telemetries
Correlation concept is straightforward. If you have child telemetry, pass them the parent information. I’ll show you an example. Although some metric is automatically tracked, I’ll write custom code to understand the behavior.
I implement the sample using Azure Functions. It has two functions. The Function1 receives the HttpRequest then send a queue to the second function. In this case, we need to track three telemetries. For the first functions, it receives a request, then sends queue to other application which means a RequestTelemetry and a DependencyTelemetry is required. Function2 receive a queue. So a RequestTelemetry is necessary.
You can see the example of the correlation. For each telemetry, we have three properties — Id, ParentId, and RootId. Id is the unique Id of the Telemetry. ParentId is the Id of the parent telemetry. RootId is the unique Id for the correlation.
If you implement it, you can see the correlation on the Application Insights on the Azure.
You can see the three telemetries is correlated. You can ignore the 127.0.0.1’s telemetry. Azure Functions track these automatically. Also, you can find the same Id, ParentId, RootId, on the right bottom, although the property names are different.
Implementation
I want to explain the basic usage of the Activity and related class. Activity class manages the Id, ParentId, and RootId for you.
Sending Request telemetry
Since the Activity class manage the Ids, let’s create an instance. It resides in the System.Diagnostics namespace. The parameter is operation name which you can find later.
var requestActivity = new Actvity("Sample: Function 1 HttpRequest");Then start the Activity. The Start() method starts the timer for the Activity and generates the Ids. You can’t modify the Ids after you start the Activity.
NOTE: You can omit this Start() method execution. For more detail, I’ll explain on the next post.
requestActivity.Start(); // You can omit this line.Then you can pass the Activity instance to the TelemetryClient which help you to send telemetry to your Application Insights. You can pass the Activity to the telemetryClient.
The method starts the timer for this operation then create a RequestTelemetry object from the Activity instance. As you see in the Application Insights, the format of the Id, ParentId, RootId is different from the Application Insights screen.
It says Request Id, Parent Id, Operation Id. RequestTelemetry class represents the telemetry. You need to convert from the Activity to RequestTelemetry. StartOperation<T>(Activity activity) method do it for you. You can also manage it by your self.
var requestOperation = telemetryClient.StartOperation<RequestTelemetry>(requestActivity);Finally, you need to stop the operation. It calls Dispose() method. It stops the timer and sends telemetry to the Application Insights. Inside of the method, it calls telemetryClient.Track() method.
Sending dependency telemetry
Dependency telemetry has a parent. Which is the RequestTelemetry. You can correlate it using, SetParentId() method. The method should be placed before the Start() method. If you set the ParentId, Start() method automatically set the RootId as well. Also the Id is automatically generated from the ParentId. You can find the relation between Id, ParentId, and RootId on the Correlation between the telemetries diagram.
var dependencyActivity = new Activity("Sample: Function 1 Enqueue");
dependencyActivity.SetParentId(requestActivity.Id); // You can omit.
dependencyActivity.Start(); // you can omit this as well.Once you call the SetParentId() , the activity becomes the children of the parent activity. In this case, it is the child of the requestActivity.
NOTE: You can omit the SetParentId() and Start() in this case. However, I explain the behavior. Also, these method calls are not harmful. it works.
How to correlate between two applications?
All you need to do is the pass the Id to the child application as an Parent Id.
See the example below. You can find that I pass the dependencyActivity.Id into the Queue message. Then the Function2 receives it and pass it to the requestActivity.
You can find whole sample code on this repo.
Conclusion
I successfully create an app for the correlation. On the next post, I’m going to explain the detail of the relationship between Activity, Request/DependencyTelemetry, and TelemetryClient.
Resources
Activity User Guide
The concept background, usage, and examples.
Track custom operations with Application Insights .NET SDK
The example how to track the custom telemetry.
Application Insights API for custom events and metrics
API for the TelemetryClient.
Telemetry correlation in Application Insights
The document for the correlation concept that Application Insights supports.
Application Insights telemetry data model
The data model and basic concept of the Application Insights telemetry.