Correlation with Activity with Application Insights (2) Activity deep dive

I wrote the overview of the distributed tracing with Application Insights.

Correlation with Activity with Application Insights (1) Overview

On this post, I want to expand the deep dive knowledge of the Activity and related class, Request/Dependency telemetry, and the background concept.

HTTP Correlation protocol

There are several correlation protocols. Activity is building on the top of the HTTP Correlation protocol. You can refer the detail on the link below. You can say that Activity is the implementation of the HTTP Correlation protocol.

I’ll explain the other correlation protocol in the next post.

dotnet/corefx

Sharing Activity among the call graph

You might want to pass the correlation information through your application. For this reason, Activity has a static property Activity.Current. If you start an Activity, it sets the Activity instance to Activity.Current. Inside the Activity.Current, they keep the Activity instance using AsyncLocal. You can share the Activity instance among the call graph. Also, you can refer to it by Activity.Current.

var activity = new Activity("HTTP Request");// create an instance
activity.Start();  // create a new Id and add to AsyncLocal

You can refer the activity anywhere on the call graph like this.

var activity = Activity.Current;

I help to share the Activity instance. One thing you need to care is, if you start the call graphs, it is valid only the child async context.

Call graph and scope of the Activity.Current

Let me expand. There are some methods. An async method A call async method B, and method B calls async method C. If you start an activity at Method A, It is shared among the three methods.

However, if you start the Activity on Method C, It is only shared among Method C or the MethodC’s children.

AsyncLocal Class (System.Threading)

SetParentId()

Activity helps us to create the correlation using HTTP Correlation protocol. Let’s see the more detail. If you set the Id to the SetParent(), it automatically set the RootId, and ParentId, but how? The secret is the HTTP Correlation protocol.

The first Activity

The Id of the newly created Activity is like this.

|0S9vTv1SnAI=.fa0f9df2_

Parent Id

0S9vTv1SnAI=

RootId

0S9vTv1SnAI=

As you can see, the RootId is the part of the Id. Then you SetParentId() to the child Activity with the Id.

Child Activity

Id

|0S9vTv1SnAI=.fa0f9df2_2. // This is generated after Start().

Parent Id

|0S9vTv1SnAI=.fa0f9df2_

RootId

0S9vTv1SnAI=

As you can see, the Activity can get RootId from ParentId. Id is null at first. However, once you call Start() method, it is automatically generated with adding new value after the ParentId. You can set the ParentId without using SetParentId(). I’ll explain that in the following chapter.

Activity.Start()

If you have an Activity which is already started, you don’t need to use SetParentId() Activiy.Start() refer the Activity.Current then automatically process SetParentId() equivalent.

If you don’t have the existing Activity, you need to use SetParentId().

var activity = new Activity("Some operation name"); // the first
activity.SetParentId("|0S9vTv1SnAI=.fa0f9df2_");
activity.Start();

However, if you already have one, it is automatically set the SetParentId();

var firstActivity = new Activity("Some operation one");
firstActivity.Start(); // Set the instance to Activity.Current
var secondActivity = new Activity("Some operation two);
secondActivity.Start(); // Set the instance to Activity.Current
                        // with SetParent form firstActivity

What it does?

If you start the Activity,

• Set the StartTimeUtc with the UtcNow()
• Set ParentId and Parent if the Activity.Current is not null
• Generate Id according to the HTTP correlation protocol
• Set the new Activity instance to the Activity.Current

You can’t modify the instance after you start the Activity.

Activity.Stop()

For the Activity.Stop(), the behavior is following.

• Set EndTime with the UtcNow()
• Set Activity.Current with the Parent

At the first time, I was confused by Request/DependencyTelemetry and TelemetryClient. They also have Start()/Stop() method. Also, I can write the same code using them. Let’s investigate the details.

Request/Dependency Telemetry

Since the Activity is the model of the HTTP correlation protocol, the Request/Dependency telemetry is the model of the Application Insights. The property is the same as you can see on the Application Insights on Azure. For the first blog post, I didn’t use Request/DependnecyTelemetry directly. In this section, I’ll use them to understand the detail.

Manually add the Activity to the Request/Dependency Telemetry

I wrote a code using Activity and TelemetryClient.

var requestActivity = new Activity("Sample: Queue Request");
requestActivity.SetParentId(parentId);
requestActivity.Start(); // You can omit this code.
var requestOperation = telemetryClient.StartOperation<RequestTelemetry>(requestActivity);
// do some operation
telemetryClient.StopOperation(requestOperation);

You can rewrite this code using RequestTelemetry. It helps to understand what happens inside. telemetryClient.Track() is just sending telemetry to the Application Insights.

var requestActivity = new Activity("Sample: Queue Request");
requestActivity.SetParentId(parentId);
requestActivity.Start(); 
var requestTelemetry { Name = requestActivity.OperationName };
requestTelemetry.Id = requestActivity.Id;
requestTelemetry.Context.Operation.Id = activity.RootId;
requestTelemetry.Context.Operation.ParentId = activity.ParentId;
requestTelemetry.Start();
// do some operation
requestTelemetry.Stop();
telemetryClient.Track(requestTelemetry);

DepndencyTelemetry is the same. Request/Dependency Start(), Stop() is simple. It is tracking the Start/End time. I can see these send some Logs. However, we can’t see on the Application Insights.

Start()

• Set Timestamp with UtcNow

Stop()

• Set Timestamp with UtcNow
• Set Duration

TelemetryClient

TelemetryClient is the client of the Application Insights. It sends telemetry to the Application Insights. It also supports extension methods which help you to reduce the code. Let’s understand the behavior.

TelemetryClient Class (Microsoft.ApplicationInsights)

StartOperation<T>(Activity activity)

Generic type T = RequestTelemetry or DependnecyTelemetry. This method starts an Operation. The StartOperation() doing something like this.

• activity.Start(); // parameter
• Create T instance with passing activity.OperationName, RootId, ParentId, and Id
• Set Timestamp with UtcNow

If you use Activity, this option might be good.

StartOperation<T>(T operationTelemetry)

Generic type T = RequestTelemetry/DependencyTelemetry. This method starts an Operation. This one is a little complex. If you want to go with Activity, the first option might be better. The behavior is

• Store the T instance to the Operation
• Create a new Activity instance
• set ParentId from telemetryContext.Id if Activity.Current is null
• start Activity
• set operationTelemetry.Id with activity.Id;
• Set Timestamp with UtcNow
• Save the OperationContext to the AsyncLocal

StopOperation<T>(IOperationHolder<T> operation)

This method stops the operation. It delegates to the IOperationHolder.Dispose(). Which means you can use using

• Stop operationTelemetry
• Stop the Activity.Current
• Set Duration
• Set Timestamp as UtcNow
• Send telemetry

Relationships

You can see which property is which. This chart might help when you debug your app.

Correspondence between the instances

Sample code

You can refer the sample code in my repository.

Basic Correlation sample

The sample for the simplest one. It uses Activity with TelemetryClient.

TsuyoshiUshio/ActivitySpike

Request/Dependency telemetry sample

When you want to understand the behavior of the Request/Dependency telemetry. There is some improper code there. Please refer the Request/Dependency telemetry and Activity parts. However, it works.

TsuyoshiUshio/ActivitySpike

Activity Tips

Some Tips for Activity

• Activity can’t be serializable. If you want to serialize some of the property like ParentId, create a custom object to copy it.
• Once you Start the Activity, You can’t change the property.
• Activity.Current has the scope; if you need to pass it over the call graph, you need to manage by yourself.
• You can use Baggage or Tags on your Activity. However, avoid sensitive information add to the baggage. It passes to the other systems.
• Request/DependencyTelemetry and Activity has no relationship. TelemetryClient’s Extension methods know these two.

Conclusion

I explain the detailed behavior of the Activity and related classes. You might understand the behavior more deeply. On the last post, I’ll write about the W3C TraceContext correlation.

Next Step

Correlation with Activity with Application Insights (3) W3C TraceContext