Reading AsyncLocal values from a memory dump
This article explains how AsyncLocal values are stored in .NET and how to retrieve them from a memory dump. Note that the code provided is written for .NET 5, but should be able to work for .NET Framework with minor modifications (the name of some internal fields have changed across versions).
Preparing the memory dump
First thing first, we need to prepare the memory dump that will serve as example for the whole article. For that, I used the following code:
The program creates an array of AsyncLocal<int> of 17 elements. 17 is not a random number, its significance will become clear in a moment. Then the program creates a few threads, and have them store a different number of asynclocal values, such as thread 1 will store 1 asynclocal value, thread 2 will store 2, and so on until thread 17. When all the values are initialized, the program displays a prompt and pauses to give time to capture a memory dump. This can be done using the procdump tool:
procdump -ma <pid>Inspecting the memory dump with WinDbg
Now that we have our memory dump, let’s start by inspecting it with WinDbg. We’ll start by retrieving the array of AsyncLocal<int>. Recent versions of WinDbg automatically load the SOS extension when dealing with .NET Core memory dumps, so that’s one less thing to do. We can use the dumpheap command to locate our instance of Program, and from there access the array.
For that, we first use dumpheap -stat -type ClientApp.Programto get the address of the MT, and then we feed it to dumpheap -mt to get the address of the instance:
Then we use dumpobj to dump the instance and from there we get the address of the value stored in the _asyncLocals field:
Now we can dump the contents of the array using dumparray and inspect the individual items:
Immediately we can see what the issue is going to be: in our instance of AsyncLocal there is… nothing. Literally nothing at all is stored inside, unless you provided a valueChanged handler (in which case, you will only find the address of your handler).
So where are those values stored? On the Thread object itself. The instance of AsyncLocal is just used as a key to retrieve the value from the execution context. If we take one thread at random, we can find all the asynclocal values from that thread stored in the m_LocalValues field of the execution context:
The _key1 field contains the address of the AsyncLocal instance, and the _value1 field contains the value. So to check all the values stores in an AsyncLocal instance, we need to inspect every single thread. Not only that, but the way the values are stored is going to change depending on how many different AsyncLocal values are associated to a given thread. Managing that in WinDbg is going to be very tricky, so it’s time to switch to ClrMD.
Inspecting the memory dump with ClrMD
Our ClrMD program will open the memory dump, find the AsyncLocal array, and extract all the values for every thread. To make things easier, we’re going to use the DynaMD library.
First thing first, we open the memory dump, and we locate the first (and only) instance of AsyncLocal<int>[] thanks to the GetProxies extension method. Then we iterate on every instance and give them to a helper method that will extract a list of thread ids and their associated values:
The helper method will iterate on every thread, inspect the AsyncLocal values stored in the execution context, and filter them to only return the one associated to the given instance:
Now we just need to extract the values from the AsyncLocal storage… Which is where things get fancy. It turns out there’s not one but many different storage implementations, depending on how many values the current thread is storing.
From 1 to 3 values
If a given thread stores one to three different AsyncLocal values, it will use an instance of OneElementAsyncLocalValueMap, TwoElementAsyncLocalValueMap, or… ThreeElementAsyncLocalValueMap. Those types just have a list of fields to store the keys and the values, _key1 / _key2 / _key3 and _value1 / _value2 / _value3 respectively. Extracting the values is just a matter of reading those fields:
From 4 to 16 values
If a given thread stores four to sixteen values, it will thankfully stop declaring a field for every value and instead use an array-backed storage: MultiElementAsyncLocalValueMap. All the keys and values are stored in the _keyValues field as an array of KeyValuePair<IAsyncLocal, object>. The array is allocated with exactly the right size, so we just need to iterate on it to return the contents. It’s very straightforward with DynaMD:
More than 16 values
To read from an AsyncLocal storage, the runtime has to find the value associated to a given key. Iterating on an array becomes less and less efficient as the number of values increases, so the runtime fallbacks on a ManyElementAsyncLocalValueMap when that number gets above 16. This type directly inherits from Dictionary<IAsyncLocal, object>, so to extract the values we need to iterate on the _entries field:
Putting everything together
At this point, you may be wondering how everything fits together. Every time you add or remove a value from an AsyncLocal instance, the runtime will allocate a new storage based on the new number of elements, and copy everything from the old to the new storage. This makes adding/removing values very inefficient, so use AsyncLocal with care.
In any case, we now have helper methods for every possible AsyncLocal storage, so we can just call the appropriate one in our ReadAsyncLocalStorage method:
And that’s it! Now we can run the program and inspect the values for every AsyncLocal instance.
The full code is available here.
