More dump forensics, understanding !locks, in this case a filter driver problem

Written by Jeff Dailey: 

 

Hello NTDebuggers, one of the most important things to understand in kernel debugging hung servers is the output of !locks.  There can be a lot of data and it’s not always clear what is going on.  One of the things I like to do in order to better understand the output is to use a visual representation of the resources involved and the threads that are blocking on those resources.   Before we can do that we need to understand what to look for so we can document it in our diagram. 

 

It’s a good idea to understand ERESOURCEs in general l before jumping into !locks.  The following MSDN article goes into lots of great detail.  http://msdn2.microsoft.com/en-us/library/aa490224.aspx

 

Simply put, you will typically see threads either with access to or trying to gain access to resources.   If a thread has access to a resource it will be marked by <*>.   Threads that have access to a resource can block other threads from gaining access to said resource.

 

You will see threads waiting for shared access.  These threads do not have the <*> and listed above the threads that are Waiting on Exclusive Access.

 

You will also see threads that are Waiting on Exclusive Access.  These threads are typically blocked waiting for the threads that have access or ownership of the resource to release it.

 

Let’s take a look at one section of !locks output and annotate each thread section...

 

Resource @ 0x896d2a68    Shared 1 owning threads  << This info is the ERESOURCE in question.      Contention Count = 15292  << The amount of contention for the object.

    NumberOfSharedWaiters = 1  << This is self explanatory

    NumberOfExclusiveWaiters = 39 << Number of exclusive waiters in the Ex Waiter List

     Threads: 89bd1234-01<*> 896d2020-01   << We have two threads here.  The owner, or shared owner <*>89bd1234 and the shared Waiter 896d2020

     Threads Waiting On Exclusive Access:

              888ed020       87c036f8       885dc7a0       8bc538b0  << All of these threads are waiting on exclusive access.    

              88e8cda0       88796988       8905fda0       8974dc10      

 

 

Note the following output is completely fabricated, so alignment and variable names may not be valid.

 

The following is some sample output from !locks.  In this scenario I document any ERESOURCE that has any threads waiting on exclusive access.  I also document the ERESOURCES as nodes and show the relationship to the Threads.  The key point is to show the threads involved, the resources they own, and the resources they are blocked on or trying to get exclusive access to.  Ultimately you need to work your way toward the head of the blocking chain of events to figure out what is holding up the entire chain of execution from moving forward.

Read more: Ntdebugging

Posted via email from Jasper-net