Friday, January 3, 2014

Debugging memory corruption

It is so difficult to analyze a memory dump caused by memory corruption.
I'd like to introduce a case we are able to analyze.

The basic concept is "The suspect is right in front of you".

There is a memory dump that tells MyDrv.sys caused the crash.

0: kd> !analyze -v
*******************************************************************************
*
* Bugcheck Analysis *
*
*******************************************************************************

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made ​​to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 00000008, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: 8c032fa5, address which referenced memory

Debugging Details:
------------------
READ_ADDRESS: 00000008
CURRENT_IRQL: 2

FAULTING_IP:
MyDrv!GetListEntry+85
8c032fa5 3b5108 cmp edx, dword ptr [ecx+8]

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT
BUGCHECK_STR: 0xD1
PROCESS_NAME: SomeProcess.exe

TRAP_FRAME: bdf939b4 - (.trap 0xffffffffbdf939b4)
ErrCode = 00000000
eax = 00000000 ebx = 8c0344b0 ecx = 00000000 edx = c339fdc0 esi = 0101ed94 edi = bdf93b60
eip = 8c032fa5 esp = bdf93a28 ebp = bdf93a4c iopl = 0 nv up ei pl nz na pe cy
cs = 0008 ss = 0010 ds = 0023 es = 0023 fs = 0030 gs = 0000 efl = 00010207
MyDrv!GetListEntry+0x85:
8c032fa5 3b5108 cmp edx, dword ptr [ecx+8] ds: 0023:00000008=????????
Resetting default scope

LAST_CONTROL_TRANSFER: from 8c032fa5 to 83248b7f

STACK_TEXT:
bdf939b4 8c032fa5 badb0d00 c339fdc0 00000000 nt!KiTrap0E+0x1b3
bdf93a4c 8c03331d c339fdc0 a0fa8b08 bdf93a68 MyDrv!GetListEntry+0x85
bdf93a5c 8c0325fc c339fdc0 bdf93a7c 8c0364e5 MyDrv!GetEntryTable+0xd
...
bdf93c10 76e270f4 0101f08c 00020019 0101eda0 nt!KiSystemServicePostCall
WARNING: Frame IP not in any known module. Following frames may be wrong.
0101ef94 00000000 00000000 00000000 00000000 0x76e270f4

STACK_COMMAND: kb

FOLLOWUP_IP:
MyDrv!GetListEntry+85
8c032fa5 3b5108 cmp edx, dword ptr [ecx+8]

SYMBOL_STACK_INDEX: 1
SYMBOL_NAME: MyDrv!GetListEntry+85
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: MyDrv
IMAGE_NAME: MyDrv.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 5099ff75
FAILURE_BUCKET_ID: 0xD1_ MyDrv!GetListEntry+85
BUCKET_ID: 0xD1_ MyDrv!GetListEntry+85
Followup: MachineOwner
---------


Use .trap command to restore the thread context when the crash occurred.

0: kd> .trap 0xffffffffbdf939b4
ErrCode = 00000000
eax = 00000000 ebx = 8c0344b0 ecx = 00000000 edx = c339fdc0 esi = 0101ed94 edi = bdf93b60
eip = 8c032fa5 esp = bdf93a28 ebp = bdf93a4c iopl = 0 nv up ei pl nz na pe cy
cs = 0008 ss = 0010 ds = 0023 es = 0023 fs = 0030 gs = 0000 efl = 00010207
MyDrv!GetListEntry+0x85:
8c032fa5 3b5108 cmp edx, dword ptr [ecx+8] ds: 0023:00000008=????????


Looking at the local variables of GetListEntry function, the problem occurred because pListEntry is NULL.

0: kd> dv
pListHead = 0x8c038d00 [0x89e95988 - 0x864f3ea8]
pListEntry = 0x00000000


pListEntry is linked from pListHead, we need to take a look at the list from pListHead.

0: kd> dl 0x8c038d00
8c038d00 89e95988 864f3ea8 8c038d08 8c038d08
89e95988 00000000 00000000 00000000 00000000


Because the list is double-linked list, it should have a memory pointer but 89e95988 that is forward link of pListHead has 00000000.

The data in 89e95988 might be overwritten to 00 by some other module.

0: kd> db 89e95988
89e95988 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
89e95998 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
89e959a8 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
89e959b8 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
89e959c8 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
89e959d8 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
89e959e8 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
89e959f8 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................


I am going to chase the module that overwrote the linked list memory of MyDrv.sys.
First, we need to investigate pool information of 89e95988.

0: kd> !pool 89e95988
Pool page 89e95988 region is Nonpaged pool

89e95808 doesn't look like a valid small pool allocation, checking to see
if the entire page is actually part of a large page allocation ...

89e95808 is freed (or corrupt) pool
Bad previous allocation size @ 89e95808, last size was 0

...

Pool page [89e95000] is __ inVALID.

Analyzing linked list ...
[89e95000]: invalid previous size [0x43] should be [0x0]
[89e95000 -> 89e95c10 (size = 0xc10 bytes)]: Corrupt region

Scanning for single bit errors ...
None found


The information is not so useful because the pool header is also corrupted.

Second step is to check the information on the preceding page.
Usually, memory corruption is caused by heap overflow, it means that some module using the memory in front of my memory can overwrite my memory area. 

0: kd> !pool 89e94378
Pool page 89e94378 region is Nonpaged pool
*89e89000: large page allocation, Tag is Ddk, size is 0xc808 bytes
 Pooltag Ddk: Default for driver allocated memory (user's of ntddk.h)


We can see the pool information having Ddk pool tag and here is not corrupted. 
My hypothesis is that the module using this memory is likely to overwrite my memory.

When I look at the memory 89e89000, it has a specific data pattern and the pattern continues over the size of this memory (0xc808).
Therefore, I am sure that the owner of this memory corrupts my memory beyond the boundary of their memory.

Unfortunately, Ddk is default memory tag so we can't use it to find the suspect module.

Third, looking for a module that has the memory pointer 89e89000 in their module address space.

0: kd> !for_each_module s @#Base @#End 00 90 e8 89
Page 69a9 not present in the dump file. Type ". Hh dbgerr004" for details
Page 6414d not present in the dump file. Type ". Hh dbgerr004" for details
Page 63b60 not present in the dump file. Type ". Hh dbgerr004" for details
bebb5424 00 90 e8 89 21 00 00 00-00 00 00 00 01 00 00 00 ....! ...........


Fortunately, there is one result bebb5424.
89e89000 is stored at the location bebb5424.

Make sure which module has the address bebb5424.

0: kd> u bebb5424
NotMyDrv+0x6424:
bebb5424 0090e8892100 add byte ptr [eax +2189 E8h], dl
bebb542a 0000 add byte ptr [eax], al
bebb542c 0000 add byte ptr [eax], al
bebb542e 0000 add byte ptr [eax], al
...


It is inside of NotMyDrv.sys.
Make sure which section in the driver contains it.

0: kd> lmvm NotMyDrv
start end module name
bebaf000 bebbe780 NotMyDrv (no symbols)
Loaded symbol image file: NotMyDrv.sys
Image path: \??\C:\Windows\system32\Drivers\NotMyDrv.sys
Image name: NotMyDrv.sys

0: kd> !dh bebaf000

File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
14C machine (i386)
5 number of sections
4D880A01 time date stamp Tue Mar 22 11:31:29 2011

0 file pointer to symbol table
0 number of symbols
E0 size of optional header
10E characteristics
Executable
Line numbers stripped
Symbols stripped
32 bit word machine

OPTIONAL HEADER VALUES
10B magic #
6.00 linker version
6300 size of code
9200 size of initialized data
0 size of uninitialized data
50D3 address of entry point
280 base of code
----- New -----
00010000 image base
80 section alignment
80 file alignment
1 subsystem (Native)
5.00 operating system version
5.00 image version
1.10 subsystem version
F780 size of image
280 size of headers
...

SECTION HEADER # 1
.text name
5D18 virtual size
280 virtual address
5D80 size of raw data
280 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
68000020 flags
Code
Not Paged
(No align specified)
Execute Read


Debug Directories (1)
Type Size Address Pointer
cv 8d 0 f780 [Debug data not mapped]


SECTION HEADER # 2
.data name
8680 virtual size
6000 virtual address
8680 size of raw data
6000 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
C8000040 flags
Initialized Data
Not Paged
(No align specified)
Read Write

SECTION HEADER # 3
INIT name
556 virtual size
E680 virtual address
580 size of raw data
E680 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
E2000020 flags
Code
Discardable
(No align specified)
Execute Read Write

SECTION HEADER # 4
.rsrc name
4A8 virtual size
EC00 virtual address
500 size of raw data
EC00 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
42000040 flags
Initialized Data
Discardable
(No align specified)
Read Only

SECTION HEADER # 5
.reloc name
61E virtual size
F100 virtual address
680 size of raw data
F100 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
42000040 flags
Initialized Data
Discardable
(No align specified)
Read Only



bebb5424 is located in data SECTION started from bebaf000 + 6000.
We can assume that it is global variable location of NotMyDrv.sys.

Eventually, we can understand where this crash is from, NotMyDrv.sys overwrote the memory used by MyDrv.sys.