VulnScan – Automated Triage and Root Cause Analysis of Memory Corruption Issues

Our engineering team’s task is to evaluate the severity, impact, and underlying causes of these issues after receiving reports about potential vulnerabilities in our products from the Microsoft Security Response Center ( MSRC ). In reality, memory corruption problems account for a sizable portion of these reports. An MSRC security engineer typically needs to examine the crash and attempt to determine what went wrong in order to identify the root causes of these problems. This workflow is n’t just for externally reported vulnerabilities; internal vulnerabilities found through fuzzing and variant investigation are frequently memory corruption problems that must also be evaluated for exploitability.

For these reasons, MSRC has invested a lot of money in creating tooling that will enable us to automate the root cause analysis process over the course of many years. To assist security engineers and developers in identifying the type of vulnerability and underlying causes of memory corruption bugs, MSRC created and developed VulnScan. Debugging Tools for Windows ( WinDbg ) and time travel debugging ( TTD ) are two internally developed tools on which it is based.

An internally developed framework called Time Travel Debugging keeps track of and replays Windows application execution. During CPPCon 2017, this technology was made available.

VulnScan can automatically identify the underlying causes of the most prevalent memory corruption problems by utilizing WinDbg and TTD. Page Heap, a mechanism used by Application Verifier, initiates an access violation that is closer to the problem’s underlying cause. Starting at the scene of the crash, the analysis moves on to the underlying cause. VulnScan supports the following categories of memory corruption problems:

Read/write outside the bounds
The origin of the invalid pointer value is determined. VulnScan tries to identify which instruction changed and why if the origin indicates a valid allocation but the pointer later becomes invalid.
As a result, the tool is also capable of detecting basic out-of-bounds accesses brought on by incorrect loop counter values as well as integer overflows.
Use following free
A backwards lookup uses the invalid pointer’s value at the access violation to determine when it becomes valid in the execution timeline. From this point forward, VulnScan tracks all memory-free operations to find the location of the pointer’s release.
Before choosing the method mentioned above, we conducted a few experiments. All memory allocations and frees were initially logged, but doing so took a lot of time and resources. The map of heap objects was prepared even for non-use-after-free bugs, which had a negative impact on the speed of triage for other bugs. Without PageHeap enabled, we were able to triage use-after-free bugs using this method. Although it is by default disabled, it can still be used.
confusion of the type
The tool employs a heuristic to determine whether the pointer size and alignment are consistent for this vulnerability type. A type confusion vulnerability ( different structure member type ) may be present if a tainted pointer value in the reverse execution flow is partially overwritten ( misaligned structure members memory write ) or modified ( with values of different sizes ).
The following section of this blog post contains an illustration of a type confusionbug.
use of uninitialized memory
By placing a memory breakpoint at the memory read’s address, each memory reading operation is checked for initialization. The code is then run backwards to where it was written. We notify the user of an uninitialized memory use and continue the analysis if the write operation is absent.
An uninitialized memory pointer triage example:

[*] Current instruction: cmp qword ptr [r8+00000558h],rax[*] Current position: 0x2B3DC0000001[*] Source memory value: 0x2390E31B940[*] Tainting back register: r8[*] Register value: 0x0[*] ----------------------------------------------------------------------[*] Current instruction: mov r8,qword ptr [rcx+00000410h] <- uninitialized memory[*] Current position: 0x2b3d80000144[*] Source effective address: 0x2417f9a2690[*] Source memory value:0x0[*] ----------------------------------------------------------------------[*] Uninitialized heap object vulnerability detected !!!

Pointer dereference that is null and constant
All values are tracked back to their initialization by the VulnScan multi-branches tainting engine. We notify the user of a null or constant pointer dereference if all branches return unchanged values.
A null pointer dereference bug triage example:

[*] Current instruction: test byte ptr [rcx+4Ch],1[*] Current position: 0x6328B80000001[*] Source memory value: 0x1[*] Tainting back register: rcx[*] Register value: 0x0[*] ----------------------------------------------------------------------[*] Current instruction: mov rcx,qword ptr [rcx+20h][*] Current position: 0x6328B4000014E[*] Source effective address: 0x1E3A54FDC20[*] Source memory value: 0x0[*] Memory is initialized @TTTPos: 1744423515849038.[*] Memory was initialized![*] Tainting back memory: 0x1E3A54FDC20[*] ----------------------------------------------------------------------[*] Current instruction: mov qword ptr [rbx+20h],rax[*] Current position: 0x6327FC00002AE[*] Source memory value: 0x0[*] Tainting back register: rax[*] Register value: 0x0[*] ----------------------------------------------------------------------[*] Current instruction: xor eax,eax[*] Current position: 0x6327FC00002AD[*] Tainted register got zeroed![*] ----------------------------------------------------------------------

VulnScan is a component of the Sonar automation framework used by MSRC. On all supported platforms and software versions, it automatically processes externally submitted proof-of-concept files. The root cause analysis is carried out and reproduced using Sonar. In order to achieve this, we use a variety of environments and attempt to repeatedly reproduce the problem using various configurations.

VulnScan will be a part of the Microsoft Security Risk Detection service ( Project Springfield ), which will use it to eliminate duplicate crashes and offer thorough fuzzing-based vulnerability analysis.

VulnScan was used to evaluate all memory corruption problems for Microsoft Office, Microsoft Internet Explorer, and Microsoft Edge over the course of ten months. For MSRC engineers, it had a success rate of about 85 %, saving them an estimated 500 hours of engineering time.

Case study: triaging confusion bug ( CVE-2017-0134 )

We can examine code in both directions of the timeline of code execution with the aid of time travel debugging ( TTD ). To track writes to the memory, we use taint techniques to track register changes and memory breakpoints. To identify the potential root cause of the problem and the bug class, every instruction during the tainting process is examined in the context of previously carried out instructions.

Because VulnScan taint analysis is multi-branched, all values obtained from a single instruction can be tracked in order. A queue of registers and memory addresses for particular execution timeline positions is present in VulnScan. For each branch, a separate taint analysis is carried out. Application data flow can be fully recreated using this method. To expedite the analysis process, we’ve made a few simplifications and optimizations over time. The analysis that follows is merely an example to highlight the tool’s fundamental ideas and features.

The illustration comes from a Chakra vulnerability that Jordan Rabet ( Microsoft Offensive Security Research Team ) provided to the MSRC.

The key positions in the trace are as follows:

The access violation’s location is indicated by position 0x2D0780000001.

Address ( 0xA0000000A ) does not indicate a valid memory location, according to the mov instruction. By tainting back from the register (rcx ) used in this pointer calculation, we begin our analysis.

Position 0x2CFA8000014D: The heuristic is activated for the first time.

This analysis’s most crucial point is probably this one. Up until this point, the invalid pointer value was tracked back as a 64-bit value; however, in the memory write operation, it is now referred to as 32-BIT value. The analysis also triggers this heuristic a few more times, but these are unimportant because they have no bearing on the invalid pointer value we observed where the access violation occurred.

The tainted value changes between the positions 0x1D420000037E and 0.x0D40400000A7.

This suggests that an attacker might be able to control the value because it was externally set by the NtReadFile system call. Further back tracing reveals that the memory is set to a Page Heap-specific constant value, indicating that we are dealing with heap allocation.

Call stack:00 ch!memcpy <- Position 0x1D420000037E01 ch!memcpy_s02 ch!_fread_nolock_s <- syscall03 ch!fread_s04 ch!fread05 ch!Helpers::LoadScriptFromFile. . .0n <- Position 0x1D40400000A7

Taint analysis of the main branch ends at position 0x2A8C80001709.

A read-only global variable in the ChakraCore binary is the dereferenced address ( 0x7FFC239B2358 ). From here, other branches ( known as junctions ) are analyzed. The next branch’s analysis will continue because the destination operand of the instruction is an arithmetic operation.

The source code for this operation is represented by the global variable g_rgdblTens [lwExp].

It was worthwhile to look into other branches ( positions 0x2A8C800016F9 ) to make sure we did n’t miss any crucial information even though they result in constant and null values.

db db db db db d8b db .d8888. .o88b. .d8b. d8b db88 88 88 88 88 888o 88 88' YP d8P Y8 d8' `8b 888o 88Y8 8P 88 88 88 88V8o 88 `8bo. 8P 88ooo88 88V8o 88`8b d8' 88 88 88 88 V8o88 `Y8b. 8b 88~~~88 88 V8o88 `8bd8' 88b d88 88booo. 88 V888 db 8D Y8b d8 88 88 88 V888 YP ~Y8888P' Y88888P VP V8P `8888Y' `Y88P' YP YP VP V8P[*] Loading Trace.[*] Exception found in trace file![*] Current instruction: mov rax,qword ptr [rcx+8][*] Current position: 0x2D0780000001[*] Source effective address: 0xA0000000A[*] Tainting back register: rcx[*] Register value: 0xA00000002[*] ----------------------------------------------------------------------[*] Current instruction: mov rcx,qword ptr [rsp+00000088h][*] Current position: 0x2D0740000FE7[*] Source effective address: 0x99C17FDCF8[*] Source memory value: 0xA00000002[*] Memory is initialized @TTTPos: 49509161766887.[*] Memory was initialized![*] Tainting back memory: 0x99C17FDCF8[*] ----------------------------------------------------------------------[*] Current instruction: mov qword ptr [r15],rax[*] Current position: 0x2D0740000FD0[*] Source memory value: 0xA00000002[*] Tainting back register: rax[*] Register value: 0xA00000002[*] ----------------------------------------------------------------------[*] Current instruction: mov rax,qword ptr [rcx+18h][*] Current position: 0x2D0740000FCC[*] Source effective address: 0x2967D8CC0C0[*] Source memory value: 0xA00000002[*] Memory is initialized @TTTPos: 49509161766860.[*] Memory was initialized![*] Tainting back memory: 0x2967D8CC0C0[*] ----------------------------------------------------------------------[*] Current instruction: mov dword ptr [r10+rax*4+18h],r11d[*] Current position: 0x2CFA8000014D[*] Source memory value: 0xA[*] Pointer size mismatch detected![*] Tainting back register: r11d[*] Register value: 0xA[*] ----------------------------------------------------------------------[*] Current instruction: mov r11d,r8d[*] Current position: 0x2CFA8000013E[*] Tainting back register: r8d[*] Register value: 0xA[*] ----------------------------------------------------------------------[*] Current instruction: mov r8d,r9d[*] Current position: 0x2CFA8000013A[*] Tainting back register: r9d[*] Register value: 0xA[*] ----------------------------------------------------------------------[*] Current instruction: mov r9d,dword ptr [rdi+rax*4+18h][*] Current position: 0x2CFA8000012C[*] Source effective address: 0x2967D8B4898[*] Source memory value: 0xA[*] Memory is initialized @TTTPos: 49454400930092.[*] Memory was initialized![*] Tainting back memory: 0x2967D8B4898[*] ----------------------------------------------------------------------[*] Current instruction: mov qword ptr [rax],rcx[*] Current position: 0x2CC280001D5A[*] Source memory value: 0x140000000A[*] Pointer size mismatch detected![*] Tainting back register: rcx[*] Register value: 0x140000000A[*] ----------------------------------------------------------------------[*] Current instruction: mov rcx,qword ptr [rdx][*] Current position: 0x2CC280001D59[*] Source effective address: 0x2967E1B4070[*] Source memory value: 0x140000000A[*] Memory is initialized @TTTPos: 49213882768729.[*] Memory was initialized![*] Tainting back memory: 0x2967E1B4070[*] ----------------------------------------------------------------------[*] Current instruction: movups xmmword ptr [rcx-10h],xmm0[*] Current position: 0x2C68400005CB[*] Source memory value: 0x140000000A[*] Tainting back register: xmm0[*] Register value: 0x140000000A[*] ----------------------------------------------------------------------[*] Current instruction: movups xmm0,xmmword ptr [rdx+rcx][*] Current position: 0x2C68400005C7[*] Source effective address: 0x2967D713D30[*] Source memory value: 0x140000000A[*] Memory is initialized @TTTPos: 48826261964231.[*] Memory was initialized![*] Tainting back memory: 0x2967D713D30[*] ----------------------------------------------------------------------[*] Current instruction: mov dword ptr [rax+8],ecx[*] Current position: 0x2C4A80000537[*] Source memory value: 0x14[*] Pointer size mismatch detected![*] Tainting back register: ecx[*] Register value: 0x14[*] ----------------------------------------------------------------------[*] Current instruction: mov ecx,dword ptr [rdx+8][*] Current position: 0x2C4A80000535[*] Source effective address: 0x2967E1BC078[*] Source memory value: 0x14[*] Memory is initialized @TTTPos: 48698486687029.[*] Memory was initialized![*] Tainting back memory: 0x2967E1BC078[*] ----------------------------------------------------------------------[*] Current instruction: mov dword ptr [rbx+rcx*4+4],eax[*] Current position: 0x2C4A800004E5[*] Source memory value: 0x14[*] Tainting back register: eax[*] Register value: 0x14[*] ----------------------------------------------------------------------[*] Current instruction: mov eax,dword ptr [rbp+18h][*] Current position: 0x2C4A800004E0[*] Source effective address: 0x2967DDB9AA8[*] Source memory value: 0x14[*] Memory is initialized @TTTPos: 48698486686944.[*] Memory was initialized![*] Tainting back memory: 0x2967DDB9AA8[*] ----------------------------------------------------------------------[*] Current instruction: mov dword ptr [rax+18h],ebp[*] Current position: 0x2A8C80001858[*] Source memory value: 0x14[*] Tainting back register: ebp[*] Register value: 0x14[*] ----------------------------------------------------------------------[*] Current instruction: mov ebp,edx[*] Current position: 0x2A8C80001827[*] Tainting back register: edx[*] Register value: 0x14[*] ----------------------------------------------------------------------[*] Current instruction: mov edx,dword ptr [rdi+000000E0h][*] Current position: 0x2A8C8000181F[*] Source effective address: 0x99C17FEE00[*] Source memory value: 0x14[*] Memory is initialized @TTTPos: 46782931277855.[*] Memory was initialized![*] Tainting back memory: 0x99C17FEE00[*] ----------------------------------------------------------------------[*] Current instruction: mov dword ptr [rax],edx[*] Current position: 0x2A8C80001738[*] Source memory value: 0x14[*] Tainting back register: edx[*] Register value: 0x14[*] ----------------------------------------------------------------------[*] Current instruction: cvttsd2si edx,xmm1[*] Current position: 0x2A8C80001728[*] Tainting back register: xmm1[*] Register value: 0x4034000000000000[*] ----------------------------------------------------------------------[*] Current instruction: movsd xmm1,mmword ptr [rbp+58h][*] Current position: 0x2A8C80001725[*] Source effective address: 0x99C17FE550[*] Source memory value: 0x4034000000000000[*] Memory is initialized @TTTPos: 46782931277605.[*] Memory was initialized![*] Tainting back memory: 0x99C17FE550[*] ----------------------------------------------------------------------[*] Current instruction: movsd mmword ptr [rdi],xmm0[*] Current position: 0x2A8C80001719[*] Source memory value: 0x4034000000000000[*] Tainting back register: xmm0[*] Register value: 0x4034000000000000[*] ----------------------------------------------------------------------[*] Current instruction: movaps xmm0,xmm1[*] Current position: 0x2A8C8000170D[*] Tainting back register: xmm1[*] Register value: 0x4034000000000000[*] ----------------------------------------------------------------------[*] Current instruction: mulsd xmm1,mmword ptr [rdx+rax*8][*] Current position: 0x2A8C80001709[*] Source effective address: 0x7FFC239B2358[*] Source memory value: 0x4024000000000000[*] Adding junction to taint register: xmm1[*] ----------------------------------------------------------------------[*] Processing junction..[*] Current instruction: mulsd xmm1,mmword ptr [rdx+rax*8][*] Tainting register: xmm1[*] Register value: 0x4000000000000000[*] Current instruction: cvtsi2sd xmm1,rax[*] Current position: 0x2A8C80001701[*] Tainting back register: rax[*] Register value: 0x2[*] ----------------------------------------------------------------------[*] Current instruction: mov eax,esi[*] Current position: 0x2A8C800016FF[*] Tainting back register: esi[*] Register value: 0x2[*] ----------------------------------------------------------------------[*] Current instruction: lea esi,[rax+rsi*2][*] Current position: 0x2A8C800016FA[*] Adding junction to taint register: rsi[*] Tainting back register: rax[*] Register value: 0x32[*] ----------------------------------------------------------------------[*] Current instruction: movzx eax,al[*] Current position: 0x2A8C800016F8[*] Tainting back register: al[*] Register value: 0x32[*] ----------------------------------------------------------------------[*] Current instruction: movzx eax,byte ptr [rbx][*] Current position: 0x2A8C800016F4[*] Source effective address: 0x28E6922DCA8[*] Tainting back memory: 0x28E6922DCA8[*] ----------------------------------------------------------------------[*] Current instruction: rep movs byte ptr [rdi],byte ptr [rsi][*] Current position: 0x1D420000037E[*] Source effective address: 0x28E692302E8[*] Source memory value: 0x0[*] Tainting back memory: 0x28E692302E8[*] ----------------------------------------------------------------------[*] Current instruction: mov qword ptr [rcx+28h],rdx[*] Current position: 0x1D40400000A7[*] Source memory value: 0xC0C0C0C0C0C0C0C0[*] Pointer size mismatch detected![*] Tainting back register: rdx[*] Register value: 0xC0C0C0C0C0C0C0C0[*] ----------------------------------------------------------------------[*] Current instruction: imul rdx,r9[*] Current position: 0x1D404000001C[*] Adding junction to taint register: rdx[*] Tainting back register: r9[*] Register value: 0x101010101010101[*] ----------------------------------------------------------------------[*] Current instruction: mov r9,101010101010101h[*] Current position: 0x1D404000001B[*] Tainted register is set to constant value![*] ----------------------------------------------------------------------[*] Processing junction..[*] Current instruction: lea esi,[rax+rsi*2][*] Tainting register: rsi[*] Register value: 0xFFFFFFE8[*] Current instruction: lea esi,[rsi-18h][*] Current position: 0x2A8C800016F9[*] Tainting back register: rsi[*] Register value: 0x0[*] ----------------------------------------------------------------------[*] Current instruction: lea esi,[rsi+rsi*4][*] Current position: 0x2A8C800016F7[*] Tainting back register: rsi[*] Register value: 0x0[*] ----------------------------------------------------------------------[*] Current instruction: xor esi,esi[*] Current position: 0x2A8C80001696[*] Tainted register got zeroed![*] ----------------------------------------------------------------------[*] Processing junction..[*] Current instruction: imul rdx,r9[*] Tainting register: rdx[*] Register value: 0xC0[*] Current instruction: movzx edx,dl[*] Current position: 0x1D404000001A[*] Tainting back register: dl[*] Register value: 0xC0[*] ----------------------------------------------------------------------[*] Current instruction: mov edx,0C0h[*] Current position: 0x1D4040000011[*] Tainted register is set to constant value![*] ----------------------------------------------------------------------[*] Type Confusion vulnerability detected !!![*] Loading Symbols.[*] Building output.[*] Writing findings to file: n:CVE-2017-0134ch.run.html[*] Analysis finished in 00:01:08

We can analyze the bug’s assembly language from this initial output. The register values, source code, local variables, and call stack are then produced in a more thorough report by VulnScan. This makes it easier for product developers to fix problems accurately and completely.

We can begin looking into the source code between the triggered heuristic and crashing location using the information from both reports. The value that serves as a pointer is set as an array element in Js: JavascriptArray:: DirectSetItemInLastUsedSegmentAt, which is derived from the EntryConcat function. By using a custom getter on an IntArray object property, Jordan was able to switch the array type to VarArray. ConcatIntArgs ( as depicted on the call stack in the report ) and JavascriptArray (uses the Symbol ) are the two functions. The array object’s isConcatSpreadable property is a custom getter that can be used to modify the array type. As a result, after the type change, concatenated IntArray items were written to VarArrays rather than ToStrings, which resulted in array object corruption. This ChakraCore commit introduced a solution to this problem.

That concludes today. Please leave feedback if you are interested in learning more about this tool, its heuristics, and the taint techniques used. The community’s response will have a significant impact on our next steps. Users of the Microsoft tool frequently request updates with new heuristics and features.

For assistance in writing this blog post, please contact the MSRC UK team, Jordan Rabet for the bug he discovered, Steven Hunter, Matt Miller, and Gavin Thomas. For the incredible tools they created, Kudos should also be given to the WinDbg, TTD, Sonar, and Microsoft Security Risk Detection teams.

Microsoft Security Response Center ( UK) employee Mateusz Krzywicki.