Styjun

There is a kernel level driver installed on a terminal server.It works fine for certain period of time on that terminal sever. later on
that terminal server itself getting into freezed state where noboday can RDP & web console to connect with server. In my case,
CPU is always hitting to 100% in freezed state and i had to hard reboot only by using VM option "power off". After unstalling that driver the terminal server works fine or even responds properly always.Even if it is 100% CPU usage and gets slow but still reponds to the RDP & web console.

That scenario is kind of hard to reproduce it. but still i got successful to fetch complete memory dump out of that machine in that scenario then i analyzed full memory dump using microsoft WinDbg tool. WinDbg tool displayed faulty driver module name and call stack as below

Module Name: MMTEProxy (Installed Driver)

 0: kd> !analyze -v
 *******************************************************************************
 * *
 * Bugcheck Analysis *
 * *
 *******************************************************************************

 NMI_HARDWARE_FAILURE (80)
 This is typically due to a hardware malfunction. The hardware supplier should
 be called.
 Arguments:
 Arg1: 00000000004f4454
 Arg2: 0000000000000000
 Arg3: 0000000000000000
 Arg4: 0000000000000000

 Debugging Details:
 ------------------
 KEY_VALUES_STRING: 1

 PROCESSES_ANALYSIS: 1

 SERVICE_ANALYSIS: 1

 STACKHASH_ANALYSIS: 1

 TIMELINE_ANALYSIS: 1

 DUMP_CLASS: 1

 DUMP_QUALIFIER: 402

 BUILD_VERSION_STRING: 9600.17415.amd64fre.winblue_r4.141028-1500

 SYSTEM_MANUFACTURER: VMware, Inc.

 VIRTUAL_MACHINE: VMware

 SYSTEM_PRODUCT_NAME: VMware Virtual Platform

 SYSTEM_VERSION: None

 BIOS_VENDOR: Phoenix Technologies LTD

 BIOS_VERSION: 6.00

 BIOS_DATE: 04/05/2016

 BASEBOARD_MANUFACTURER: Intel Corporation

 BASEBOARD_PRODUCT: 440BX Desktop Reference Platform

 BASEBOARD_VERSION: None

 DUMP_TYPE: 0

 BUGCHECK_P1: 4f4454

 BUGCHECK_P2: 0

 BUGCHECK_P3: 0

 BUGCHECK_P4: 0

 CPU_COUNT: 2

 CPU_MHZ: bb8

 CPU_VENDOR: GenuineIntel

 CPU_FAMILY: 6

 CPU_MODEL: 3e

 CPU_STEPPING: 4

 CPU_MICROCODE: 6,3e,4,0 (F,M,S,R) SIG: 42C'00000000 (cache) 42C'00000000 (init)

 DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

 BUGCHECK_STR: 0x80

 PROCESS_NAME: svchost.exe

 CURRENT_IRQL: 0

 ANALYSIS_SESSION_HOST: INPN01LAP107

 ANALYSIS_SESSION_TIME: 03-26-2019 16:30:13.0120

 ANALYSIS_VERSION: 10.0.18317.1001 amd64fre

 LAST_CONTROL_TRANSFER: from fffff8005ae205b2 to fffff8009a6601a7

 STACK_TEXT: 
 nt!KxWaitForLockOwnerShip+0x27
 MMTEProxy!SVSessionLutTranslatePort+0x2c2 [c:usersdkelonegitMMTEMMTEMMTEdriversessionlut.c @ 873] 
 MMTEProxy!PerformProxySocketRedirection+0xba7 [c:usersdkelonegitMMTEMMTEMMTEdriverfilteralebindredirect.c @ 247] 
 MMTEProxy!TriggerProxyByALERedirectInline+0x244 [c:usersdkelonegitMMTEMMTEMMTEdriverfilteralebindredirect.c @ 690] 
 MMTEProxy!DDProxyBindRedirectClassify+0x537 [c:usersdkelonegitMMTEMMTEMMTEdriverfilteralebindredirect.c @ 881] 

 THREAD_SHA1_HASH_MOD_FUNC: 03f7fb5fd041c46c9b4dff8f1685ccff753d3642

 THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 7f4a5e830d38804e610244f134268d53640c97a0

 THREAD_SHA1_HASH_MOD: 2a8f232a3e3c38ad2a6b44b0d2253b97c2ac4b2a

 FOLLOWUP_IP: 
 MMTEProxy!SVSessionLutTranslatePort+2c2 [c:usersdkelonegitMMTEMMTEMMTEdriversessionlut.c @ 873]
 fffff800`5ae205b2 c644244000 mov byte ptr [rsp+40h],0

 FAULT_INSTR_CODE: 402444c6

 FAULTING_SOURCE_LINE: c:usersdkelonegitMMTEMMTEMMTEdriversessionlut.c

 FAULTING_SOURCE_FILE: c:usersdkelonegitMMTEMMTEMMTEdriversessionlut.c

 FAULTING_SOURCE_LINE_NUMBER: 873

 FAULTING_SOURCE_CODE: 
 No source found for 'c:usersdkelonegitMMTEMMTEMMTEdriversessionlut.c'

 SYMBOL_STACK_INDEX: 1

 SYMBOL_NAME: MMTEProxy!SVSessionLutTranslatePort+2c2

 FOLLOWUP_NAME: MachineOwner

 MODULE_NAME: MMTEProxy

 IMAGE_NAME: MMTEProxy.sys

 DEBUG_FLR_IMAGE_TIMESTAMP: 5a60d5f0

 STACK_COMMAND: .thread ; .cxr ; kb

 BUCKET_ID_FUNC_OFFSET: 2c2

 FAILURE_BUCKET_ID: 0x80_MMTEProxy!SVSessionLutTranslatePort

 BUCKET_ID: 0x80_MMTEProxy!SVSessionLutTranslatePort

 PRIMARY_PROBLEM_CLASS: 0x80_MMTEProxy!SVSessionLutTranslatePort

 TARGET_TIME: 2019-02-26T11:15:36.000Z

 OSBUILD: 9600

 OSSERVICEPACK: 0

 SERVICEPACK_NUMBER: 0

 OS_REVISION: 0

 SUITE_MASK: 16

 PRODUCT_TYPE: 3

 OSPLATFORM_TYPE: x64

 OSNAME: Windows 8.1

 OSEDITION: Windows 8.1 Server TerminalServer

 OS_LOCALE: 

 USER_LCID: 0

 OSBUILD_TIMESTAMP: 2014-10-29 06:08:48

 BUILDDATESTAMP_STR: 141028-1500

 BUILDLAB_STR: winblue_r4

 BUILDOSVER_STR: 6.3.9600.17415.amd64fre.winblue_r4.141028-1500

 ANALYSIS_SESSION_ELAPSED_TIME: 685

 ANALYSIS_SOURCE: KM

 FAILURE_ID_HASH_STRING: km:0x80_MMTEProxy!svsessionluttranslateport

 FAILURE_ID_HASH: c64b7e97-0bf3-daf1-ad95-9f39cbf37a9a

 Followup: MachineOwner
 ---------

Since i am not expert in kernel level driver development,But i tried to google about driver. Internally it uses the following lock to perform any operation at process table or session table

 #Code snippet

 PLIST_ENTRY processTableListHead = NULL;

 
 ....

 KLOCK_QUEUE_HANDLE processTableLockHandle;
 KLOCK_QUEUE_HANDLE sessionTableLockHandle;

 PLIST_ENTRY tempNode = 0;
 ....
 ...

 KeAcquireInStackQueuedSpinLock(&gProcessTableLock,&processTableLockHandle);

 tempNode = processTableListHead;

 ...
 ...
 ..
 //Releases lock
 KeReleaseInStackQueuedSpinLock(&sessionTableLockHandle);
 KeReleaseInStackQueuedSpinLock(&processTableLockHandle);

 

With help of WinDbg tool, What i observed here, Mostly it is failling at source line no where assinging the value to a variables and that variables defined before accuiring the lock. You can see it in above driver code snippet. my driver is a WFP ALE filtered driver. it inspects traffic it works in a multhreaded environment and my driver allocates/freed memory in non-paged pool

And I also checked that there is no any deadlock condition or curretnly held lock by any thread. still i am not getting what causing this issue. whether its lock is not handled properly at code level or some particular situation.

Can you please help me with pointer or direction?