Server Reboot-Kernel Power / Bug Check
TDR Host Sensor, Server 2016
This morning I noticed my backup replications failed because the replication destination could not be found. The moment I logged onto the destination server it hard rebooted on me.
In the Event logs there is a Critical Kernel-Power System error, and in the Error logs there is a Bug-Check Event, and interestingly there are 1372 Service Control Errors in the past 10 hours with this error message:
"The Threat Detection and Response Network Driver service failed to start due to the following error:
The system cannot find the file specified."
In the Host Sensor log file there is this:
2021-01-19 16:02:50.197 [Error] [thread:2968] [BehaviorPreventorProcessEvents] Caught exception in handleProcessNotification: Not connected to core driver
2021-01-19 16:02:50.369 [Information] [thread:10308] [ProcessEventListener] Process created: pid=52400 name=powershell.exe image=c:\Windows\System32\windowspowershell\v1.0\powershell.exe fallback=N
2021-01-19 16:02:50.369 [Error] [thread:2540] [BehaviorPreventorProcessEvents] Caught exception in handleProcessNotification: Not connected to core driver
2021-01-19 16:02:50.525 [Information] [thread:7060] [ProcessEventListener] Process created: pid=52332 name=conhost.exe image=c:\Windows\System32\conhost.exe fallback=N
2021-01-19 16:02:50.697 [Information] [thread:9556] [RegistryScanner] Registry scan found 937 keys
2021-01-19 16:02:51.415 [Information] [thread:6072] [ProcessEventListener] Process deleted: 52332
2021-01-19 16:02:51.415 [Information] [thread:2968] [ProcessEventListener] Process deleted: 52356
2021-01-19 16:02:51.415 [Information] [thread:10680] [ProcessEventListener] Process deleted: 52400
2021-01-19 16:02:51.635 [Error] [thread:7060] [PEUtilities] CryptQueryObject failed with 2148081673
2021-01-19 16:02:51.635 [Error] [thread:9652] [PEUtilities] CryptQueryObject failed with 2148081673
2021-01-19 16:02:51.635 [Error] [thread:5268] [PEUtilities] CryptQueryObject failed with 2148081673
2021-01-19 16:02:51.744 [Error] [thread:7040] [ProcessHeuristicsChecker] getModuleInfoList() : Failed to get module snapshot, error 24
2021-01-19 16:02:51.744 [Error] [thread:7040] [ProcessHeuristicsChecker] Process module information is empty
2021-01-19 16:02:51.790 [Information] [thread:7060] [ProcessEventListener] Process created: pid=52580 name=powershell.exe image=c:\Windows\System32\windowspowershell\v1.0\powershell.exe fallback=N
2021-01-19 16:02:51.790 [Error] [thread:6092] [BehaviorPreventorProcessEvents] Caught exception in handleProcessNotification: Not connected to core driver
2021-01-19 16:02:51.806 [Error] [thread:9592] [Process] getUserInfo: OpenProcess failed, pid: 41436code: 87
2021-01-19 16:02:51.806 [Error] [thread:9592] [Process] getUserInfo: OpenProcess failed, pid: 52252code: 87
2021-01-19 16:02:51.900 [Information] [thread:9652] [ProcessEventListener] Process created: pid=52020 name=cmd.exe image=c:\Windows\System32\cmd.exe fallback=N
2021-01-19 16:02:51.900 [Error] [thread:10680] [BehaviorPreventorProcessEvents] Caught exception in handleProcessNotification: Not connected to core driver
2021-01-19 16:02:52.025 [Information] [thread:5268] [ProcessEventListener] Process created: pid=45660 name=conhost.exe image=c:\Windows\System32\conhost.exe fallback=N
2021-01-19 16:02:52.025 [Information] [thread:3948] [ProcessEventListener] Process deleted: 41436
2021-01-19 16:02:52.025 [Information] [thread:2968] [ProcessEventListener] Process deleted: 52252
2021-01-19 16:02:52.837 [Error] [thread:9592] [Process] getUserInfo: OpenProcess failed, pid: 44952code: 87
2021-01-19 16:02:52.837 [Information] [thread:9640] [ProcessEventListener] Process deleted: 44952
2021-01-19 16:02:52.837 [Error] [thread:9592] [Process] getUserInfo: OpenProcess failed, pid: 52356code: 87
2021-01-19 16:02:52.837 [Error] [thread:9592] [Process] getUserInfo: OpenProcess failed, pid: 52332code: 87
2021-01-19 16:02:52.837 [Error] [thread:9592] [Process] getUserInfo: OpenProcess failed, pid: 52400code: 87
2021-01-19 16:02:52.837 [Warning] [thread:6276] [ProcessEventListener] Failed to get name for parent process 52356: Failed to open process 52356 | GetLastError reported: The parameter is incorrect. (Error code 87)
2021-01-19 16:02:52.837 [Error] [thread:9592] [Process] getUserInfo: OpenProcess failed, pid: 52400code: 87
2021-01-19 16:02:52.837 [Error] [thread:10308] [ProcessEventListener] Failed to get actual path for 52400:powershell - Failed to open process 52400 | GetLastError reported: The parameter is incorrect. (Error code 87)
2021-01-19 16:02:52.837 [Information] [thread:10056] [ProcessEventListener] Process deleted: 52400
2021-01-19 16:02:52.837 [Warning] [thread:10308] [ProcessEventListener] Failed to get name for parent process 52356: Failed to open process 52356 | GetLastError reported: The parameter is incorrect. (Error code 87)
2021-01-19 16:02:52.837 [Error] [thread:9592] [Process] getUserInfo: OpenProcess failed, pid: 52356code: 87
2021-01-19 16:02:52.837 [Error] [thread:10308] [PEUtilities] mapFile() - Failed to map view of file "powershell" with Error: 2
2021-01-19 16:02:52.837 [Error] [thread:9592] [Process] getUserInfo: OpenProcess failed, pid: 52332code: 87
2021-01-19 16:02:52.837 [Warning] [thread:10308] [FileScanEngine] File not found
2021-01-19 16:02:52.837 [Error] [thread:1492] [ProcessHeuristicsChecker] Can't open process for pid 52400, error 87
2021-01-19 16:02:52.837 [Error] [thread:2096] [PEUtilities] CryptQueryObject failed with 2148081673
2021-01-19 16:02:52.837 [Error] [thread:1492] [ProcessHeuristicsChecker] Process module information is empty
2021-01-19 16:02:52.837 [Error] [thread:6276] [PEUtilities] CryptQueryObject failed with 2148081673
2021-01-19 16:02:52.837 [Information] [thread:10308] [ProcessEventListener] Process deleted: 52332
2021-01-19 16:02:52.837 [Error] [thread:7040] [ProcessHeuristicsChecker] Can't open process for pid 52356, error 87
2021-01-19 16:02:52.837 [Error] [thread:7040] [ProcessHeuristicsChecker] Process module information is empty
2021-01-19 16:02:52.853 [Error] [thread:10308] [PEUtilities] CryptQueryObject failed with 2148081673
2021-01-19 16:02:52.853 [Error] [thread:9336] [ProcessHeuristicsChecker] Can't open process for pid 52332, error 87
2021-01-19 16:02:52.853 [Error] [thread:9336] [ProcessHeuristicsChecker] Process module information is empty
2021-01-19 16:02:52.869 [Error] [thread:2096] [PEUtilities] CryptQueryObject failed with 2148081673
The backup replication failed right around the same time TDR started experiencing its issues.
Has anyone seen this before or experienced it?
Could this be caused by the TDR network driver corrupting and causing other services to fail?
After the hard reboot the server seems to be functioning normally with no further error messages.
Any thoughts appreciated.
- Doug
It's usually something simple.
Comments
I just installed the host sensor on a Server 2008 R2 SP1 and the alert is "Host Sensor has a problem". The log has numerous entries:
[Error] [thread:1728] [PEUtilities] CryptQueryObject failed with 2148081673
So you're not alone with this error.
Hi @shaazaminator I'd suggest opening a support ticket. From what I can see, it looks like TDR is looking at a process that is ending very quickly for whatever reason -- it's hard to tell what it might be based on the verbosity there. The support team can look into that more closely and help with a better answer.
-James Carson
WatchGuard Customer Support
This just happened again last week right before I lost power for the past five days.
TDR driver corrupts, causing WMI issues, making server unresponsive, resulting in Kernel panic.
Only this time the server didn't reboot, I had to use the integrated management system to reboot the server. Ended up corrupting a database running in a VM on that server. (that is for pity points)
Nice to know I'm not the only one @CraigS
I'll open a ticket once I get caught up.
It's usually something simple.
Opened up a ticket. Once there is a resolution I'll post it.
It's usually something simple.