US-12625961-B2 - Malicious direct syscall call detection

Abstract

Methods, storage systems and computer program products implement embodiments of the present invention for protecting a computer by first deploying in a memory of the computer a hooked version of a syscall used by an operating system kernel of the computer A notification of a call to the hooked version of the syscall from a user mode of the computer is received from the hooked version of the syscall, the notification including a return address in the memory and a set of features extracted from the call. The return address and the received features are analyzed so as to classify the call as benign or malicious, and an alert is generated for the computer upon classifying the new call as malicious.

Inventors

• Or Chechik
• Ofir Ozer
• Ori Damari
• Yuval Zan

Assignees

• PALO ALTO NETWORKS (ISRAEL ANALYTICS) LTD.

Dates

Publication Date

20260512

Application Date

20240731

Claims (20)

1. 1 . A method for protecting a computer, comprising: deploying in a memory of the computer a hooked version of a syscall used by an operating system kernel of the computer; receiving, from the hooked version of the syscall, a notification of a call to the hooked version of the syscall from a user mode of the computer, the notification comprising a return address in the memory and a set of features extracted from the call; analyzing the return address and the received features so as to classify the call as benign or malicious; and generating, for the computer, an alert upon classifying the new call as malicious, wherein classifying the call as malicious comprises classifying the call as a direct call, and wherein the call comprises a new call, and wherein the features further comprise additional features from previous direct calls to the hooked version of the syscall by execution entities executing in the memory of the computer and additional computers, wherein the new call and previous calls were conveyed during multiple days, and wherein one or more organizations comprise the computers.
2. 2 . The method according to claim 1 , wherein analyzing the return address comprises identifying the return address as belonging to a specified block in a user-mode segment of the memory.
3. 3 . The method according to claim 2 , wherein the specified block is not allocated to a specified DLL.
4. 4 . The method according to claim 1 , wherein a given feature comprises a mapped block in the memory comprising the execution entities, wherein analyzing the features comprises identifying a number of distinct execution entities that generated the direct calls from the mapped block, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
5. 5 . The method according to claim 1 , wherein a given feature comprises a mapped block in the memory comprising the execution entities, wherein analyzing the features comprises identifying a number of distinct days when any of the computers having the given feature, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
6. 6 . The method according to claim 1 , and further comprising defining a key comprising first and second features, the first feature comprising a given execution entity that conveyed one or more of the direct calls, and the second feature comprising a mapped block in the memory comprising an execution entity that generated one or more of the direct calls, wherein analyzing the features comprises identifying a number of distinct computers having the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
7. 7 . The method according to claim 1 , and further comprising defining a key comprising first and second features, the first feature comprising a given execution entity that conveyed one or more of the direct calls, and the second feature comprising a mapped block in the memory comprising an execution entity that generated one or more of the direct calls, wherein analyzing the features comprises identifying a number of distinct days when any of the computers have the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
8. 8 . The method according to claim 1 , and further comprising defining a key comprising first and second features, the first feature comprising a given execution entity that conveyed one or more of the direct calls, and the second feature comprising a mapped block in the memory comprising an execution entity that generated one or more of the direct calls, wherein analyzing the features comprises identifying a number of distinct organizations comprising the computers having the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
9. 9 . The method according to claim 1 , and further comprising defining a key comprising first and second features, the first feature comprising a given execution entity that conveyed one or more of the direct calls, and the second feature comprising the hooked version of the syscall, wherein analyzing the features comprises identifying a number of distinct memory blocks comprising the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
10. 10 . The method according to claim 1 , and further comprising defining a key comprising first and second features, the first feature comprising a process that conveyed one or more of the direct calls, and the second feature comprising the hooked version of the syscall, wherein analyzing the features comprises identifying a number of distinct computers comprising the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
11. 11 . The method according to claim 1 , and further comprising defining a key comprising first and second features, the first feature comprising a given execution entity that conveyed one or more of the direct calls, and the second feature comprising the hooked version of the syscall, wherein analyzing the features comprises identifying a number of distinct days when any of the computers have the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
12. 12 . The method according to claim 1 , wherein one or more of the computers execute shellcodes comprising respective shellcode headers.
13. 13 . The method according to claim 12 , wherein a given feature comprises a given execution entity that one or more of the computers executed from any of the shellcodes, and that conveyed one or more of the direct calls, wherein analyzing the features comprises identifying a number of distinct computers having the given feature, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
14. 14 . The method according to claim 12 , wherein a given feature comprises a given execution entity that one or more of the computers executed from any of the shellcodes, and that conveyed one or more of the direct calls, wherein analyzing the features comprises identifying a number of distinct days when any of the computers have the given feature, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
15. 15 . The method according to claim 12 , wherein a given feature comprises a given execution entity that one or more of the computers executed from any of the shellcodes, and that conveyed one or more of the direct calls, wherein analyzing the features comprises identifying a number of distinct shellcode headers having execution entities identical to the given feature, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
16. 16 . The method according to claim 12 , and further comprising defining a key comprising first and second features, the first feature comprising a given execution entity that one or more of the computers executed from any of the shellcodes, and the second feature a given shellcode header for a given shellcode that spawned a given execution entity, wherein analyzing the features comprises identifying a number of distinct computers having the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
17. 17 . The method according to claim 12 , and further comprising defining a key comprising first and second features, the first feature comprising a given execution entity that one or more of the computers executed from any of the shellcodes, and the second feature a given shellcode header for a given shellcode that spawned a given execution entity, wherein analyzing the features comprises identifying a number of distinct days when any of the computers have the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
18. 18 . The method according to claim 12 , and further comprising defining a key comprising first and second features, the first feature comprising a given execution entity that one or more of the computers executed from any of the shellcodes, and the second feature a given shellcode header for a given shellcode that spawned a given execution entity, wherein analyzing the features comprises identifying a number of distinct organizations comprising the computers having the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
19. 19 . The method according to claim 12 , and further comprising defining a key comprising first and second features, the first feature comprising the hooked version of the syscall, and the second feature a given shellcode header for a given shellcode that spawned a given execution entity, wherein analyzing the features comprises identifying a number of distinct computers having the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.
20. 20 . The method according to claim 12 , and further comprising defining a key comprising first and second features, the first feature comprising the hooked version of the syscall, and the second feature a given shellcode header for a given shellcode that spawned a given execution entity, wherein analyzing the features comprises identifying a number of distinct days when any of the computers in a given organization have the key, and wherein classifying the new call as malicious comprises identifying that identified number is less than a specified threshold.

Description

FIELD OF THE INVENTION The present invention relates generally to computer security, and particularly to identifying direct calls to operating system kernel syscalls, and classifying the detected direct calls as benign or malicious. BACKGROUND OF THE INVENTION System calls, also known as syscalls, serve as a bridge between user-level programs and an operating system kernel. They enable programs to request essential services from the kernel, such as file operations, process management, and network communication. When a program initiates (i.e., calls) a syscall, it transitions from user mode to kernel mode, granting access to privileged operations. The syscall function in the kernel executes the requested task by interacting with kernel data structures. After completing the operation, the syscall function returns a value (e.g., a file descriptor or an error code) to the program, indicating the outcome of the operation. To return from the syscall, a special instruction is executed to switch control back to user mode, allowing the program to access return values stored in registers or memory locations for further processing. The description above is presented as a general overview of related art in this field and should not be construed as an admission that any of the information it contains constitutes prior art against the present patent application. SUMMARY OF THE INVENTION There is provided, in accordance with an embodiment of the present invention, a method for protecting a computer, including deploying in a memory of the computer a hooked version of a syscall used by an operating system kernel of the computer, receiving, from the hooked version of the syscall, a notification of a call to the hooked version of the syscall from a user mode of the computer, the notification including a return address in the memory and a set of features extracted from the call, analyzing the return address and the received features so as to classify the call as benign or malicious, and generating, for the computer, an alert upon classifying the new call as malicious. In one embodiment, the return address includes identifying the return address as belonging to a specified block in a user-mode segment of the memory. In some embodiments, the specified block is not allocated to a specified DLL. In another embodiment, classifying the call as malicious includes classifying the call as a direct call. In some direct call embodiments, the call includes a new call, and wherein the features further include additional features from previous direct calls to the hooked version of the syscall by execution entities executing in the memory of the computer and additional computers, wherein the new call and previous calls were conveyed during multiple days, and wherein one or more organizations include the computers. In a first profile embodiment, a given feature includes a mapped block in the memory including the execution entities, wherein analyzing the features includes identifying a number of distinct execution entities that generated the direct calls from the mapped block, and wherein classifying the new call as malicious includes identifying that identified number is less than a specified threshold. In a second profile embodiment, a given feature includes a mapped block in the memory including the execution entities, wherein analyzing the features includes identifying a number of distinct days when any of the computers having the given feature, and wherein classifying the new call as malicious includes identifying that identified number is less than a specified threshold. In a third profile embodiment, the method further includes defining a key including first and second features, the first feature including a given execution entity that conveyed one or more of the direct calls, and the second feature including a mapped block in the memory including an execution entity that generated one or more of the direct calls, wherein analyzing the features includes identifying a number of distinct computers having the key, and wherein classifying the new call as malicious includes identifying that identified number is less than a specified threshold. In a fourth profile embodiment, the method further includes defining a key including first and second features, the first feature including a given execution entity that conveyed one or more of the direct calls, and the second feature including a mapped block in the memory including an execution entity that generated one or more of the direct calls, wherein analyzing the features includes identifying a number of distinct days when any of the computers have the key, and wherein classifying the new call as malicious includes identifying that identified number is less than a specified threshold. In a fifth profile embodiment, the method further includes defining a key including first and second features, the first feature including a given execution entity that conveyed one or more of the direct calls, an