US-20260127284-A1 - Systems and Methods for Detecting Malicious Shellcode

US20260127284A1US 20260127284 A1US20260127284 A1US 20260127284A1US-20260127284-A1

Abstract

Kernel-mode security software detects a trigger event indicative of a specific stage in the lifecycle of a target software entity executing in user mode. In response, the security software identifies a target object residing in memory (e.g., loaded library, chunk of code, etc.) according to a current content of the user-mode call stack, and determines whether the target object is malicious. Various methods described herein detect malicious modifications of a loaded module, such as overload, stomping, and unhooking, among others. Other methods described herein detect dynamically swapped libraries and malicious shellcode, among others.

Inventors

Andrei V. LUTAS
Ionel C. ANICHITEI
Nicolae BODEA

Assignees

Bitdefender IPR Management Ltd.

Dates

Publication Date: 20260507
Application Date: 20250107

Claims (20)

1 . A computer system comprising at least one hardware processor configured to: in response to an occurrence of a trigger event caused by an executing process, identify a region of memory used by the process, the region of memory identified according to a current content of a call stack of the process; determine whether the region of memory is currently allocated for any software module used by the process; in response, if the region of memory is not allocated for any software module used by the process, analyze a sequence of processor instructions stored within the region of memory to determine whether the sequence includes instructions referencing valid memory addresses; and in response to analyzing the sequence, if the sequence does not include any instructions referencing valid memory addresses, determine that the region of memory stores malicious software.
2 . The computer system of claim 1 , wherein the at least one hardware processor is further configured, in response to determining that at least one instruction of the sequence references a valid memory address, determine that the region of memory stores benign software.
3 . The computer system of claim 1 , wherein analyzing the sequence of processor instructions comprises: determining whether a selected instruction of the sequence includes a memory reference; and in response to determining that the selected instruction includes the memory reference, determining whether the memory reference comprises a valid absolute address or a valid relative address located within a memory image of another software module.
4 . The computer system of claim 1 , wherein analyzing the sequence of processor instructions comprises: determining whether a selected instruction of the sequence includes an immediate value; in response, if yes, determining whether the immediate value comprises a valid memory address.
5 . The computer system of claim 1 , wherein the at least one hardware processor is further configured to: in response to determining that the memory region is not allocated for any software module, determine a type of content that the memory region is allocated for; and if the type of content does not comprise executable code, determine that the region of memory comprises malicious software.
6 . The computer system of claim 5 , wherein determining the type of content comprises scanning the region of memory for a content-type signature indicative of the type of content.
7 . The computer system of claim 5 , wherein determining the type of content comprises determining whether the region of memory stores an item selected from a group consisting of a document, a compressed archive, and a media file.
8 . The computer system of claim 5 , wherein determining the type of content comprises determining whether the region of memory stores an image of a portable executable (PE) file.
9 . The computer system of claim 1 , wherein the at least one hardware processor is configured to determine whether the region of memory is currently allocated for any software module according to an allocation type of the region of memory.
10 . The computer system of claim 9 , wherein the at least one hardware processor is configured to determine that the region of memory is currently allocated for a software module if the allocation type of the region of memory is ‘image’.
11 . A computer security method comprising employing at least one hardware processor of a computer system to: in response to an occurrence of a trigger event caused by an executing process, identify a region of memory used by the process, the region of memory identified according to a current content of a call stack of the process; determine whether the region of memory is currently allocated for any software module used by the process; in response, if the region of memory is not allocated for any software modules used by the process, analyze a sequence of processor instructions stored within the region of memory to determine whether the sequence includes instructions referencing valid memory addresses; and in response to analyzing the sequence, if the sequence does not include any instructions referencing valid memory addresses, determine that the region of memory stores malicious software.
12 . The method of claim 10 , further comprising employing the at least one hardware processor to, in response to determining that at least one instruction of the sequence references a valid memory address, determine that the region of memory stores benign software.
13 . The method of claim 10 , wherein analyzing the sequence of processor instructions comprises: determining whether a selected instruction of the sequence includes a memory reference; and in response to determining that the selected instruction includes the memory reference, determining whether the memory reference comprises a valid absolute address or a valid relative address located within a memory image of another software module.
14 . The method of claim 10 , wherein analyzing the sequence of processor instructions comprises: determining whether a selected instruction of the sequence includes an immediate value; in response, if yes, determining whether the immediate value comprises a valid memory address.
15 . The method of claim 10 , further comprising employing the at least one hardware processor to: in response to determining that the memory region is not allocated for any software module, determine a type of content that the memory region is allocated for; and if the type of content does not comprise executable code, determine that the region of memory comprises malicious software.
16 . The method of claim 15 , wherein determining the type of content comprises scanning the region of memory for a content-type signature indicative of the type of content.
17 . The method of claim 15 , wherein determining the type of content comprises determining whether the region of memory stores an item selected from a group consisting of a document, a compressed archive, and a media file.
18 . The method of claim 15 , wherein determining the type of content comprises determining whether the region of memory stores an image of a portable executable (PE) file.
19 . The method of claim 10 , comprising determining whether the region of memory is currently allocated for any software module according to an allocation type of the region of memory.
20 . The method of claim 19 , comprising determining that the region of memory is currently allocated for a software module if the allocation type of the region of memory is ‘image’.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. patent application Ser. No. 19/012,315, filed Jan. 7, 2025, titled “Systems and Methods for Detecting Malicious Software Libraries,” which is a continuation of US patent application Ser. No. 19/012,275, filed Jan. 7, 2025, titled “Systems and Methods for Detecting Malicious Modifications of a Loaded Software Module,” which claim the benefit of the filing date of U.S. provisional patent application No. 63/715,705, filed on Nov. 4, 2024, titled “Anti-Malware Systems and Methods,” the content of all of which is incorporated by reference herein. BACKGROUND OF THE INVENTION The invention relates to computer security, and in particular to protecting computers against malicious software (malware). Malicious software affects a great number of computer systems worldwide. In its many forms such as computer viruses, worms, rootkits, and spyware, malware presents a serious risk to millions of computer users, making them vulnerable to loss of data and sensitive information, identity theft, and loss of productivity, among others. Security software employs a variety of methods and strategies to combat malicious software. Some such methods try to match the contents of memory against a pre-determined library of malicious code snippets and patterns, commonly known as signatures. Other exemplary methods monitor the execution of selected software entities, looking for patterns of behavior that are indicative of malice. Malware detection and mitigation face substantial technical challenges. Activities related to behavior monitoring and signature matching are computationally costly and may adversely affect productivity and user experience. Some components of the security software are themselves vulnerable to attack. Last but not least, malware constantly evolves to evade detection. Therefore, there is a persistent interest in developing novel, efficient, and robust anti-malware systems and methods. SUMMARY OF THE INVENTION According to one aspect, a computer system comprises at least one hardware processor configured to, in response to an occurrence of a trigger event caused by an executing process, identify a region of memory used by the process, the region of memory identified according to a current content of a call stack of the process, and to further determine whether the region of memory is currently allocated for any software module used by the process. The at least one hardware processor is further configured, if the region of memory is not allocated for any software modules used by the process, to analyze a sequence of processor instructions stored within the region of memory to determine whether the sequence includes instructions referencing valid memory addresses, and in response, if the sequence does not include any instructions referencing valid memory addresses, to determine that the region of memory stores malicious software. According to another aspect, a computer security method comprises employing at least one hardware processor of a computer system to, in response to an occurrence of a trigger event caused by an executing process, identify a region of memory used by the process, the region of memory identified according to a current content of a call stack of the process, and to further determine whether the region of memory is currently allocated for any software module used by the process. The method further comprises, if the region of memory is not allocated for any software modules used by the process, analyzing a sequence of processor instructions stored within the region of memory to determine whether the sequence includes instructions referencing valid memory addresses, and in response, if the sequence does not include any instructions referencing valid memory addresses, determining that the region of memory stores malicious software. According to another aspect, a non-transitory computer-readable medium stores instructions which, when executed by at least one hardware processor of a computer system, cause the computer system to, in response to an occurrence of a trigger event caused by an executing process, identify a region of memory used by the process, the region of memory identified according to a current content of a call stack of the process, and to further determine whether the region of memory is currently allocated for any software module used by the process. The instructions further cause the computer system, if the region of memory is not allocated for any software modules used by the process, to analyze a sequence of processor instructions stored within the region of memory to determine whether the sequence includes instructions referencing valid memory addresses, and in response, if the sequence does not include any instructions referencing valid memory addresses, to determine that the region of memory stores malicious software. BRIEF DESCRIPTION OF DRAWINGS The foregoing aspects and advantages of the presen