Malware authors use a number of tricks to avoid detection and analysis. One of the most popular methods is to employ a packer, a tool that compresses, encrypts, and/or modifies a malicious file’s format. (Packers can also be used for legitimate ends, for example, to protect a program against cracking or copying.) All these tricks decrease the chance of detection by antimalware products and help to avoid analysis by security researchers.
Packers can both make it harder for security staff to identify the behavior of malware and increase the amount of time required for an analysis. Unpacking malware is the first challenge to understanding a threat. The complexity of packers varies.
Types of packers
A packer can act simply as armor to protect the binary. It is more convenient for attackers to use a packer rather than to directly implement protection inside the code. Advanced malware coded by organized cybercriminal groups, however, employ custom packers or implement complex protection inside malicious files.
For packers that encrypt or compress a file, a stub (a piece of code that contains the decompression or decryption routine) acts as a loader, which executes before the malware.
A packer compresses or encrypts data. The original file is passed in the packer routine and stored in a packed section in the new .exe. Once the file is running, the decompression stub stored in the packed file will decompress the packed section. The original .exe file is then loaded into memory.
In these cases the original entry point, the memory address where the program starts, is relocated in the packed section. The analyst has to retrieve it to recover the original file.
Other packers act as a proxy and protect the import address table (IAT). The IAT references functions that are used by a program and are available by the Windows API. During runtime the IAT is resolved dynamically and used by the program when necessary. This protection increases the difficulty of unpacking the malware. Indeed, a memory dump does not work because the IAT is not complete. Without the IAT, it is more difficult to correctly analyze malware because it cannot be properly disassembled.
A packer obfuscating an API. The packer stub acts as a proxy and intercepts every call to the API. The call is translated from the API to the stub and is run by the payload.
In such cases, a malware analyst must fix the import by locating the function and tracing the return routine. A stub redirects the execution to the original API. In some cases, the address of the API is located in the EIP register.
Still other techniques virtualize the code and emulate the behavior of a processor to run the original file. This technique adds an abstraction layer between the executed code and its behavior. The original code is translated in the byte code of the virtual machine. This technique is one of the most difficult to analyze.
A code virtualization packer. The original file shows x86 assembly when reversed. Once the file is protected, the reversed code will be in the virtual machine’s byte code.
To remove this kind of protection, a malware analyst must study the behavior and the byte code of the virtual software to retrieve and analyze the original file.
Other packers will combine techniques such as anti-debugging or anti-sandbox and add more functions to protect the malware. Some malware use several layers to protect themselves against detection and analysis. To analyze them we must remove each layer until we recover the original file. The fastest recovery technique is to run the malware and dump it directly from memory. This can be difficult because some packers add antidumping tricks.
Packers remain an effective way to slow down analysis and decrease detection by antimalware products. They are also the favorite tool of attackers for protecting malware. Nevertheless, manual analysis usually defeats packers.