X86 architecture for Reverse Engineering | Malware analysis

Hello everyone, welcome back to another learning session on X86 architecture. In this blog, we aim to enhance your understanding of this architecture to help you analyze files more effectively. If you find this topic interesting and wish to explore similar content, please consider following our page. Your feedback is highly valuable to me, and I am excited to hear from you. Throughout this blog, I will not only delve into theory but also provide practical examples to facilitate your learning experience.

Introduction

Malware frequently exploits inherent vulnerabilities within system architectures to achieve its malicious objectives. Therefore, gaining a comprehensive understanding of how malware operates necessitates delving into the intricacies of the underlying systems in which they propagate. In this session, we will provide a concise yet insightful overview of the x86 architecture, specifically focusing on aspects crucial for malware analysis. It is important to acknowledge that, due to the nature of this discussion’s focus, certain intricate details of the x86 architecture may be omitted, as they are not directly pertinent to the examination of malware.

CPU architecture

Certainly! Let’s provide a more detailed explanation of the components of a computer system and their roles in executing a program:

1. Control Unit

The Control Unit is a vital component of the central processing unit (CPU). Its primary function is to direct and coordinate the execution of instructions within the CPU. It receives instructions from the main memory, which is located outside the CPU. These instructions are fetched sequentially, and the address of the next instruction to be executed is stored in a special register known as the Instruction Pointer (IP). In 32-bit systems, this register is called the Extended Instruction Pointer (EIP), while in 64-bit systems, it is called the RIP (Instruction Pointer). The Control Unit ensures that instructions are fetched, decoded, and executed in the correct order to perform the desired operations.

2. Arithmetic Logic Unit (ALU)

The Arithmetic Logic Unit is a fundamental part of the CPU responsible for executing arithmetic and logical operations. When the Control Unit fetches an instruction from memory, it is passed to the ALU for execution. The ALU performs operations such as addition, subtraction, multiplication, division, bitwise AND, OR, XOR, and logical comparisons (e.g., greater than, less than). The results of these operations are then stored either in the CPU’s registers or in the main memory.

3. Registers

Registers are small, high-speed storage locations within the CPU that are used to hold data temporarily during program execution. They are much faster to access than the main memory, which is located outside the CPU. Registers are essential for optimizing the CPU’s performance because they provide quick access to frequently used data and help in speeding up the execution of instructions. Some common types of registers include:
— Data Registers: Used to hold operands and intermediate results during arithmetic and logical operations.
— Address Registers: Store memory addresses for data retrieval and storage.
— Control Registers: Used by the Control Unit to manage and control CPU operations.

4. Memory (Main Memory or RAM)

The Memory, also known as Main Memory or Random Access Memory (RAM), is a large pool of storage where all the code and data required for a running program are stored. When a user executes a program, its instructions and data are loaded into the memory from secondary storage devices (e.g., hard disks) through a process called “loading” or “loading into memory.” The CPU then fetches instructions one by one from the memory according to the value stored in the Instruction Pointer (IP/RIP) and executes them using the Arithmetic Logic Unit. The memory’s random access nature allows the CPU to access any location in memory directly, enabling efficient data retrieval and storage.

5. I/O Devices:

Input/Output devices (I/O devices) are peripheral devices that allow the computer to interact with the outside world. These devices include keyboards, mice, displays (monitors), printers, scanners, external storage devices (e.g., hard disks, USB drives), network adapters, and more. When a program needs to interact with these devices or transfer data between the CPU and external devices, it does so through Input/Output operations. The Control Unit manages these interactions, and data is exchanged between the CPU and I/O devices through designated pathways.

Registers Overview

Registers serve as the CPU’s high-speed storage units, allowing it to access data faster than other storage mediums. Due to their limited capacity, they need to be utilized efficiently. Registers can be categorized into various types, each serving specific purposes:

1. Instruction Pointer:

The Instruction Pointer (IP) is a crucial register that holds the memory address of the next instruction to be fetched and executed by the CPU. It essentially points the CPU to the next operation it needs to perform, ensuring the correct sequence of instructions during program execution.

2. General Purpose Registers:

General Purpose Registers (GPRs) are versatile registers capable of holding different types of data, such as integers, addresses, or pointers. They are used by the CPU to perform various arithmetic and logical operations. GPRs are crucial for temporary data storage during calculations, making them essential for efficient program execution.

3. Status Flag Registers:

Status Flag Registers (also known as Condition Code Registers or Flag Registers) are special registers that contain individual flags representing the status of specific CPU operations. These flags are binary indicators, and each one signals a specific condition resulting from the execution of an instruction. For example, flags may indicate whether a calculation resulted in zero, a negative number, or if there was a carry or overflow during an arithmetic operation. Program control flow and decision-making often rely on these flags.

4. Segment Registers:

Segment Registers are specific to the x86 architecture and have a role in memory segmentation. While modern operating systems and processors largely abstract the complexity of segmentation, segment registers still play a role in certain compatibility modes. They were historically used to manage memory access by providing an offset to a specific segment of memory. In contemporary systems, their direct use is limited, but their influence is seen in compatibility and certain security mechanisms.

x86 Registers (32-bit):

• EAX (Accumulator Register): Used for arithmetic operations and storing return values of functions.
• EBX (Base Register): Typically used as a base pointer for memory access.
• ECX (Counter Register): Often used as a loop counter.
• EDX (Data Register): Used for arithmetic operations and I/O operations.

x86–64 Registers (64-bit):

• RAX (Accumulator Register): Used for arithmetic operations and storing return values of functions.
• RBX (Base Register): Typically used as a base pointer for memory access.
• RCX (Counter Register): Often used as a loop counter.
• RDX (Data Register): Used for arithmetic operations and I/O operations.
• RSI (Source Index): Used as a source pointer in string operations.
• RDI (Destination Index): Used as a destination pointer in string operations.
• RBP (Base Pointer): Frame pointer for referencing function parameters and local variables.
• RSP (Stack Pointer): Points to the top of the stack.

Register Overview-2

In a computer’s central processing unit (CPU), when it performs operations, there are often situations where we need to know the outcome or status of those operations. For example, we may want to know if a mathematical calculation resulted in a negative or zero value, if an operation caused an overflow, or if two values are equal or not.

To keep track of these outcomes, CPUs have a special set of individual flags called “Status Flags.” These flags are single bits that can be either set to 1 (meaning true) or 0 (meaning false). Each flag represents a specific condition or outcome of an operation.

In 32-bit systems, there is a register called EFLAGS, and in 64-bit systems, it’s extended to a 64-bit register called RFLAGS. These registers store the status flags and allow programs to access and modify them as needed.

Here are some important flags and their meanings:

1. Carry Flag (CF): Indicates if an arithmetic operation resulted in a carry or borrow. For example, in an addition, if the result overflows the maximum value, the carry flag is set.
2. Zero Flag (ZF): Set when the result of an operation is zero. It helps to check if two values are equal.
3. Sign Flag (SF): Indicates the sign of the result. If the result is negative, the sign flag is set.
4. Overflow Flag (OF): Set when an arithmetic operation causes an overflow, usually when the result is too large to be represented with the available number of bits.

Segment Registers

In a computer’s central processing unit (CPU), the memory is organized as a flat, linear address space. However, to provide efficient and secure memory access, the CPU uses a concept called “segmentation.” Segmentation divides the memory into smaller logical segments, allowing different parts of a program to be accessed separately.

To manage these segments, the CPU has special 16-bit registers called “Segment Registers.” These registers are used to store segment addresses and allow the CPU to translate logical addresses into physical addresses.

There are six segment registers, each serving a specific purpose:

1. Code Segment (CS): This register points to the segment in memory that contains the program’s instructions (code). When the CPU fetches instructions to execute, it uses the Code Segment to determine the starting address of the instruction in memory.

2. Data Segment (DS): The Data Segment register points to the segment in memory that holds the program’s data, such as variables, arrays, and other data structures. It is used by the CPU to access data elements during program execution.

3. Stack Segment (SS): The Stack Segment register points to the segment in memory that represents the program’s stack. The stack is used to store temporary data, function return addresses, and local variables. The CPU uses the Stack Segment to manage the stack efficiently.

4. Extra Segments (ES, FS, and GS): These are additional segment registers that can be used by the program to access more data sections. The specific use of these registers depends on the architecture and operating system.

The use of segment registers is mainly seen in older x86 systems with real mode or protected mode memory addressing. In modern x86–64 systems and newer CPU architectures, the concept of segmentation is less prevalent, and the memory is mostly addressed using a flat memory model.

Memory Overview

1. Code:

The “Code” section, also known as the “Text” section, is where the executable code of the program resides. This section contains the instructions that make up the program’s logic and functionality. The code is usually read-only and is loaded into memory from the program’s executable file when the program starts. During execution, the CPU fetches instructions from the code section, decodes them, and executes them sequentially, following the program’s control flow.

2. Data:

The “Data” section contains initialized global and static variables used by the program. Global variables are accessible throughout the entire program, while static variables are only accessible within their own scope (e.g., a function). When the program starts, the data section is initialized with the specified values for these variables. This section is typically writable, allowing the program to modify the values of variables during runtime. The data section includes both initialized global and static variables (data segment) and uninitialized variables (bss segment).

3. Heap:

The “Heap” is an area of memory used for dynamic memory allocation. It is a region of memory where the program can request additional memory space during runtime for storing data structures like arrays or objects whose size is not known at compile time. Memory allocation and deallocation in the heap are typically handled using functions like `malloc()` and `free()` in C/C++. Proper management of the heap is essential to prevent memory leaks and unnecessary memory fragmentation.

4. Stack:

The “Stack” is a special area of memory used for managing function calls and local variables. Each time a function is called, its local variables and function call information (return address, saved registers, etc.) are stored on the stack. This creates a “stack frame” for the function. When the function returns, its stack frame is removed, and control returns to the calling function. The stack operates in a last-in-first-out (LIFO) manner, meaning that the most recently called function’s stack frame is at the top of the stack, and it gets removed first when the function returns. The stack is crucial for maintaining the execution flow of a program and is automatically managed by the CPU.

1. When main is called, its local variable mainVar is stored on the stack.
2. main then calls functionA with mainVar as an argument.
3. Inside functionA, a new local variable localVarA is declared, and it's also stored on the stack.
4. functionA calls functionB with localVarA as an argument.
5. Inside functionB, another local variable localVarB is declared and stored on the stack.It’s important to note that in modern operating systems, memory protection mechanisms and virtual memory are employed to provide isolation and security between processes. Each process has its own virtual memory space, which is mapped to physical memory by the operating system. This allows multiple processes to run concurrently while being isolated from each other’s memory. The memory layout described above is a logical representation of a process’s memory, and the physical memory organization is managed by the operating system using techniques like paging and segmentation.

The anatomy of Memory

It is important to note that the stack is a critical component in memory management, and understanding its significance is essential. Let me explain the details briefly.

1. Code Execution and Payload Injection: Malware often aims to execute arbitrary code on a target system. One common technique employed by malware is called “stack-based buffer overflow.” In this attack, malware exploits a vulnerability in a program to overwrite the stack with malicious code, effectively injecting its payload into the program’s execution flow. When the compromised program returns from its current function, it unintentionally jumps to the injected code on the stack, leading to the execution of the malware’s malicious instructions.

2. Shellcode Injection: Malware may use the stack to inject shellcode into a vulnerable program. Shellcode is a small piece of code that typically provides a malicious actor with unauthorized access to a compromised system. By exploiting stack-based buffer overflows or other vulnerabilities, malware can place its shellcode on the stack and trick the program into executing it, providing the attacker with control over the system.

3. Return-Oriented Programming (ROP) Attacks: In modern systems with security mechanisms like Data Execution Prevention (DEP) and Address Space Layout Randomization (ASLR), executing arbitrary code directly from the stack becomes challenging. Malware adapts to these defenses by using Return-Oriented Programming (ROP) attacks. In ROP attacks, malware constructs a sequence of return addresses (gadgets) from the program’s existing code, effectively chaining them together to achieve specific actions, such as bypassing security mechanisms or executing unintended instructions.

4. Stack-based Data Storage: Malware may use the stack to store critical data temporarily during its execution. This data might include encryption keys, configuration information, or addresses of critical functions. Storing data on the stack can help malware hide its presence and avoid detection by antivirus software, as the stack is an essential and legitimate part of the program’s memory layout.

5. Information Leakage: Malware might exploit vulnerabilities that allow it to access data stored on the stack or obtain sensitive information leaked onto the stack by other programs. This information can then be used to perform further attacks or exfiltrate data.

You can add more attacks to abuse the stack. I had already written some blogs about abusing stack. You can check my previous blogs.

It’s important to note that the misuse of the stack for malicious purposes is a consequence of software vulnerabilities rather than the stack itself being inherently dangerous. Software developers and security professionals continually work to identify and fix vulnerabilities that could be exploited by malware to manipulate the stack. Regular software updates and security patches are crucial for maintaining the integrity and security of computer systems.

Stack Frame

1. Stack Frame:
In computer programming, a stack frame is a portion of the call stack used to store information related to a function call. It includes local variables, function arguments, and the necessary information for the function’s execution. Each function call creates a new stack frame, which is removed when the function returns. This process ensures that each function call is isolated, and local variables do not interfere with other functions.

2. ESP and EBP:
ESP (Extended Stack Pointer) and EBP (Extended Base Pointer) are registers in the x86 architecture used to manage the stack frame.

The ESP register holds the memory address of the top of the stack (the current stack pointer). When a function is called, the stack pointer is adjusted to allocate space for the function’s local variables and other information in the new stack frame.

The EBP register acts as a frame pointer, pointing to the base of the current function’s stack frame. It is often used to access function parameters and local variables relative to a fixed reference point.

3. Old Base Pointer and Return Address:
During function calls, the EBP register plays a crucial role in maintaining the integrity of the stack frame. Before a function is called, its caller pushes the EBP value onto the stack, effectively creating a new base pointer for the callee. This EBP value is often referred to as the Old Base Pointer.

Additionally, the caller pushes the return address (the address to which control should return after the callee finishes) onto the stack before calling the function. The return address is used to continue the execution in the calling function after the callee returns.

4. Function Prologue and Epilogue:
The function prologue and epilogue are code sequences responsible for setting up and tearing down the stack frame when a function is called and returned, respectively.

Summary

The x86 architecture is a widely used CPU design, powering most personal computers. It consists of the Control Unit, which fetches and executes instructions, and the Arithmetic Logic Unit, responsible for calculations. Registers store essential data for quick access, while Memory holds code and data during program execution. Input/Output devices allow interaction with computers.

Malware analysis involves studying malicious software to understand its behavior and defend against it. Understanding the architecture is crucial for effective analysis. Malware often exploits vulnerabilities to abuse system design, infecting and compromising systems. Analyzing malware helps in developing security measures and protecting against cyber threats.

In brief, the x86 architecture underpins modern computing, while malware analysis is vital for cybersecurity, ensuring safe and secure digital environments.

Thank you for taking the time to read this blog. I hope you have understood the concept well. If you have any doubts or if anything is unclear, please feel free to ask. I am always open to hearing your feedback. If I made any mistakes, please point them out so that I can make corrections. As a malware analyst, it is crucial to have a thorough understanding of the topic.

You can follow me on

Twitter: https://twitter.com/lockpin010_?s=09

LinkedIn: https://www.linkedin.com/in/ahmetgoker/

Ahmet Göker | Threat Cases Operator