The cdecl (C Declaration) calling convention is one of the most commonly used calling conventions in C and C++ programming. It is the default calling convention for these languages on many platforms and compilers. Understanding cdecl is crucial for systems programming, low-level debugging, and interfacing with assembly language.

-: Key Characteristics of cdecl :-

1. Argument Passing: Arguments are passed to functions on the stack in right-to-left order.
2. Stack Cleanup: The caller is responsible for cleaning up the stack after the function call.
3. Return Values: The return value is typically placed in the EAX register (on x86 systems).
4. Name Mangling: Simple, often involves adding an underscore prefix to function names in some platforms (e.g., _functionName).

-: Detailed Breakdown :-

1 Argument Passing

In the cdecl convention, function arguments are pushed onto the stack in reverse order (right to left). This means that the last argument is pushed first, and the first argument is pushed last. This approach helps in supporting functions with a variable number of arguments.

void myFunction(int a, float b, char c);

The arguments would be pushed onto the stack as follows:

push c      ; Push the char argument
push b      ; Push the float argument
push a      ; Push the int argument

2 Stack Cleanup

The cdecl convention mandates that the caller cleans up the stack after the function call. This means the caller must adjust the stack pointer to remove the arguments that were pushed.

call myFunction    ; Call the function
add esp, 12        ; Clean up the stack (3 arguments * 4 bytes each)

3 Return Values

The return value of a function in cdecl is typically placed in the EAX register on x86 systems. For functions that return larger data types (e.g., structs), the return value may be handled differently.

int sum(int x, int y);

In this case, the result of sum would be in EAX:

mov eax, [ebp+8]   ; Load first argument (x)
add eax, [ebp+12]  ; Add second argument (y)

4 Name Mangling

In some platforms, particularly on Windows, the cdecl convention involves name mangling by adding an underscore prefix to function names. This helps in distinguishing between different calling conventions and avoiding symbol clashes.

void _myFunction();

Stack Illustration in x86

In x86 architecture stack grows from higher memory address to lower memory address. Stack in x86 is managed by two registers (stack pointer and base pointer).

• Stack pointer (SP) always points to the top of the stack.
• Base pointer (BP) always points to the bottom of the stack.

A stack frame is a structured way of organizing data related to function calls, including parameters, local variables, return addresses, and saved registers. This organization allows for consistent and reliable access to these elements during function execution.

When we call a function its stack frame is set up, and when it returns the control, its stack is tear down.

Anatomy of a Stack Frame

A typical stack frame in x86 assembly consists of the following components:

1. Function Arguments: Parameters passed to the function.
2. Return Address: The address to which the function should return after execution.
3. Saved Base Pointer (EBP): The previous base pointer, saved to restore the caller's stack frame.
4. Local Variables: Variables local to the function.
5. Saved Registers: Registers that the function needs to preserve.

Stack Pointer (ESP)

The stack pointer (ESP) always points to the top of the current stack. It is automatically adjusted during push and pop operations, function calls, and returns.

• Push Operation: Decreases ESP and stores the value at the new ESP address.
• Pop Operation: Retrieves the value from the current ESP address and then increases ESP.

When we push a value onto the stack in x86 assembly, the stack pointer is decremented by the size of the value before the value is stored. This means that the stack pointer will point to the starting address of the value that is pushed.

Let's have a Illustration:

; Initial Stack Pointer
	|		|
	|-------| 0x0000 <-- Higher Memory address | Stack Pointer
	|		|
	|_______| 0x0004
	|		|
	|		|
	
;; When we do 
; push 5
; the stack pointer first decreases by 4 bytes in 32-bit system
; and 8 bytes in 64-bit system.

	|		|
	|-------| 0x0000 <-- Higher Memory address | Stack Pointer
	|		|
	|_______| 0x0004  <-- Stack Pointer
	|		|
	|		|
	(First Stack Pointer is decremented by 4)
	
	|		|
	|-------| 0x0000 <-- Higher Memory address | Stack Pointer
	| 0000  | 0x0001
	| 0000  | 0x0002
	| 0000  | 0x0003
	| 0101  | 0x0004  <-- Stack Pointer
	|-------|
	|		|
	(After that value 5 is inserted starting from the address
	where Stack Pointer is pointing from lower to high memory address)
	;; 5 = 0000 00000 00000 0101 (5 in binary)

Example:

push eax      ; Pushes EAX onto the stack
              ; ESP is decremented by 4 (assuming a 32-bit system)
              ; add eax value is inserted
pop ebx       ; Pops the top value of the stack into EBX
              ; ESP is incremented by 4

Base Pointer (EBP)

The base pointer (EBP) is used to create a stable reference point within a function. It typically remains constant throughout the function execution, pointing to the base of the current stack frame.

• Function Prologue: Sets up the stack frame by saving the old base pointer and setting the new base pointer.
• Function Epilogue: Restores the old base pointer before returning to the caller.

push ebp      ; Save the old base pointer
mov ebp, esp  ; Set the base pointer to the current stack pointer
sub esp, 16   ; Allocate space for local variables
; ... function code ...
mov esp, ebp  ; Restore the stack pointer
pop ebp       ; Restore the old base pointer
ret           ; Return to caller

Example :-

;; Initial Stack, Empty

	|_______| 0x7E00 <-- Higher Memory address | Stack Pointer | Base Pointer
	|_______|
	|_______|
	|_______|
	|_______|
	|		| <- Memory Address

1 Caller Setup:

push 5              ; Push second argument (b)
push 3              ; Push first argument (a)

;; Stack after push statement
	|		|
	|-------| 0x7E00 <-- Higher Memory address | Base Pointer
	|	5	|
	|_______| 0x7E04
	|		|
	|	3	|
	|-------| 0x7E08 <-- Base Pointer
	|		|
	|		| <-- Lower Memory Address

• Pushes arguments onto the stack.

call add            ; Call the function

;; This will be discussed after the function return.
add esp, 8          ; Clean up the stack (2 arguments * 4 bytes each)

• Calls the function.
• Clean up the stack after the call

2 Function Prologue:

It sets up the stack frame.

add:
    push ebp        ; Save the old base pointer
    mov ebp, esp    ; Set the new base pointer to the current stack pointer
    sub esp, 4      ; Allocate space for local variable 'sum' (4 bytes)

;; Stack after entering function
	|		|
	|-------| 0x7E00 <-- Higher Memory address | Base Pointer
	|	5	| second argument (pushed first)
	|_______| 0x7E04
	|		|
	|	3	| first argument (pushed second)
	|-------| 0x7E08
	|return | return address
	|address|
	|-------| 0x7E0C <-- Stack Pointer
	|		|
	|		|
	|-------|

;; Stack after push ebp
	|		|
	|-------| 0x7E00 <-- Higher Memory address | Base Pointer
	|	5	| second argument (pushed first)
	|_______| 0x7E04 
	|		|
	|	3	| first argument (pushed second)
	|-------| 0x7E08
	|return |
	|address|
	|-------| 0x7E0C
	|		|
	|  ebp	|
	|-------| 0x7E10 <-- Stack Pointer
	|		|
	|		|
	|-------|

;; Stack after mov ebp, esp
	|		|
	|-------| 0x7E00 <-- Higher Memory address
	|	5	|
	|_______| 0x7E04
	|		|
	|	3	|
	|-------| 0x7E08
	|return |
	|address|
	|-------| 0x7E0C
	|		|
	|  ebp	|
	|-------| 0x7E10 <-- Stack Pointer | Base Pointer
	|		|
	|		|
	|-------|


;; Stack after sub esp, 4
	|		|
	|-------| 0x7E00 <-- Higher Memory address
	|	5	|
	|_______| 0x7E04
	|		|
	|	3	|
	|-------| 0x7E08
	|return |
	|address|
	|-------| 0x7E0C
	|		|
	|  ebp	| (old ebp address)
	|-------| 0x7E10 <-- Base Pointer
	|		|
	|		| (space for 1 local variable)
	|-------| 0x7E14 <-- Stack Pointer
	

• Saves the old base pointer.
• Sets the new base pointer.
• Allocates space for local variables.

3 Accessing Arguments and Local Variables:

;; accessing first argument as base pointer act as a hanger in stack frame.
; ebp => old ebp address
; ebp + 4 = return address
; ebp + 8 = 3, first parameter
; ebp + 12 = 5, second parameter
; ebp - 4  = Accesses the local variable
	|		|
	|-------| 0x7E00 <-- Higher Memory address
	|	5	|
	|_______| 0x7E04
	|		|
	|	3	|
	|-------| 0x7E08
	|return |
	|address|
	|-------| 0x7E0C
	|		|
	|  ebp	| (old ebp address)
	|-------| 0x7E10 <-- Base Pointer
	|		|
	|		| (space for 1 local variable)
	|-------| 0x7E14 <-- Stack Pointer

mov eax, [ebp+8]    ; Load first argument (a) into EAX
add eax, [ebp+12]   ; Add second argument (b) to EAX
mov [ebp-4], eax    ; Store result in local variable 'sum'

• Arguments are accessed relative to the base pointer.
• Local variables are also accessed relative to the base pointer.

4 Returning the Result:

• The result is placed in EAX for retuning to the caller.

mov eax, [ebp-4]    ; Load local variable 'sum' into EAX

5 Function Epilogue:

Stack Frame teardown.

• Restores the old base pointer.
• Returns to the caller.

mov esp, ebp        ; Restore the stack pointer
pop ebp             ; Restore the old base pointer
ret                 ; Return to caller

;; stack before this thing
	|		|
	|-------| 0x7E00 <-- Higher Memory address
	|	5	|
	|_______| 0x7E04
	|		|
	|	3	|
	|-------| 0x7E08
	|return |
	|address|
	|-------| 0x7E0C
	|		|
	|  ebp	| (old ebp address)
	|-------| 0x7E10 <-- Base Pointer
	|		|
	|		| (space for 1 local variable)
	|-------| 0x7E14 <-- Stack Pointer

;; stack after mov esp, ebp
	|		|
	|-------| 0x7E00 <-- Higher Memory address
	|	5	|
	|_______| 0x7E04
	|		|
	|	3	|
	|-------| 0x7E08
	|return |
	|address|
	|-------| 0x7E0C
	|		|
	|  ebp	| (old ebp address)
	|-------| 0x7E10 <-- Base Pointer | Stack Pointer
	|		|
	|		| Local variable track deleted
	|-------| 0x7E14

;; Stack after pop ebp
	|		|
	|-------| 0x7E00 <-- Higher Memory address | Base Pointer
	|	5	|
	|_______| 0x7E04
	|		|
	|	3	|
	|-------| 0x7E08
	|return |
	|address|
	|-------| 0x7E0C <-- Stack Pointer
	|		|
	|   	|
	|-------| 0x7E10
	|		|
	|		|
	|-------| 0x7E14

;; ret instruction, pops the return address from the stack.
; Jump to return address: Control is transfered back to the return address,
resuming execution in the caller function.
; Stack after ret statement
	|		|
	|-------| 0x7E00 <-- Higher Memory address | Base Pointer
	|	5	|
	|_______| 0x7E04
	|		|
	|	3	|
	|-------| 0x7E08 <-- Stack Pointer
	|		|
	|		|
	|-------| 0x7E0C
	|		|
	|   	|
	|-------| 0x7E10
	|		|
	|		|
	|-------| 0x7E14

6 Caller Clear the Arguments:

add esp, 8		; Clean up the stack (2 arguments)
;; Stack before this step
	|		|
	|-------| 0x7E00 <-- Higher Memory address | Base Pointer
	|	5	|
	|_______| 0x7E04
	|		|
	|	3	|
	|-------| 0x7E08 <-- Stack Pointer
	|		|
	|		|
	|-------| 0x7E0C
	|		|
	|   	|
	|-------| 0x7E10
	|		|
	|		|
	|-------| 0x7E14

;; Stack after 'add esp, 8'
	|		|
	|-------| 0x7E00 <-- Higher Memory address | Base Pointer | Stack Pointer
	|	5	|
	|_______| 0x7E04
	|		|
	|	3	|
	|-------| 0x7E08
	|		|
	|		|
	|-------| 0x7E0C
	|		|
	|   	|
	|-------| 0x7E10
	|		|
	|		|
	|-------| 0x7E14

Note the data is still in memory but we lost the track that stack pointer
is adjusted.

Calling C Function From Assembly:

Remember when we call c function from assembly it uses famous cdecl calling convention. Here the calling function is Assembly function and the caller function is C function.

• Parameter are pushed onto the stack from right to left by the caller function. (RTL = Right To Left).
• The caller is responsible for cleaning by the stack after the callee return the control.
• The return value is typically placed in the EAX register for integer return values.

// C code callee
int add(int a, int b) {
	return a + b;
}

;; Assembly calling code
extern add	; declare external c function

push 10		; Push second argument
push 20		; Push first argument
call add	; Calling c function

; Clean up the stack
add esp, 8		; 2 arguments * 4 bytes each

Calling Assembly Function From C

Calling an assembly function from c need to declare it as the `extern, Which means the function's prototype and its implementation is in assembly language.

;; Assembly callee code
sum:
    ; Function prologue (optional in simple functions)
    push ebp        ; Save the base pointer
    mov ebp, esp    ; Set up a new base pointer

    ; Function body
    mov eax, [ebp+8]    ; Load first argument into EAX
    add eax, [ebp+12]   ; Add second argument to EAX

    ; Function epilogue (optional in simple functions)
    pop ebp         ; Restore the base pointer
    ret             ; Return to caller

// C caller

// Declare external assembly function
extern int sum(int a, int b);

int a = 10, b = 20;
int result = sum(a, b);