Calling Conventions

What is a Calling Convention?

A calling convention defines how function calls are made at the binary level. It answers questions like:

• Who passes the arguments (caller or callee)?
• In which order are arguments passed?
• How are arguments passed? (Registers, stack, etc.)
• Who cleans up the stack after the function returns?
• How is the return value passed back?

A calling convention is a standardized method for function calls in programming. It defines how arguments are passed, how the return value is managed, and how the call stack is utilized. This consistency is crucial for ensuring that functions execute correctly, particularly when dealing with different programming languages or low-level system operations.

Why Does It Matter?

Calling conventions are important because:

• They allow binary compatibility across modules or libraries.
• Misusing or mismatching calling conventions can lead to crashes, corrupted memory, or undefined behavior.
• They can impact performance, especially in low-level or real-time systems.

Key Components of Calling Conventions

1 Argument Passing: This specifies how arguments are passed to functions. They can be passed via registers, the stack, or a combination of both.

2 Return Value Handling: This defines how the function's return value is provided back to the caller, often in a specific register.

3 Stack Management: This describes the use and management of the call stack, including the order of pushing arguments and the return address.

4 Stack Clean Up: It refers to the process of restoring the call stack to its state before the function call was made. This ensures that the stack does not grow uncontrollably, which could lead to stack overflows or corrupted memory. The responsibility for stack cleanup can either fall on the caller (the function that calls another function) or the callee (the function that is being called), depending on the calling convention used.

4 Caller/Callee-Saved Registers: These indicate which registers must be preserved across function calls. Caller-saved registers are saved by the caller if needed after the call, while callee-saved registers are preserved by the function being called.

5 Name Mangling: This involves transforming function names to include additional information such as namespaces or argument types, common in languages like C++.

Different Calling Conventions:

1 cdecl (C Declaration):

• Usage: Default in C on x86 platforms.
• Arguments: Pushed onto the stack in right-to-left order.
• Stack Cleanup: Done by the caller.
• Return Values: Placed in a register (e.g., EAX in x86).

; Caller code
push 5                  ; Push second argument
push 3                  ; Push first argument
call add_numbers        ; Call the function
add esp, 8              ; Clean up the stack (2 arguments * 4 bytes)

2 stdcall (Standard Call):

• Usage: Windows API functions.
• Arguments: Pushed onto the stack in right-to-left order.
• Stack Cleanup: Done by the callee.
• Return Values: Placed in a register.

; Callee code
add_numbers:
    push ebp            ; Save base pointer
    mov ebp, esp        ; Set base pointer to current stack pointer
    mov eax, [ebp+8]    ; Load first argument into EAX
    add eax, [ebp+12]   ; Add second argument to EAX
    mov esp, ebp        ; Restore stack pointer
    pop ebp             ; Restore base pointer
    ret 8               ; Clean up 8 bytes (2 arguments * 4 bytes) and return

3 fastcall:

• Usage: Performance-critical functions.
• Arguments: First few arguments in registers, remaining on the stack.
• Stack Cleanup: Varies by implementation.

4 thiscall:

• Usage: C++ member functions.
• Arguments: this pointer passed in a register (e.g., ECX in x86).
• Stack Cleanup: Similar to cdecl or stdcall.

5 vectorcall:

• Usage: Functions with vector arguments.
• Arguments: Passed in vector registers when possible.
• Stack Cleanup: According to the calling convention.

x64 Calling Convention

Windows x64 Calling Convention (Microsoft ABI)

• First 4 arguments: passed in RCX, RDX, R8, R9 (integers/pointers)
• Additional args: passed on the stack
• Caller allocates shadow space (32 bytes) for callee to use
• Return values in RAX

System V AMD64 ABI (Linux/macOS x64)

• First 6 arguments: RDI, RSI, RDX, RCX, R8, R9
• Return in RAX or RDX:RAX for large types
• Stack alignment to 16 bytes required