What is a Calling Convention?

A calling convention is a standardized method for function calls in programming. It defines how arguments are passed, how the return value is managed, and how the call stack is utilized. This consistency is crucial for ensuring that functions execute correctly, particularly when dealing with different programming languages or low-level system operations.

Key Components of Calling Conventions

1 Argument Passing: This specifies how arguments are passed to functions. They can be passed via registers, the stack, or a combination of both.

2 Return Value Handling: This defines how the function's return value is provided back to the caller, often in a specific register.

3 Stack Management: This describes the use and management of the call stack, including the order of pushing arguments and the return address.

4 Stack Clean Up: It refers to the process of restoring the call stack to its state before the function call was made. This ensures that the stack does not grow uncontrollably, which could lead to stack overflows or corrupted memory. The responsibility for stack cleanup can either fall on the caller (the function that calls another function) or the callee (the function that is being called), depending on the calling convention used.

4 Caller/Callee-Saved Registers: These indicate which registers must be preserved across function calls. Caller-saved registers are saved by the caller if needed after the call, while callee-saved registers are preserved by the function being called.

5 Name Mangling: This involves transforming function names to include additional information such as namespaces or argument types, common in languages like C++.

Different Calling Conventions:

1 cdecl (C Declaration):

• Usage: Default in C and C++.
• Arguments: Pushed onto the stack in right-to-left order.
• Stack Cleanup: Done by the caller.
• Return Values: Placed in a register (e.g., EAX in x86).

; Caller code
push 5                  ; Push second argument
push 3                  ; Push first argument
call add_numbers        ; Call the function
add esp, 8              ; Clean up the stack (2 arguments * 4 bytes)

2 stdcall (Standard Call):

• Usage: Windows API functions.
• Arguments: Pushed onto the stack in right-to-left order.
• Stack Cleanup: Done by the callee.
• Return Values: Placed in a register.

; Callee code
add_numbers:
    push ebp            ; Save base pointer
    mov ebp, esp        ; Set base pointer to current stack pointer
    mov eax, [ebp+8]    ; Load first argument into EAX
    add eax, [ebp+12]   ; Add second argument to EAX
    mov esp, ebp        ; Restore stack pointer
    pop ebp             ; Restore base pointer
    ret 8               ; Clean up 8 bytes (2 arguments * 4 bytes) and return

3 fastcall:

• Usage: Performance-critical functions.
• Arguments: First few arguments in registers, remaining on the stack.
• Stack Cleanup: Varies by implementation.

4 thiscall:

• Usage: C++ member functions.
• Arguments: this pointer passed in a register (e.g., ECX in x86).
• Stack Cleanup: Similar to cdecl or stdcall.

5 vectorcall:

• Usage: Functions with vector arguments.
• Arguments: Passed in vector registers when possible.
• Stack Cleanup: According to the calling convention.