System V AMD64 Calling Convention

The System V AMD64 ABI (Application Binary Interface) is the calling convention used on Unix-like operating systems for 64-bit x86 processors (x86-64 architecture). It defines how functions are called, how parameters are passed, how return values are handled, and how the stack is managed.

Key Aspects of System V AMD64 ABI

1. Register Usage

The System V AMD64 ABI designates specific registers for passing function arguments and returning values:

Integer and Pointer Arguments:

• The first six integer or pointer arguments are passed in registers RDI, RSI, RDX, RCX, R8, and R9.
• Additional arguments are passed on the stack.

Floating Point Arguments:

• Floating-point arguments are passed in registers XMM0 through XMM7.

Return Values:

• Integer and Pointer Return Values:
  • Single-value returns (up to 64 bits) are returned in RAX.
  • For larger return values, the caller allocates space for a pointer in RDI, which points to where the return value will be stored.
• Floating Point Return Values:
  • Floating-point return values are returned in XMM0.

2. Stack Frame

Stack Alignment:

• The stack pointer (RSP) must be aligned to a 16-byte boundary at the point of function call.

Stack Management:

• The callee (the function being called) is responsible for preserving the base pointer (RBP) if it uses it.
• The callee allocates space for local variables on the stack.
• The caller is responsible for stack cleanup after the function call.

3. Function Calling

Parameter Passing:

• Integer and pointer arguments are typically passed in registers.
• Additional arguments beyond the first six are passed on the stack.

Return Address:

• The return address is automatically pushed onto the stack by the call instruction.

Stack Cleanup:

• The caller cleans up the stack after the function call by adjusting RSP to remove arguments passed on the stack.