Calling Conventions

x86 C Convention (cdecl)

  • Arguments to the function are put on the stack (in reverse order) prior to the call of the function.

 

  • Local variables are given space after the function is called

 

  • Caller must preserve EAX, ECX, EDX

 

  • Callee must preserve EBX, EDI, ESI

 

  • Stack must be cleaned* to return

Source

stdcall (x86 Windows)

  • Exactly the same as cdecl except callee cleans up the stack
; int __cdecl func(int x, int y)
func:
   mov eax, [esp+4] ; x
   add eax, [esp+8] ; x += y
   ret
; int __stdcall func(int x, int y)
func:
   mov eax, [esp+4] ; x
   add eax, [esp+8] ; x += y
   ret 8

Uses special ret instruction that removes n bytes off the stack AFTER it pops EIP off the stack

Do not confuse this with C style returns, this has nothing to do with the return value

fastcall (x86 Windows)

  • Exactly the same as stdcall except first two arguments from the LEFT go into ECX and EDX
; int __fastcall func(int a, int b, int c, int d, unsigned char e)
func:
   mov eax, [esp+4] ; eax = c
   add eax, [esp+8] ; eax += d
   and eax, [esp+12]; eax &= e
   add eax, ecx ; eax += a
   add eax, edx ; eax += b
   ret 12 ; why is it 12 even though e is a char?
; int __stdcall func(int a, int b, int c, int d, unsigned char e)
func:
   mov eax, [esp+4] ; eax = a
   add eax, [esp+8] ; eax += b
   and eax, [esp+12]; eax &= c
   add eax, [esp+16]; eax += d
   add eax, [esp+20]; eax += e
   ret 20

ECX and EDX do not need to be cleaned off the stack

x64 C Convention (SYS V)

  • Arguments are passed in through registers, then on the stack. First argument in RDI, then RSI, RDX, RCX, R8, R9, then stack


  • Locals are same as x86 (on stack after function call)


  • Caller must preserve RAX, RDX, RCX, RSI, RDI, R8-11




  • Stack must be cleaned* to return

Notice the offsets to each argument are now multiples of 8 instead of 4 like in x86

Source

  • Callee must preserve RBX, RSP, RBP, R12-15

x64 C Convention (Windows)

  • Arguments are passed in through registers, then on the stack. First argument in RCX, then RDX, R8, R9, then stack
  • Formally: Windows x64 ABI 
  • Same as System V AMD64 ABI except only 4 registers used for arguments 
  • No more stdcall, fastcall, cdecl, etc.

Linker Name Mangling

  • How does Windows know which calling convention a function is using?  

_func

_func@8

@func@8

  • No mangling in x64
     
  • Names only need to be mangled for linking
    • What does this mean for us?
int __cdecl func(int x, int y)
int __stdcall func(int x, int y)
int __fastcall func(int x, int y)

Other calling conventions

The type of calling convention can vary from compiler to compiler.

  • Object oriented compilers like g++ need to mangle names and pass the calling object in a register (source)

 

  • And many others, but they all share persisting notions that are recognizable!

 

  • Calling conventions can be changed to make reversing difficult :(

x86 Calling Conventions

By Drake P

Made with Slides.com

x86 Calling Conventions

  • 280

More from Drake P