Week 1

Agenda

• Calling conventions
• Linking
• Buffer overflow

Calling Conventions

• Arguments to the function are put on the stack (in reverse order) prior to the call of the function.

• Local variables are given space after the function is called

• Caller must preserve EAX, ECX, EDX

• Callee must preserve EBX, EDI, ESI

• Stack must be cleaned* to return

Source

x64 C Convention

• Arguments are passed in through registers, then on the stack. First argument in RDI, then RSI, RDX, RCX, R8, R9, then stack

• Locals are same as x86 (on stack after function call)

• Caller must preserve RAX, RDX, RCX, RSI, RDI, R8-11

• Callee must preserve RBX, R12-15

• Stack must be cleaned* to return

Notice the offsets to each argument are now multiples of 8 instead of 4 like in x86

Source

Other calling conventions

The type of calling convention can vary from compiler to compiler.

• Object oriented compilers like g++ need to mangle names and pass the calling object in a register (source)

• WINAPI (aka STDCALL) convention used to call Windows API. Cleans the stack of parameters passed; does not allow variable arguments (source)

• And many others, but they all share persisting notions that are recognizable!

• Calling conventions can be changed to make reversing difficult :(

Linking/API Calls

How do multiple source files become a single executable?

ELF file formats:

• Executable file
• Shared Object file
• Relocatable file
• and some others

ELF Header specifies the file format

 + Executable: specifies how to load the program into a process image (remember exec and forking?)

 + Relocatable: specifies how to include it's own code and data into an Executable or Shared object. Object files waiting to be included.

 + Shared Object: Dynamic library that links with an executable on load by a linker. Think printf, Libc, stdio.h 

How do multiple source files become a single executable?

ELF file formats:

• Executable file
• Shared Object file
• Relocatable file

Linker links objects with shared libraries.

What does the whole pipeline look like then?

1. Gcc compiles into ELF Relocatables

2. Linker links Relocatables and attaches necessary information for SO linking into an Executable

3. Loader execs the Executable, then the dynamic linker actually links to the shared objects for code execution. 

Source

If a Shared Object is a Dynamic Library, then what is a Static Library?

Static libraries are a collection of Relocatables!

Where in the pipeline should they take part?

Buffer Overflow

What is a Buffer Overflow?

What is a Buffer Overflow?

Putting more data into a buffer than the buffer intended.

What problems can this lead to?

Attacks?

If we know enough about the buffer, can we exploit this an attack the program? YES.

What damage can we achieve?

See Dave Levin's beautiful walk through on a Buffer overflow attack

Code example here