ENPM809V

Race Conditions

Final Project Structure

Individual-based
Will contain four challenges
- You will need to complete three of them
Will contain a writeup - one for each challenge
Will count for 25% of total grade!

Overview

Sources

Some of the information of the information has come from pwn.college's lecture's as this was the resource that helped me the most learning about race conditions

History of Computers

Originally was a single CPU

Initalize()

do_action_1()

do_action_2()

do_action_3()

finalize()

History of Computers

Later added multiple cores per processor
- More computation power

Initalize()

do_action_1()

do_action_2()

do_action_3()

finalize()

pInitalize()

do_action_1()

do_action_2()

do_action_3()

finalize()

History of Computers

What it actually means is this...

p2_Initalize()

p2_do_action_1()

p2_do_action_2()

p2_finalize()

p1_Initalize()

p1_do_action_1()

p1_do_action_2()

p1_finalize()

Order of Execution

p2_Initalize()

p2_do_action_1()

p2_do_action_2()

p2_finalize()

p1_Initalize()

p1_do_action_1()

p1_do_action_2()

p1_finalize()

p2_Initalize()

p2_do_action_1()

p2_do_action_2()

p2_finalize()

p1_Initalize()

p1_do_action_1()

p1_do_action_2()

p1_finalize()

p2_Initalize()

p2_do_action_1()

p2_do_action_2()

p2_finalize()

p1_Initalize()

p1_do_action_1()

p1_do_action_2()

p1_finalize()

What implications does it have?

As a result...

More cores to compute process
But they can execute in any order
- Programs can execute concurrently
- No guarantee about which process completes first
A race condition is formed!

Linux Filesystem

What is it?

The Linux Filesystem is the way that data is stored to the harddrive
Also contains information regarding processes and connected devices
Root directory is /. User directories are located in /home/<username>

What is it?

Often called Virtual Filesystem because it is an abstraction layer for FS like EXT4, FAT32, etc.
Why is it done this way?
- Kernel can support various implementation
- Easy interoperation between various file system types

What is it?

Often called Virtual Filesystem because it is an abstraction layer for FS like EXT4, FAT32, etc.
Why is it done this way?
- Kernel can support various implementation
- Easy interoperation between various file system types

Application

Write()

Read()

sys_write

sys_read

Firmware

EXT4, FAT32 ...

Hierarchy

/bin - Essential cmd line utilities
/boot - Boot loader files
/dev - Physical and Virtual Device Files
/etc - Static configuration files
/home - User home directories
/lib(64) - Library files
/media - Mount points for removable devices
/mnt - Temporarily mounted FS
/opt - Additional Software
/proc - Virtual Filesystem (we will get more into this)

Hierarchy

/root - Home directory for root user
/run - Run time variable data
/sbin - System binaries (fsck, init, route)
/srv - Served data (e.g. FTP, HTTP servers)
/sys - Information about drives, kernel
/tmp - Temporary FS (memory backed)
/usr - Multiuser binaries
/var - Variable Files

File System Implementations

ext4 - Extended File System version 4
- Used by most Linux distributions, originally developed in 1993
- Slides will be covering this file system primarily
XFS - Extended File System
- High performance 64 bit journaling FS
- Default for RedHat/CentOS
SquashFS
- Read-only Filesystem for low-memory devices
JFS
- 64 bit journaling Filesystem
- Default for AIX

Filesystem Layout

Disk

Partition

Block

Filesystem Layout

Block - basic unit of a filesystem
- Boot Block
- Superblock
- i-node table
- Data block
Partition - Where filesystems are contained
Disk - Contains multiple partitions

Filesystem Layout

Boot block
- First block in the file system
- Used to help the OS boot
Superblock
- Follows the boot block
- Stores metadata for the file system
  - Number of blocks
  - Size of Blocks
  - Size of i-node table
  - type of file system
  - etc

Filesystem Layout

I-node table
- List of index nodes
- Contains metadata about a file
Data block - well... it's exactly as it sounds

How to View Filesystem Metadata

I-nodes (Inodes)

Used to represent files and directories
Contains metadata about the file such as size, physical location, owner, group, etc.
Files are assigned an I-node number on creation
- Unique identifier to help with indexing
The number of i-nodes on a system is fixed

https://en.wikipedia.org/wiki/Inode_pointer_structure

A Very Deep Dive

https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout

A race condition is...

the ability to change the state of the program by an external context, making behavior nondeterministic.

Race Conditions in The File System

How can they be caused?

Multiple processes accessing the same file
- Reading & Writing a file at the same time
A file opened twice
Having a symlink to a file so that an access can occur twice.
How can we achieve this?

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

int main(int argc, char **argv)
{
    int fd = open(
    	argv[1], 
    	O_WRONLY | O_CREAT | O_TRUNC,
        0755);
    
    write(fd, "#!/bin/sh\necho SAFE\n", 20); 
    close(fd);
    
    execl("/bin/sh", "/bin/sh", argv[1], NULL);
    return 0; 
}

Example Race Condition

while /bin/true; do cp -v ./catflag asdf; done

wittsend2@wittsend2-virtual-machine:[~/Documents/race_condition_test]
$ ./main asdf
SAFE
wittsend2@wittsend2-virtual-machine:[~/Documents/race_condition_test]
$ ./main asdf
SAFE
wittsend2@wittsend2-virtual-machine:[~/Documents/race_condition_test]
$ ./main asdf
SAFE
wittsend2@wittsend2-virtual-machine:[~/Documents/race_condition_test]
$ ./main asdf
SAFE
wittsend2@wittsend2-virtual-machine:[~/Documents/race_condition_test]
$ ./main asdf
FLAG

Source pwn.college

#!/bin/sh
# catflag script
cat /flag

The first classwork is an example of this.

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

int main(int argc, char **argv)
{
    int echo_fd = open("/bin/echo", O_RDONLY);

    int fd = open(
    	argv[1],
    	O_WRONLY | O_CREAT,
        0755);

    sendfile(fd, echo_fd, 0, 1024*1024);
    close(fd);

    execl(argv[1], argv[1], "SAFE", NULL);
    return 0;
}

Less Winnable Race Condition

Source pwn.college

Much smaller window of opportunity to win!
- Copying echo as our payload
- Close the file descriptor
- Then executing "echo safe"
Why?
- Less initialization is happening (previous example had to initialize sh)

Time of Checck to Time of Use (TOCTOU)

A race condition where between checking if a file exists and the usage, the state of the file changes.
- There is a gap between when checking the file condition vs. utilizing the file.
How it works:
- A condition where it checks a value occurs
- In another thread/context, the value gets modified.
- The program continues with new value

Time of Checck to Time of Use (TOCTOU)

// Check if file is safe to write
if (access("/tmp/myfile", W_OK) == 0) {
    // Time gap here - attacker can act!
    FILE *f = fopen("/tmp/myfile", "w");
    fprintf(f, "sensitive data");
    fclose(f);
}

Strategies for Being More Winnable

Scheduling Priorities:

System Calls like Nice can slow programs down by lowering scheduling class and priority
- Check man nice and man ionice to see how it works

Path Complexity:

Shorter paths tend to be faster than longer paths (with many directories)
Kernel needs to process each directory
Symbolic Links can increase that complexity too
Example:
- cat a/b/c/d/e/f/g/h/i/1/2/3/4/5/6/7/8/9/target_file is slower than cat target_file (remember, max path size is 4096)

Strategies for Being More Winnable

if (access("file", W_OK) != 0) {
    exit(1);
}

fd = open("file", O_WRONLY);
write(fd, buffer, sizeof(buffer));

What is wrong here?

Strategies for Being More Winnable

if (access("file", W_OK) != 0) {
    exit(1);
}

fd = open("file", O_WRONLY);
write(fd, buffer, sizeof(buffer));

symlink("/etc/passwd", "file");

Victim

Attacker

Preventing Filesystem Race conditions

flock - a library call and utility to lock files.

Before We Continue...

Lets Learn about Processes and Threads!

What's a Process?

It is a currently running program
They are in complete isolation of each other
- Their own Virtual Memory
- Registers
- File Descriptors
- Process IDs
- Security Properties

What's a Thread?

A thread is a single sequential flow of control within a program
Threads Share Virtual Memory (excluding the stack) and File Descriptors
They are independent in every other way.

Every process has atleast one thread

Various Implementations

Threads:

System V
Posix - This is what we will focus on today

Example

void *threadFunc(int arg)
{
	printf("Thread %d, PID %d, TID %d, UID %d\n", arg, getpid(), gettid(), getuid());
}

void main()
{
	pthread_t thread1, thread2;
    pthread_create(&thread1, NULL, threadFunc, 1);
    pthread_create(&thread1, NULL, threadFunc, 2);
    printf("Main thread PID %d, TID %d, UID %d", getpid(), gettid(), getuid());
    pthread_join(thread1, NULL); 
    pthread_join(thread2, NULL); 
}

Underneath the hood

clone(
  child_stack = 0x7f81afdc7fb0,
  flags = CLONE_VM|		// parent and child will run in the same memory space
          CLONE_FS|		// parent and child will share filesystem info (chroot(), chdir(), and similar effects will be shared between parent and child)
          CLONE_FILES|	// parent and child will share file descriptors
          CLONE_SIGHAND|	// parent and child will share signal handlers
          CLONE_THREAD|	// signify that the child is a fellow thread of the parent
          CLONE_SYSVSEM|	// parent and child will share semaphore information
          CLONE_SETTLS|	// set a unique "Thread Local Storage" area for the new thread
          CLONE_PARENT_SETTID|	// store the new thread ID at the memory location pointed to by the tid arg below
          CLONE_CHILD_CLEARTID,	// zero out the thread ID at the location pointed to by the tid arg below when the child exits
  parent_tid=[1926535],
  tls=0x7f81afdc8700,
  child_tidptr=0x7f81afdc89d0
)

pthread_create uses the clone system call to create its thread.
- What it's actually doing is creating a child_process that shares the same memory space as the parent process.

Some things to note

Although libc functions (like libpthread) and the system call interface are generally similar, there are a couple of discrepancies
- setuid - the libc syscall wrapper will set the UID of all thread; however the system call itself only sets UID of the caller
- exit - The libc wrapper for exit() calls exit_group() meaning it will terminate all threads. The exit() system call only terminates the caller thread.

Some things to note

Threads run until they are finished executing!
The only way to exit a thread early is to call the pthread_kill function (not recommended).
Some common ways of handling this:
- Global variables (though this could be unsafe if not careful).
- Using signal handlers
- Calling exit() (though better to let threads terminate before calling it).

Races in Threads and Memory

How are memory race conditions possible?

Memory is a shared resource between processes and threads
- Shared memory API (POSIX and System V)
- Passing pointers around
  - Global variable that is a pointer to some location in memory (must NOT BE STACK)

How are thread race conditions possible

Developer Mistakes
Threads accessing memory unsafely
Kernel Space not checking memory correctly
Not utilizing locking mechanisms correctly

How Can They Happen

unsigned int size = 128;
void read_data() 
{
    char buffer[16];
    if (buffer < 16) 
    {
    	printf("Valid size!");
        read(0, buffer, 16); 
        printf("%s\n", buffer); 
    }
}

void *start_thread()
{
    while (true)
    {
    	read_data();
    }
}

void main()
{
    pthread_t thread; 
    pthread_create(&thread, NULL, thread_allocator, 0);
    
    while(size != 0)
    {
    	read(0, &size, 1);
    }
    
 	return; 
}

#define SECRET "123456" 

bool canAccess = false;
char readData[64]; 
void *accessSensitiveInfo()
{
	char password[64];
    printf("Enter password: ");
    scanf("%s", buffer); 
    if (password == SECRET)
    {
    	canAccess = true;
    }
	if (canAccess) 
    {
    	int fd = open("/path/to/sensitiveFile", O_RDONLY);
        read(fd, readData, 64);
        close(fd);
    }
}

void main()
{
	pthread_t user1, user2, user3; 
    pthread_create(&user1, NULL, accessSensitiveInfo, 0);
    pthread_create(&user2, NULL, accessSensitiveInfo, 0);
    pthread_create(&user3, NULL, accessSensitiveInfo, 0);
    pthread_join(&user1, NULL);
    pthread_join(&user2, NULL);
    pthread_join(&user3, NULL);
    exit(0); 
}

How can They Happen

int check_safety(char *user_buffer, int maximum_size) {
    int size;
    copy_from_user(&size, user_buffer, sizeof(size));
    return size <= maximum_size;
}

static long device_ioctl(struct file *file, unsigned int cmd, unsigned long user_buffer) {
    int size;
    char buffer[16];
    if (!check_safety(user_buffer, 16)) return;
    copy_from_user(&size, user_buffer, sizeof(size));
    copy_from_user(buffer, user_buffer+sizeof(size), size);
}

copy_from_user and copy_to_user are functions within kernel space to copy data from/to the user from/to kernel space.

How can They Happen

int check_safety(char *user_buffer, int maximum_size) {
    int size;
    copy_from_user(&size, user_buffer, sizeof(size));
    return size <= maximum_size;
}

static long device_ioctl(struct file *file, unsigned int cmd, unsigned long user_buffer) {
    int size;
    char buffer[16];
    if (!check_safety(user_buffer, 16)) return;
    copy_from_user(&size, user_buffer, sizeof(size));
    copy_from_user(buffer, user_buffer+sizeof(size), size);
}

Example of Time of Check to Time of Use race condition in the Linux kernel

How to prevent them

#include <stdio.h>
#include <pthread.h>

unsigned int n = 0; 
void *thread_function()
{
    while(1)
    {
        n++; 
        n--;
        printf("Num = %d\n", n);
    }
    
}

int main()
{
    pthread_t thread1;
    pthread_t thread2;
    pthread_t thread3;
    
    pthread_create(&thread1, NULL, thread_function, 0);
    pthread_create(&thread2, NULL, thread_function, 0);
    pthread_create(&thread3, NULL, thread_function, 0); 

    getchar();    
    return 0; 
}

Without Mutex

How to prevent them

unsigned int n = 0; 
pthread_mutex_t mutex l;

void *thread_function()
{
    while(1)
    {
        pthread_mutex_lock(&l);
        n++; 
        n--;
        printf("Num = %d\n", n);
        pthread_mutext_unlock(&l); 
    }
}

int main()
{
    pthread_t thread1;
    pthread_t thread2;
    pthread_t thread3;
    
    pthread_create(&thread1, NULL, thread_function, 0);
    pthread_create(&thread2, NULL, thread_function, 0);
    pthread_create(&thread3, NULL, thread_function, 0); 

    getchar();    
    return 0; 
}

With Mutex

#include <stdio.h>
#include <pthread.h>
#include <semaphore.h>
#include <unistd.h>

sem_t mutex;

void* thread(void* arg)
{
    //wait
    sem_wait(&mutex);
    printf("\nEntered..\n");

    //critical section
    sleep(4);
    
    //signal
    printf("\nJust Exiting...\n");
    sem_post(&mutex);
}


int main()
{
    sem_init(&mutex, 0, 1);
    pthread_t t1,t2;
    pthread_create(&t1,NULL,thread,NULL);
    sleep(2);
    pthread_create(&t2,NULL,thread,NULL);
    pthread_join(t1,NULL);
    pthread_join(t2,NULL);
    sem_destroy(&mutex);
    return 0;
}

With Semaphores

Tools To Help

Valgrind - keeps track of memory and ensures proper access
- Need to have proper test cases to trigger it
- Also good for checking for memory leaks
Clang thread-sanitizer
- Built in thread sanitizer for debugging
- Utilize -fsanitize-thread when compiling with clang

Races in Signals and Reentrancy

What are signals?

Signal is a kind of interrupt used to announce asynchronous events that happen on the system.
- Think of things like when you press ctrl+c to send a SIGINT
- There is a limited list of possible signals that we do not create on our own.

Handling Signals

Processes can register signal handlers to perform an action when a signal is sent to it.
What happens? Signal pauses process execution and calls into signal handler. After completion:
- Process resumeds
- Process terminates via an exit() call.
Global Variables are the mechanism for signals to send data back to the process
You can send Any Sginal to Any Process that has the same ruid as you (even if their eUID is root)

Triggering Signals

Time Slices: the kernel gives a process a limited time to run before it hijacks the process to see if signals are thrown (or execute another process's code).
- It actually hijacks the CPU from your process at the end of the time slice
- If a system call is made near the end of your time slice, the kernel might just confiscate it early.
Signal checking: the kernel will check for recieve signals to your process. It then triggers the signal handler
- Then Schedules your process for execution (after a system call or the beginning of the next time slice

This means any program can suddenly and unexpectedly divert execution to the signal handler!

Some Crazy Signal Handlers

bool adminUser = false;
void signal_handler(int signum) {
    	adminUser = true;
}

int main() {
    signal(SIGUSR1, signal_handler);
    while (1) {
		if (adminUser) printf("Key is abcd1234\n");
	}
}

Some Crazy Signal Handlers

int hold;
void swap(int *x, int *y)
{
	hold = *x; 
    *x = *y;
    *y = tmp;
}
int call_swap()
{
	int x = 1; y = 2;
    swap(x, y);
}
int main()
{
	signal(SIGUSER1, call_swap);
    call_swap();
}

Reentrant function are functions that would operate properly even when interrupted with an instance of themselves.

Are any of these functions reentrant?

No! All of these functions call swap(), which is a non-reentrant function. The signal handler is call_swap too!

Some Crazy Signal Handlers

void swap(int *x, int *y)
{
	int hold = *x; 
    *x = *y;
    *y = tmp;
}
int call_swap()
{
	int x = 1; y = 2;
    swap(x, y);
}
int main()
{
	signal(SIGUSER1, call_swap);
    call_swap();
}

Reentrant function are functions that would operate properly even when interrupted with an instance of themselves.

Are any of these functions reentrant?

Yes they are! All variables are non-global and no global variables are modified by the signal handler

How do we call signals?

Utilizing the kill() system call...
- Yes I know, this is not intuitive
- h ttps://www-uxsup.csx.cam.ac.uk/courses/moved.Building/signals.pdf - list of signals
Look at man kill and man signal to see more

Preventing Race Conditions via Signals

Do not call non-reentrant functions in your signal handler
- They might have been internrupted mid-execution when sending your signal
- Other signals can interrupt the signal handler

Note: malloc and free are non reentrantn

man signal-safety

Takeaways

Protect your code!

Good coding practices will go a long way. if you have asynchronous code, make sure it does it intended behavior.
Never assume asynchronous code will be executed in the same order.
- If it needs the same result every time, make sure you are using memory, locking, and re entrant functions properly.

ENPM809V

Final Project Structure

Overview

Sources

History of Computers

History of Computers

History of Computers

Order of Execution

What implications does it have?

As a result...

Linux Filesystem

What is it?

What is it?

What is it?

Hierarchy

Hierarchy

File System Implementations

Filesystem Layout

Filesystem Layout

Filesystem Layout

Filesystem Layout

How to View Filesystem Metadata

I-nodes (Inodes)

A Very Deep Dive

A race condition is...

the ability to change the state of the program by an external context, making behavior nondeterministic.

Race Conditions in The File System

How can they be caused?

Example Race Condition

Less Winnable Race Condition

Time of Checck to Time of Use (TOCTOU)

Time of Checck to Time of Use (TOCTOU)

Strategies for Being More Winnable

Strategies for Being More Winnable

Strategies for Being More Winnable

Preventing Filesystem Race conditions

Before We Continue...

Lets Learn about Processes and Threads!

What's a Process?

What's a Thread?

Various Implementations

Example

Underneath the hood

Some things to note

Some things to note

Races in Threads and Memory

How are memory race conditions possible?

How are thread race conditions possible

How Can They Happen

How can They Happen

How can They Happen

How to prevent them

How to prevent them

Tools To Help

Races in Signals and Reentrancy

What are signals?

Handling Signals

Triggering Signals

This means any program can suddenly and unexpectedly divert execution to the signal handler!

Some Crazy Signal Handlers

Some Crazy Signal Handlers

Some Crazy Signal Handlers

How do we call signals?

Preventing Race Conditions via Signals

Takeaways

Protect your code!

Homework

Exploit a Threaded Race Condition

ENPM809V - Race Conditions

More from Ragnar Security