Syscalls in linux

How to add a custom syscall to linux?

Index

What is a SysCall?

User mode vs Kernel mode

SysCall interface

strace

Adding a new custom syscall to linux

Compile and boot the kernel

What is a SysCall?

Interface to the services made available by kernel

User mode vs Kernel mode

Generally, CPU can operate in one of these modes:

  • User mode
  • Kernel mode (privileged mode)

`Cp source dest`

System Call interface

OS provides SysCalls to user programs

man syscalls

man fork

man exec

man read

...

strace

Track Signals and System Calls

  • read
  • write
  • open
  • close
  • stat
  • fstat
  • ...

x86_64

  • restart_syscall

  • exit    

  • fork  

  • read  

  • write  

  • open  

  • ...

x86

System call table

  • restart_syscall

  • exit    

  • fork  

  • read  

  • write  

  • open  

  • ...

arm

Syscall tables

x86_64: arch/x86/entry/syscalls/syscall_64.tbl

x86: arch/x86/entry/syscalls/syscall_32.tbl

arm: arch/arm/tools/syscall.tbl

...

Add a SysCall function

SYSCALL_DEFINE0(syscall_name)
{
        printk("CUSTOM SYSCALL FOR FumLug\n");
        return 0;
}

SYSCALL_DEFINE1(syscall_name, const char *, argument1)
{
        printk(argument1);
        return 0;
}

...

Update makefiles

  • obj-y in local Makefile 
    • obj-y := syscall_name.o
  • core-y in main Makefile 
    • core-y += syscall_folder/

Compiling linux

-j$(nproc)

parallel compilation considering the number of your CPU threads

  • make defconfig/menuconfig/tinyconfig/...
  • make
  • make modules (optional in this scenario)

 We'll Use 'tinyconfig' For faster kernel compilation

  • 64-bit kernel ---> yes

  • General setup ---> Initial RAM filesystem and RAM disk (initramfs/initrd) support ---> yes

  • General setup ---> Configure standard kernel features ---> Enable support for printk ---> yes

  • Executable file formats / Emulations ---> Kernel support for ELF binaries ---> yes

  • Executable file formats / Emulations ---> Kernel support for scripts starting with #! ---> yes

  • Device Drivers ---> Character devices ---> Enable TTY ---> yes

Compile options needed using `make menuconfig`:

  • Device Drivers ---> Generic Driver Options ---> Maintain a devtmpfs filesystem to mount at /dev ---> yes

  • Device Drivers ---> Generic Driver Options ---> Automount devtmpfs at /dev, after the kernel mounted the rootfs ---> yes

  • File systems ---> Pseudo filesystems ---> /proc file system support ---> yes

  • File systems ---> Pseudo filesystems ---> sysfs file system support ---> yes

Optional

Booting the kernel

-enable-kvm

Faster virtualization using Kernel-based virtual machine

-m 1g

ram size

  • Create a minimal initramfs using busybox
  • Add your binaries to the initramfs

  • Boot the kernel with:

  • qemu-system-x86_64 -kernel ./bzImage -initrd ./initramfs

initramfs

minimal initramfs using busybox

You can use this small script (written by Alireza Arzehgar)

#!/usr/bin/env bash
URL="https://www.busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-x86_64"
[ -d /tmp/root ] && rm /tmp/root/boot

mkdir -p /tmp/root/{bin,dev,etc,lib,mnt,proc,sbin,sys,tmp,var}

# Copy custom binaries
cp ./a.out /tmp/root/

# Install Busybox
cd /tmp/root || exit
wget ${URL} --no-clobber -O bin/busybox
chmod +x bin/busybox

# Create init file
echo "#!/bin/busybox sh
/bin/busybox --install -s /bin
clear
# optional (please enable support in kernel options menu if you want these filesystems...)
#mount -t devtmpfs  devtmpfs  /dev
#mount -t proc      proc      /proc
#mount -t sysfs     sysfs     /sys
#mount -t tmpfs     tmpfs     /tmp
setsid cttyhack sh
exec /bin/sh" >init
chmod +x init

# Create initramfs
find . | cpio -ov --format=newc | gzip -9 >initramfs


cd - || exit
cp /tmp/root/initramfs .

Test your syscal

#include <unistd.h>

int main(){
	syscal(SYSCALL_NUMBER);
}
(nix-shell -p gcc musl)
gcc test.c -static

1

Add your custom syscall

2

Update Makefiles

(obj-y)

(core-y)

3

compile kernel

  • make
  • make modules (optional)

4

Create a minimal initramfs using busybox

(remember to add your binaries)

6

Test your syscall

5

Boot the kernel using qemu

qemu-system-x86_64 -kernel ./bzImage -initrd ./initramfs

Credits

Email


M.Erfan Arasteh (github.com/erfanara)

Alireza Arzehgar (github.com/alirezaarzehgar)

Extra

https://kernelnewbies.org/FAQ

https://tiny.wiki.kernel.org/use_cases
https://tiny.wiki.kernel.org/faq
https://opensource.com/article/18/10/kbuild-and-kconfig
https://z49x2vmq.github.io/2020/12/24/linux-tiny-qemu/
https://gist.github.com/chrisdone/02e165a0004be33734ac2334f215380e
https://docs.kernel.org/kbuild/makefiles.html
https://medium.com/@jeremyphilemon/adding-a-quick-system-call-to-the-linux-kernel-cad55b421a7b
https://www.kernel.org/doc/html/v6.5/process/adding-syscalls.html
https://github.com/armi3/custom_syscall#3-config-the-kernel
https://www.nicktriller.com/blog/implementing-a-syscall/

Made with Slides.com