Week 7 System Calls, Kernel Threads, Kernel Debugging

1
Week 7 System Calls, Kernel Threads, Kernel
Debugging

• Sarah Diesburg
• Florida State University

2
First

• Any questions on
• Part 1 5 system calls
• Part 2 xtime proc module

3
Story of Kernel Development

• Some context

4
In the old days

• There were no modules or virtual machines
• The kernel is a program
• Has code, can compile, re-compile, make
  executable
• When changes needed to be made, developers make
  changes in source and re-compile

5
How is the kernel different from a regular
program?

• Mostly in how it is executed
• Boot loader loads the kernel image/executable
  during boot time
• Sets kernel mode
• Jumps to the entry point in the image/executable
• Remember the generic booting sequence?

6
Quick Question

• How would you make changes to the kernel and run
  those changes?
• Make changes to the source
• Re-complie the kernel source
• Re-install the kernel source
• Make sure the bootloader sees the new kernel
  image (grub)
• Reboot and profit!

7
Getting more modern..

• Modules were created as bits of code that can be
  loaded and unloaded by the kernel in kernel mode
• Made development easier
• Instead of re-compiling, re-installing, and
  rebooting into the new kernel, one could just
  re-compile and load a module

8
Quick Question

• How would you make changes to a module and run
  those changes?
• Make changes to module source code
• Re-compile the module
• Load the new module

9
Present Day

• Reboots into new kernels and loading new modules
  often freezes machines
• Enter virtual machine software
• Process that emulates the hardware necessary to
  run an OS in user-space
• Guest OS is executed inside the virtual machine
  process!

10
New System Calls

• Fun but tricky!

11
Implementing System Calls
int start_elevator(void) int issue_request(int
1, int 2, int 3) int
stop_elevator(void)

• Need to implement the functions above. But how?

12
Adding a System Call to Kernel

• Files to add
• LINUX_DIR/PROJECT_NAME/Makefile
• LINUX_DIR/PROJECT_NAME/PROJECT_NAME.c
• LINUX_DIR/PROJECT_NAME/NEW_SYSCALLS.c
• Files to modify
• LINUX_DIR/arch/x86/kernel/syscall_table_32.S
• LINUX_DIR/include/asm-generic/unistd.h
• LINUX_DIR/include/linux/syscalls.h
• LINUX_DIR/Makefile

13
Sample System Call

• Lets add a sample module to the kernel that
  defines a sample system call
• test_newsyscall(int test_int)
• Takes an int test_int and issues a printk on it
• Returns test_int
• Problem If our module isnt loaded, what
  happens if we call our sample system call?

14
Project 2 System Call Model
Elevator module
Core kernel
User program
15
Project 2 System Call Model
Elevator module
Core kernel
User issues system call, core kernel looks up
system call in system call table
User program
16
Project 2 System Call Model
Elevator module performs system call action
Elevator module
Core kernel
User program
17
Project 2 System Call Model
Elevator module returns result of system call
Elevator module
Core kernel
User program
18
Project 2 System Call Model
Elevator module
Core kernel
Core kernel forwards result of system call to
user program
User program
19
What happens if elevator module is not loaded?
Core kernel
User program
20
What happens if elevator module is not loaded?
Core kernel
User issues system call, core kernel looks up
system call in system call table
User program
21
What happens if elevator module is not loaded?
Elevator module is not loaded to perform the
action. OOPS!
Core kernel
User program
22
Module System Calls

• We must create a wrapper system call!
• Wrapper will call module function if module
  loaded, else returns an error
• Must be created in a separate, built-in kernel
  file in the project folder

23
Function Pointers

• We will implement our system call wrapper with a
  function pointer
• Pointer to a function
• Function pointer can point to any function that
  you implement that
• Takes the same input variable types
• Returns the same return type

24
Function Pointers

• long (STUB_test_newsyscall)(int test_int)
  NULL
• Function pointer that
• Returns a long
• Name is STUB_test_newsyscall
• Takes parameter int test_int
• Function pointer set to NULL
• Can set function pointer to a local function you
  implement

25
Elevator Project

• Create a file in your elevator project that just
  contains the system call information
• KERNEL_DIR/PROJECT_DIR/newsyscalls.c

26
KERNEL_DIR/PROJECT_DIR/newsyscalls.c

• include ltlinux/linkage.hgt
• include ltlinux/kernel.hgt
• include ltlinux/module.hgt
• / System call stub. We initialize the stub
  function to be NULL. /
• long (STUB_test_newsyscall)(int test_int)
  NULL
• EXPORT_SYMBOL(STUB_test_newsyscall)
• / System call wrapper. If the stub is not NULL,
  it will be run, otherwise returns -ENOSYS /
• asmlinkage long sys_test_newsyscall(int test_int)
• if (STUB_test_newsyscall)
• return STUB_test_newsyscall(test_int)
• else
• return -ENOSYS

27
KERNEL_DIR/PROJECT_DIR/newsyscalls.c

• include ltlinux/linkage.hgt
• include ltlinux/kernel.hgt
• include ltlinux/module.hgt
• / System call stub. We initialize the stub
  function to be NULL. /
• long (STUB_test_newsyscall)(int test_int)
  NULL
• EXPORT_SYMBOL(STUB_test_newsyscall)
• / System call wrapper. If the stub is not NULL,
  it will be run, otherwise returns -ENOSYS /
• asmlinkage long sys_test_newsyscall(int test_int)
• if (STUB_test_newsyscall)
• return STUB_test_newsyscall(test_int)
• else
• return -ENOSYS

Function pointer
28
KERNEL_DIR/PROJECT_DIR/newsyscalls.c

• include ltlinux/linkage.hgt
• include ltlinux/kernel.hgt
• include ltlinux/module.hgt
• / System call stub. We initialize the stub
  function to be NULL. /
• long (STUB_test_newsyscall)(int test_int)
  NULL
• EXPORT_SYMBOL(STUB_test_newsyscall)
• / System call wrapper. If the stub is not NULL,
  it will be run, otherwise returns -ENOSYS /
• asmlinkage long sys_test_newsyscall(int test_int)
• if (STUB_test_newsyscall)
• return STUB_test_newsyscall(test_int)
• else
• return -ENOSYS

Export the pointer so we can access it later
29
KERNEL_DIR/PROJECT_DIR/newsyscalls.c

• include ltlinux/linkage.hgt
• include ltlinux/kernel.hgt
• include ltlinux/module.hgt
• / System call stub. We initialize the stub
  function to be NULL. /
• long (STUB_test_newsyscall)(int test_int)
  NULL
• EXPORT_SYMBOL(STUB_test_newsyscall)
• / System call wrapper. If the stub is not NULL,
  it will be run, otherwise returns -ENOSYS /
• asmlinkage long sys_test_newsyscall(int test_int)
• if (STUB_test_newsyscall)
• return STUB_test_newsyscall(test_int)
• else
• return -ENOSYS

System call wrapper
30
Elevator Project

• Next create a separate file that
• Holds your module code
• Registers the system call pointer
• Actually implements the system call behavior

31
Inside KERNEL_DIR/PROJECT_DIR/PROJECT_NAME.C

• / Extern system call stub declarations /
• extern long (STUB_test_newsyscall)(int
  test_int)
• long my_test_newsyscall(int test)
• printk("s Your int is i\n",
  __FUNCTION__, test)
• return test
• my_module_init()
• STUB_test_newsyscall(my_test_newsyscall)
• return 0
• my_module_exit()
• STUB_test_newsyscallNULL

32
Inside KERNEL_DIR/PROJECT_DIR/PROJECT_NAME.C

• / Extern system call stub declarations /
• extern long (STUB_test_newsyscall)(int
  test_int)
• long my_test_newsyscall(int test)
• printk("s Your int is i\n",
  __FUNCTION__, test)
• return test
• my_module_init()
• STUB_test_newsyscall(my_test_newsyscall)
• return 0
• my_module_exit()
• STUB_test_newsyscallNULL

Gain access to stub function pointer.
33
Inside KERNEL_DIR/PROJECT_DIR/PROJECT_NAME.C

• / Extern system call stub declarations /
• extern long (STUB_test_newsyscall)(int
  test_int)
• long my_test_newsyscall(int test)
• printk("s Your int is i\n",
  __FUNCTION__, test)
• return test
• my_module_init()
• STUB_test_newsyscall(my_test_newsyscall)
• return 0
• my_module_exit()
• STUB_test_newsyscallNULL

Local function that implements syscall
34
Inside KERNEL_DIR/PROJECT_DIR/PROJECT_NAME.C

• / Extern system call stub declarations /
• extern long (STUB_test_newsyscall)(int
  test_int)
• long my_test_newsyscall(int test)
• printk("s Your int is i\n",
  __FUNCTION__, test)
• return test
• my_module_init()
• STUB_test_newsyscall(my_test_newsyscall)
• return 0
• my_module_exit()
• STUB_test_newsyscallNULL

Set stub function pointer to local function in
init
35
Inside KERNEL_DIR/PROJECT_DIR/PROJECT_NAME.C

• / Extern system call stub declarations /
• extern long (STUB_test_newsyscall)(int
  test_int)
• long my_test_newsyscall(int test)
• printk("s Your int is i\n",
  __FUNCTION__, test)
• return test
• my_module_init()
• STUB_test_newsyscall(my_test_newsyscall)
• return 0
• my_module_exit()
• STUB_test_newsyscallNULL

Reset stub function pointer to NULL on module
unload
36
KERNEL_DIR/PROJECT_DIR/Makefile

• obj-m my_module.o
• obj-y newsyscalls.o
• KDIR /lib/modules/2.6.32/build
• PWD (shell pwd)
• default
• (MAKE) -C (KDIR) SUBDIRS(PWD) modules

37
KERNEL_DIR/PROJECT_DIR/Makefile
Compile as a module

• obj-m my_module.o
• obj-y newsyscalls.o
• KDIR /lib/modules/2.6.32/build
• PWD (shell pwd)
• default
• (MAKE) -C (KDIR) SUBDIRS(PWD) modules

38
KERNEL_DIR/PROJECT_DIR/Makefile

• obj-m my_module.o
• obj-y newsyscalls.o
• KDIR /lib/modules/2.6.32/build
• PWD (shell pwd)
• default
• (MAKE) -C (KDIR) SUBDIRS(PWD) modules

Compile as kernel built-in
39
Core Kernel Additions

• Add the new system call to the core kernel system
  call table
• Modify three files
• Add the project directory to the main Makefile
• Modify one file

40
Modifying syscall_table_32.S

• .long sys_preadv
• .long sys_pwritev
• .long sys_rt_tgsigqueueinfo / 335 /
• .long sys_perf_event_open

41
Modifying syscall_table_32.S

• .long sys_preadv
• .long sys_pwritev
• .long sys_rt_tgsigqueueinfo / 335 /
• .long sys_perf_event_open
• .long sys_test_newsyscall / 337 /

• Add new system call to the end of the file.
• Remember the number youll need it in
  userspace!

42
Modifying unistd.h

• / midfile /
• define __NR_perf_event_open 241
• __SYSCALL(__NR_perf_event_open,
  sys_perf_event_open)
• undef __NR_syscalls
• define __NR_syscalls 242
• / midfile /

• Can be found around line 623

43
Modifying unistd.h

• / midfile /
• define __NR_perf_event_open 241
• __SYSCALL(__NR_perf_event_open,
  sys_perf_event_open)
• define __NR_test_newsyscall 242
• __SYSCALL(__NR_test_newsyscall,
  sys_test_new_syscall)
• undef __NR_syscalls
• define __NR_syscalls 242
• / midfile /

44
Modifying unistd.h

• / midfile /
• define __NR_perf_event_open 241
• __SYSCALL(__NR_perf_event_open,
  sys_perf_event_open)
• define __NR_test_newsyscall 242
• __SYSCALL(__NR_test_newsyscall,
  sys_test_new_syscall)
• undef __NR_syscalls
• define __NR_syscalls 243
• / midfile /

45
Modifying syscalls.h

• asmlinkage long sys_perf_event_open(
• struct perf_event_attr __user
  attr_uptr,
• pid_t pid, int cpu, int group_fd,
  unsigned
• long flags)
• endif
• / EOF /

46
Modifying syscalls.h

• asmlinkage long sys_perf_event_open(
• struct perf_event_attr __user
  attr_uptr,
• pid_t pid, int cpu, int group_fd,
  unsigned
• long flags)
• asmlinkage long sys_test_newsyscall(int
  test_int)
• endif
• / EOF /

47
Modifying KERNEL_DIR/Makefile

• Objects we will link into vmlinux / subdirs we
  need to visit
• init-y init/
• drivers-y drivers/ sound/ firmware/
• net-y net/
• libs-y lib/
• core-y usr/
• endif KBUILD_EXTMOD

48
Modifying KERNEL_DIR/Makefile

• Objects we will link into vmlinux / subdirs we
  need to visit
• init-y init/
• drivers-y drivers/ sound/ firmware/
• net-y net/
• libs-y lib/
• core-y usr/ my_module/
• endif KBUILD_EXTMOD

• Found around line 475
• Can replace my_module with the name of your
  PROJECT_DIR

49
Getting it all to work

1. Re-compile the kernel
2. Install modules, install kernel
3. Make new initramfs image
4. Reboot
5. Test with a user-space program

50
Sample User-space Program

• include ltstdio.hgt
• include ltstdlib.hgt
• include ltsys/syscall.hgt
• include ltlinux/unistd.hgt
• define __SYS_TEST_ELEVATOR 337
• int main()
• int test5
• long ret
• retsyscall(__SYS_TEST_ELEVATOR, test)
• if(retlt0)
• perror("system call error")
• else
• printf("Function successful,
  returned i\n", ret)
• return 0

51
syscall()

• int syscall(int number, ...)
• Performs the system call based on the system
  calls number
• Number can be found in the syscall_table_32.S
  file (our example was 337)

52
User-space Program Output

• Output when my_module not loaded
• system call error Function not implemented
• Output when my_module loaded
• Function successful, returned 5

53
Kthreads

• Run the main logic of your module in a kthread!

54
Refresher hello.c

• include ltlinux/init.hgt
• include ltlinux/module.hgt
• MODULE_LICENSE(Dual BSD/GPL)
• static int hello_init(void)
• printk(KERN_ALERT Hello, world!\n)
• return 0
• static void hello_exit(void)
• printk(KERN_ALERT Goodbye, sleepy world.\n)
• module_init(hello_init)
• module_exit(hello_exit)

55
Kernel Modules

• Remember, kernel modules are very event-based
• We need a way to start an independent thread of
  execution in response to an event
• e.g. start_elevator() for project 2

56
kthread_run

• kthread_run(threadfn, data, namefmt, ...)
• Creates a new thread and tells it to run
• threadfn the name of the function the thread
  should run
• data data pointer for threadfn (can be NULL if
  the function does not take any args)
• namefmt name of the thread (displayed during
  ps command)
• Returns a task_struct

57
kthread_run example

• struct task_struct t
• t kthread_run(run, NULL, my_elevator")
• if (IS_ERR(t))
• retPTR_ERR(t)

58
kthread_stop

• int kthread_stop(struct task_struct k)
• Sets kthread_should_stop for k to return true,
  wakes the thread, and waits for the thread to
  exit
• Returns the result of the thread function

59
kthread_stop_example

• retkthread_stop(t)
• if(ret ! -EINTR)
• printk("Main logic tread stopped.\n)

60
Thread Function Example

• static int run(void arg)
• / Lock here /
• while(!kthread_should_stop())
• / Do stuff /
• / Unlock here /
• schedule()
• / Lock here /
• / Unlock here /
• printk("s kernel thread exits.\n",
  __FUNCTION__)
• return 0

61
Thread Function Example

• static int run(void arg)
• / Lock here /
• while(!kthread_should_stop())
• / Do stuff /
• / Unlock here /
• schedule()
• / Lock here /
• / Unlock here /
• printk("s kernel thread exits.\n",
  __FUNCTION__)
• return 0

schedule() is very important here. Why?
62
Inefficient Solution

• Thread will continue to run even though it has
  nothing to do
• Eats up resources
• Investigate the kthread interface to find ways to
• Put thread to sleep
• Wake up thread
• There is more than one way to do this

63
Debugging
64
Kernel Debugging Configurations

• Timing info on printks
• __depreciated logic
• Detection of hung tasks
• SLUB debugging
• Kernel memory leak detector
• Mutex/lock debugging
• Kmemcheck
• Check for stack overflow
• Linked list debugging

65
Select Kernel Hacking
66
Enable Debugging Options
67
Debugging through procfs

• Necessary for elevator project!
• General process
• Identify data to monitor in your module
• Create a proc entry to monitor this data
• Run your module
• Query /proc/ltentrygt for that information at any
  time

68
Kernel Oops and Other Errors

• Kernel errors often only appear on first tty
  (terminal interface)
• Why?
• How can I see my first tty?
• On regular system CTRLALTF1
• CTRLALTF7 to go back to X screen
• On VMware CTRLALTSPACEF1
• CTRLALTSPACEF7 to go back to X screen

69
Oops!
70
Reason for failure
71
Current drivers
72
Call Trace
73
Call Trace
74
Failed command
75
Defensive Programming

• Infinite loops and deadlocks at the kernel level
  hang your machine
• Ctrl-Alt-Del has NO effect
• Ctrl-C does not matter
• Ctrl-D does not matter
• You may only reboot
• How do you protect yourself?
• Use schedule() strategically
• Use preemptable versions of functions

76
Debugging Tools not Covered

• LTT Linux Tracing Framework
• gdb Invoking gbd on the kernel image
• kgdb A remote debugger for the kernel
• Magic SysRq
• printk Rate limiting, turning on/off

77
Next Time

• Locks
• Linked lists
• Elevator algorithms