Achieving hard RealTime with General Purpose Operating Systems

1
Achieving (hard) Real-Time with General Purpose
Operating Systems
2
Motivation

• There is a conflict between the requirements of
  real-time systems and the need to run common day
  applications.
• Can we use a general purpose OS for real-time
  applications?

3
Outline

• Basic definitions
• The challenges
• Can we use existing systems
• Proposed solutions
• Case studies of real-time oss.

4
Basic definitions

• A task is the basic unit of execution. Each task
  has three important properties
• 1. release time the point in time from which
  the task can be executed.
• 2. deadline the point in time by which the task
  must complete.
• 3. execution time the time the task takes to
  execute
• E.g. brakes controller

5
example

• If we have the following set of tasks
• task1 release0 deadline3 exec time2
• task2 release2 deadline5 exec time1
• task3 release6 deadline10 exec time3
• Then they must be scheduled during these periods

6
Basic definitions

• A processor is a concept used to denote a
  resource that is active in the execution of a
  task
• Usually refers to the CPU.

7
Classification of task timing constraints

• There are two types of timing constraints hard
  and soft.
• Definitions
• if missing a deadline is a fatal error or has
  disastrous consequences it is considered hard
  (missing an occasional deadline is not
  acceptable)
• otherwise (if some percentage of missed deadlines
  is acceptable) it is soft.
• Classic examples
• braking systems controller hard
• multimedia streaming soft

8
Soft real time systems

• Contain only tasks with no hard timing
  constraints.
• Also known as best effort systems
• Most modern operating systems can serve as the
  base for a soft real time systems.
• Examples
• multimedia transmission and reception,
  networking, telecom (cellular) networks,
• web sites and services, computer games.

9
Hard real time systems

• Contains tasks with hard timing constraints.
• Requires formal verification/guarantees of being
  to always meet its hard deadlines (except for
  fatal errors).
• Examples
• air traffic control , vehicle subsystems
  control, medical systems

10
Classifications of real time systems

• The type of system can be either a uni-processor
  , mp or distributed system.
• Moving to an mp or distributed real time systems
  adds lots of difficulty by requiring real time
  communication and synchronization mechanisms
  between the processors.

11
Classifications of real time systems

• There are two different execution models
• In a preemptive model of execution a task may be
  interrupted (preempted) during its execution and
  another task run in its place.
• In a non-preemptive model of execution after a
  task that starts executing no other task may
  execute until this task concludes or yields the
  CPU.

12
Classifications of real time systems

• The task model for a real time system has two
  main types
• 1. purely periodic every task is cyclic and
  executes periodically, even IO is polled. The
  nature of tasks doesnt vary much between cycles.
  
• 2. aperiodic/asynchronous most tasks in the
  system arent periodic.

13
Classifications of real time systems

• A static system is a system where all tasks are
  knows at design time including their release
  times (full apriory knowledge).
• A dynamic systems is a system that can
  dynamically create and destroy tasks at runtime
  (No full apriory knowledge).

14
Classifications of real time systems

• In many real time systems the tasks have
  different priorities.
• There are two possible models
• static-priorities the priorities of tasks dont
  change during execution
• dynamic-priorities the priority of tasks may
  change during execution

15
Classifications of real time systems

• Sets of tasks are classified according to the way
  they share resources
• We will call a task set dependent if the tasks in
  it share resources.
• We will call a task set independent if they do
  not.

16
Outline

• Basic definitions
• The challenges
• Can we use existing systems
• Proposed solutions
• Case studies of real-time oss.

17
The main challenges

• Processor sharing scheduling
• Resource sharing
• Handling external input / communication

18
Scheduling

• a scheduler needs to decide when to run each task
  in order to meet the deadlines imposed on the
  tasks.
• In order to do this the scheduler needs to base
  its operation upon the properties of the tasks
  (release times, deadlines ,etc..).

19
Scheduling

• If scheduling is done online, the overhead of
  scheduling needs to be reasonable (this can be
  hard because most of the optimal scheduling
  algorithms are np-hard, even for a single
  processor).
• If the task model is dynamic we need the
  scheduler to decide if adding some new task would
  prevent it from filling obligations to currently
  running tasks (admissibility testing).

20
Resource sharing

• When tasks share resources they become dependant
  on one another and this limits our flexibility
  when scheduling, requiring us to use
  synchronization mechanisms to protect critical
  sections in the tasks.
• This raises the issues of deadlocks, process
  starvation and priority inversion.

21
Handling external input / communication

• In real time system there will often be
  constraints on the minimal time until the system
  processes input/communication.
• These constraints must be reflected in the
  scheduling of tasks, the handling of external
  interrupts and the communication infrastructure
  and protocols chosen.

22
Outline

• Basic definitions
• The challenges
• Deficiencies in modern operating systems
• Proposed solutions
• Case studies of real-time oss.

23
Why most modern OSs are unsuitable

• The basic guiding concept for most operating
  systems and schedulers is to strive for good
  average performance while in real time systems
  meeting the deadlines is far more important than
  average performance.
• many modern operating features are problematic in
  a hard real time setting.

24
Problematic modern features

• Protected memory while important for the
  stability of the system it carries a high
  overhead in a real time environment because of
  the added cost to a context switch and the caches
  that need to be flushed each context switch ( the
  tlb for example).
• User and kernel modes while protecting the
  systems from user processes adds significant
  overhead because each mode switch incurs a
  performance penalty because of the trap
  instructions and cache flushes.
• memory paging can increase the cost of a context
  switch by several magnitudes if the memory pages
  we need were switched out. This causes a lot of
  unpredictability in the timing.

25
Problematic modern features

• Dynamic memory allocation this is a critical
  feature used by most modern application, however
  it creates memory fragmentation and as a result
  adds to timing unpredictability.

26
Linux

• Linux Is based on UNIX which is a time sharing
  system designed to optimize average performance
  (as weighted by priorities) and Linux preserves
  this nature. This means that as the load
  increases the real time processes will suffer the
  most.
• Linux currently doesnt support an interface for
  telling the OS that a task is real time or
  informing it of any timing constraints on a task.

27
Linux

• The basic Linux scheduler is a time slice based
  on priorities, however it is a fair scheduler
  and favors interactive (IO bound) programs
  rewarding programs who havent used up all of
  their CPU share.
• This presents several problems
• 1. the time slice resolution may be higher than
  requires for the timing and so it may be
  impossible to meet the timing demands.
• 2. raising the priority of real time tasks is
  insufficient because enough lower priority tasks
  will cause us to miss deadlines anyway.
• 3.fairness is not a desirable quality in
  real-time systems.

28
Linux

• Linux does not provide a reliable mechanism to
  wake a task up at a certain time. The sleep timer
  can only promise to wake a task up after a
  certain time.
• The kernel cant be preempted so long system
  calls executed in the kernel can interfere with
  timing constraints.
• External interrupts are disabled at several
  sections in the kernel to protect OS data from
  corruption (saves the effort of making them
  reentrant). This adds unpredictability to the
  amount of time it takes to respond to IO.

29
Linux kernel 2.6 changes

• These has been some progress on some of these
  issues in the 2.6 Linux kernel
• 1.the kernel is now preemptable although not all
  of it is.
• 2.improvements in IO handling.
• despite these changes 2.6 is still not suitable
  for hard real time performance.

30
Windows

• Windows (NT based) suffers from most of the same
  problems Linux does.
• Windows doesnt support the specification of
  timing constraints for tasks.

31
Windows

• Interrupts may be delayed for an arbitrarily long
  time in critical sections of the kernel.
• Handling of interrupts is done in two parts the
  second part the deferred procedure call is
  executed later in a FIFO manner and this may
  cause delays if a lot of DPCs are awaiting
  dispatch.
• Requests for synchronization objects such as
  semaphores are also done in FIFO, leading to
  similar unpredictable delays.

32
Windows

• The windows scheduler supports threads on an OS
  level. Each thread is given a priority (0-31).
• The scheduler has compensation for input bound
  threads and anti-starvation mechanisms.
• Windows has a real time priority for threads
  but other real-time threads and the kernel can
  preempt a task with real time priority (15-26).
• A real time priority is higher than some parts
  of the OS, causing most non-real-time apps to
  become unusable

33
Fixing the problems

• Fixing these problems in either OS would require
  either some form of hack into the operating
  systems design or a massive rewrite of the
  kernel.
• This has been tried using both of the operating
  systems as we will see in later case studies
  (rt-linux, rialto).

34
Outline

• Basic definitions
• The challenges
• Deficiencies in modern operating systems
• Proposed solutions
• Case studies of real-time OSs.

35
Scheduling

• Scheduling algorithm properties
• An algorithm is said to be optimal if for each
  task set for which a feasible scheduling is
  possible the algorithm will find one (assumes
  tasks are independent).
• Schedulable utilization is the maximal
  utilization of a task set such that a task load
  under this utilization will be scheduled
  successfully by the algorithm.

36
Cyclic executive

• This is a static table driven approach to
  scheduling. It assumes full apriori knowledge
  meaning tasks are static and periodic.
• The schedule is created offline, so scheduling
  doesnt load the system during runtime.
• performance guarantees can be given before system
  is built.
• The algorithm analyze and schedule tasks
  offline using some algorithm which creates a
  static schedule for running all tasks during
  runtime.

37
Cyclic executive

• The cyclic executive approach can incorporate
  tasks of which there is no foreknowledge by
  scheduling them in empty spaces in the cycle,
  however it cant make any guarantees about their
  timing. This approach works for the other
  algorithms as well.

38
Priority driven algorithms

• This is a family of algorithms which gives the
  tasks priorities according to some criterion and
  schedules at each point the task with the highest
  priority. They are efficient in that if there are
  tasks ready to execute the CPU is always busy.
• This approach is attractive because in case of
  performance degradation the most important tasks
  will be affected the least.

39
Rate monotonic

• This algorithm assumes a periodic task model
  with preemption.
• The Algorithm
• is a priority driven algorithm where the tasks
  are assigned priorities proportional to their
  cycle length, the shorter the cycle the higher
  the priority.

40
Rate monotonic

• The algorithm is optimal in a way if there is a
  feasible schedule for a periodic task load where
  Pipriorities then RM will find a feasible schedule.
  
• Liu and Leyland show that the schedulable
  utilization of RM is bounded by
• where N is the number of real time tasks

41
example

• If we have the following set of tasks
• task1 perioddeadline3 execution time1
• task2 perioddeadline5 execution time3
• Then the schedule will look like this

42
The problem with static priority driven algorithms

• This simple example shows that a schedulable set
  of tasks cant be scheduled based on static
  priorities

task1 perioddeadline4 execution
time2 task2 perioddeadline10 execution
time5
1
3
2
1
1
Task 2 must have higher priority
43
EDF algorithm

• The Earliest Deadline First algorithm is a
  priority driven algorithm in which the tasks get
  priorities according to how close their nearest
  deadline is. The closer the deadline the higher
  the priority.

44
example

• If we have the following set of tasks
• task1 perioddeadline3 execution time1
• task2 perioddeadline5 execution time 3
• Then the schedule will look like this

4
1
2
3
1
2
2
45
EDF properties

• The algorithm is optimal.
• The schedulable utilization of the algorithm is 1
  .
• Priorities must only be recalculated when a new
  task arrives, as time advances equally for all
  tasks.
• the algorithm behaves unpredictably at overloads,
  and suffers more from overloads than RM.

46
LST algorithm

• The slack time of a task is the amount of time
  until the task must be scheduled or miss its
  deadline.
• The least slack time algorithm is a priority
  driven algorithm in which the tasks receive
  priorities according the their slack times. The
  shorter the slack time the higher the priority.
• The LST algorithm is optimal

47
Priority inversion

• The priority inversion problem exists in the
  preemptive model when tasks are dependant on one
  another (share resources). it causes a high
  priority task to wait for a low priority task to
  conclude because it is holding a critical
  resource.
• This problem can be at least partially solved by
  priority inheritance, or the priority ceiling
  protocol based on a similar concept.

48
Outline

• Basic definitions
• The challenges
• Deficiencies in modern operating systems
• Proposed solutions
• Case studies of real-time OSs

49
Case study

• We will go over how some actual implementations
  of real time operating systems tackled these
  problems.
• Rt-Linux
• Rialto
• VxWorks

50
Rt-linux

• RT-Linux is a system developed as a way to adapt
  Linux to hard real time performance and still
  give the user most of the advantages of the Linux
  environment.
• The main idea add a hard real time scheduler
  into (under) the kernel which will schedule the
  real time tasks and will run the rest of Linux
  as the lowest priority task.

51
Rt-linux

• the main philosophy is this
• Real time programs need to be split into parts ,
  a small light parts with hard real time
  constraints and larger parts that do most of the
  processing.
• the light real time parts will be scheduled in
  the real time scheduler, while the heavier parts
  will run as normal processes under Linux.
• The two parts can communicate through a
  real-time-FIFO, a non-blocking queue which allows
  the real time parts to read and write data from
  normal processes.

52
Rt-linux

• Real time tasks are implemented as kernel modules
  whose init function sends the real time scheduler
  all the information about their release times,
  periods and deadlines.
• Rt-Linux allows for swappable schedulers for the
  real time tasks, a scheduler based on the EDF
  algorithm and another based on the RM algorithm.
• Linux is run as the lowest priority process in
  either case and will only execute if the real
  time scheduler has nothing else to do.When Linux
  is running it uses its own scheduler to schedule
  tasks running under it.

53
Rt-Linux

• The Linux kernel runs as a process under the real
  time scheduler and as a result can be preempted
  by it at any time, and while this allows for good
  sleep timers and scheduling flexibility it
  dictates that system calls cant be used by real
  time tasks in order to preserve the consistency
  of the kernels internal state.

54
Rt-Linux

• All the memory needs of the real time processes
  are handled by the real time subsystem (no system
  calls) and memory is allocated in non-pageable
  blocks saving the uncertainty of swap file usage.
  
• As all the real time tasks run in the same kernel
  memory space we lose the benefits of protected
  memory for them but context switches are much
  faster because we save the cost of flushing the
  TLB cache and using trap instructions to enter
  and exit kernel mode for scheduling.

55
Rt-Linux

• A very serious problem is the disabling of
  interrupts by the kernel this is handled by
  replacing the function call to disable interrupts
  in the kernel source with a macro that simply
  masks these interrupts from the Linux process but
  not from the other real time processes, so
  interrupts are never blocked as far as the real
  time scheduler or tasks are concerned.
• The masking of the interrupts is done by having
  the real time portions of the system initially
  handle all interrupts and pass the ones intended
  to the Linux process on to it as software
  interrupts (in case Linux thinks the interrupts
  are enabled).

56
Rt-Linux - limitations

• Doesnt handle the problem of scheduling
  dependant real time tasks.
• Not all applications can be split into a light
  real-time parts and non-real-time parts.
• Not suited for dynamic real time systems.

57
Rt-Linux - summary

• Demonstrates the hack in to the architecture
  approach to adding real time.
• Is a functional system used in the wild.
• Its requirements form the real time portions of
  the tasks may make it unsuitable for some
  applications.

58
Rialto

• Rialto is a real time system developed at
  Microsoft Research based on the windows NT
  architecture.
• The main idea rewriting the kernel and scheduler
  so that they support real time deadlines.

59
Rialto

• Unlike rt-linux it doesnt require the
  separation of the hard real time functionality
  from the rest of the code.
• It uses the beginConstraint and endConstriant
  function calls that pass the parameters of the
  real time portion of task to the scheduler.

60
Rialto

• The parameters passed to the scheduler
• - Release time
• - Deadline
• - Running time estimate
• - Criticality (criticalhard constraint ,
  non-criticalsoft )
• The scheduler returns the feasibility of
  scheduling the task. This allows the task to
  perform load shedding if possible, or fail
  gracefully if not.

61
Rialto

• the scheduling in Rialto is quite complicated
• Tasks can be granted CPU time reservations. This
  is the average amount of CPU time they should
  get.
• The scheduler is based on a variation of
  preemptive LST which uses the maximal amount of
  time before a task has to start running
  (deadline-running time estimate).
• Critical tasks always run before non-critical
  tasks.

62
Rialto

• The rialto system is system-resource aware.
• Tasks can negotiate with the resource manager in
  order to reserve the resources they need.
• The resource manager arbitrates the use of
  resources, and can cause an existing task to
  relinquish resources allocated to it.

63
Rialto - limitations

• Rialto doesnt handle the problem of the kernel
  disabling interrupts.
• The schedulability test was not fully implemented
  originally.

64
Rialto -summary

• Rialto uses the approach of rewriting an existing
  kernel in order to allow support for real time
  performance.
• Built as academic work ,was never widely used.
• Uses a novel scheduling approach, But because it
  is complicated it is sensitive to many
  implementation details.

65
VxWorks

• VxWorks is a commercial hard real time operating
  system developed by wind river systems.
• The main idea use monolithic kernel to schedule
  user tasks according to user defined priorities.
  Maximize kernel timing predictability.
• Gives the users maximal control.

66
VxWorks

• A dedicated real time system, not intended as a
  general purpose OS.
• lacks many modern os features that interfere with
  real time performance (flat memory model, no
  paging).

67
VxWorks

• Scheduling is done using a preemptive priority
  driven approach, priorities are chosen
  arbitrarily by the user (0-256).
• Priorities can be changed by the user at runtime
  but this is discouraged.
• A user can lock a task so that it cant be
  preempted even by higher priority tasks or
  interrupts.
• This allows the use of the fixed priority
  response time analysis to check schedulability
  offline.

68
VxWorks

• Is resource sharing aware and has a priority
  inheritance built in.
• Optimizations in implementation of the context
  switch and the return from interrupts.
• The kernel never disables NMI (non-maskable
  interrupts) so they are always available to the
  user.

69
VxWorks - limitations

• Lacks many modern OS features.
• Guaranteeing the deadlines is the responsibility
  of the user at design time.
• Doesnt support most modern applications and APIs
  (only a small subset of POSIX).
• Despite the flat memory model dynamic memory
  allocation still cases memory fragmenting, which
  increases timing unpredictability.

70
VxWorks - summary

• A dedicated and widely used real time system.
• Offers the user maximal control , but also passes
  him responsibility for the deadlines.
• Lacks many modern OS features.

71
Summary

• This talk reviewed
• - The basic concepts of real time systems
• - The challenges of building a real time system
• - The use of generic OSs as real time systems
• - Some Solutions to the Challenges.
• - The approaches taken by some real systems

72
questions
?
73
Thank you