Part 6: System Security

1
Part 6 System Security

• Malware
• Safe Coding
• Software Trust
• Virus Detectors
• Software Signatures
• Kernel Integrity Checkers
• Application Integrity Checkers

2
Computer Systems

• Security keep the systems safe
• No malware
• No applications that can be attacked
• No data stolen
• No corruption or harmful activity
• User accounts are not hacked
• .. HOW?

Safe software Virus checks Sandboxing Integrity
checking Crypto File Systems Smartcards
3
Safe Coding

• Not well understood
• Software has vulnerabilities, ensure attack free
  code at implementation time
• Stack Guard and Lib Safe and a host of techniques
  are available
• Type safe languages are available
• Not clear if these approaches work
• Teach programmers to be aware of attack
  techniques on software
• Safe coding practices

4
Bounds Checking

• Array bounds often get exceeded and yet nothing
  happens
• Very common in C and C code
• Optimized compilers for all languages disable
  bounds checking
• Have to be checked in software, at all points
• Buffer overflow of many kinds use this
• For every data transfer we need to check the size
  of the data
• Function calls
• Data copying
• Reading input
• more

5
Input Validation

• Input to a program has to be validated
• From network
• From files
• From user
• Almost never done
• A program expects the data to be formatted
  correctly, specially if that program generated
  the data
• Suppose WORD creates a DOC file
• When WORD reads it, the data is expected to be in
  WORD format
• Simple checks are NOT enough
• This vulnerability is ingrained into all legacy
  software

6
Calling Routines

• When routines are called, the caller and the
  callee must perform sanity checks on arguments
• Specially when system calls and library routines
  are called
• Many stack and heap smashing attacks use such
  vulnerabilities
• If a program has a call to the system routine
  in Unix
• If arguments are tampered with, any program can
  be run by an attacker

7
Root Access Attacks

• set uid programs can gain root access
• Used heavily for Unix
• Similar features in all operating systems
• Many attacks have been designed to misuse this
  feature in many non-intuitive ways
• Databases are particularly attackable via setuid
  attacks

8
Heap tricks

• Put crafted data on heap
• Most input is read into the heap area anyway
• Ensure that a particular pointer to an argument
  to another function call has been tampered with
• Next time the function is called, it will be
  called with attacked arguments
• Hard to detect and protect against

9
Open Source vs. Closed Source

• What is safer Open Source or Closed Source?
• Open Source
• Many eyes theory
• When found, vulnerabilities quickly fixed
• Closed Source
• Stringent code review and coding security
  policies may be in effect
• Even if vulnerabilities exist, they may not be
  found, as the code is unknown
• Vendor may have liability
• This debate has no obvious outcome, both have
  advantages and disadvantages

10
Data Execution

• Many attacks inject executable code as data and
  then transfer control to it
• Solution is to ensure data is never executed
• Code resides in code segment, data resides in
  data segment
• Code segment must be non-writable
• Data segment must be non-executable
• Hardware support needed NX or XD bits in page
  tables
• Existed in the old VAX architecture

11
Software Issues

• Software is inherently untrustable
• Even if source code is visible (e.g. Thompson
  Attack)
• Many techniques for obfuscation
• Easy to make software do things other than what
  the source code seems to do
• Programmers can inject things into programs that
  can be harmful
• By design
• By accident
• How do we know that software is to be trusted?
• OS
• Applications
• Drivers
• Downloaded things

No good answer
12
Feature Creep

• Features of a program causes unforeseen
  vulnerabilities
• e.g. Visual Basic Scripting in Outlook
• Why does an email reader have to be programmable?
• .because it can be done!
• Complex features of programs are attractive to
  designers and to end users
• Impossible to analyze the security impacts
• Many tricks are then invented by attackers
• Old adage Keep it simple
• Code will be cleaner
• Features would be understandable
• Unforeseen consequences will be minimized

13
Malware

• Virus Trojan Worm
• Adware, spyware
• Rootkits

Can be detectable, but only if detector in
operating system
Not detectable
14
Virus Detectors

• Current state of the art in virus prevention is
  detection of viruses in all incoming files and
  existing files
• Any file that a computer accesses is scanned by
  a virus detector
• Hooks to the open and other system calls
• Makes system performance degrade
• Scanning via hash matching
• Needs to know all the hashes of known viruses
• Detects known viruses
• Does not detect changes to existing programs
• Have to frequently download virus definitions

15
Downside of Virus Detectors

• Can be easily disabled (via buffer overflow, for
  example)
• Rootkits can uninstall it, or disable it
• Cannot detect rootkits
• Performance is slow
• New viruses can spread without fear for some time
• The virus detector is itself a virus?
• Yes, but a good one
• No, but can be, via a little social engineering
• Can store hashes of good files, but.
• A virus can change the definitions and escape
  detection

16
Smart Viruses

• Polymorphic Viruses
• Viruses that attack detectors
• Viruses that hide from detectors
• Camouflaged
• Hidden names

17
Rootkits

• Rootkits are kernel patches processes
• Processes may run with Admin/root privileges
• Special routines are embedded in the kernel
• Many kernel routines may be changed or patched
• How to detect
• Take checksum of kernel
• Take checksum of routines that may change
• Look at kernel tables and watch for changes
• Check for perturbations

18
Undetecatable

• If a rootkit is not worried about rootkit
  detectors then it is detectable
• Rootkits can be designed to be evasive
• If we take hashes of the kernel, it will patch
  the routine that computes the hash
• If we track kernel tables, the tracking routine
  will be patched
• A new rootkit detector will find an old rootkit
• As long as the rootkit writer does not know of
  the detector and its techniques
• A rootkit can patch itself later to become
  invisible to new rootkit detector
• Cat and Mouse game

19
Sandboxing

• Run a program with some flags set in the kernel,
  designating it to be a sandboxed process
• Every time this process calls a system call
• Check whether it is an allowable operation
• Check whether the arguments are allowed
• Interpret the call and ensure it falls under the
  safe category
• Policies have to be stated and enforced
• . Loopholes?

20
Integrity Checking

• Every executable file should have a pre-computed
  hash and a signature and an issuer certificate
• Hash ensures any changes to the executable is
  detectable
• Signature ensures that no one updates the hash
• Issuer certificate ensures the signature is not
  faked
• This information is stored, separately from the
  file itself (hopefully securely)
• Integrity Checking
• Check hash and signature every time the file is
  executed
• Check hash occasionally while the program
  execution is in progress (to detect dynamic
  corruption)
• Problem Who does the checking?

21
Application Integrity Checking

• Assume Operating System is not compromised
• OS does Application Integrity Checking
• OS stores hashes and signatures
• Unsigned application either not allowed, or
  signed by OS
• Dynamic checking is also done
• Ensures freedom from downloaded viruses
• Ensures some freedom from buffer overflow attacks
• Cannot be completely detected, but the harm
  caused is limited
• Runs only trusted applications
• Either signed applications or authorized by the
  user
• Help prevention of undetected downloads, spyware
  and many such malware

22
Kernel Integrity Checking

• How to check the OS for rootkits?
• Use a hardware device
• TCG Trusted Computing Group approach
• Co-Pilot approach
• Use a software device
• The VMM Virtual Machine Manager approach
• Concept
• A trusted device has keys and signatures for the
  operating system components
• This device can access the OS and check it for
  integrity
• Possible to detect rootkits, as long as the
  checker is not compromised

23
Hardware Checking

• TPM (trusted platform module) from TCG
• A hardware device, that contains keys and hashes
  for checking integrity of operating systems
• Attestation of software and digital signatures
  possible
• An elaborate framework of security modules,
  protocols and monitoring of applications
• Can be used to enforce software licenses
• Co-Pilot Project
• Hardware on the PCI bus, can take over the
  computer
• Performs memory checks on the OS and interrupt
  routines
• Can detect rootkits, even dynamic ones

24
Software Checking

• Run the Operating System as a host on a Virtual
  Machine Manager
• The VMM is hard to compromise
• Small, hence free of vulnerabilities (assumption)
• OS has to be rootkitted to attack VMM, but the
  VMM can detect such rootkits
• Check OS integrity from the VMM (Terra Project)
• Check OS integrity from a separate host operating
  system called the Security Monitor
• ASU project
• Work in progress, not tested well yet

25
Secure I/O

• All security solutions outlined so far suffers
  from a spoofing vulnerability
• A piece of software can be flagged to be
  trusted by faking human input by a program
  (virus)
• TO ensure spoof freedom there is a need for a
  secure I/O channel from the core security system
  to the human user
• TCG-TPM, Co-Pilot, Terra --- all are vulnerable
• Secure I/O needs separate hardware channel,
  separate keypad
• An SSL connection to a small communicator may be
  a solution.

26
Crypto File Systems

• All files are stored encrypted
• Where is the key?
• Derived from a passphrase
• Everything is weak as along the computer is
  running
• MS-CFS uses PKI
• Advantages
• Lost/stolen disks are secure
• Proper implementations may disallow some viruses
  from accessing some files
• Diadvantrages
• Files are decrypted on the fly and CFS is
  transparent to users and hance viruses
• Attacking the key is possible

27
Current Status

• Integrity checking is almost non-existent in
  deployed software
• Hence, all systems are soft against viral attacks
• Big security problem,
• Device driver signing is used in Windows, quite
  ineffective
• The TCG system is the closest to deployment
• Also the weakest system
• Has political implications
• No other solutions are being deployed ?
• may not be entirely true