Data deletion

1
Data deletion
Out damn spot, out! http//seifried.org/security/
Kurt Seifried, kurt_at_seifried.org
2
TOC

• The basic problem
• The attacker
• Some examples of failure
• Wiping hardware
• Wiping files
• Wiping information
• Wiping memory
• Encrypting information
• Common failure modes
• The failure of containment
• The future

3
The basic problem

• Data is valuable, some data increases in value
  with age, some decreases (Tobacco company studies
  for example)
• Increasingly powerful data recovery tools
• Deleting data rarely gets rid of it, instead
  freeing up storage space, formatting does not
  destroy data either in most cases

4
The basic problem (cont.)

• Existing tools such as EnCase make data recovery
  very easy
• As data becomes more abstracted it becomes more
  difficult to locate where it has been stored
• More data is being stored on network file systems
• Copies are perfect, file fragments are perfect
  partial copies

5
The attacker

• Different types of attackers, with various
  resource levels and attack methods, this must be
  taken into account when creating a security
  policy and protection mechanisms
• Unintended recipient
• Malicious insider
• Outside hacker
• Civil litigants
• Law enforcement

6
Some examples of failure

• Hardware devices not sanitized due to weak
  magnetic fields
• Failure to wipe alternate data streams attached
  to files
• Data being replicated in unexpected places due to
  defragmentation, backups, etc

7
Wiping hardware

• Wiping hard drive, floppy disks and tapes
• Wiping cd-roms and other optical media
• Wiping memory

8
Wiping hard drive, floppy disks and tapes

• Raid issues
• RAID examples
• Bad block / clusters
• Destroying hardware
• Hard and soft 0's and 1's
• Degaussing issues
• Verification of wiping

9
RAID issues

• If a drive in a volume set fails most of the data
  available on it will still be available
• If a stripped drive without or without parity
  (RAID 0, 3, 5) fails chances are large pieces of
  data can be retrieved, depending on cluster size
  used (up to 64k in some cases)
• Mirrored drives (RAID 1) have a complete copy of
  the data

10
RAID examples

• If RAID level 3 or 5 operation is interrupted,
  e.g. the data blocks have been scrubbed, but
  parity has not been regenerated it may be
  possible to regenerate the data from parity and
  data on the other drives
• RAID level 1 can be done in software and
  hardware, scrubbing clusters may not get the
  correct clusters on both drives

11
Bad blocks / clusters

• Blocks or clusters that show damage are
  eventually marked as bad, this can be done by
  the hardware itself (i.e. SCSI hard drives) or by
  software (the OS)
• Impossible to scrub bad blocks in many cases (the
  hard drive itself makes them inaccessible), the
  drive must be physically destroyed

12
Destroying hardware

• Destroys resale value (bad pun)
• Grinding requires reasonably small particles,
  especially as data density rises
• Punching a hole disks with a power drill will
  deter most attackers and is easily verified
  visually
• Use of hazardous materials can make proper
  disposal difficult and expensive

13
Hard and soft 0's and 1's

• Data is either 0 or a 1 on the physical medium,
  expressed as orientation by magnetic particles
• Hard drive heads wander, data is written on a
  track, this track can move slightly, thus data on
  the outside or inside of the track may not be
  overwritten

14
Hard and soft 0's and 1's (cont.)

• Data that is a 0 and then overwritten as a 1 will
  be a soft 0, some 1's remain
• Data that is a 0 and then overwritten with a 0
  will be a hard 0, very few 1's remain
• Multiple passes help, however data on the outside
  or inside of the track may remain intact, this
  requires physical inspection to retrieve however

15
Degaussing issues

• Degaussing requires strong magnetic fields
• Hard drives use increasingly dense data storage
  and much stronger and tightly focused magnetic
  fields, old degaussing equipment may not generate
  enough field strength to wipe data
• May not be possible to reformat and verify that
  data is wiped

16
Verification of wiping

• Hard drives have serial numbers, individual
  platters do not, harder to track
• Visual verification is possible with grinding,
  folding and so on, however without serial numbers
  it could be any drive
• Verification is never 100, some unknown
  technique may restore data

17
Wiping cd-roms and other optical media

• Media must usually be destroyed by grinding or
  shredding
• Huge volumes of media, easily lost or mixed up
  with other disks
• Machines to declassify cd-roms are expensive

18
Wiping memory

• Numerous hardware related issues
• Potential business issues when decommissioning
  older systems
• Please see wiping memory section

19
Wiping files

• Wiping memory
• ATA protected storage
• Verification of wiping
• Wiping free space
• Microsoft issues
• UNIX issues

20
Wiping memory

• Files are loaded into memory, consequently they
  can end up in a number of interesting locations
• Please see wiping memory section

21
ATA protected storage

• Protected area of hard drive, not accessible to
  BIOS or OS, used to store recovery data (i.e. OS
  installation files)
• MBR must be modified or special boot media used
  to access the protected areas
• Not wiped by most software packages including
  hardware wiping software
• Tools such as dd will not copy the data reliably

22
Verification of wiping

• Checking the media
• Disk defragmentation
• Looking for data

23
Checking the media

• You must check individual clusters/etc for data,
  this means using a known pattern (such as all
  0's) and then checking for any 1's for example
• This of course assumes there is only one copy of
  the data file, data can be copied as a result of
  being in swap space or swap files.

24
Disk defragmentation

• Disk defragmentation results in data being copied
  and the original space being marked as free.
• Operating system does not store disk
  defragmentation data, the wiping software
  consequently has no idea of where the data has
  been

25
Looking for data

• Verifying data has been wiped requires a search
  to ensure no file fragments or copies exist,
  pattern matching partial strings and so on is
  expensive computationally and may not be possible
  on large storage arrays, this of course requires
  a copy of the data (which requires wiping...),
  use of signatures (i.e. MD5 sums) or watermarks
  is possible but this will not catch partial data
  fragments

26
Looking for data (cont.)

• Data may have been copied to temporary files on
  other file systems (local or remote)
• Files can be very large and contain multiple
  copies of data (i.e. MS Word with auto save)

27
Wiping free space

• As a consequence of not being able to verify data
  has been copied before it was wiped all unused
  space must be wiped, this include slack space
  (partially unused inodes), free space, swap file
  space, and so on.
• Modern hard drives are huge, 160 gigabytes and
  growing, wiping free space cane take hours or
  even days, may not be possible at all on busy
  systems

28
Wiping free space (cont.)

• Free space cannot be locked, free space may be
  reserved by another process for a file and thus
  be inaccessible, but not overwritten yet, you
  would effectively need to stop the system, boot
  from different media, wipe all free space and
  slack space to guarantee destruction

29
Microsoft issues

• NTFS and NTFS5
• File locking
• File replication services

30
NTFS and NTFS5

• NTFS and NTFS5 Overview
• Slack space
• Defragmentation
• Alternate Data Streams
• Master File Table
• Encrypted File System
• Journaling
• Sparse files
• Compressed files and directories

31
NTFS and NTFS5 Overview

• NTFS5 needed to support new features such as disk
  quotas, file encryption, reparse points,
  directory junctions, volume mount points, sparse
  files, and the change journal
• NTFS can be converted to NTFS5, NTFS5 cannot be
  converted to NTFS
• NTFS is a journaling file system with database
  style components

32
Slack space

• Most files do not fully use the clusters they are
  allocated, thus even when a file is overwritten
  parts of it may survive
• Difficult to wipe slack space since it has been
  allocated, not all software wipes slack space
  properly

33
Defragmentation

• Files are copied around the disk, in essence you
  end up with multiple copies of any defragmented
  file
• Often runs as an automated task on servers
• Must wipe all free space to deal with this issue

34
Alternate Data Streams

• Few wiping programs properly wipe alternate data
  streams (e.g. PGP wipe has not been fixed)
• Used by default in Explorer to store thumbnails
  of images, and by Excel 2000 and others to store
  temporary files
• Must wipe all free space to deal with this issue

35
Master File Table

• Small files (under 1k) stored directly in MFT
  sometimes
• MFT cannot be safely modified directly, damage to
  MFT can destroy the file system (many products
  make no attempt to touch the MFT)
• MFT never grows smaller, small files stored in
  MFT only overwritten by other MFT events

36
Encrypted File System

• Encrypts files and directories, existing files
  and directories marked for encryption leave plain
  text copies
• If only files marked as encrypted they may be
  written in decrypted form to hard drive when you
  access them
• Microsoft advises creating an encrypted folder,
  and then creating files inside of it

37
Journaling

• File data is stored in a journal before being
  committed, this increases the number of locations
  data is stored
• Journal areas may be cleaned with wipe free
  space, however this is problematic

38
Sparse files

• Large files containing long strings of zeros can
  be created, but only the actual data (i.e. not
  the 0s) is stored, resulting in significant
  space savings
• Should not interfere with wipe free space (but
  untested as of yet)
• Sparse files cannot be changed to normal files

39
Compressed files and directories

• Files stored in compressed format, files are
  automatically decompressed when opened and
  compressed when saved
• Large number of file copies executed (to
  decompress and compress file), essentially each
  time you open or save a file

40
File locking

• Locked files cannot be deleted or modified (can
  be scheduled for after a reboot takes place
  however)
• Difficult to remove a lock, easy to create a lock
• Dlock from 32bits can be used to lock files

41
File replication services

• Data files are automatically replicated when
  written to
• When deleted the remote copy is simply deleted,
  files cannot be wiped on remote systems
• Files are staged in a temporary directory as well
  on remote servers

42
UNIX issues

• Wiping free space is not possible on most systems
  due to lack of utilities, utilities that do exist
  generally do not wipe slack space, leaving file
  fragments
• Extensive use of network file storage via NFS,
  AFS and others

43
Wiping information

• Overview
• Application issues
• Protocol issues

44
Wiping information overview

• All the problems of wiping files and media come
  into play
• More difficult then wiping files as information
  typically gets copied, moved, merged and shared
  in many forms
• Existence of information can be as useful to an
  attacker as the actual information

45
Wiping information overview (cont.)

• Non-existence of information can also be useful
  to attackers
• Tracking information is nearly impossible, file
  moves, copies, defragmentation, emails containing
  data, cutting and pasting data (data is stored in
  clipboard) and so on

46
Application issues

• Databases
• Printers / Print servers
• Search engines
• Exchange server

47
Database issues

• Data storage is heavily abstracted, even if an
  item is deleted wiping free space may not work as
  the database is still using the file space on the
  disk
• Database optimization tools, data integrity and
  so forth can also cause data to be moved around,
  resulting in multiple copies on the disk

48
Printers / Print servers

• Modern prints servers typically have solid state
  storage for print spools, wiping is rarely
  supported (do any?)
• Many are easily broken into, someone contain full
  operating systems such as Linux with webservers
  and so on

49
Search engines

• Often contain large part of the data, certainly
  enough to look for keywords
• Some cache documents (such as google.com)
• Removing data can be difficult depending upon
  implementation

50
Exchange server

• Stores messages in a database, impossible to
  ensure they are wiped
• Incoming and outgoing messages are stored in
  temporary areas resulting in multiple copies

51
Protocol issues

• Most network file sharing protocols used to
  transfer data are not encrypted by default, SMB,
  CIFS, NFS, etc.
• Network printing protocols do not support
  encryption, very few end devices (printers)
  support IPSec/etc.
• Proxy servers commonly cache data in memory and
  on disk

52
Wiping memory

• Wiping RAM
• Hibernation / suspend mode
• Swap space / file

53
Wiping RAM

• Memory can be volatile or non-volatile (i.e.
  Requires a charge to hold data)
• Volatile memory (conventional computer memory
  typically) can retain data even without a charge,
  when the power is cycled (i.e. the system is
  turned on) the data is actually wiped at this
  point, as opposed to when the system is turned off

54
Wiping RAM (cont.)

• Flash memory can hold data indefinitely (embedded
  devices, flash cards in routers, digital cameras,
  etc.)
• Replacing old memory is difficult at best,
  voltages and other issues, physical destruction
  may render the system unsaleable

55
Hibernation / suspend mode

• Many modern systems support suspend or
  hibernation modes
• The system is put into a minimal power
  consumption mode
• Memory (both system RAM and video) is fed a
  trickle charge or copied to a physical file which
  is read back into memory when the system is
  brought back up

56
Swap space / file

• Data is moved from memory back onto a disk
• Swap files can migrate and become fragmented,
  leaving traces all over the disk
• Swap partitions when used heavily will leave data
  at the end, unless heavy usage occurs again
  data can remain resident for several years
  (surviving formats and OS reinstallation)

57
Encrypting information

• Many file encryption packages encrypt the file
  but do not wipe the original
• When file is decrypted into memory it may be
  written to swap space / file, few applications
  use memory only flag
• Key management and storage issues, weak
  passphrases, easily attacked applications
• Lack of complete disk encryption programs

58
Encrypting information (cont.)

• Legal aspects, data deletion vs. Destruction of
  evidence, laws like the U.K. RIP bill
• Requirements for key and data recovery in most
  organizations (otherwise data dies with the user)

59
The failure of containment

• Few commercial operating systems support data
  classification (i.e. SECRET, TOP SECRET)
• Software to encrypt / control distribution
  expensive, requires deployment onto semi secure
  systems

60
Common failure modes

• Most software fails when dealing with bad blocks
• Most software does not scrub slack space by
  default
• Most software fails when dealing with NTFS ADS or
  the MFT
• Most software fails with network storage devices
  such as NFS/SAMBA/SANS

61
Common failure modes (cont.)

• Disk wiping utilities such as East-Tec eraser
  fail to overwrite all sectors on hard drives
  (Redemtech report).

62
The failure of containment

• Few commercial operating systems support data
  classification (i.e. SECRET, TOP SECRET)
• Software to encrypt / control distribution
  expensive, requires deployment onto semi secure
  systems

63
The future

• Extremely large drives wiping free space, slack
  space will take huge amounts of time, data will
  survive extended periods
• Microsoft DFS - Distributed File System - do you
  know where your data is?
• Database style file systems such as Microsofts
  OFS, due out in Longhorn, data is heavily
  abstracted and difficult to trace down

64
The future (cont.)

• Increased storage of data on network servers
  through protocols such as SMB, CIFS, HTTP, HTTPS
  and so on
• Cross platform interaction with large back end
  storage such as SANS that do not allow wiping
  software to be used
• Reliance on encryption and DRM systems to secure
  data, wiping may not be supported

65
The future (cont.)

• Network storage arrays, SANS, SWAN, acronym soup
• iSCSI protocol becoming mainstream
• IBM storage bricks and other huge data
  repositories that are disposable
• Mobile devices with distributed storage,
  PerosnalRAID

66
URLs

• http//seifried.org/security/presentations/
• NTFS resources http//linux-ntfs.sourceforge.net
  / http//www.sysinternals.com/ntw2k/source/ntfsin
  fo.shtml http//www.win2000mag.com/Articles/Index
  .cfm?ArticleID8294 http//www.pcguide.com/ref/hd
  d/file/ntfs/index.htm
• UNIX filesystem information http//www.fish.com/f
  orensics/advanced-files.pdf

67
URLs (cont.)

• Microsoft file replication service
  http//www.microsoft.com/windows2000/techinfo/res
  kit/samplechapters/dsdh/dsdh_frs_bnyr.asp
• Dlock (windows file and folder locking)
  http//www.32bits.co.uk/prods/dlock
• ATA protected space paper http//www.techpathways
  .com/uploads/Protected20Area20Analysis.pdf
• Redemtech report on disk wiping
  http//www.etestinglabs.com/main/reports/redemtec
  h.pdf

68
URL's (cont.)

• slack space wiping in UNIX ftp//ftp.scyld.com/p
  ub/bmap/ http//www.jetico.com/index.htm/linux/
  
• http//seifried.org/security/articles/20010910-pro
  tecting-information-from-exposure.html
• Basics of magnetic recording - http//www.infomrt.
  com/readrite/magbasic.html
• IBM Storage Bricks http//www.usenix.org/publicati
  ons/library/proceedings/fast02/morris.pdf
• PersonalRAID http//www.usenix.org/publications/li
  brary/proceedings/fast02/sobti.html

69
MS knowledge base

• Q221111, Q103657, Q310749, Q231388

70
Remediation tips

• Wiping slack space on UNIX find / -type f -exec
  bcwipe -S \
• Wiping free space in UNIX create a large file
  and then wipe it, this significantly impacts
  server availability however and is not reliable
  at all.

71
The End

• Question and answers if time permits
• Run for emergency exit if crowd is hostile