Karl Larson

1
Improving Availability inVERITAS Environments

• Karl Larson
• karl_at_tellme.com
• Todd Stansell
• todd_at_certaintysolutions.com

2
Introduction

• Collect information to understand your situation
• Know that certain architectural characteristics
  are difficult to manage
• There are solutions to help you with your storage
  management
• VERITAS provides excellent tools to help with
  edge cases, if used correctly

3
Collecting Information

• Understand your usage patterns to understand how
  you can improve performance and availability
• Tools
• vxstat2gnuplot
• Cricket
• save-vxlayout
• symcli/emcprint

4
vxstat2gnuplot

• Converts vxstat data to gnuplot data
• Can provide a clear, graphical view of
  performance statistics of VxVM objects

5
Cricket

• Free, trends-based monitoring tool
• Use the trend information to understand what is
  normal
• When the patterns become abnormal, you know you
  have a problem
• http//cricket.sourceforge.net/

6
save-vxlayout

• Saves all Volume Manager configuration details
  necessary to rebuild your volumes
• Must still be archived to another host for
  maximum safety
• Typically run via cron, once a day
• Should only be necessary for disaster recovery
  and remote troubleshooting

7
symcli/emcprint

• symcli available from EMC2
• Allows retrieval of Symmetrix internal
  configuration details
• emcprint custom script
• Requires EMC2s symcli utility
• Extension to vxprint
• Shows backend director, controller, and device
  numbers for each subdisk

8
The small files problem

• The size of the files is not as troublesome as
  the number of files
• Keep directories a manageable size
• Metadata updates become expensive
• Backups can be hard
• Restores are much harder

9
Metadata performance bottleneck (VxFS)

• VxFS uses metadata to store directory
  information and intent logs
• Changing millions of small files causes far more
  metadata updates than changing several large
  files does
• By default, all metadata information is located
  at the beginning of the filesystem

10
Quicklog And Metadata Updates

• If metadata updates are dominated by intent log
  writes, moving them to dedicated storage can help
• QuickLog allows you to move just the intent log
  portion of the metadata to other disks
• If most of your problems are due to directory
  updates, your only option is to stripe across
  more disks

11
Backup and Recovery Problems

• Sequentially backing up millions of individual
  files can take days
• Consistent, point-in-time backups are not
  practical without some sort of snapshot
  technology
• VERITAS provides for the ability to do snapshots
  at the volume and filesystem layers

12
Volume Manager snapshots

• Pros
• You get this feature for free with Volume Manager
• You can easily revert an entire filesystem to its
  previous state by mounting the snapshot
• Cons
• It requires as much disk space as the original
  filesystem

13
Filesystem Snapshots

• Pros
• Cheaper than volume snapshots since you only need
  space for changes that occur while snapshot is
  mounted
• Cons
• Most implementations dont allow easy rollback to
  snapshot copy
• Not persistent between reboots
• Performance overhead while snapshot exists
• Users cant easily restore their own files like
  they can on, say, a NetApp

14
FlashBackup is one solution

• You get basic filesystem snapshot capability free
  with VxFS
• The FlashBackup extension to NetBackup is an
  automated way to do filesystem snapshot-based
  backups
• FlashBackup works with both VxFS and UFS
  filesystems

15
How FlashBackup Works

• Uses internal kernel driver to create a snapshot
  on a common volume
• Builds an inode map of filesystem
• Dumps data via the raw device for maximum
  performance
• For incremental backups, raw device is still
  used, but only the changed blocks are transferred

16
FlashBackup

• FlashBackups, full and incremental, are always
  done using the raw device
• The number of files only increases the time
  required to build the inode map
• Full restores are done to the raw device too, and
  are usually faster than backups
• Incremental restores must be done through the
  filesystem, and can require weeks to complete

17
What does this mean?

• Understand your usage patterns
• Avoid running into edge cases
• For systems with lots of small files use several
  small filesystems
• Modularize your systems to avoid over-sized
  systems
• Use a hashed directory structure to keep
  directories a reasonable size
• There are tools available to help you when you
  cant avoid edge cases

18
Questions?http//www.vxideas.org/

• Karl Larson
• karl_at_tellme.com
• Todd Stansell
• todd_at_certaintysolutions.com