Before%20we%20start - PowerPoint PPT Presentation

About This Presentation
Title:

Before%20we%20start

Description:

Reads data from disk into buffers in memory. Writes buffers from memory ... sav [.exes]/repair ... Restoring file: UPS$:[000000.EXES]BACKUPSHR.EXE;16. Compression ... – PowerPoint PPT presentation

Number of Views:46
Avg rating:3.0/5.0
Slides: 53
Provided by: Bara90
Learn more at: https://de.openvms.org
Category:

less

Transcript and Presenter's Notes

Title: Before%20we%20start


1
(No Transcript)
2
A New Star Is Born
  • Before we start
  • Introducing Maklee Engineering Solutions
  • High quality services in the area of
  • Performance improvements
  • Oracle services / SQL tuning
  • Platform migration
  • Programming / Custom engineering
  • Custom training

3
OpenVMS Backup V8.3 New Features
  • Guy Peleg
  • President
  • Maklee Engineering guy.peleg_at_maklee.com

4
Agenda
  • Main Algorithms
  • Performance considerations
  • V8.3 New Features

5
30,000 feet overview
  • Reads data from disk into buffers in memory
  • Writes buffers from memory to target saveset
  • Savesets are sequential files
  • Fixed length record
  • RMS is used when writing/reading savesets to/from
    disks
  • SYSQIO is used when writing/reading savests
    to/from tape drives

6
BACKUP
  • BACKUP.EXE
  • DCL Utility to parse qualifiers and call the main
    shareable image
  • SYSUPDATEBACKUP.CLD
  • BACKUPSHR.EXE
  • parses API and branches to action routines
  • BACKUP operates in 3 main phases
  • Init phase
  • Open File phase
  • Map I/O phase

7
Init Phase
  • Decides on useable buffer space
  • WSQUOTA - WSQUOTA/64 300 JPI_FILCNT 512
  • Creates each buffer using EXPREG
  • Watch out This is R/W virtual address space
    charged against PGFLQUOTA
  • Opens and reads in first directory

8
Open File Phase
  • Looks for the next input file
  • Open file and read file header for mapping
    pointers
  • Map file to buffers
  • Maps each file extent
  • Breaks down I/O to fill a buffer
  • Max. buffer size is 63.5 KB
  • Even for a nicely contiguous file
  • Stops when either FILLM has been reached or all
    buffers have file I/O mapped (WSQUOTA)
  • No file data I/O at this point
  • No I/O to output save set at this point

9
Map and I/O Phase
  • Sort all mapped I/Os by increasing LBN numbers
  • Without it performance decreases by 10
  • Submit I/Os in LBN order in a tight loop
  • Pre V8.3 algorithm (or latest V7.3-2, V8.2 ECOs)
  • If DIOLM is reached wait 10ms
  • Large DIOLM significantly impacts performance
  • Wait for I/Os to complete
  • Write data to output device
  • Goto phase 1 if last buffer written

10
Disk Queue Load
  • BACKUP uses a large buffer pool to read file data
    in an optimized manner
  • Algorithm was designed 20 years ago
  • All reads for the entire buffer pool are issued
    concurrently
  • Over time buffer pools have gotten larger the
    I/O subsystems tolerance of being flooded with
    large number of I/Os has decreased
  • Especially true with EVA XP storage controllers
  • A picture is worth a thousand words.

11
FC Queue Depth
Very high peak, BACKUP blocks production I/O
12
FC Queue Depth Better Balance
Low peak, BACKUP no longer blocks production I/O
13
Port Queue Depth vs. Read KBS
Note the Read KBS (blue) value on this Port
14
Port Queue Depth vs. Read KBS
Read KBS on this Port remains the same. Lower
DIOLM value keeps a similar BACKUP throughput.
15
Impact of High FC Queue Depth
During periods of high queue depth, VDisk
Latencies exceed 100 milliseconds (thats
1/10th of a second!!)
16
Impact of Lowered FC Queue Depth
With lowered queue depth, VDisk Latencies only
once exceed 10 milliseconds (thats 1/100th of
a second much better)Production I/O workload
does not suffer due to BACKUP jobs.
17
Disk Queue Load
  • Pre V7.3-2 V8.2 (plus patches) behavior
  • issue as much I/Os possible allowed by DIOLM
  • Continue issuing I/Os until we hit SS_EXQUOTA
  • Wait for I/Os to complete and flood the I/O
    subsystem again
  • The New algorithm
  • Issue the disk reads in n parallel AST threads
  • The completion AST of each thread issues the next
    I/O
  • By default use 8 parallel I/O threads
  • Number of threads controlled by new /IO_LOAD
    qualifier

18
Disk Queue Load
  • Better performance
  • Idle EVA controller showed 15 reduction in
    elapsed time
  • Results are not linear busy controllers will
    witness more significant (dramatic !) improvement
  • Direct attached SCSI disk showed 5 improvement
  • YMMV

19
Performance Considerations
20
Performance Considerations
  • Why BACKUP performance is an issue?
  • Only 24 hours a day for incremental BACKUP
  • Small window for backup
  • Even smaller window for restore in case of
    emergency
  • Always test your restore procedures !

21
Process Quotas
  • System Managers Manual Volume 1 Essentials
  • Chapter 11.7
  • WSQUOTA
  • Initial value of 10,000 pagelets, increment by
    5,000
  • Over 100,000 performance decreases
  • FILLM
  • Equal to 100 (mostly sufficient), increment by 10
  • Use larger values if input disk contains many
    very small files or is highly fragmented
  • Effectiveness limited by WSQUOTA
  • DIOLM
  • Equal to 100 (mostly sufficient), increment by 10
  • 8 for Modern storage controllers
  • Larger values can cause disk I/O subsystem hangs
    or resets without any performance advantage

22
Process Quotas
Second option is 15 faster
  • Less is more
  • DIOLM32767, BIOLM32767, WSDEFWSQUOTAWSEXTENT3
    000000 VS.
  • DIOLM8, BIOLM150, WSDEF10000,
    WSQUOTA20000, WSEXTENT30000

23
Media Compaction
  • INIT/MEDIACOMPACTION
  • BACKUP/MEDIA_FORMATCOMPACTION
  • Compaction performed by the hardware
  • Default is no compaction
  • Compaction ratio depends on data and drive
  • Tape drives are optimized to use compaction

24
SDLT 320 Compaction Performance
MBs Per Second (More is better)
25
Performance Considerations
  • BACKUP / GROUP
  • Defines the number of blocks backup places in
    each redundancy group.
  • /GROUP_SIZE 10 is the default
  • Modern drive technology makes the XOR group
    feature obsolete
  • /GROUP_SIZE 0 to save 10 tape space
  • BACKUP/CRC
  • Stores CRC in each backup buffer
  • Does not increase the tape utilization as cell is
    part of the record header
  • Increases CPU consumption
  • With modern tape drives consider using /NOCRC
    (but need to understand the possible risk)

26
Performance Considerations
  • Tape drives are getting faster.
  • Ultrium-960 can write _at_160 MB/sec
  • With Ultrium drives elapsed time for Backup will
    be shorter but during this time the CPU may get
    overloaded (calculating CRC)
  • 90 CPU utilization on DS25 writing to
    Ultrium-460 drive (_at_ 40MB/sec)
  • May impact the availability of other applications
    on the system

27
Performance Considerations
  • /BLOCK_SIZE
  • Specifies the size of output block size in bytes
  • Larger blocks -gt less I/Os -gt better performance
  • Default of 32256 when writing to disks
  • Default of 8192 when writing to tapes
  • Modern tape drives optimized for large I/Os use
    maximum supported block size 65535
  • BACKUP is optimized to save large files
  • Same amount of data in small files vs. one large
    file may show up to 40 times difference.

28
RMS Optimizations
  • SET RMS
  • /BLOCK127 124
  • /BUFFER 8
  • /EXTEND65535 (for large savesets)
  • /NETWORK127
  • RMS_SEQFILE_WBH (Prior to V8.3)
  • With recent versions VMS now sets RAH WBH

29
BACKUP/FAST
  • Fast file scan
  • Large number of input files
  • Small number of files meet selection criteria

Elapsed time to backup all files created today
(less is better)
30
New Features
31
Improved CTRL-T information
  • Traditional CTRL-T information showed the name of
    the current file being saved/restored and the
    total number of save-set blocks processed
  • backup IA64KITSI64XB37.BCK/sav
    5dka100/ima
  • MIKAXP_VTA61 143638 BACKUP
    CPU000000.86 PF908 IO2192 MEM256
  • Restoring file 5DKA100DWMOTIF_SUPPORT_I64XB3
    7.KITHP.SICOMPRESSED1
  • Saveset volume1, saveset block266 (32256 byte
    blocks)
  • Can you tell how much data restored so far?
  • Can you tell when the restore will be done?
  • Introducing the new improved CTRL-T

32
Improved CTRL-T information
  • IPL31_VTAT7 144627 BACKUP CPU000003.40
    PF6298 IO18408 MEM465
  • Restoring file DKA100DWMOTIF_SUPPORT_I64XB37.K
    ITHP.PCSICOMPRESSED1
  • Saveset volume1, saveset block720 (32256 byte
    blocks)
  • 22.14MB restored out of 1.18GB, 1 completed
  • Restore rate 965KB/sec, estimated completion
    time 150731.85

33
Improved CTRL-T information
  • Note the message says ESTIMATED !!!
  • Fancy CTRL-T message displayed when
  • Restoring a saveset
  • Creating an image backup
  • Counters are updated when a file marked nobackup
    is encountered
  • When the amount of data to be saved is unknown,
    only the rate and total amount of data processed
    so far is displayed
  • CTRL-T can not be used in batch.and BACKUP is
    usually running in batch

34
BACKUP/PROGRESS_REPORTn
  • /PROGRESS_REPORTS writes CTRL-T style message to
    the output device every given interval
  • n is the number of seconds between intervals
  • back IA64KITSI64XB37.BCK/sav
    dka100/ima/progress10
  • BACKUP-I-PROGRESS, progress report generated at
    4-JAN-2006 150054.47
  • Restoring file DKA100DWMOTIF_SUPPORT_I64XB37.K
    ITHP1.PCSICOMPRESSED1
  • Saveset volume1, saveset block170 (32256 byte
    blocks)
  • 5.22MB restored out of 1.18GB, 0 completed
  • Restore rate 535KB/sec, estimated completion
    time 153928.28

35
lost saveset attributes
  • A saveset transferred using FTP or compressed and
    decompressed using ZIP will lose its RMS
    attributes
  • An attempt to process the saveset will fail
  • BACKUP-F-NOTSAVESET
  • Fortunately the correct RMS settings are stored
    in the saveset header

36
BACKUP / REPAIR
  • /REPAIR instructs BACKUP to attempt and restore
    the correct RMS attributes
  • backup images.bck/sav .exes/repair
  • BACKUP-I-REPAIRED, saveset attributes changed to
    RFMFIX, MRS32256
  • IPL31GUY 145858 BACKUP CPU000010.89
    PF7765 IO71628 MEM409
  • Restoring file UPS000000.EXESBACKUPSHR.EXE16

37
Compression Support in BACKUP
  • BACKUP supports data compression using ZLIB.
  • ZLIBs advantages
  • Free
  • Legally unencumbered
  • May be included in commercial applications
  • Lossless data compression library
  • Never expands the data
  • Unlike DCX may be used to compress stream of
    bytes
  • Does not need to analyze the file in advance
  • Maximum compression factor 11000
  • 50MB file filled with zeros compressed to 49KB
  • Realistic numbers are in the range of 21 to 51
  • http//www.zlib.net

38
Compression support in BACKUP
  • ZLIB routines are shipping in a new shareable
    image
  • SYSLIBRARYCOMPRESSSHR.EXE
  • SYSLIBRARYCOMPRESSSHR_EV6.EXE (alpha only)
  • New qualifier - /DATA_FORMATCOMPRESSED
  • Instructs BACKUP to create a compressed save-set
  • Does not need to be specified on the restore
    command
  • Initially viewed as a feature to save space.but
    turned out to be a significant performance
    improvement (wellat least in some cases)

39
Compression support in BACKUP
  • The slower the I/O subsystem the bigger the
    performance win
  • Big win for network operations
  • Big win for MSCP served devices
  • doubles the CPU consumption
  • On average BACKUP completes 2-5 times faster
  • Useful when shipping dump files
  • 800MB dump file was compressed to 75MB
  • No support for writing compressed savesets to
    tapes
  • Do not encrypt compress..at least for now

40
Compression support in BACKUP
  • Compressed save-sets have variable length records
  • Instead of fix
  • First record is not compressed
  • Detect compressed savesets
  • Force largest record value of /BLOCK_SIZE
  • To fix attributes after file transfer
  • SET FILE X.CBCK/ATTRIB(RFMVAR,LRL32256,MRS322
    56)

41
  • IPL31gt dir cxx.exe/sizall
  • Directory SYSCOMMONSYSEXE
  • CXXCOMPILER.EXE1 39.68MB/39.68MB
  • CXXDEMANGLE.EXE1 41KB/42KB
  • CXXLINK.EXE1 166KB/166KB
  • Total of 3 files, 39.89MB/39.89MB
  • IPL31gt backup cxx.exe cxx.bck/sav/log
  • BACKUP-W-NOFILES, no files selected from
    DSA5SYS0.SYSEXECXX.EXE
  • BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
    SEXECXXCOMPILER.EXE1
  • BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
    SEXECXXDEMANGLE.EXE1
  • BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
    SEXECXXLINK.EXE1
  • IPL31gt dir cxx.bck/siz

42
  • IPL31gt dir cxx.exe/siz
  • Directory SYSCOMMONSYSEXE
  • CXXCOMPILER.EXE1 39.68MB
  • CXXDEMANGLE.EXE1 41KB
  • CXXLINK.EXE1 166KB
  • Total of 3 files, 39.89MB
  • IPL31gt backup cxx.exe cxx.cbck/sav/datacomp/log
  • BACKUP-W-NOFILES, no files selected from
    DSA5SYS0.SYSEXECXX.EXE
  • BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
    SEXECXXCOMPILER.EXE1
  • BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
    SEXECXXDEMANGLE.EXE1
  • BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
    SEXECXXLINK.EXE1
  • BACKUP-I-COMPRESS, data compressed by 66
  • IPL31gt dir cxx.cbck/siz

Compression support in action
43
Encryption support in BACKUP
  • BACKUP uses the Encrypt LP for data encryption
  • How does it work?
  • At run-time, BACKUP generates a random encryption
    key used for encrypting the save-set records
  • The encryption key is being encrypted using a
    user provided key and stored in the save-set
    header
  • Decryption the encryption key is retrieved by
    decrypting the key stored in the header using the
    user provided key
  • BACKUP/ENCRYPT()
  • NAME
  • ALGORITHM
  • VALUE

44
Encryption support
  • Existing limitations
  • Uses DESCBC algorithm (old, slow expensive)
  • User provided algorithm is only used for
    encrypting the key
  • DESCBC is still used for encrypting the data
  • V8.3 adds AES encryption support to BACKUP
  • Modern stronger encryption
  • User provided algorithm is used for encrypting
    the data
  • DESCBC is still used by default (to maintain
    backword compatibility)
  • The following algorithms supported by BACKUP
  • AESCBC128, AESCBC192, AESCBC256, AESECB, AESCFB
    and AESOFB

45
Encryption support - examples
  • Create an encrypted save-set, specify the
    encryption key at run-time
  • backup .com coms.bck/sav/encryptalgorithmaes
  • Enter key value
  • Verification
  • Create an encrypted save-set, create the
    encryption key from DCL
  • encryp/create_key guy "This is a nice key
    called guy"/aes
  • backup .com coms.bck/sav/encrypt(algaes,name
    guy)
  • backup coms.bck/sav /encrypt(algaes,nameg
    uy)

46
Compaction Encryption
  • Compaction Encryption require additional CPU
    work
  • Elapsed time will be impacted

47
Saving 2 GB file
rx2600 1.3 GHz OpenVMS V8.3 HSG80
Seconds to save a file (Less is better)
48
BACKUP DVE
  • OpenVMS V7.3-2 added Dynamic Volume Expansion
    (DVE) support
  • Two new terms introduced Logical Volume size and
    Expansion Volume size
  • Controlled by the /SIZE /LIMIT qualifiers to
    the INITIALIZE SET VOLUME commands
  • Error count 0 Operations
    completed 8936
  • Owner process "_VTAT7" Owner UIC
    NPAR_BUILD
  • Owner process ID 2817CDDC Dev Prot
    SRWPL,ORWPL,GR,W
  • Reference count 2 Default
    buffer size 512
  • Current preferred CPU Id 0 Fastpath
    1
  • Total size 33.91GB Sectors
    per track 96
  • Total cylinders 7719 Tracks
    per cylinder 96
  • Logical Volume Size 2.38GB Expansion
    Size Limit 190.96GB
  • Allocation class 5

49
BACKUP DVE
  • BACKUP had no knowledge about DVE
  • DVE characteristics of a device were lost when
    image backup performed
  • Customers had to manually set DVE characteristics
    (assuming they noticed it was lost -)
  • BACKUP now fully supports DVE
  • VMS732_BACKUP-V0600

50
BACKUP DVE Expansion size
  • The volume expansion size is being recorded in
    the save-set header
  • BACKUP/LIST displays the expansion size if it
    exists in the save-set
  • When restoring a save-set (using /image) or
    performing disk-to-disk image backup, the target
    device inherits the expansion size limit of the
    input device
  • New qualifiers
  • /IGNORELIMIT prevents the target device from
    inheriting the expansion size
  • /LIMITn added to allow overriding the expansion
    size stored in the save-set header
  • Corresponding to INIT/LIMIT

51
BACKUP DVE Logical size
  • By default logical size is not preserved
  • Restoring image backup of 4GB disk to a 36GB disk
    will only result in 4GB of usable disk space
  • BACKUP/SIZE
  • Instructs BACKUP to preserve the logical volume
    size during a restore operation
  • BACKUP/SIZEn
  • Instructs BACKUP to initialize the target device
    to have a logical volume size of n

52
Questions?
  • See us at www.maklee.com for
  • Performance improvements
  • Oracle Tuning
  • Platform Migration
  • Custom Engineering solutions
  • Custom Training
Write a Comment
User Comments (0)
About PowerShow.com