Title: Before%20we%20start
1(No Transcript)
2A New Star Is Born
- Before we start
- Introducing Maklee Engineering Solutions
- High quality services in the area of
- Performance improvements
- Oracle services / SQL tuning
- Platform migration
- Programming / Custom engineering
- Custom training
3OpenVMS Backup V8.3 New Features
- Guy Peleg
- President
- Maklee Engineering guy.peleg_at_maklee.com
4Agenda
- Main Algorithms
- Performance considerations
- V8.3 New Features
530,000 feet overview
- Reads data from disk into buffers in memory
- Writes buffers from memory to target saveset
- Savesets are sequential files
- Fixed length record
- RMS is used when writing/reading savesets to/from
disks - SYSQIO is used when writing/reading savests
to/from tape drives
6BACKUP
- BACKUP.EXE
- DCL Utility to parse qualifiers and call the main
shareable image - SYSUPDATEBACKUP.CLD
- BACKUPSHR.EXE
- parses API and branches to action routines
- BACKUP operates in 3 main phases
- Init phase
- Open File phase
- Map I/O phase
7Init Phase
- Decides on useable buffer space
- WSQUOTA - WSQUOTA/64 300 JPI_FILCNT 512
- Creates each buffer using EXPREG
- Watch out This is R/W virtual address space
charged against PGFLQUOTA - Opens and reads in first directory
8Open File Phase
- Looks for the next input file
- Open file and read file header for mapping
pointers - Map file to buffers
- Maps each file extent
- Breaks down I/O to fill a buffer
- Max. buffer size is 63.5 KB
- Even for a nicely contiguous file
- Stops when either FILLM has been reached or all
buffers have file I/O mapped (WSQUOTA) - No file data I/O at this point
- No I/O to output save set at this point
9Map and I/O Phase
- Sort all mapped I/Os by increasing LBN numbers
- Without it performance decreases by 10
- Submit I/Os in LBN order in a tight loop
- Pre V8.3 algorithm (or latest V7.3-2, V8.2 ECOs)
- If DIOLM is reached wait 10ms
- Large DIOLM significantly impacts performance
- Wait for I/Os to complete
- Write data to output device
- Goto phase 1 if last buffer written
10Disk Queue Load
- BACKUP uses a large buffer pool to read file data
in an optimized manner - Algorithm was designed 20 years ago
- All reads for the entire buffer pool are issued
concurrently - Over time buffer pools have gotten larger the
I/O subsystems tolerance of being flooded with
large number of I/Os has decreased - Especially true with EVA XP storage controllers
- A picture is worth a thousand words.
11FC Queue Depth
Very high peak, BACKUP blocks production I/O
12FC Queue Depth Better Balance
Low peak, BACKUP no longer blocks production I/O
13Port Queue Depth vs. Read KBS
Note the Read KBS (blue) value on this Port
14Port Queue Depth vs. Read KBS
Read KBS on this Port remains the same. Lower
DIOLM value keeps a similar BACKUP throughput.
15Impact of High FC Queue Depth
During periods of high queue depth, VDisk
Latencies exceed 100 milliseconds (thats
1/10th of a second!!)
16Impact of Lowered FC Queue Depth
With lowered queue depth, VDisk Latencies only
once exceed 10 milliseconds (thats 1/100th of
a second much better)Production I/O workload
does not suffer due to BACKUP jobs.
17Disk Queue Load
- Pre V7.3-2 V8.2 (plus patches) behavior
- issue as much I/Os possible allowed by DIOLM
- Continue issuing I/Os until we hit SS_EXQUOTA
- Wait for I/Os to complete and flood the I/O
subsystem again - The New algorithm
- Issue the disk reads in n parallel AST threads
- The completion AST of each thread issues the next
I/O - By default use 8 parallel I/O threads
- Number of threads controlled by new /IO_LOAD
qualifier
18Disk Queue Load
- Better performance
- Idle EVA controller showed 15 reduction in
elapsed time - Results are not linear busy controllers will
witness more significant (dramatic !) improvement - Direct attached SCSI disk showed 5 improvement
- YMMV
19Performance Considerations
20Performance Considerations
- Why BACKUP performance is an issue?
- Only 24 hours a day for incremental BACKUP
- Small window for backup
- Even smaller window for restore in case of
emergency - Always test your restore procedures !
21Process Quotas
- System Managers Manual Volume 1 Essentials
- Chapter 11.7
- WSQUOTA
- Initial value of 10,000 pagelets, increment by
5,000 - Over 100,000 performance decreases
- FILLM
- Equal to 100 (mostly sufficient), increment by 10
- Use larger values if input disk contains many
very small files or is highly fragmented - Effectiveness limited by WSQUOTA
- DIOLM
- Equal to 100 (mostly sufficient), increment by 10
- 8 for Modern storage controllers
- Larger values can cause disk I/O subsystem hangs
or resets without any performance advantage
22Process Quotas
Second option is 15 faster
- Less is more
- DIOLM32767, BIOLM32767, WSDEFWSQUOTAWSEXTENT3
000000 VS. - DIOLM8, BIOLM150, WSDEF10000,
WSQUOTA20000, WSEXTENT30000
23Media Compaction
- INIT/MEDIACOMPACTION
- BACKUP/MEDIA_FORMATCOMPACTION
- Compaction performed by the hardware
- Default is no compaction
- Compaction ratio depends on data and drive
- Tape drives are optimized to use compaction
24SDLT 320 Compaction Performance
MBs Per Second (More is better)
25Performance Considerations
- BACKUP / GROUP
- Defines the number of blocks backup places in
each redundancy group. - /GROUP_SIZE 10 is the default
- Modern drive technology makes the XOR group
feature obsolete - /GROUP_SIZE 0 to save 10 tape space
- BACKUP/CRC
- Stores CRC in each backup buffer
- Does not increase the tape utilization as cell is
part of the record header - Increases CPU consumption
- With modern tape drives consider using /NOCRC
(but need to understand the possible risk)
26Performance Considerations
- Tape drives are getting faster.
- Ultrium-960 can write _at_160 MB/sec
- With Ultrium drives elapsed time for Backup will
be shorter but during this time the CPU may get
overloaded (calculating CRC) - 90 CPU utilization on DS25 writing to
Ultrium-460 drive (_at_ 40MB/sec) - May impact the availability of other applications
on the system
27Performance Considerations
- /BLOCK_SIZE
- Specifies the size of output block size in bytes
- Larger blocks -gt less I/Os -gt better performance
- Default of 32256 when writing to disks
- Default of 8192 when writing to tapes
- Modern tape drives optimized for large I/Os use
maximum supported block size 65535 - BACKUP is optimized to save large files
- Same amount of data in small files vs. one large
file may show up to 40 times difference.
28RMS Optimizations
- SET RMS
- /BLOCK127 124
- /BUFFER 8
- /EXTEND65535 (for large savesets)
- /NETWORK127
- RMS_SEQFILE_WBH (Prior to V8.3)
- With recent versions VMS now sets RAH WBH
29BACKUP/FAST
- Fast file scan
- Large number of input files
- Small number of files meet selection criteria
Elapsed time to backup all files created today
(less is better)
30New Features
31Improved CTRL-T information
- Traditional CTRL-T information showed the name of
the current file being saved/restored and the
total number of save-set blocks processed - backup IA64KITSI64XB37.BCK/sav
5dka100/ima - MIKAXP_VTA61 143638 BACKUP
CPU000000.86 PF908 IO2192 MEM256 - Restoring file 5DKA100DWMOTIF_SUPPORT_I64XB3
7.KITHP.SICOMPRESSED1 - Saveset volume1, saveset block266 (32256 byte
blocks) - Can you tell how much data restored so far?
- Can you tell when the restore will be done?
- Introducing the new improved CTRL-T
32Improved CTRL-T information
- IPL31_VTAT7 144627 BACKUP CPU000003.40
PF6298 IO18408 MEM465 - Restoring file DKA100DWMOTIF_SUPPORT_I64XB37.K
ITHP.PCSICOMPRESSED1 - Saveset volume1, saveset block720 (32256 byte
blocks) - 22.14MB restored out of 1.18GB, 1 completed
- Restore rate 965KB/sec, estimated completion
time 150731.85
33Improved CTRL-T information
- Note the message says ESTIMATED !!!
- Fancy CTRL-T message displayed when
- Restoring a saveset
- Creating an image backup
- Counters are updated when a file marked nobackup
is encountered - When the amount of data to be saved is unknown,
only the rate and total amount of data processed
so far is displayed - CTRL-T can not be used in batch.and BACKUP is
usually running in batch
34BACKUP/PROGRESS_REPORTn
- /PROGRESS_REPORTS writes CTRL-T style message to
the output device every given interval - n is the number of seconds between intervals
- back IA64KITSI64XB37.BCK/sav
dka100/ima/progress10 - BACKUP-I-PROGRESS, progress report generated at
4-JAN-2006 150054.47 - Restoring file DKA100DWMOTIF_SUPPORT_I64XB37.K
ITHP1.PCSICOMPRESSED1 - Saveset volume1, saveset block170 (32256 byte
blocks) - 5.22MB restored out of 1.18GB, 0 completed
- Restore rate 535KB/sec, estimated completion
time 153928.28
35lost saveset attributes
- A saveset transferred using FTP or compressed and
decompressed using ZIP will lose its RMS
attributes - An attempt to process the saveset will fail
- BACKUP-F-NOTSAVESET
- Fortunately the correct RMS settings are stored
in the saveset header
36BACKUP / REPAIR
- /REPAIR instructs BACKUP to attempt and restore
the correct RMS attributes - backup images.bck/sav .exes/repair
- BACKUP-I-REPAIRED, saveset attributes changed to
RFMFIX, MRS32256 - IPL31GUY 145858 BACKUP CPU000010.89
PF7765 IO71628 MEM409 - Restoring file UPS000000.EXESBACKUPSHR.EXE16
37Compression Support in BACKUP
- BACKUP supports data compression using ZLIB.
- ZLIBs advantages
- Free
- Legally unencumbered
- May be included in commercial applications
- Lossless data compression library
- Never expands the data
- Unlike DCX may be used to compress stream of
bytes - Does not need to analyze the file in advance
- Maximum compression factor 11000
- 50MB file filled with zeros compressed to 49KB
- Realistic numbers are in the range of 21 to 51
- http//www.zlib.net
38Compression support in BACKUP
- ZLIB routines are shipping in a new shareable
image - SYSLIBRARYCOMPRESSSHR.EXE
- SYSLIBRARYCOMPRESSSHR_EV6.EXE (alpha only)
- New qualifier - /DATA_FORMATCOMPRESSED
- Instructs BACKUP to create a compressed save-set
- Does not need to be specified on the restore
command - Initially viewed as a feature to save space.but
turned out to be a significant performance
improvement (wellat least in some cases)
39Compression support in BACKUP
- The slower the I/O subsystem the bigger the
performance win - Big win for network operations
- Big win for MSCP served devices
- doubles the CPU consumption
- On average BACKUP completes 2-5 times faster
- Useful when shipping dump files
- 800MB dump file was compressed to 75MB
- No support for writing compressed savesets to
tapes - Do not encrypt compress..at least for now
40Compression support in BACKUP
- Compressed save-sets have variable length records
- Instead of fix
- First record is not compressed
- Detect compressed savesets
- Force largest record value of /BLOCK_SIZE
- To fix attributes after file transfer
- SET FILE X.CBCK/ATTRIB(RFMVAR,LRL32256,MRS322
56)
41- IPL31gt dir cxx.exe/sizall
- Directory SYSCOMMONSYSEXE
- CXXCOMPILER.EXE1 39.68MB/39.68MB
- CXXDEMANGLE.EXE1 41KB/42KB
- CXXLINK.EXE1 166KB/166KB
- Total of 3 files, 39.89MB/39.89MB
- IPL31gt backup cxx.exe cxx.bck/sav/log
- BACKUP-W-NOFILES, no files selected from
DSA5SYS0.SYSEXECXX.EXE - BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
SEXECXXCOMPILER.EXE1 - BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
SEXECXXDEMANGLE.EXE1 - BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
SEXECXXLINK.EXE1 - IPL31gt dir cxx.bck/siz
42- IPL31gt dir cxx.exe/siz
- Directory SYSCOMMONSYSEXE
- CXXCOMPILER.EXE1 39.68MB
- CXXDEMANGLE.EXE1 41KB
- CXXLINK.EXE1 166KB
- Total of 3 files, 39.89MB
- IPL31gt backup cxx.exe cxx.cbck/sav/datacomp/log
- BACKUP-W-NOFILES, no files selected from
DSA5SYS0.SYSEXECXX.EXE - BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
SEXECXXCOMPILER.EXE1 - BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
SEXECXXDEMANGLE.EXE1 - BACKUP-S-COPIED, copied DSA5SYS0.SYSCOMMON.SY
SEXECXXLINK.EXE1 - BACKUP-I-COMPRESS, data compressed by 66
- IPL31gt dir cxx.cbck/siz
Compression support in action
43Encryption support in BACKUP
- BACKUP uses the Encrypt LP for data encryption
- How does it work?
- At run-time, BACKUP generates a random encryption
key used for encrypting the save-set records - The encryption key is being encrypted using a
user provided key and stored in the save-set
header - Decryption the encryption key is retrieved by
decrypting the key stored in the header using the
user provided key - BACKUP/ENCRYPT()
- NAME
- ALGORITHM
- VALUE
44Encryption support
- Existing limitations
- Uses DESCBC algorithm (old, slow expensive)
- User provided algorithm is only used for
encrypting the key - DESCBC is still used for encrypting the data
- V8.3 adds AES encryption support to BACKUP
- Modern stronger encryption
- User provided algorithm is used for encrypting
the data - DESCBC is still used by default (to maintain
backword compatibility) - The following algorithms supported by BACKUP
- AESCBC128, AESCBC192, AESCBC256, AESECB, AESCFB
and AESOFB
45Encryption support - examples
- Create an encrypted save-set, specify the
encryption key at run-time - backup .com coms.bck/sav/encryptalgorithmaes
- Enter key value
- Verification
- Create an encrypted save-set, create the
encryption key from DCL - encryp/create_key guy "This is a nice key
called guy"/aes - backup .com coms.bck/sav/encrypt(algaes,name
guy) - backup coms.bck/sav /encrypt(algaes,nameg
uy)
46Compaction Encryption
- Compaction Encryption require additional CPU
work - Elapsed time will be impacted
47Saving 2 GB file
rx2600 1.3 GHz OpenVMS V8.3 HSG80
Seconds to save a file (Less is better)
48BACKUP DVE
- OpenVMS V7.3-2 added Dynamic Volume Expansion
(DVE) support - Two new terms introduced Logical Volume size and
Expansion Volume size - Controlled by the /SIZE /LIMIT qualifiers to
the INITIALIZE SET VOLUME commands - Error count 0 Operations
completed 8936 - Owner process "_VTAT7" Owner UIC
NPAR_BUILD - Owner process ID 2817CDDC Dev Prot
SRWPL,ORWPL,GR,W - Reference count 2 Default
buffer size 512 - Current preferred CPU Id 0 Fastpath
1 - Total size 33.91GB Sectors
per track 96 - Total cylinders 7719 Tracks
per cylinder 96 - Logical Volume Size 2.38GB Expansion
Size Limit 190.96GB - Allocation class 5
49BACKUP DVE
- BACKUP had no knowledge about DVE
- DVE characteristics of a device were lost when
image backup performed - Customers had to manually set DVE characteristics
(assuming they noticed it was lost -) - BACKUP now fully supports DVE
- VMS732_BACKUP-V0600
50BACKUP DVE Expansion size
- The volume expansion size is being recorded in
the save-set header - BACKUP/LIST displays the expansion size if it
exists in the save-set - When restoring a save-set (using /image) or
performing disk-to-disk image backup, the target
device inherits the expansion size limit of the
input device - New qualifiers
- /IGNORELIMIT prevents the target device from
inheriting the expansion size - /LIMITn added to allow overriding the expansion
size stored in the save-set header - Corresponding to INIT/LIMIT
51BACKUP DVE Logical size
- By default logical size is not preserved
- Restoring image backup of 4GB disk to a 36GB disk
will only result in 4GB of usable disk space - BACKUP/SIZE
- Instructs BACKUP to preserve the logical volume
size during a restore operation - BACKUP/SIZEn
- Instructs BACKUP to initialize the target device
to have a logical volume size of n
52Questions?
- See us at www.maklee.com for
- Performance improvements
- Oracle Tuning
- Platform Migration
- Custom Engineering solutions
- Custom Training