Performance SeNTry: Development Status Report

1
Performance SeNTry Development Status Report

• Demand Technology Software
• A DataCore Software company
• 1020 Eighth Avenue South, Suite 6, Naples, FL
  34102
• phone [phone] fax [phone]
• e-mailsupport_at_demandtech.com
• http//www.demandtech.com

2
Agenda

• Performance SeNTry version 2.4.2 highlights
• NTSMF Collector
• SeNTry Administration
• Version 3 roadmap
• Alerts and Traps
• File System Collector
• Improved Systems Console for real-time
  notification, management, and control

3
Performance SeNTry version 2.4.2Topics

• Windows 2000 features
• NTSMF features
• SeNTry Administration features
• Event log use
• New Enhancements SQL Server 2000
• New Enhancements - Exchange 2000 support
• New collection service architecture
• Threaded collector
• Improved Exception handling

4
Performance SeNTry version 2.4.2Windows 2000
features

• System level objects now in visible performance
  libraries
• System object missing some NT4 counters
• Total Processor Time, Total User Time,
  Total Privileged Time, Total DPC Time, Total
  Interrupt Time, Total Interrupts/sec
• Delay in some objects reporting data at system
  start going through RegQueryValueEx()
• New values in Performance Registry keys
• Timeouts
• DisablePerformanceCounters

5
Performance SeNTry version 2.4.2 New fields in
Registry HKLM\\Services\PerfDisk\Performance
6
Performance SeNTry version 2.4.2 Additional
NTSMF features

• Install option to run under a user account
• -install -account ltaccount namegt -password
  ltpasswordgt
• Program defaults with filters active - NT4 and
  Win2K have specific sets
• Memory size field in header fixed for Win 2000
  systems with 4GB or more of physical memory
• No fix for Windows NT4 systems with 4GB or more
  of physical memory since the GlobalMemoryStatus
  api call returns only a 32 bit number

7
SeNTry Administration enhancements

• New default DCS for Win2K and Performance Gallery
  Gold templates
• SQL Server 2000 object support
• MS Exchange 2000 objects support
• Tool to compact the Performance SeNTry Database
  ('Tools' menu item in the 'Network Browser'
  window)
• More filters
• More options

8
SeNTry Administration enhancementsCollection
Filters - Process

• Process
• New Process Filter on Process Working Set Size
• Always write named processes availability
  reporting
• Threads
• Add 'Ready Threads' counter in the 'Process'
  object
• No longer need to write Thread objects into .smf
• Terminal Server
• suppress Idle User record
• write User and Session records (automatic) only
  when activity occurs

9
Performance SeNTry version 2.4.2Collection
filters, Thread
10
SeNTry Administration enhancementsParameters
Administration, Collection Cycle

• On Error Command capability for launching
  commands when errors are encountered
• Same rules as Cycle End commands
• poor mans alert something as simple as Net
  Send user-name message

11
SeNTry Administration enhancementsParameters
Administration, Scheduling

• Flexible DCS chaining specifying Next DCS in
  chain
• (e.g., prime shift.DCS chains to nighttimeDCS,
  which chains back)
• Collection Period to specify the period that the
  collector is active and reporting data
• Provide basic scheduling options without
  requiring a 3rd party package

12
Performance SeNTry version 2.4.2Parameters
Administration, File Contents

• Support for summarization program
• Collect Inventory data from Registry keys

13
Performance SeNTry version 2.4.2Parameters
Administration, File Contents

• New option Write denominator counters
• Used by .smf Summarization (DMSumSMF.exe) uses in
  calculations to compute correct average values of
  compound Counters
• e.g., Copy Read Hits
• One more option to manage the volume of data that
  can be collected effectively
• Base appended to denominator counter name
  strings for identification in .smf file

14
Performance SeNTry version 2.4.2Parameters
Administration, File Contents

• Summarization utility (based on Counter Type)
• Requires Type 6 records!
• Designed to run at Cycle End, but can be run
  standalone
• Creates sum.smf files
• Identical in format to detail records
• aged like any other .smf file in
  Current/Previous/Archive folders
• Calculates the arithmetic mean for most Counters
• Calculates interval peak (min, max) where
  appropriate
• Some counters simply save the last observed value
• (e.g. elapsed time)

15
SeNTry Administration enhancementsParameters
Administration, File Contents

• Include Registry Key Values to get registry key
  value information into configuration records in
  the SMF data file.
• Used to collect hardware and software inventory,
  versioning, etc.
• No restrictions inventory as many values as you
  need
• Must specify the full Registry key name and field
  name
• Use in conjunction with regedit Copy Key Name
  function
• Key Not Found earns a Warning message, but no
  other side effect
• Invalid spec earns a Warning message, but no
  other side effect

16
SeNTry Administration enhancementsParameters
Administration, File Management

• PKZIP compression option at Cycle End
• Another way to manage data volume

17
Performance SeNTry version 2.4.2Event Log
messages

• Informational, Warning and Error messages are
  placed in the Application Event log
• Also in ltsystemgt.ntsmf.log (plain text)
• Date, time User, System, source filled in by
  event facility
• Type, category, event id, description defined by
  developers
• I and W will not

18
Performance SeNTry version 2.4.2Event Log
messages - Warning
19
Performance SeNTry version 2.4.2Event Log
messages ntsmf.log example
10/24/01-000001 - Event ID 210, Category
Discovery, Severity Warning The Performance
Library, "C\Program Files\Exchsrvr\bin\exprfdll.d
ll", for service "ExIPC" (ExIPC), failed to
load. The system error was "The specified module
could not be found (126)". Performance data for
this service will not be available. The Event
Viewer (source Perflib) may have more details
20
Performance SeNTry version 2.4.2Event Log
messages unhandled exception
10/13/01-000002 - Event ID 96, Category N/A,
Severity Error Unhandled exception
encountered 10/13/01-000002 - Ended Performance
SeNTry service 10/13/01-000002 - Event ID 104,
Category Initialization, Severity Info The
Performance SeNTry service stopped with error
code, -1073741728. Version2.3.6
21
Performance SeNTry version 2.4.2New Enhancements
SQL Server 2000

• Lots of new SQL Server 2000 records
• A single machine can be run multiple instances of
  SQL Server
• No instance names for separate SQL Server
  processes, as delivered by MS
• MSSQLdb-nameobject-name
• e.g. MSSQLSUPBuffer Manager,
• MSSQLSUP1Buffer Manager,)
• Anticipate object selection and reporting
  problems
• We write a MSSQLBuffer Manager Object instance
  where the db-name token becomes an Instance name
• 2.2.2,0,5,2001,4,17,20,36,45,859,0,MSSQLBuffer
  Manager,1,18,22,1,SQL Server DB,Buffer cache hit
  ratio,

22
Performance SeNTry version 2.4.2New Enhancements
- Exchange 2000 support

• New objects visible when optional components
  installed (e.g. Instant Messaging)
• Cases where a perflib loads another .dll
• exprfdll.dll loads exchmem.dll
• Objects have many counters
• MSExchange Web Mail 75 counters
• MSExchangeIMAP4 96 counters
• MSExchangeIS 107 counters (actual)
• Database .. 127 Counters (many of no
  data type)

23
Performance SeNTry version 2.4.2New Architecture
- Threaded collector

• Separate thread for each performance library
  found in the Registry
• 3rd party Perflib DLLs perform the actual
  performance data gathering
• Windows 2000, XP expose the system perflibs,
  including perfos.dll, perfdisk.dll, perfproc.dll,
  perfnet.dll
• Improved error handling, especially in 3rd party
  code
• Additional collection service threads manage
  dispatching, file writes, other tasks

24
NT Performance monitoring API

• Perflib Dlls are loaded into the dmperfss process
  address space when opened
• 3rd party Perflib Dlls might need to synchronize
  with a separate monitored address space

25
NT Performance monitoring API

• Open and Collect Timeout values exceeded trigger
  DisablePerformanceCounters setting, which must be
  disabled manually!
• HKLM\SOFTWARE\Microsoft\Windows
  NT\CurrentVersion\Perflib ExtCounterTestLevel can
  be used to verify integrity of 3rd party Perflib
  dlls.

26
Performance SeNTry version 2.4.2New Architecture
- Exception handling

• Try to catch exceptions that occur in a thread,
  i.e., inside code in a Perflib DLL
• The failing thread and perflib (and the
  performance Objects it supplies) are deactivated,
  while other Objects continue to be collected
• Problem perflib is automatically retried every
  new cycle or on demand (Notify Command)
• No manual intervention required
• Unless the Object is on the critical list
• System, Memory, Cache, Process, Thread, Processor
• Critical missing means log and stop

27
Performance SeNTry version 2.4.2New Architecture
- Exception handling

• On exception
• system error text message
• the loaded modules with addresses
• and the call stack
• Helps to identify rogue Perflib dlls
• Very compact and accurate

28
Performance SeNTry version 2.4.2Log message and
module list
10/26/01-102005 - Event ID 1300, Category
PerfData, Severity WarningCould not collect
performance data for object, "Database gt
Instances (L28698, G3370), counter, Pages
Converted/sec (L28700, G3372)".The system error
was "Win32 exception, 0xC0000005, encountered at
location 0x0043FBB1Read attempt at location
0x00000020Module List(0x00400000-0x00480000)
D\NTSMFv24224\DMPerfss.exe(0x01000000-0x01017000
) F\WINNT\System32\BEPerfDll.dll(0x01140000-0x01
14E000) D\Program Files\Exchsrvr\bin\davexpc.dll
(0x01630000-0x0163B000) D\Program
Files\Exchsrvr\bin\eseperf.dll(0x02000000-0x02010
000) F\WINNT\system32\dssdata.dll(0x02C40000-0x0
2C46000) D\Program Files\Exchsrvr\bin\srsperf.dll
. (lots more of these)
29
Performance SeNTry version 2.4.2Stack Trace
Stack Trace (ThreadId 4540) DMPerfss.exe
PSSPerfLocalCounterRealizeCache 111 bytes
E\DmPerfss 2.4.2\Collector\PSSPerfLocalCounter.c
pp, Line 372 15 bytes DMPerfss.exe
PSSPerfObjectCacheRealizeCache 821 bytes
E\DmPerfss 2.4.2\Collector\PSSPerfObjectCache.cp
p, Line 247 11 bytes DMPerfss.exe
PSSPerfObjectCacheOnCollectionComplete 12
bytes E\DmPerfss 2.4.2\Collector\PSSPerfObjectCa
che.cpp, Line 408 9 bytes DMPerfss.exe
PSSPerfDataSinkCollect 597 bytes E\DmPerfss
2.4.2\Collector\PSSPerfDataSink.cpp, Line 395
14 bytes DMPerfss.exe PSSPerfDataSinkCollect
15 bytes E\DmPerfss 2.4.2\Collector\PSSPerfDataS
ink.cpp, Line 463 15 bytes DMPerfss.exe
PSSPerfObject_Collect 75 bytes E\DmPerfss
2.4.2\Collector\PSSPerfObject.cpp, Line 1688 26
bytes DMPerfss.exe PSSPerfObjectCollect 79
bytes E\DmPerfss 2.4.2\Collector\PSSPerfObject.c
pp, Line 1748 20 bytes
30
Performance SeNTry version 2.4.2Stack Trace
DMPerfss.exe PSSCollectionCycleStart 1483
bytes E\DmPerfss 2.4.2\Collector\PSSCollectionCy
cle.cpp, Line 1500 45 bytes DMPerfss.exe
PSSPerformanceSeNTryStartCollection 248
bytes E\DmPerfss 2.4.2\Collector\PSSPerformanceS
eNTry.cpp, Line 1571 11 bytes DMPerfss.exe
PSSPerformanceSeNTryRun 109
bytes E\DmPerfss 2.4.2\Collector\PSSPerformanceS
eNTry.cpp, Line 1254 22 bytes"
31
Performance SeNTry version 2.4.2 New
Architecture - Exception handling

• New dmperfss.exe ships with debug symbols
  included for extended error diagnostics report
• Size of the image file is now approximately 5.8
  MB
• The larger image file installs by default
• A dmperfss.exe without symbols is also on the CD
• E.g., a floppy disk distribution/install
• dmperfss.exe with symbols is preferred,
  especially if new applications or new versions
  are discovered

32
Performance SeNTry Version 3

• BREAK

33
Performance SeNTry Version 3 Roadmap

• Real-time Alerts
• Event log SNMP support
• File System Collector
• Revamped Systems Console

34
Performance SeNTry Version 3Alerts

• Alert is written into the Application event log
• Triggered by system condition (e.g. cpu busy
  threshold)
• Frequency of alerts is also thresholded wont
  fill up the Event Log unnecessarily
• Trigger, Release mechanism
• Alerts are cleared when condition is absent
• (based on the lower threshold)

35
Performance SeNTry Version 3SNMP - Traps

• Like Alerts, defined by same rules
• SNMP extension agent notified of the alert by
  dmperfss.exe
• SNMP extension agent generates a Trap message
• Trap is sent to management entities by SNMP
  facility
• Management entities would be OpenView, Tivoli,
• CA-Unicenter, others

36
Performance SeNTry Version 3File System Collector

• At prototype stage now
• Probably another thread in the collector
• Designed to run infrequently, say once/day
• (dont know how long it takes yet)
• Report files that are above threshold
• (e.g. file size gt 2 Logical disk size)
• Report folders that are above threshold when
  files in them are totaled
• (dont also report parents, just siblings)

37
Performance SeNTry Version 3File System Collector

• Summary records size of files remaining
• NTSMF header record format
• Pseudo object id
• Records are instances where the UNC path is the
  instance name
• Probably optioned - allow include in .SMF file or
  write to a file of its own

38
Performance SeNTry Version 3System Console

• Processes real-time alerts
• Can examine objects and counters related to the
  alerts
• Can change the alert definition, collection
  interval, etc., as needed for the display
• Display results graphically
• Capture the data and graphs for later review

39
Performance SeNTry Version 3Exchange 2000
references

• http//www.exchange-mail.org/whitepapers.html
• http//www.microsoft.com/technet/treeview/default.
  asp?url/technet/prodtechnol/exchange/reskit/ex00r
  es/resguide/c29monit.asp
• (Microsoft's online version of the Exchange
  Resource Kit Documentation)
• http//www.netiq.com/Downloads/Library/White_Paper
  s/NetIQ_Monitoring_and_Managing_Exchange.pdf

40
Performance SeNTryreach us at

• Demand Technology Software
• A DataCore Software company
• Address 1020 Eighth Avenue South, Suite 6,
  Naples, FL 34102
• Phone [phone] fax [phone]
• E-mail support_at_demandtech.com
• Web site http//www.demandtech.com

41
Performance SeNTry Version 3 Roadmap

• Questions?