Tuning and Capacity Planning for WebLogic Server

1
Tuning and Capacity Planning for WebLogic Server

• by
• Doug Leeper, Technical Manager
• Chicago, Professional Services
• 03/12/2002

2
Covered Topics

• Tuning
• What do I look at first?
• What can be tuned?
• Debugging
• Core dumps
• Active Processes
• Shared libraries
• Signals and Error numbers
• Tuning References
• Exercise in Capacity Planning
• Stake in the ground
• Thread based model

3
A little about myself

• Technical Manager within BEA Professional
  Services based in Chicago
• Founder of Bauhaus Technologies which was
  acquired by BEA in Oct 2000
• Over 11 years of experience designing and
  developing large scale enterprise applications
• Industries have spanned finance, banking, credit
  institutions, airline and others in varying
  segments
• Specialties include OO Analysis, Design and
  Development and troubleshooting and tuning.

4
Tuning Checklist

• Operating System
• Network and Hardware
• Database
• WebLogic Server
• Application

5
Few Notes

• Solaris is used for examples. Other Unix OS have
  similar commands/tools. Check with respective OS
  documentation or expert.
• For more information about Unix tools, please
  refer to man pages or consult system
  documentation. Some I have used and others have
  been recommended.
• Need to employ Administrators at each layer
  (System, Network, Database, and Software
  Administrator) during tuning exercises.
• Most tuning problems will not be found unless
  load testing is applied. Perform load testing as
  early as possible.

6
Tuning Solaris

• RAM
• Monitor Scan Rate and Paging with vmstat 30, Use
  iostat to monitor IO on swap devices!
• MemTool (http//www.solarisinternals.com)
• CPU Utilization (Too much or Not enough)
• Run queue, run queue, run queue
• vmstat 30 (r / CPUs gt4 and idle lt 15)
• I/O
• Look for a busy device with high service times
  (iostat -x 30 svc_t gt 50 and b gt 20)
• Look for a busy SCSI bus (iostat -x 30 w gt 5)
• vmstat -s (total name lookups lt 90) try
  increasing ncsize
• sar -g 30 (shows ufs_ipf gt 0) try increasing
  ufs_ninode

7
Tuning Solaris cont.

• How do I look at network I/O ? Try snoop(1),
  netstat(1).
• Lock Contention
• mpstat(1) shows smtx (spins on mutex per CPU)
• Ensure latest patches have been applied

8
Utilities

• uname -a shows basic info such as SunOS release
• /etc/sysdef -i can be used to show tunable values
• prtconf(1) shows device configuration
• psrinfo(1) and psradm(1) show CPUs and allow
  state change (on-line, off-line, no-intr)
• psrset(1) allows creation of processor groups and
  binding of processes to CPUs or sets of CPUs
  (2.6)
• nice(1) and priocntl(1) will modify scheduling
  priority
• ps -lfLp pid (shows all info about JVM LWPs)
• mpstat 30 (show locks and interrupts per CPU)

9
Utilities cont.

• vmstat 30
• run queue
• paging activity and scan rate
• cpu system, user, idle (ideal is about 15/75/10
  but there are exceptions. NFS servers have high
  system time.) vmstat -s (Since boot stats)
• vmstat -c (Hardware Address Translation Cache)
• /etc/swap -l and /etc/swap -s
• /usr/proc/bin/pfiles pid (Monitor open files and
  sockets)
• /usr/proc/bin/pmap -x pid (Monitor resident sizes)

10
Utilities cont.

• sar (sar -u shows wio)
• iostat -x 30
• Use /etc/path_to_inst to map device names or use
  iostat -xn on Solaris 2.6
• traceroute(1M) is bundled with Solaris 2.7
• plimit(1) allows root user to set active process
  limits (e.g. plimit -n 1024,4096 sets fd
  limits)

11
Truss and sotruss

• truss(1) traces process system calls
• -u option traces user functions
• -d option provides time stamps for profiles
• -D provides time deltas
• sotruss(1) monitors system call activity in
  shared libraries
• Can be used to determine what WebLogic is doing
• System calls can be matched to our source code to
  help isolate problem area

12
Solaris Tuning References

• Books
• SUN Performance and Tuning by Cockcroft Pettit
  (the porche book)
• Solaris Internals (Core Kernel Architecture) by
  Mauro McDougall
• Web Sites
• www.sun.com
• www.sunworldonline.com / www.unixinsider.com
• www.setoolkit.com
• www.solarisinternals.com
• www.sunfreeware.com
• http//www.sean.de/Solaris/tune.html

13
Tuning Database

• Index utilization
• Are the SQL requests efficient?
• Can indexes be added or removed to gain
  performance?
• Physical Data Model
• Raw device utilization
• Table striping across many disks
• Segment sizes and hash algorithms
• RDBMS Tuning
• Shared Memory (Solaris offers Intimate Shared
  Memory)

14
Tuning WebLogic Server

• Threads
• Goal should be least number that allows full CPU
  utilization or 50 if strict fail over
  requirement when employing multiple machines
• Monitor with vmstat 30 and/or mpstat 30
• Ideal maximum CPU utilization is when run queue
  lt 4 of CPUS and idle lt 10
• Heap
• Goal minimum that allows server to handle high
  load situations with out java.lang.OutOfMemoryErro
  r
• -Xms and -Xmx the same if possible
• Native IO
• Default for Solaris but some have it off
• WL_HOME/lib/solaris/libmuxer.so

15
Tuning WebLogic Server cont.

• JDBC Drivers
• Type 2 should give better performance but raises
  the possibility of server crashes. Open CRs on
  Oracle jDriver and core dumps.
• Oracle and other RDBMS vendors have Type 2 and
  Type 4 drivers that will work with WebLogic.
• Initial/Max sizing (same if possible to avoid
  connection time)
• Minimize connection testing
• EJBs
• Enable pass by reference or utilize
  LocalInterfaces (EJB2.0)
• WLS 6.x Database vs. Exclusive concurrency
• dbIsShared
• Fine grain vs Course grained access

16
Tuning Application

• Biggest gains found at the application level
• Typically 90 of performance problems related to
  poor design and/or implementation
• Design is critical
• Need to know behaviors of J2EE components and how
  to best use them
• Reducing RDBMS requests (fine grain vs. course
  grain EJB access)
• DOM vs SAX XML parsing
• Implementation is also key
• Visit Java Developers Connection, The Server Side
  and many others on-line resources for best
  practices for implementation optimization
• Code profiling to identify and tune most utilized
  routines (profile architectural layers to
  identify potential bottlenecks)

17
Tuning Application cont.

• Changes impact platform tuning
• Impacts RDBMS usage
• Impacts WLS Services
• Impacts OS utilization (Memory, CPU, I/O, Lock
  Contention)
• Impacts Network and hardware utilization

18
Tuning Summary

• Tuning implies multiple areasnot just one
• Many tools to be utilized to help find
  bottlenecks and problem areas
• No one tool is the end all solution
• Design and implementation are the most critical
  factors that affect performance
• Perform load testing early as possible to
  identify problem areas

19
Capacity Planning Methodology

• Stake in the ground
• An attempt to put some science into the art
• A good starting point
• Numerous assumptions
• Thread based algorithm
• Determines expected of CPU
• Utilizes best practices

20
Capacity Planning Methodology cont.

• Required Input (expected peak)
• of concurrent business transactions
• of WLS transaction in a business transactions
• Transaction definition is thread of execution in
  WLS
• A single thread can span multiple J2EE components
• JSP -gt EJB -gt JMS
• JMS -gt EJB
• Transaction response time
• Defined by business stakeholders
• Technologists must achieve this
• of concurrent user sessions
• of stateful EJB and amount of memory stored in
  Http Sessions

21
Capacity Planning Methodology cont.

• Best Practices
• Depending upon CPU and WLS version
• 1 CPU 1 WLS 6.x instance
• 2 CPU 1 WLS 5.x instance
• 15 100 Threads per CPU (context switching)
• Should change as CPU processing power increases
• Memory (RAM)
• 80 of remaining memory (after OS and other
  critical applications) divided between number of
  WLS instances on a machine

22
Capacity Planning Methodology cont.

• Algorithm
• CPU WLS instances TPS / (TC / TRT)
• TPS Transactions per Second
• TC Thread Count
• TRT Transaction Response Time
• TPS is a business requirement and not actual
  numbers. It usually represents the worst case
  scenario.
• TC is the only variable that can be adjusted.
  Adjusting this number affects system response
  time, aka CPU utilization.
• A different perspectivewhat is the maximum
  transaction throughput (TPS) for a single thread
  if the transaction is 2 seconds? Answer .5
  txn/s
• REMEMBER STAKE IN THE GROUND!!!! TO PROVE,
  APPLY LOAD TO YOUR SYSTEM!!!

23
Example Capacity Planning

• See document

24
Anecdotes

• Development Methodology
• Prove it can be done
• Get it to work
• Get it to perform
• Get it to scale
• Add the flash