Hardware Layouts for LAMP Installations

1
Hardware Layouts for LAMP Installations

• John Allspaw, Flickr Plumbr

Flickr (Yahoo)
jallspaw_at_yahoo.com
October 18, 2005
2
Hardware Layouts for LAMP Installations

• Hardware requirements for LAMP installs have to
  do with
• A decent amount about the actual hardware
  (in-box stuff)
• A bit more about the hardware architecture
• Which should complement the application
  architecture

3
Hardware Layouts for LAMP Installations

• What well talk about here
• Database (MySQL) layouts and considerations
• Some miscellaneous/esoteric stuff (lessons
  learned)
• Caching content and considerations

4
Hardware Layouts for LAMP Installations

• Growing Up, One Box solution
• Basic web application (discussion board, etc.)
• Low traffic
• Apache/PHP/MySQL on one machine
• Bottlenecks will start showing up
• Most likely database before apache/php
• Disk I/O (Innodb) or locking wait states (MyISAM)
• Context switching between memory work (apache)
  and CPU work (MySQL)

5
Hardware Layouts for LAMP Installations
ONE BOX
6
Hardware Layouts for LAMP Installations

• Growing Up, Two Box solution
• Higher traffic application (more demand)
• Apache/PHP on box A, MySQL on box B
• Same network bad (or is it ?), separate
  network good
• Bottlenecks with start to be
• Disk I/O on MySQL machine (Innodb)
• Locking on MyISAM tables
• Network I/O

7
Hardware Layouts for LAMP Installations
TWO BOX
8
Hardware Layouts for LAMP Installations

• Growing Up, Many Boxes with Replication
  solution
• Yet even higher traffic
• Writes are separated from reads (master gets
  IN/UP/DEL, slaves get SELECTs)
• Diminishes network bottlenecks, disk I/O, and
  other in-box issues
• SELECTs, IN/UP/DEL can be specified within the
  application,
• OR.
• Load-balancing can be used

9
Hardware Layouts for LAMP Installations
MANY BOX
10
Hardware Layouts for LAMP Installations

• Slave Lag
• When slaves cant keep up with replication
• Theyre too busy
• Reading (production traffic)
• Writing (replication)
• Manifests as
• Comments/photos/any user-entered data doesnt
  show up on the site right away
• So users will repeat the action, thinking that it
  didnt take the first time, makes situation
  worse

11
Hardware Layouts for LAMP Installations
Insert funny photo here about slave lag
slave lag isnt funny
12
Hardware Layouts for LAMP Installations

• Hardware Load Balancing MySQL

13
Hardware Layouts for LAMP Installations

• How Its Usually Done
• Standard MySQL master/slave replication
• All writes (inserts/updates/deletes) from
  application go to Master
• All reads (selects) from application go to a
  load-balanced VIP (virtual IP) spreading out
  load across all slaves

14
Hardware Layouts for LAMP Installations
15
Hardware Layouts for LAMP Installations

• What Is Good About Load Balancing
• you can add/remove slaves without affecting
  application, since queries are atomic
  (sorta/kinda)
• additional monitoring point and some automatic
  failure handling
• you can treat all of your slave pool as one
  resource, and makes capacity planning a lot
  easier if you know the ceiling of each slave

16
Hardware Layouts for LAMP Installations

• How do you know the ceiling (maximum QPS
  capacity) of each slave ?
• First make a guess based on benchmarking (or look
  up some bench results from Toms Hardware or
  anandtech.com, etc.
• Then get more machines than that )
• Scary in production during a lull in traffic,
  remove machines from the pool until you detect
  lag
• The QPS you saw right before slave lag set in
• THAT is your ceiling

17
Hardware Layouts for LAMP Installations
18
Hardware Layouts for LAMP Installations

• What Can Be Bad/Tough About Load Balancing
• not all load-balancers are created equal, not all
  load-balancing companies expect this product use,
  so support may still be thin
• not that many people are doing it in high-volume
  situations yet, so support from community isnt
  large either
• Gotchas
• port exhaustion,
• health checks,
• and balance algorithms

19
Hardware Layouts for LAMP Installations

• Port Exhaustion
• PROBLEM
• LB is basically a traffic cop, nothing more
• Side effect of having a lot of connections only
  64,511 ports per each IP (VIP) to use
• 64,511 ports/120 sec per port.
• 535 max concurrent connections per IP

Not really, but close to it tcp_tw_recycle
and tcp_tw_reuse
20
Hardware Layouts for LAMP Installations
21
Hardware Layouts for LAMP Installations

• Port Exhaustion (contd)
• SOLUTION
• Use a pool of IPs on the database slave/farm side
  (Netscaler calls these subnet IPs, Alteon calls
  them PiPs)
• Monitor port/connection usage, know when its
  time to add more

22
Hardware Layouts for LAMP Installations

• Health checks
• LB wont know anything about how well each MySQL
  slave is doing, and will pass traffic as long as
  port 3306 is answering
• Load balancers dont talk SQL, only things like
  plain old TCP, HTTP/S, maybe FTP

23
Hardware Layouts for LAMP Installations

• Health checks (contd)
• Two options
• 1. Dirty, but workable
• Have each server monitor itself, and shut
  off/firewall its own port 3306, even if MySQL is
  still running

24
Hardware Layouts for LAMP Installations

• Health checks (contd)
• 2. Cleaner, but a bit more work
• Have each server monitor itself, and run a check
  via xinetd (for example, a nagios monitor)
• So the LB can tickle that port, and expect back
  an OK string. If not, itll automatically take
  that server out of the pool
• Good for detecting and counteracting isolated
  incidents of slave lag and automatically
  handling it

25
Hardware Layouts for LAMP Installations
Health Checks
26
Hardware Layouts for LAMP Installations

• Balancing Algorithms
• Load balancers know HTTP, FTP, basic TCP, but not
  SQL
• Two things to care about
• Should the server still be in the pool ? (health
  checks)
• How should load get balanced ?
• least connections or least bandwidth or
  least anything BAD
• Because not all SQL queries are created equal
• Use round-robin or random
• What happens if you dont Evil Favoritism

27
Hardware Layouts for LAMP Installations
Evil Favoritism
28
Hardware Layouts for LAMP Installations
29
Hardware Layouts for LAMP Installations

• Meanwhile.for in-the-box considerations
• Interleaving memory does make a difference
• Always RAID10 (or RAID0 if youre crazy) but
  NEVER RAID5 (for Innodb, anyway)
• RAID10 has much more read capacity, and a write
  penalty, but not as much as RAID5
• Always have battery backup for HW RAID write
  caching
• Or, dont use write caching at all

30
Hardware Layouts for LAMP Installations

• IN-THE-BOX considerations (contd)
• Always have proper monitoring (nagios, etc.) for
  failed/rebuilding drives
• SATA or SCSI ? SCSI ! Its worth it!
• 10k or 15k RPM SCSI ? 15k! Its worth it!
• (20 performance increase when youre disk
  bound)
• For 64bit Linux (AMD64 or EM64T)
• Crank up the RAM for Innodbs buffer pool
• Swapping very very bad either
• Turn it off (slightly scary)
• Leave it on and set /proc/sys/vm/swapiness 0

31
Hardware Layouts for LAMP Installations

• 10k versus 15k drives ?
• Does it really matter that much ?
• Some in-the-wild proof.

32
Hardware Layouts for LAMP Installations
Slave Lag in production
10K drives
15K drives
33
Hardware Layouts for LAMP Installations

• Using MySQL with a SAN (Storage Area Network)
• Do layout storage same as if they would be local
• Do make sure that the HBA (fiber card) driver is
  well supported by Linux
• Dont share volumes across databases
• Dont forget to correctly tune Queue Depth Size,
  which should be increasing, from server HBA -
  switch - storage

34
Hardware Layouts for LAMP Installations

• Caching your static content

35
Hardware Layouts for LAMP Installations

• Caching Static Content
• SQUID good
• Relieve your front-end PHP machines from looking
  up data that will never (or rarely) change
• Generate static pages, and cache them in squid,
  along with your images

36
Hardware Layouts for LAMP Installations

• Caching Static Content (contd)
• Use SQUID to accelerate plain-old origin
  webservers, also known as reverse-proxy HTTP
  acceleration
• Described here and elsewhere
• http//www.squid-cache.org/Doc/FAQ/FAQ-20.html

37
Hardware Layouts for LAMP Installations
Basic SQUID layout

• squid accepts requests on 80
• passes on cache misses to apache on 81
• apache uses as its docroot an NFS mounted dir
• should be on local subnet, or dedicated net

38
Hardware Layouts for LAMP Installations

• Good HW layout for high-volume SQUIDing
• Do use SCSI, and many spindles for disk cache
  dirs
• Dont use RAID
• Do use network attached storage, or place the
  origin servers on separate machines
• Do use ext3 with noatime for disk cache dirs
• Do monitor squid stats

39
Hardware Layouts for LAMP Installations
Flickr How We Roll
40
Hardware Layouts for LAMP Installations

• Yummy SQUID stats
• 2800 images/sec, 75-80 are cache hits
• 10 million photos cached at any time
• 1.5 million cached in memory

41
Hardware Layouts for LAMP Installations

• The End