Squid Proxy Fest

1
Squid Proxy Fest
?e??µß???? 2004
2
?a??? ???ate...

• Node 799 ocean
• squid.ocean.awmn3128

3
???e?? ??aß??µ?s? !!!

• aDSL 384 -gt 48KB/s

• aDSL 384 Squid

aDSL 512 (64KB/s) !!!
1MB Data ?p e??e?a? 70 - ?a??t?ta 48KB/s
?????? 14.58 Sec Squid 30 - ?a??t?ta 256KB/s
?????? 1.17 Sec S???????? ??????
15.75 1MB/15.75 507,93 Kb/s 63,49KB/s
4
Agenda

• ??sa???? Web Caching/Of??e?e?
• Squid Proxy E??at?stas?/???µ?se??
• ?a?aµet??p???s?
• G?a p??????µ?????...
• Peering
• ACLs
• Delay Pools
• Redirect Programs
• Squid Peering st? AWMN
• Concept/?a??de??µa Configuration
• ??? ?a µ??? pe??ss?te?a

5
Web Proxy

• ??a Proxy e??a? e?a µ?????µa p?? µetaf??e?
  a?t?µata p??sßas?? se pe??e??µe?? t?? Web ap?
  ???a µ??a??µata pe??te?
• ???s? ?ta? ? pe??t?? de? µp??e? ?a ??e? aµes?
  p??sßas? st? Web
• ?e?t???e? t?? asf??e?a, t?? d?a?es?µ?t?ta ?a? t??
  ap?d?s? t?? p??sßas?? st? Web

6
Web Proxy ??? ?e?t????e?

• ?? Proxy ap????e?e? a?t???afa t?? se??d?? p??
  ??t????a? (?a? t?? a?t??e?µ???? p?? t??
  ap?te????) t?p???
• ?ta? ?? se??de? a?t?? ??t????? ?a??, ? proxy t??
  pa???e? ap? t?? ap????e?t??? t?? ???? (cache)

7
Web Proxy Why Bother ?

• ?????te??? ?????? ap????s??
• ??a??st?p???s? t?? ?pa?t??µe??? Bandwidth
• ?e??ss?te?? ta??t?ta
• ?e?a??te?? d?a?es?µ?t?ta
• ?a??te?? control

8
Squid Proxy ??? t???e?

• Sta pe??ss?te?a Unix
• (Source code C)
• Windows (!!!)
• (?e ???s? Cygwin/Mingw)
• http//www.cygwin.com
• http//www.mingw.org
• Mac OSX
• (ets? ?a? a?????? a?t? e??a? BSD ? )

9
Squid Proxy ??? ?a t? ß??

• Source
• http//www.squid-cache.org
• Binaries
• http//www.squid-cache.org/binaries.html
• Linux (Debian, RH)
• BSD (Net, Free)
• Solaris

10
Squid Proxy ?pa?t?se?? HW

• CPU
• ?????? apa?t?se??
• ?a Delay Pools a??????? t? f??t?
• 450MHz PII ??eta 1Mb/s
• http//hermes.wwwcache.ja.net/servers/squids.html
• Disk
• ?ata p??t?µ?s? SCSI
• OXI Raid5 ??? Cache_Dir se ?e????st? d?s??

11
Squid Proxy ?pa?t?se?? HW

• ???µ?
• 48MB/1GB Cache Squid Process 8MB
• ???s??? st? swapping
• ???a s?µe?a
• Single Proceess
• Max_open_fds 2048

12
Squid Proxy - ??s? ????

• ???????st?? 2 ?µ??e? Data
• ??a eßd?µ?da ?a??te?a
• ?p?????sµ??
• aDSL 384 12??e? 2GB
• 30 Hit-rate .6GB
• 2 ?µ??e? Data 1.2GB
• 7 ?µ??e? 4.2GB
• T?µ??e?te t?? µ??µ?....
• Ta s?st? refresh patterns ß?????e...

13
Squid Proxy - Compilation

• Standard Compile µe Make
• ./configure
• make
• make install
• ??t?a options pa??µet??? st? configure
• p.?. ./configure enable-delay-pools
• ??? p?e? t?
• Installation Dir./configure --prefix/usr/local/s
  quid
• Binaries /usr/local/squid/bin
• Config file /usr/local/etc/squid.conf
• Startup Script /etc/rc.d/init.d/squid (Linux)

14
Squid Proxy Packages/Ports

• Linux
• Debian apt-get install squid (??µ??? linux
  help wanted here ? )
• RH-rpm rpm -Uvh squid_package_name.rpm
• Slackware ASK Spirosco !!!
• FreeBSD
• pkg_add r squid
• ?p? source
• Cd /usr/ports/www/squid
• Make make install make clean
• Mac OSX
• http//www.osxgnu.org - Download ?a? click st?
  icona?? ?

15
Squid Proxy ?????? ???µ?s?

• Config file squid.conf
• 125 pa??µet??? ???µ?s?? !

• Dont Panic
• ?pa??e? pa??de??µa squid.conf.default
• ??t????f??µe t? pa??de??µa ?a? t??p?p????µe µ???
  t?? apa?a?t?te? pa?aµ?t???? ...a????a ?
• ?? ???sµ??e? pa??µet??? pa?????? t?? default t?µ??

16
Squid Proxy ?????? ???µ?s?

• ?as???? ???µ?se??
• cache_dir ??? ?a e??a? t? object cache ?a? t?
  µ??e??? ?a ??e?
• Effective User and Group ID
• Access Control Lists and Access Control Operators
• Email for the Cache Administrator

cache_dir ufs /squid-cache 2048 16 256
cache_effective_user squid cache_effective_group
squid
acl mynet1 src 10.0.0.0/255.0.0.0 http_access
allow mynet1
cache_mgr webmaster_at_my.computer.net
17
Squid Proxy ?????? ???µ?s?

• ???t? F??? ??µ??????a Directories
• ???? ?e????sete ?a?????? t? squid ??a p??t? f???
  p??pe? ?a t?? ep?t???ete ?a d?µ??????se? ta
  subfolders µesa st? cache directory . ?f?? ??ete
  ft???e? t? configuration file d?ste ap? t?
  command line t?? e?t??? squid -z -D

18
Squid Proxy ?????? ???µ?s?

• ?a?????? ??a???
• ?p??e?te ?a ?e????sete t? squid ?ta? ?e????e? t?
  s?st?µa sa? µe e?a startup script
• /etc/rc.d/init.d/squid (Linux)
• /usr/local/etc/rc.d/squid.sh start (FreeBSD)
• ? µe t? ???? ap? t? command line
• ? e?t??? ??a ?a ?e????se? t? squid e??a? squid D

19
Squid Proxy
G?a p??????µ?????
20
Squid Proxy - Peering

• ?? e??a?
• ? s?????s? d?? ? pe??ss?t???? Squids
• ???te??µata
• ????s? t?? Hit-Ratio 10
• ???p?? d??µ?????s?
• ?e???e?t?µata
• ??? d?s???? configuration
• ?e?a??te?? ?a??st???s? sta Cache Miss

21
Squid Proxy - Peering

• ??d? Peering
• ?e?a????? (Tree-Like)
• Mesh (?µ?t?µa Proxies)
• S??d?asµ?? ?a? t?? d??
• ???s??? sta Forwarding Loops !

22
Squid Proxy - Peering

• ??d? Peers
• Parent
• ???p??ete? requests p?? e?te ?p?????? st?? cache
  e?te ???
• Sibling
• ???p??ete? ???? requests p?? ?p?????? ?d? st??
  Cache
• ?p????µe ?a ????µe s??d?asµ? ?a? t?? d??

cache_peer 10.11.12.13 parent 3128 3130
cache_peer 10.11.12.14 sibling 3128 3130
cache_peer 10.11.12.13 parent 3128
3130 cache_peer 10.11.12.14 parent 3128
3130 cache_peer 10.11.12.15 sibling 3128
3130 cache_peer 10.11.12.16 sibling 3128 3130
23
Squid Proxy - Peering

• ?p????????a µeta?? peers
• ICP
• UDP based / Ping like
• (Cache1 Request Cache2, ??e?? t? t?de Object
• Cache2 Reply ?a? t? ??? ? Oxi ? ?a? a??a
  e??a? ...µpa???t??? ?)
• ICP packet ??a ???e request
• ? p?? ??????? ap??t?s? pa???e? t? request
• ICP Multicast
• ?a?? ??a µe?a??te?? a???µ? proxies
• ??s????te?? st? configuration
• ?p??e? ?a ??pe? ap? routers/firewalls

24
Squid Proxy - Peering

• ?p????????a µeta?? peers (S????e?a)
• Cache Digests
• ?at?????? objects p?? ?p?????? st? cache
• MD5 based Hash
• ?etaf??? µeta?? proxies ???e 10 ?ept?
  (???µ??eta?)
• ?a?? ??a ????? proxies µe s?et??? ???a objects
  st? ???e ??a
• ?a?? ??a d??t?a µe µe???? latency (?p?? t? AWMN)
• ??µ?????e? ???a False Hits

25
Squid Proxy Access Control

• Access Control Lists
• ?e?????f??? ?µ?de? (???st??, d?e????se??, ????
  ?.?.p.)
• Access Control Operators
• ?fa?µ????ta? ep??? st?? ?µ?de? ?a? pe?????f???
  e????e?e? p?? ep?t??p??ta? ? apa???????ta?

acl mynet1 src 10.1.0.0/255.0.0.0
http_access allow mynet1
26
Squid Proxy Access Control

• ??d? Access Control Lists

• Source/Destination IP address
• Source/Destination Domain
• Regular Expression match st? domain
• ???e?? st? ??t??µe?? URL
• ???e?? st? source ? destination domain
• ?µ??a/O?a
• Port ??????sµ??
• ???t?????? (FTP, HTTP, SSL)
• Method (HTTP GET ? HTTP POST)
• ??d?? Browser
• ???µa ???st? (Ident protocol)
• Autonomous System (AS) number
• Username/Password
• SNMP Community

27
Squid Proxy Access Control

• ??d? Access Control Operators

• http_access
• icp_access
• cache_access
• no_cache
• ident_lookup_access
• miss_access
• always_direct, never_direct
• snmp_access
• delay_classes
• broken_posts

28
Squid Proxy Access Control

• Se??? e?????? ACL
• ?e t?? se??? p?? eµfa?????ta? st? squid.conf
• To p??t? match te?µat??e? t?? ??e???
• ?? de? ß?e?e? match t? squid ???e? t?
  a?t?st??f? ap? a?t? p?? ??e?e ? te?e?ta?a ??aµµ?
  p?? d??ßase

29
Squid Proxy Access Control

• ??????? s??d?asµ??
• ?a st???e?a e??? ACL s??d???ta? µe OR
• ?a ACLs se e?a Access Control Operator s??d???ta?
  µe AND
• T?µ??e?te ?t? t? Squid a?t?st??fe? t? te?e?ta??
  action a? de? ß?e? match

acl myNets src 10.0.0.0/255.255.255.0
10.1.0.0/255.255.255.0
acl myNets src 10.0.0.0/255.255.255.0
10.1.0.0/255.255.255.0 acl work_hours time
0800-1700 http_access allow myNets work_hours
30
Squid Proxy Delay Pools

• ?p?t??p??? t? sharing µ??? µ????? t?? Bandwidth
• ?p????? ?a efa?µ?st??? se µeµ???µ????? pe??te? ?
  ?a? se ???????a subnets
• ??? d?af??et??a ???a
• ???? ta??t?ta? (se Bytes/s) - Restore
• ???? µe?????? µeta t? ?p??? e?e???p???ta? t? ????
  ta??t?ta? (se Bytes) - Max

31
Squid Proxy Delay Pools

• ??d? Delay Pools
• Class1
• efa?µ??e? e?a s??????? ???? ??a ??a ta requests
  p?? p?ft??? st? pool
• Class2
• ?fa?µ??e? e?a s??????? ???? a??a ?a? e?a
  ep?µ????? ???? a?a ???st?
• Class3
• ?fa?µ??e? s??????? ????, ???? a?a class C subnet
  ?a? ???? a?a ???st?

delay_parameters 1 8000/32000
delay_parameters 1 8000/32000 4000/16000
delay_parameters 1 32000/128000 16000/64000
4000/16000
32
Squid Proxy Delay Pools

• ???s? Delay Pools
• ?e??µ??? ta??t?ta p??sßas?? ???sµ??e? ??e? t??
  ?µ??a? (time based ACLs)
• ?e??µ??? ta??t?ta ??a ???sµ????? µ??? p?????sµ???
  (dst ? dstdomain ACLs)
• ?e??µ??? ta??t?ta ??a ???ste? p?? de? ????? ???e?
  authenticate µe t?? proxy (username/password
  ACLs)
• ?e??µ??? ta??t?ta ??a ???sµ??a p??t?????a (p.?.
  FTP)
• ?e??µ??? ta??t?ta e??p???t?s?? peers
• ??????ta? µe???? ???, µp????µe ?a af????µe t???
  ???ste? ?a ß??p??? ??????a web se??de? a??? ?a
  ?ateß????? µe???a a??e?a p?? a???

33
Squid Proxy Redirect Progs

• ?? e??a?
• ??????µµata p?? f??t?????? ta ??t??µe?a URLS ß?s?
  ?a????? ?a? a?????a ep?t??p??? ? apa???e???? t??
  p??sßas? ? µetaf????? t?? ???st? se ?ap??a ????
  s???da ap? a?t? t?? a?????? p?????sµ??
• G?at? ta ?????µe
• ????µ? d?af?µ?se??
• ????µ? e?????t???? sites
• Mirror s???? ep?s?ept?µe??? sites

34
Squid Proxy Redirect Progs

• SquidGuard
• http//www.squidguard.org/
• ACL Based
• ?t??µe? ??ste?/ACLs ??a p????? ?at?????e?
• DansGuardian
• http//dansguardian.org
• Multiple method based
• Content Phrase filtering
• ??? epa??e?µat??? ap? t? squidGuard
• ??? d?s???? st? configuration
• ??? ßa??

35
Squid Proxy Redirect Progs

• SquidGuard ?a??de??µa config

dbhome /var/db/squidGuard logdir /var/log
SOURCE ADDRESSES src noadsawmn ip
10.0.0.0/8 DESTINATION CLASSES dest ads
domainlist ads/domains urllist
ads/urls acl noadsawmn pass
!ads all redirect http//www.ocean.aw
mn/icons/blank.gif default
pass all
36
Squid Proxy AWMN Peering

• ??p?????a
• ? t?p?????a p?? p??te??eta? ??a ???s? st? AWMN
  e??a? a?t? t?? semi-hierarchical mesh ?a?
  fa??eta? st? pa?a??t? s??µa

37
Squid Proxy AWMN Peering

• ??d? Proxies
• Level 1
• ?? ??µß?? a?t?? t?? t?p?? e??a? ???e ??µß?? p??
  ??e? aDSL s??des? ?a? ???e? ?a t?? µ?????e? st?
  AWMN
• Level 2
• ??µß?? ????? t?p??? aDSL p??sßas? a??a µe
• s??des?µ?t?ta se ??µß??? Level1

38
Squid Proxy AWMN Peering

• ?a???e?
• ?? Level1 ??µß?? µp????? ?a ????? µ??? sibling
  relationships µe ?????? Level1
• (µ??? e?a??es? e??a? mutual parenting
  µeta?? Level1 nodes ?at?p?? s?µf???a? ?a? ???s?
  nonhierarchical_direct off, prefer_direct
  on ?a? cache_access ACLs)
• ?? Level2 ??µß?? µp????? ?a ????? parent
  relationships µ??? µe Level1 ?a? µ??? sibling
  relationships µe a????? Level2
• ?? ??a? ??µß?? e?e? pe??ss?te???? ap? ??a?
  Parents, t?te ?a p??pe? ?a ???s?µ?p???se? CARP
  selection a?????a µe t? Bandwidth t?? ???e Parent
• ???s? Cache Digests

39
Squid Proxy AWMN Peering

• Of???
• ????µ??? Hit-Ratio
• Internet from everywhere
• Redundancy/failover
• ?e??s? t?? default routes
• S?µe?a p??s????
• Masking a?ep???µ?t?? clients p?s? ap? Level2
  Proxies
• ?p??e? ?a a?t?µet?p?ste? µe ???s? delay pools
  ?ste ?a d??eta? µ??? µ???? ??µµ?t? BW st???
  untrusted Level2 peers
• ???s? t?? proxies ??a ep???se??/e???????
  e????e?e?
• ?p??e? ?a a?t?µet?p?ste? µe s?st? ???s?
  logging ?a? sta d?? proxy Levels

40
Squid Proxy AWMN Peering

• ?a??de??µa Configuration
• Level2 Proxy µe e?a Parent
• Level2 Proxy ?e d?? Parents ?? ?p???? ????? DSL
  ??aµµ?? ?d?a? ????t???t?ta?
• Level2 Proxy µe d?? Parents ?? ?p???? ????? DSL
  ??aµµ?? d?af??et???? ????t???t?ta?

cache_peer XXX.XXX.XXX.XXX parent 3128 3130
default
cache_peer XXX.XXX.XXX.XXX parent 3128 3130
carp-load-factor0.50 cache_peer ???.???.???.???
parent 3128 3130 carp-load-factor0.50
cache_peer XXX.XXX.XXX.XXX parent 3128 3130
carp-load-factor0.30 cache_peer ???.???.???.???
parent 3128 3130 carp-load-factor0.70
41
Squid Proxy AWMN Peering

• Autoconfig Clients Script
• Netscape type (PAC Proxy Auto Config)
• Proxy.pac file
• WPAD (Web Proxy Auto-Discovery Protocol)
• http//www.squid-cache.org/Doc/FAQ/FAQ-5.htmlss5.
  10

function FindProxyForURL(url, host) if
(isInNet(host, 10.XXX.XXX.0", "255.255.255.0"))
return DIRECT return "PROXY
cache.domain.awmn3128 DIRECT"
42
G?a ?e??ss?te?e? p????f???e?

• Main Site
• http//www.squid-cache.org
• Handy Reference
• http//squid.visolve.com
• RTFM
• http//squid-docs.sourceforge.net
• ?s??µata
• http//squid.ocean.awmn

43
St? ep?µe?? ?pe?s?d??

• Integration µe CISCO/WCCP Protocol
• Storage Algorithms G?at? t? GDSF e??a? ?a??te??
  ap? t? LRU ?a? t? LFUDA ....
• eee ??? t? ??e? ? a??a??? ??? ?
• Refresh Patterns ??p?? t? cache sa? e??a?
  ?eµ?t? Junk ???
• Advanced Peering/Request Routing
• (? p?? ?a ???ete windows update µes? ?aµ?a?)
• ?a? ???a p????...

44
???t?se?? / S???t?s?