UW Medicine Networking Update

1
UW Medicine Networking Update

• Terry Gray
• Associate Vice President, IT Infrastructure
• University of Washington
• 16 April 2004

2
Key Elements of the Partnership

• Changed CC now responsible for...
• In-building network implementation
  andoperational support for med ctrs, clinics
• Med center network design for real
• Not Changed CC still responsible for...
• Network backbone, routers
• Regional and Internet connectivity
• SoM and Health Sciences networking

3
Why the Partnership Makes Sense

• Consistency, interoperability, manageability
• Leverage CC networking expertise
• Clinical/research hi-performance network needs
• 24x7 Network Operations Center (NOC)
• Advanced network management tools
• Avoid design/build organizational conflicts
• Beyond the network...hope to share distributed
  system architecture and network computing
  expertise

4
Near-term Progress and Plans

• Created Top 10 list --now up to Top 20 )
• Agreement on standard maintenance window
• Static addressing work-around (sDHCP)
• FDDI, VLAN elimination
• Subnet splits/upgrades (1500 computers)
• Equipment upgrades
• Router consolidation, dedicated subnets, separate
  med center backbone
• Equipment, outlet location database updates
• Initial wireless deployment
• NetVersant and Cisco external studies

5
The Challenge

• Create a network computing environment
• with excellent security
• excellent supportability
• that users find reliable and responsive

6
Context A Perfect Storm

• Increased dependency on network apps
• Decreased tolerance for outages
• Decades of deferred maintenance...
• Inadequate infrastructure investment
• Some old/unfortunate design decisions
• Some fragile applications
• Fragmented host management
• Increasingly hostile security environment
• Increasing legal/regulatory liability
• Increasing importance of research/clinical
  leverage

7
Context Some Numbers
8
Network Device Growth
Note Most dips reflect lower summer use last
one is a measurement anomaly
9
Network Traffic Growth (linear)
10
Network Traffic Growth (log)
11
System Elements

• Environmentals (Power, A/C, Physical Security)
• Network
• Client Workstations
• Servers
• Applications
• Personnel, Procedures, Policy, and
  ArchitectureFailures at one level can trigger
  problems at another level need Total System
  perspective

12
Systemic Network Problems(some of these go back
decades)

• Old infrastructure (e.g cat 3 wire)
• Non-supportable technologies (e.g. FDDI)
• Non-supportable (non-geographic) topology
• Expensive shortcuts (e.g. cat5 mis-terminated)
• Security based on individual IP addresses
• Subnets with clients and critical servers
• Documentation deficiency
• Contact database
• Device location database
• Critical device registry

13
Systemic General Problems

• Ever-increasing system complexity, dependencies
• Ever-increasing threats, liabilities
• Departmental autonomy
• Un-controlled hosts
• Un-reliable power and A/C in equipment rooms
• No net-oriented application procurement
  standards
• Are HA and DRBR expectations realistic?
• Are backup plans workable?

14
Key Operational Objectives

• simplicity
• lower cost
• higher MTBF (modulo redundancy)
• lower MTTR (quicker diagnosis)
• consistency
• deterministic outlet behavior (Network Utility
  Model)
• connection transparency (open/deterministic
  Internet)
• easier problem diagnosis
• These objectives conflict with other goals

15
Design Tradeoffs

• Networks Connectivity Security Isolation
• Fault Zone size vs. Economy/Simplicity
• Reliability vs. Complexity
• Prevention vs. (Fast) Remediation
• Security vs. Supportability vs.
  FunctionalityDifferences in NetSec approaches
  relate to
• Balancing priorities (security vs. ops vs.
  function)
• Local technical and institutional feasibility

16
Tradeoff Examples

• Defense-in-depth conjecture (for N layers)
• Security MTTE (exploit) ? N2
• Functionality MTTI (innovation) ? N2
• Supportability MTTR (repair) ? N2
• Perimeter Protection Paradox (for D devices)
• Firewall value/efficiency ? D
• Firewall effectiveness ? 1 / D
• Border blocking criteria
• Threat cant reasonably be addressed at edge
• Wont harm network (performance, stateless block)
• Widespread consensus to do it
• Security by IP address

17
Network Security Chronology

• 1990 Five anti-interoperable networks
• 1994 Nebula shows network utility model viable
• 1998 Defined border blocking policy
• 2000 Published Network Security Credo
• 2000 Added source address spoof filters
• 2000 Proposed med ctr network zone
• 2000 Proposed server sanctuaries
• 2001 Ban clear-text passwords on CC systems
• 2001 Proposed pervasive host firewalls
• 2001 Developed logical firewall solution
• 2002 Developed Project-172 solution
• 2003 Slammer, Blaster death of the Internet
• 2003 Developed flex-net architecture

18
Next-Gen Network Architecture

• Parallel networks more redundancy
• Supportable (geographic) topology
• Med center subnets separate backbone zone
• Perimeter, sanctuary, and end-point defense
• Higher performance
• High-availability strategies
• Workstations spread across independent nets
• Redundant routers
• Dual-homed servers

19
Success Metrics

• Toms
• Nobody gets hurt
• Nobody goes to jail
• Steves
• Four Nines or bust!
• High ROI (Return On Investment)
• Terrys
• Low ROI (Risk Of Interruption)
• Low MTTR (Quick to Fix)
• High predictability (No surprises)

20
Lessons

• Net reliability host security are inextricably
  linked
• Five 9s is hard (unless we only attach phones?)
• for , best security investment is central host
  management
• Nebula existence proof security in an open
  network
• Watch out for unfair cost shifting
• The cost of static IP configuration is very high
• Controlling net access is hard --hublets,
  wireless
• Even host firewalls dont guarantee safety
• Perimeter firewalls may increase user confusion,
  MTTR
• It only takes one compromise inside to defeat a
  firewall
• Next-generation threats firewalls wont help
• Even so defense-in-depth is a Good Thing

21
Questions? Comments?
22
Network Security Addendum
23
Recent Events

• attacks
• slammer (Jan 2003)
• blaster (Aug 2003)
• sobig (Sep 2003)
• mydoom (Feb 2004)
• witty (Mar 2004)
• impact
• demise of the open/transparent/deterministic
  Internet
• demise of the network utility model
• demise of the unmanaged/autonomous PC
• demise of reliable email

24
Seven Security Axioms

• 1. Network security is maximized when we assume
  there is no such thing.
• 2. Large security perimeters mean large
  vulnerability zones.
• 3. Firewalls are such a good idea, every
  computer should have one. Seriously.
• 4. Remote access is fraught with peril, just like
  local access.
• 5. One person's security perimeter is another's
  broken network.
• 6. Private networks won't help (Limits of
  isolation).
• 7. Network security is about psychology as well
  as technology.

25
Network Security Credo

• Focus first on the edge(Perimeter Protection
  Paradox)
• Add defense-in-depth as needed
• Keep it simple (e.g. Network Utility Model)
• But not too simple (e.g. offer some policy
  choice)
• Avoid
• one-size-fits-all policies
• cost-shifting from guilty to innocent
• confusing users and techs (broken by design)

26
Preserving the Net Utility Model

• What is it?
• Why important?
• Incompatible with perimeter security?
• Too late to save?
• NUM-preserving perimeter defense
• Logical Firewalls
• Project 172
• Foiled by static IP addressing
• Requires all hosts be reconfigured

27
Conflicting Perspectives

• System administrator view
• some prefer local control/responsibility
• some prefer central/big-perimeter defense
• some underestimate cost impact on others
• User view
• want just enough openness to run apps
• prefer unlisted numbers?
• Network operator view
• concerned about increased support costs and
  repair times due to growing complexity and
  unpredictability
• concerned about loss of network functionality

28
Generic Security Toolkit

• host choice truly thin clients species
  diversity
• host configuration management
• conventional firewalls
• logical firewalls
• private addressing (e.g. project 172)
• IDS, IPS, ADS
• vulnerability scanning, anti-virus tools
• QoS (to protect critical traffic types)
• isolated networks (physical, VLAN, VPN)
• non-technical policies, education, staff

29
Lines of Defense

• network isolation for critical services
• host integrity (Make the OS net-safe)
• host perimeter (integral ACLs/firewalling)
• cluster/lab perimeter (sanctuary, FW, LFW)
• network zone perimeter (P172, FW)
• real-time attack detection and containment
• user education

30
Perimeter Firewalls

• increase time-to-infection
• increase time-to-repair
• provide defense-in-depth
• may look like a broken network to users
• are defeated by a single hacked host
• are defeated by tunneling/encryption
• often give a false sense of security
• encourage backdoors
• may be a performance bottleneck
• may inhibit legitimate activities, innovation
• create a vulnerability zone that is hard to
  protect
• vpns, laptops, wifi, usb drives, social engr
  attacks
• the more you depend on perimeter defense, the
  more you must invest in defending the perimeter

31
Operational Impact by firewall type

• host -- best case user interaction w/FW
  possible
• cluster -- no impact on net diagnosis beyond
• logical -- low impact on basic net diagnosis
• subnet -- impacts almost all diagnosis
• zone -- impacts inter-zone diagnosis
• border --impacts inter-enterprise diagnosisNB
  cost of maintaining firewall config depends on
  who is doing it, and how many rules/exceptions
  there are.

32
Limits of Isolation attack gateways

• hosts connected to two different networks can
  become attack gateways between the two
• example home PCs with VPN connection to
  protected network
• safer remote access SSH, SSL, K5, RDP, SSL VPNs

33
Med Center Zone Perimeter

• purpose
• time to defend against zero-day events
• protect the otherwise unprotected
• defense-in-depth
• reduced annoyance/noise traffic
• DOS attack mitigation
• options
• conventional inline firewall
• private addressing NAT or proxies
• both

34
Protecting Non-fixable Devices

• FDA-approved devices, printers, etc
• protection options (besides zone perimeter)
• private addressing
• individual firewall, VPN, or NAT box (25 -
  2500)--depending on performance requirements
• cluster/lab perimeter firewalls
• logical firewalls

35
NOC view of Firewall Approaches

• EPFW End-Point Firewall
• LFW Logical Firewall w/masquerading NAT
• SFW Subnet Firewall
• BZFW Border or Zone Firewall
• P172 Project 172-phase III (Private
  addresses with NAT)
• 
  IDEAL EPFW LFW P172 SFW
  BZFW
• Policy Enforcement Point? Host
  Host Subnet Zone Subnet Zone
• Requires host reconfigure? No
  Yes Yes Yes No No
• Requires network reconfig? No
  No No No Yes
  Yes
• Destroys E2E transparency? No
  No No No Yes Yes
• Assured NOC access to switches? Yes
  Yes Yes Yes No No

36
Network Security Trends
stealth / advanced scanning techniques
Blendedattacks
High
denial of service
DDOS attacks
sniffers
www attacks
automated probes/scans
back doors
packet spoofing
Attack Sophistication
disabling audits
hijacking sessions
burglaries
exploiting known vulnerabilities
password cracking
self-replicating code
password guessing
Low
37
Impact of Recent Security Events

• more perimeter firewalls (demise of open
  Internet, NUM)
• more VPNs
• more tunneling (firewall friendly apps)
• more encryption (thanks to RIAA)
• more collateral damage (from attacks remedies)
• worse MTTR (complexity, broken tools)
• constrained innovation (e.g. p2p, voip)
• cost shifted from guilty to innocent
• pressure to fix computer security problems in
  network
• pressure for private nets
• pressure to make network topology match org
  boundaries
• blaster triggered more perimeter defense, but
  showed weakness of conventional perimeter defense