DCE and DFS at PaineWebber'''

1
DCE and DFS at PaineWebber...

• Ron Smetana
• Divisional Vice President
• October 16, 2000

2
Establishing Some Credibility

• Worked for IBM for 17 years
• International Tech Support Center - Austin
• Wrote the Distributed File System for AIX/6000
  Redbook (gg24-4255) circa 1994
• Worked for Deutsche Bank in NYC
• DCE, CICS/6000, DFS
• Currently Work for PaineWebber in Weehawken, NJ
• DCE, CICS/6000, DFS

3
As of October, 2000 the PW DCE cell.

• contains over 23,696 users of DCE and DFS, of
  which
• 18,708 users have DFS file space allocated
  (branch users)
• 4,988 users have no DFS space allocated (web
  users)
• contains over 371 remote DFS fileservers
• operates under a must be available environment
• is maintained by a small group of administrators
• is still growing!

4
Our configuration is unique because

• 371 remote branch sites are connected via WAN
  links
• A DFS File server is at each site
• Majority of brokers log in within a short period
  of time
• Rapid Rollout occurred at 15 branches/weekend
• Implemented across RS/6000s, PCs and Sun machines
• Largest deployment of DCE/DFS clients that WE
  know of

5
DCE/DFS Chronology at Painewebber

• Plan the DCE design (circa 1996)
• Configure a development cell
• Development of the MAC DCE application
• Configure the production DCE cell
• Rollout the 300 branches (finished 6/99)
• Addition of Webservers as DCE/DFS clients
• Access/improve scalability/reliability/availabilit
  y
• Addition of Bradford branches (finished 7/00)
• Addition of SUN DCE/DFS clients
• Upgrade to DCE/DFS version 3.1 (ongoing)

6
What We Have Today

• DCE/DFS cells that we maintain
• Experimental cell
• Development/QA cell
• Production cell
• Special Projects cell(s)
• All Servers are administered by 2 DCE/DFS
  sysadmins (in addition to the normal branch
  support sysadmins)

7
Production DCE Cell - Enterprise Sites

• Four DCE Security Servers (2-NYC and 2-LNH)
• Three Fully replicated Cell Directory Servers
• Three FLDBs
• Various DCE/DFS clients and fileservers
• AIX DCE/DFS clients
• NT DCE/DFS clients
• Solaris DCE/DFS clients
• Data recovery site is NYC

8
Storage on Enterprise DFS Fileserver

• Implemented as RS6000 connected to EMC Symmetrix
• Applications Resident
• Portfolio Reporting Application (highly r/w)
• PW Research Documents (read mostly)
• Misc DFS base (read mostly)
• DFS Features used
• Point in time backups remain on-line for quick
  data retrieval
• DFS base fileset is replicated to NYC for
  availability
• Log based recovery enables a quick startup after
  reboot
• All DFS enterprise data can be moved from LNH to
  NYC while users are accessing data!!!

9
High Availability for PW DCE/DFS

• What we CAN do
• Branch users can log on when a DCE security
  server, cell directory server, enterprise DFS
  fileserver goes down
• Branch users can log on if all of LNH or if all
  of NYC is down
• Keep an weekly updated on-line copy of a users
  storage
• What we CANNOT do
• Immediate failover from LNH to NYC of
  enterprise-based highly writeable DFS storage
• Replicate a branch users read/write storage

10
Production DCE Cell - Branch Sites

• 371 branches currently deployed
• 1 DFS File server/branch
• Between 10 and 228 DCE/DFS clients/branch
• 2 enterprise branches with 473 and 910 clients
• Average of 50 DCE/DFS clients/branch
• Hardware/Software
• DFS File Srvr RS/6000, AIX 4.2.1, DCE/DFS 2.1
  PTF set 18
• Clients PC, NT4.0, Gradient DCE-2.0.5, Transarc
  DFS-1.0.3c

11
Current Challenges

• Updating to current versions of DCE/DFS
• NT DCE clients Gradient 2.0.5--gt2.2
• NT DFS clients Transarc 1.03c--gt1.5
• AIX DCE/DFS filesrvs 2.1,ptf18--gtDCE3.1,ptf2
  DFS3.1,ptf1
• AIX DCE/DFS servers 2.1,ptf29--gtDCE3.1,ptf2
  DFS3.1,ptf1
• Solaris DCE/DFS 2.0--gtDCE3.1,DFS3.1
• Too many machines, Too many versions, Too Little
  Time.
• 19085
• 11
• 0

12
Why We are Upgrading to DCE/DFS 3.1

• IBM cannot support the DCE/DFS 2.1 version
• No use sending in DCE/DFS 2.1 based system dumps
  or complex problems
• Obviously better to be on a current STABLE
  version
• Problems that we currently have with DCE/DFS 2.1
• System dumps during fts clone operation
• FLDBs are sluggish under load
• fts dumps face a 2.1GB limit
• fts dumps do not complete when systems are under
  heavy load
• Unsure how this version could scale to a larger
  environment
• Take advantage of new features and performance

13
Vendor Challenges

• Transarc and IBM
• Vendor Commitment Concerns
• Support Issues
• Software Quality Issues
• IBM and Transarc have separated DCE/DFS 2.1 into
  two distinct products
• Two sets of ptfs to track and put thru our QA
  cycle
• Two sets of documentation
• PMRs disappear from IBMs web-based SupportLine
  database after 28 days

14
PMRs, PMRs and PMRs

• DFS 3.1
• PMR38601 PE_SITE not being updated
• PMR38177 start.dfs did not complete successfully
  message
• PMR33404 start.dfs does not migrate 2.1 DFS
  version to 3.1
• PMR32722 missing SMIT config/unconfig menus
• PMR37825 DFS dumps core during unix find command
• PMRXXXX DFS3.1 ships on CD w/incorrect .toc file
• DCE 3.1
• PMR36807Can not install CDS or SS
• PMR38172 lsdce says we have 2 initial directory
  servers
• PMR34220 dce3.1.0.1 missing prereqs

15
Some Items On Our Radar Screen.

• Upgrade all AIX DCE/DFS software to Version 3.1
• Upgrade all NT DCE clients to use new supported
  versions
• Improve NT client failover times via additional
  DCE/DFS configuration profiles and features
• Enable fileset replication of DFS data from LNH
  to NYC for availability (read-mostly enterprise
  storage)
• Mount day-old DFS filesets in backup site data up
  for availability (read/write enterprise storage)
• Rollout out DCE/DFS Solaris platform for projects
  involving Vignette Story Server, et al.

16
How Can...?

• How can we help IBM and Transarc?
• How can we convince IBM and Transarc to supply a
  better product and support experience?
• How can we help our DCE community outside of
  IBM/Transarc?

17
A Few Misc Things...

• The PaineWebber Scalibility Viewpoint
• Acquiring brokerage firms means you need to scale
• Being acquired means you need to scale
• Steve Mills, IBM VP said in a recent trade
  magazine that IBM wants to make middleware easier
  to use

18
Summary of DCE/DFS

• Good news
• Scaled nicely for the current number of users
• Handled the JC Bradford merger without problems
• Highly reliable---100 Enterprise Server uptime
  since rollout!
• DCE/DFS version 2.1 has served us well
• Not so Good News
• Must Migrate to DCE and DFS 3.1
• Migration to DCE 3.1 and DFS 3.1 not smooth
• Vendor Commitment not readily apparent