Title: SECURITY PERS Best Practices and an Assessment
1SECURITYPERS Best Practices and an Assessment
PRISM 2006
- Tom Roark, Technical Services Manager
- Mississippi Public Employees Retirement System
2The Session Game Plan.
- The Big Question about Security
- An Opportunity (Problem)
- An Undesirable Surprise
- Change/Upgrade Strategy
- Security Assessment
- Best Practices and Lessons Learned
3Security - The Big Question
- Is there a way to totally secure your computing
devices and data sources?
4Security The Answer
- The only way to completely secure any computer
device or data source is to disconnect it from
the network and place it in a locked vault where
no one has the key. In this case, the data would
be completely secure but totally inaccessible.
5Free Stuff
- A wise person learns from his/her own mistakes
and experiences. - An even wiser person learns from others mistakes
and experiences.
6An Opportunity (Problem)
- We were receiving 800 diskettes a month to
collect Wage and Contribution Data - Along with the diskettes we were also receiving a
signed paper Form 8 as an official submission
document - Both the diskette and paper Form 8 were submitted
to PERS via regular mail - It was the year 2002 and PERS was still using a
DOS based application. (The application was
developed in Foxpro 2.0 (Foxpro 2.0/1990))
7Our Assignment
- Upgrade the application from a DOS based platform
to a Windows based platform - Move from diskette submission via regular mail to
secure electronic submission via the internet - Eliminate the paper Form 8 required
- Distribute the new application via the internet
8The Solution
- Upgraded to Visual Foxpro 5.0
- Decided to use FTP to transfer data files
- Since FTP was not a secure protocol, we
researched and discovered a FREE Secure FTP
client and server software solution - Researched and integrated an FTP Activex control
into our Visual Foxpro application. This provided
for a seamless submission to PERS via FTP - Eliminated the paper Form 8 by including a
special data line in the transferred file - Provided a download link on our web site to
distribute the new software (unpublished, login
required)
9Distribution Configuration
10Submission Configuration (Secure FTP)
11Bomb (New State Security Policy)
NO MORE FTP ALLOWED
12Go to Plan B (we had no plan B)
- Researched using an allowed protocol (HTTP) to
transfer the data files and at the same time keep
as much of our previously completed work as
possible. We found an HTTP activex control for
Foxpro. - Since HTTP is not secure without SSL and PERS was
not yet SSL enabled and savvy, we researched and
found a file encryption activex control that
could be used to secure the data files prior to
transfer via HTTP (DES, 3DES, AES encryption) - We also decided to transfer the data files to
PERS web server via an unpublished URL which
required a login for added security - We researched and wrote PERL scripts to transfer
the files from our web server to our inside FTP
server and then immediately delete the files from
the web server for another layer of security - We also found a real plus to plan B in that it
removed the secure FTP client and server software
requirement
13Submission Configuration (Plan B HTTP/FTP)
14PERS Best Practices
- Always communicate with your ISP on projects that
could be impacted by your dependence on them. - Make sure any future plans you have will not be
hindered by any future plans they have.
15An Undesirable SurpriseMonday Morning 700 am
4/1/2002
- This really happened
- Dont let it happen to you!!!!!!
16(No Transcript)
17www.pers.state.ms.us was hacked
- Gained Administrator access to server
- Replaced the Root structure of PERS web site with
hacked pages - Changed Administrator passwords and locked us out
of our own servers - It was NOT an April Fools Joke!!!!
18What did we do??????
- We panicked like most IT staff where
- The web server was deployed by a third party
- Web site was developed by another third party
- MIS staff had minimal web server experience
- Backups were done infrequently
- We started rebuilding our web server
19Solving the Web server problem
- Rebuilt the server from the ground up
- Renamed administrator account
- Put Complex Password on administrator account
- Disabled the Guest account
- Used netscape best practices on web server
configurations - As an interim protective strategy we installed
and configured a desktop firewall on the web
server - Implemented a new backup strategy for our web
server
20The Interim Configuration
21PERS Best Practices
- Dont EVER leave an administrator password
blank!!!! Due Diligence - Take ownership in products delivered by 3rd party
companies - Play a vital role in all installations/deployments
. Make sure your understand everything - Have computing standards in place and make sure
any installations/deployments done by 3rd parties
meet your standards - Make sure every computer asset you have has added
security beyond that of an OS (Firewall, AV,
etc) - Make sure you have an adequate backup strategy
- Have documented server build procedures (Disaster
Recovery for WHEN it happens)
22The straw that broke the camels back
23Time to Make Some Changes
- We did some serious evaluations and took an
honest inventory at where we were - 3 years ago an honest inventory indicated that
PERS was - Using a basically unprotected web site
- Old Version of Netscape Suitespot Web Server
- Using an outdated backup technology
- Very Slow Network
- Outdated Servers and Desktops
- Using Windows NT 4.0 on desktops and servers
- Using Novell 5.1 (File and Print sharing)
- Using Office 97
- Exchange 5.0
- SQL Server 6.5
- Norton Anti-Virus 6 or 7
- No web security
- No spy-ware/mal-ware security
- No email security
- No Desktop/Server/Backoffice software maintenance
agreements - Uncontrolled user environment
24Time to Make Some Changes
- Did some research about available products and
possible upgrade paths - Set some goals, objectives and priorities
- Decided on a change strategy
- Got to work and started making changes
25Change Strategy Pyramid
26Hardware - Network and Firewall UpgradeWhat it
was like
- Outdated Slow Network (10mb half duplex hubs and
an unsupported core LANPLEX (SPOF)) - Multiple protocol network (TCP/IP, IPX/SPX, etc)
- No DMZ or Service Network for exposed servers
(www.pers.state.ms.us) - Software firewall on Windows NT with 2 interface
cards, inside and outside
27Hardware - Network and Firewall UpgradeWhat we
did
- Purchased 10/100/1000 MB Ethernet
state-of-the-art full duplex switches - Purchased 10/100/1000 MB Ethernet nics for
servers - Researched and Purchased a firewall appliance
with multiple functionalities including Intrusion
Detection/Prevention, Content Filtering, AV, VPN,
FW - Made sure our firewall appliance had multiple
interface capabilities, minimum of three (inside,
outside, service)
28The Current Configuration
29Service Network Configuration
30Hardware - Network and Firewall UpgradePERS Best
Practices
- Firewalls
- Use Appliances for firewalls
- Identify SPOFs and implement a fault tolerant
strategy in case of failure (Cluster, spare, next
business day maintenance contract) - Implement a service network or DMZ zone for
exposed servers - Use a different network segment on each firewall
interface - Eliminate path from exposed/outside servers to
inside servers/network. (i.e. Implement pulls
from inside instead of pushes from
outside/service network.) - Restrict Administration access to firewall to
internal specified machines and users, NO
external configuration allowed - Lock down firewall rules to interfaces and
entities, make them as tight as you can get them - Disable all services not being used on firewall
- Configure any alerts to go to a firewall
administrators email group - Lock down VPNs to pass traffic to proxies so
that rules must be created to allow exact data,
not just anything - Lock down smtp to ISP relays only or equivalent,
dont allow smtp from universe - Network
- Only patch network drops actually being used to a
switch port - Require user/password security to administrater
your switches - Disable all services not being used on your
switches - Keep your network switches and firewall up to
date with patches
31Hardware Desktops/Servers/Backups/OtherWhat it
was like.
- Old outdated equipment
- Servers and desktops didnt even meet minimum
requirements for new OSs - Several different models adapters and monitor
types - 10 MB desktop nics/100 MB server nics
- Old Slow Scanners
- Very old backup solution
32Hardware Desktops/Servers/Backups/OtherWhat we
did
- We upgraded our desktops, servers, backups, etc
- Standardized on equipment manufacture
- Standardized on server type across the board
- Standardized on desktop type across the board
- Standardized on monitor types
- Standardized on network access with 10/100/1000
MB nics - Standardized on scanner types
- Implemented a D2D2T high speed backup solution
(Really Good Decision)
33D2D2T Backup Solution
- Before D2D2T
- 4 Backup Servers
- 8 Tape drives with 5 tape auto-changers each
(DLT) - 4 Racks of space
- Slow 100 MB Network
- 12-16 Hour Backup window (incremental on some)
- Very Problematic
- After D2D2T
- 2 Backup Servers
- 2 High Speed Tape Cache Systems (1.5 TB each)
- 2 Tape drives and no auto-changers necessary
(LTO-2) - 1 Rack of space
- Fast Gigabit Network
- 5 Hour Backup window
- 1-2 Hour Offline tape copy
34D2D2T Backup Solution
35Hardware Desktops/Servers/Backups/OtherPERS
Best Practices
- Purchase similar equipment where possible in
order to have standardization and swappable parts
in case of emergencies - Use hardware RAID disk configurations on all
servers - Place all servers on UPSs
- Implement UPS power failure graceful shutdowns
- Configure equipment with more resources than
software minimum requirements - Disable unnecessary devices on desktops (USB,
diskette, etc) - Implement a D2D2T backup strategy
- Standardize, Standardize, Standardize !!!!!!!!
36Picture of server room
37Desktop and Server OSWhat it was like
- Windows NT4 sp6
- Users could do almost anything, uncontrolled
environment - Users could run any .exe
- Users could make all types of desktop preference
changes - Had Control Panel access
- Minimal group policy usage
- Many different configurations
- Take 1 days to completely rebuild a pc from the
ground up - The bottom line is we had inconsistent, unstable,
problematic and unsecured configurations
38Desktop and Server OSWhat we did
- Implemented an enterprise agreement for desktop
- Implemented software assurance for servers
- Windows XP sp2
- Windows Server 2003 sp1
- Made good use of Active directory GPO.
- Minimize the number of different configurations
as much as possible - Implemented the use of Ghost images for builds.
We can have a pc completely rebuilt in less than
an hour
39Desktop and Server OSPERS Best Practices
- Standardize configurations
- Minimize the number of different configurations
as much as possible - Implemented Disk quotas per user
- Segregated MIS and Business user home directories
- Rename administrator accounts and deleted
descriptions - Disable guest accounts
- Delete other windows help accounts
- Disable all unnecessary services
- Control drive mappings with group policy and/or
login scripts - Implement user time restrictions
- Documented desktop and server builds
- Used Microsoft Security Baseline Analyzers (MBSA)
and followed recommended best practices - Made copies of all software and build procedures
and placed offsite for disaster recovery - Control the desktop experience for backgrounds,
screen savers, colors, etc - Use AD GPO.
40Desktop and Server OSPERS Best Practices AD GPO
- Configure Active Directory according to your
organizations structure - Implement global policies and then implement
departmental policies - No Control Panel Access
- Standard Background
- Standard screen savers with password protected
Resume - Standard Color Scheme
- Standard Start Menu View and settings
- Redirected Start Menus
- Redirected My Documents
- Run only allowed Windows Applications !!!!!!
41Back Office ApplicationsWhat it was like
- No Web Security
- No Mail Security
- Netscape Suite Spot Server
- SQL Server 6.5
- Exchange Server 5.0
- Microsoft Office 97
- Adobe 4.0
- Flash (???)
- Java (1.???)
- Anti Virus 6.0 - 7.0
- Legato (4.???)
- No spyware or malware
- No Web IDE
42Back Office ApplicationsWhat we did
- We upgraded to.
- Symantec Web Security
- Symantec Mail Security with Brightmail
- IIS 6.0
- SQL Server 2000/2005
- Exchange Server 2003
- Microsoft Office 2003
- Adobe (latest)
- Flash (latest)
- Java (latest)
- Antivirus 10.0 with spyware/malware
- Arcserve 11.5
- Dreamweaver and Coldfusion
43Back Office ApplicationsPERS Best Practices
- Standardize on network file locations for all
users for Word, Powerpoint, Excel, etc - Standardize on other settings as much as possible
(auto archive, empty deleted items, empty
temporary internet files on exit, etc) - Lets look at a couple of items in more detail
(Web Security, Mail Security)
44Back Office ApplicationsWeb Security - What it
was like...
- Had each user read and sign an internet security
policy - Lock in IP address with DHCP reservation
- Create a firewall user entity matching the DHCP
IP address - Add the firewall entity to the internet access
group - Users could browse anywhere in the world, NO
restrictions - Users could log into any computer on PERS network
- Basically our web security consisted of allowed
or not allowed web access - Whenever supervisors wanted an access listing for
a particular user, I had to browse through TONS
of firewall logs. - Any sites that I wasnt sure of their content, I
actually had to go to the site to determine its
content - I eventually wrote a program to parse the logs
and give a report for a specific IP address
45Web security Configuration
46Guess what happened one weekend ???
- Someone was looking at things they were not
supposed to be looking at ??? - From the Executive Directors assistants
workstation
47Back Office ApplicationsWeb Security What we
did/Best Practices
- Implemented a web security proxy server (Symantec
Web Security) with automatic content filtering,
reporting features and that was Active Directory
aware - Configure internet browsers connection settings
via active directory group policy to use a proxy
server - Used AD GPO to prevent users from changing
browser proxy settings - Configure your proxy to require logins. This
reminds the users they are being monitored - Set up denied categories such as sex, games,
gambling, etc - Set up filtering on all allowed categories
- Set up some allowed sites lists for state
government sites for non-internet users - Implemented an autolock policy for repeated
violations - Do NOT allow temporary overrides for content
filtering - Make sure users can only login to the network on
their workstation - As a Result..
- We were able to create one firewall entity and
rule for the web security proxy server only - Eliminated individual firewall configuration by
using a product that integrated with our windows
active directory - Eliminated searching through firewall logs for
violations - Eliminated DHCP IP address reservations
- Eliminated foxpro program to parse firewall logs
for an individual sites visited because product
had reporting features by user - Stopped spyware that doesnt use IE proxy settings
48Web security Configuration
49Back Office ApplicationsEmail Security - What it
was like...
- File system AV on email server
- No spam detection, getting all types of garbage
- Exchange 5.0 which even allowed .exe file
attachments - Firewall allowed smtp traffic (email) from
universe - No access remotely via the web (secure or
unsecured), directors and managers wanted it - Basically, we had no email security
50Email Security Configuration
51Back Office ApplicationsEmail Security - What we
did...
- Upgraded to exchange 2003 (SP2), skipped 5.5 and
2000 - Implemented Symantec Mail Security (SMS) for
Exchange 2003 with spam and AV protection for the
email database - Started adding BAD words to SMS match lists
- Used real time black lists (RBL) of know spammers
- Created Blank subject/sender filters
- Reconfigured firewall to allow email from ISP
relays only, not the universe (cut down on
internal state attacks) - After a year or so of fighting manual match list
maintenance, we upgraded to SMS with Brightmail
technology subscription (WOW!!!!) - Configured suspect spam threshold and began
routing suspect spam email to a spam catcher
email account that we are monitoring - Configured whitelist for bank clients, etc
- Used exchange baseline analyzers and followed
best practices recommendations - Implemented SSL webmail and only allow the
firewall to connect
52Email Security Configuration
53Back Office ApplicationsEmail Security PERS
Best Practices...
- Implement a computer usage policy that has an
email usage section. - Require users to sign the computer usage policy
agreement page and keep in personnel files - Use an anti-spam, anti-virus, spyware/malware
aware product on your email server - Use Blank subject/sender filtering rules as well
as any others deemed necessary - Restrict attachment types
- Use whitelist for important customers that should
not go through the email filtering process - When implementing a webmail solution over the
internet, use SSL (https) - Only give secure webmail privileges to users that
need it - Implemented Mailbox quotas per user
54Business ApplicationsWhat it was/is like
- Using Foxpro 2.0 to develop small miscellaneous
applications - Small applications were everywhere in our
directory structure, no organization - Using no longer supported forteg3 OOP language
for LOB application - No adhoc reporting for users
- Had 2 environments, Test and Production
55Business ApplicationsWhat we did and Plan to
do
- Upgraded to Visual Foxpro for small apps and
adhoc reporting - Upgraded to Visual Studio for API small apps
- Plan to upgrade LOB application from forteg3 to a
new development platform (java, .net, ???) - Plan to implement some type of adhoc reporting
for users (Data warehouse, BI)
56Business Applications PERS Best Practices
- Have multiple environments, three if possible
(Test, User Acceptance Test and Production) - Have a designated directory structure for
applications - Implement good organization with a one to one
correlation between the source code and user
accessible application - Use as few development platforms as possible
- Standardize and stick with the standard
- Have procedures and follow them as much as
possible
57UsersWhat it was like
- Users were changing colors and you could not see
certain things - Users would change display resolutions and font
sizes - All type of cursors, backgrounds, screen savers,
gremlins, etc - Browsers were being taken over by spyware
- Systems would respond slow and/or erratically
- Users were constantly complaining of computer
problems (email garbage, etc..)
58UsersWhat we did
- Implemented a new business policy and decided to
upgrade the users every 3-5 years (Just Kidding) - Implemented standards via AD GPO
- Took away the ability to do anything they wanted
- Provided training and education
59UsersPERS Best Practices
- Involve the users in the upgrades
- Have them test and signoff that everything is
working properly before the upgrades are
implemented in production - Educate your users on the new changes in security
- Train your users on the new aspects of the
upgrades (OS, Office, etc..)
60Quick Summary
61Security Assessment
- We decided to wait until most of our upgrades
were in place before we had our assessment
performed - We inquired about Homeland security money and
were able to have the assessment done at no
expense to PERS - We got bids from three different companies and of
course selected one - We met with the selected vendor and agreed on an
assessment strategy - Our strategy was to perform an assessment on a
subset of our network instead of the entire
network. However, we made sure our subset had one
of each type of machine/device configuration in
the assessment (multiple environments scenario) - We also decided to keep the assessment very quit
to all personnel in order to try and get an
accurate picture of where we really were. Very
few people new the assessment was being
performed. - The assessment took about 3 months total
62Security Assessment
- The vendor assessed 10 different categories and
identified three levels of risk factor within
each category - High Risk, a severe security problem that could
cause loss of service or immediate access to
critical severs and file systems - Medium Risk, less severe problem and by itself
would not be an issue, but remediation would
provide incremental improvements in security - Low Risk, a vulnerability that is either very
rare or would require significant skill to
exploit or the potential exposure would be minimum
63Security Assessment
- At first, several of the technical services staff
posed questions after seeing unusual activity in
logs and on administrator email notifications - After a couple of days, the vendor had to ask for
an administrator account to get access - They had to ask how to get to our web server in
the service network - Initially, they said everything looks real
good!!!! - At the end of the assessment, the vendor said we
have performed assessments on 10-15 state
agencies and approximately 50 other entities in
Mississippi and that PERS was one of the best
they had seen. - They also mentioned that the assessment will
sometimes not be a fair indicator of an
agencies overall security status because just a
few missing patches will warrant a low score in a
category and drive the overall rating down
64Security Assessment
- PERS Overall score was a 2.5 out of a possible
4.0 - However, PERS knew that because of legacy systems
that not yet been upgraded, some of our internal
hosts and database assessments would receive
lower scores. - The vendor asked PERS if we were sure we wanted
older servers/OSs/etc.. scanned since we were
planning to upgrade them - What every security manager wants to hear
- Pileum was unable to compromise the PERS network
through the available open services
65Security Assessment Graph
66Security Assessment Recommendations
- The only way to completely secure any computer
device or data source is to disconnect it from
the network and place it in a vault where no one
has the key. In this case, the data would be
completely secure but totally inaccessible.
Therefore, there is a risk that must be assumed
with any computing device or data source that is
made accessible via network connections. It
should be a security managers goal to minimize
and be aware of the security risks, but not to
assume that they can or will ever be eliminated
completely.
67Security Assessment Recommendations
- PERS investigate all High risk or critical
vulnerabilities and their remedies - Review our security policies and procedures and
how they are enforced - Have ongoing assessments quarterly
- Implement an effective patch management solution
- Implement a Windows event log management system
68Going Forward PERS plans too
- Maintain software maintenance agreements on all
software - Finish our remaining planned upgrades
- Upgrade to SQL Server 2005 across the board
- Upgrade to Windows Server 2003 on legacy Systems
- Implement IE and other GPO settings
- Upgrade LOB application to current technologies
- Implement Self Service via the web
- Implement Security Assessment recommendations
- Implement a patch management solution
69Final Thoughts Remember !!!!!!!
- A wise person learns from his/her own mistakes
and experiences. - An even wiser person learns from others mistakes
and experiences. - Visit the PRISM website at www.prism-assoc.org
and download this presentation if you find any of
this information helpful - Take one of my business cards and shoot me an
email or give me a call if you would like to
discuss something in more detail