Detecting and Defending against WebServer Fingerprinting

1

• Detecting and Defending against Web-Server
  Fingerprinting
• Annual Computer Security Applications Conference
  2002
• http//www.acsac.org/2002/abstracts/96.html
• Presented by Lee Hui Huang

2
Outline

• Introduction
• How to perform HTTP server fingerprinting
• Tools for fingerprinting
• How to detect fingerprinting attempts
• How to protect against HTTP server fingerprinting

3
Definition of Fingerprinting

• Fingerprinting is a heuristic method of observing
  the behavior of a software component for the
  purpose of determining its identity.
• In general, it involves sending specific requests
  to the component and observing the response.

4

• For HTTP servers
• Send specifically chosen HTTP Request messages
  and observe the Response messages.

5
Information gathering Goals

• Some of the information an attacker hopes to
  obtain
• open ports
• services running on these open ports
• Operating System

6
Why fingerprinting is necessary

• Without the relevant information, an attacker
• Wont know what exploits to use
• Have to try all possible attacks.
• Trying all possible attacks
• Time consuming
• will generate a lot of suspicious traffic

7
Why fingerprinting is necessary

• With the necessary information, an attack can
  be more efficient.
• For example
• I know a server is running IIS 5.0 on Windows
  2000. So try IPP buffer overflow (CA-2001-10)
  attack to gain a command prompt on the server.
• If an Apache web server 1.2.2 is running, try to
  exploit Apache Web Server Chunk Handling
  Vulnerability (CA-2002-17) to execute arbitrary
  code on it.

8
Why fingerprinting is necessary

• Malicious programs like worms also need to do
  some probing (like port scans) in order to select
  potential victims to attack.

9
Why web servers are attractive targets

• Very common. Easy to find and connect to one.
• Much vulnerability exists in existing web
  services.
• Not all web servers have all the necessary
  patches

10
How Fingerprinting is possible

• To generate a web servers fingerprint a set of
  characteristics that differentiates a specific
  servers use of HTTP from that of other servers
  must be identified.

11
How Fingerprinting is possible

• Specifications defined in the HTTP RFCs are
  agreed upon, not enforced.
• Words like MUST, SHOULD and MAY used to described
  features. E.g.
• All responses to the HEAD request method MUST
  NOT include a message-body,
• The field value MAY be preceded by any amount
  of LWS, though a single SP is preferred.
• The difference in degree of compliance allows
  HTTP servers to be fingerprinted

12
Fingerprinting Methodology

• A list of characteristics to be use for
  fingerprinting can be assembled.
• For each of these characteristics, design a HTTP
  Request that will provoke a Response exhibiting
  the characteristic.

13
Fingerprinting Methodology

• Characteristics can be divided into 3
  categories
• Lexical specific words, phrases and punctuation
  used.
• Syntactic the ordering and context of elements
  in the response
• Semantic a servers specific interpretation of a
  Request from among the possible interpretations.

14
Fingerprinting Methodology

• Lexical category variations in the actual words
  used, capitalizations and punctuation

15
Fingerprinting Methodology

• Methods
• Check for difference in response code message
• Each HTTP response message contains a number
  indicating whether the attempt to satisfy the
  request has succeeded or not as well as the
  corresponding text message
• E.g. for the error code 404,
• Apache uses Not Found
• Microsoft IIS/5.0 uses Object Not Found.

16
Fingerprinting Methodology

• 2. Difference in the header words
• variation occurs in the capitalization
• For example
• some servers use Content-Length
• Others use Content-length

17
Fingerprinting Methodology

• 3. Difference in use Line Terminators
• RFC specified behavior is to use \r\n to
  separate elements of the header
• But some older servers use only \n
• More recent servers use \r\n

18
Fingerprinting Methodology

• The HTTP response message also contains a field
  called Server.
• Example
• HTTP/1.1 200 OK
• Connection close
• Date Thu, 06 Aug 1998 120015 GMT
• Server Apache/1.3.0 (Unix)
• Last-Modified Mon, 22 Jun 1998 092324 GMT
• Content-Length 6821
• Content-Type text/html

19
Fingerprinting Methodology

• Syntactic category the difference in the
  ordering and format of Request elements like the
  headers and the contents

20
Fingerprinting Methodology

• Methods
• Difference in ordering of Headers
• For example
• For Apache servers, the Date header is before
  Server header.
• For Netscape-FastTrack/4.1, it is the opposite.

21
Fingerprinting Methodology

• 2. Difference in ordering of list items
• Sometimes a list of items is returned as the
  contents of a header
• E.g. If the OPTIONS method is sent in an HTTP
  Request, a list of allowed methods for the given
  URI are returned in an Allow header.

22

• The order of these elements tends to vary between
  servers.
• But not all headers with lists can be used. E.g.
  the Content-Language header which identifies the
  language types

23
Fingerprinting Methodology

• Difference in formatting of some elements.
• Some elements have formats that are variable or
  unspecified by the RFCs.
• E.g. ETag header which provides a unique
  identifier for a given document
• Apache/1.3.11 uses 0-574 383791543a5b7811
• Jigsaw/2.1.2 uses mvancts0jndthg

24
Fingerprinting Methodology

• Semantic category When a request message is
  received, the server has to decide on a
  interpretation for it before it can respond to
  it.
• Many variation on how servers interpret both
  well-formed and mal-formed Requests

25
Fingerprinting Methodology

• Methods
• Check for the Existence of Response Line and
  Headers
• E.g. A specially crafted request message will
  cause some servers to believe the requester is an
  HTTP/0.9 based client
• These server will respond with a message without
  any headers.

26
Fingerprinting Methodology

• 2. Look for specific headers
• A server chooses what headers to include in a
  Response.
• Many of them (e.g. ETag) are optional.
• E.g. When there is a 501 Method Not Implemented
  error,
• Apache servers send an Allow header with a
  list of the allowed methods for the designated
  URI
• Jigsaw/2.1.2 does not.

27
Fingerprinting Methodology

• 3. Compare the response code from different
  servers for ad Hoc requests
• Given a malformed request, different servers may
  assign it a different type of error.

28

• E.g. When a random text string (e.g. hi) is
  sent,
• Apache interprets it as a bad Request from a
  HTTP/0.9 client.
• Will respond with a header less message warning
  that the method hi is not implemented.
• Microsoft IIS interprets it as a malformed
  Request from an HTTP/1.X client.
• Will respond with a 400 Bad Request.

29

• Many ways of developing such ad hoc test
  requests.
• Can try changing the method line, the headers and
  the body size
• For example
• "GET
• "GET / HTTP/999.99
• "GET / HTTP/9.Q"
• "HEAD /////////// HTTP/1.0"
• "HEAD /.\ HTTP/1.0"
• "HEAD /asdfasdf/../ HTTP/1.0"
• "HEAD /./././././././././ HTTP/1.0

30
Fingerprinting Methodology

• Make use of the different error ranges of the
  different servers
• Server URL Length Response
• --------------------------------------------------
  ---------------------------------Apache/1.3.12
  (Win) 1-216 404 Not Found
• 217-8176 403 Forbidden
• 8177-up 414 Request-URI Too Large
• --------------------------------------------------
  ---------------------------------
• Netscape-FastTrack/4.1 1-4089 404 Not found
• 4090-8123 500 Server Error
• 8124-8176 413 Request Entity Too
  Large
• 8177-up 400 Bad request

31
Tools for Fingerprinting

• HMap was written by the authors of this paper
  http//wwwcsif.cs.ucdavis.edu/leed/hmap/
• HMap uses the test described above.

32

• Sample HMap output
• matches mismatches unknowns
• Apache/1.3.22 (Win32) 116 0 7
• Apache/1.3.12 (Win32) 113 3 7
• Apache/1.3.14 (Win32) 113 3 7
• Apache/1.3.17 (Win32) 113 3 7
• Interpreting the result
• 116 of the tested characteristics were a direct
  match for Apache/1.3.22
• None of them were different
• 7 of them couldnt be determined.

33
Tools for Fingerprinting

• NMap (http//www.insecure.org/nmap/)
• NMap is a versatile port scanner that can perform
  ping sweeps, OS fingerprinting, among others.
• Command to use
• nmap O sV www.website.com
• -O option performs OS fingerprinting
• -sV option performs version detection

34

• Sample output
• Starting nmap 3.50 ( http//www.insecure.org/nmap
  ) at 2004-03-29 2002 Malay Peninsula Standard
  Time
• Interesting ports on www.website.com
  (192.168.123.144)
• (The 1650 ports scanned but not shown below are
  in state closed)
• PORT STATE SERVICE VERSION
• 21/tcp open ftp Microsoft ftpd
• 80/tcp open http Microsoft IIS
  webserver 5.1
• 135/tcp open msrpc Microsoft Windows
  msrpc
• 139/tcp open netbios-ssn
• 443/tcp open https?
• 445/tcp open microsoft-ds Microsoft Windows XP
  microsoft-ds
• Device type general purpose
• Running Microsoft Windows 95/98/MENT/2K/XP
• OS details Microsoft Windows Millennium Edition
  (Me), Windows 2000 Professional or Advanced
  Server, or Windows XP
• Nmap run completed -- 1 IP address (1 host up)
  scanned in 42.972 seconds

35
Tools for Fingerprinting

• Telnet can be used to connect to port 80 of a web
  server to get its name.
• Example
• Use telnet www.website.com 80 to connect to the
  server.
• Then issue a GET request GET / HTTP/1.1

36

• Sample response message received
• HTTP/1.1 200 OK
• Date Mon, 29 Mar 2004 135224 GMT
• Server Apache/1.3.29 (Unix) mod_ssl/2.8.16
  OpenSSL/0.9.7d
• Transfer-Encoding chunked
• Content-Type text/html
• Via 1.1 Application and Content Networking
  System Software 5.0.7
• Connection Close
• If source code of the web server is available, it
  can be modified to return a false value in the
  Server field.

37
Tools for Fingerprinting

• Netcraft (www.netcraft.com) conducts periodic
  surveys to determine the market share of various
  web servers.
• One of their feature allows a user to find out
  information about a web server. E.g.
• Web service running
• Operating System
• Uptime of the server

38
(No Transcript)
39
Stealth Techniques

• Running the full suite of HTTP server
  fingerprinting tests is very effective in
  identifying a server.
• But it is easy to be detected by an IDS.
• An attacker can try a few techniques to minimize
  detection

40
Stealth Techniques

• 1. Have several computers and run a subset of
  tests from each of them and then correlate the
  data.
• IDSes might not detect a pattern of behavior if
  it analyzes records on a host by host basis.

41
Stealth Techniques

• 2. Run the tests over a long period of time.
• If the IDS notice one unusual Request then it
  might classify it as a random error and flush the
  observation after some time.

42
Stealth Techniques

• 3. Do only the short forms of large Request tests
  
• Server URL Length Response
• --------------------------------------------------
  ---------------------------------Apache/1.3.12
  (Win) 1-216 404 Not Found
• 217-8176 403 Forbidden
• 8177-up 414 Request-URI Too Large
• --------------------------------------------------
  ---------------------------------
• Netscape-FastTrack/4.1 1-4089 404 Not found
• 4090-8123 500 Server Error
• 8124-8176 413 Request Entity Too
  Large
• 8177-up 400 Bad request
• A URL of length 217 is enough to differentiate
  the servers.

43
Stealth Techniques

• 4. Change the contents of these large test
  requests so they vary in more than just the
  length.
• Currently the long requests are composed of all
  the same characters and do not look like legal
  Requests.

44
Stealth Techniques

• 5. Make a search tree of characteristics.
• Do the minimal subset of tests that can identify
  a server, instead of doing every test

45
Stealth Techniques

• 6. Use URL encoding to mask the contents of
  Requests.
• URL encoding rewrites the request URL in
  hexadecimal.
• Some IDSes do not perform URL decoding.
• Pattern matching involving URL encoding is more
  cpu intensive

46
Detecting Fingerprinting Activities

• Currently it is difficult to find an IDS that
  looks for web server fingerprinting activity.
• IDSes that looks for CGI attack probes do exist
  and presumably could be extended to include
  detection of HTTP-based identity probing.

47
Detecting Fingerprinting Activities

• Probing does not necessary mean an attack will
  follow.
• Hence IDS shouldnt issue an alert every time
  probing is detected.
• Ideally, there should be a more comprehensive
  system to determine what to do with this
  information.

48
Detecting Fingerprinting Activities

• Some suspicious HTTP request messages an IDS can
  look for
• Unusual Request Element Size
• Some tests use very large elements.
• E.g. large URIs and large numbers of headers

49
Detecting Fingerprinting Activities

• 2. Use of Unknown or Unusual methods
• Watch out for use of Unknown methods (e.g.
  QWERTY) or Unusual methods that normal browsers
  rarely or never send (e.g. TRACE).
• The same applies to unknown or unusual header
  fields.

50
Detecting Fingerprinting Activities

• 3. Look for unusual constructions
• Most Requests have a fairly simple format
• E.g. method line of METHOD URI HTTPVERSION
  followed by common headers
• Unusual request e.g. those with an inappropriate
  body or those that use incorrect line terminators
  should be examined

51
Detecting Fingerprinting Activities

• 4. Look for Unusual method line syntax
• Most browsers are fairly well behaved when they
  issue a Request
• Watch for unusual spacing or corrupted version
  information
• E.g. "GET / HTTP/9.Q"

52
Detecting Fingerprinting Activities

• 5. Verify the web clients identity
• If a requests User-Agent field indicates one
  type of browser but the client behaves like
  another, suspicion should be raised
• May result in many false alarms since some
  privacy applications purposely hide the name of
  the browser

53
Detecting Fingerprinting Activities

• Some Suspicious HTTP Reponses an IDS can look
  for
• 1. Headerless Responses
• HTTP/0.9 type clients are rare these days.
• If a server sends back a headerless response, it
  is possible that someone is pretending to be such
  a client or a request that has confused a server
  was received

54
Detecting Fingerprinting Activities

• 2. Unusual and Repeated Errors
• Many of the fingerprinting tests are attempts to
  provoke error responses. (non-200 OK).
• Rare error Reponses e.g. 413 Request Entity Too
  Large should raise suspicion.
• Errors that are common in other servers but rare
  in a particular server should also be noted.

55
Defending against Web Server Fingerprinting

• 1. Decide on a set of Common Interface and
  Behavior for web servers
• Fingerprinting succeeds because each vendor has a
  slightly different interpretation of how an HTTP
  server should behave
• If all the developers agree on a common
  interface, then fingerprinting would be more
  difficult
• As web server attacks become more frequent and
  severe this sort of strategy may become more
  attractive

56
Defending against Web Server Fingerprinting

• 2. The web server software should have
  configuration options to allow the administrator
  to have more control over the lexical, syntactic
  and semantic elements of Responses.
• E.g. by modifying a configuration file, the
  server can be made to behave like another server
  or act like no known server.

57

• Currently, it is uncommon to find any server
  configuration options that allow identity hiding.
• Apaches ServerTokens directive allows an
  administrator to vary the Server headers level
  of detail, but doesnt hide it entirely.
• E.g. the server can either return
• Server Apache or
• Server Apache/2.0.41 (Unix)

58
Defending against Web Server Fingerprinting

• 3. Use Variable Responses
• E.g. instead of returning the same Not Found
  message all the time, a number of different
  messages could all be used randomly. E.g. File
  Not Found or Not found

59

• Fingerprinting becomes more difficult since a
  static picture of the target can not be developed
  
• Ability of a client and server to converse will
  not be affected since each of the variations
  would be semantically equivalent.

60
Conclusion

• Hiding the identity of an HTTP server is no easy
  task.
• The response to the carefully created HTTP
  requests contains enough information to identify
  a web server with high certainty.

61

• However by using the same fingerprinting
  methodology as reference, IDS that can detect
  fingerprinting attempts can be developed
• Web server developers can increase the security
  of their web servers by modifying the servers so
  that fingerprinting attempts will be unreliable

62

• References
• Detecting and Defending against Web-Server
  Fingerprinting
• ACSAC 2002
• http//www.acsac.org/2002/abstracts/96.h
  tml
• Hmap web page
• http//wwwcsif.cs.ucdavis.edu/leed/hmap/
• Nmap web page
• http//www.insecure.org/nmap/
• NetCraft
• http//www.netcraft.com
• CERT Advisories
• http//www.cert.org/advisories/