Usage Monitoring

1
Usage Monitoring

• Once a web site has been created (possibly in
  early alpha form) and deployed to be available
  across the Internet or an Intranet we have
  various options for monitoring the traffic that
  this site generates.

2
(No Transcript)
3
Why monitor usage?

• low level - is anyone visiting the site? (which
  might justify its existence).
• for usability purposes, such as
• identifying navigation problems
• determining user paths through a site
• examining exit and entry points to the site.
• for marketing reasons, such as
• determining which content is viewed, and which
  not
• which marketing activities are attracting users
• which activities are associated with sales.

4
Why monitor usage, cont.

• The most important point is the usage monitoring
  provides EVIDENCE (however imperfect) of how your
  site is being used.
• This evidence can be used as an agent for CHANGE
  (either a re-design of your site, or as a tool to
  persuade clients or managers for continued or
  greater support).

5
What data can we monitor?

• Firstly who is coming to the site (minimally from
  what company, e.g. bmw.co.uk but if the user
  filled in a form at your site we can link in the
  information about names, phone numbers etc...)
• where the user came from - did they use a link at
  another site or a search engine - if so what
  keywords did they type in?
• which pages they are viewing and what paths they
  take through the site.
• how long they visit a site for - and when they
  visit.
• which customers purchase the most and where they
  come from.

6
Where does the information about visitors come
from?

• There are a number of techniques, which rely on
  analysing the following data sources
• server log files (a popular technique)
• network packets
• page tagging.

7
Server log file analysis.

• Each web server will keep a record of all files
  sent to clients.
• This includes basic information about each file
  that is sent, including (but not limited to)
• the network address of the client
• the web browser used
• the referring page
• the status code and bytes sent
• the file that was sent.

8
Example server log
9
Log files analysis, cont.

• Log files record info on every file sent, as such
  a lot of information gets logged. (consider all
  the text on the previous slide represents one web
  page transmission - the html file associated
  images).
• Log files are often quite large even a low volume
  site might generate a 250,000 line long file each
  day (about 10MB in size).
• This raw data almost always needs software tools
  to help us analyse it.

10
Problems with server log analysis

• Accurately assessing really who is visiting a
  given site.
• the real path visitors take (caching, back
  button)
• referring site (bookmarks, typed in url, email
  link)
• how much time a user really spends on a page.

11
(No Transcript)
12
How are some limitations overcome?

• One of the major limitations is finding out who
  visits a site.
• the log file only contains information about
  files transferred from server to client - not
  directly who is visiting a site.
• Most log analysers infer from the log file and
  hostname or I.P. address what represents a user
  session.
• Some of the limitations can be overcome using
  cookies.

13
Using cookies to identify users

• A typical HTTP request

• This represents a request for the page
  /products/NetTracker/index.html, using the HTTP v
  1.0 protocol and from a the Netscape version 4
  web browser.
• The web server will send back a response, which
  will (hopefully) be the requested page, preceded
  with some header information.

14
Using cookies, pt 2

• Typical response

This response contains information about a cookie
(a piece of info generated by a server and
subsequently stored on a user machine)
15

• Analyse this!

16

• --------------------------------------------------
  ------------------Time of visit Wednesday, 7
  Nov 2007 , 160811Hostname ip-122-152-128-48.a
  sianetcom.netIP 122.152.128.48Referer Page
  Browser Baiduspider(http//www.baidu.com/sear
  ch/spider_jp.html)
• --------------------------------------------------
  ------------------
• --------------------------------------------------
  ------------------Time of visit Wednesday, 7
  Nov 2007 , 154308Hostname hos003-59440-rtr-ad
  sl-93.altohiway.comIP 84.252.215.93Referer
  Page http//www.google.co.uk/search?hlenieISO
  -8859-1qcameraforschoolsmetaBrowser
  Mozilla/4.0 (compatible MSIE 5.17 Mac_PowerPC)
• --------------------------------------------------
  ------------------
• --------------------------------------------------
  ------------------Time of visit Tuesday, 6 Nov
  2007 , 102912Hostname 89.122.213.193IP
  89.122.213.193Referer Page http//www.einternet
  directory.co.uk/Holidays.phpBrowser
  Mozilla/4.0 (compatible MSIE 6.0 Windows NT
  5.2 WOW64 SV1)
• --------------------------------------------------
  ------------------

17

• --------------------------------------------------
  ------------------Time of visit Monday, 5 Nov
  2007 , 114241Hostname 87.113.38.236.plusnet.p
  te-ag1.dyn.plus.netIP 87.113.38.236Referer
  Page Browser Mozilla/5.0 (Windows U
  Windows NT 5.1 en-GB rv1.8.1.8) Gecko/20071008
  Firefox/2.0.0.8
• --------------------------------------------------
  ------------------
• Time of visit Monday, 5 Nov 2007 ,
  114051Hostname 83-37-17.netrun.cytanet.com.cy
  IP 83.168.37.17Referer Page
  http//www.google.com/search?sourceidnavclientie
  UTF-8rlsWZPA,WZPA2007-42,WZPAenqmonthlycon
  tractBrowser Mozilla/4.0 (compatible MSIE
  7.0 Windows NT 5.1 .NET CLR 2.0.50727
  InfoPath.2)
• --------------------------------------------------
  ------------------
• --------------------------------------------------
  ------------------Time of visit Monday, 5 Nov
  2007 , 05835Hostname indesit.netcraft.comIP
  194.72.238.61Referer Page http//www.netcraft
  .com/survey/Browser Mozilla/4.0 (compatible
  Netcraft Web Server Survey)
• --------------------------------------------------
  ------------------
• --------------------------------------------------
  ------------------Time of visit Monday, 5 Nov
  2007 , 05101Hostname lj512117.crawl.yahoo.net
  IP 74.6.20.80Referer Page Browser
  Mozilla/5.0 (compatible Yahoo! Slurp
  http//help.yahoo.com/help/us/ysearch/slurp)
• --------------------------------------------------
  ------------------

18
Using cookies, pt 3

• The cookie information is saved on the users
  machine and is resent by the web browser with
  subsequent requests, e.g.

Log analysis tools can capture the cookie info
(which would be unique for each machine) and use
it to uniquely identify a user - because it is
sent on all subsequent requests we can also use
it to identify the path through a site, and
repeat visits.
19
Common steps in usage monitoring

• Gathering data (server log files, network
  packets, page tagging). For log files data is
  parsed at regular intervals, the other techniques
  often gather data in real time.
• Analyse the data (usually using complex heuristic
  methods to infer user paths, sessions etc..)
• Report on the analysed data (often the analysed
  data will be held in a separate database, reports
  can then be generated from this).

20
Usage monitoring software

• Often large companies will develop in-house
  solutions to analyse traffic, often written in
  the PERL language.
• Inexpensive shareware tools are available (e.g.
  AccessWatch at www.accesswatch.com)

21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
Commercial software

• There are a number of commercial tools for usage
  analysis, a common one is NetTracker
  (www.sane.com) which is a server log analysis
  tool, but with additional web server plug-ins.
• It allows the user to create a large number of
  custom reports of the database created by
  analysis of the log files.

26
Discussion Questions

• Hit vs Visit

27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)