An Introduction to TMT Trunk Monitoring Tool

1
An Introduction to TMT (Trunk Monitoring Tool)

• David Parish
• Loughborough University
• D.J.Parish_at_lboro.ac.uk

2
Overview

• What is TMT?
• What it does and how it works.
• Detailed Operation
• Using TMT
• Important Issues
• Questions

3
What is TMT

• A highly visualised network performance display
  tool.
• Uses size and colour to highlight important
  aspects of the network performance.
• Presents a pictorial view of Node Error, Node
  Discard and Trunk Throughput.

4
Background and History

• Designed by Loughborough University in response
  to a need from SMDS Operations staff.
• Aims to provide a fast overview of SMDS error and
  throughput performance by collecting existing
  SNMP data from the switches.
• Designed to be used in conjunction with existing
  tools.

5
Concept of TMT System
BT Management Network
SMDS Network
TMT Display (Walsall)
TMT Data Gatherer (Oswestry)
6
TMT Machine
Data Processing and display generation
Database
SNMP Data Collection
7
Main Features

• Shows relative value of data.
• Uses size and shape of objects to convey
  information.
• Two level hierarchy
• Top Level for general view
• Lower Level shows more localised nodes
• User configurable thresholds
• Historical replay for post-event investigation (3
  hour history)

8
TMT Top Level (No problems)
9
Main Front Panel Objects

• Meshes
• Nodes (and node names) in Mesh A and Mesh B
• Non-MSP Trunks
• Regions
• Two Main Nodes (and names) in each Region
• A selection of other Nodes and Trunks connecting
  to the Main Nodes
• User Controls

10
Nodes

• Colour
• Blue CBX500 Green Other.
• Yellow/Red Node Errors (Worst Case).
• Grey No Response from ANY port on node.
• Does NOT mean that the node is down.
• Size/Shape
• Rectangle Main Regional Node.
• Circle Size Relative Node Error.

11
Top Level Node Errors

• Error Thresholds (2) set via slide control
• Yellow First Threshold
• Red Second Threshold
• Red Threshold also controls relative size
• Node Error defined as The Worst Case Error or
  Discard performance seen on any monitored
  interface of the displayed node.

12
TMT Top Level (Some Problems)
13
TMT Top Level (Serious Problems)
14
Top Level Trunks

• Shows the relative throughput in each direction
  as a set of moving arrows/dashes above and below
  a solid line.
• Speed of movement has no significance.
• Colour
• Green Line ALL trunk data collected.
• Green Arrow Throughput below threshold.
• Red Arrow Throughput above threshold.
• Grey No SNMP response for specific object.
• Does NOT mean trunk is down!

15
Regions (1)

• A region colour may appear as a box around the
  objects in the given region.
• No box All Node Error data in the region below
  preset thresholds and some data available.
• Grey Box No data for ANY interface in the
  Region available.
• Yellow Box AT LEAST 1 Port on one Node has an
  error performance above the yellow threshold.

16
Regions (2)

• Red Box AT LEAST 1 Port on one Node has an
  error performance above the red threshold.
• Mouse Click on the Region Nodes to move to Lower
  Level sections to investigate further.
• Note that Lower Level Trunk Throughput
  information is NOT SHOWN on the Top Level.

17
User Controls

• Lower section of display has user controls for
• Node Error Thresholds
• Move sliders with mouse.
• Red MUST BE gt Yellow.
• History
• Slider to determine Minutes Ago.
• Click on OK to update display.
• System will automatically update data from the
  Gatherer every 10 seconds.

18
Lower Level Introduction

• Shows more detail in a specific section of the
  network.
• Additional Local Nodes and Trunks.
• Additional Node Error information.
• Lower Level called by mouse click on Top Level
  Node.
• TMT dynamically computes Lower Level Topology.
• Calculated with a predetermined second node.

19
TMT Lower Level
20
Lower Level

• Direct Trunks and those connected via one
  intermediate Node are displayed.
• All Node Names are displayed, but not Trunk
  Names.
• User set information (thresholds, history) are
  carried forward from the Top Level.
• Change via a return to the Top Level.
• New performance data are continually collected as
  for the Top Level.

21
Lower Level Trunks

• Provides an additional Dotted Line feature.
• Shows Worst Case Error Performance on either Port
  connected to the given trunk.
• Discards are NOT considered.
• The greater the Space/Mark ratio, the greater the
  Error rate.
• Press BACK to return to the Top Level.

22
Issues to note (1)

• Grey. Basically means data not available via
  SNMP, but
• Node only Grey if ALL DATA UNAVAILABLE.
• Region only Grey if ALL DATA UNAVAILABLE.
• Trunk element Grey if SPECIFIC DATA UNAVAILABLE.

23
Grey Example
24
Issues to note (2)

• Node Errors. WORST CASE shown!
• This may or MAY NOT relate to the Trunk shown.
• Regions. Top Level shows some Nodes and
  summarises Error Performance for the others.
• Worst Case Node Error Performance shown via
  Region Colour Box.
• Lower Level Throughput NOT represented via Region
  Colour.

25
Issues to Note (3)

• Lower Level Dotted Trunk feature shows Node
  Error only (eg NOT discards).
• Do NOT Minimise the Viewer Window. (There is
  nothing behind it!!)
• If you do, best to turn the power off then on.
• Then follow Startup operations.
• Lower Level may not show EVERY trunk in the given
  Region, but all nodes are shown.

26
Issues to Note (4)

• Trunk FAILURES are NOT directly shown.
• A trunk which is down will appear as a green
  trunk with no traffic.

27
Suggestions for Using TMT (1)

• Assuming appropriate thresholds have been set
• Node error breaches indicate high error/discard
  rates on one or more interfaces on the switch.
• Lower Level may show more detail.
• Dotted line shows worst case error on trunk.
• Use SPICES to get numerical data for the
  interfaces. (TMT has identified the Node)
• High error rates may be seen on one interface BUT

28
Suggestions for Using TMT (2)

• If trunk has shut down, traffic may reduce.
• Look for zero throughput.
• If a colour box appears around a region
• Find the similarly coloured nodes.
• Look for the largest node.
• May be on the Lower Level, so step through with
  mouse.
• Follow previous suggestions.

29
Suggestions for Using TMT (3)

• Look for trunks with high throughput.
• May indicate re-routed traffic
• Could indicate other problems
• Look for associated erroring nodes
• Use history feature to replay network events
• Remember to reset to current time afterwards!

30
Possible Scenarios (1)

• Scenario 1
• A trunk generates significant errors, then shuts
  down. Assume traffic is rerouted.
• Initially
• A region colour will go red.
• Two red regions MAY indicate an inter-region
  trunk.
• One or more Top Level nodes MAY go red and become
  large.
• If not, search lower levels

31
Possible Scenarios (1)

• If trunk shown on Lower Level, dotted feature may
  indicate trunk.
• If not, find error data for all ports on red node
  using other tools (TMT has indicated node).
• High error rate may indicate problem.
• If trunk shuts down, traffic may reroute
• Node/trunk errors may return to normal
• Search for trunks with abnormally low throughput.

32
Possible Scenarios (1)

• Otherwise, use history feature to identify large
  red node.
• Then use other tools to find port with zero or
  very low throughput.

33
Possible Scenarios (2)

• Scenario 2
• A trunk becomes overloaded
• A Region should go red
• Two red Regions would indicate an inter-region
  trunk overload
• One or more Top Level Nodes MAY be large and red.
• Search the Lower Levels until at least one large
  red node is found.
• Associated Trunk should have large red arrows.

34
Possible Scenarios (2)

• Trunk may NOT show the dotted feature!
• This does not respond to discards.
• Other tools may be used to check ports on large
  red node/nodes to identify ports with high
  discard rates.

35
Possible Scenarios (3)

• Scenario 3
• One Node Fails totally
• Search Lower Levels to find grey node.
• Yellow/Red nodes may also be seen as a result of
  lost/error traffic.
• But, be aware that grey simply means the Gatherer
  gets no response from the node.

36
Possible Scenarios (4)

• Scenario 4
• A Trunk Fails
• This should be indicated by other tools.
• Initially, one or more large red nodes should
  appear. Record these for further investigation.
• Later, if traffic is redistributed
• Look for Trunks/Ports with no traffic

37
Possible Scenarios (5)

• Scenario 5
• SMDS Management Network fails
• All objects become grey!

38
Possible Scenarios (6)

• Scenario 6
• Walsall/Oswestry Management Network fails
• Top Level will be green or blue
• Warning window will appear
• Watch out if this is behind Top Level Window!

39
Starting and Stopping TMT

• The Gatherer should run continuously.
• Can be shutdown from the Viewer if necessary.
• ssh tmt_at_gather
• enter TMT's password
• su root
• enter root's password
• poweroff

40
Viewer Shutdown

• Log into the Viewer as super user
• su root
• enter the root password.
• Type poweroff
• After a few seconds the Viewer will display
  "Power Down". Both machinescan now be powered
  down.

41
Startup

• Turn on the Gatherer (should already be on)
• At the Viewer
• Login as tmt
• Enter Password
• Type y ltrtngt if asked if user is authorised to
  use system
• Type startx ltrtngt

42
Changing Passwords (1)

• On the Viewer, press Ctrl-Alt-F2.
• Login as TMT
• Login to Gatherer
• tmt_at_WAL-WS-TMT-display ssh gather
• Enter current Gatherer password

43
Changing Passwords (2)

• tmt_at_OW-TMT-gather passwd
• Changing password for tmt
• (current) UNIX password ltPasswordgt
• Enter new UNIX password ltNew password for tmt
  accountgt
• Retype new UNIX password ltNew password for tmt
  accountgt

44
Changing Passwords (3)

• Change the root passwords
• tmt_at_OW-TMT-gather su
• Password ltPasswordgt
• OW-TMT-gather/tmt passwd root
• Enter new UNIX password ltNew password for rootgt
• Retype new UNIX password ltNew password for rootgt

45
Changing Passwords (4)

• Now exit
• OW-TMT-gather/tmt exit
• tmt_at_OW-TMT-gather exit
• logout
• tmt_at_WAL-WS-TMT-display exit
• Login prompt will be displayed.
• Change back to the normal TMT display
• Press Ctrl-Alt-F7.

46
Communication Problems

• If, on Startup, the Viewer cannot see the
  Gatherer to collect configuration data a warning
  window is generated
• Connection attempts will continue

47

• If communication is lost during normal operation,
  a warning window is displayed.
• Connection attempts will continue.

48

• Thank you..
• Any Questions?
• Suggestions/Comments to BT/LU team
• Help and support to LU Team
• Mark Sandford J.M.Sandford_at_lboro.ac.uk
• Tel [phone]
• David Parish D.J.Parish_at_lboro.ac.uk
• Tel [phone]
• Iain Phillips I.W.Phillips_at_lboro.ac.uk
• Tel [phone]