What is Fault Tolerant Ethernet (FTE)?

1
TOPICS

• What is Fault Tolerant Ethernet (FTE)?
• How does FTE work?
• How is FTE implemented?

2
Why FTE?

• Needed a next generation control network for
  Experion PKS
• gt10x performance, determinism, security, fault
  tolerance
• Reduce cost of communication infrastructure and
  support
• Reduce cost of connection to PCs and IT networks
• Ethernet preferred
• Industry trend to industrial Ethernet
• Industry bus protocols migrating to Ethernet
• FF H1 ? FF HSE
• Profibus ? ProfiNet
• DeviceNet/ControlNet ? Ethernet/IP
• Modbus ? Modbus/TCP
• Etc.
• Ethernet equipment perceived as not industrial
  enough
• No suitable fault tolerance approach
• FTE provides the required fault tolerance, using
  Cisco switches to provide determinism and
  security.

3
Experion PKS

• FTE is the control network of Experion PKS.
• Analogous to TPS LCN/UCN and PlantScape
  ControlNet.

Operator Stations
to production management equipment
Supervisory Controller
System Server
Remote Users
Engineering Station
Remote Server
FTE
Multi-loop Controller
Direct I/O Module

• Performance
• Determinism
• Fault tolerance
• Security

Other Industrial Devices
4
What is Fault Tolerant Ethernet (FTE)?

• FTE is the control network of Experion PKS.
• Analogous to TPS LCN/UCN and PlantScape
  ControlNet.
• Dedicated to the control mission
• Fault-tolerant
• Fast performance
• Deterministic
• Secure
• Not an IT network, but leverages IT technology to
  lower cost of
• FTE network infrastructure
• Connection to IT networks
• Connection to 3rd party Ethernet devices
• Maintenance and support
• 3 Cisco switches qualified for R200
• 100/1000 Mbps single and multi-mode optical
  fiber up to 70 km
• Security and determinism functions required for
  control mission
• Preferred supplier by many customers

.
5
What is FTE?

• An FTE network has redundant switches and cables.
• Topology 2 parallel trees joined at the top to
  form one network.

• An FTE node connects to both trees.

• An Ethernet node (non-FTE) connects to either
  tree.

6
What is FTE? With Ethernet nodes

• An FTE network has redundant switches and cables.
  
• 1 communication path between Ethernet nodes

7
What is FTE? With FTE Ethernet nodes

• An FTE network has redundant switches and cables.
  
• 1 communication path between Ethernet nodes
• 2 communication paths between an FTE node and an
  Ethernet node

8
What is FTE? With FTE Ethernet nodes

• An FTE network has redundant switches and cables.
  
• 1 communication path between Ethernet nodes
• 2 communication paths between an FTE node and an
  Ethernet node

9
What is FTE? With FTE nodes

• An FTE network has redundant switches and cables.
  
• 1 communication path between Ethernet nodes
• 2 communication paths between an FTE node and an
  Ethernet node
• 4 communication paths between FTE nodes

A - A
10
What is FTE? With FTE nodes

• An FTE network has redundant switches and cables.
  
• 1 communication path between Ethernet nodes
• 2 communication paths between an FTE node and an
  Ethernet node
• 4 communication paths between FTE nodes

A - B
11
What is FTE? With FTE nodes

• An FTE network has redundant switches and cables.
  
• 1 communication path between Ethernet nodes
• 2 communication paths between an FTE node and an
  Ethernet node
• 4 communication paths between FTE nodes

B - B
12
What is FTE? With FTE nodes

• An FTE network has redundant switches and cables.
  
• 1 communication path between Ethernet nodes
• 2 communication paths between an FTE node and an
  Ethernet node
• 4 communication paths between FTE nodes

B - A
13
TOPICS

• What is Fault Tolerant Ethernet (FTE)?
• How does FTE work?
• How is FTE implemented?

14
How Does FTE Work? FTE path status

• Each FTE node continually issues short diagnostic
  messages to test each path to every other node,
  and builds a status table.
• Below is a nodes status table. The first 2 nodes
  are FTE nodes, and the last 4 nodes are
  singly-connected Ethernet nodes.
• A bad link displays as SILENT.
• If the B cable to the FTE-GUS node fails, AgtB and
  BgtB ? SILENT.
• If the crossover cable fails, AgtB and BgtA ?
  SILENT on all nodes.

15
How Does FTE Work? - Software Architecture
Operational Details
CCA Fault Tolerant Ethernet
Intermediate Driver

• For Microsoft applications an intermediate driver
  is inserted between layers of NDIS (Network
  Driver Interface Standard)
• The TCP stack and its connection to the OS are
  unaffected.
• The FTE software has the ability to intercept,
  analyze, and redirect messages
• The location of this driver in the Kernel while
  still using standard calls is what gives FTE its
  openness and performance.

Config
Tool /
User Interface
TPS System Management MMC Plug-in
WinSock2
User Mode
Kernel Mode
TCP / UDP
Transport
IP
Driver
protocol
Ethernet frame
Ethernet frame
miniport
WDM
NIC Switch
FT Manager
Intermediate
Driver
Virtual
Virtual
Driver A
Driver B
Fault-Tolerant
Ethernet
protocol
Software
Ethernet frame
NDIS
Control flow
Ethernet frame
Miniport
Data flow
miniport
miniport
Driver
Driver A
Driver B
Ethernet frame
16
How Does FTE Work? - Fault Detection
Operational Details
CCA Fault Tolerant Ethernet
FTE nodes

• There are four possible paths between two FTE
  Nodes
• Sequence numbered messages are periodically sent
  from both ports to exercise all four paths
• The status of the four paths is sent back in the
  next test message transmission
• Loopback (shown in red) is also checked to
  diagnose the inter-switch link.
• From this data the best possible path from Node 1
  to Node 2 is determined and used.
• Link carrier status can also be checked for rapid
  detection (hardware dependent).

6
17
How Does FTE Work? - Fault Detection
Operational Details
CCA Fault Tolerant Ethernet
Ethernet nodes

• There are two possible paths between an FTE and a
  non-FTE Node
• The best viable path is chosen (using the data
  derived from the test messages)
• No test messages are required from the non-FTE
  nodes.
• Singly attached nodes can receive and use test
  message data to determine LAN health.
• If the inter-switch link is bad, multicast
  messages are sent via both ports to maximize the
  connectivity to singly attached nodes

FTE Node 1
SW A
SW B
Ethernet Node 2
Ethernet Node 1
7
18
How Does FTE Work? Security and
Determinism
Plant Automation System Levels
CDA Control Data Access DSA Distributed
Systems Architecture
Domain Controller
PKS Server
APC
PHD Server
Station
Station
Management Level 3
Operation Level 2
FTE
Switch A
Switch B

Control Level 1

• This diagram shows levels of the plant automation
  system (level 0 field devices is not shown).
• FTE is the Experion PKS network for the control
  and operation levels (1 and 2 ).

19
How Does FTE Work? Security and
Determinism
Firewall hides all but servers

• Firewall hides/secures Level 2 and Level 1--
  Only L2 PKS Servers are visible

• PKS server on L3 consolidates and makes available
  L2 data / alarms for applications via DSA

20
How Does FTE Work? Security and
Determinism
Firewall hides all but servers
L1 L2 Broadcast, Multicast, Unicast Storm
Suppression
L1 L2 Bandwidth Allocation
L2 CDA Traffic Prioritized High
L1 Restricted to CDA and FTE Traffic Only

• Firewall hides/secures Level 2 and Level 1--
  Only L2 PKS Servers are visible

• PKS server on L3 consolidates and makes available
  L2 data / alarms for applications via DSA

• FTE switches provide
• Port Filtering between L2 and L1 to allow only
  CDA and FTE messages for control
• L2 bandwidth allocation to ensure that L2
  supervisory traffic is not disrupted
• L1 bandwidth allocation to ensure that L1 control
  is not disrupted
• Broadcast, Multicast, Unicast storm suppression
  to maximize FTE network availability

21
TOPICS

• What is Fault Tolerant Ethernet (FTE)?
• How does FTE work?
• How is FTE implemented?

22
Basic FTE Configurations

• An FTE network interconnects clusters of nodes.
  
• A cluster is a group of nodes with high
  intercommunication, typically associated with the
  same process unit.

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Experion Stations
Application Control Environment
RedundantServers
Engineering Tools
FTE
FTE
FTE
FTE
FTE
FTE
FTE
FTE
C200
FTE
23
Basic FTE Configurations

• A minimum FTE network is one pair of cluster
  switches larger networks could have several
  cluster switch pairs connected to backbone
  switches.

Firewall
To Plant Information Network (PIN)
Backbone Switches
History, Advanced Control
Cluster Switches
UNIT 1 CLUSTER
24
How Is FTE Implemented?

• Cabling CAT5 STP copper single and multi-mode
  fiber optic.
• Cisco switches 24/48 STP ports 2 GBIC ports
  10 GBIC ports.
• Switches are expandable up to 436 ports can mix
  switch types
• GBICs plug-in converters for 0.5 / 10 / 70 km
  fiber optic
• FTE software and dual Network Interfaces per PC
  node

Backbone Switches
Media Converters
GBICs
Cluster Switches
Software
Dual NIC card
FTE
FTE
Ethernet
Ethernet
FTE
FTE
25
Main Configuration Rules

• Switches 24/48 ports, expandable up to 432 ports
• FTE network
• up to 200 FTE nodes (dual-connected)
• Up to 99 of those can be C200 controllers
• up to 511 Ethernet nodes (singly-connected)
• Firewall/router required to connect to other
  networks
• FTE network is a separate IP subnet
• Private IP addresses only servers are visible
  externally
• Cable shielded twisted pair (STP) or fiber optic
  recommended for best noise immunity and
  performance
• Required for CE Mark

26
Fault Tolerant Ethernet (FTE)-----

• FTE is the control network of Experion PKS.
• Analogous to TPS LCN/UCN and PlantScape
  ControlNet.
• Dedicated to the control mission
• Fault-tolerant
• Fast response
• Deterministic
• Secure
• Not an IT network, but leverages IT technology to
  lower cost of
• FTE network infrastructure
• Connection to IT networks
• Connection to 3rd party Ethernet devices
• Maintenance and support

FTE
.