SPC PortLevel Processing: the MSR Kernel

1
(SPC) Port-Level Processingthe MSR Kernel

• Fred Kuhns
• Washington University
• Applied Research Laboratory

2
Overview

• Introduction to hardware environment
• APIC core processing and buffer management
• Overview of SPC kernel software architecture and
  processing steps
• Plugin environment and filters
• Command Facility

3
Port Processors SPC and/or FPX
ControlProcessor
Switch Fabric
ATM Switch Core
IPP
OPP
IPP
OPP
IPP
OPP
IPP
OPP
IPP
OPP
IPP
OPP
FPX
FPX
FPX
FPX
FPX
FPX
Port Processors
SPC
SPC
SPC
SPC
SPC
SPC
LC
LC
LC
LC
LC
LC
Line Cards (link interfaces)
4
Using Both and FPX and SPC
Shim contains results of classification step
SPC
FPX
DQ Module
Z.2
Active processing
IP Classifier
X.1
shim
NID
APIC
Flow Control
5
Focus on SPC as Port Processor
ControlProcessor
Switch Fabric
SPC
SPC
OutputPortProc.
InputPortProc.
. . .
6
The SPC an Embedded Processor
Switch Interface
DRAM
Link Interface
CPU Module
APIC
PCI Bus
System FPGA
Serial Ports
7
Typical Pentium PC Architecture
Addr/Data
Ctrl
Ctrl
CPU
North- Bridge
Cache
DRAM
Addr/Data/Ctrl
PCI Bus
Intr
NMI
INIT
SouthBridge (PIIX3) (PIC, PIT, )
PCI Devices
ISA Bus
ISA Devices
BIOS
Super-IO
BIOS
RTC Uarts Kbd/Mse Floppy Parallel ...
8
SPC Hardware Architecture
Addr/Data
Ctrl
Ctrl
CPU
North- Bridge
Cache
DRAM
Addr/Data/Ctrl
Intel Embedded Module
PCI Bus
NMI
INIT
Intr
APIC
PIT
PIC
RTC
UART1 Interface
BIOS ROM
Link Interface
UART2 Interface
System FPGA
Switch Interface
9
SPC Components

• APIC - PCI Bus Master
• Pentium Embedded Module
• 166 MHz MMX Pentium Processor
• L1 Cache 16KB Data, 16KB Code
• L2 cache 512 KB
• NorthBridge
• 33 MHz, 32 bit PCI Bus
• PCI Bus Master
• System FPGA - PCI Bus Slave
• Xilinx XC4020XL-1 FPGA
• 20K Equivalent Gates, 75 used

10
SPC Components (continued)

• Memory
• EDO DRAM
• 64MB (Max for current design)
• SO DIMM
• Switch Interface - 1 Gb Utopia
• Link Interface - 1 Gb Utopia
• UART
• Two Serial Ports
• NetBSD system console
• TTY port

11
Overview

• Introduction to hardware environment
• APIC core processing and buffer management
• Overview of SPC kernel software architecture and
  processing steps
• Plugin environment and filters
• Command Facility

12
APIC Descriptors

• APIC uses a data structure called a descriptor to
  describe available buffers and their status.
• The hardware and software follow a well defined
  protocol for jointly managing the descriptors.
• The APIC controls one or more Free Descriptor
  chains, with each chain representing buffers
  available for Rx for a predefined set (one or
  more) of RX channels.

13
APIC Descriptor Management

• Some details are left out for brevity
• Software
• allocates descriptors to free pools for the APIC
  HW
• Reads receive descriptor chains for each VC, free
  descriptor after it is read, then returns it to a
  free pool.
• Adds descriptors to send queues for each VC, then
  recycles them after APIC sends corresponding
  frame/cell.
• Hardware
• allocates descriptors from free pools for
  frames/cells received on a corresponding VC. Then
  places them on a receive chain.
• Reads descriptors from send queues for each VC
  and notifies software when packet is sent.

14
APIC Descriptors and Buffers
Flags O - Read Only, E - EOF, C - CRC OK, T -
Type, Y - Valid Bits
Buffer Length or Amount Left Unused
Desc. Table Index
Physical Address of Data Buffer

• Frame must be multiple of 48 B.
• Buffers are 2048 B.
• Max size 2016 B, or 42 cells.
• Reserve 8 B for shim, 8 B for trailer
• IP Datagram MTU must be 2000 B
• At output port, max 2016 B frame received, offset
  8 bytes in buffer.
• At most the 2024 B of buffer are used.
• 24 B at end of buffer not used.

15
Descriptor Notes
0
8
16
24
31
Match/Checksum
- - - -
V
I
S
O
E
C
L
X
T
Y
MatchFlags
BufLen
NextDesc
SizeNext
Index into Desc Table
BufAddrLo
Physical Address of Data Buffer
Low32Addr
BufAddrHi
V Volatile Buffer I Interrupt/Notify on
Read S SAM Enable O Read Only E End of
Frame C CRC OK, RX L Loss Priority (CLP of
last cell), RX X Congestion indication from
last cell's PTI, RX T BufType, 0 - Data 1 -
RM 2 - segment OAM 3 end-2-end OAM Y Sync
0 - Done, Valid Link 1 - Done, InValid
Link 2 - Not Ready 3 - Ready
Possible values for First Word CAFE0083 Tx,
EoF, Ready (Driver) CAFE0080 Tx, EoF,
DoneValidLink (APIC) CAFE0002 Tx,
NotReady CAFE0003 Rx, Ready, No Interrupt on
Read CAFE0403 Rx, Ready, Interrupt on
Read xxxx00C0 Rx, EoF, CRC OK,
DoneValidLink xxxx0040 Rx, CRC OK,
DoneValidLink xxxx00C1 Rx, EoF, CRC OK,
DoneInValidLink xxxx0041 Rx, CRC OK,
DoneInValidLink
16
APIC Descriptors
Free Descriptor chain used by APIC during
receive,each descriptor contains the physical
address of an available buffer.
Pool X Chain Head
CAFE003 or if end of chain CAFE002
CAFE003
buffer
buffer
17
Descriptors on a Receive Queue
VC 101 Queue
DoneValidLink
DoneInvalidLink
1000 Byte frame
2016 Byte frame
18
RX Descriptor to Buffer Mapping
Buffers (replace Mbufs)
First RX descriptor j
0
Buffer
Descriptors
2KB
j
j1
1
Buffer
j2
j3
jN
N
Buffer
19
Descriptor Layout
Invalid Descriptor
Starting address desc_area
Index
local_start
aal5_pool
local_count
RX/TX Shared IP Packet Buffers
local_end
aal5rx_start
aal5_count
aal5rx_end
aal5tx_start
aal5_count
msr_descr_count
aal0_pool
aal5rx_end
RX - Cell Buffers
aal0rx_start
RX channel 0, aal0_count_vci
aal0_count
aal0rx_end
aal0_count
RX channel 1, aal0_count_vci
TX - Cell Buffers
aal0tx_start
TX channel 0, aal0_count_vci
aal0_count
aal0tx_end
TX channel 1, aal0_count_vci
unallocated
20
Descriptor Buffer Relationships
APIC
Descriptor Table (DT)
Rx desc bound (same offset) to specific buffer
Descriptors
Rx
current rx offset
registers
Global
notification
channel
Tx
TX desc allocated dynamically and bound to the
RX desc and buffer
Tx Offset
MSR Buffers (MB)
Buffers
same as rx offset
port 0
port 2
port 1
21
Receiving a Packet
APIC
Descriptor Table (DT)
DT base
Rx
indx
registers
Tx
1) AAL5 frame is received APIC allocates and
reads desc from RX pool. Then the previous Rx
desc is written back (updated).
Driver and IP code
MSR Buffers (MB)
MB base
indx
port 0
port 2
port 1
22
Completing the Receive
APIC
Descriptor Table (DT)
1) APIC writes (updates) current desc.
DT base
Rx
indx
registers
2) APIC updates notification register
Tx
Driver and IP code
3) Last Assert Interrupt
port 0
APIC disables interrupts on Rx channel
port 2
port 1
23
Sending Packet
1) allocate Tx desc and bind to Rx desc and buffer
APIC
2) a) write to current descs next index b) Write
to resume Tx channel register
Descriptor Table (DT)
IP Lookup Table
DT base
Rx
indx
registers
Tx
Driver and IP code
port 0
port 2
port 1
24
Enhancements to Buffer Management

• Currently enhancing plugin interface to support
  both sinking and sourcing of packets.
• This is combined with the flow table for reserved
  and bound flows.
• We also support 64 datagram queues for packet
  scheduling.
• Changes to buffer management are shown in the
  next few slides

25
Relationships between Buffers
Pkt Buffers
index by rx,txindx
desc_bindings
APIC Descriptors
RX Start
shim
a, refcnt 2
IP Packet
x, rxindx a
rxdesc (a)
rxindx
AAL5 trailer
y, rxindx a
TX Start
Header and buffer offset are same as rxindx.
shim
z, rxindx 0
IP Packet
txdesc (x)
txdesc (y)
AAL5 trailer
Buffer Headers
txdesc (z)
txindx
desc_bindings index equal to descriptor index
Header has start of frame pointer
26
Buffer Headers and FTEs
Buffer Header
Buffer
pkt // pointer to start of frame fid // flow
table entry reference qlist // linked list for PS
queue gid // nonexclusive gm filter qid // queue
id used by PS fwdkey // (sid, outVIN), route plen
// AAL5 frame length atmlen // plen cell
headers rxcid // VC packet received on txcid //
VC to send frame on flags // CPY, Active, FPX,
Ingress/Egress, IPO,
Shim, RATM, Cntl, Drop
shim
Flow Table Entry (FTE)
IP Packet
alist // list of active FTEs fwdkey // pinned
route qid // Unique flow id for PS firm_req
// requested rate firm_act // allocated rate
soft_req // requested rate soft_act //
allocated rate weight // PS weight res // LFS
reservation flags // Deny, Active,
Reserv, LFS, Reclaim plugin //
pointer to plugin filter // exact match 5
tuple refcnt // pkts in system pktcnt // total
pkts matching
plen
AAL5 Padding
AAL5 trailer
Packet Scheduler Queues
queue index
head - list of buff headers
27
Classifier Table Relationships
General Match Filters
Route Table (RT priority)
Hash Table
filter // with masks prio // priority of
filter fte // if exclusive ilist // if
nonexclusive flags // Exclusive, ...
...
default route
filter // with masks prio // priority of
filter fte // if exclusive ilist // if
nonexclusive flags // Exclusive, ...
...
Exclusive Filters reference a flow table entry.
lookup destination
Hash IP Header
Flow Table (FT priority)
... (10 filters)
hlist // hash list alist // active list filter //
with masks fte //
list of filters with same hash value
filter // with masks prio // priority of
filter fte // if exclusive ilist // if
nonexclusive flags // Exclusive, ...
hlist // hash list alist // active list filter //
with masks fte //
NULL
Nonexclusive filters are used for monitoring and
do not reference a fte nor can they alter
application packets.
28
Overview

• Introduction to hardware environment
• APIC core processing and buffer management
• Overview of SPC kernel software architecture and
  processing steps
• Plugin environment and filters
• Command Facility

29
SPC Software Architecture
debug messages
CP command processor and debug message
command reply
DRR Service
commands
Paced APIC TX queues
interrupt
Plugin Environment
Egress
Sub Port 0
Plugin
plugin
plugin
Commands
Sub Port 1
...
Sub Port 2
SP1
handler() send budget per interval
Sub Port 3
SP2
APIC
APIC
...
APIC Specific Driver Code
Ingress/ Egress ?
APIC TX Qs DQ Adjusts VOQ Pacing
IP processing
insert/process shim
periodic callback interrupt (D ?sec)
SPN
Ingress
Route Lookup (FIPL, Simple)
DQ Reports
DQ Service
handler() read cells, set pacing, broadcast
report
Read DQ Report Cells
Broadcast Report
30
SPC Data Path - Simplified View
Plugin Environment
Plugin
plugin
plugin
DQ/ In Queuing
...
Flow Classifier/ (channel map)
NM Filter
...
Ingress/ Egress ?
Frame/Buffer and IP Processing
Route Lookup (Shim, FIPL, Simple, cache)
DRR/ Out Queuing
...
31
SPC Input (Ingress) Processing
Plugin Environment
Manage
IP Options
X.1
Z.1
W.1
X.2
Y.1
Local Resource Manager and PCU Interface
Z.2
APIC TX Qs DQ Adjusts VOQ Pacing
PCU Framework
interrupt
Replace IntraShim with InterShim. Update trailer
and IP header
Flow Classifier/ (channel map)
NM Filter
IP Processing
Insert InterPort Shim
APIC Specific Driver Code
APIC
APIC
Route Lookup (FIPL, Simple)
Broadcast Report
Distributed Queuing callback read cells, set
pacing, broadcast report
DQ Reports
Read DQ Report Cells
periodic callback interrupt (D ?sec)
32
Input (Ingress) Processing
Plugin Environment
Manage
IP Options
X.1
Z.1
W.1
X.2
Y.1
Local Resource Manager and PCU Interface
Z.2
APIC TX Qs DQ Adjusts VOQ Pacing
PCU Framework
interrupt
Replace IntraShim with InterShim. Update trailer
and IP header
Flow Classifier/ (channel map)
NM Filter
IP Processing
Insert InterPort Shim
APIC Specific Driver Code
APIC
APIC
Route Lookup (FIPL, Simple)
Broadcast Report
Distributed Queuing callback read cells, set
pacing, broadcast report
DQ Reports
Read DQ Report Cells
periodic callback interrupt (D ?sec)
33
Output Port (Egress) Processing
interrupt
Paced APIC TX queues
DRR Service
Sub Port 0
Sub Port 1
Determine Out VC Remove Shim update AAL5 trailer
and IP header
APIC
...
Flow Classifier/ (channel map)
NM Filter
IP processing
process shim
APIC Specific Driver Code
Classifier
Sub Port 2
handler send budget per flow
Sub Port 3
DQ report Tx queue lengths
periodic callback interrupt (D ?sec)
34
Output Port (Egress) Processing
General and Exact match classifier same as
ingress, except route is obtained from output VIN
in Shim
interrupt
Paced APIC TX queues
DRR Service
Sub Port 0
Sub Port 1
Determine Out VC Remove Shim update AAL5 trailer
and IP header
APIC
...
Flow Classifier/ (channel map)
NM Filter
IP processing
process shim
APIC Specific Driver Code
Classifier
Sub Port 2
handler send budget per flow
Sub Port 3
DQ report Tx queue lengths
periodic callback interrupt (D ?sec)
35
What about Ethernet?
For more detail see the GigE talk on Tuesday
Router
MSR
Ethernet Switch
Router
Router
36
GgE Link Interfaces
Map multicast or broadcast to ethernet address
If ARP table lookup fails, send ARP request to
broadcast address, drop packet. No retries are
made.
Send to pkt-dst if bcast or mcast map to
eaddr else resolve w/ARP
ARP Table (M Entries)
MAC
IP
No ARP entry aging!
IP1
MAC1
...
...
IP Header
Ethernet
IPM
MACM
data
IP Header
From FPX/SPC
AAL5 trailer
data
To Next Hop or Endstation
Add Ethernet header using the derived destination
address and out source address. Protocol is IP.
if VC ! 50, Lookup VC in VIN table returns IP
used for ARP lookup (support N 4)
Software creates VIN table at boot time by
writing to interface.
37
GigE Link Interface - Ingress
ARP Table (M Entries)
receive ethernet frame eth if (eth-type
ARP) if (eth-arp-has ! Ethernet/0001) Drop
Frame if (eth-arp-pas ! IP/0800) Drop
Frame update eth-arp-spa, eth-arp-sha in
ARP table if (eth-arp-tpa NOT in MyIP0,
MyIP1, MyIP2) Drop Frame // target IP not
ours if (eth-arp-op Request/01) swap
source and target ARP info set operation to
Reply set ether header src and dst
address send reply // Already handled
eth-arp-op Reply/02 // when updated cache
above else if (eth-type IPv4) remove
ethernet header, padding and CRC add AAL5
trailer and required padding break into cells
and send on default Base VC else Error, drop
packet Unicast MAC address filtering
MAC
IP
IP1
MAC1
...
...
IPM
MACM
Ethernet
IP Header
IP Header
From Next Hop or Endstation
data
To FPX/SPC
data
AAL5 trailer
38
Overview

• Introduction to hardware environment
• APIC core processing and buffer management
• Overview of SPC kernel software architecture and
  processing steps
• Plugin environment and filters
• Command Facility

39
Packet Classification Plugins

• Classification provides and opportunity to bind
  flows to registered plugin instances.
• General classifier - Network Management
• classification using 5-tuple
• ,
• Prefix match on address, exact match port and
  proto
• 0 is a wildcard for all fields
• input and output ports
• filters added/removed via the command facility

40
Flow Bound to a Plugin
Plugin Environment
Plugin
plugin
plugin
DQ/ In Queuing
...
Exact Match active processing same as general
match. The AAL5 length is and IP header checksum
are calculated so plugin does not have to perform
these operations.
Flow Classifier/ (channel map)
NM Filter
...
Ingress/ Egress ?
Frame/Buffer and IP Processing
Route Lookup (Shim, FIPL, Simple, cache)
DRR/ Out Queuing
...
41
General Match Classifier Notes

• General Match Classifier Linear search of
• src_addr, dst_addr, src_port, dst_port, proto
• General Classifier options First, Last, All
• Rule Actions Deny, Permit, Active.
• Rule flags All, Copy, Stop

Search
42
Exact Match Classifier Notes

• Exact Match Classifier Hash followed by linear
  search - src_addr, dst_addr, src_port, dst_port,
  proto.
• Exact Match Classifier options None.
• Rule Actions Deny, Permit, Active, Reserve.
• Rule flags Pinned, Idle, Remove

43
Active Processing Environment
Class A plugin x
Class B plugin y
Class C plugin z
Instance 1 Active
Instance 1 Deny
Instance 1 Active
Instance 2 Active, All
General/Exact Match Classifier
Rule N
Rule P

• Plugin instance maps to at most one rule/filter.
• General classifier rule maps to at most 5
  instances.
• Exact match classifier rule maps to at most 1
  instance.

44
Creating an Instance
Class A classid 100 inst_t create_instance(clas
s_t , inst_id)
Return reference to instance
create class instance
Instance of Class A - (Base Class extended by
Developer) Class class_t class inst_t next inst_id id fid_
t bound_fid void (handle_packet) (inst_t ,
ip_t , flag32_t) void (bind_instance)
(inst_t ) void (unbind_instance) (inst_t
) void (free_instance) (inst_t ) int
(handle_msg) (inst_t , buf_t , flag8_t,
seq_t, len_t ) ...
45
Plugin Class Specific Interface

• All plugins belong to a class. At run time a
  class (i.e. plugin) must be instantiated before
  it vcan be referenced.
• Plugin is passed its instance pointer (like c)
  as the first argument.
• Developer may extend the base class (struct
  rp_instance) to include additional fields which
  are local to each instance.
• Plugin developer must implement the following
  methods
• void(handle_packet)(struct rp_instance , struct
  ip , u_int32_t)
• void(bind_instance)(struct rp_instance )
• void(unbind_instance)(struct rp_instance )
• void(free_instance)(struct rp_instance )
• int (handle_msg)(struct rp_instance , void ,
  u_int8_t, u_int8_t, u_int8_t)

46
Plugin Framework Enhancements

• Integrated with Command framework
• send command cells to PCU
• create instance, free instance, bind instance to
  filter, unbind instance
• Send command cells to particular plugin instances
• Send command cells to plugin base class
• Enhanced interface to address limitation noticed
  in crossbow
• instance access to plugin class, instance id,
  filter id
• pcu reports describing any loaded classes,
  instances and filters

47
Overview

• Introduction to hardware environment
• APIC core processing and buffer management
• Overview of SPC kernel software architecture and
  processing steps
• Plugin environment and filters
• Command Facility

48
Command Facility Highlights

• Overview
• High level description - Application Layer
• MSR Command Interface Overview
• Cell format and field definitions
• Example

49
Definitions

• Session Open connection between the CP and a
  specific SPC. Intended to represent open
  connections and command state
• Transaction Represent a complete command. A
  transaction terminates with either an EOF is
  received by the CP or and error occurs.
• EOF End of File is returned to CP with the last
  bit of command data is returned or in response to
  a Cancel message (or an error occurs)

50
Overview - Cmd Interface on CP

• Synchronous Request/Response protocol
• Timeout can be specified as well as the number of
  retries - Per session option
• Essentially provides a reliable service
• Issue if no reply, cmd/reply msg lost in port,
  channel or CP. Retries may be a bad thing.
• Address - MSR Port and Command
• Message destination - Callback function within
  the Ports kernel (implements command)

51
Command Interface on CP

• Types of messages
• New Command, Get Next set of reply data Command,
  Cancel Command
• Error Reply, EOF Reply, Continued Reply
• Message Identifiers - Only requires a sequence
  number initialized to 0 for each New Command
• One sending entity on CP,
• One outstanding command for each port,
• Ports send exactly one reply msg per command msg,
• Command must fit within one cell,
• Replies may span multiple cells.

52
Command Interface on Port

• Callback function registered with MSR kernel and
  called under 3 cases
• New Command
• Flags Command Sequence 0 Length valid
  bytes in buffer Buffer application data
• Next Command
• Flags Command Next Sequence previous1
  Length valid bytes in buffer Buffer
  application data
• Cancel Command
• Flags Command Cancel Sequence previous1
  Length 0 Buffer contains no valid data

53
Command Interface on Port

• Callback function must
• Read from/Write to supplied buffer
• Set length Bytes written to buffer (in/out
  param)
• Indicate if an error occurred (return -1)
• Whether more data exists (return 0 EOF, return
  0 Not EOF, return ERROR EOF)
• Framework
• generates reply message using same Command value
  and Sequence number.
• sets flags indicating status (EOF, Error etc)

54
Failure Modes

• Library support for lost messages
• if (timeout 0, Replies 0), then CP API
  library will re-send with RETRY flag set.
• if (timeout 0, Replies 0 or all replies
  failed), then API library returns error to
  application
• If (timeout 0 - No Timeout), then send
  operation blocks indefinitely.
• Lost Command message -
• if (timeout 0 and retries 0), CP resends
  command same sequence number but RETRY flag set.
  Command buffer and flags passed to callback fn.

55
Failure Modes

• Lost Reply message,
• if no retries, Any issues?
• if retries then CP resends
• New Command - Port knows this is a duplicate
  command (RETRY flag). Application responsible for
  handling retries. If an issue can use unique
  message ids. Extreme case use a history (last
  reply message).
• Next Command - Port receives Command w/Sequence
  0, w/RETRY flag. Passed to application which
  chooses the correct course of action. The intent
  is to ensure there are no holes in the reply data
  received by the CP.
• Cancel message - same as Next command.

56
Possible Enhancements

• Support asynchronous messaging
• Multiple outstanding commands per port
• Asynchronous I/O on CP
• Speed up boot process and dynamic configuration
• Facilitates implementing port monitoring (ping or
  heartbeat) for fault detection and recovery.
• two methods for reporting results
• upcall - function registered by application is
  called when results arrive
• poll - application periodically polls library for
  results.
• Support Broadcast and/or Multicast

57
MSR Command Layer

• Simple messaging facility optimized for MSR.
• Command message (CP sends)
• Sent by CP to a specific MSR port (unicast)
• Must fit within one AAL0 cell.
• Message header, includes
• protocol version
• Command
• Sequence number
• flags
• Application data follows header
• Library implements Request/Reply protocol.

58
MSR Command Layer

• Reply Message (Port sends)
• Port must send reply message in response to a
  Command message.
• Reply message Header
• version and sequence number same as command msg.
• Includes application data and flags indicating if
  command was successful and if more data exists
  (EOF).
• Application registers command specific callback
  function at port.
• Callback function must conform to specified
  interface.

59
MSR Command Overvew

• Command Protocol description
• Control Processor sends command messages to a
  specific port and expects to receive a reply
  message indicating either Success or Failure.
  This is termed a Command Cycle.
• There is the notion of a Command Transaction
  which may include one or more command cycles. A
  command transaction is terminated when the target
  (port) responds with a reply msg containing an EOF

60
MSR Command Overview

• Command Protocol description, continued
• CP processing of Reply msg, depends on EOF flag
• If EOF is set then no further reply data is
  available and the command transaction is closed.
• If EOF is not set then there is remaining data
  and the command transaction is still open.
• If remaining data (Not EOF), then CP must follow
  with a either a Next or Cancel command message.
• Sequence number indicates the chunk of data to
  be returned.
• Command indicates the messages destination
• sequence number previous 1

61
APIC Cell Format
8
16
24
0
cid
ld
cc
pout
x
xx
pin
x x x x x x x x x x x x x
cl
gfc
vpi
vci
pti
ver
length
command/status
sequence number
flags
MSR Command Message
Cell payload contains the MSR Command Command
header is 4 Bytes, leaving 44 Bytes for
sub-commands and data.
62
ATM/APIC Header

• pin (Ports-In) - Port cell arrived
• Tx not used (set to 000b)
• Rx port cell arrived on (is the below correct?)
• 001 Port 0, 010 Port 1, 100 Port 2, etc.
• pout (Ports-Out) - Set of output ports.
• Tx Command library sets
• 001 Fiber/Link, 010 Ribbon/Switch, 011 Both
• 101 Loopback MV0, 110 Loopback MV1
• Rx Set by VCXT, see pin above.

63
ATM/APIC Cell Format

• cc (Control Cell Indicator) - Not used, set to
  0b
• ld (Low Delay) - Not used, set to 0b.
• Should we use low delay?
• cid (Connection Identifier) - set to vci value.
• gfc (Generic Flow Control) - set to 0000b.

64
ATM/APIC Cell Format

• vpi (Virtual Path Identifier) - Set to 0x0.
• vci (Virtual Circuit Identifier) - Equal to cid.
• See presentation on MSR configurations for a
  complete list of VCI assignments.
• pti (Payload Type) - Set to 000b (data cell)
• cl (Cell Loss Priority) - Set to 0b (High
  Priority)

65
MSR Command Header
ver
length
command/status
sequence number
flags

• Version (2 bits) - Protocol version. Allows for
  at most 4 versions. Current version set to 0.
• field width was a trade off with the length
  field.
• Length (6 bits) - Number of valid data bytes.
• 0
• This field is indirectly set by the application
  or command implementation. The CP library and
  kernel interfaces allow for applications to pass
  a buffer pointer and indicate the number of valid
  data bytes.

66
MSR Command Header
ver
length
command/status
sequence number
flags

• Command/Status (8 Bits) CP inserts command value,
  SPC/port inserts status information.
• Valid Commands are listed in SYS/msr/msr_ctl.h,
  also see MSR/utils/command/.c,h
• Library API on CP accepts Command as argument.
  implementation in kernel - array of function
  pointers, uses Command as index
• Reply msg Status indicating success or an error
  code (Upcall, ATM, Cmd Invalid, Cmd Not
  Implemented, or Other Cmd Error).

67
MSR Command Header
ver
length
command/status
sequence number
flags

• Sequence Number (8 Bits) - Is of primary use by
  the applications.
• When command message first sent, sequence 0.
• If the reply does not include an EOF flag, then
  CP increments sequence by one for each subsequent
  command message.
• When EOF is received the Command Transaction is
  complete and the sequence number is reset to 0.

68
MSR Command Header
ver
length
command/status
sequence number
flags

• Flags (8 bits) - Bit field, valid flags are
• Invalid flag 0, should not occur
• CMD - cell contains a valid command from CP
• REPLY - cell contains reply from Port
• ERROR - Reply only, error processing on Port
• EOF - No reply data remains, end of cmd
  transaction
• NEXT - get next set of reply data
• CANCEL - cancel current cmd transaction
• RETRY - set if cp resend a command after it was
  lost

69
CP Library API

• Library API for application on CP,
• int sendcmd(int sid, int cmd, char data, int
  flags, int dlen)
• sid session id,
• cmd - Command to execute on port
• data buffer pointer,
• flags
• RETRY (reply timeout),
• CANCEL (cancel current command),
• Next (get next set of reply data)

70
MSR Kernel API

• MSR kernel interface SYS/msr/msr_ctl.h,c
• Callback function signature
• msr_ctl_ (void buf, u_int8_t flags,
  u_int8_t seq, u_int8_t dlen)
• buf command buffer w/application data,
• flags
• CMD,
• NEXT,
• RETRY or
• CANCEL,
• seq sequence number indicating reply data set,
  and
• dlen is input/output parameter, data length in
  bytes.

71
Kernel State Diagram
Next
Retry
Command
Command
Command
Cancel
EOF
Proto Error
Closed
Idle
Proto Error
Retry
72
CP Library State Diagram
73
Example Sending Cmd to Port
Lookup sub-command perform function call then
report results
wugs
192.168.200.X
192.168.204.X
P0
P4
SPC/FPX
SPC/FPX
Next/Prev Hop
Next/Prev Hop
msr_ctl
DQ
DQ
192.168.205.X
192.168.201.X
P1
P5
SPC/FPX
SPC/FPX
reply() plugin instance created Status, Instance
ID
Next/Prev Hop
Next/Prev Hop
DQ
DQ
sendcmd() create plugin instance port id
0, PluginID 200
192.168.202.X
192.168.206.X
P2
P6
Report command completion status to application.
SPC/FPX
SPC/FPX
Next/Prev Hop
Next/Prev Hop
DQ
DQ
192.168.202.2
192.168.203.X
192.168.207.X
P3
P7
SPC/FPX
SPC/FPX
Next/Prev Hop
CP
DQ
DQ
192.168.203.2