CS 838: NetFPGA Tutorial

1
CS 838 NetFPGA Tutorial

• Theophilus Benson

2
Outline

• Background What is the NetFPGA?
• Life cycle of a packet through a NetFPGA
• Demo

3
What is the NetFPGA?
NetworkingSoftware running on a standard PC
CPU
Memory
PCI
A hardware accelerator built with Field
Programmable Gate Arraydriving Gigabit
network links
4
NetFPGA Router

• Function
• 4 Gigabit Ethernet ports
• Fully programmable
• FPGA hardware

• Open-source FPGA hardware --
• Verilog base design
• Open-source Software -- Linux user Level
• Drivers in C and C

5
NetFPGA Platform

• Major Components
• Interfaces
• 4 Gigabit Ethernet Ports
• PCI Host Interface
• Memories
• 36Mbits Static RAM
• 512Mbits DDR2 Dynamic RAM
• FPGA Resources
• Block RAMs
• Configurable Logic Block (CLBs)
• Memory Mapped Registers

6
NetFGPA Router Design

• Pipeline of modules
• FIFO queues between each module
• Inter module communication
• CTRL Send on ctrl bus (8 bits)
• Metadata about the data being send
• DATA Send on data bus (64 bits)
• RDY Signifies ready to receive packet (1 bit)
• WR Signifies packet being send(1bit)

7
NetFPGA
Linux user-level processes
Linux Processes
Verilog on NetFPGA PCI board
FGPA Modules 1
FGPA Modules 2
8
Example An IP Router on NetFPGA
Management CLI
Linux user-level processes
Routing Protocols
Exception Processing
Routing Table
Verilog on NetFPGA PCI board
Switching
Forwarding Table
9
Life of a Packet through the hardware
192.168.102.y
192.168.101.x
IP packet
10
Router Stages
11
Inter-module Communication

• Using Module Headers

Data Word (64 bits)
Ctrl Word (8 bits)
Module Hdr
x
Contain information such as packet length, input
port, output port,


Last Module Hdr
y
Eth Hdr
0
IP Hdr
0

0
Last word of packet
0x10
12
Inter-module Communication
Module i1
Module i
data
ctrl
wr
rdy
13
MAC Rx Queue
MAC Rx Queue
14
Rx Queue
Rx Queue
15
Input Arbiter
Rx Q 7
Input Arbiter
Pkt

Rx Q 1
Pkt
Rx Q 0
Pkt
16
Output Port Lookup
Output Port Lookup
17
Output Port Lookup
5- Add output port module
1- Check input port matches Dst MAC
Output Port Lookup
output port 4
0x04
6- Modify MAC Dst and Src addresses
2- Check TTL, checksum
Pkt length, input port 0
0xff
EthHdr Dst MAC 0 Src MAC x, Ethertype IP
0
EthHdr Dst MAC nextHop Src MAC port 4,
Ethertype IP
3- Lookup next hop IP output port (LPM)
IP Hdr IP Dst 192.168.2.3, TTL 64, Csum0x3ab4
0
7-Decrement TTL and update checksum
IP Hdr IP Dst 192.168.2.3, TTL 63, Csum0x3ac2
4- Lookup next hop MAC address (ARP)
Data
0
18
Output Queues
Output Queues
OQ0
OQ4
Pkt
OQ7
19
MAC Tx Queue
MAC Tx Queue
20
MAC Tx Queue
MAC Tx Queue
output port 4
0x04
Pkt length, input port 0
0xff
EthHdr Dst MAC nextHop Src MAC port 4,
Ethertype IP
0
0
IP Hdr IP Dst 192.168.2.3, TTL 64, Csum0x3ab4
IP Hdr IP Dst 192.168.2.3, TTL 63, Csum0x3ac2
Data
0
21
NetFPGA-Host Interaction

• Linux driver interfaces with hardware
• Packet interface via standard Linux network stack
• Register reads/writes via ioctl system call (with
  convenience wrapper functions)
• readReg(nf2device dev, int address, unsigned
  rd_data)
• writeReg(nf2device dev, int address, unsigned
  wr_data)
• eg
• readReg(nf2, OQ_NUM_PKTS_STORED_0, val)

22
NetFPGA-Host Interaction
Register access
PCI Bus
1. Software makes ioctl call on network socket.
ioctl passed to driver.
23
NetFPGA-Host Interaction

• Packet transfers shown using DMA interface
• Alternative use programmed IO to transfer
  packets via register reads/writes
• slower but eliminates the need to deal with
  network sockets

24
DEMO Life of a Packet through the hardware
192.168.2.y
192.168.1.x
IP packet
25

• Programming the FPGA with your code
• nf2_download NF2/bitfiles/reference_router.bit
• Mirror linux arp
• ./NF2/projects/router_kit/sw/rkd
• Helpful tool
• ./NFlib/C/router/cli
• Shows forwarding tables arp table, ip table
• Allows to modify tables

26
Useful Links

• NetFPGA Website
• NetFPGA Wiki
• NetFPGA Guide
• Walkthrough the Reference Designs
• The Verilog Golden Reference Guide

27
Questions
28
Verilog
29
Hardware Description Languages

• Concurrent
• By Default, Verilog statements evaluated
  concurrently
• Express fine grain parallelism
• Allows gate-level parallelism
• Provides Precise Description
• Eliminates ambiguity about operation
• Synthesizable
• Generates hardware from description

30
Verilog Data Types

• reg 70 A // 8-bit register, MSB to LSB
  // (Preferred bit order for
  NetFPGA)
• reg 015 B // 16-bit register, LSB to MSB
• B A70,A07 // Assignment of bits
• reg 310 Mem 01023 // 1K Word Memory
• integer Count // simple signed 32-bit
  integer
• integer K164 // an array of 64 integers
• time Start, Stop // Two 64-bit time variables
  

From CSCI 320 Computer Architecture Handbook on
Verilog HDL, by Dr. Daniel C. Hyde
http//eesun.free.fr/DOC/VERILOG/verilog-manual.
html
31
Signal Multiplexers
Two input multiplexer (using if / else) reg y
always _at_    if (select)       y a    else
      y b
Two input multiplexer (using ternary operator ?)
wire t (select ? a b)

• From http//eesun.free.fr/DOC/VERILOG/synvlg.html

32
Larger Multiplexers
Three input multiplexer reg s always _at_   
begin    case (select2)       2'b00 s a
      2'b01 s b       default s c    
endcase    end
33
Synchronous Storage Elements

• Values change at times governed by clock

• Clock
• Input to circuit
• Clock Event
• Example Rising edge

Din
t0

• Flip/Flop
• Transfers Value From Din to Dout on Clock event

Clock Transition
Dout
A
B
S0
t0
34
Finite State Machines
35
Synthesizable Verilog Delay Flip/Flops

• D-type flip flop
• reg q
• always _at_ (posedge clk)   q lt d

• D type flip flop with data enable
• reg q
• always _at_ (posedge clk)   if (enable)     q lt
  d

• From http//eesun.free.fr/DOC/VERILOG/synvlg.html
  

36
More on NetFPGA System
37
NetFPGA System
User Space Linux Kernel
Browser Video Client
Monitor Software
CAD Tools
Web VideoServer
Packet Forwarding Table
PCI-e
PCI

VI
VI
VI
VI
NIC
NetFPGA RouterHardware
GE
GE
GE
GE
GE
GE
(eth1 .. 2)
(nf2c0 .. 3)
38
NetFPGA System Implementation

• NetFPGA Blocks
• Virtex-2 Pro FPGA
• 4.5MB ZBT SRAM
• 64MB DDR2 DRAM
• PCI Host Interface
• 4 Gigabit Ethernet ports
• Intranet Test Ports
• Dual or Quad Gigabit Etherents on PCI-e
• Internet
• Gigabit Ethernet on Motherboard
• Processor
• Dual-Core CPU
• Operating System
• Linux CentOS 4.4

39
NetFPGA Lab Setup
CPU x2
Dual NIC
Client
Eth2 Server
GE
PCI-e
(eth1 .. 2)
Eth1 Local host
GE
Server
Net-FPGA
Nf2c3 Adj. Server
GE
PCI
NetFPGA Control SW
Nf2c2 Local Host
Internet Router Hardware
GE
Nf2c1 Adjacent
GE
Nf2c0 Adjacent
GE
CAD Tools
40
Exception Path
41
Exception Packet

• Example TTL 0 or TTL 1
• Packet has to be sent to the CPU which will
  generate an ICMP packet as a response
• Difference starts at the Output Port lookup stage

42
Exception Packet Path
Ethernet
43
Output Port Lookup
1- Check input port matches Dst MAC
Output Port Lookup
output port 1
0x04
2- Check TTL, checksum EXCEPTION!
Pkt length, input port 0
0xff
EthHdr Dst MAC 0, Src MAC x, Ethertype IP
0
IP Hdr IP Dst 192.168.2.3, TTL 1, Csum0x3ab4
0
3- Add output port module
Data
0
44
Output Queues
Output Queues
OQ0
OQ1
OQ2
Pkt
OQ7
45
CPU Tx Queue
CPU Tx Queue
46
CPU Tx Queue
CPU Tx Queue
output port 1
0x04
Pkt length, input port 0
0xff
EthHdr Dst MAC 0, Src MAC x, Ethertype
IP
0
0
IP Hdr IP Dst 192.168.2.3, TTL 1, Csum0x3ab4
Data
0
47
ICMP Packet

• For the ICMP packet, the packet arrives at the
  CPU Rx Queue from the PCI Bus
• Follows the same path as a packet from the MAC
  until the Output Port Lookup.
• The OPL module seeing the packet is from the CPU
  Rx Queue 1, sets the output port directly to 0.
• The packet then continues on the same path as the
  non-exception packet to the Output Queues and
  then MAC Tx queue 0.

48
ICMP Packet Path
Ethernet
49
NetFPGA-Host Interaction
NetFPGA to host packet transfer
PCI Bus
50
NetFPGA-Host Interaction
NetFPGA to host packet transfer (cont)
PCI Bus
6. Driver passes packet to network stack
51
NetFPGA-Host Interaction
Host to NetFPGA packet transfers
PCI Bus
1. Software sends packet via network sockets.
Packet delivered to driver.