Onchip File Systems to Support Complex Embedded Systems

1
On-chip File Systems to Support Complex Embedded
Systems

• Geoffrey Brown

2
Reuse- and SW-centric SoC design is increasingly
partitioned into (a) the creation of programmable
platforms by a team of hardware-centric and
hardware-dependent software-centric designers,
and (b) use of the platform in derivative
products by a team of applications and
software-centric designers. International
technology roadmap for semiconductors 2001
edition
3
Problem Were Addressing

• With SoC based systems software development
  dominates time to market
• System development and SoC development often in
  different companies
• Software development made more difficult by
  opaque nature of SoC architecture (its hard to
  see internal interfaces)

4
(No Transcript)
5
Example SoC
6
Inspiration Plan 9 (slide from Dennis Ritchie
talk)

• Make all resources accessible by file system
  names/dev/console/dev/consctl/dev/mouse/proc
  (information about processes) mem stack
  status control .../net dns (domain
  name service) tcp clone (create
  connections) tcp05 data
  (send/receive data) control
  status tcp10 ....

7
Inspiration -- Plan 9/Inferno

• Distributed system built upon distributed file
  system
• Used in Pathstar Access Server (a Lucent
  telephone switch)
• All resources accessible as files
• Simple file system protocol (9P) with 20
  messages (essentially rpc)
• User controlled name spaces, with ability to
  mount remote files
• Servers handle file requests below them
• New servers can be created to support foreign
  protocols (e.g. ftp connections)
• /proc supports remote, heterogeneous debugging
  (ACID debugger)
• IOCTL dumped in favor of explicit, named control
  files

8
Model for Exploration
9
On-chip File System

• /DataMultiplexor/
• Classifier/
• Table/
• insert
• remove
• QueueManager/
• 0/
• length
• discards
• 1/

10
System Level File System

• /System/DataMux0/ Table/
• QueueManager/
• DataMux1/ DataDemux0/ DataDemux1/

11
On-chip File System Goals

• Access to chip level resources, devices, and
  applications through file operations
• Stackable hierarchy multiple file systems can
  be merged, system software can create new file
  systems. We can have system level, board level,
  chip level, block level file systems merged into
  a single hierarchy
• Portable protocol (should be easy to write
  libraries to import file system)

12
Applications

• Management
• Debugging and tracing
• Testing
• Performance measurement
• Resource proxies

13
Stackable Hierarchy
Export
Import
FileSystem
Board Level
System Level
Chip level
14
Export can be bidirectional

• -- host can proxy services for embedded system

Import FS
Import FS
Export FS
Export FS
15
9P and 9P2000 Protocols
Server (file system)
Client
Point-point Connection

• Communication through custom RPC
  protocol attach(rootfid, uid) returns
  qidestablishes connection

16
File Operations

• Conventional
• fd open(/a/b, OREAD)
• Becomes sequence of RPC commands
• clone(rootfid, newfid) walk(newfid,
  a) walk(newfid, b) open(newfid, OREAD)

17
RPCs, Messages

• There is a small set of RPCs,attach, clone,
  walk, open, read, write, create, remove, stat,
  wstat, clunk, flush. Directories read like
  normal files
• Simple message format

type1 tag1 fid2 contents lt 120
Contents depends upon request
fid selected by client
tag selected by client enables concurrent requests
Tclone,Rclone,Tattach,Rattach
18
Example Walk Messages
type1 tag1 fid2 name28
Twalk
qid8
Rwalk
qid picked by server, uniquely identifies file,
(path, version, type)
19
Remainder of Talk

• Applications for exporting file systems from
  embedded devices
• Debugging
• Tracing
• Applications for importing file systems to
  embedded devices
• System wide storage
• System wide network interface
• Components for Constructing Distributed File
  Servers
• What does it cost to do this ?

20
Embedded Software Development Support Issues

• Debugging - support for multiple heterogeneous
  devices
• Event Monitoring/Tracing Support for high speed
  hybrid monitoring
• System configuration/provisioning
• Unit testing inject/receive test traffic

21
Embedded Debugging
22
Issues With Traditional Interface

• JTAG piggybacks on test scanchain
• Low level bit protocol custom for each processor
  (goal of NEXUS standard effort)
• Doesnt support naturally support multiple
  on-chip blocks (goal of P1500 standard effort)
• Requires direct access to chip

23
Basic JTAG Interface
24
MSP430 Debug Unit

• Read/Write RAM/FLASH
• Read/Write Registers
• Read/Write Breakpoints
• CPU Control
• Halt
• Step
• Read/Write MAR
• Read/Write MDR

25
MSP430 JTAG Instructions
26
MSP430 Setting PC

• Debug Interface Software
• Proprietary
• Contains 10,000 code lines
• Many bug fixes in code

27
An Alternative Model

• /system/
• proc/
• processor1/
• control
• registers
• memory
• breakpoints
• processor2/
• DSP/

28
Experimental Platform
Stratix
GDB
Nios
MSP430
Proxy
Host FS
Client FS

• Host FS can simply pass through client, or
  extend/refine the interface of the client FS

29
Solution Range

• Minimal Remote Functionality (e.g. interface to
  raw JTAG)
• Minimal Local Functionality
• Split Functionality (e.g.) common operations
  remotely, rare operations locally

30
Minimum Remote Functionality

• /MSP430
• /Signals
• tck
• tdi
• tms
• DR
• IR

• /MSP430
• Command
• DR
• IR

Command String (e.g. tck(0)tck(1))
Too many File Ops
Both variations limited by roundtrip latency
(its possible to resolve this by storing Return
data stream remotely and accessing in batches).
31
Maximum Remote Functionality

• /MSP430
• control
• status
• registers
• memory

Command strings like set breakpoint, run,
stop
Read/Write access to processor memory
32
Solution Cost

• Basic Server 1500 lines C
• Simple Model
• File system 250 lines C
• Complex Model
• File system 500 lines C
• TI supplied debugger functionality 7000 lines C

33
Tracing/Monitoring

• Processor Tracing
• Event Monitoring
• Research Issues

34
ARM Processors Support Tracing at
Instruction/Data Bus Level

• Supports instruction and data traces
• Complex Triggering (when), Filtering (type)
• Address and data comparators, 16-bit counters,
  three-state sequencers
• Transmits trace packets

35
Multi-processor SoC Instruction Tracing Issues

• Each processor needs dedicated trace buffer
• Each processor needs high speed trace interface
  capability

36
VxWorks Embedded OS provides System Event Tracing
37
System Event Tracing

• Need constant time runtime penalty for real
  time systems this suggests always on policy
• Need filtering to extract events of interest
• Interesting applications
• Detecting real-time constraint violation
• Detecting mutual exclusion violations

38
Hardware Support for Multiprocessor
Instruction/Transaction Tracing

• Constant time hybrid transaction trace
• Programmable filters (bpf-like)
• Trace/DMA channels to support processor buffer
• Need support for high speed file transfer

39
Lightweight Proxy Services
Host
Embedded Processor
Internet
Application
Client Library
File Server
Host might be another embedded processor,
workstation, etc., so
long as it speaks file system protocol
40
TCP-IP networking with files

• /net/tcp
• /clone ? create new socket
• connection_number read(/net/tcp/clone)
• /net/tcp
• /0 ? connection number
• /ctl ? connect/disconnect/co
  nfigure
• /data ? send/receive data
• write(/net/tcp/0/ctl,connect
  129.79.246.181!1234)
• Only textual commands, no byte order issues

41
internet
LAN
gateway
export /net/tcp
mount ( gwip ! port ) /net/tcp
Tattach
Rattach
con_id read(/net/tcp/clone)
Topen
Ropen
write(/net/tcp/0/ctl,connect
129.79.246.181!1234)
Twrite
Rwrite
42
Throughput
43
NTP
1 unsigned char msg4810,0,0,0,0,0,0,0,0 2
clonefd open(conn,"/net/udp/clone",ORDWR) ...
.......................... 7 write(conn,
clonefd, 0, 23, "connect203.50.77.6!123") 8
write(conn, datafd, 0, sizeof(msg), msg) 9
read(conn, datafd, 0, sizeof(msg), msg) 10
seconds ((time_t) msg10)
44
Hosted Mass Storage
1 void FileServe(char filename,int datafd)
2 sprintf(path,"/http/pages/s",filename
) 3 pagefd open(conn,path,OREAD) 4
while (1) 5 n
read(conn, pagefd ,0,100, filedata) 6
if (nlt0) break 7
write(conn, datafd,offset,n,filedata)
Web server on embedded processor utilizes disk
and I/O of host
45
FPGA Based Custom Computing
FPGA
Custom Component
Support File System
Custom Component
46
File System Components

• Server Library
• Single/multi-threaded file systems
• mux, demux components
• Client Library
• -- Single/multi-threaded client libraries

47
Client Complexity (lt 500 lines)

• styx_attach(Conn c, Fid f, char uid)
• Fcall fcall
• Tag t tagalloc()
• fcall.type Tattach
• fcall.fid f
• fcall.uname uid
• styx_send(c, fcall)
• styx_recv(c, fcall)
• tagfree(t)
• -- error handling elided

• FID sys_open(Conn c, char path, int flags)
• char n
• Fid newfid fidalloc()
• styx_clone(c, root, newfid)
• for (n strtok(path, /) n
• n strtok(NULL, /))
• styx_walk(c, newfid, n)
• styx_open(c, newfid, flags)
• return newfid

48
Servers

• Server needs (at least)
• Way to maintain a directory structure
• Procedure to navigate directory structure
• Procedure to read a directory
• Procedure to read/write files
• Simple static table based lt 500 lines
• Dynamic system with ability to mount foreign
  file systems -- lt 2000 lines

49
Table driven approach

• enum
• Qdir,
• Qdata,
• static
• Dirtab testtab
• ".", Qdir, 0, QTDIR, 0, 0555,
• "data", Qdata, 0, 0, 0666,
• .
• Dev testdevtab
• 'T',
• "test",
• devinit,
• testattach,
• testwalk,

50
Component Approach
Customizable File System
Mount File System
Imported file systems
Multiplexer
Imported File System
clients/ mounts
51
What does it all cost ?

• 16 to 32 bit processor
• 10K ram (100k would allow sophisticated system

Note 68000 was 68000 transistors,
MIPS R2000 was 125,000 Thats 17,000/31,000
gates respectively
52
Issues

• Differences between Plan 9 protocols
• Limitations of Plan 9 protocols
• Advantages/Disadvantages of this approach

53
9P vs 9P2000

• 9P2000 enables multi-step walk in single message
• 9P2000 doesnt impose name lengths
• 9P2000 has better authentication model
• 9P has bounded communication buffer requirements
  (121 bytes)
• 9P messages are easier to pack/unpack

54
Limitations of Plan 9 Protocols

• Connection based (can be a problem in sensor
  networks)
• Require reliable connection -- no recovery
  mechanisms
• Not well suited for streaming data

55
Advantages of File System Structure

• Portable protocol (no special transport
  requirements)
• Familiar interface
• Natural naming conventions
• Natural hierarchy
• Extensible

56
Disadvantages of File System Structure

• Based upon RPC which makes asynchronous events
  (more) difficult to support
• Not reflexive you get a name space, but no
  semantics

57
What Were Working On

• Portable libraries
• Partitioned Servers (reduced functionality at
  leaf, full file functionality at intermediate
  server
• Supporting high performance streaming data
• Management of clusters of sensors
• File systems for aggregating WSDL based services
• Managing custom compute engines
• Adapting user space file system interfaces for
  linux

58
Questions ?

• I have funding to support a post doc for up to
  two years. See me if youre interested.

59
Other Research Issues

• File System Architecture
• Portable Protocol (hybrid of Sprite/Plan 9)
• Support multiple physical interfaces (also
  simultaneously)
• Support bridging to other technologies (e.g.
  CORBA)
• Support merging with user level file systems on
  chip
• Generation of On-chip file systems
• Given standard chip level interconnect and a
  library of components, could we generate
  significant portions of file system ?
• Software support
• Initial boot file system, software download
• User level applications (e.g. code to interface
  to debuggers event monitors)
• Hardware support (e.g. specialized DMA channels),
  trace filters, unit testing infrastructure

60
Development Applications

• Configuration/provisioning
• Support processor needs r/w access to registers,
  memories etc. for on chip devices
• Debugging embedded heterogeneous processors
• Event Tracing/Monitoring
• Supporting Unit Testing

61
Inspiration Sprite OS

• Used single network wide name space (less
  flexible than Plan 9, but simpler)
• Simple file protocol based upon RPC
• Name lookup based upon prefix caches (eliminates
  inefficiency of accessing files through a chain
  of servers)

62
Inferno examplehttp//www.vitanuova.com/mkt/press
/Inferno_overview.pdf
63
(No Transcript)
64
Inspiration Unix /proc file system

• System V, BSD, Solaris, Linux all provide a /proc
  file system (Try looking at /proc/pid for some
  pid)
• /proc file system lets user level programs
  explore/control state of running processes (e.g.
  start/stop, read/write registers, memory, etc.)

65
ITRS Quote

• The cost of test application (tester time, and
  ATE speed limitations) will increasingly motivate
  SW-driven, on-chip self-test technologies
  dependencies between system design and test
  architecture design must be planned from initial
  system conception .... significant advances and
  use of BIST and/or embedded software-based
  self-testing for larger portions of test, are
  required.

66
A Simple ExampleBhanu Nagendra, Shashwat
Srivastav

• /pico/
• control/
• RunStop
• registers/
• s1

67
Very Capable Programmable Logic Platforms for
Experimentation

• Xilinx
• Up to 100,000 Logic cells
• Up to 3Mbits of ram
• Microblaze 32 bit RISC processor, UltraController
  ppc405
• Altera
• Up to 80,000 Logic elements
• Up to 7Mbits of ram
• Nios 32/16 bit RISC processors 1000/1500 Logic
  elements user extensible
• Chip level multi-master bus (Avalon) supports
  streaming DMA

68
Move Away From Custom ASIC

• Application Specific Standard Parts (e.g. NPUs
  Intel IXP1200)
• Structured ASICS
• NEC
• Fujitsu AccelArray
• IBM Customizable Control Processor
• ASICs organized round standard IP interconnect
• Sonics bus
• AMBA (arm)
• VSI Alliance
• Open Channel Protocol (OCP)
• Platform Based Design (e.g. set top box)

69
Deficiency with Debugging

• Real-time code should not be stopped
• Stream based processing (e.g. mpeg) shouldnt be
  stopped (data will go in trash)
• In concurrent program, interactions are more
  interesting than execution of individual lines of
  code
• Solution is Tracing/Event Monitoring

70
Prefix Tables
Figures from The Sprite NetworkOperating System
71
(No Transcript)
72
Quotes from International Technology Roadmap for
Semiconductors (ITRS)

• As HW platforms emerge and become increasingly
  fixed, embedded SW will become the main vehicle
  for product differentiation. Embedded SW will
  then become more than a first-class citizen in
  system design it becomes the focus
• Reuse- and SW-centric SoC design is increasingly
  partitioned into (a) the creation of programmable
  platforms by a team of hardware-centric and
  hardware-dependent software-centric designers,
  and (b) use of the platform in derivative
  products by a team of applications and
  software-centric designers

73
Example Continued

• Unix library to support user applications
• Styx file protocol
• Message based
• Attach, clone, walk, open, 13 messages
• Approximately 10K bytes code and data on support
  processor

74
Importing Resources to Chip Level

• System level storage
• System level TCP/IP

75
Remainder of Talk

• Technology Trends
• Embedded File System Example
• Supporting Embedded Software Development
• Related Work (Sources of Inspiration) in File
  Systems
• Research Issues

76
Unit Testing

• Run software directed BIST (vendor supplied)
• Support level unit testing (may require
  interfacing support processor to dedicated
  hardware)
• Inject traffic into subsystems
• Remove traffic from subsystems
• Provide access to vendor supplied boot time
  diagnostic routines

77
Note 68000 was 68000 transistors,
MIPS R2000 was 125,000 Thats 17,000/31,000
gates respectively
78
Example TCP

• Import from Inferno on Host
• /net
• /tcp
• /udp
• To create a connection
• Write address, port to udp/clone
• Read udp/clone to obtain connection ID (e.g. 3)
• /udp
• /3
• /ctl
• /data

79
TCP Example
1 conn styx_connect() 2 clonefd
open(conn,"/net/tcp/clone",ORDWR) 3 read(conn,
clonefd ,0,sizeof(connid), conid) 4
sprintf(data_string,"/net/tcp/s/data",conid) 5
datafd open(conn,data_string,ORDWR) 6
write(conn, clonefd,0,28,"connect
129.79.246.195!10101") 7 for(counter0
counterlt100 counter) 8 n read(conn, datafd
,0, 4, (char) (valuescounter))