Infopipes

1
Infopipes and the Infopipe Stub
Generator Galen Swint, Calton Pu, Younggyun
Koh, Wenchang Yan
2
Overview

• The Infosphere project
• Goals of the ISG
• Implementation
• Current results
• Infopipes and OEP
• Future Work

3
The Infosphere Project

• Distributed computing with RPC is hard
• RPC semantics do not fit streaming applications
• Need services to be more composable
• QoS concerns
• expanded QoS
• eliminate redundant QoS coding
• Enhance portability
• Same specification can be made for several
  different comm. machines

4
Motivation

• RPC
• Procedure call abstraction hides network
• Application must address QoS
• Does not understand application packets
• Request/Response is an unnatural fit for
  streaming media (movies/sound)
• Composable?
• Basis for CORBA, RMI, Sun RPC, SOAP
• We need an information-centric abstraction that
  is network aware to complement RPC

5
Goals for ISL/ISG

• Form the core of an Infopipe toolkit
• ISL
• Simple description for flows
• Support datatypes
• Support basic composition
• QoS requirements
• ISG
• Generate datatypes, communication stubs
• Support multiple communication layers
• Support multiple languages

6
Implementation

• ISG becomes part of a process
• XSLT C
• Brand new XIP (Young)
• New comm. added more quickly (but needs more
  typing!)
• XSLT allows easy modularity in templates
  (xslinclude)
• Must re-implement the aspects
• Buzzword compliant

7
Implementation v0.2
8
µ-benchmarks

• Round trip time (ping-pong)
• One number sent and received on separate simplex
  channels
• 100 experiments of 1000 ping-pongs

9
Experiment 1 - UAV

• UAV demo
• DoD project
• ECho
• Remote camera sends data via wireless link
• Demonstrates code uploading
• ISG generates replacement comm. code from SIP/XIP

10
Experiment 1 - UAV - Results

• Initialization
• 101 runs, discard first
• No statisical difference

Same performance with 36 fewer LOC!
11
Conclusions

• No performance hit
• Substantial LOC savings
• Time savings for development
• Potential portability
• LOC savings probably varies with App

12
OEP Integration Recap

• Participation in BBN OEP
• Source-based filters for adaptive constrained
  resource management in the Multi-UAV demo
• New filters as Infopipes (compression,
  encryption)
• Better DRE application programs and better QoS
  support (more platforms and dimensions)
• Infopipe software tools
• Infopipe Stub Generator, program viewer, other
  tools
• Demo apps (BBN OEP, Boeing OEP, etc)

13
New Infopipe Functionality

• Three kinds of filters (and more)
• Image filters (lower/higher resolution, BW)
• Compression (gzip, lzo, JPEG)
• Encryption (Rijndael)
• Two underlying platforms
• BSD sockets/TCP and TAO AVStreams/UDP
• And more event channels, publish/subscribe
• Automated QoS management
• Adaptation by choosing appropriate combination of
  filters (also controlled manually at runtime)

14
Abstract Infopipe View
Defragmenter
Assembler
BBN Video Distributor
ATR (PPM)
UDP
Wireless Link-TCP
QuO Contract
Filter Control GUI
Source Filter
15
Single Filter Experiments
Application
GUI for Infopipe Specification
Infopipe Spec. Lang. (ISL)
(evolving)
ISL2XIP
XIP
(fixed, extensible)
16
Multiple-Filter Experiment (1)

• Refining the information flow by combining
  Infopipes through connectors

ATR
17
Multiple-Filter Experiment (2)

• Self-configured Infopipes (when data stream is
  unencrypted, system bypasses the decryption stage)

Image Filter
Compress
Decompress
ATR
18
Multiple-Filter Experiment (3)

• Adaptive and flexible distribution of Infopipe
  stages at runtime

Source
Image Filter
Distributor
ATR
19
Connector Specialization
Different machines
Socket Connector
IPC Connector
Function Call Conn.
Same machine
Same process/CPU
Function Call Conn.
20
Future Work

• Stabilize comm. layers
• Some basic type checking
• Use aspects for QoS (w/Lenin S.)
• Wrap in specialization (w/Young K.)
• Use aspects for performance adaptation

21
Planned Experiments

• Infopipe (filter) development and integration
• Development cost/time, variety of filter code
• Code quantity (ISL/XIP module size), quality
  (runtime overhead), portability (variety of
  underlying platforms)
• Integration into OEP, making QoS work with new
  filters (measure QoS dimensions below)
• QoS dimensions and trade-offs
• Performance (e.g., latency, bandwidth, image
  resolution)
• Security (e.g., encryption level)
• Other platforms in OEP that support QoS

22
Credits

• Initial stub generator Morimori
• ECho Greg Eisenhauer, Karsten Schwan
• Java socket template Younggyun Koh
• C socket template Volkan Altuntas
• Initial QoS Wei Han
• Demo harnesses, demo integration Wenchang Yan
• Initial UAV demo Fabian Bustamante, P_at_rick
  Widener

23
XIP Example

• Creating a data type
• From the UAV demo, this holds a frame

ltdatatype name"Raw_data"gt   ltarg type"integer"
name"tag" /gt   ltarg type"char" name"ppm1" /gt
  ltarg type"char" name"ppm2" /gt   ltarg
type"integer" name"size" /gt   ltarg
type"integer" name"width" /gt   ltarg
type"integer" name"height" /gt   ltarg
type"integer" name"maxval" /gt   ltarrayArg
type"byte" name"buff" size"155000" /gt
lt/datatypegt
24
XIP Example

• Creating a filter
• From the UAV example, this crops an image to half
  size

ltfilter name"cropImage1Cto2C"
uri"cropImage.ecl" inType"Raw_data1C"
outType"Raw_data2C" takesParams"1"
paramType"Params_rectangle" /gt
25
XIP Example

• Declaring a simple, singular pipe
• From UAV example, the sender and receivers
• Note each is only a half-pipe

ltpipe name"uavSender" absMachine"ECho2"gt  
ltoutport name"send1" type"Raw_data" /gt
lt/pipegt ltpipe name"uavReceiverNormal"
absMachine"ECho2"gt   ltinport name"receive1"
type"Raw_data" /gt lt/pipegt
26
XIP Example

• Connecting pipes together

ltcomposedPipe name"multiUAV"gt ltdeclarationsgt  
ltpipe name"sender" class"uavSender" /gt  
ltpipe name"receiver" class"uavReceiverNormal"
/gt   lt/declarationsgt   ltports /gt
ltconnectionsgt ltconnectiongt   ltfrom
pipe"sender" port"send1" /gt   ltto
pipe"receiver" port"receive1" /gt  
lt/connectiongt   lt/connectionsgt lt/composedPipegt
27
ISG Template

• Snippet for current version of ISG
• Submit function for sending data
• for-each command generates for multiple outputs
• nodePipeOutConnect captures some semantic data
  about the template

ltxslfor-each select"/xip/pipes/pipe_at_namethisP
ipeClass/outport"gt ltxslvariable name"portName"
select"_at_name"/gt ltnodePipeOutConnect
mark"true"gt int ltxslvalue-of select"thisPipeNa
me"/gt_ltconnType/gt_ltxslvalue-of
select"portName"/gtltconnName/gtSubmit(
) lt/nodePipeOutConnectgt lt/xslfor-eachgt
28
Et voila!

• The code produced

//int sender__send1Submit( CarrierStruct
) int sender__send1Submit( )
29
µ-benchmarks

• Single Integer
• Loop 100,000 times sending 1 integer (4 bytes)