Distributed Computing with DAFFIE

1
Distributed Computing with DAFFIE

• Glenn Bresnahan
• Boston University
• Scientific Computing and Visualization
• (http//scv.bu.edu/DAFFIE/Earlab-Oct02)

2
What is DAFFIE?

• Distributed Applications Framework For Immersive
  Environments
• System for easily creating distributed and shared
  virtual environments
• Authored by Glenn Bresnahan, Erik Brisson, Robert
  Putnam

3
DAFFIE Messaging System

• Application protocol
• Client services
• Client application library (API)
• Message router (event server)

4
DAFFIE Message routing
client
client
client
client
proxy
client
event server
client
proxy
client
client
client
5
Application Architecture

App thread
App thread
App thread
App thread

Services (timing, streaming, file sharing)
Event Messaging
Message Transport
6
API - Control

• id event_join(server, nc)
• rc event_leave()
• id event_id()
• nc event_nclient()
• millisec event_clock()
• event_sleep(millisec)
• event_register(class, species, instance)

7
Event headers information

• EventType(event)
• EventSize(event)
• EventFrom(event)
• EventTo(event)
• EventSeq(event)

8
System Events Join Leave

• typedef struct
• EVENT_HEAD head
• int id
• int nmember
• EVENT_JOIN
• typedef struct
• EVENT_HEAD head
• int id
• short detail
• short exitcode
• EVENT_LEAVE

9
System Events Identification

• typedef struct
• EVENT_HEAD head
• int tag
• EVENT_IDENTITY
• typedef struct
• EVENT_HEAD head
• char hostMAXIDSTR
• char classMAXIDSTR
• char speciesMAXIDSTR
• char instanceMAXIDSTR
• int tag
• EVENT_IDENTIFICATION

10
API Send

• rc event_send(to, type, event, size)
• to is a client id or one of the following
• EVENT_BCAST_ALL
• EVENT_BCAST_NOTME
• EVENT_BCAST_SERVER
• EVENT_BCAST_LOCAL
• Macros
• SEND(to, type, event)
• BCAST(type, event)
• event_drain()

11
API - Receive

• rc event_receive(event)
• event_receive_type(n, type1, )
• event_select(on_off, et_low, et_high)
• event_wait()
• event_wait_timeout(millisec)

12
API Receive queue

• event_scan_type(n, type1, )
• rc event_scan(event)
• n event_queued()
• event_flush(flag)
• event_flush_sync()
• event_receive_enable(on_off)

13
API Active messages

• event_callback(type, func)
• event_add_callback(type, func)
• event_delete_calllback(type, func)
• event_protect_type(on_off, et_low, et_high)

14
Client Services

• Streaming
• Data chunking
• Bandwidth management
• Dynamic tuning
• File sharing
• File servers
• Conditional upload/download

15
Job Initiation Monitoring

• Rely on external authentication and job
  initiation (e.g. ssh, Globus)
• Launch scripts (ad hoc)
• cgl-video visible.bu.eduvgrab -compress
  -process -vswirl -steps 9999999 -np 4
  -vbroadcast .vdirect -object poolanim_grp
• Agent initiation daemon
• Logging daemon

16
Message Transports

• TCP/IP
• Single channel or striped
• Shared memory
• Myrinet
• IP Multicast

17
Defining Upper Layer Protocol

• Components
• Event type ID (define in a .h)
• Data layout (compile-time) (C struct in a .h)
• Data layout (run-time) (event_define)
• Needed for canonical byte ordering
• Types statically (compile time) defined
• Protocol compiler

18
Message Routing

• Message router (event server)
• Multiple delivery service classes
• Globally sequenced, reliable delivery
• Client management
• Message subscription
• Multiprocessor, high bandwidth
• Hierarchical servers
• Proxy routers
• Fan-in/fan-out
• Tunneling
• Direct routing for P2P

19
Messaging API

• Multithreaded, non-blocking
• Dynamic join and leave
• Point-to-point and broadcast messages
• Polled or callback (active messages)
• Subscription based receive
• Low level services
• Registration, timing, synchronization, query

20
Service Discovery

• Client registration/identification
• Class (Viewer, Audio, Video, Agent)
• Species (video source, broadcaster, sink,
  processor)
• Instance (video stream-name)
• ID
• Hostname
• Client location services
• Service request/acknowledgement

21
Client Monitoring

• Dynamic clients
• JOIN, EXIT, IDENTIFICATION events
• Event server monitoring
• Heart beat
• Latency monitor
• Congestion control
• Statistic generation
• Client monitoring
• Process level monitoring
• Congestion management
• Remote process termination

22
Leaky Example Headers

• typedef struct
• EVENT_HEAD head
• char filename200
• EVENT_GETPARAM
• typedef struct
• EVENT_HEAD head
• int logger, frame_size, sample_rate)
• EVENT_START
• typedef struct
• EVENT_HEAD head
• char msg1000
• EVENT_LOGINFO

23
Leaky Example Headers

• typedef struct
• EVENT_HEAD head
• int client
• int size
• float pressure1
• EVENT_ADVANCE

24
Leaky Example Headers

• define ET_START 1001
• define ET_GETPARAM 1002
• define ET_LOGINFO 1003
• define ET_ADVANCE 1004
• define ET_STOP 1005
• event_define(ET_START, start, 1, 3,
  EVENT_DATA_INT)
• event_define(ET_GETPARAM, getparam, 1, 200,
  EVENT_DATA_CHAR)
• event_define(ET_ADVANCE, advance, 3,
• 1, EVENT_DATA_INT,
• 1, EVENT_DATA_INT,
• 0, EVENT_DATA_FLOAT)

25
Leaky Example Initialization

• id event_join(earlab.bu.edu, nc)
• printf(Joined as client id d of d\n, id, nc)
• event_register(integrator, leaky, 1.0)
• event_receive_type(ET_GETPARAM, ET_START,
  ET_ADVANCE, ET_STOP)
• event_callback(ET_START, do_start)
• event_callback(ET_GETPARAM, do_getparam)

26
Leaky Example File Service

• GotParamFile 0
• efs_init()
• efs_chdir_receive(/tmp/earlab)

27
Leaky Example GetParameters

• do_getparams(EVENT_GETPARM egp)
• efs_download_cond(egp-gtfilename,
• egp-gtfilename, xfer_done, 0)
• return(1)
• xfer_done(int reqid, int fsid, inr rc, char
  file, int tag)
• char path1000
• strcpy(path, /tmp/earlab/) strcat(path,
  file)
• GotParamFile 1
• Leaky_GetParameters(path)
• )

28
Leaky Example Start

• do_start(EVENT_START est)
• Logger est-gtlogger
• while (!GotParamFile) event_sleep(100)
• Leaky_Start(egp-gtframe_size, egp-gtsample_rate)
• return(1)

29
Leaky Example LogInfo

• void LogInfo(char msg)
• EVENT_LOGINFO einfo
• strcpy(einfo-gtmsg, msg)
• event_send(Logger, ET_LOGINFO, einfo,
  sizeof(einfo))

30
Leaky Example Advance

• do_advance(EVENT_ADVANCE ein)
• int bytes
• EVENT_ADVANCE eout
• bytes sizeof(EVENT_ADVANCE)
  (ein-gtsize-1)sizeof(float)
• eout (EVENT_ADVANCE ) malloc(bytes)
• Leaky_Advance(ein-gtpressure, eout-gtpressure)
• eout-gtclient event_clientid()
• eout-gtsize ein-gtsize
• event_send(ein-gtclient, ET_ADVANCE, eout,
  bytes)
• free(ein)
• return(1)

31
DAFFIE Message routing
client
client
client
client
proxy
client
event server
client
proxy
client
client
client