Building a Configurable Pub/Sub Notification Service

1
Building a Configurable Pub/Sub Notification
Service

• Cristian G. Fiorentino
• Advisor Mariano Cilia
• UNICEN, June 2006

2
Previous Work - Foundations

• Event-based MMPOG (J2EE, JMS)
• JMS limitations
• Rebeca NS extension Channels
• Research Initiation Scholarship
• DVS Group, TU-Darmstadt, Germany
• Paper Accepted at IFIP DAIS 05
• International Conference on Distributed
  Applications and Interoperable Systems
• Athens, Greece, June 2005
• NS Performance and Reliability Analysis
• Framework Development / Documentation

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

3
Agenda
Index

• Introduction
• Motivation
• Goals
• Framework Design
• Deployment Runtime
• Configuration and Management
• Case Study / Framework Usage
• Conclusions and Future Work

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

4
Agenda
Index

• Introduction
• Motivation
• Goals
• Framework Design
• Deployment Runtime
• Configuration and Management
• Case Study / Framework Usage
• Conclusions and Future Work

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

5
Introduction Event-Based Paradigm
Introduction

• Since more than a decade commercial applications
  do apply the Client/Server architectural style
• New Technology Trends have emerged
• imposing convergence of technologies (sensor
  miniaturization, intelligent adaptative
  autonomous systems)
• business trends like the supply chain
  management, or zero-latency enterprises,
  real-time enterprises
• They have begun mistakenly using the
  Client/Server architectural style.
• Request/Reply paradigm not suited for event-based
  systems
• Event-based paradigm has emerged as the proper
  solution
• allowing the flow of data/events from producer to
  consumer(s)
• the producer of data is responsible to initiate
  the interaction when a situation of interest is
  observed
• incorporation of new applications do not imply
  code modification

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
1
6
Introduction Pub/Sub Notification Services
Introduction

• In distributed systems, the responsibility of
  data dissemination is delegated to the middleware
• We focus on Pub/Sub NS as a particular case of
  this middleware, that implements the event-based
  paradigm
• Pub/Sub mechanisms offer loosely coupled exchange
  of asynchronous notifications, facilitating
  extensibility and application scalability
• Internally Network of brokers, Routing
  Algorithms, Transport

C1
C3
C1
C3
C3
NS
NS
NS
sub(a,b)
pub(a)
notif(b)
notif(a)
sub(a)
pub(b)
C2
C4
C2
C4
notif(a)
7
Introduction Pub/Sub Notification Services
Introduction

• Addressing models
• a main classification for NS. Define subscription
  expressivity and are intimately related with
  routing strategies.
• They have evolved from channel-based, to more
  flexible subject-based, and expressive
  content-based and concept-based approaches
• Many commercial products and specifications
• WebSphereMQ, TIB-Rendezvous, and various JMS
  compliant implementations (Sun, JBoss, OpenJMS,
  ActiveMQ)
• Content-based research prototypes
• Rebeca, Siena, JEDI, Gryphon, etc.
• More recently have emerged Topic-based Pub/Sub
  systems built on top of a P2P overlay network.
• Examples of these NSs are Scribe, Hermes,
  Fiorano.

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
3
8
Introduction Rebeca
Introduction - Pub/Sub Notification Services

• Open Source Notification Service Framework
• Rebeca Event-Based Electronic Commerce
  Architecture
• Research prototype at DVS Group, TU Darmstadt,
  Germany
• Principal purpose
• Test Routing Algorithms and Performance
• Routing Algorithms
• Flooding, Simple Routing, Identity Routing,
  Covering, Merging
• Characteristics
• Filters
• Advertisements
• Extended functionality
• Transactions, Concept-based Addressing, Caching,
  JMS interface, Channels

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
4
9
Introduction Peer-to-Peer (P2P)
Introduction

• P2P an Overlay Network topology
• Offer failure resilience
• Self managing system
• Allow high Scalability
• Nodes hold a circular Ring address
• Point-to-point O(log N) connections
• DHT Solutions
• Pastry Scalable Distributed Object Location,
  Ring O(log(n)), RT, Leaf, Neighbors, Locality
• Bamboo Pastry DHT, Hi Level Churn, Static
  Resilence, Failure Detection, Recovery Algorithms
• P2P Content-based Routing Algorithms
• Spanning Tree approach Tree over P2P Graph, Edge
  Filter, Invariant, Pub/Sub over Trees
• Bit Zipper Data placement for P2P, Rendezvous
  Problem

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
5
10
Agenda
Index

• Introduction
• Motivation
• Goals
• Framework Design
• Deployment Runtime
• Configuration and Management
• Case Study / Framework Usage
• Conclusions and Future Work

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

11
Motivation NS Required Features
Motivation

• To provide applications, the NS needs to
  interpret, aggregate, filter and analyze streams
  of messages to adequately make messages flow from
  any data producer to interested consumers
• Applications require many issues of interest
• Academia research has focused on several of them
• efficient dissemination
• addressing models
• message correlation
• mobility
• scalability
• fault-tolerance

• access controls
• data integration
• security
• privacy protection
• transactions
• caching, etc.

• But still many others need to be offered

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
6
12
NS
?

13
Motivation Todays Problems
Motivation

• Different applications require different NS
  features and combinations
• News Ticker, Supply-chain management, MMPOG .
• Existent Publish/Subscribe middleware supports
  the basic stream of messages
• but are typically monolithic and includes only a
  subset of features

• Today no product offers all required features and
  they are hardly extendible

A problem arises when developers need a
middleware that (completely) fulfills their
application requirements
Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
8
14
Agenda
Index

• Introduction
• Motivation
• Goals
• Framework Architecture
• Lower-level Design Decisions
• Deployment Runtime
• Configuration and Management
• Case Study / Framework Usage
• Conclusions and Future Work

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

15
Goals
Goals

• Offer a NS solution that accomplish
• Clear NS Design
• The inclusion of all desired features BUT only
  the required ones
• Extensible Configurable Features, Usability
• Multi-platform Infrastructure, Technology-independ
  ent
• Offer a facility for generating pub/sub NSs
  that supports the combination of required issues
  based on application requirements
• Allows the configuration of a Pub/Sub solution
  based on a reusable and extensible set of
  building blocks
• Offer a testbed to experiment and probe different
  ideas together with other approaches

Framework
Solution...
Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
9
16
C1
D2
A2
C3
A3
D3
B3
B3
D2
C1
B3
A2
17
But also
Goals

• Allow the NS user to easily generate a deployable
  NS, based on provided or user generated building
  blocks
• Provide tools for NS deployment
• Provide tools for management at runtime
• Allow Functionality Adaptability
• Tuning
• Remote management
• Monitoring
• Offer multiple brokers in the same machine
• Allow different deployment strategies
• Provide different routing algorithms, overlay
  networks and transmission protocols

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
11
18
Proposed Approach
Goals

• FRANSCESCA
• FRAmework for a Notification Service
• ComponEnt-baSed Configurable Architecture
• A framework for Notification Services that
  enables the construction of a NS instance
  according to user needs,
• selecting among a predefined and extensible set
  of components,
• that mainly include broker type, routing
  algorithm and network structure.
• The resulting NS must be then adaptable,
  configurable and easily deployable and
  manageable.

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
12
19
Proposed Approach in three steps
Goals
Elementary Building Blocks
(1) NS Specification
Resulting NS
Adm
Framework
User
Framework
Framework
Configs. Services QoS
(2) NS Deployment
(3) NS Management
Deployment Tool
20
Agenda
Index

• Introduction
• Motivation
• Goals
• Framework Design
• Deployment Runtime
• Configuration and Management
• Case Study / Framework Usage
• Conclusions and Future Work

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

21
Framework Design

• Main Architecture
• Components
• Component Configuration
• Interfaces
• Events/Messages
• Extensibility

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

22
Main Architecture
Framework Design

• Analysis, organization and classification of all
  required NS Features ?
• Layered Design Orthogonal Functionality
• Layered design offers
• Clear, equilibrated organization
• Cohesive independent components
• Goal modifiability and portability
• Designed layers include
• Application specific application
• Broker brokers funct., filters
• Routing router, routing algorithm
• Topology overlay networks
• Network transport, serialization

• Orthogonal Functionality includes Management
  issues like Life Cycle Management, Adaptation,
  Monitoring, Tuning

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
14
23
Main Architecture
Framework Design (cont. 1)

• Layers group a set of components with similar
  functionality sharing the same common interface.
• Components represent specific building blocks.
  Component types
• Main component
• Filter
• Strategy
• A service is a specific combination of components
  crossing (possibly) all layers
• Represent a Broker instance
• Not all component combinations are possible
• Allow to reuse components
• in different services

ServiceA
ServiceB
Management
App1
App2
Application
Concept-B Broker
Content-B Broker
Filter
Broker
Scopes
Bit Zipper
Spanning Tree P2P
Rebeca Routing
Ants
Routing
DHT
Static tree
Channels
Topology
Network UDP
Network TCP
Serializer
Network
Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
15
24
Interfaces
Framework Design

• A common set of interfaces between layers was
  crucial to easily allow to add and change
  functionality
• It has been defined two kinds of interfaces
• Asynchronous they define the main message
  passing system events
• Divided between pair-of-layers protocol,
  connection and interface events
• Also divides as upwards and downwards
• Synchronous define a direct communication
  between layers and components
• Used to get/set state of components directly
• Lower overhead approach
• Message design
• Specific Message interfaces
• Message Models
• Layering message structure (serialization
  required)
• Pipeline message filtering (XML transmission)

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
16
25
Application
HandleEx(Ex)
Broker
Exception Detect()
Routing
ContRoute()
Forward(Rm,k,nh)
Forward()
Topology
errorDet()
-m Message -i Information -k destination
Key -nh nextHopNode -D Destination -St State
-Ev Event -Ae App Event -Ex Exception
Network
errorDet()
26
(No Transcript)
27
Events/Messages
Framework Design

• Definition of different Framework Events
• Management Events configuring, monitoring
• Interlayer Events hold Layer Events
• Layer Events transport
• Messages
• Interface Parameter
• Message design
• Specific Message interfaces
• Message Models
• Layering message structure
• (serialization required)
• Pipeline message filtering
• (XML transmission)

Layering
Pipeline
Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
28
FrameworkEvent (InterlayerEvent)
LayerEvent (ProtocolEvent)
Framework Parameters
LayeringMessage (layer X)
Interface Parameters
Specific Message (Layer X)
Destina tion
Event Type
Framework Message (layer X-1)
Layering
LayerEvent (Layer X1)
e
a
A
F
a
e
a
a
d
b
B
E
e
b
e
a
d
a
b
b
b
b
c
C
D
d
b
c
c
b
d
c
29
FrameworkEvent (InterlayerEvent)
LayerEvent (ProtocolEvent)
Framework Parameters
PipelineMessage (layer X)
Interface Parameters
ltNS XMLgt ltBROKER/gt ltSUBS/gt
ltROUTEgt ltTOPOLOGYgt ... ltNS XML/gt
Pipeline
Unique Message
LayerEvent (Layer X1)
c
A
D
c
a
a
b
a
a
a
b
b
B
C
a
b
a
a
b
a
a b b
30
Extensibility Component Creation
Framework Design

• Define component type
• Decide if include
• configurable strategies
• Identify Data Agreements
• Extensible Representation
• abstract common data types
• For each component class
• specific parameter format
• and data types
• Implement main class
• Implementing layer interfaces
• Possibly include Legacy code
• Include any programming
• languaje code using JNI
• Framework behavior binding
• UserToolkit interface.

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
18
31
Extensibility Component Creation
Framework Design (cont.1)

• Define component configurations and subscriptions
  descriptively
• Layer
• Type
• Class
• Name
• Identifier
• Services
• Message Type
• Subscriptions
• (LDAP syntax)
• Create component
• bundle

32
Agenda
Index

• Introduction
• Motivation
• Goals
• Framework Design
• Deployment Runtime
• Configuration and Management
• Case Study / Framework Usage
• Conclusions and Future Work

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

33
Deployment Runtime

• Deployment Options
• Deployment Runtime
• Component-based Infrastructures
• Container Analysis

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

34
Deployment Options
Deployment Runtime
Global NS View
Broker
Client-Broker
Client
Router
35
Deployment Runtime
Deployment Runtime

• Different deployment and runtime strategies
  offered
• Upwards and downwards event shortcuts in the
  layering stack them allow different deployment
  options.
• Component selection at creation time and runtime
  implicates a variable pipeline of components with
  dynamic wiring
• Multi-platform deployment (Mainframes, PCs,
  Mobile Devices like PDA, telephones, and embedded
  systems)
• How to manage homogeneously different component
  instantiation and how to automate component
  wiring?

Runtime Environment
Solution..
Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
21
36
Component-based Infrastructures
Deployment Runtime

• Lightweight containers Alternative to the
  heavyweight complexity in the mainstream J2EE
• Back to POJOs (Plain Old Java Objects)
• Inversion of Control (IoC) ? Dependency Injection
• Lightweight containers offers a suited Runtime
  Environment
• Instantiating component respect to descriptive
  configurations and management tools
• Component wiring achieved automatically
• Important Deployment decisions
• Container independent code portable code to
  different containers
• Also allow non container-based deployment (clear
  interfaces)

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
22
37
Container Analysis and Selection Criteria
Deployment Runtime
Com- plexity
Life Cicle Manag.
Size
Dyn. Remote Deploy
Config.
Characteristics
Comp
Feature Platform

/- Startable interface

- Coded instance
/-- Get/set inteface
Container, start- able, dep. iny. Inv. of Control
Class
Pico Container
/-
Web
/-

Web
Agents, adaptor Connectors, Mbean Server
MBean
JMX

/- interface

- Coded instance
-
Stage subscribe to events. Build over SEDA
Stage
Bamboo DustDevil

Comm. line
J2ME

Get/set service
Platform, bundles (Jar, Manifest)
Bundle
OSGi
38
OSGi / Oscar
Deployment Runtime

• OSGi
• Open Services Gateway Initiative
• Light weight Container (J2ME compliant)
• Life Cycle Management
• Everything is a service (bundles)
• Service Binder
• Unification and Business Code separation
• Component subscription
• XML configuration file
• Configuration
• Dynamic configuration
• Dynamic update with Service Binder

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

39
Agenda
Index

• Introduction
• Motivation
• Goals
• Framework Design
• Deployment Runtime
• Configuration and Management
• Case Study / Framework Usage
• Conclusions and Future Work

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

40
Configuration and Management

• Configuration and Management
• Architecture Configuration Verification
• Management Design

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

41
Configuration and Management
Configuration and Management

• Formalization of the main Configuration
  Capabilities
• Layer Selection
• Component Types
• Functionalities Selection
• Component Settings (configuration and tuning)
• Orthogonal Functionality includes Configuration
  and Management functions
• Life Cycle Management
• Configuration
• Monitoring
• Includes Runtime Environment and Wiring
• Management components treated as container
  components

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
24
42
Architecture Configuration Verification
Configuration and Management

• Architecture-based Verifications assure
  architecture coherence restricting the evolution
  of architectural design
• For every configuration capability a related
  Verification
• Layer Selection ? Layering Verifications
• Component Type ? Component Type Verification
• Functionality Selection ? Functionality
  Arch.Consistency
• Component Settings ? Component Settings Verif.
• Map NS Framework architecture into Families and
  Systems of some ADL (ACME)
• Checks creation or runtime configurations
  (system) against predefined architectural family
  constraints

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
25
43
Management Design
Configuration and Management
Management
Manual Configuring
Monitoring
input
monit
Comp. Config.
Config. Data
Main Manager
update
Life Cycle Manag.
output
Event Translation
Layer Style Checker
Main Checker
Comp Type Checker
output
Settings Checker
Configuring Verification
44
Agenda
Index

• Introduction
• Motivation
• Goals
• Framework Design
• Deployment Runtime
• Configuration and Management
• Case Study / Framework Usage
• Conclusions and Future Work

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

45
Case Study / Framework Usage

• Rebeca Encapsulation
• Encapsulation Implementation
• Framework Usage Tools
• NS Creation
• Component Life Cycle Manag.
• Component Configuration

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

46
Rebeca Encapsulation
Case Study/Framework Usage

• Relying on Franscesca, reuse Rebeca code
• A way to validate Framework design and defined
  interfaces
• Encapsulate Rebeca code into Franscesca component
  for each layer
• Just encapsulating components, Rebeca gains
  Franscesca advantages management, different
  message models, extensibility, etc.

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
27
47
Encapsulation Implementation
Case Study/Framework Usage

• Different design found
• Similitudes Routing, Application
• Lower design intersections
• Broker/Network, Topology/Routing
• Not all interfaces used
• Map data types, Main comp. types

Application
Routing
Topology
Broker
Network
48
Framework Usage Tools
Case Study/Framework Usage

• NS Creation

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
29
49
Framework Usage Tools
Case Study/Framework Usage

• NS Creation
• Component Life Cycle Management

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
30
50
Framework Usage Tools
Case Study/Framework Usage

• NS Creation
• Component Life Cycle Management
• Component Configuration

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
31
51
Agenda
Index

• Introduction
• Motivation
• Goals
• Framework Design
• Deployment Runtime
• Configuration and Management
• Case Study / Framework Usage
• Conclusions and Future Work

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

52
Conclusions
Conclusions and Future Work

• Main Contributions
• Obtained a solution to satisfy specific NS
  requirements
• Flexible framework, covering wide range of NSs
• Well suited layered architecture, with a accurate
  definition Interfaces for each layer
• Allow Different deployment configurations
• Main Characteristics
• Allow multi-brokers (optimizing computational
  resources)
• Different message representations (structured,
  XML)
• Message Definition and Common data agreements
  allow to implement components using to legacy
  code
• Different component types, offering design
  flexibility
• Descriptive configuration allow reusable
  components
• Good extensibility due to framework behavior
  binding, UserToolkit

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
32
53
Conclusions
Conclusions and Future Work

• Facilities and Tools offered
• Allows to generate NS instances, ready to be
  deployed in a Runtime Environment
• Remote deloyment tools (NS instances or
  components)
• Components life-cycle Management (Component
  Adaptation)
• Manual or Automated component tuning
• Other interesting features
• The architecture based configuration verification
  formalize component relationship
• Scalability and self managing NSs using P2P
  overlays
• Reliability protocols are ready to be included
• Management components treated as layering ones .

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
33
54
Future Work
Conclusions and Future Work

• Offer a wide component catalog
• Support with P2P Overlays and related Routing
  algorithms
• Include Concept-based addressing components
• Implement SOAP-based protocols (in network layer)
  to transmit XML messages
• Analysis and comparison
• Use framework as a testbed to compare different
  combinations
• Test framework overhead comparing monolithic with
  framework-integrated solutions.
• Include QoS metrics based on recent research
• Establish NS Security models based on scopes
• Based on the SOAP implementation, implement and
  test WSNotifications specifications
• Implement aspects of reliability, based on JMS
  approaches and Web Services Reliability
  specifications.

Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service
34
55
The End.. Thank you!
Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service

56
The End.. Thank you!
Questions?
Fiorentino Cristian, 2006 Building
a Configurable Pub/Sub Notification Service