Metadata Driven Data Services for SOA

1
Metadata Driven Data Services for SOA

• Paul Anderson
• Technical Sales Director18 May 2015

2
Agenda

• SOA Review
• Data Integration
• Data Services
• Use Cases
• Wrap-Up

3
Overview
Data architecture and its management are evolving
toward an data services management system that
delivers information as a service.
4
SOA Timeline

• Personal involvement since 1998
• Candle's Roma product of 1998 is the ESB's most
  direct ancestor" (Roy Schulte, Gartner)
• SOA concepts have existed for a long time
• Business needs did not drive SOA adoption
• Fragmented technology slowed adoption
• Gained widespread acceptance in last 5 years
• Web Service Technology has enabled broad adoption
  of the concept

5
What is SOA?

• Gartner
• "Web services are about technology
  specifications, whereas SOA is a software design
  principle. (Yefim V. Natis)
• SOA Concepts
• Decoupling of service production from consumption
• Service Interchange
• Service Discovery
• SOA is an architectural pattern
• The architectural pattern has/does appeared in
  many guises
• ESB tools represents the most widely accepted
  toolset for SOA implementation
• Just because you use an ESB does not mean you
  have an SOA architecture
• SOA benefits from the platform neutrality of Web
  Services
• Enables interchange of services
• Requires decoupling
• Provides discovery
• The changing business environment is driving SOA
• The scope of business functionality is changing

6
SOA Business Drivers

• Business Process enhancements
• Migration of customer management to relationship
  management
• Risk Management across/between organizations
• Order/Shipment tracking across/between
  organizations
• Tighter Partner Management
• Cross-Sell / Up-sell
• Based on history/demographics/relationship/value
• Business Agility
• Services are re-usable building blocks
• Cataloging of available services
• Enhanced time to market
• Focus on information rather than data
• Integrated Businesses
• Transactions span multiple organizations

7
Characteristics of SOA

• Requires Horizontal integration
• Scope crosses lines of business boundaries
• Scope crosses corporate boundaries
• Requires Horizontal understanding
• Requires understanding of multiple lines of
  business
• Requires development of common vocabulary
• Require concept generalization
• Exploitation of commonality
• Drilldown into specifics
• Requires easy access to information
• Need to remove the complexity of differing
  semantics, location, and access methods

8
SOA often slowed by

• Vertical Fragmentation
• Differing technologies across verticals
• Differing semantics across verticals
• Infrastructure Fragmentation
• Application fragmentation
• Hardware fragmentation
• Data fragmentation
• LOB focus
• Multiple technologies
• Replicated data
• Organizational fragmentation
• Ownership Issues
• Organizational issues
• Governance issues

9
Common Misconceptions 1

• SOA is an technology issue

• Web Services are a suite technology
  specifications
• You can implement a web service based stove pipe
• SOA does not have to be web service based
• Looks like an arcitectural diagram but is just a
  list of standards

10
Common Misconceptions 2

• SOA is an application issue

• Focuses on services but ignores the underlying
  data
• Characteristics of an application layer are very
  different from the characteristics of a data
  layer
• Focuses on reuse of functionality and ignores
  reuse of data

11
Common Misconceptions 3

• An ESB tool is all you need

12
Common Misconceptions 3 (Continued)

• Fragile implementation
• Data access is embodied in the business process
• Adding an additional data source requires
  business process change
• The data services are source specific
• Poor performance
• Data reduction is likely happening in the
  Business Process layer
• XML well suited for business process layer but
  not as well suited to the data services layer
• Unmanageable
• Difficult to audit data access
• Difficult to secure data access

13
SOA reality

• SOA is fueled by data
• SOA combines People, Machines
• SOA embodies Business Process
• SOA is based on data
• SOA is fueled by metadata
• You can only reuse what you know exists
• You can only reuse what you understand
• SOA based applications need to co-exist with
  existing applications

14
Data Integration Implementation Evolves Toward
Data Services Gartners View
Users
Applications
Processes
Business Services
Get customer
Close account
Calc lifetime value
Archive history
Get single view of product
Request for data operations(Type, Format,
Latency, Quality)
Response (Data or Metadata)
Data Services
Metadata
Sync
Transform
Aggregate
Access
Profile
Move
15
(No Transcript)
16
Agenda

• SOA Review
• Data Integration
• Data Services
• Use Cases
• Wrap-Up

17
Enterprise Information Problem
The current state of corporate data
Disparate Information Silos
?
Data Services
Enterprise Information Consumers
Enterprise Information Sources
18
Data Integration Challenges

• Organizations cannot address basic questions
• What information do I have?
• Where is it?
• How do I access the most current information?
• How do I manage it?
• What is the impact of change during and after
  implementation of my system?
• Information is stored in myriad data stores
  across the Enterprise.
• Different formats/data types
• Different structures/semantics
• Different access methods (APIs)

19
Data Integration Landscape

• EAI - Target is the application
• Even Driven data movement between apps
• Data movement pre-wired
• Message based.
• Requires application coordination / workflow.
• Requires complex translation / programming.

• ETL - Target is the database
• Scheduled extraction of data
• Bulk loading of data warehouses.
• Batch driven, Large Volumes.
• Inflexible, unable to meet business agility
  needs
• Historical data analysis.
• One-way data movement, read-only access.

CRM
ERP
ETL

• Data Services - Target is the end-user
• On-demand delivery of information
• Real-time integration of disparate data.
• Universal data access layer.
• Integrates multiple data sources.
• Push / Pull any data across the enterprise.
• Single View of Customer.
• Supports dynamic / evolving reporting.
• Supports / Enables SOA / Web Services.

20
Integration Technologies
Process
Integration Style
Data
DataServices
Real Time
EAI
Data Integration Timeliness
ETL
Batch
21
What if.
Reporting, Analytics
CustomApplications
Web Services,Business Processes
Packaged Applications
EAI, ESB, BPM
Enterprise Data Services
Virtual Data
Client sees single database containing enterprise
data Clients are freed from the specifics of
underlying sources
Each client has their own desired view Clients
issue queries against their view
22
Enterprise Data Services
Applications / Dashboards
ESB/ Portals
Reporting / Analytics

• High Performance Infrastructure
• Universal data access layer.
• Integration of structured/unstructured data.
• Data access security enforcement.
• Insulates data consumers from sources.
• Real-time integration of disparate data.
• Federation of operational historical data.
• Provides a virtual representation of data.
• Extract Combine information on-demand.
• Supports dynamic reporting / dashboards.

Web Service/ Relational
Relational
Web Service
23
Model-driven Data services
Understand, relate, harmonize, rationalize, and
use
24
Agenda

• SOA Review
• Data Integration
• Data Services
• Use Cases
• Wrap-Up

25
SOA Challenges

• Majority of data is held in relational sources
• Relational structure optimized for LOB usage
• Probably do not want the existing data structure
  to drive data service structures
• Performance concerns may rule out XML federation
• Granularity Issues
• Granularity of data needed to support SOA may be
  different than current use cases
• Typically there will be data reduction moving
  from existing data sources to data service
  presentation
• Web-Service data is XML based
• Data is organized hierarchically
• Need to map from relational to Hierarchical
• Scope of application typically wider that
  traditional applications
• Spanning lines of business and/or corporate
  boundaries

26
SOA Challenges (continued)

• Performance
• XML based data is exceptionally verbose
• Exposing existing data in XML clearly not viable
• Security
• Broader access to data drives security concerns
• Applications that span political boundaries may
  be subject to local confidentiality regulations
• May need to integrate data security with
  application security infrastructure
• Auditability
• Regulations demand access audit trails

27
From Application Silos to SOA
As-is Application Silos
Application
Application
Application
Server
Server
Server
DBMS
DBMS
DBMS
28
What is a Data Service?

• Decouple data sources from application
• Data implementation shielded from application
• Semantic/Format Mediation
• Standard vocabulary
• Single access point
• Web Service/XML
• SQL
• Federation
• Scalability
• Security, performance

SOAP/XML
Bridge the Gap
Data Service
API Call
SQL
SQL
Master Data
SAP
Divisional Operational Application
29
Data Service Layer in SOA
Client Process Applications
App
App
App
App
App
App
Business Process Services
Business Services
Message Services (ESB)
Data Service
Data Service
Data Service
Data Service
Data Service
Data Service
Data Sources
30
Obtain XML From Non-XML Sources
GIVEN Data Sources containing Information to
integrate
GIVEN Fixed XML Schema WANT Data complying to
schema
NEED Mapping from Data to XML
?
31
XML Modeling to achieve business agility
The Problem
32
Data Service Design

• Three typical use scenarios
• Bottom-up expose relational-style sources in
  table-like form
• Business view starting from XSD/XML-based
  business views
• Top-down starting from WSDL definition of Web
  service operations

Bottom Up
WSDL
WSDL descriptor
VDB Container
Web Svc Operation
Web service operation
XSD IN
XSD OUT
XSDs - in/out
ltXgt
XML views
Relational views
Source models
Import sources
Start
33
Data Service Design

• Three typical use scenarios
• Bottom-up expose relational-style sources in
  table-like form
• Business view starting from XSD/XML-based
  business views
• Top-down starting from WSDL definition of Web
  service operations

Business View
WSDL
WSDL descriptor
VDB Container
Web Svc Operation
Web service operation
XSD IN
XSD OUT
XSDs - in/out
Start
ltXgt
XML views
Relational views
Source models
Import sources
34
Data Service Design

• Three typical use scenarios
• Bottom-up expose relational-style sources in
  table-like form
• Business view starting from XSD/XML-based
  business views
• Top-down starting from WSDL definition of Web
  service operations

Top Down
WSDL
WSDL descriptor
Start
VDB Container
Web Svc Operation
Web service operation
XSD IN
XSD OUT
XSDs - in/out
ltXgt
XML views
Relational views
Source models
Import sources
35
Data Service Design

• Three typical use scenarios
• Bottom-up expose relational-style sources in
  table-like form
• Business view starting from XSD/XML-based
  business views
• Top-down starting from WSDL definition of Web
  service operations

Bottom Up
Business View
Top Down
WSDL
WSDL descriptor
Start
VDB Container
Web Svc Operation
Web service operation
XSD IN
XSD OUT
XSDs - in/out
Start
ltXgt
XML views
Relational views
Source models
Import sources
Start
36
Data Services Design Time
Metadata Modeler
Metadata Reports

• Model and Manage metadata.
• Combines technical business metadata
• Enrich models with custom metadata.
• Infer relationships.
• Infer ownership.
• Maintain consistent data dictionary
• Understand overlaps in business areas
• Reduce time to perform impact analysis.
• Metadata open standards
• MOF
• XMI

Metadata Relational View
Security
Runtime Metadata
Design time Metadata
37
Data Services Run Time

• Model driven Integration
• GUI based modeler
• Metadata Repository

MetaData Modeler
Console (Admin)
Query Builder

• Data Connectivity via
• SQL-92
• JDBC
• ODBC
• SOAP/HTTP
• SOAP/JMS

• Robust security and access control
• Row column level
• Supports existing authentication solutions

Security/Audit

• Distributed query optimization processing
• Data Caching vs. Real Time

Query Processing / Optimise Engine
Data Cache

• Connectivity to
• Databases
• ERP
• MOM
• Data Warehouses
• Apps / Legacy
• Files

Connector Framework
38
Data Services benefits

• On-demand information
• Real time data integration
• Information sharing between business units
• Federation of disparate Information
• Structured, unstructured
• Relational XML Enterprise Apps Legacy
• Faster time to market
• Integrated information in days, weeks
• Tight coupling of design implementation phases
• Leveraging the skill-set of the data architects
  for integration
• Costs across application lifecycle reduced
• Model-driven abstraction layer between
  information sources and applications eases
  development and maintenance

39
Agenda

• SOA Review
• Data Integration
• Data Services
• Use Cases
• Wrap-Up

40
Typical Use Cases

• Financial Services
• Market Reference Data
• Risk Management BIS - Basel II
• Transaction Monitoring Anti-money Laundering,
  Patriot Act
• Supply Chain
• Visibility
• Reporting
• Customer Service
• Single View of Customer
• Account Aggregation Cross-marketing
• Financial Reporting
• Corporate Governance Compliance
• Sarbanes-Oxley, Executive Dashboard
• Homeland Security
• Watch Lists

41
Securities Reference Data

• Challenge
• Securities data is inconsistent
• Consistent pricing other data needed for better
  trades accounting
• Hard-coding was too expensive and inflexible
• Solution
• Models define common view of securities data
• Transform data to XML formats
• Publish to pub/sub system
• Support over 50 front-end applications
• ROI
• Consistent data better trades, better
  accounting, better risk management
• New data sources online faster (weeks)
• 3 million in IT savings

Front-End Applications
IBM MQ Series
MetaMatrix Server
Security Price
Security Description
Security Price
42
Implementing Web Services Service Oriented
Architecture

• Business Drivers
• Information sources available and discoverable as
  location-independent Services on the CSFB Network
• CSFB Lines of business have access to information
  when and where it is needed
• Web Services Loosely Coupled IT Architecture
• Technology Use Case Universal Data Services
• Web Service consumers integrated to discoverable
  Services
• Encapsulate existing/legacy functionality
• Solution Benefits ROI
• 70 of IT Budget spent on integrating CSFB
  systems
• Found new uses for old data
• Squeezed more value out of legacy systems
• Embrace heterogeneity
• Increase business agility

43
Single View of Customer

• Challenge
• Call center application
• Full view of customer interaction
• Data distributed across multiple systems
• Data inconsistent
• Need a enterprise-wide data layer for more
  applications
• MetaMatrix Solution
• Real-time access
• Virtual views of data
• Federated queries across systems
• Common re-usable data definitions
• Enterprise-wide data layer for re-use
• ROI
• Better service of customers
• Lower data integration costs

Call Center Application
MetaMatrix
Data Sources
44
Conceptual Architecture
Information Consumption
Application
Integration
XML /
XML
XML
XML / Binary
XML / Binary
Binary
Enterprise Information Integration
Information Services
Information Bus
Metadata Management
Data Quality
Extract, Transform and Load
Information Transport Protocols
Near Real Time
Information Assets
LOB App(s)
Meta
Rules
ODS
Content
Data Staging
External Data
Data
(structured)
(unstructured)
Data Warehouse
LOB Data
Data Marts
45
Agenda

• SOA Review
• Data Integration
• Data Services
• Use Cases
• Wrap-Up

46
Recommendations

• Start small, incrementally grow toward
  information as a service
• Build the infrastructure on a project by project
  basis
• Data security must be a top priority
• Ensure performance as you go
• Keep it in SOA context information fabric is a
  means to an end, promoting information as a
  service
• But ability to access and use information is a
  longer-term vision

47
Thank You
MetaMatrix Paul Anderson Technical Director 508
720 9266 panderson_at_metamatrix.com
48
(No Transcript)