SDMX Implementation and Support

1
SDMX Implementation and Support

• Background
• SDMX standard
• Projects Census Hub, SODI
• Objectives
• Improve development of SDMX for Data Providers
• Define a reference architecture (focus on Census
  Hub, SODI) and support member states to implement
  it
• Facilitate re-usability of software

2
Objectives of the Reference Architecture

• SDMX data provisioning from existing NSI
  infrastructure
• Reference architecture independent of platform
• Provide reusable components
• Java and .NET implementations
• Components released as Open Source Software

3
Existing NSI Dissemination Architectures

• ConIstat
• Istats Dissemination System
• Modules for SDMX data exchange and dissemination
  according to the SODI project
• SDMX Data Web Service implements the
  functionalities of the NSI Web Service, SDMX
  Query Parser, Data Retriever and SDMX-ML Data
  Generator all together in one module
• Loader assumes the roles of both Data Loader and
  RSS Generator
• SDMX compliant databases
• Straightforward integration with SDMX Reference
  Architecture

4
Existing NSI Dissemination Architectures (contd)

• PC-Axis
• Two alternative implementations for data
  dissemination
• Connect to relational common macro MetadataBase
  and retrieve data from subject (dissemination)
  SQL databases
• Metadata stored in relational MetadataBase
• Connect to PC-Axis database comprised of PX
  files organized in a file catalogue structure
• Metadata stored in PX file
• Majority of PC-Axis users adhere to the second
  approach using PC-Axis files
• PC-Axis file database to be used as the basis for
  integration with the SDMX Reference Architecture

5
Context of the SDMX NSI Reference Architecture
NSI Dissemination DB
NSI
External
NSI SDMX Dissemination Environment
Data Consumer (ESTAT)
PC-Axis
SDMX Registry
NSI Production System
6
NSI SDMX Environment

• Provision of Web Service for serving SDMX-ML
  datasets given SDMX-ML Query
• The Web Service follows the SDMX guidelines for
  web services
• Message style (not RPC)
• Adopt wrapper operation names (implementation
  issue Java vs .NET)
• Provision of a feed for new data published (SODI)
• Store and manage metadata for mapping SDMX
  constructs to the local storage
• Query access to the local storage
• supported RDBMS vendors SQL Server, MySQL, Oracle
• PC-Axis database

7
NSI Web Service
NSI Web Service Provider
Web Service
SDMX Data Generator
Data Retriever
Query Parser
Mapping Assistant
PC-Axis DB
Mapping Store
SDMX Model
8
Whole architecture view
9
Interaction with PC Axis

• Mapping Assistant
• Downloads remote (FTP) PC Axis files
• Parses PC Axis files (for metadata and sample
  content data)
• Creates and stores mappings for specific files
• Data Retriever
• Reads stored mappings
• Downloads remote (FTP) PC Axis files
• Parses PC Axis files (for requested content)

10
Interaction with PC Axis (contd)
11
Proof of Concept Generation of temporary CSV
files

• Advantages
• Quick implementation for Madrid workshop
• Drawbacks
• Applications automatically convert PC-Axis files
  to local disk .csv files
• Large .csv files can be produced from a PC-Axis
  file (hard disk space consuming)
• Error prone (access rights on local drive,
  generated file locking)
• Next Steps
• Replace with a more appropriate method

12
1st Alternative Approach
Load Data
Custom Data ODBC Driver
Custom Data JDBC Driver
Metadata
In-memory database
In-memory database
Custom Metadata Driver
ODBC
JDBC
ODBC
ODBC
JDBC
Retrieve MetaData with SQL
Retrieve Data with SQL
Mapping Assistant
.Data Retriever
SDMX IS Applications
13
Evaluation of 1st alternative approach

• Advantages
• No need to write data to local disk
• Using already implemented ODBC,JDBC drivers (less
  bugs)
• Support for non-trivial SQL queries
• (i.e. nested SELECTs, aliases in FROM, UNION
  ALL, sub-string functions, concatenation
  functions)
• SQL execution performance
• Reasonable implementation effort

14
Evaluation of 1st alternative approach (contd)

• Drawbacks
• Memory-consuming (Size of PC-Axis files may be
  considerable)
• DB generation is required every time a query is
  performed (DB not stored in disk)
• Next Steps
• Decide which In-Memory RDBMS (H2, SQLLite, other)

15
2nd Alternative Approach
Fetch Data
Metadata
Custom Metadata Driver
Custom Data ODBC Driver
Custom Data JDBC Driver
ODBC
ODBC
JDBC
Retrieve MetaData with SQL
Retrieve Data with SQL
Mapping Assistant
.Data Retriever
SDMX IS Applications
16
Evaluation of 2nd alternative approach

• Advantages
• No need to write data to local disk
• Memory efficient retrieves only the data that is
  needed from PC-Axis files

17
Evaluation of 2nd alternative approach (contd)

• Drawbacks
• Lengthy implementation
• ODBC, JDBC drivers for PC-Axis files must be
  implemented from scratch
• Support for non-trivial SQL
• (i.e. nested SELECTs, aliases in FROM, UNION
  ALL, sub-string functions, concatenation
  functions)
• File streaming slower than database querying
• Next Steps
• Assess required effort