OECD Expert Group on Statistical Data and Metadata Exchange

1
OECD Expert Group on Statistical Data and
Metadata Exchange

• Geneva 6-7 April
• OECD.Stat SDMX Web Service
• Jens Dossé, OECD

2
Vision for future design of international
Statistical Data Metadata eXchange
2. Get query parameters for requesting annual
data for GDP for France for 2002from any
international sources
3. There are two registry entries one form OECD
(data in ), and one from Eurostat (data in ).
Here are the query parameters
5. Query datasetMEI with SUBGDP, COUFRA,
FREQA, TIME2002 and MEAUSD
4. ltSDMXQuerygtGDP France Annual 2002
lt/SDMXQuerygt
5. Query GDP, France, Annual, 2002,

• OECD.Stat

6. 88888.88
6. 99999.99
7. ltSDMXgt 88888.88lt/SDMXgt
1. Query GDP, France, Annual, 2002,from any
international sources
8. ltDATA OECDgt99999.99lt/DATAgt ltDATA
EUROSTATgt88888.88lt/DATAgt

• OECD

• other orgs

3
OECD SDMX web service

• Conforms to SDMX standards v1 and v2
• Currently one method to extract data in
  GenericData message
• Currently developing methods to extract Data
  Structure Definitions (DSD), concepts and code
  lists
• Secured access integrates authorization to
  OECD.Stat
• Needed additional conventions bridging current
  SDMX-ML limits

4
Additional conventions I

• For time series data, independently of the
  presence of a FREQUENCY dimension, the requested
  time frequency must be specified using a
  TIME_FORMAT attribute ltqueryAndgt ltqueryAtt
  ribute name"TIME_FORMAT"gtP1Mlt/queryAttributegt
  ltqueryTimegt ltqueryStartTimegt1960-01lt/querySt
  artTimegt ltqueryEndTimegt2005-12lt/queryEndTimegt
  lt/queryTimegt lt/queryAndgtFormat values
  P1Yannual, P6Msemi-annual, P3Mquarterly and
  P1Mmonthly data Time formats yyyy, yyyy-mm or
  yyyy-mm-dd with digits yyear, mmonth
  and dday, e.g. '2005-09-01'. This attribute is
  then also present in the resulting generic data
  message at the Series level of time-series
  data.
• Non-time-series data omit all time
  specifications in the query message The
  response messages will use 9999 as (obligatory)
  time periods
• If the ltqueryEndTimegt node is not present then
  currently only the values for the
  ltqueryStartTimegt are returned

5
Additional conventions II

• OECD data warehouse (OECD.Stat) data model
• Not based on the SDMX 'Key Family' model
• But implemented as a number of multidimensional
  datasets based on a set of common and non-common
  dimensions
• Therefore, applied back to the SDMX structure,
  each OECD dataset has its own 'Key Family' (Data
  Structure Definition)

6
Additional conventions III

• Several datasets within one message
• Query message can request data from several
  datasets
• But SDMX standard still disallows returning
  several datasets in GenericData message
• Therefore, the effective dataset codes are
• not specified in the datasetID attribute of the
  ltmessageDataSetgt node
• neither in the ltgenericKeyFamilyRefgt node
• but as DataSet concepts in the
  ltgenericAttributesgt node of each
  ltgenericSeriesgt nodeltDataSetgt ltgenericKeyFami
  lyRef gtnonelt/genericKeyFamilyRefgt ltgenericSerie
  sgt ltgenericSeriesKeygt ltgenericValue
  concept"COU" value"AUS"/gt ltgenericValue
  concept"SUB" value"EVOPOP_T1"/gt lt/genericSeri
  esKeygt ltgenericAttributesgt ltgenericValue
  concept"DataSet" value"CSP5"/gt ltgenericValue
  concept"TIME_FORMAT" value"P1Y"/gt lt/genericA
  ttributesgt ltgenericObsgt ltgenericTimegt2000lt/
  genericTimegt ltgenericObsValue
  value"19153"/gt lt/genericObsgt lt/genericSeries
  gtlt/DataSetgt

7
Additional conventions IV

• For simplicity during the start-up of this
  generic web service, the returned message headers
  only contain required information
• The required ltmessageIDgt item currently only
  contains a placeholder ltHeadergt ltIDgtnonelt/ID
  gt ltTestgtfalselt/Testgt ltTruncatedgtfalselt/Truncat
  edgt ltPreparedgt2006-03-29T115140lt/Preparedgt lt
  Sender id"OECD"gt ltName xmllang"en"gtOrganisat
  ion for Economic Co-operation and
  Developmentlt/Namegt ltName xmllang"fr"gtOrganisa
  tion de coopération et de développement
  économiqueslt/Namegt lt/Sendergt lt/Headergt

8
Additional conventions V

• Maximum size of generic data message 5 MB
• Message is truncated if necessary
• Requests need to be split if this limit is
  exceeded

9
Future challenges about SDMX web services

• Big message sizes for any data format need
  standard for compression
• Security need standard for authentication/encrypt
  ion
• Need for another message format generic
  cross-sectional format ? remove the special
  status of the Time dimension (as not all
  statistical data are time series)
• Registering SDMX web services in the SDMX
  registry
• Multilaterally connecting organisations SDMX web
  services to build international networks

10
OECD SDMX web service client example

• Simple web application
• Pre-filled with example query message
• Source code freely available
• Secured access to all data in data warehouse
  (integrates authentication and authorization to
  OECD.Stat)

11
Contacts/Links

• jens.dosse_at_oecd.orgdouglas.paterson_at_oecd.org
• http//stats.oecd.org/OECDStatWS_SDMX/SDMX.asmx
• http//stats.oecd.org/OECDStatWS_SDMX/Test_SDMX.as
  px