Introduction to Data Warehousing and Data Landscaping Concepts

1
Introduction toData Warehousing and Data
Landscaping Concepts

• Project Management
• Night School
• December 18, 2001
• Presented by
• Qwest Interactive, Eastern Region Technical
  Services Department

2
Agenda

• Overview of an Enterprise Data Landscape
• What is a Data Landscape?
• What is a Data Warehouse?
• Data Lifecycle Management
• Project Management Issues for Data Warehousing
• Fundamentals of Data Modeling
• Example Application Demonstration

3
Overview of anEnterprise Data Landscape
4
What is a Data Landscape?

• Data
• Factual information, especially information
  organized for analysis or used to reason or make
  decisions.
• Computer Science. Numerical or other information
  represented in a form suitable for processing by
  computer.
• Landscape
• An expanse of scenery that can be seen in a
  single view a desert landscape.
• A picture depicting an expanse of scenery.
• An extensive mental view an interior prospect
  They occupy the whole landscape of my thought
  (James Thurber).

5
Sample Enterprise Data Landscape
Data Mining Engine
Processing Transformation
Web Application
Enterprise Data Warehouse
Exploration Warehouse
Transactional Database
Transactional Database
Reporting Applications
Transactional Database
Operational Data Store
Feedback Loop (After Data Reconciliation)
6
What is a Data Warehouse?

• Data Warehouse
• A generic term for a system for storing,
  retrieving and managing large amounts of any type
  of data. Data warehouse software often includes
  sophisticated compression and hashing techniques
  for fast searches, as well as advanced filtering.
• A database, often remote, containing recent
  snapshots of corporate data. Planners and
  researchers can use this database freely without
  worrying about slowing down day-to-day operations
  of the production database.

7
Transactional Systems vs. A Data Warehouse

• A Transactional System is an application and
  database that supports a specific business
  function through transaction oriented data.
• Asset Trade Transactions
• Sale Transactions
• Etc
• A Data Warehouse has a slightly different goal
  to facilitate the process of decision support.

8
Transactional System
Transactional Database

• Transactional Systems
• Support Specific Business Processes
• Traditionally Operate in Silos
• Can be legacy apps, custom apps
• or product implementations

9
Basic Warehouse System
Transactional Database
Transactional Database
Transactional Database
Data Warehouse

• Data Warehouses
• Import Data from
• Transactional Systems
• Provide Consolidated Data Access
• Across Silos

10
Segregating Data for Specific Uses
Data Mart
Data Mart
Data Mart

• Data Marts
• Allow for Functional Segregation
• of Warehouse Data
• Provide Focused Data Sets
• Optimized for Reporting

Data Mart
11
The Arrow ETL Processing

• ETL Processing
• Extraction
• Transformation
• Loading

12
Why Develop DifferentDatabase Systems?

• Provides an Approach for Business and Technical
  Risk Mitigation
• Does not Hamper the Stability of Business
  Critical Systems
• Optimized for Analysis and Querying
• Allows for Data Transformation
• Allows for Data Integration

13
ETL Processing

• Extraction Pulling data from its source system.
• May come from mainframe flat files, relational
  databases, web logs, etc
• Transformation Mapping data types, formats and
  structures to a common warehouse format.
• I.E. Changing date formatting from M/D/YY to
  MM/DD/YYYY
• Loading Placing the data into the warehouse.
• Mapping from the source schema to the target
  schema.

14
Data Lifecycle Management
15
What is a Data Lifecycle?

• A Strategy that Encompasses Each of
• the Following Traits
• Ownership
• Acquisition
• Storage
• Summarization
• Expiration

16
Example Data Lifecycle
17
Levels of Data Ownership
18
Project Management Issuesin Data Warehousing
19
Political Environment

• Warehouse projects cross organizational borders.
• Not viewed as revenue streams (ROI difficult to
  quantitatively describe).
• Complex relationships increase project risk and
  require enhanced communications.

Dept. A
Web Application
Dept. B
Dept. C
Transactional Database
20
Projects Need to Become Disciplines,Not be
One-Off Projects
21
Drive RequirementsThrough Business Goals
Corporate Mission
Stakeholder Objectives
Key Decisions
Information Requirements
Data Requirements
22
Fundamentals ofData Modeling
23
Entity Relationship (ER) Modeling
24
Warehouse Modeling
25
Application Demonstration