Generic Change Management: The SERF Project and Beyond

1
Generic Change ManagementThe SERF Project and
Beyond

• Elke A. Rundensteiner

In collaboration with students at WPI
including Kajal Claypool, Hong Su, Amy Lee, and
others.
Dept. of Computer Science Worcester
Polytechnic Institute
2
Nothing is as constant as change itself
Christina Baldwin
3
Motivation Change is Everywhere

• Persistent data everywhere!
• And it changes ...
• Why?
• Mistakes in database design or modeling
• Changes in real world (understanding of domain)
• Inherent in design and exploratory processes
• Addition of new applications to system
• Support for comprehensive and generic change
  management needed !!!

4
Issues in Change Management

• How to specify change?
• How to execute change?
• How to manage effect on dependent information?

DBAs
5
Specification of Change
6
Schema Evolution

• What?
• Changes to structure of class and class hierarchy
• Object transformations to conform to schem
  changes
• Why?
• Complex applications
• Requirements change over time

Root
Person
Employee
Student
7
Schema Evolution
add_attribute(address, Person)
Changes to structure of class and class hierarchy
add_attribute(address, Person)
add_attribute(address, Employee)
add_attribute(address, Student)
Object transformations to conform to schema
changes
8
Issues Specification of Change

• Traditional change
• Structural change and data change
• Schema evolution
• Pre-defined fixed operations Objectstore,Versant,
  Objectivity
• E.G. Add or delete of an attribute
• Pre-defined complex operations Breche96,
  Lerner96
• E.G. Merging two classes, split a class
• Not enough hard to meet all users needs...
• Goal 1 need flexible mechanism

9
Issues Specification of Change

• Many types of change
• Structural, data, constraint, rules, time, model
• Related work
• Schema evolution Objectstore94, Versant92,
  Objectivity93, Breche96, Lerner96, O2-94
• Constraint evolution Mayol99
• Time evolution Nascimento95
• Problem
• Individual similar solutions for each change type
  ER -Roddick99
• Goal 2 Need to different kinds of changes

10
Limitations of Current Evolution Technology

• lack of flexibility
• a fixed set of primitives
• lack of extensibility
• pre-determined semantics
• lack of re-usability
• ad-hoc programs for complex transformations

11
SERF Schema Evolution through an Extensible,
Re-usable and Flexible Framework
12
Goal Generic and Flexible Change Specification
Framework

• Flexible
• user or domain-specific transformation
• Extensible
• user can specify new semantics
• Reusable
• generic and not application-specific library
• Portable
• not tied to a particular system

13
SERFSchema Evolution Through an Extensible,
Re-usable and Flexible Framework

• Schema evolution framework
• Sits on top of existing OODB systems
• SERF transformation
• SERF template
• Name, parameters, generalized

combination of evolution primitives
specification of object transformations
OQL as the glue logic
14
State of Art in Schema Evolution

• Commercial databases
• O2, ObjectStore, Versant
• taxonomy of simple evolution operations
• e.g., add-attribute, delete-attribute,
  rename-attribute
• Research systems
• BrecheBre97, LernerLer96
• a fixed number of advanced primitives
• e.g., merge, inline, difference

15
SERF Approach - The Goals

• Flexibility
• define user or domain specific transformations
• Extensibility
• not a closed taxonomy of primitives
• a transformation language
• Re-usability
• generically applicable transformations
• re-usable library of transformations
• Based on standard (ODMG)

16
SERF Transformation

• SERF Transformation
• combination of evolution primitives
• specification of object transformations
• OQL as the glue logic
• SERF schema evolution framework sits on top of
  existing OODB systems.

17
Example of Inlining
18
Inline Transformation
// Add required attributes to Person
class add_attribute (Person, Street, String,
"") add_attribute (Person, City, String,
"") add_attribute (Person, State, String,
"") // Get all objects for Person class define
extents() as select c from Person c //
Update all objects for all obj in extents()
obj.set (obj.Street, valueOf(obj.address.Street))
obj.set (obj.City, valueOf(obj.address.City))
obj.set (obj.State,valueOf(obj.address.State))
// Delete the address attribute delete_attri
bute (Person, address)
19
SERF Template

• SERF Transformation
• Re-callable (Named)
• Generally Applicable (MetaData)
• Re-usable (Parameters)

20
Naming and Parameterizing a Transformation

• Re-callable
• named
• begin template inline (..)
• end template
• Parameterized
• input and output parameters
• (className, refAttr)
• Example of Instantiation
• inline (Person, address)

21
Generalizing a Transformation

• Remove dependency to specific classes by using
  metadata (ODMG System Repository)

define localAttrs(cName) as select
c.localAttrList from MetaClass c
where c.metaClassName cName for all attrs in
localAttrs(refClass) add_attribute
(className, attrs.attrName,
attrs.attrType, attrs.attrValue)
22
Inline Template
begin template inline (className, refAttrName)
refClass element ( select
a.attrType from MetaAttribute a
where a.attrName refAttrName
and a.classDefinedIn className
) define localAttrs(cName) as select
c.localAttrList from MetaClass c
where c.metaClassName cName for all attrs
in localAttrs(refClass) add_attribute
(className, attrs.attrName,
attrs.attrType, attrs.attrValue) define
extents(cName) as select c from cName c
for all obj in extents(className) for all
Attr in localAttrs(refClass) obj.set
(obj.Attr, valueOf(obj.refAttrName.Attr))
delete_attribute (className, refAttrName)
end template
23
Flow of a Template
inline(className, refAttrName)
Template
className Class Person, refAttrName
Attribute address
error
error
inline(Person, address)
Pure oql statements
error
pass
24
SERF Architecture
Template Manager
User Interface
Template Editor
Instantiates and executes
uses
Template Library
SERF Framework
uses
uses
uses
OODB System
Query Engine
uses
operates on
queries
queries
Object Repository
25
SERF Architecture
26
Advantages of SERF

• Flexible
• define semantics of choice
• Extensible
• define new transformations
• Re-usable
• define embedded transformations
• Portable
• applicable to different ODMG systems

27
Research Issues

• Optimization of Templates
• SE primitives in templates expensive
• Application of other language features
• exceptions
• Semantic consistency
• contracts
• Evolution of complete ODMG model
• relationships, keys, behavior

28
Conclusions

• Identify limitation of fixed SE primitives
• Developed SERF solution
• Transformations
• primitives OQL system update methods
• Templates
• generalized, named and parameterized
  transformations
• Library of Templates (from Case Study)
• ODMG based SERF prototype

29
Does it give us Consistency?
name
name
inline
street
Person
Person
city
address
street
state
Address
city
Department
Department
state
Office
Office
30
Consistency Responsibilities

• Who should be responsible for guaranteeing
  consistency?
• SERF Templates or
• Core Schema Evolution Primitives or
• End User or Application Developer

31
Concept of Contracts into Templates

• Mechanism for declaratively expressing
  constraints of schema in template
• System-enforced consistency guarantee
• Use for consistency of relationship evolution --gt
  ER2000
• Use for software upgrade itself for better
  maintainability --gt TOMORROW

32
Performance of such Complex Templates

• Schema evolution are expensive methods
• Evolution is embedded within multiple query
  statements
• CHOP ECOOP OODB workshop2000 on an optimizer
  for evolution sequences
• CHOP-OQL optimizer for embedded oql queries MS
  thesis, Ming Li, 2000

33
Supporting Other Applications
34
Issues Supporting Other Applications
Views

• Current approaches
• flexibility in structure of queries
  Lieberherr94
• Adapting queries using available redundant
  information EVE97
• Hiding change from other applications TOSE98
• Versioning Zdonik, Lauteman, Sommerville

Applications
?
View Evolution change
?
?
DBMS
Structure change
DBAs
35
Schema Evolution

• Examples
• Add-attribute capacity-augmenting (CA)
• Delete-attribute capacity-reducing (CR)
• Change-class-name capacity-equivalent (CE)

36
Application Interoperability ProblemI Deleting
Data
old
new
Schema
Data
37
Transparent Object Schema Evolution View-based
Solution
old
View
View
new
Schema
Data
38
Application Interoperability ProblemII Adding
Data
old
Data Added
Schema
Data
39
TOSE for Capacity-Augmenting Changes
old
View
View
View
Data Added
Schema
?
Data
?
40
Previous SolutionTransparent Schema Evolution

• For Capacity-Reducing/-Equivalent
• Derive target view schema from input view schema
  by e.g. hiding
• For Capacity-Augmenting
• Require capacity-augmenting views to store
  additional capacity

Capacity-Augmenting Views not (yet) Commercially
Available!
41
TOSE Three-Pronged Process for
Capacity-Augmenting Schema Changes
1. Base augmentation step (schema change) 2.
Target generation step (view) 3. Original base
restoration step (view)
Requires only conventional view- and schema
evolution tools!
42
Degenerate Process for
Capacity-Reducing/-Equivalent SCs
1. Base Augmentation Step (skip) 2. Target
Generation Step (from VS) 3. Original Base
Restoration Step (skip)
CR or CE SC
VS
VS
User-defined SC
BSBS
View Derivation
43
Update Semantics Preservation(USP)

• Given Users work on view schemas
• instead of on shared database
• Goal Provide base-like working environment
• Requires
• Updates on views translated into updates on base
  schema
• Back-propagation from base to views have same
  effects as if updates directly performed on views

44
USP Virtual Class

• Definition Update on virtual class (VC) has same
  effect as if VC were base class
• Mechanism Create, delete, and set update applied
  to VC is translated to its source class, and then
  propagates back to VC.
• Example Person is usp.

hide(income,Person)
name
Person
Person
name
income
(base class)
(hide virtual class)
45
USP Case Study

• Use MultiViews object algebra as basis
• Identify set of algebra operators generating
  virtual classes sufficient for TOSE, namely
  ident, hide, union, difference, intersect.
• Prove this set of view derivation operators to be
  USP.

46
USP Definitions

• Definition (VCi,VCj) is a usp pair, if (VCj,VCi)
  possesses typical update behavior as if they were
  base classes.
• Definition view schema VS is usp if every
  pair of classes in VS is a usp pair.
• Definition VD(Si) Si, is usp derivation if Si
  is guaranteed to be usp whenever Si is usp.

47
Capacity-Augmenting Schema Change Example
VS
VS
add-attr(birthday, Person)
Root
Root
name
Person
income
Person
birthday
Student
Student
Root
Dept
Student
BS
48
Step 1 Base Augmentation
VS
VS
add-attr(birthday, Person)
Root
Root
name
Person
income
Person
birthday
Student
Student
1
Root
Root
name
income
Person
Dept
Dept
Person
birthday
Student
Student
BS
BS
49
Step 2 Target Generation
VS
VS
add-attr(birthday, Person)
Root
Root
name
Person
income
Person
birthday
Student
Student
2
1
Root
Root
name
income
Person
Dept
Dept
Person
birthday
Student
Student
BS
BS
50
Step 3 Original Base Restoration
VS
VS
add-attr(birthday, Person)
Root
Root
name
Person
income
Person
birthday
Student
Student
2
1
Root
Root
name
income
Person
Dept
Dept
birthday
3
Student
Student
BS
BS
51
Capacity-Reducing Schema Change Example
del-attr(income, Person)
Root
Root
Person
Person
name
Student
Student
VS
VS
VS VDdel-attr(income,Person)(VS(BS))
52
Capacity-Reducing Schema Change Example
del-attr(income, Person)
Root
Root
Person
Person
name
hide(income, Person)
Student
Student
VS
hide(income, Student)
VS
VS VDdel-attr(income,Person)(VS(BS))
53
TOSE Transparency Theorem

• Study complete taxonomy of schema evolution
  primitives
• If capacity-reducing or -equivalent give
  corresponding usp view derivation
• If capacity-augmenting transformation a.
  give three steps of process
  b. prove their correctness

54
Advantages of TOSE

• Applications can interoperate on same set of
  objects flexibly
• Shared schema can evolve without affecting
  existing applications (migration)
• Data consistency is guaranteed, since there is
  only one copy of base objects
• TOSE relies on standard tools only

55
TOSE Implementation

• SPARCstation 10
• GemStone 4.0
• SunOS 4.1.3

TOSE
MultiView View System
MultiView Object Model
GemStone
56
Related Work

• Banerjee et al. regular in-place schema changes
• Breche et al. / Ra et al. rely on CA views
• Tresch et al. tackle CR/CE changes only
• Lautemann use versioning approach
• Kaushal et al. handle schema changes from base
• Claypool et al. define complex change
  operations using primitive change operations

57
Conclusions

• Development of three-step methodology to achieve
  schema changes transparently
• Identification of update semantics preservation
  requirement of view schema case study
• Formalization of the TOSE methodology
• Development of TOSE algorithms for standard
  taxonomy of SE primitives
• Feasibility demonstration via prototype

58
Next Research Tasks

• Propagate SERF Transformations through Views
• Expand work to complex views (usp?)
• Apply TOSE to relational databases (e.g.,
  object-views by Oracle-8)
• Extend to distributed environments modeling
  management technology

59
Research Direction

• Model the Mapping between two Systems as
  First-Class Citizen
• Model Models as First-Class Citizens
• Support Operations on Mapping and Models
  themselves Change Propagation is one such
  operator

60
Issues Execution of Change
Views
Applications
Structure change

• Base change - process somewhat understood.
• View change - limited work done.

Constraint change
Web Pages
DBMS
61
Execution of View ChangeRelated Work

• Schema change
• Object-preserving OO views Rundensteiner98,
  Bertino96
• Data change Keller82, Scholl91, Bancilhon81
  ,EBG97, Dayal89
• Other types of changes
• Meta-data Sciore94
• Security Rosenthal97
• Problem
• Individual similar solutions for each change type
  

62
Change by Propagation Process
U
V(DB)
U(V(DB)) V(DB)
T
V
V
T(U)
DB
T(U)(DB) DB

• Common process
• Find the right source
• Translate the change and apply it
• Re-write the view/propagate the change back up

63
Goals

• Goal 1 Framework that is
• Extensible.
• Propagation of different information.
• Flexible.
• Propagation algorithms can be altered easily.
• Portable.
• Applicable for heterogeneous sources and views.
• Goal 2 Apply for evolving object-generating
  views.

64
PURE - Propagation Rule Framework

• Rule-based
• Handle different information types
• Proposal PR-OQL
• rule extension for OQL
• Object-oriented
• easier application to heterogeneous sources and
  views
• Middle-layer
• Propagate from source to view, view to source
• applicable in multi-tier environment

65
PURE Architecture
66
Goal 2 - Evolution of Object-Generating Views

• Existing work Rundensteiner98, Bertino96
• evolution of select-project views
• Complex applications (e-commerce, design)
  Abiteboul99, Peckham95
• views are often join views
• Evolution more complicated than select-project
  views!

67
Join Example
JOINCLASS
JOINCLASS'
AdvisorEmail studentName grade
AdvisorEmail studentName grade dateOfGrad
Add-attribute(JOINCLASS,
dateOfGrad, DATE,
"1999")
select a.AdvisorEmail, s.studentName,
s.grade from Advisor a, Student s where
a.advisorName s.advisorName
Derive/ up propagation
Propagate
J
Advisor
Student'
AdvisorName AdvisorEmail phoneNumber
studentName advisorName grade dateOfGrad
Translate /Apply
Add-attribute(Student,
dateOfGrad, DATE,
"1999")
68
Model Management

• Goal 1 Framework that is
• Extensible.
• Propagation of different information.
• Flexible.
• Propagation algorithms can be altered easily.
• Portable.
• Applicable for heterogeneous sources and views.
• Goal 2 Apply for evolving object-generating
  views.

69
PURE - Propagation Rule Framework

• Rule-based
• Handle different information types
• Proposal PR-OQL
• rule extension for OQL
• Object-oriented
• easier application to heterogeneous sources and
  views
• Middle-layer
• Propagate from source to view, view to source
• applicable in multi-tier environment

70
PURE Architecture
71
Goal 2 - Evolution of Object-Generating Views

• Existing work Rundensteiner98, Bertino96
• evolution of select-project views
• Complex applications (e-commerce, design)
  Abiteboul99, Peckham95
• views are often join views
• Evolution more complicated than select-project
  views!

72
Join Example
JOINCLASS
JOINCLASS'
AdvisorEmail studentName grade
AdvisorEmail studentName grade dateOfGrad
Add-attribute(JOINCLASS,
dateOfGrad, DATE,
"1999")
select a.AdvisorEmail, s.studentName,
s.grade from Advisor a, Student s where
a.advisorName s.advisorName
Derive/ up propagation
Propagate
J
Advisor
Student'
AdvisorName AdvisorEmail phoneNumber
studentName advisorName grade dateOfGrad
Translate /Apply
Add-attribute(Student,
dateOfGrad, DATE,
"1999")
73
Finding the right Source

• Left or right outer join
• Key
• Meta-properties Rosenthal94
• allow user to specify dependencies at evolution
  time
• E.G. dateOfGrad ltfunctionalDependentgt on
  studentName

74
Relationship Example

• Complex changes require
• finding the right source
• finding the right view
• E.G add a relationship between
• two views

Example 2
75
Tasks

• Propagation Re-write algorithms
• Equivalence Model
• rigid equivalence too strict
• user preference model for equivalence with
  threshold equivalence
• Cost Model to evaluate the choices

76
Evaluation and Validation

• Implementation using PURE
• Validation
• Show algorithms are
• Equivalence preserving
• Confluent and terminating
• Update semantics preserving
• Compare cost model with experimental results

77
Applications of SERF
78
Re-WEB - Re-Usable Templates for Web View
Generation and Restructuring

• WIDM98 Workshop, Kajal T. Claypool, Elke A.
  Rundensteiner, Li Chen, Bhupesh Kothari
• WWW Journal 2000, Li Chen, Kajal Claypool, and
  Elke A. Rundensteiner

Dept. of Computer Science Worcester Polytechnic
Institute
Funded in part by NSF, IBM and ODI Inc.
79
Motivation Re-WEB - Re-Usable Templates for Web
View Generation and Restructuring

• The Digital Age!
• web-sites - a large data repository
• Generating web pages from database
• exploit database and optimization technology
• easier to manage dynamic generation
• easier to propagate changes
• Issue
• Diverse web-site representations needed

80
Re-WEB Approach Internal Views
Html / XML
Html/ XML
Web Mapping semantics
SERF Templates
View classes
81
Re-WEB Approach (contd.)

• Advantages
• no new web language required for restructure,
  since re-use OQL
• change management is reduced to propagating
  changes to views
• apply SERF (re-usable re-structuring templates)
• Disadvantages
• define view schema in database

82
Re-WEB Re-using SERF
SERF Template
Original schema
New schema
Original Web Page View
New Web Page View
83
Original Schema Structure
Obj4
Obj1
Obj5
Obj2
Obj6
Obj3
Data
84
Original Web Pages
Course Web Pages
Professor Web Pages
85
To Get Desired Web Pages ?
86
Re-structuring of DB Convert-To-Literal
SERF Transformation
create_view_class (Course, CourseView,
select c from Course
c) create_view_struct (Professor,
ProfessorStruct) add_attribute(CourseView,
struct-taughtBy, set, null) for all objects in
taughtBy flatten to struct of type
struct-taughtBy delete_attribute(CourseView,
taughtBy)
87
begin template convert-to-literal (mainclassName,
mainViewName, attributeToFlatten, structName)
refClass element( select
a.attrType from MetaAttribute a
where a.attrName
attributeToFlatten and
a.classDefinedIn mainclassName ) define
ViewDef ( viewClass ) select c from
viewClass c create_view_class (
mainclassName, mainViewName, View (
mainclassName )) create_view_struct (
classToFlatten, structName) add_attribute
( mainViewName, structName, collectionltstructNa
megt, null) define Extents (cName)
select c from cName c define
flattenedCollection( object ) select
(structName)p. from refClass p
where exists( p in p.attributetoFlatten )
for all obj in Extents( mainViewName )
obj.set(obj.structName, flattenedCollection(obj))
delete_attribute( mainViewName,
attributeToFlatten)
88
Advantages of Re-WEB

• Web views are consistent with one another
• change management is easier (database)
• One step generation of web site structure
• Easy to generate diversely restructured web-sites
• Re-usable library of Re-WEB templates
• Uses standard technology ODMG,OQL,XML

89
More Information.
http//davis.wpi.edu/dsrg/OOSE/SERF
90
Some project publications

• SERF Framework
• (Claypool, Jin and Rundensteiner 1998) CIKM,
  journal in submission
• SERF Prototype
• (Rundensteiner, Claypool, Chen, Koenoki et al.)
  SIGMOD99, CASCON99
• Validation
• Web site re-structuring
• (Rundensteiner, Claypool, Chen et al.) SIGMOD00,
  WWWJournal00
• Object model evolution
• (Claypool, Rundensteiner and Heineman) ER2000,
  TR99-15, TR00-09
• Other Relevant Work
• Consistency management in SERF
• (Claypool and Rundensteiner) TR99-21
• Optimization of schema evolution sequences
• (Claypool, Natarajan and Rundensteiner)
  ECOOP-WS00

91
(No Transcript)