Design and Users

1
Design and Users Evaluation of a Topic Map-Based
Korean Folk Music (Pansori) Retrieval System

• Sam Oh
• SungKyunKwan University, Seoul Korea
• samoh_at_skku.edu
• Oknam Park
• University of Washington
• parko_at_u.washington.edu

2
Outline

• Research Questions
• Related Works
• Research Design
• Samples and Variables
• Search Task Types
• Modeling Korean Folk Music Pansori
• Using Polygons
• Using UML
• Two Retrieval Systems
• TM Pansori Retrieval System (TMPRS)
• Current Pansori Retrieval System (CPRS)
• Research Results
• Conclusion

3
Research Questions

• Are there objective performance differences
  between TMPRS (Topic Map Pansori Retrieval
  System) and CPRS (Current Pansori Retrieval
  System)?
• Are there subjective performance differences
  between TMRPS and CPRS?

4
Related Works

• Guo et al. (2004)
• Evaluated four OWL-based systems
• Query response time, Search completeness,
  Soundness ( of the answers for each query)
• Kim (2005)
• Compared an ontology based system to a free text
  system
• 10 domain experts and 20 queries. Search time and
  relevance.
• An ontology system A better precision and less
  search time
• Sure and Losif (2002)
• Compared two ontology based systems to a free
  text system
• An ontology system Fewer mistakes and less time

5
Limitations of Related Works

• Relatively only a few evaluation studies of
  ontology based systems
• Few studies applying diverse task types
• Limited objective measurements
• Limited evaluation of ontology based systems vs.
  free text system
• No user study of Topic Map-based systems

6
Research Design

• Subject Samples
• Twenty LIS Students in Korea
• Repeated Measure
• Canned Queries 7 different search tasks.
• Their own query
• Preventing order effects
• 10 subjects searched TMPRS first, then CPRS
• 10 subjects searched CPRS first, then TMPRS
• Questionnaire/Screen Recording/Observation
  Note-Taking

7
Research Variables
Independent Variables
Dependent Variables
Conceptual Level

1. Objective Measurement
2. Subjective Measurement

Two Retrieval Systems
Operational Level

1. Topic Map-Based Pansori Retrieval System TMPRS
2. Current Pansori Retrieval System CPRS

1. Search steps, Search Time
2. Completeness, Ease of use, Efficiency,
   Appropriateness, Users satisfaction

Controlled Variables

1. Search Tasks
2. Subjects

8
Pansori Terms Explained

• Pansori (???)
• A type of Korean Folk Music
• Dae-Mok (??)
• A special part of a pansori (A long song is a
  part of the pansori ChunHyanGa)
• Yoo-Pa/Je (??/?)
• Four types of Korean Folk Music - Pansori
  (Dong-Pyon, Seo-Pyon, Chung-Go, Kang-San)
• Myung-Chang (??)
• A person who is well-known singer of Pansori
• Go-Soo (??)
• A person who has expertise in playing Korean drums

9
Pansori Terms Explained

• Dunum (??)
• Famous Myung-Chang Je A special part of
  Pansori (E.g,Kwon, Sam Duk-Je-A love song)
• Ba-Di (??)
• Famous Myung-Chang Je Pansori (E.g,
  E.g,Kwon, Sam Duk-Je-ChunHyanGa)
• Jo (?)
• Pansori Melody
• Jang-Dan (??)
• A special kind of Pansori rhythm
• Sa-Seol (??)
• A form of lyric written for Pansori

10
Search Task Types
Task Group 1 Simple Task Search for information about Jang-Dan (??)
Task Group 2 Complex Task 1 Search for Myung-Chang and works of Dong-Pyon-Je (???)
Task Group 3 Complex Task 2 Search for the birth year for SoHee Kim (???)
Task Group 4 Hierarchical Relationship Task Search for hierarchical category related to Seo-Pyon-Je (???)
11
Search Task Types ...
Task Group 5 Association and Cross Reference related Task 1 Search for generation which NokJu Park (???) belongs to, and find three Myung-Changs in the same generation
Task Group 6 Association and Cross Reference related Task 2 Search for a famous Myung-Changs for a Je-Bi (????? ??? ??) Dae-Mok and find the birth place for that Myung-Chang.
Task Group 7 User Own Query Search for information in your own interest area.
12
Modeling Korean Folk Music - Pansori
13
Topic Map Modeling of Pansori
Soonchang(Region)
KangSan-Je (Yoo-Pa)
Present (Genealogy)
Famous in
Belongs to
Classified as
Man-Jung ChunHyang Editorials (Sa-Seol)
Has editorials
Is a member of
ChunHyang-Ga(Pansori)
Park, Nokju
Has-Teachers
Cho, Sanghyun(Myung-Chang)
Composed-By
Shin, Jaehyo (Composer)
Well-Known for
Song, Kwangrok
Consists of
Is a dunum of
Singer-Part
Mathes with
Singer-Tone
Played by
Is a body of
Part-Rhythm
An, Suksun's Love Song (Dunum)
Part-Tone
Master of
Kim, Myunghwan (Go-Soo)
Jung-Mo-Li (Jang-Dan)
SeolRyong-Ge(Jo)
Han, Aesoon's Chunghyang-Ga (Ba-Di)
Sarang-Ga (Dae-Mok)
14
Pansori TM Association Types
member of
Genealogy
Myung-Chang
????/??? ??
Has-Teachers
Myung-Chang
??/??
Belongs to
Myung-Chang
Yoo-Pa
?? ??/??? ??
Singer-Tone
Myung-Chang
Jo
???/?? ????
Singer-Part
Dae-Mok
Myung-Chang
????/??? ????
Well-known for
Myung-Chang
Pansori
??????/????????
15
Pansori TM Association Types
Played by
Pansori
Go-Soo
????/???????
Contains Bodies
Pansori
Ba-Di
?? ??/?? ???
Contains Dunums
Pansori
Dunum
?? ??/?? ???
Composed by
Composer
Pansori
???/?? ???
Has Editorials
Sa-Seol
Pansori
????/??????
Famous in
Region
Pansori
????/????????
Classified as
Yoo-Pa
Pansori
?????/????????
Pansori-Rhythm
Pansori
Jang-Dan
???????/????????
Consists of
Pansori
Dae-Mok
????/??? ???
16
Pansori TM Association Types
Part-Rhythm
Dae-Mok
Jangdan
?? ????/??? ????
Part-Tone
Dae-Mok
Jo
?? ???/?? ????
Master of
Go-Soo
Jangdan
?? ????/?? ????
17
Pansori Modeling and Occurrences
Soonchang(Region)
KangSan-Je (Yoo-Pa)
Present (Genealogy)
Famous in
Belongs to
Classified as
Man-Jung ChunHyang Editorials (Sa-Seol)
Has editorials
Is a member of
ChunHyang-Ga(Pansori)
Park, Nokju
Has Teachers
Composed-By
Shin, Jaehyo (Composer)
Well-Known for
Song, Kwangrok
Consists of
Is a dunum of
Singer-Part
Mathes with
Singer-Tone
Played by
Is a body of
Part-Rhythm
Part-Tone
Master of
Kim, Myunghwan (Go-Soo)
Jung-Mo-Li (Jang-Dan)
SeolRyong-Ge(Jo)
Sarang-Ga(Dae-Mok)
18
Pansori Occurrence Types

• Myung-Chang, Go-Soo, Composer
• Description(??), Real Name(??), Pen Name(?), Nick
  Name(??), Birth Place(???), Date of Birth(????),
  Active Period (????), Homepage (????),
  Albums(??), Sound(????), Image(??), Video(???),
  Paper(??), Articles(??), Critique (??), Book(?)
• Pansori
• Description(??), Contents(??), Work
  Sturcture(??), Paper(??), Articles(??), Critique
  (??), Book(?), Editorials(??), Product
  Year(????), Albums(??), Sound(????),
  Website(????)
• Dae-Mok
• Description(??), Contents(??), Paper(??),
  Articles(??), Critique (??), Albums(??),
  Sound(????)
• Jangdan, Jo, Dunum, Ba-Di
• Description(??), Albums(??), Sound(????)
• Yoo-Pa
• Description(??), Paper(??), Articles(??),
  Critique (??), Book(?)

19
Pansori Occurrence Types Displayed
Soonchang(Region)
KangSan-Je (Sect)
Present (Genealogy)
Famous in
Belongs to
Real name
Pen name
Classified as
Man-Jung ChunHyang Editorials (???)
Has editorials
Is a member of
Biography
Nick name
ChunHyang-Ga(Pansori)
Birthplace
Activity year
Park, Nokju
Has Teachers
Composed-By
Shin, Jaehyo (Composer)
Position
Date of birth
Exponent of
Website
Image
Song, Kwangrok
Consists of
Sound
Is a dunum of
Album
Singer-Part
Mathes with
Production year
Singer-Tone
Played by
Description
Is a body of
Part-Rhythm
Contents
Video
Part-Tone
Tambour-Rhythm
Genealogy
Book
Kim, Myunghwan (Tambour)
Jung-Mo-Li (Rhythm)
Article
Structure
SeolRyong-Ge
Sarang-Ga
Critique
Paper
20
UML RENDERING of PANSORI MODELING
21
TM vs. ER/UML Modeling

• No need to lose meaningful relationships captured
  by UML and ER modeling
• It may lead to better performance for navigation
  and retrieval of information
• For a database designer, less time and effort in
  changing schemas
• Ability to implement complex relationships
  explicitly and use them for retrieval

22
Two Retrieval Systems Compared
TMPRS (Topic Map Pansori Retrieval System) -
Authors (TM Modeling) and INEK (The Leading DL
Vendor in Korea, Web Implementation)vs. CPRS
(Current Pansori Retreival System)- Current
Pansori Website (Widely Used)
23
TMPRS Topic Types Example
24
TMPRS Search Example
25
CPRS Top Categories
26
CPRS Search Example
27
Research Results

• Objective Measurements
• Subjective Measurements

28
Subjective Measurement (TMPRS - CPRS) Normal
Distribution
Measure\ Task T1 T2 T3 T4 T5 T6 T7 Overall
Completeness Mean -0.15 0.85 1.1 1.55 0.15 1.9 -1.55 1.25
Completeness SD 1.42 1.42 1.37 1.82 1.67 2.35 3.08 1.68
Ease of Use Mean 0.4 1.4 2.5 1.35 1.31 3.15 0 2.45
Ease of Use SD 1.98 1.72 1.5 1.84 2.26 1.69 3.41 1.53
Efficiency Mean 0.15 1.45 2.85 1.35 1.36 3.6 -0.55 2.2
Efficiency SD 1.81 1.73 2.03 1.42 2.26 1.81 3.77 1.85
Appropriateness Mean -0.05 1.5 2.7 1.7 1.26 3.5 -0.35 1.75
Appropriateness SD 2.11 1.60 1.65 1.49 2.35 1.82 3.77 2.02
Satisfaction   Mean 0.6 1.8 2.8 1.85 1.31 3.4 -0.05 1.85
Satisfaction   SD 2.32 1.70 1.76 1.63 2.35 2.11 3.36 1.95
Table mean differences between two systems
their standard deviations
29
Subjective Measurement (Wilcoxs Singed Ranked
Test - S Value)
Measure\ Task T1 T2 T3 T4 T5 T6 T7 Overall
Completeness 0.75 0.021 0.003 0.001 0.647 0.003 0.052 0.003
Ease of Use 0.427 0.003 lt.0001 0.003 0.023 lt.0001 0.993 lt.0001
Efficiency 0.703 0.001 lt.0001 0.001 0.033 lt.0001 0.606 0.001
Appropriateness 0.851 0.001 lt.0001 0.001 0.042 lt.0001 0.702 0.001
Satisfaction 0.247 0.001 lt.0001 0.001 0.031 lt.0001 0.839 0.001
Table S value
30
Subjective Measurement (TMPRS - CPRS)

• Task 2 through Task 6 TMPRS is significantly
  better than CPRS in general.
• Ease of use and satisfaction TMPRS is
  significantly better than CPRS for task 1.
• Completeness TRMPS is significantly better than
  CPRS for task 5.
• Task 3 and Task 6 show better performance for
  TMPRS than Task 2 and Task 5.
• No significant difference for Task 1 and Task 7
  (Simple Query and User Queries)
• TMPRS is significantly better than CPRS for
  complex tasks than simple tasks
• Limitation of the findings
• Domain users are not employed and lack of diverse
  user groups so limited generalization

31
Objective Measurement(TMPRS - CPRS)
Measures/Task Measures/Task T1 T2 T3 T4 T5 T6 T7
Time Mean -5.15 24.47 58.78 31.15 21.16 135.10 11.63
Time SD 12.65 42.31 62.14 48.99 57.01 140.86 50.45
Search Steps Mean -3.94 1.421 3.789 0.78 2.88 7.52 0.15
Search Steps SD 3.51 4.658 3.40 2.61 4.58 8.75 3.67
32
Objective Measurement(TMPRS - CPRS)
Measures/Task T1 T2 T3 T4 T5 T6 T7
Time t 0.092 S 0.004 S lt.0001 t 0.0126 S 0.031 S lt.0001 S 0.3549
Search Steps S lt.0001 S 0.430 S 0.0001 S 0.302 S 0.019 S 0.0002 S 0.9119
33
Objective Measurement(TMPRS - CPRS)

• Task 2, 3, 4, 5, 6 Less time in TMPRS
• Task 3, 5, 6 Less steps taken in TMPRS
• Task 1 Less steps taken in CPRS
• Task 3 and Task 6 show better performance for
  TMPRS than Task 2 and Task 5
• TMPRS is significantly better than CPRS for
  complex tasks than simple tasks

34
Users Reaction

• CPRS
• Related information not found in the structure
• Hard to guess where information is in
  hierarchical relationships
• Took more time and energy
• Fragmented or Not related
• Required to combine fragmented information from
  different places of the site
• Needed to read a long text for information
• Not appropriate for linked and complex
  information
• Serendipitous findings are rare

35
Users Reaction

• TMPRS
• Fewer clicks and browsing required
• More specific and detailed information structure
  are provided
• Could find more information in one page
• Related information at one page
• More information, Flexible,
  Well-structured, or Easy to find related
  information
• Serendipitous findings are well-supported
• No need to read long text
• Gave pictures of domain knowledge conceptually
  with much linked categories

36
Conclusion

• TMPRS showed Higher performance for objective and
  subjective measurements in general
• TMPRS was good for more complex task
• TMPRS was much better than CPRS for task 2
  through 6
• No difference between task 1 and user queries
• Most user own queries were very simple (similar
  to task 1)

37
Conclusion

• Task 3 and Task 6 show better performance for
  TMPRS than Task 2 and Task 5
• Users felt TMPRS more flexible, more related and
  well structured
• Future Aspects
• Diver User Study needed
• Task 2 and Task 5 (Association Include) and Task
  3 and Task 6 (Occurrence Include) Generalization
  Study
• Other Domain Study

38
Q Asamoh_at_skku.eduparko_at_u.washington.edu