Towards Evaluation of P2P-based DKMS

1
Towards Evaluation of P2P-based DKMS

• AMKM-2003
• Christoph Tempich, M. Ehrig, C. Schmitz, S.
  Staab, J. Tane
• University Karlsruhe

2
Agenda

• Introduction
• Evaluation process
• Test data
• Current status and future work
• Conclusion

3
Introduction

• Distributed Knowledge Management
• No. of Conceptualizations
• No. of Documents
• No. of Domains

Evaluation of algorithms necessary
4
Evaluation Input
Peers, Documents,nm Ontologies

• 1m Ontologies
• disjoint
• overlapping
• 11 Ontologies

• nm Ontologies
• n1 Ontologies

• Representation in RDF(S) / OWL

5
Evaluated Methods

• Alignment of ontologies
• Merging of ontologies
• Creation of ontologies
• Semantic routing

6
Evaluation results

• Relevance
• Precision
• Recall
• No. of Hops
• Network load
• Reliability
• Information loss

Ontology specific
7
Test data
Corpus name no. doc text type
Reuters 21678 21.578 yes classification
DMOZ 190.000.000 yes hierarchical
Medline 1.141.893 no hierarchical ontology
self parameter no pure ontology
8
Current status

• Implementation of Test environment
• Inclusion into KAON tool suit
• Testing of routing algorithms

9
Future work

• Test framework
• Creation of a standard test ontology
• Enabling techniques like
• merging algorithms
• semantic routing

10
Conclusion

• Evaluation is important for DKMS
• Combination of evaluation measures from different
  disciplines
• Test dataset must be created

11
Thank you!
Acknowledgements
12
Evaluation Principles

• User - based vs. System based
• Combination of Techniques from
• Information Retrieval
• Ontology specific
• Peer-to-Peer

13
Evaluation function

• n gt 1
• problem of alignment
• information loss