Ra

1
Benchmarking in Knowledge Web
Raúl García-Castro, Asunción Gómez-Pérez ltrgarcia,
asun_at_fi.upm.esgt
Jérôme Euzenat ltJerome.Euzenat_at_inrialpes.frgt
September 10th, 2004
2
Industrial Benchmarking
Research Benchmarking
?
WP 1.2 (From T.A. page 26) WP 2.1 (From T.A. Page 41)
Point of view Tool recommendation Research progress
Criteria Utility Scalalability Robustness Interoperability
Tools Ontology development tools Annotation tools Querying and reasoning services of ontology development tools Merging and alignment tools Ontology development tools Annotation tools Querying and reasoning services of ontology development tools Semantic Web Service technology
3
Index Benchmarking activities in Knowledge
Web Benchmarking in WP 2.1 Benchmarking in WP
2.2 Benchmarking information repository Benchmarki
ng in Knowledge Web
4
Benchmarking activities in KW

• Overview of the benchmarking activities
• Progress
• What to expect from them
• What are their relationships/dependencies
• What could be shared/reused between them

5
Benchmarking timeline
?
D1.2.1 Utility of ontology development tools
Utility of merging, alignment, annotation
Performance of querying, reasoning
WP 1.2 Roberta Cuel
D1.31 Best practices and guidelines for industry
Best practices and guidelines for business cases
WP 1.3 Luigi Lancieri
D2.1.4 Benchmarking Methodology, criteria, test
suites
D2.1.1 Benchmarking SoA
D2.1.6 Benchmarking building tools
Benchmarking querying, reasoning, annotation
Benchmarking web service technology
WP 2.1 Raúl García
D2.2.2 Benchmarking methodology for alignment
D2.2.4 Benchmarking alignment results
Progress
Finished
WP 2.2 Jérôme Euzenat
Started
Not started
0 6 12
18 24
30 36
42 48
6
Benchmarking relationships
T 1.3.1 Best Practices and Guidelines
Best Practices
Benchmarking methodology Benchmark suites
T 1.2.1 Utility of ontology- based tools
Benchmarking overview SoA ontology tech.
evaluation
T 2.1.4 Definition of a methodology, general
criteria for benchmarking
T 2.1.6 Benchmarking of ontology building tools
T 2.1.1 SoA on the technology of the
scalability WP
Benchmarking methodology alignment Benchmark
suite alignment
Benchmarking methodology
T 2.2.4 Research on alignment techniques and
implementations
T 2.2.2 Design of a benchmark suite for
alignment
6
12
18
24
7
Index Benchmarking activities in Knowledge
Web Benchmarking in WP 2.1 Benchmarking in WP
2.2 Benchmarking information repository Benchmarki
ng in Knowledge Web
8
Benchmarking in WP 2.1
0 6
12 18
24 ... 36
... 48
T 2.1.6 Benchmarking of ontology building tools
T 2.1.4 Definition of a methodology, general
criteria for ontology tools benchmarking
T 2.1.1 State of the Art
T2.1.x Benchmarking querying, reasoning, annotati
on, web service
Benchmarking methodology
Type of tools to be benchmarked Ontology building tools Annotation tools Querying and reasoning services of ontology development tools Semantic Web Services technology
General evaluation criteria Interoperability Scalability Robustness
Test suites for each type of tools
Benchmarking supporting tools

• Overview of benchmarking, experimentation, and
  measurement
• SoA of ontology technology evaluation

Specific evaluation criteria Interoperability Scalability Robustness
Test suites for ontology building tools
Benchmarking supporting tools
9
T 2.1.1 Benchmarking Ontology Technology in D
2.1.1 Survey of Scalability Techniques for
Reasoning with Ontologies
Ontology Technology/Methods
Experimentation
Evaluation
Benchmarking
Measurement
Desired attributes Weaknesses Comparative
analysis ...
Continuous improvement Best practices
Recommendations

• Overview of benchmarking, experimentation, and
  measurement
• State of the Art of Ontology-based Technology
  Evaluation

10
T 2.1.4 Benchmarking methodology, criteria, and
test suites
Methodology
Plan 1 Goals identification 2 Subject
identification 3 Management involvement 4
Participant identification 5 Planning and
resource allocation 6 Partner selection
Experiment 7 Experiment definition 8 Experiment
execution 9 Experiment results analysis
Improve 10 Report writing 11 Findings
communication 12 Findings implementation 13
Recalibration

• Benchmark suites for
• Ontology building tools
• Annotation tools
• Querying and reasoning services
• Semantic Web Services technology

• General evaluation criteria
• Interoperability
• Scalability
• Robustness

• Benchmarking supporting tools
• Workload generators
• Test generators
• Statistical packages
• ...

11
T 2.1.6 Benchmarking of ontology building tools
Partners/Tools UPM
...
...
...
...

• Benchmark suites
• Interoperability
• (x tests)
• Scalability
• (y tests)
• Robustness
• (z tests)

• Benchmarking results
• Comparative
• Weaknesses
• (Best) practices
• Recommendations

Benchmarking ontology building tools

• Interoperability
• Do the tools import/export from/to RDF(S)/OWL?
• Are the imported/exported ontologies the same?
• Is there any knowledge loss during
  import/export?
• ...

• Benchmark suites
• RDF(S) Import capability
• OWL Import capability
• RDF(S) Export capability
• OWL Export capability

• Experiments
• Import/export RDF(S) ontologies
• Import/export OWL ontologies
• Check for knowledge loss
• ...

• Experiment results
• test 1
• test 2
• test 3
• ...

• Benchmarking results
• Comparative
• Weaknesses
• (Best) practices

NO OK OK
12
Index Benchmarking activities in Knowledge
Web Benchmarking in WP 2.1 Benchmarking in WP
2.2 Benchmarking information repository Benchmarki
ng in Knowledge Web
13
T 2.2.2 Design of a benchmark suite for alignment

• How evaluate?
• Take a real life case and set the deadline
• Take several cases normalizing them
• Take simple cases identifying what they
  highlight
• (benchmark suite)
• Build a challenge (MUC, TREC)

• Why evaluate?
• Comparing the possible solutions
• Detecting the best methods
• Finding out where we are bad.
• Two goals
• For the developer improving the solutions
• For the user choosing the best tools
• For both testing compliance with a norm.
• Results
• Benchmarking methodology for alignment
  techniques
• Benchmark suite for alignment
• First evaluation campaign
• Greater benchmarking effort.

14
T 2.2.2 What has been done?

• Information Interpretation and Integration
  Conference (I3CON), to held at the NIST
  Performance Metrics for Intelligent Systems
  (PerMIS) Workshop focuses on "real-life" test
  cases and compare algorithm global performance.

• Facts
• 7 ontology pairs
• 5 participants
• Undisclosed target alignments (independently
  made)
• Ask for the alignments in normalized format
• Evaluation on the F-measure.

• Results
• Difficult to find pairs in the wild (they have
  been created)
• No dominating algorithm, no most difficult case
  for all
• 5 participants was the targetted number, we must
  have more next time!

The Ontology Alignment Contest at the 3rd
Evaluation of Ontology-based Tools (EON)
Workshop, to be held the International Semantic
Web Conference (ISWC) aims at defining a proper
set of benchmark tests for assessing
feature-related behavior.

• Facts
• 1 ontology and 20 variations (15 hand-crafted on
  some particular aspects)
• Target alignment (made on purpose) published
• Ask for a paper, with comments on the tests and
  on the achieved results (as well as the results
  in normalized format).

Results We are currently benchmarking the
tools! See you at EON Workshop, ISWC
2004, Hiroshima, JP November
15
T 2.2.2 Whats next?

• More consensus on whats to be done?
• Learn more
• Take advantage of the remarks
• Make a more complete
• real-worldbench suitechallenge?
• Provide automated procedures

16
Index Benchmarking activities in Knowledge
Web Benchmarking in WP 2.1 Benchmarking in WP
2.2 Benchmarking information repository Benchmarki
ng in Knowledge Web
17
Benchmarking information repository

• Web pages inside the Knowledge Web portal with
• General benchmarking information
• (methodology, criteria, test suites, references,
  ...)
• Information about the different benchmarking
  activities in Knowledge Web
• Benchmarking results and lessons learned
• ...
• Objectives
• Inform
• Coordinate
• Share/reuse
• ...
• Proposal for a benchmarking working group in the
  SDK cluster.

18
Index Benchmarking activities in Knowledge
Web Benchmarking in WP 2.1 Benchmarking in WP
2.2 Benchmarking information repository Benchmarki
ng in Knowledge Web
19
What is benchmarking in Knowledge Web?

• In Knowledge Web
• Benchmarking is performed over products/methods
  (not processes)
• Benchmarking is not a continuous process
• Ends with findings communication, there is no
  findings implementation or recalibration
• Benchmarking technology involes evaluating
  technology
• Benchmarking technology is NOT just evaluating
  technology
• We must extract practices and best practices
• Benchmarking results
• Comparative
• Weaknesses
• (Best) practices
• Benchmarking results are needed!
• Both in industry and research
• ...

Recommendations (Continuous)
Improvement
20
How much do we share?

• Is the view about benchmarking from industry
  similar to the view from research?
• Is it viable to have a common methodology? Will
  anyone use it? ?
• Can the test suites be reused between
  industry/research?
• Can be useful a common way of presenting test
  suites?
• ...
• Can research benchmarking results be (re)used by
  industry, and viceversa?
• Can be useful a common way of presenting
  results?
• ...

Benchmarking methodology, criteria, and test
suites Benchmarking results
21
Next steps

• Provide the benchmarking methodology to industry
• First draft after Manchester Research meeting.
  1st October.
• Feedback from WP 1.2. End of October.
• (Almost) final version by half-November.
• Set up web pages with benchmarking information in
  the portal
• Benchmarking activities
• Methodology
• Criteria
• Test suites
• Discuss in a mailing list and agree on a
  definition of best practice.
• Next meeting? To be decided (around November)
  (with O2I)

22
Benchmarking in Knowledge Web
Raúl García-Castro, Asunción Gómez-Pérez ltrgarcia,
asun_at_fi.upm.esgt
Jérôme Euzenat ltJerome.Euzenat_at_inrialpes.frgt
September 10th, 2004