Diapositiva%201

1
TrebleCLEFEvaluation package and language
resourcesCLEF Evaluation PackagesNicolas
Moreau / Khalid Choukri - ELDA
2
TrebleCLEF and LRs

• CLEF evaluation campaigns produced large volumes
  of valuable data
• A major objective of TrebleCLEF make CLEF
  collections publicly available
• ELDA responsible for

• Inventory of CLEF resources
• IPR issues
• Production of CLEF Evaluation Packages

3
Evaluation Packages

• CLEF test suites consist of
• Multilingual document collections
• Sets of topics
• Sets of relevance assessments
• Scoring tools
• Participant results and related publications
• Step-by-step documentation on how to perform a
  system evaluation
• Reusable test suites for system benchmarking

4
CLEF Packages

• Already distributed (ELRA/ELDA catalogue)
• CLEF 2000-2003 (AdHoc on news domain specific)?
• Produced or being produced
• CLEF AdHoc/News 2004-2008 gt Preliminary version
• CLEF AdHoc/TEL 2008
• CLEF QA 2003-2008 CLEF QAST 2007-2008
• CLEF Domain specific 2004-2008
• GeoCLEF 2005-2008
• Still in discussion
• ImageCLEF, VideoCLEF
• WebCLEF, CLEF-IP, and others ...?

5
Distribution and Pricing Policy

• Distribution
• ELRA/ELDA catalogue http//catalog.elra.info/
• Restricted to evaluation use
• Pricing policy
• Low prices for academics
• Cover the management of data (catalogue
  management, DVD shipping, etc.)?