Chesley Robinette

1
WekaMetal Meta-learning extension to Weka

• By
• Chesley Robinette
• Hemant Joshi

2
Project Outline

• What is WekaMetal?
• Introduction
• Download and Installation
• Configuration and launch
• Get to know the MetaL
• Known Bugs and Limitations
• Demo and Questions

3
What is WekaMetal?

• WekaMetal is a Meta-Learning extension to the
  data-mining package Weka. It offers advice on
  algorithm selection, based on expected accuracy
  and time performance

4
Introduction

• WekaMetal provides a ranking which predicts how
  well a set of algorithms will perform on a
  particular dataset. (Roughly speaking, this is
  what meta-learning is about.)

5
Introduction

• The ranking is generated using knowledge of the
  performance of those algorithms on some
  "benchmark" datasets. The ranker first
  "characterizes" the previously unseen dataset.
  Based on this characterization, and a similar
  characterization of the benchmark datasets,
  datasets are selected which are similar to the
  unseen dataset.

6
Introduction

• WekaMetal uses its knowledge of the performance
  of the algorithms on the benchmark datasets to
  predict the performance of the algorithms on the
  unseen dataset.

7
Download and Installation

• You can download WekaMetal from
• http//www.cs.bris.ac.uk/7Efarrand/wekametal/Weka
  Metal-0.11.jar
• Jar file ensures portability to all available
  platforms.

8
Download and Installation

• Unzip this jar file into a separate folder for
  example c\wekametal and then put both weka.jar
  from \\program files\weka 3.2\ folder and
  wekametal.jar from \\wekametal folder in the
  system class path.
• WekaMetal uses compact version of Weka and not
  all classes are loaded if not required.

9
Configuration and launch

• You can run WekaMetal through a batch file or
  through a link that has following command.
• Java cp c\program files\weka 3.2\weka.jar
  c\wekametal\wekametal.jar weka.gui.explorer.Exp
  lorer

10
Configuration and launch

• WekaMetal out of the box does not need any
  special configuration to start with and the Weka
  window that is launched will have an extra tab
  called MetaL. The initial screenshot looks like
  shown on the next slide.

11
Configuration and launch
12
Get to know the MetaL

• The top-centre widget allows different rankers to
  be selected. Currently there are two available.
  The SRR (Simple Ratio of Ratios) ranker ranks
  algorithms based on expected accuracy. The ARR
  (Adjusted Ratio of Ratios) ranks algorithms based
  on expected accuracy and algorithm run time.

13
Get to know the MetaL

• Characteristics can be (de)activated in the
  central panel. A characteristic is a meta-level
  attribute which describes a dataset. If unsure,
  leave them all selected. The right hand panel
  selects the algorithms which should be ranked.

14
Get to know the MetaL

• Clicking "Start" causes the ranker to run. The
  Ranking will be generated in a few seconds,
  providing that WekaMetal has previously computed
  the performance of the selected algorithms on the
  benchmark datasets. If you are told that the
  benchmark data is not available you have two
  options

15
Get to know the MetaL

• If you think that the benchmark data really is
  available, uncheck the "strict Benchmark
  checking" option. This occurs because the
  benchmarks have a different timestamp that when
  WekaMetal last ran the algorithms.

16
Get to know the MetaL

• If on the other hand you think that the benchmark
  data really isn't available check the "Allow
  Benchmark Generation" option. This occurs because
  you have added characterizers and/or algorithms
  and/or benchmark datasets. It is time consuming
  process.

17
Known Bugs and Limitations

• Rankings are not yet very good - need more
  benchmarks and characterizers.
• The time component of rankings is taken from
  doing 10-fold cross validation, rather than
  training time.
• Can't be run from the command line. (I.e.,
  ranking needs to be called from the GUI or from
  another Java program.)

18
Known Bugs and Limitations

• Some sanity checks are probably missing. (For
  example, WekaMetal will give a nonsense ranking
  if you deselect all characterizers.)
• Caching of characteristics/algorithm performance
  is a little flakey - it's sometimes too strict in
  strict mode, and sometimes not strict enough in
  non-strict mode.

19
Known Bugs and Limitations

• The stop button doesn't work.
• Countless more waiting to be discovered.

20
Demo and Questions