Regression Systems For Knowledge

1

• Regression Systems For Knowledge
• Discovery
• Conor Nugent

2
Overview

• Regression Systems and Knowledge Discovery
• A quick review of different regression systems
  and approaches taken
• A look at the performance of some methods in
  particular
• A look at the a particular domain problem, the
  Met Éireann project
• Future Directions- some ideas about providing an
  explanation facility

3
Regression

• The problem is to build a model that will predict
  a continuous various
• e.g. an exchange rate, life expectancy, melting
  point, temperature etc
• Traditionally the focus has been on building
  accurate models but model interpretability is now
  considered a important factor in many
  applications

4
Knowledge Discovery Explanation Facilities

• The extraction of rules allows for User
  Explanation of the system
• e.g Safety Critical domains such as health
  care systems
• Knowledge Discovery- Regression Systems may learn
  previously unnoticed complex non-linear
  relationships in the input data. Using rule
  extraction techniques these relationships may be
  exposed to the user

5
Symbolic Approaches

• Symbolic
• Tree approaches
• CART (Breiman) , M5 (Quinlan)
• Rule Induction algorithms
• Rule (Sholom M. Weiss and Nintin Indurkhya )
• FORS (Aram Karalic and Ivan Bratkko )
• R2 (Luis Torgo)

6
(No Transcript)
7
R2, Luis Torgo
 
8
Regression Systems

• Statistical
• MARS (Multivariate Adaptive Regression Splines),
  Jerome Friedman. Uses simple basis functions to
  iteratively build up a model, very nasty

9
Lazy Learning Regression Methods

• Simple k-neighest neighbour- finds k nearest data
  sample and averages them
• Weighted Averaging- takes a weighted average of
  each smaple wihtin a certain range
• Local Linear Regression- fits a regression model
  to the weighted set of data points sround the
  query point

10
Conectionist Approaches, Neural Networks and
Ensembles

• Neural Networks
• Universal approximators
• Lack of transparency!

11
Ensembles

• Improved Accuarcy and stabilty
• But even less interpretable

12
Is there any need to use Ensembles of Neural
Networks?

• Tested Neural Network and Bagged ensembles of
  Neural Network models on a series of UCI data
  sets
• Compared results with Linear Models, 5-NN and
  ensmbles of these models

13
Experiment Settings

• 10 Fold cross validation was used as the
  evaluation method
• NeuralBag Algorithm was used to generate the
  ensembles and simple averaging was used as the
  aggregating method
• Feature subset selction was used as the ensemble
  genertaing method and Dynamic Selection and SR
  were used as aggregation techniques

14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
Met Éireann Post Processing Problem

• Met Éireann have numerical models that predict
  the weather 24 hours in advance
• Ireland is broken in to a number of different
  regions and a forecast is given for each.
• The Forecasting models are good but they could be
  better and their performance at critical
  temperatures i.e. freezing point is important
• The model dont take into account local geography
  but using historical data we might be able to
  build models that implicitly take this into
  account

18
(No Transcript)
19
(No Transcript)
20
Neural Net Explanation Systems

• Different Categories
• Decompositional- take an in-depth view of the
  architecture of a network, decomposing the very
  units of the network in order determine to
  relationships and dependencies
• Black Box- view the ANN as an oracle or black
  box and try to extract global relationships
  between inputs and the outputs without knowledge
  of the inner workings of the ANN
• Ecletic somewhere in between

21
Decompositional

• REFANN (Rule Extraction from Function
• Approximating Neural Networks)

Rule 1 Rule 2 Rule 3 Rule 4
22
Fuzzy Rule-Based Systems (FRBS)
 
23
Black Box Techniques

• Trepan
• Tree based approach that differs significantly
  from from other popular decision-tree such as
  CART and C4.5
• Membership Queries and the Oracle
• Tree Expansion
• Splitting Tests m-of-n tests

24
 
25
Future Work

• An ensembe of linear models might be easier to
  intreprete.
• Black box approaches, Trepan for regression?
  could a neural network model an ensemble of
  networks?
• Build a really large casebase and use local
  regression methods? Generate cases on the fly?