ENTROPY-BASED CONCEPT SHIFT DETECTION

1
ENTROPY-BASED CONCEPT SHIFT DETECTION

• PETER VORBURGER, ABRAHAM BERNSTEINIEEE ICDM 2006

1
2
OUTLINE

• Introduction
• Entropy and Concept Shift Adaption
• Calculating Entropy on Data Streams
• Algorithm Control Strategy using Entropy Measure
• Experimental setup and results
• Application to a Real-World Problem
• Limitations, Future Work, and Conclusion

2
3
INTRODUCTION

• Motivation the analysis of sensor data on
  wearable devices.
• Problem
• Concepts may drift (i.e., change) over time.
• Increasing amount of data
• Limitation of computing power due to
  miniaturization
• Faster and more resource friendly algorithms.

3
4
INTRODUCTION

• Context-awareness A Scenario-based Approach for
  Direct Interruptablity Prediction on Wearable
  Devices
• Based on sensory input
• Contexts switch rather than gradually change
• Contextual information could be reused
• An ongoing monitoring of the sensor stream is
  needed

4
5
ENTROPY AND CONCEPT SHIFT ADAPTION

• Assumptions
• Similar distribution ? no concept drift occurred
• Measure the distribution inequality
• Entropy 1?If two distributions are equal
• Entropy 0?If they are absolutely different

5
6
CALCULATING ENTROPY ON DATA STREAMS

• Sliding window technique
• one presenting older
• and the other representing more recent instances
  in the stream
• Discretize the range of instance values to a
  fixed number of bins to take the approximate
  value distribution into account

6
7
CALCULATING ENTROPY ON DATA STREAMS

• Data stream sequentially ordered tuples in
  time
• i (1, 2, 3, )
• where is the vector of all feature stream
  instances sni at time ti
• The domain of the label stream l is discrete and
  contains all class values c C

7
8
CALCULATING ENTROPY ON DATA STREAMS

• Hi the resulting entropy at time ti
• S the number of feature-streams
• His is calculated from the entropies Hiscb

9
CALCULATING ENTROPY ON DATA STREAMS

• Hiscb represent the entropy of each class (c
  C) and bin (b B) given the stream s at time ti
• Bins discrete aggregation of the values
• the probability that an instance
  occurs in the old window at time ti, belong to
  class c, with feature domain of stream s in bin b
• wiscb depend on i, s, c, b

10
ALGORITHM CONTROL STRATEGY USING ENTROPY MEASURE

• Instance selection style algorithm

11
EXPERIMENTAL SETUP

• Real concept drifts changes in the actual target
  concepts
• Virtual concept drifts changes in the
  distribution
• P. Vorburger and A. Bernstein. Entropy-based
  detection of real and virtual concept shifts.
  Working Paper University of Zurich, Department
  of Informatics, 2006

12
EXPERIMENTAL SETUP

• A representative set of benchmarks
• Perfect benchmark assumes an oracle-given ideal
  window size ? for any point in time
• A selection of ensemble classifiers the
  literature so far showed to have the highest
  accuracy and robustness against noise

13
EXPERIMENTAL RESULTS
14
EXPERIMENTAL RESULTS

• The prediction quality against increasing noise
  levels

15
EXPERIMENTAL RESULTS

• The elapsed time
• Three committee classifiers 2031.615s
• Entropy based algorithm 148.6s
• Entropy calculation without Naïve Bayes model
  building 1.10.1s

16
APPLICATION TO A REAL-WORLD PROBLEM CONTEXT
SWITCHES IN SENSOR DATA

• Data set
• Audio decomposed into 10 features
• accelerometer data recorded over a time of
  15381s merged in one single feature
• The wearable data acquisition set up a
  microphone and three three-dimentional
  accelerometers attached on the subjects
  shoulder, wrist, and leg

17
APPLICATION TO A REAL-WORLD PROBLEM CONTEXT
SWITCHES IN SENSOR DATA

• (A)walking (B)streetcar (C)office work
• (D)lecture (E)cafeteria (F)meeting

18
LIMITATIONS, FUTURE WORK, AND CONCLUSION

• Gradual concept drifts
• Recognize recurring concepts and exploit this
  information
• Formulation of entropy on data streams is capable
  to detect and measure concept shifts
• Algorithm with an entropy based instance
  selection strategy outperformed ensemble based
  algorithms on real concept shift data sets