BlueEyes Human-Operator Monitoring System

1
BlueEyesHuman-Operator Monitoring System

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Poznan University of Technology, Poland
• http//www.put.poznan.pl

2
Agenda

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Motivation
• What is BlueEyes (and what is not) ?
• Physiological foundations
• System design
• System implementation
• Future improvements
• Project work summary
• System demonstration

3
Motivation

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Human error - a frequent reason of catastrophes
  and ecological disasters

• weariness
• oversight
• Monitoring of consciousbrain involvement
• Capable sensors available

• Lack of complex solution exploiting wireless link
  technology, database systems and AI algorithms

4
What is BlueEyes ?

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• The complex solution for human-operator
  monitoring
• conscious brain involvement monitoring
• triggering user-defined actions
• data recording

What is BlueEyes not?

• Doesnt predict nor interfere with operators
  thoughts
• Cannot force directly the operator to work

5
Who is BlueEyes for ?

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Generic control rooms
• Power station
• Captain bridge
• Flight control centers
• Operating theatres anesthesiologists
• Common application
• A simpler system version for drivers

6
BlueEyes - benefits

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Prevention from dangerous incidents

• The reconstructionof the course ofoperators
  work

7
Physiological foundations

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

Human brain

• Preconscious brain

Conscious brain

• manages the knowledge
• logical reasoning
• makes decisions

• alogical
• automatic

8
Conscious brain - operation modes

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

Exploratory mode
Planning mode
9
Eye movements

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Central vision andthe conscious brain

Saccade - a rapid eye jump to a new location
assigned by the conscious attention process.
Saccades presence ? Exploratory mode
10
The eye movement sensor

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Eye position measuring - direct infrared
  oculography
• Oxy- and deoxyhemoglobin measurment
• Two axialaccelerometer
• Ambient lightsensor

11
Wires, wires, wires...

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• A serious limitation of
• mobility
• comfort
• safety

12
Personal Area Network !

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

Solution
13
Design objectives

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• A mobile data acquisition device

• A reliable real-time buffering, processing and
  recording
• A clear visualization interface
• Mechanisms for creating custom data analyzers
• To ensure a possibility of distributing the
  processing among a number of computers

• Objectives
• System overview
• DAU
• CSU

14
System overview

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

Connection Manager
Visualization Module
Atmel 89C52 microcontroller

• Objectives
• System overview
• DAU
• CSU

Data Analysis
Jazz Multisensor
15
DAU - features

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Lightweight
• Runs on batteries - low power consumption
• Easy to use - does not disturb the operator
  working
• ID cards for operator authorization
• Voice transmission using hardware PCM codec

Atmel 89C52 microcontroller
Bluetooth

• Objectives
• System overview
• DAU
• CSU

Jazz Multisensor
16
CSU - features

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Connection management
• Data processing
• Visualization
• Data recording

Connection Manager
Visualization Module
Connection Manager
Bluetooth

• Objectives
• System overview
• DAU
• CSU

Data Analysis

• Accessverification
• System maintenance

17
CSU - saccade detector

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Saccade detector and conscious brain involvement
  level calculation

Raw eye movement data

• Objectives
• System overview
• DAU
• CSU

18
CSU - Visualization Module

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Raw and processed data visualization using

• Objectives
• System overview
• DAU
• CSU

19
Data security

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Only registered mobile devicescan connect to the
  system
• Bluetooth connection authentication
• Bluetooth connection encryption
• Access rights restrictions
• Personal and physiological data encryption

• Objectives
• System overview
• DAU
• CSU

20
Implementation - DAU

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Bluetooth module inside
• Jazz sensorinterface
• Microphone andearphone sockets
• 5-key keyboard
• Two lines, 8characters LCD
• Powered from6 AA batteries

• DAU
• CSU

21
Implementation - DAU

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• DAU
• CSU

22
Implementation - CSU

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

• Custom-builtBluetoothcommunicationroutines
• Intel USB camera
• ID card programmer
• Developed using Visual C 6.0
• Deployment environmentWindows 2000,MS SQL
  Server

• DAU
• CSU

23
Future improvements

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Systemdemonstration
• Futureimprovements
• Project worksummary

• DAU
• small CMOS camera to monitor the operators point
  of gaze
• single PCB (SMD technlogy)
• low voltage ICs - LiIO batteries power
• CSU
• data mining algorithms
• advanced database encryption using e.g. AES
  algorithm

24
Project work summary

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Systemdemonstration
• Futureimprovements
• Project worksummary

• Bluetooth and PAN technology acquired
• New skills on designing software and hardware
  systems gained
• A mobile device implemented
• 31 KLOC of C code and4 KLOC of 8051 assembler
  code written
• All parts of the Project Kit efficiently used
• New approach to system safety proposed and
  implemented

25

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Futureimprovements
• Project worksummary
• Systemdemonstration

BlueEyesHuman-Operator Monitoring System
System demonstration
26
Thank you for your attention

• Motivation
• What isBlueEyes ?
• Physiological foundations
• Systemdesign
• System implementation
• Systemdemonstration
• Futureimprovements
• Project worksummary

BlueEyesHuman-Operator Monitoring System
Poznan University of Technology,
Poland http//www.put.poznan.pl
27
DAU - components
MC 145483 PCM codec
Atmel 89C52 microcontroller
Jazz Multisensor
Beeper
UART
LCD display
MAX232
UART
LED indicators
ID card
ID card interface
Simple keyboard
28
CSU - Connection Manager

• Initializes Bluetooth modules
• Inquires for new devices in range
• Establishes connections
• Supports operator roaming
• Feeds Operator Manager with physiological data

Operator Data Manager

• Data demultiplexer
• Data producer queues

Bluetooth Connection Manager
Separated data streams
Operator Manager
Raw Data Buffer
Alerts
Alert buffer
... ...
29
CSU - Data Analysis

• Custom data analyzers - decision tree based
  classifiers

Example Operator gesture recognition (simplified)
No answer
NO
YES
30
CSU - Data Analysis

• Custom data analyzers - decision tree based
  classifiers

• Automatic decision tree coefficients generation -
  C4.5 based algorithm

... ...
31
CSU - intermodule communication

• Uses buffered thread-safe queues and
  producer-consumer scheme

Data consumer
Data queue
creates
Data producer
Data consumer
Data consumer
Data consumer
32
CSU - intermodule communication
Separated physiological data streams
Alerts
Data Analysis
Visualization Module
Data Logger
Processed data
Recorded (off-line) data