Multimodal sensors

1
Multimodal sensors digital interfaces
2
Credits

• The original Multimodal Project developed by and
  credit for
• Zhigang Zhu and Weihong Li (Integration of Laser
  Vibrometry with Infrared Video for Multimedia
  Surveillance Display)

3
Outline

• Multimodal System Overview
• Multimedia Sensors
• Infrared camera
• The LDV sensor
• PTZ camera
• Multimodal System Components
• System Design Concept
• Design Issues
• Integration Issues

4
Multimodal System Overview

• The object of this system is to provide a
  multimodal integration of audio, visible, thermal
  for human signature detection.
• The goal is to use the sensing technologies for
  perimeter surveillance.
• Sensors, alarm, response.
• Multimodal system interface
• The environment, the sensors, and the events.

5
Multimedia Sensors Infrared Camera

• Infrared camera
• FLIR ThermoVision A40M IR camera
• Temp Range of -20º to 500ºC, accuracy ( of
  reading) 2ºC or 2
• 320x240 Focal Plane Array
• 24º FOV Lens
• Firewire Output IEEE 1394
• Video output RS170 EIA/NTSC or CCIR/PAL
  composite video for monitoring on a TV screen
• ThermoVision System Developers Kit (C)
• Each thermal image is built from 76,800
  individual picture elements that are sampled 60
  times per second by the camera's on-board
  electronics.

6
Multimedia Sensors Infrared Camera

• Samples

Figure 1. A person sitting in dark room can be
clearly seen in the IR image. The temperature at
Sp1 on the face is 33.1ºC
Figure 2. Two IR images before and after a person
standing at about 200 feet. The reading of the
temperature at Sp1 changes from 11C to 27C.
7
Multimedia Sensors Infrared Camera

• Thermographic measurement techniques
• An infrared camera measures and images the
  emitted infrared radiation from an object.
• The radiation measured by the camera does not
  only depend on the temperature of the objects but
  is also a function of the emissivity.
• Radiation also originates from the surroundings
  and is reflected in the object
• Radiation from the object and reflected radiation
  will also be influenced by the absorption of the
  atmosphere
• Parameters need to take care
• The emissivity of the object
• The reflected temperature
• The distance between the object and the camera
• The relative humidity

8
Multimedia Sensors Infrared Camera

• Emissivity
• How much radiation is emitted from the object
• Object materials and surface treatments exhibit
  emissivity ranging from approximately 0.1 0.95
• Highly polished (mirror) surface lt 0.1
• Human skin exhibits an emissivity close to 1
• Metal low, increase with temperature
• Non-metal high, decrease with temperature

9
Multimedia Sensors Infrared Camera

• Reflected ambient temperature
• To compensate for the radiation reflected in the
  object and the radiation emitted from the
  atmosphere between the camera and the object.
• If the emissivity is low, the distance very long
  and the object temperature relatively close to
  that of the ambient it will be important to set
  and compensated for the ambient.
• Distance
• The distance between object and the front lens of
  the camera.
• Relative Humidity
• Normally, default 50

10
Multimedia Sensors Infrared Camera

• History of Infrared Technology
• Sir William Herschel (1738-1822)
• Discover of infrared spectrum
• Marsilio Landriani (1746-1815)
• As the blackened thermometer was moved slowly
  along the colors of the spectrum, the temperature
  readings showed a steady increase from the violet
  end to the red end.
• Macedonio Melloni (1798-1854)
• Rock salt (NaCl) (to be made into lenses and
  prisms) is remarkably transparent to the
  infrared.
• Sir John Herschel
• The first heat-picture in 1840, thermograph
• Samuel P. Langley (1834-1906)
• Inventor of the bolometer (1880)

11
Multimedia Sensors LDV sensor

• Vibrometer types
• Single Point Vibrometers
• Measure the vibration of an object in the
  direction of laser beam
• Differential Vibrometers (dual beam)
• Allow vibration measurement between two points
  vibrating relative to each other.
• Rotational Vibrometers
• Measure angular vibrations on rotating
  structures.
• In-plane Vibrometers
• measure continuous (DC) velocity and superimposed
  variable (AC) components perpendicular to the
  central axis of two converging laser beams.
• 3D Vibrometers

12
Multimedia Sensors LDV sensor

• Laser Doppler Vibrometer (LDV)
• Optical instruments for accurately measuring
  velocity and displacement of vibrating structures
  completely without contact.
• Sensor head OFV-505
• HeNe laser, ? 633.8 nm.
• OFV-SLR lens (f30mm) 1.8m 200m, auto focus
• Controller OFV-5000 with a digital velocity
  decode card VD-6
• RS232 interface for computer control
• Telescope VIB-A-P05
• 1º vertical tilt and 1.5º horizontal tilt

13
Multimedia Sensors LDV sensor

• Measurement Principle

S is the light source f is frequency P is the
moving with velocity v and reflects the light O
is the receiver (f ?fD)
Resultant frequency shift
For vibrometers SO ("backscatter") ?1 - ?2,
therefore,
14
Multimedia Sensors LDV sensor

• LDV schema
• Velocity is directly obtained by demodulation ?
  2?fm
• Voice frequency f 300Hz 3000Hz
• LDV can detect vibration at a magnitude as low as
  m ?/2?f 1/(23.14300) 0.5µm

15
Multimedia Sensors PTZ camera

• Pan/tilt/zoom (PTZ) camera
• Human and other target detection at a large
  distance
• Canon PTZ
• 26X optical zoom lens 12X digital zoom
• Pan 100º, Tilt 90º/-30º
• Built-in IR light (effective up to 9 feet)
• BNC video output
• RS-232 computer control interface

16
Multimedia Sensors PTZ camera

• PTZ Samples

Two images of a person at a distance of about 200
feet, captured by changing the zoom factors of
the PTZ camera.
17
Multimodal System Components

• Three components
• The IR/EO imaging video surveillance component
• Human motion tracking, human face detection
• Thermal Camera for daytime and nighttime
• Visible camcorder (sony), Web cam (logitech)
• The LDV audio surveillance component
• Audio signal capture, voice recognition
• The human-computer interaction component
• Cognitive understanding of the environment, the
  sensors, and the events

18
System Design Concept

• The overall goal the project is to design a human
  computer interface for human-centered multimodal
  (MM) surveillance.

19
Design Issues

• Issues need to be considered
• how to use EO camera tracking human motion
• how to incorporate IR imaging with existing EO
  captured image
• how to use IR imaging to help the laser Doppler
  vibrometer to select the appropriate targets
• how to select optimal viewpoint from audio
  detection.

20
Integration Issues

• Target detection and localization via IR/EO
  imaging
• Set up an IR/EO imaging system with an IR camera
  and a PTZ camera for finding vibration targets
  for LDV listening
• Registration between the IR/EO imaging system and
  the LDV system.
• Two types of sensors need to be precisely aligned
  so that we can point the laser beam of the LDV to
  the target that the IR/EO imaging system has
  detected
• Future research on automated targeting and
  focusing.

21
References

• Main multimodal system technical report
• http//www-cs.ccny.cuny.edu/zhu/LDV/FinalReportsH
  TML/CCNY-LDV-Tech-Report-html.htm
• Polytec Laser Vibrometer
• http//www.polytec.com/
• FLIR Systems Security ThermoVision Cameras
• http//www.flir.com/
• Paper
• Z Zhu, W Li, Integrating LDV Audio and IR Video
  for Remote Multimodal Surveillance
• Others

22
(END)