AWEAR 2.0 System: Omnidirectional AudioVisual Data Acquisition and Processing

1
AWEAR 2.0 System Omni-directionalAudio-Visual
Data Acquisition and Processing

• Michal Havlena 1, Andreas Ess 2, Wim Moreau 3,
  Akihiko Torii 1,Michal Jancoek 1, Tomá Pajdla
  1, and Luc Van Gool 2,3

1)
2)
3)
2
AWEAR 2.0 System
3
AWEAR 2.0 Components

• 3 D2703-S Mini-ITX motherboard
• AMD mobile Turion 64 X2 Dual Core
• 2 AVT Stingray F-201C camera
• 2Mpixel, 14fps, IEEE-1394b
• 2 Fujinon FE185CO86HA1 lens
• fish-eye with 150114 FOV
• Focusrite Saffire Pro soundcard
• 10-channel, 96kHz
• 4 T.Bone EM700 microphone
• condenser microphone, -42dB

4
AWEAR 2.0 Components

• Power distribution unit
• 4 7.5Ah 12V lead-gel battery
• Hardware trigger
• synchronizing cameras and audio
• Gigabit Ethernet connection
• Wi-Fi remote operation
• Rigid frame backpack
• connectors, cables,
• Software
• Ubuntu, recording SW

5
AWEAR 2.0 Schema

• VIDEO subsystem
• 2 PC, 2 cameras
• AUDIO subsystem
• 1 PC, 4 microphones
• POWER subsystem
• 4 batteries
• TRIGGER

6
Camera Calibration

• Equi-angular projection model Micusik PAMI06
• transformation between rays and points
• calibration target box

7
Wide Baseline SfM

• Camera pose estimation
• Keyframe selection
• sufficient dominant apical angle Torii CVPR08
• Trajectory from keyframes
• chaining epipolar geometries Torii PSIVT09
• Gluing non-keyframe images
• camera resectioning using the 3D point cloud
• Interpolating remaining images

t
8
Trajectory from Keyframes

• SfM pipeline
• Feature region detection description
• MSER Matas IVC04, SURF Bay CVIU08
• Descriptor matching by FLANN Muja VISAPP09
• Epipolar geometry by PROSAC Chum CVPR05
• soft-voting for epipole position Torii
  VISAPP08
• 5-point minimal problem Nister PAMI04

9
Trajectory from Keyframes

• Chaining epipolar geometries
• fixing the scale using 1 point correspondence
• cone intersection test
• L1-triangulation feasibility Kahl ICCV05

10
Image Sequence Stabilization

• Fixing gravity vector direction w.r.t. the first
  camera
• use known camera pose to rotate
• rectify to cylindrical projection

n o n - c e n t r a l
c e n t r a l
11
Image Sequence Stabilization
w i t h s t a b i l i z a t i o n
w / o s t a b i l i z a t i o n
o r i g i n a l i m a g e
12
High-level Visual Data Processing

• Pedestrian tracking Ess CVPR08
• Detection by classification
• HOG Dalal CVPR05
• Multi-hypothesis tracking
• TODO Action recognition
• Dense 3D reconstruction Jancosek CMP-TR08
• 3D mesh reconstruction
• independently for each camera
• Fusion of reconstructed meshes
• TODO Obstacle detection

13
Pedestrian Tracking

• Obtaining space-time pedestrian trajectories
• use known camera trajectory (from SfM)
• detect pedestrians in each frame
• generate an overcomplete set of possible
  explanations
• select the best mutually consistent subset
• Trajectory model
• Motion model constant velocity (EKF)
• Appearance model color histogram (HSV)

14
Pedestrian Tracking

• Track hypotheses generation
• Parallel generation of new and alternative
  explanations
• detections from next timeslot
• independent search down the timeline
• finding new trajectories
• Extension of existing trajectories
• predicting selected hypotheses
• greedy assignment
• Global optimization
• favouring explanations with fewer trajectories

15
Dense 3D Reconstruction

• Obtaining a candidate 3D mesh for each camera
• 3D seeds
• Harris on a grid Furukawa CVPR07, guided
  matching
• Growing
• online orientation refinement using PCA based
  plane fitting
• Filtering
• MRF based filtering approach Campbell ECCV08
• Final mesh reconstruction (Meshing)
• part of a mesh is accepted if at least 3 cameras
  agree
• no further candidate mesh reconstruction for
  places contained in the final mesh already

16
Dense 3D Reconstruction
17
Experimental Results

• PEDCROSS data set
• 228 frames at 12fps

18
Experimental Results

• Stabilization and pedestrian tracking

19
Experimental Results
/
20
Conclusions

• Wearable audio-visual sensor platform
• running cognitive supportive applications for
  elderly
• Current embodiment
• capturing egocentric audio-visual sequences
• running state-of-the-art computer vision methods
  off-line
• speed-up needed for actual cognitive feedback