Distributed Display Approach using PHMD with Infrared Camera

1
Distributed Display Approach using PHMD with
Infrared Camera

• Ryugo KIJIMA, Kenji HAZA, Yoshihiro TADA, Takeo
  OJIKA
• Virtual System Laboratory/Department of
  Information Science, Faculty of Engineering,Gifu
  University, JAPAN

2
Contents

• Motivation
• Development of Projection HMD
• Development of Camera Module and Image Processing
• Demo and Application
• Future Work

3
Person of Interface - Analogy of Mirror -
(I-P)

• First Person Interface Full Immersion HMD CAVE
  and its Clones
• Real Hand drives Virtual Hand that manipulates
  objectVirtual Eye (camera model) watches the
  world
• As if Users Consciousness Passes ThroughBody
  Mirror to Virtual Worldand Attached to the Model
  of User
• First Persons View

I am Replaced Replacing the self/world by Body
Mirror
4
Person of Interface -Analogy of Mirror- (II-P)

• Second Person Interface .. most of todays
  computing environment Usual Desktop Computer
  Environment PDA Cellular Phone .
• Users Focus is on One Display Like as a Hand
  Mirror Shows One Aspect of World
• Efficient, Matured but Limited

Talking to You World appears in Hand Mirror
5
Person of Interface - Analogy of Mirror-
(III-P)

• Third Person Interface isLike as Fragments of
  Mirror
• Each Mirror Shows Each Aspect of World
• Users Focus is Moving from One to the Other
• Attention is Invoked by the Other One
• Our Daily Life

Surrounded by Them Many fragments of Mirror
6
Goal -Display Everywhere- Ubiquitous Display

• Display Fixed to Location on the Wall, Desk,
  Floor, .
• Display on Object Digital Real Stationery,
  Information Tag
• Display on Body True Palm Computing
• Display Freely Handled More Interactivity

7
Basic Idea of System

• Projection HMD with Infrared Camera Multiplexing
  Input and Output

Retro Reflector as 1 Visual Screen Visible
Region 2 Hi-Contrast Marker IR
Region Relative Location between User and
Screen 3 Backlight for Finger Interaction
IR Wave Length
8
Development of PHMD
Principle of PHMDRetro-reflectorSmall PHMD
Development
9
Principle of Projection HMD Conjugation
between Projection Center and Viewpoint
(Kijima et. al. HIS 94, ICAT/VRST 95, VRAIS97)
Projection Transformation

• Optical Conjugation Projection
  Center View Point
• Projection Transformation x Viewing
  Transformation I

10
Principle of Projection HMD Conjugation
between Projection Center and Viewpoint
(Kijima et. al. HIS 94, ICAT/VRST 95, VRAIS97)
Viewing Transformation

• Optical Conjugation Projection
  Center View Point
• Projection Transformation x Viewing
  Transformation I

11
Principle of Projection HMD Conjugation
between Projection Center and Viewpoint
(Kijima et. al. HIS 94, ICAT/VRST 95, VRAIS97)
Optical Conjugation Cancels Distortion
User see the pixel in the direction in that the
projector throw it

• True image (HMDfalse image)
• Larger Field of View
• Robust Optics against offset from designed eye
  point

12
PHMD Ver. 0 (Kijima, Hirose 1994, VRAIS97)

• Merit- long eye relief- robustness against
  incorrect wearing
• Demerit- brightness .. dark (30W halogen
  bulb)- weight .. heavy (1300g)

13
Introducing Retro-Reflective Screen (Ishikawa,
1996, .. RRZ)(Inami, Kawakami, Tachi, 1997 ..
HMP)(Rolland,J., 1997 .. HMPD)

• Compensation of Brightness (Spooner, 1982) -gt
  smaller projector enabled
• Stereo viewing by Retro-Reflection Zoning
  
  (Ishikawa, 1996)
• Small pupil optical system for long focus depth
  (Inami, Kawakami, Tachi, 1997)
• Occlusion like as Back Projection
  System (Godzilla movie, 1960s)

RRZ (Ishikawa Optics)
HMP (Inami, Kawakami, Tachi)
14
PHMD-RR Ver. 1 (Oct. 98, Kijima)
To design the basic parameter of the optical
system Ver. 1a pin-hole model Ver. 1b
projection lens model
15
PHMD-RR Ver. 2 (Mar. 99, Kijima Haza)

• Head Mounted Configuration
• Monocular Display
• Light Source 5W Halogen Lamp caused the heat
  problem

Brightness-Heat RatioBrightness-Weight Ratio
is Key Problem
16
PHMD-RR Ver. 3 (Jul. 99, Haza Kijima)

• Halogen Lamp is isolated by Optical Fiber
• Design of LCD-Lens Block is Refined Heavier than
  Ver.2, better weight balance

17
PHMD-RR Ver.4 (Dec. 99, Haza, Miwa, Kijima)

• Stereoscopic Configuration
• Bright LED Block for Light Source (2cd x 8 x 5
  80cd) (Nitia Chem Industry Co)
• FOV 1545 deg (one side)
• Removable Lens Unit
• Weight 110g min

18
Robust PHMD (Aug. 2001, Suzuki, Kijima)

• Based on small projector in consumer market
  (Toshiba Videoball LZP2)
• Heavy (2kg) but Robust
• Constant Spring to compensate the weight

19
Screen as Input Device

• Multiplexing I/O using Infra-red Camera Module

20
Locating Screen with IR-Camera System Feature

• Infra-red Camera multiplexing I/O
  (display and sensing)
• Calculate Relative Location to Users Head
• Calculate Fingertip Position on Screen

Strategy Control the environment
before Swelling Image Processing
21
Infra-red Light Source (I)

• Camera-IR LED Conjugated Module
• Heavy
• Field of View is Limited (lt90 deg)
• A LED cannot generate enough power Low Contrast
• Background Noise

Captured Image
Camera-IR LED Conjugated Module
22
Infra-red Light Source (II)

• Non-Conjugated several LEDs
• Contrast is OK but sensitive to the location of
  screen

23
Simulation of Reflected Energy Distribution

• LED Characteristics (specification)
• Retro-Reflector Characteristics (measurement)
• Simulation

Angle of Retro-reflector Vs Intensity
Reflected Ray Distributionof Retro-Reflector
Energy Distribution of LED
24
Simulation of Reflected Energy Distribution
Incoming Energy at R from L (mW)
LED Characteristic Function
Incoming Energy at O from R (mW)
Reflector Characteristic Function
25
Infra-red Light Source (III)
26
Differential Image Capture
IR Light On
IR Light Off
Difference of Images
Processed Image
27
Locating Screen with IR-Camera Image Processing

• Calbrate the camera and remove the optical
  distortion
• Trace all the edge in captured image
• Choose the longest edge as the screen
• Estimate as a convex polygon and find vertexes
  
• Shrink Convex Polygon donw to requtabgular,
  based on the distance and angle between the
  neighbourring vertex

• Original image

Find Edge, Corners
4 Corners
28
Locating Screen with IR-Camera Image Processing

• Divide Screen edge into 4 regions using the
  result aboveChoose the longest edge as the screen
  
• Perform recursive line fitting for each region
• Calculate the crossing pointas the corner of
  screen
• Calculate the position and orientation of the
  screen in 3D

Divide into 4 region

• Line Fitting

29
Locating Screen with IR-Camera
Calculate 3D Location from 2D Image

• Screens Shape and Size
• Position of Vertices on the Image Plane

Known as the PnP (Perspective n-Points) Problem
30
Demos and Apps

• (Kondo, Miwa, Isomura, Kijima 1999-2001)

31
Demos

• Ambient Location Based Display
• Ambient Window demo (Video)
• Showcase in a wall demo (Video)
• Personalized bulletin board / schedule board
• Product Tag in Market
• Self explanation object
• Infrared filter film tag
• Manipulat-able Display Object
• Medical Image Slicer demo (Video)
• Finger Draw demo (Video)
• Copied Remote Controller
• TV Controller, Phone,

32
Demos (I) Information on Object
33
Demos (II) relative location sensing
Viewpoint Moves
Screen Moves
Finger Moves
34
Demos (II) Virtual Arthroscopy
35
Summary Future Work

• III rd Person Interface and Distributed Display
• Implementation PHMD with Infrared
  Camera Retro-Reflective Screen
• Applications / Demos
• ____________________________
• Sensing with Multiple Screen, Registration
  Framework, Sensor Fusion with Gyroscope Sensor
• Reflex PHMD to Compensate Time Delay
• IR Screen ID Tag
• Further Development of PHMD

36
Thank you