AugmentedReality

1
Augmented-Reality

• Ping Gai
• HFE 760

2
Augmented-Reality

• Augmented- Reality Definition
• Augmented Reality vs. Virtual Reality
• Visual Display Systems for AR
• Video Keying and Image Registration
• System Design Issues
• Augmented Reality Application

3
Augmented Reality Definition

• Augmented Reality is a growing area in virtual
  reality area.
• An Augmented Reality system generates a composite
  view for the user. Its a combination of the real
  scene viewed by the user and a virtual scene
  generated by the computer that augments the scene
  generated by the computer that augmented the
  scene with additional information.

4
Augmented Reality Definition

• Typically, the real-world visual scene in an AR
  display is captured by video or directly viewed.
• Most current AR displays are designed using
  see-through HMDs which allow the observer to view
  the real world directly with the naked eye.
• If video is used to capture the real world, one
  may use either an opaque HMD or screen-based
  system to view the scene.

5
AR vs. VR

• Virtual Reality a computer generated,
  interactive, three-dimensional environment in
  which a person is immersed.(Aukstakanis and
  Blatner, 1992)
• Virtual Environment is a computer generated three
  dimensional scene which requires high performance
  computer graphics to provide an adequate level
  of realism.
• The virtual world is interactive. A user requires
  real-time response from the system to be able to
  interact with it in an effective manner.
• The user is immersed in this virtual environment.

6
AR vs . VR

• VR the user is completely immersed in an
  artificial world and becomes divorced from the
  real environment. The generated world consists
  entirely of computer graphics.

7
AR vs. AR

• VR strives for a totally immersive environment.
  The visual, and in some systems aural and sense
  are under control of the system.
• In contrast, an AR system is augmenting the real
  world sense of presence in that world. The
  virtual images are merged with the real view to
  create the augmented display.

8
AR vs. VR

• For some applications , it may be desirable to
  use as much as possible real world in the scene
  rather creating a new scene using computer
  imagery. For example, in medical applications,
  the physician must view the patient to perform
  surgery, in telerobotics the operator must view
  the remote scene in order to perform tasks.

9
AR vs. VR

• A main motivation for the use of AR relates to
  the computational resources necessary to generate
  and update computer-generated scene. In VR, The
  more complex the scene, the more computational
  resource needed to render the scene.
• AR can maintain the high-level of detail and
  realistic shading that one finds in the real
  world.

10
AR vs. VR

• NO simulator sickness. Vertigo, dizziness
  introduced by sensory mismatch within display
  environment can be a problem when one uses an HMD
  to view a virtual world.
• If the task is to show an annotation to the real
  world.

11
Visual Display System for AR

• Hardware for display visual images
• A position and orientation sensing system
• Hardware for combining the computer graphics and
  video images into one signal
• The associated system software

12
Visual Display System for AR

• There are two main ways in which the real world
  and the computer generated imagery may be
  combined to form an augmented scene.
• Direct viewing of the real world with overlaid
  computer generated imagery as an enhancement.In
  this case, the the real world and the CG images
  are combined optically.
• Combining the camera-captured video of the real
  world with CG imagery viewed using either an
  opaque HMD, or a screen-based display system.

13
Visual Display System for AR

• Two basic types of AR system
• Opaque HMD or screen-based AR.
• These systems can be used to view local or remote
  video views of real world scenes, combined with
  overlaid CG.The viewing of a remote scene is an
  integral component of telepresence applications.
• Transparent HMD AR.
• This system allows the observer to view the real
  world directly using half-silvered mirrors with
  CG electronically composited into the image. An
  advantage id that the real-world can be directly
  viewed and manipulated.

14
Visual Display System for AR
15
Video Keying

• Relevant when an opaque HMD with video input is
  used to create an AR scene. Video and synthetic
  image are mixed using a video keyer to form an
  integrated scene.
• Video Keying is a process that is widely used in
  television, film production and CG. (weather
  report)

16
Video Keying

• When using video keying to design AR scenes, one
  signal contains the foreground image and the
  other one contains the background image. The
  keyer combines the two signal to produce a
  combined video which is then sent to the display
  device.

17
Video Keying

• Keying can be done using composite or component
  video signals.
• A composite video signal contains information
  about color, luminance, and synchronization, thus
  combining three piece of information into one
  signal.
• With component video, luminance synchronization
  are combined, but chroma information is delivered
  separately.

18
Video Keying

• Chroma keying involves specifying a desired
  foreground key color. Foreground areas containing
  the keying color are then electronically replaced
  with the background image. This results in the
  background image being replaced with the fore
  ground image in areas where the background image
  contains chroma color.
• Blue is typically used for chroma keying
  (Chromakey blue) rarely shows up in human skin
  tones.

19
Video Keying

• If a video image of the real world is chosen as
  the foreground image, parts of the scene that
  should show the computer-generated world are
  rendered blue.
• In contrast, if video of the real world is chosen
  as the background image, the computer generated
  environment will be located in the foreground.

20
Video Keying
21
Video Keying

• A luminance keyer works in a similar manner to a
  chroma keyer, however, a luminance keyer combines
  the background image wherever the luminance
  values are below a certain threshold.
• Luminance and chroma keyers both accomplish the
  same function but usa of a chroma keyer can
  result in a sharper key and has greater
  flexibility, whereas a luminance keyer is
  typically lower resolution and had less
  flexibility.

22
Z-keying
23
Z-keying

• Figure is a schema of the z-key method. The
  z-key method requires images with both depth
  information (depth map) as inputs. The z-key
  switch compares depth information of two images
  for each pixel, and connects output to the image
  which is the nearer one to the camera. The result
  of this is that real and virtual objects can
  occlude each other correctly. This kind of
  merging is impossible by the chroma-key method,
  even if it is accompanied with some other
  positioning devices such as magnetic or acoustic
  sensor, since these devices provide only a gross
  measurement of position.

24
Image Registration

• Its required that the computer generated images
  accurately register with the surroundings in the
  real world. In certain applications, image
  registration is crucial.
• In terms of developing scenes for AR displays,
  the problem of image registration, or positioning
  of the synthetic objects within the scene in
  relation to real objects, is both a difficult and
  important technical problem to solve.

25
Image Registration

• With applications that require close
  registration, accurate depth information has to
  be retrieved from the real world in order to
  carry out the calibration of the real and
  synthetic environments. Without an accurate
  knowledge of the geometry of the real world and
  computer-generated scene, exact registration is
  not possible.

26
System Design Issues

• Frame rate, update rate, system delays, and the
  range and sensitivity of the tracking sensors.
• Frame rate is a hardware-controlled variable
  determining the number of images presented to the
  eye per second. AR displays which show stereo
  images alternatively to the left and right eye
  typically use a scan rate doubler to transmit 120
  frames per second so that each eye has an
  effective frame rate of 60 Hz.

27

System Design Issues

• Update rate of the display is the rate at which
  new images are presented to the viewer.
• With a low update rate, if the user using an AR
  display moves his head, the real and
  computer-generated images will no longer be
  registered until the next update. Small errors in
  registration are easily detectable by the visual
  system.
• What limits the update rate is the relationship
  between the complexity of the scene and the
  computational power of the computer system used
  to generate the scene. This relationship is esp.
  important for computationally intensive
  applications such as medical imaging.

28
System Design Issues

• The lag in image generation and tracking is
  noticeable in all HMDs but is dramatically
  accentuated with see-through HMDs. This is an
  crucial problem if exact image registration is
  required.
• There are two types of system delays which will
  affect performance in AR computational and
  sensor delays.
• As the complexity of the CG image increases, the
  computational delay is a major factor determining
  the update of a display.
• In addition, sensor delay, the time requires
  updating the display, is an important variable in
  determining performance in augmented reality.
• Many HMD-based systems have combined latencies
  over 100ms, which become very noticeable.

29
System Design Issues

• Sensor sensitivity
• The head-tracking requirements for AR displays.
• A tracker must be accurate to a small fraction of
  a degree in orientation and a few millimeters in
  position.
• Errors in head orientation(pitch, roll, yaw)
  affect image registration more so than error in
  position(x, y, z), leading to the more stringent
  requirements for head-orientation tracking.
• Positional tracking errors of no more than 1 to 2
  mm are maximum for AR system.

30
System Design Issues

• In addition to visual factors, cognitive factors
  should be considered in the design as well.
• Users of systems form mental models of the system
  they interact with and the mental model they form
  influence their performance.
• With AR displays the designer must take into
  account two mental models of the environment, the
  mental model of the synthetic imagery and of the
  real image.
• The challenge will be to integrate the two
  stimuli in such a way that a single mental model
  will be formed of the augmented scene.

31
System Design Issues
Integrated Mental Model
Mental Model of real envoronment
Mental Model of synthetic envoronment
Virtual world stimuli Auditory, haptic, visual
Real world stimuli Auditory, haptic, visual
32
Augmented-Reality Application

• Medical
• Entertainment
• Military Training
• Engineering Design
• Robotics and Telerobotics
• Manufacture, Maintenance and Repair
• Consumer Design

33
Augmented-Reality Application

• Medical
• Most of the medical applications deal with image
  guided surgery. Pre-operative imaging studies,
  such as CT or MRI scans, of the patient provide
  the surgeon with the necessary view of the
  internal anatomy. From these images the surgery
  is planned. Visualization of the path through the
  anatomy to the affected area where, for example,
  a tumor must be removed is done by first creating
  a 3D model from the multiple views and slices in
  the preoperative study. AR can be applied so that
  the surgical team can see the CT or MRI data
  correctly registered on the patient in the
  operation theater while the procedure is
  progressing. Being able to correctly register the
  images at the point will enhance the performance
  of the surgical team and eliminate the need for
  the painful and cumbersome stereotactic frames
  currently used for registration.

34
Augmented-Reality Application
35
Augmented-Reality Application

• Entertainment
• Weather report
• Virtual studio
• Movie special effect
• Advertisement

36
Augmented-Reality Application

• Military Training
• The military has been using display in cockpits
  that present information to the pilot on the
  windshield of the cockpit or the visor of their
  flight helmet. This is a form fo AR display.

37
Augmented-Reality Application

• Engineering Design
• Distributed Coollaberation
• Product visualizatoin

The scenario for this application consists of an
office manager who is working with an interior
designer on the layout of a room. The office
manager intends to order furniture for the room.
On a computer monitor the pair see a picture of
the room from the viewpoint of the camera. By
interacting with various manufacturers over a
network, they select furniture by querying
databases using a graphical paradigm. The system
provides descriptions and pictures of furniture
that is available from the various manufactures
who have made models available in their
databases. Pieces or groups of furniture that
meet certain requirements such as colour,
manufacturer, or price may be requested. The
users choose pieces from this "electronic
catalogue" and 3D renderings of this furniture
appear on the monitor along with the view of the
room. The furniture is positioned using a 3D
mouse. Furniture can be deleted, added, and
rearranged until the users are satisfied with the
result they view these pieces on the monitor as
they would appear in the actual room. As they
move the camera they can see the furnished room
from different points of view.
38
Augmented-Reality Application

• Robotics and Telerobotics

39
Augmented-Reality Application

• Manufacturing, Maintenance, and Repair
• One application area that is currently being
  explored involves mechanical maintenance and
  repair. In this scenario a mechanic is assisted
  by an AR system while examining and repairing a
  complex engine. The system may present a variety
  of information to the mechanic. Annotations may
  identify the name of parts, describe their
  function, or present other important information
  like maintenance or manufacturing records. AR may
  lead the mechanic through a specific task by
  highlighting parts that must be sequentially
  removed and showing the path of extraction. The
  system may also provide safety information. Parts
  that are hot or electrified can be highlighted to
  constantly remind the mechanic of the danger of
  touching them. The mechanic may also be assisted
  by a remote expert who can control what
  information is displayed on the mechanic's AR
  system.

40
Augmented-Reality Application

• Consumer Design
• House Design
• Fashion, beauty industry
• .

41
Reference

• http//www.cs.rit.edu/jrv/research/ar/
• Virtual Environments and Advanced Interface
  Design, edited by Woodrow Barfield, Thomas
  A.Furness III

42

• Augmented Reality Sites - North America
• MIT
• Image Guided Surgery home page
• Intelligent Room project
• J P Mellor's home page
• Media Lab Wearable Computer page
• CMU
• Z-Key project
• Magic Eye project
• Columbia University
• Virtual Worlds research
• Architectural Anatomy
• University of North Carolina - Chapel Hill
• Ultrasound Visualization Research
• Hybrid Tracking Research
• Latency in Augmented Reality
• Ronald Azuma's Augmented Reality page
• Telepresence Research Group
• Rich Holloway's Home Page