A Real-Time Augmented Reality System for Industrial Tele-Training

1
A Real-Time Augmented Reality System for
IndustrialTele-Training  
Department of Computing Science University of
Alberta

•  
• Pierre Boulanger1, Nicholas Geoganas2, Xiaowei
  Zhong2, and Peiran Liu2
• 1 Department of Computing Science,
• University of Alberta, Alberta
• 2School of Information Technology and
  Engineering, University of Ottawa, Ontario

2
What is Augmented Reality?

• A combination of a real scene viewed by a user
  and a virtual scene generated by a computer that
  augments the scene with additional information.

3
What is the Goal of AR?

• To enhance a persons performance and perception
  of the world
• But, what is the ultimate goal????

4
The Ultimate Goal of AR

• Create a system such that no user CANNOT tell
  the difference between the real world and the
  virtual augmentation of it.

5
Augmented Reality vs. Virtual Reality

• Augmented Reality
• System augments the real world scene
• User maintains a sense of presence in real world
• Needs a mechanism to combine virtual and real
  worlds

• Virtual Reality
• Totally immersive environment
• Visual senses are under control of system
  (sometimes aural and proprioceptive senses too)

6
Combining the Real and Virtual Worlds

• We need
• Precise models
• Locations and optical properties of the viewer
  (or camera) and the display
• Calibration of all devices
• To combine all local coordinate systems centered
  on the devices and the objects in the scene in a
  global coordinate system

7
Combining the Real and Virtual Worlds (cont)

• Register models of all 3D objects of interest
  with their counterparts in the scene
• Track the objects over time when the user moves
  and interacts with the scene

8
Realistic Merging

• Requires
• Objects to behave in physically plausible manners
  when manipulated
• Occlusion
• Collision detection
• Shadows
• All of this requires a very detailed
  description of the physical scene

9
Components of an Augmented Reality System
10
Display Technologies

• Monitor Based
• Head Mounted Displays
• Video see-through
• Optical see-through

11
Monitor Based Augmented Reality

• Simplest available
• Little feeling of being immersed in environment

12
Optical see-through HMD
13
Video see-through HMD
14
AR For ATM Training
15
AR for ATM Maintenance Training

• Imagine some trainees in different locations are
  trained to repair an ATM switch.
• The task consists of removing a switchboard from
  an ATM switch and installing a chip on the
  switchboard.
• As an ATM switch is an expensive piece of
  equipment, only one trainee has access to it.
• The other trainees would like to participate in
  the training exercise to get some experience.
• In addition to trainees, there are some trainers
  in remote locations who are knowledgeable experts
  in the domain.

16
AR for ATM Maintenance Training

• During the training exercise, the trainee who is
  equipped with the ATM switch may need to ask a
  trainer for advice.
• For example, the trainee may contact the trainer
  to ask for general advice (How do I?) or for
  particular pieces of information (which piece of
  the switchboard should I work with? Should I
  install the chip this way or that way?).
• On the other hand, the trainer may give the
  trainees some directions or point out mistakes.

17
Problems with the Current Practice

• Costly the ATM switch is an expensive device.
  Not all of the trainees have had access to this
  device. Furthermore, errors in installation could
  lead to damage to the chip or the pins of the
  switchboard. Therefore, it could be costly to do
  a real training exercise.
• Inefficiencies Since the remote trainer is not
  always available, the trainees often have to
  interrupt the current training exercise until
  they have an opportunity to talk to a trainer in
  person.

18
Problems with the Current Practice

• Limitation of Voice-only conversations When a
  trainee needs to ask the remote trainer for
  advice, phone conversations can be helpful in
  asking for a specific type of information, but
  they are not effective in situations where an
  expert has to provide a trainee with step-by-step
  directions, depending on the situation. Sometimes
  video is indispensable in interactive
  conversation.
• Informative Support is expected A problem arises
  when trainees would like to be shown how to do
  the job but the trainers are far away or not
  available. It is not informative for a remote
  trainer to tell a trainee via voice conversation
  to pull the switchboard in this direction in
  this way.
• Less hands-on experience The trainees who are
  not equipped with ATM switches and other
  facilities are not able to carry out an effective
  training exercise.

19
AR System Workflow For Training
20
Binary Markers vs Augmentation
21
Coordinates frames in our calibration procedure
22
Camera Calibration Finding TCT
The relationships between the camera screen
coordinates, the camera coordinates and the
calibration target coordinates can be
represented as where P represents the
perspective transformation, Tct represents the
translation and rotation transformation from the
calibration target coordinates to the camera
coordinates and C is the transformation matrix
obtained by combining P and Tct.
23
Calibration Parameters Solved Using ShapeCapture
4.0
Lens Aberrations
24
Binary Square Marker
The 7 bits code of the pattern can identify 27
different objects.
25
Image Preprocessing

• Image binarization. The program uses an adaptive
  threshold to binarize the video image. Binary
  images contain only the important information,
  and can be processed very rapidly.
• Selecting the quadrilateral regions. These
  regions become candidates for the square marker
• Searching and recording the four corners of the
  quadrilateral region found in step (2).

26
Pattern Recognition 
(1)    The system recognizes the 2D binary code
pattern inside the square marker region. (2)   
Extracting the binary code   Our algorithm is
based on the following observation Given the
four vertices of the marker region, the
projection of every point on the marker can be
computed as a linear combination of the
projections of the four vertices.
27
Marker Coding
The reference frame is defined as a
non-Euclidean, affine frame. In the system, the
marker pattern is defined as a 4x4 matrix. Every
grid in the matrix represents one bit of the
binary code of the marker. The whole pattern is
in black and white color. The grid in black
represents 1, and the grid in white represents 0
28
POSITION AND POSE ESTIMATION OF MARKERS
29
POSITION AND POSE ESTIMATION OF MARKERS
When two parallel sides of a square marker are
projected on the image, the equations of those
line segments in the camera screen coordinates
are the following
30
POSITION AND POSE ESTIMATION OF MARKERS
For each of markers, the value of these
parameters has been already obtained in the
line-fitting process. Given the perspective
projection matrix P that is obtained by the
camera calibration then
31
Final OpenGL Matrix
Given that normal vectors of these planes are n1
and n2 respectively, the direction vector of
parallel two sides of the square is given by the
outer product n1xn2.
32
Current Collaboration Functions

•  
• Audio Conference 
• Remote Presence
• Remote Manipulation
• Remote Pointing
• Image Freezing

33
Performance Issues

• Augmented Reality systems are expected
• To run in real-time so that the user can move
  around freely in the environment
• Show a properly rendered augmented image
• Therefore, two performance criteria are placed on
  the system
• Update rate for generating the augmented image
• Accuracy of the registration of the real and
  virtual image

34
Limitations for Updating the Generated Images

• Must be at 10 times/second
• More photorealistic graphics rendering
• Current technology does not support fully lit,
  shaded and ray-traced images of complex scenes

35
Design Implementation
36
AR ATM Training Session
37
Augmented Reality Demo
38
Multimedia Information Rendering

• Currently, our system is implemented to embed
  the following multimedia information into the
  real world from the users view.
• Text, such as character description of physical
  objects and instruction for performing physical
  tasks in form of annotation.
• Audio, such as speech instructions.
• Image.
• Real-time video stream.
• 3D model, such as OpenGL model and VRML model.

39
Problems with Current Implementation

• The experimental results show that a wearable
  computer without 3D hardware capacity, for
  instance the Xybernaut Mobile IV wearable
  computer, is not able to satisfy the requirements
  of a real-time AR application.
• A powerful wearable computer with a good 3D
  accelerator is recommended. A possible
  alternative could be a powerful laptop carried by
  the user in a backpack and equipped with a NVIDIA
  GetForce4 graphic card and an I-glass see-through
  stereoscopic color Head Mounted Display mounted
  with a camera.

40
Conclusion

• In this presentation we have addressed the
  problem of identifying a great number of real
  world objects with a robust marking approach by
  using computer vision AR techniques.
• The binary code error detection and correction
  functions used in the marker recognition
  algorithm make the algorithm more robust to
  lighting conditions.
• We also presented a prototype developed for
  industrial Tele-training which features augmented
  reality and collaboration.

41
What Next Collaborative AR over Satellite
Connection
Anik E2
University of Ottawa Ottawa
University of Alberta Edmonton
Encoder H323
Decoder H323
Network Controller
Encoder H323
Decoder H323
Network Controller
Side By Side MPX
Side By Side MPX
Control PC
Control PC
DTI Glassless Stereo Display
DTI Glassless Stereo Display