2nd Joined Advanced Student School

1
2nd Joined Advanced Student School

• Ubiquitous Tracking based AR-Setups
• Benjamin Fingerle
• Christian Wachinger

2
Procedure

• We will present a scenario in which a user -
  Gerhard - gets into the benefits of an AR-enabled
  Environment
• First the scenario will be presented from the
  users perspective
• In a 2nd stage well look behind the surface and
  see how the AR-environment can be modeled using
  the DWARF Service concept
• Finally well go even deeper into detail and
  present how spatial relationships of objects can
  be represented using the Ubiquitous Tracking
  Framework

3
The Users Equipment

• Gerhard wears an Optical See Through Head Mounted
  Display (OST-HMD)
• Up on this HMD two 6DOF markers - one optical,
  one magnetic - are rigidly mounted
• Additionally a stereo-vision camera is installed
  on top of his head
• Gerhard wears special gloves with attached
  optical 6DOF markers

4
A Day in Gerhards Life

• Gerhard strolls down the TUM hallway greeting
  colleagues of his through closed doors
• Eventually he reaches his own office and steps in
• Sitting down in front of the desk he reads a
  virtual message of his Russian friend Vladimir
  from St. Petersburg asking for a chess game
• He starts the RemoteChess application whereupon a
  virtual chess board appears aligned on the desk
  and the virtual counterpart of Vladimir takes a
  seat across from him

5
A Day in Gerhards Life

• Suddenly an emergency request for instant help by
  his befriended Spanish colleague José shows up
• José asks Gerhard - a renowned surgeon - for
  advises regarding a difficult surgery José is
  currently conducting
• Just where a minute ago the chess board was
  visible a 3D - Model showing the patient crops up
• Gerhard studies the patient while having a look
  from different sides and getting computer
  tomography images registered on the patient on
  demand
• In vivid discussion with José a life is saved

6
The Scenario Can Be Modeled With Services

• Services
• Have needs
• Refined by predicates
• Offer abilities
• Refined by attributes
• Connectors
• Offer interfaces for data exchange
• Service Managers
• One for each network node
• Detect mutually satisfying services
• Provide services with connectors

ltservice nameOpticalTrackergt ltneed
namevideo typeVideoStreamgt ltconnector
protocolsharedMemorygt lt/needgt ltneed
namemarker typeMarkerDatagt ltconnector
protocolObjectReferencegt lt/needgt ltability
namemarkerPose typePoseDatagt ltattribute
namelocation value(marker.location)gt lt
attribute nameidentity value(marker.ident
ity)gt ltconnector protocolNotificationPushgt
lt/abilitygt lt/servicegt
Context location, identity, activity, time
can be modeled using Predicates and Attributes
7
Services are Specified Using XML

• ltservice nameHMDOpticalTrackergt
• ltneed name
• ...
• lt/needgt
• ltability nameHeadPose typePoseDatagt
• ltattribute namelocation
• value(landmark.location)gt
• ltattribute nameidentity
• valueGerhardgt
• ltconnector protocolNotificationPushgt
• lt/abilitygt
• ltability namemarkerPose typePoseDatagt
• ...
• lt/abilitygt
• lt/servicegt

8
Hallway Services
Service Name Needs Abilities
GerhardConfigData -- landmarkDescription
HallwayConfigData -- landmarkDescription
HMDCamera -- videostream
HMDOpticalTracker videostream markerPose, HMDPose
HMDVideoShow videostream --
WhatsBehind pose videostream
ContextEstimator landmark context
9
Office Services
Service Name Needs Abilities
GerhardConfigData -- landmarkDescription
HMDCamera -- videostream
HMDOpticalTracker videostream markerPose, HMDPose
HMDVideoShow videostream --
ContextEstimator landmark context
RoomConfigData -- landmarkDescription
RoomCamera -- videostream
roomMagneticTracker landmark markerPose
roomOpticalTracker videostream markerPose
Table 3D-content --
VirtualChess chessPartner 3D-content
VirtualSurgery handPose 3D-content
VirtualCommunication communicationPartner 3D-content
10
Mutually Satisfying Services are Found by
ServiceManager

• For each network node (e.g. Room, Hall, ) one
  ServiceManager exists
• This ServiceManager observes needs and abilities
  of those services belonging to his particular
  network node
• When a match is found the ServiceManager provides
  all involved parties with connector-objects
• These connectors offer push-, pull-, or shared
  memory access to abilities via various interfaces

11
Matching of Mutually Satisfying Services
12
Ubiquitous Tracking - The Formal Layer

• The crucial problem of Augmented Reality
  applications is the correct tracking of objects
• It is common to integrate the tracking procedure
  into the application
• Different tracking technologies are combined to
  get better results
• To release the AR application of tracking and to
  enable a seamless integration of new tracking
  devices a formal layer is introduced
• The formal framework, called UbiquitousTracking,
  forms the formal layer

13
Ubiquitous Tracking The Formal Framework

• Requests to the framework about the spatial
  relationship of objects can be send
• The answer delivers the optimal relationship
  available

• Graph-model
• nodes objects
• edges spatial relationships

14
Ubiquitous Tracking - The Underlying Graph Model

• Properties of a spatial relationship
• Represents the transformation and translation of
  the source coordinate system to the target
  coordinate system
• Attributes characterizing the quality of the
  relationship
• Three different types of graphs
• Real relationship graph
• Measured relationship graph
• Inferred Relationship

15
Real Relationship Graph

• Each pair of objects has at every point of time a
  geometric relationship

16
Measured Relationship Graph

• Estimates of relationships are just available for
  certain objects for discrete points of time
• Additionally Attributes characterizing the
  quality of the measurement exist
• Error function describing
• the quality of a relationship

17
Inferred Relationship Graph

• Knowledge about spatial relationships not just
  for discrete points of time is necessary
• Knowledge has to be inferred about the
  relationship of objects
• Error functions help to find optimal inferences

18
A Possible Graph in our Scenario
RoomCam
Landmark
HeadCam
Drawing on th flipchart Add. Edges for magnetic
tracker
19
What Is a Good Error Function?

• Aim Finding optimal paths in the graph!
• Latency,
• Update frequency
• Confidence value
• Pose accuray
• Time to live

20
Optimisation of Graph Algorithms

• Precomputing the Data Flow Graphs
• Infrequently changing structure of the spatial
  relationships
• Infrequently changing Attributes
• No dependency on the pose measurements
• Grouping Nodes
• Several nodes can be represented by a single
  supernode
• Enabling level of detail hierarchies
• ? Faster graph search
• What is the coordinate system of the new
  supernode?

21
Security Safety Aspects

• What if the WhatsBehind Service can be used in
  front of every door?
• What if your boss knows what you have done this
  weekend?
• What if other users know about every place where
  you have been this week?
• What if Indra the intern uses the AR setup of
  Gerhard?
• What if Eve corrupts the data showing you the
  false distance to the oncoming bus?
• What if you cant see the car because the latest
  sports news occluded it?

Privacy Restrictions Authenticity
22
Finding Matches Forms Crucial Performance Issue

• Matching problem of high computational complexity
• One Service Manager for each Network Node
• Number of potential matchings grows exponentially
  with the number of services available inhibiting
  scalability
• Different strategies for coping with this issue
  could include
• Heuristics, based on context information,
  realized as graph searches on the spatial
  relationship graph
• Interpretation of matching problem as predicate
  logic formula, applying specialized algorithms
  known from model checking

23
Issues Concerning Graph Representation

• Possible ways of storing the spatial relationship
  graph include
• One graph-service holding the complete Graph
• all graph algorithms applicable- contradicts
  the distributed computing paradigm
• Each Service knows its adjacency
• complies with distributed computing paradigm
• - not all graph algorithms applicable

24
Issues Concerning Access to the Graph Information

• Possible ways of accessing spatial relationships
  include
• Request formulated as need for spatialRelationship
  with predicates source and target, abilities
  offered by
• Certain GraphInformationServices
• Graph can be split up between different such
  services
• - different graph search strategies for the same
  relation cannot coexist
• One service for each relationship instantiated by
  special relationShipGenerators
• relationships instantly available
• - different graph search strategies for the same
  relation cannot coexist
• - number of services raises dramatically
• Requesting a graphInformerObject providing an
  interface getSpatialRelationShip(source, target)
• Objects implementing different graph searches
  can coexist
• - centralization

25
Open Questions

• How much information about the world do we have
  to put in the error function?
• Which graph algorithms have to be applicable?
• Which graph representation allows theses
  algorithms?
• How to perform context changes?
• How to enable new users to enter AR environments?
• What has then to be calibrated?

26
Critical success factors

• Number of AR ready buildings
• Development and acceptance of standards
• Prices and convenience
• Legal issues
• Products and applications urging the user to buy