Gaze-Contingent Displays: Review and Current Trends

1
Gaze-Contingent DisplaysReview and Current
Trends
Adaptive Displays Conference 2004
Andrew T. Duchowski
2
Acknowledgements

• National Science Foundation
• NASA Ames
• DoD / Navy / SPAWAR Charleston
• Students
• Nathan Cournia
• Hunter Murphy
• Scott Gibson (Summer Research Internship)

3
Overview

• Brief review of Gaze-Contingent Displays (GCDs)
• Tuning of GCDs via real-time metrics
• update rates
• display / interface design
• Current trends
• GPU-programmable image processing
• Eye tracking technology state-of-the-art

4
Gaze-Contingent Displays

• Motivation
• minimize display or users attentional bandwidth
• balance displayed information against visual
  information processing capacity (via eye
  tracking)
• Generally, partition display in two distinct
  regions
• high-resolution foveal region
• low-resolution peripheral surround
• Critical concerns
• how, what, and when to display in the periphery

5
GCD Strategies

• Three approaches (low- to high-level)
• screen-based displays (manage pixels)
• model-based displays (manage graphics objects)
• Attentive User Interfaces (manage interface)
• Basic assumption
• observers real-time focus of attention coincides
  with fixation point (eye movements)
• measured unobtrusively by eye tracker

6
Attentive User Interfaces
Fig.1 AUIs (a) eyeCONTACT sensor, (b) Light
fixture with eyeCONTACT sensor, (c) eyePROXY, (d)
attentive TV (Courtesy Roel Vertegaal, see Shell
et al. (2003))
7
Model-Based Approach

• Model-Based Gaze-Contingent Applications
• manipulate geometry just prior to display
• degrade resolution of peripherally located
  objects
• Not as much progress in this area (vs.
  screen-based) despite recent advancements in
  Level Of Detail (LOD) modeling techniques
• Most relevant established technique is that of
  isotropic LOD management, as originally proposed
  by Clarke (1976)

8
Model-Based Approach

• Isotropic LOD management
• pre-computed fine-to-coarse object hierarchies
• resolution is uniformly degraded as object
  recedes from view (based on projected pixel area
  coverage)
• Isotropic LOD management not always desirable,
  especially when viewing large objects close up
• Due to advancements in multiresolution modeling,
  it is now becoming feasible to extend LOD
  approach to nonisotropic object rendering

9
Model-Based Examples
Fig.2 gaze-contingent graphics

• Luebke et al.s (2002) gaze-contingent LOD
  graphics
• OSullivan et al.s (2002) temporal graphics
  degradation

10
Model-Based Examples
Fig.3 gaze-contingent terrain, model

• Our work
• Gaze-contingent terrain generation (AAAI
  Symposium 2000)
• Gaze-contingent graphics modeling (EuroGraphics
  2001)

11
Model-Based Metrics

• Key questions is it worth it?
• how much to degrade, where?
• does degradation reduce rendering time?
• how does degradation impact user?
• Parhurst and Niebur (2004) discuss classic speed
  / accuracy tradeoff
• degradation improves display speed (interaction)
• reaction times GCDs impede target indentification

12
Screen-Based Approach

• Strategy
• manipulate framebuffer just prior to display
• periphery is often masked or smoothed to compress
  image information (bits-per-pixel)
• Good deal of progress in this area since design
  of early eye-slaved flight simulators
• Research is rooted with classic Psychology
  research on reading perception and moving
  window paradigm (McConkie and Rayner 1975)

13
Screen-Based Image Coding
(a) Recorded scanpath
(b) Reconstructed image
Fig.4 HVS-matching wavelet coefficient
scaling (Haar wavelets emphasizing degradation
effects)
14
Screen-Based Video Coding

• Bergström (2003) incorporates MAR-based visual
  acuity model into transform coders

Fig.5 The Barbara image DCT coded (left) and
MDCT coded (right). The focus point is in the
centre of the MDCT coded image.
Fig.6 Barbara coded by the DWT coder (left) and
the MDWT coder (right).
15
Screen-Based Metrics
Fig.7 Gaze-contingent multi-resolution displays

• Loschky and McConkie (2000) found that to be
  imperceptible, image changes must start within 5
  ms following end of saccade
• Parkurst et al. (2000) found visual search
  performance with 5º window comparable to uniform
  resolution display

16
Current Trends

• Good deal of work recurring in screen-based
  approaches
• introduction of arbitrary visual maps
• hardware-accelerated processing
• new applications (fisheye displays)
• New research facilitated by rapid progress in eye
  tracking technology

17
Arbitrary Visual Maps

• Perry and Geisler (2002) introduced arbitrary
  visual fields
• An important advancement since degradation
  function of any shape can be created
• Simulations of visual dysfunctions possible
  (e.g., glaucoma)

Fig.8 Arbitrary Visual Fields
18
Hardware-Accelerating Imaging

• GPU programs allow per-fragment resolution
  selection from mipmap pyramid
• For image-based GCDs, image processing bottleneck
  effectively eliminated
• Peripheral color degradation now possible

Fig.9 GPU-programmable mipmap lookup
19
New Applications
Fig.10 Pliable Display Technology (PDT) lens

• PDT lens is a type of focus plus context screen
  with focal area magnified
• Gaze-Contingent applications as yet unexplored

20
Eye Tracking Technology

• 1st generation plaster-of-paris, scleral coils
• eye-in-head measurements invasive
• 2nd generation photo- and video-oculography
• 3rd generation video-based corneal reflection
• 4th generation digital video, DSP computer
  vision algorithms (for face, eye-in-head
  detection)
• much easier to use, still as accurate and fast
• still not auto-calibrating, but getting closer

21
State-of-the-Art

• State-of-the-art eye trackers
• facilitate rapid application dev.
• Key GCD research question
• perception or performance?
• Real-time applications
• simulation, telecommunication, etc.
• Other directions
• diagnostic (off-line) uses, testing purposes, etc.

Fig.11 Tobii eye tracker
22
References

• Bergström, P. (2003). Eye-Movement Controlled
  Image Coding. PhD Dissertation (No. 831),
  Institute of Technology, Linköping University,
  Linköping, Sweden.
• Clarke, J. H. (1976). Hierarchical Geometric
  Models for Visible Surface Algorithms.
  Communications of the ACM 19, 10, 547-554.
• Danforth, R., Duchowski, A., Geist, R., McAliley
  (2000). A Platform for Gaze-Contingent Virtual
  Environments. In Smart Graphics (Papers from the
  2001 AAAI Spring Symposium, Technical Report
  SS-00-04), Menlo Park, CA, AAAI, pp. 66-70.
• Luebke, D., Reddy, M., Cohen, J., Varshney, A.,
  Watson, B., and Huebner, R. (2002). Level of
  Detail for 3D Graphics. Morgan-Kaufmann
  Publishers, San Francisco, CA.
• McConkie, G. W. and Rayner, K. (1975). The Span
  of the Effective Stimulus During a Fixation in
  Reading. Perception Psychophysics 17, 578-586.

23
References

• Murphy, H. and Duchowski, A. (2001).
  Gaze-Contingent Level Of Detail. In EuroGraphics
  (Short Presentations), Manchester, UK,
  EuroGraphics.
• OSullivan, C., Dingliana, J., and Howlett, S.
  (2002). Gaze-Contingent Algorithms for
  Interactive Graphics. In The Minds Eye
  Cognitive and Applied Aspects of Eye Movement
  Research, J. Hyöna, R. Radach, and H. Duebel,
  Eds., Elsevier Science, Oxford, England.
• Parkhurst, D., Culurciello, E., and Niebur, E.
  (2000). Evaluating Variable Resolution Displays
  with Visual Search Task Performance and Eye
  Movements. In Eye Tracking Research
  Applications Symposium p.105-109. Palm Beach
  Gardens, FL.
• Parkhurst, D. J., Niebur, E. (2004). A
  Feasibility Test for Perceptually Adaptive Level
  of Detail Rendering on Desktop Systems. In
  Applied Perception and Graphics Visualization
  (APGV). ACM, Los Angeles, CA, to appear.

24
References

• Perry, J. S. and Geisler, W. S. (2002).
  Gaze-Contingent Real-Time Simulation of Arbitrary
  Visual Fields. In Human Vision and Electronic
  Imaging, San Jose, CA, SPIE.
• Shell, J. S., Selker, T., and Vertegaal, R.
  (2003). Interacting with Groups of Computers,
  Communications of the ACM 46, 3 (March), 40-46.