Context Aware Paper-based Review Instrument: A User Study

1
Context Aware Paper-based Review Instrument A
User Study

• Presented by Dipti Shah, Hoda Homayouni, Daniel
  Belcher

2
Agenda

• Introduction of CAPRI
• Related Works and Motivation
• Experimental Design
• Discussions

3
Objective

• Evaluating CAPRI (in terms of efficiency and
  perceptual measures) by comparing it to two other
  review methods

4
Traditional Review System

• Sorting
• Memory work
• Errors
• Time
• Collaboration

5
What is CAPRI ?

• Context Aware Paper Based Review Instrument
• Tangible and augmented user interface for
  architectural document review

6
CAPRI System

• Table top
• Plan
• Papers
• Projector
• Camera
• Computer with a software program (Java)

7
CAPRI Process

• Input Interface
• Medium Detector
• Context Interpreter
• Output Interface

8
Input Interface
9
Menu Selection
10
Grab and Move
11
Related Works

• CAPRI is an augmented and tangible environment
  for facilitating document review using design
  objects that are the fundamental computational
  elements.

12
Document Review Systems

• Document review is a cognitively challenging
  process.
• The hypertext-like aspect of CAPRI decreases
  the reviewers cognitive load. It also saves the
  reviewer time and effort in sorting and
  presenting the relevant details.

13
Document Review Systems

• Architectural documents are often reviewed by
  different people. Thus, the issue of authority
  always exist in design review process.
• CAPRI affords diverse teams to directly and
  intuitively interact with the design document.

14
Document Review SystemsRelated Works

• JANUS is based on a model of a computer supported
  design that utilizes a knowledge-based approach
  to link a graphic construction system to
  hypertext.

15
JANUS
16
Tangible User Interfaces

• Tangible User Interfaces (TUI) are interfaces
  that couple digital information to physical
  objects and environments.
• By using socially accepted media, CAPRI relies on
  natural behaviors to support a complex task.

17
Tangible User InterfacesRelated Works

• The metaDESK system focuses on the use of
  tangible objects as driving elements of HCI.

18
Augmented Reality Interfaces

• Augmented Reality (AR) is an emerging platform
  for interacting with computers, contextualizing
  virtual objects in a physical world.
• CAPRI is an interface that has almost no learning
  curve. Unifying the tasks space, it encourages
  focus on the design discussion instead of the
  mechanics of interactions.

19
Augmented Reality InterfacesRelated Works

• ATI is an interface device for distributed design
  review that fuses input and output that can be
  handled, passed back and forth, inspected
  visually, positioned relative to other objects
  and even assembled or disassembled
• with them.

20
Motivation

• Predictions based upon the CAPRI claims
• More efficient (less time, better performance,
  less errors)
• Less cognitive load
• Higher level of collaboration

21
Experimental Design Overview

• Three Conditions
• Condition 1 Paper-Based Condition (PB 1)
• Condition 2 Computer-Supported Condition (CS -2)
• Condition 3 CAPRI Condition (CP 3)
• 45 Tasks (15 per condition)
• 10-15 Subject (projected) 100-150 data points
  per condition.
• Data recorded
• Performance measures (time and error) recorded.
• Perceptual measures (post-experiment
  questionnaire) recorded.
• Analysis
• Compatible with three-way ANOVA design.
• Within and between conditions.
• Within and between subjects.
• Within and between tasks.

22
Paper-Based Condition (PB 1)

• Similar to traditional paper-based document
  review.
• Subjects may browse paper-based plots in any
  order.
• Uses A1 sized plotted drawing set (plan and
  section).
• Details are on Standard (8½11") paper.

23
Computer-Supported Condition (CS 2)

• A mix of paper-based and digital references.
• Uses the same A1 sized plotted drawing set (plan
  and section).
• Details are on a computer monitor.
• Subjects may browse details by clicking on a
  detail thumbnail to enlarge in a main browser
  window.

24
CAPRI Condition (CP 3)

• A mix of paper-based and digital references,
  but with automated linking and detail sorting.
• Uses the same A1 sized plotted drawing set (plan
  and section).
• Relevant details are projected on Standard
  (8½11") paper.
• Subjects may browse details by overlying paper on
  detail bugs.

25
Procedure

• Consent forms
• Instructions
• Training phase
• First condition (randomized)
• Instructions
• Training Phase
• Second condition (randomized)
• Instructions
• Training Phase
• Third condition (randomized)
• Post-experiment Questionnaire
• Debriefing (optional)
• Participants paid

26
Pre-experiment Consent Form

• PURPOSE OF THE STUDY
• The study is to test different methods of
  architectural document review. With the help of
  participants, we will assess the effectiveness
  and usability of three different tools of
  document review.
• STUDY PROCEDURES
• The study involves sitting, reading and executing
  certain tasks. Participant will be given a
  learning phase in which to understand the task,
  then 45 discrete questions related to
  architectural drawings will be presented.
• RISKS, STRESS, OR DISCOMFORT
• This study involves sitting for more than an
  hour, which might cause discomfort to some
  subjects. There is also a possibility that a
  participant may experience mild eye-strain while
  looking at the computer screen.
• If any of the above symptoms occur, and the
  participant does not wish to continue, they may
  alert the attending researcher and immediately
  stop the experiment without penalty.
• WRITTEN PERMISSION MUST BE GAINED BEFORE
  EXPERIMENT.

27
Instructions

• Example from CAPRI condition
• In this segment, you are asked to find answers
  to a set of questions relating to an
  architectural drawing set. The relevant details
  necessary to answer each of the questions are
  contained in the drawing set. Your task is to
  read the questions and find the answer.
• In this segment, you will use a standard paper
  drawing and pieces of standard paper to query the
  drawing for detail. When the upper right-hand
  corner of a piece of paper is placed on a bug,
  the relevant detail is projected from above onto
  the page
• Please answer all the questions as quickly and
  accurately as possible.

28
Sample Tasks

• What is the distance between the 1st step and
  the end of the hand-rail on stair 3B?
• What kind of sign is in between the two doors in
  the east side of the lobby?
• What is the width of central aisle of the
  auditorium?
• On the southern wall of the vestibule seminar
  room, what is the width of the whiteboard?

29
Controls - Learning
COUNTERBALANCED CONDITION ORDERS COUNTERBALANCED CONDITION ORDERS COUNTERBALANCED CONDITION ORDERS COUNTERBALANCED CONDITION ORDERS
First Condition Second Condition Third Condition
Subject 01 2 3 1
Subject 02 1 2 3
Subject 03 1 3 2
Subject 04 2 3 1
Subject 05 3 1 2
Subject 06 1 3 2
Subject 07 3 1 2
Subject 08 2 3 1
Subject 09 3 2 1
Subject 10 3 1 2
Subject 11 1 2 3
Subject 12 2 1 3
Subject 13 1 3 2
Subject 14 1 3 2
Subject 15 1 2 3
  KEY  
1 Paper-Based (PB) 2 Computer-Supported (CS) 3 CAPRI (CP)
30
Data Collected Performance

• Two performance measures
• Time (in milliseconds)
• Error (correct/incorrect)
• Measurement Systems
• Time Participants press a key to start the time
  trial and again when finished. Subjects may
  pause the experiment at any time, but not
  continue with tasks.
• Error Participants record their answers to
  questions in pencil on a worksheet. All errors
  are hand-checked and recorded.

31
Data Collected Perceptual

• The perceptual measures were recorded in the form
  of a subjective post-experiment questionnaire.
• All questions are standard Likert-scale rankings.
• Forced choice, no center (6-level scale)
• 4.) Locating a detail bug in the plan drawing is
  an easy task.
• 1 2 3 4
  5 6
• 1 Completely disagree
• 2 Disagree
• 3 Somewhat disagree
• Unstructured comment space will be provided at
  the end of the post-experiment questionnaire to
  record any notes about their experience or
  feedback about the interfaces.

4 Somewhat agree 5 Agree 6 Completely agree
32
Analysis Format (Sample Data)
33
Analysis Format (Sample Data)
34
Challenges

• Statistical Comparison
• Architectural data sets
• Research Methodology
• Human Subjects issues
• CAPRI interface issues
• Confound it!