Hold that Thought

1
Hold that Thought!

• Why I care about automating the capture of
  everyday experiences
• Gregory D. Abowd
• College of Computing GVU Center
• Georgia Tech

2
AcknowledgmentsAll in the Family

• Intellectual
• Kurt Stirewalt, Anind Dey, Jen Mankoff, Jason
  Brotherton, Bob Waters, Heather Richter, Khai
  Truong, Lonnie Harvel, Kris Nagel, Giovanni
  Iachello, Jay Summet, Gillian Hayes, Shwetak
  Patel, Julie Kientz, Mario Romero, Zhigang Hua,
  Eric Stuntebeck, Lamar Gardere, Matthias Gauger,
  Andreas Lachenmann, Sebastian Boring, Arwa
  Tyebkhan, Roman Savaryn
• Blood
• Richard G. Abowd, Jr., John, Aidan Blaise
• Financial
• NSF, NiCHD, Intel, Cure Autism Now Foundation,
  Organization for Autism Research, Nokia

3
High-level message

• The challenges and opportunities of ubiquitous
  computing are best understood through
  experimentation in living laboratories exploring
  everyday use.
• My personal life provides many significant
  challenges for ubicomp.
• This is fun, motivating, significant and HARD
  computing research.

4
Outline

• Automated capture and ubicomp
• Three examples
• Portable retrospective memory aid
• Sharing family memories
• Monitoring and detecting developmental
  disabilities

5
Georgia Techs Version of Ubicomp (since 1995)

• Application-driven ubicomp, situated in living
  laboratories
• Computationally enhanced environments where
  people do real activities.
• Classroom, office, home, body
• Using a human need to motivate a technology
  innovation.

6
Hold that Thought!

• A Common Theme
• Many interesting services rely on automated
  capture of everyday activity, with useful
  interfaces to access and replay those memories.

7
Not a New Idea

• Vannevar Bushs 1945 vision
• the memex.
• Mark Weisers 1991 vision
• Sal scenarios, Tivoli project
• Leverage computer as recorder
• Allow the human to
• engage in activity
• synthesize experience

8
Think Globally, Act Locally

• Three case studies of capture, motivated by
  personal experiences.
• Concrete applications, done carefully, can lead
  to larger research issues.
• Personal relevance, carefully restrained, can
  speed to an interesting design point.

9
Example 1The Personal Audio Loop

• Has this ever happened to you?
• Sorry about thatuh, what were we talking
  about?
• An interesting design for adoption challenge
• When/where does it occur?
• Is there a useful and practical solution?
• What are the social/legal ramifications?

10
Of Prototypes and Paratypes

• Mobile HCI 2004, IEEE Pervasive Computing
  magazine 2005, CHI 2005, CHI 2006, Giovanni
  Iachellos 2006 thesis

11
Example 2RGA Action Films

• Richard G. Abowd, Jr. was a handsome fellow.

He married Sara, a buddys sister.
He was an 8mm film hobbyist.
On Nov. 19, 1998, he died, leaving behind a
30-year archive of home movies.
That Christmas, his faithful projector also died.
12
The Design Challenge

• Create a system to annotate home movies with
  relevant metadata to facilitate browsing and
  searching.
• Effective symbiosis between manual and automated
  techniques.
• Control/accuracy vs. speed
• (Multimedia Information Retrieval 2003)

13
The Family Video Archive
14
The Family Video Archive
15
Metadata Matters

• Albums reflect content with common theme.
• Automatically author a DVD using metadata to
  support navigation.
• How to gather metadata automatically, while
  recording live?(Ubicomp 2004)

16
Why Bother?

• Cheap way to give meaningful presents
• Quick way to embarrass relatives
• John Maron Abowd, this is your life!
• But, of course, there is more

17
From Whimsical to Medical

• One evening, I was converting video from 1998
• I was shocked at what I saw.
• Automated capture has taken on a whole new
  meaning for me.

Aidan Blaise Abowd Christmas 2002 Australia
2005
18
Initial Problem

• The science behind intervention therapies for
  children with special needs.
• It is hard to track the impact of therapy on
  these children.

19
Opportunity for Automated Capture

• Diagnosis and monitoring of CWA can be augmented
• Automate data collection
• Simplify access to relevant data
• Facilitate communication among care teams
• Improve data collection reliability
• Provide data that could not otherwise be observed

Hayes, et al. Designing Capture Applications to
Support the Education of Children with Autism.
Ubicomp '04
20
Four Thrusts

• Abaris
• Helps caregivers use real data to assess progress
  in structured interventions
• CareLog
• Capture of data in unstructured activities in the
  natural environment
• Auto Recognition of Autistic Behaviors
• Detecting and analyzing non-observable data
• Early Detection
• Proactive system for alerting new parents of the
  symptoms of developmental delay

21
Julie Kientz

• A tool to support discrete trial therapy, a
  popular form of ABA in homes and schools
• Allows therapists to review videos of sessions to
  make data-based decisions on the progress of the
  child during meetings

Kientz, et al. Abaris Evaluating Automated
Capture Applied to Structured Autism
Interventions. Ubicomp 05.
Kientz, et al. From the War Room to the Living
Room Decision support for home-based therapy
teams. CSCW 06.
22
Discrete Trial Training Therapy

• Team of therapists teach basic life skills
• e.g. Word pronunciation
• One-on-one setting with one therapist and child
• Many repeated trials with a specific protocol
• Each trial starts with an exact, spoken command
• e.g. Say school bus
• Each trial ends with a written grade
• e.g. Grade I means child said school bus
  independently
• Very data-intensive
• Current system is paper-based and time consuming

23
Discrete Trial Training Therapy
24
Abaris Capture

• Leverages basic DTT protocol and minimizes
  intrusion

25
Abaris - Access
26
Abaris - Access
27
Abaris Deployment

• 4-month pilot study in the home of one child and
  his therapy team
• 4 therapists, 1 lead therapist, and 1 consultant
• Abaris was used in
• 52 therapy sessions
• 3869 trials
• 45.1 hours of recorded video
• 6 team meetings

28
ResultsUbicomp 2005, CSCW 2006 under review

• Capture interfered minimally in teaching sessions
• Time spent with child increased
• Indexing good enough Anoto most effective
• Video clarified confusion and ensure consistency
  amongst therapists

29
Next step for Abaris

• Deployment at the Experimental Education Unit in
  the Center for Human Development and Disability
  at the University of Washington.
• Hypothesis More frequent data-based
  decision-making by therapists improves results of
  DTT and generalization.

30
Lamar Gardere
Ellen Matthews

• Designed to collect rich behavioral data in the
  natural environment
• Includes
• Video clips of relevant events
• Simple remote triggering
• Tagging and reviewing of individual clips
• Graphing and querying of results over time

31
Selective Archiving of Captured Experiences

• Recording devices (e.g., cameras, microphones)
  embedded in an environment
• Always on and available but
• Require explicit user action to store anything

Hayes Abowd, Tensions in Designing Capture
Technologies for an Evidence-Based Care
Community, CHI06. Hayes Truong, Using
Wearable Devices to Take Advantage of
Environmental Services. IEEE Pervasive Computer
Magazine, 2005. Hayes, et al. Experience
Buffers A Socially Appropriate, Selective
Archiving Tool for Evidence-Based Care. CHI05
32
Illustration by Khai Truong
33
Illustration by Khai Truong
34
Different applications

• Home
• Capture and share meaningful moments with family,
  friends, doctors, teachers, therapists
• Schools
• Support direct observation of behavioral problems
• Both ideas being pursued commercially as
  Behavioral Imaging (BI) tools

35
CareLog in Schools
36
Controlled Study in School

• Use CareLog
• Custom system designed for FBAs
• Uses experience buffers architecture
• 4 teacher participants, 8 CWA participants
• 2 conditions
• Traditional pen and paper vs.CareLog
• Time to saturation, quality of analysis

Hayes, et al. Submission in preparatin for
CHI07
37
Giving a voice to CWA
Tracy Westeyn

• Some observations are hard
• e.g., self-stimulatory behaviors
• Could this be automated?

Westeyn, et al. Recognizing Mimicked Autistic
SelfStimulatory Behaviors Using HMMs. ISWC 05
38
Research Question

• Can we use sensors (in environment or on the
  body) to recognize stimming behavior?

39
Preliminary results

• Yes, we can detect stimming episodes in
  continuous stream of sensor data.
• But, this hasnt been done on real subjects yet.
• And even if we could, why would this approach
  matter?

40
Early Detection of Developmental Delay

• The earlier autism is detected, the more of an
  impact therapies can have on the childs
  development
• Most diagnoses are
• made several years after the
• first warning signs
• How can we help new parents look for the warning
  signs so they will be able to seek treatment as
  soon as possible?
• This applies beyond autism.

41
A Special Lab The Aware Home
42
A Goal

• Create technologies cheap enough to deploy in any
  home that can help track developmental progress
  for all children.
• Think about baby calendars and baby monitors and
  how they can be leveraged and improved.

Ubicomp 06 Patel, Truong Abowd, Powerline
positioning A practical sub-room-level indoor
location system for domestic use. Patel et al.,
Farther than you may think An empirical
investigation of the proximity of users to their
mobile phones.
43
Conclusions

• The ubicomp vision is best understood through
  experimentation in living laboratories exploring
  everyday use.
• My personal life (yours?) provides many
  significant challenges for ubicomp.
• This is fun, motivating, significant and HARD
  computing research.

44
Thank you!

• For more information
• http//www.gregoryabowd.com
• http//www.awarehome.gatech.edu
• http//home.cc.gatech.edu/autism
• http//www.caringtechnologies.com