CSCW Evaluation Techniques

1
CSCW Evaluation Techniques

• Presented by
• Christopher Edwards

2
Overview of Presentation

• Evaluation Techniques
• Understanding Ethnography
• Using Ethnography in CSCW
• Understanding Ethnomethodology
• Ethnomethodology and CSCW
• Technomethodology
• Conclusion

3
Evaluation Overview

• Olson and Olson.
• What are we Evaluating?
• Evaluations Techniques
• Internal/External Validity
• Conclusion

4
Characteristics of Groups

• Individuals Differ in
• Skills
• Ability
• Knowledge
• Personalities
• Motivations
• Agendas

5
Characteristics of Organizations

• System comprised of people and technology
• Social Technology
• Physical Technology
• Comprised of multiple actors
• Dependency on Communication
• Information Processing Entities

6
Characteristics of Task

• Tasks involve different types of material
• Physical, Digital or Ethereal
• Ease or Difficulty of Task
• Differ on Core Activity
• Subtasks Tightly Coupled/Loosely coupled

7
Characteristics of Environment

• Physical Environment
• Distance between Group members
• CSCW technologies designed to overcome
• Contextual Time
• When in the day the interaction occurs
• Effects on Distant Group member

8
Characteristics of Technology

• Increasingly Varied
• Technologies to Support Conversation
• Auditory/Visual
• Back channels/gestures
• Technologies to Support Shared Work
• Objects that support work
• Fit of Tool to Material

9
Process

• Technology Deployment
• How and Why
• Process Analysis
• Why outcomes were affected
• Progress of Task
• Communication process
• Examined through time scales

10
Outcomes

• The initial outcome of using technology
• Quality of work
• Measure outcomes at every level
• Group Outcome
• Organizational Productivity

11
Conceptual Framework for CSCW Studies
Group
Organization
Process
Outcomes
Task
Environment
Technology
12
Tools used to evaluate CSCW technologies
13
The Survey

• Set of questions
• Fixed Alternatives
• Statistically Analyzed
• Wording of questions problematic

14
Interview

• Structure of interview
• Formal and structured
• Unstructured
• Analysis can be complicated

15
Experimental

• Controlled Setting
• Specific Task
• Conditions
• Assignment of Participants
• Useful for making inferences about causality

16
Case Study

• Examines a single or small number of cases
• Exploratory research

17
Ethnography

• Method adopted from Anthropology
• Describing Culture
• Used originally to describe other cultures
• Misunderstood method

18
Many other Methods

• Diaries
• Analytic Field Studies
• Quasi Experimental
• Longitudinal Studies
• Historical Studies

19
Internal and External Validity
HIGH
Laboratory Experiments
Field Experiments
Level of Internal validity
Surveys
Ethnographies
LOW
HIGH
Level of External Validity
20
Conclusion to the Overview

• Different factors influence use and evaluation of
  CSCW software
• Framework of CSCW studies
• Evaluation Techniques
• Validity of Techniques

21
Short Break

• Reconvene in 3 minutes
• Upcoming Understanding Ethnography
• Using Ethnography
• Ethnography and CSCW

22
Ethnography

• Understanding Ethnography
• Sociology Adoption
• Using Ethnography
• CSCW Ethnography in Design (Hughes)

23
Understanding Ethnography

• Ethnography is loosely applied to qualitative
  research
• Home is originally from Anthropology
• Aim to describe cultural interpretation

24
Understanding Ethnography

• Understanding culture from an insiders point
  view
• Three sources of data
• Participant Observation
• Interviews
• Collection of representative artifacts

25
Sociology Adoption

• Originally used to study distant cultures
• Chicago School of Sociology
• Studies focused on exploration of groups in urban
  settings
• Cultural comparisons in USA
• Family of Ethnographic Techniques

26
Class Participation Time

• In Pairs (Saul and myself included)
• Everyone gets a Handout
• For a total of 5 Minutes (2.5 Minutes each)
• Each member of the pair (one at a time) asks the
  other questions from the sheet

27
Debrief of Class participation

• What answers were given.
• Obviously not a long term ethnography study
• Depending on your relationship to this lab
  differing perspectives
• Understanding Grouplab culture (to some extent)

28
Ethnography and CSCW

• Prominence of Ethnography in CSCW
• Insufficient attention to social context
• New problems for design of collaborative
  character of work and activities
• Ethnography and system design
• Problem of scale
• Pressure of time
• Role of the ethnographer

29
Concurrent Ethnography

• Design is influenced by on-going ethnographic
  study
• Sequenced process

Ethnographic Study
Systems Development
Debriefing Meetings
System Prototype
30
Concurrent Ethnography in action

• London Air Traffic Control Centre
• Four week Ethnography
• Each stage of fieldwork was intended to target
  designers issues
• Small research team
• What ethnography provided

31
Quick and Dirty Ethnography

• Brief Ethnographic Studies
• Duration relative to the size of the task
• Selecting aspects of work setting of importance
  to design

Outline of project Meetings
Short Focus Studies
Debriefing Meetings
Scoping Document
32
Quick and Dirty Ethnography in Action

• Ethnographic investigation of software engineers
• Challenges of Large scale setting
• Working in Industrialized Environments
• Acceptance into the setting (Key to Ethnographic
  research)

33
Evaluative Ethnography

• Ethnography used to verify formulated design
  decisions

Initial outline Design or Specification
Short Ethnographic study
Debriefing Meetings
Amended Design Or Specification
34
Evaluative Ethnography in action

• Fieldwork in Building Society
• Using research for IT developments
• Routine of work
• Finding what customers wanted
• Outlined limitations of model that had been
  proposed

35
Re-examination of previous studies

• Previous studies are re-examined to inform design
• Ethnography used for many decades
• Many studies related to work and occupation
• Can be informative

36
Re-examination in action

• Inform preliminary design of Shared Object
  Service
• Using previous Ethnographic studies on
• Social work, police work and invoice processing
  in a multi-site fast food company
• What common service should support

37
Summary of Ethnography

• Understanding Ethnography
• Ethnography and CSCW
• Uses of Ethnography
• Concurrent
• Quick and Dirty
• Evaluative
• Re-examination

38
Big Break Time 5 Minutes

• Reconvene in 5 Minutes
• Upcoming Ethnomethodology
• Understanding Ethnomethodology
• Ethnomethodology in CSCW

39
Ethnomethodology

• Understanding Ethnomethodology
• Confusing Ethnography and Ethnomethodology
• Ethnomethodology in system design
• Incorporation of Sociology and Computer Science?
  Technomethodology

40
Understanding Ethnomethodology

• Ethnomethodology literally means Peoples
  Methods
• A Shift from other Sociological Methods
• Social Life is potentially Chaotic
• Social Actors
• Members methods for making sense

41
Understanding Ethnomethodology

• Garfinkel Documentary Method
• Example of Documentary Method
• Garfinkel Indexicality
• Disrupt Technique
• Example in class

42
Understanding Ethnomethodology

• We can observe other members methods of
  construction
• Development of Conversation Analysis

43
Confusing Ethnography and Ethnomethodology

• Ethnography is a form of investigative fieldwork
• Ethnography focuses on the Members Point of
  View
• Ethnomethodology is a specific analytical
  technique

44
Confusing Ethnography and Ethnomethodology

• Confusion arises because
• Ethnomethodologist is likely to use ethnographic
  techniques
• Analytic mentality-selection of phenomena and
  topics for investigation

45
Ethnomethodology in HCI and CSCW

• Observations of work activities and interactions
  help design process
• Understanding temporal organization of activities
  and interactions and implications to design

46
Learning from Ethnomethodologists

• Division of Labour
• Field Observation conducted by ethnomethodologists
  
• Act as proxy for end users
• Hand off requirements to computer science people

47
Ethnomethodology for Critique and Design

• Ethnomethodology has provided
• Critique of the design
• Failure to support the work
• Technology doesnt allow people to engage in
  their work
• Outlines organization of work and communication
  in the real world

48
Two Paradoxes

• Paradox of system design
• Large scale activity
• Paradox of technomethodology
• Transformational nature of technology
• Analysis of practice not invention

49
Technomethodology

• Develop a stance in which ethnomethodology and
  computer science play equally significant roles
• Foundational relationships

50
Technomethodology

• Trying to exploit generalizations from
  ethnomethodology
• Abstractions from both disciplines
• Means by which such working practices arise
• Dialogical interfaces

51
Conclusion

• Understanding Ethnomethodology
• Confusing Ethnography and Ethnomethodology
• Ethnomethodology and system design
• Technomethodology

52
Conclusion of Presentation

• Overview of Evaluation Techniques used in CSCW
• Ethnography
• Ethnomethodology
• Technomethodology

53
My thoughts

• CSCW has been unfortunately caught in a battle
  between sociological techniques
• CSCW design should incorporate work context
  issues

54
It was this clear.