IT 3803 v

1
IT 3803vår 2006

• Evaluering
• Brukbarhet
• CSCW/samarbeid

2
Evaluering og anvendelse Hill, L. et al. (1997)
User Evaluation Summary of the Methodologies and
Results for the    Alexandria Digital Library,
University of California, Santa Barbara. In ASIS
'97. The Annual Meeting of the American
Association for Information Science.  
http//www.asis.org/annual-97/alexia.htm
Buttenfield, Barbara. Usability Evaluation of
Digital Libraries. I David Stern (ed)
Philosophies, technical design considerations,
and example scenarios. 21s. Norbert Fuhr,
Preben Hansen, Michael Mabe, Andras Micsik,
Ingeborg Sølvberg. Digital Libraries A generic
classification and evaluation scheme. Proc. 5th
European Conference on Research and Advanced
Technology for Digital Libraries (ECDL2001).
Lecture Notes in Computer Science Vol.2163,
Springer Verlag, 2001. pp.187-199.
http//www.springerlink.de/link.asp?idw22qmkk09nc
y59av Saracevic, T. (2000). Digital Library
Evaluation Toward Evolution of Concepts. Library
Trends, 49, (2) 350-369. Special issue on
Evaluation of Digital Libraries.
http//www.scils.rutgers.edu/tefko/LibraryTrends2
000.pdf   David M. Nichols and Michael B.
Twidale. Computer supported cooperative work and
libraries , http//www.comp.lancs.ac.uk/computing/
research/cseg/projects/ariadne/docs/vine.html
Also in Vine, 109, 10-15, (special issue on
Virtual communities and information services).
3
EvalueringSaracevic, Covi Challenges for
Digital Library EvaluationFuhr et.al Digital
Libraries A generic Classification and
Evaluation SchemeDiv. ADL/ADEPT papersog andre

• Hvorfor evaluere?
• Hva kan/skal evalueres?
• Hvordan kan/skal en evaluere?
• Hvem trenger resultatene?
• Når skal en evaluere?

4
Evaluering av Alexandria Digital LibraryLinda
Hill et.al.

• Evaluering av et system under utvikling, stadig
• interaksjon med sluttbrukere. Og dette er ikke et
  
• evaluerings-prosjekt, men prosjekt for
• Innpassing av geospatial søking i eksisterende
  IR-modeller og utvidelse av disse
• Datateknisk forskning
• Innvirkning av bruker-studier på design og
  implementasjon av romlige digitale bibliotek

5
Terminologi

• Gazetteer A list of geographic names, together
  with references to their locations and other
  descriptive information.
• Footprint (Location) Each information bearing
  object (IBO) is represented by latitude and
  longitude coordinates in the following order.

6
Foiler ADL

• Her skal det være en kort beskrivelse av ADEPT

7
Evaluering av Alexandria Digital LibraryLinda
Hill et.al.

• I ADL tre måter å finne informasjon
• Map Browser kan zoome (kart, bilder)
• Gazetteer finner stedsnavn områder med
  spesielle attributter
• Katalog finner dokumenter innen (tilnærmet)
  samme geografiske områder

8
Valgte evaluerings-metoder

• Spørreskjema
• Utvalgte brukergrupper
• Etnografiske studier
• Demografiske analyser
• Analyser av Log-filer

9
Hvorfor velge metoder disse metodene?

• Tilgjengelig ekspertise for de valgte metodene
• Mulig innenfor tilgjengelige tids- og
  kostnadsrammer
• Maksimere den forventede gevinst av innsats for
  deltagere og ansatte
• Velge metoder som utfyller hverandre

10
Demografiske analyser

• Navn, E-mail, Organisasjon, Stilling, Referanse
  (MÅ utfylles ved første innlogging)
• 2287 registrerte beta-testere, 906 ulike
  IP-adresser
• De fleste oppdaget ADL via Web
• Ca. 25 via presentasjoner, besøk og lignende.
  UCSB
• Forskere utgjorde 42
• Deltagere fra 49 land, 65 fra USA

11
Spørreskjema

• Mål
• Innhente detaljert og direkte tilbakemelding im
  brukernes erfaring med brukergrensesnittet
• Finne ut Hvem er brukerne av ADL
• Sammenheng mellom brukers erfaring med systemet
  og brukers bakgrunn
• gt 96 brukbare, utfylte spørreskjema

12
Etnografiske studierutført av Pedagoger

• Lydbåndopptak av referanse-intervju (13stk.)
• Video-opptak av ADL-brukere (8 stk.)
• Resultat avhenger av bakgrunnskunnskap data
  (plattform, www, system-design, tidl. bruk av
  ADL, bibliotek søkestrategier), fag-kunnskap
  (kart, geografi)
• Sammensatte analyser kombinasjoner av
  spørreskjema og online skjema, og denetnografiske
  undersøkelsen

13

• Etnografisk
• gir en bred forståelse for rollemønster og
• effekter av praksis i et videre sosial eller
  organisatorisk
• rammeverk
• Sosiologisk fokuserer på sosiale krefter og
  effekter
• HER System tilnærming gt ulike former for ytelse

14
Utvalgte brukergrupper

• Heldagsmøter (2 stk.) med representanter for
• utvalgte brukergrupper
• Fagspesialister (geografi og lignende.)
• Informasjons-spesialister
• Lærere
• Resultater
• Karaktertrekk (beskrivelser) av brukergruppene
• Brukerkrav, design-ideer, scenarier
• Hvordan gjennomføre denne typen møter

15
Hva lærte en?

• Om brukerne?
• Selve evalueringen og tilnærmingsmåten?
• Om ADL brukergrensesnitt?
• Funksjonalitet og innholdet i ADL?

16
Selve evalueringen og tilnærmingsmåten?

• Står resultatet i forhold til innsats/kostnader?
• Informasjon som i kvalitet og kvantitet kan
  brukes i modellering og utvikling av DL?
• Kan prosjektets begrensede evaluerings-ressurser
  brukes bedre?
• Hvordan tilbakeføre vår nye forståelse inn i vårt
  videre arbeid?
• Hva er viktigst "technology push, requirements
  drag" eller resultater av evaluering (i forhold
  til utviklingshastighet?)

17
DELOS NoE

• Digital Library Research Forum
• Workshops, ECDL, forskerutveksling
• Digital Library Evaluation Forum
• Digital Library Test Suite
• Multilingual and Cross-Language System Evaluation
  infrastructure
• Standardization Forum
• Training and Technology Transfer Forum
• International Cooperation Forum

18
Utkast. Områder og aktører for forskning (DELOS,
2.1)
19
Spørsmål

• Hva?
• Samlinger, teknologi, brukbarhet,
• Hva og hvordan måles?
• Effektivitet, brukbarhet, ..
• Hvem trenger resultatene fra evaluering?
• Systemutviklere, forskning, samlingsutviklere,
• Når skal en evaluere?
• Utviklingsfasen, ferdige DL, programvarekomponente
  r,,,

20
Proposed DL Classification and Metrics

• Classification
• Users/uses
• user/uses
• internal, general, education, professional,
  research
• domain (subject area)
• info encounter
• object seeking, browsing
• purpose
• consume, analyse, synthesize

• Metric
• Users/uses
• user/uses
• number
• distribution
• domain
• distribution
• info encounter
• distribution
• purpose
• distribution

21
Proposed DL Classification and Metrics

• Classification
• Data/Collection
• content
• none/partial/full
• audio, video, text
• 2D/3D
• meta-content
• biblio
• indexing/thesaurus /classification
• citation
• management
• rights, work flow, user management, maintenance

• Metric
• Data/Collection
• content
• diversity, age, size, quality (white/grey
  literature)
• meta-content
• media
• level of detail
• management
• doc. age, growth rate, immediacy, completeness,
  maintenance intervals

22
(No Transcript)
23
(No Transcript)
24
Conclusions Next Steps

• DL evaluation needs a broad and user-centred view
• new DL evaluation scheme
• a hierarchy of user-based metrics
• meta library of DL collections
• http//www.sztaki.hu/delos_wg21/metalib/
• extension of the meta library
• agenda setting for DL research

25

• Digital Library Classification and Evaluation A
  Publishers View
• Of the Work of the DELOS Evaluation Forum
• Michael Mabe
• Elsevier Science
• Delos Workshop, Budapest June 2002

26
Overview

• Applying Strategic Marketing
• Market Segmentation and Classification
• DELOS Evaluation Forum
• market-based classification
• metrics and evaluation
• development of the Meta-Library
• Helping shape the future agenda

27
Strategic Marketing DL Evaluation

• How can these possibly be related?
• Business is customer-centred
• the customer is king
• First Commandment of Marketing
• WHO is buying WHAT and WHY

28
(No Transcript)
29
Strategic Marketing Cycle
Segment Market
Decide on Benchmarks
Annual Planning Cycle
Evaluate Performance
Take Actions
Analyse Outcomes
30
Market segmentation

• Most important aspect of strategic marketing
• Market segments must be
• highly similar within
• distinct from other segments
• described in terms of use or purchase
• reachable
• Accuracy of segmentation affects success TQM and
  thus of the business

31
Market Segmentation
WHO is using WHAT and WHY

• WHOdemographic socioeconomicbrand
  loyaltyheavy/light userlifestyle

• WHAT( HOW)productplacepriceoutletservices

• WHYbenefitsattitudesperceptionspreferences

Customers dont buy products they seek to
acquire benefits
32
(No Transcript)
33
DL Segmentation Classification
WHO is using WHAT, HOW and WHY

• WHO
• users
• demographics
• interests (subject)
• approaches

• WHAT
• data/collection
• description
• management

• HOW
• technology
• user techn.
• info. tools
• system techn.
• document techn.

• WHY
• uses (purpose)
• encounter type

WHO and WHY predetermine WHAT and HOW
34
Utkast. Områder og aktører for forskning (DELOS,
2.1)
35
Market Segmentation and DL Metrics

• A market segment must be
• highly similar within
• distinct from other segments
• described in terms of use or purchase
• reachable
• A DL evaluation metric
• independent of other metrics
• relevant
• measurable

36
A Goal of the ADEPT Project

• integrate ADEPT into undergraduate education in
  disciplines that might make use of
  georeferenced/geospatial resources

37
ADEPT Project Organization

• Implementation team How ADEPT is constructed
• Education and Evaluation team How ADEPT is used
• Formative evaluation Conduct needs assessments
  as input to the design process
• Iterative design Assess ADEPT usability in
  learning situations and provide feedback to the
  design process

38
(No Transcript)
39
(No Transcript)
40
Over-arching research question

• Can digital library services that provide
  instructors and students with the means to
  discover, manipulate, and display dynamic
  geographical processes contribute positively to
  undergraduate instruction and to the development
  of scientific reasoning skills?
• What is the best way to teach science to students?

41
Studying digital libraries in context

• Educational applications of digital libraries
• Deliver content to classrooms, labs, dorms,
  offices, homes
• Facilitate instruction
• Provide primary and secondary resources
• Provide services to construct lectures, labs,
  lessons
• Provide learning environment for students to
  utilize digital resources
• Provide access to digital content for research
  and teaching
• University infrastructure for digital libraries
• Supplement library resources and services
• Supplement instructional development resources
  and services

42
Geography as domain of study

• A scholarly discipline that
• Relies on a rich array of primary data sources
• Maps
• Satellite Observations
• Remote Sensing
• Physical observations
• Studies dynamic processes
• Models
• Simulations

43
(No Transcript)
44
Educational goals in geography

• Systems thinking
• interaction of earth processes and atmospheric
  processes
• Scientific thinking
• observation, hypothesis construction, hypothesis
  testing
• Acquiring geographic knowledge
• asking geographic questions
• acquiring geographic information
• organizing geographic information
• analyzing geographic information
• answering geographic questions

45
Desired outcomes of ADEPT project

• Instructors and teaching assistants
• Promote scientific thinking in students
• Incorporate primary sources in instruction
• Use digital library services to enrich
  instruction
• Students
• Become more active learners
• Learn to think like scientists
• locating relevant information
• balancing evidence
• synthesizing knowledge
• testing hypotheses
• developing their own conclusions

46
Research Methods

• Classroom-based studies
• Faculty and Teaching assistant interviews
• Classroom observation
• Student performance and demographic data
• Student interviews
• Laboratory-based studies
• Assessment of mental models and scientific
  thinking processes
• Office-based studies
• Faculty interviews about information seeking

47
ADEPT instruction scenario

• Topic river networks
• Instructor prepares class lecture with ADEPT
• Discovers relevant geographic objects
• Constructs metadata to describe objects for
  personal and shared use
• Annotates objects as necessary
• Integrates objects into Iscapes (personal digital
  libraries)
• Instructor presents lecture to students using
  ADEPT
• Teaching assistants review selected topics in lab
  sessions using ADEPT
• Students use ADEPT for lab exercises and to study
  for exams

48
(No Transcript)
49
(No Transcript)
50
ADEPT Education and Evaluation The Course
Topography Hypothesis Pre-ADEPT a high number
of concepts, with sparse number of associations
among concepts, with a reliance on textbooks and
other low quality instructional
aids. Post-ADEPT fewer topics explored in
greater depth, with a comparatively richer set of
associations among concepts, using primary
scientific evidence for illustration.
51
ADEPT Education and Evaluation Course
Topography Purpose Detecting the impact of
ADEPT in classrooms. The course topography
method will provide baseline data
for assessing the impact of
ADEPT. Approach a concept mapping method
adapted for the evaluation of
instruction.
52

• ADEPT Education and Evaluation
• Course Topography
• Domain concepts,
• documents and instructional materials,
• assessment instruments,
• other instructional materials (e.g., homework
  assignments),
• and the relationships that connect them
• are modeled as first-class objects. Specific
  instances of these objects are expressed as nodes
  within a large graph.

53

• ADEPT Education and Evaluation
• Examples of Concepts
• Instances of concepts in geography, for example,
  include
• climate
• orbital forcing
• seasonal change
• Rough definition an instructor typically
  introduces a new concept every 5-10 minutes

54

• ADEPT Education and Evaluation
• Examples of Documents
• Instances of documents include
• specific texts (both traditional and Web
  accessible),
• examples of the critical arguments (Popper 1979,
  p. 107) that express the domain concepts and are
  the vehicle of their conveyance
• learning standards

55

• ADEPT Education and Evaluation
• Describe the now
• Accomplishments to date qualitative and
  quantitative data collection
• Developed classroom data collection instrument
• Developed course topographies via classroom
  observations (extended concept maps)
• Interviewed faculty on pedagogical goals
• Collected baseline student demographic data and
  conducted interviews on pedagogical issues
• Created rapid prototype modules on hydrological
  processes
• Determined how faculty use personal collections
  in instruction

56
ADEPT Education and Evaluation to
understand the future Through comparative
evaluation, we will discover how ADEPT
affects Scientific thinking Student
outcomes Teaching strategies
57
ADEPT Education and Evaluation Current
Effort Topography of Geography Courses Concept
mapping A discipline represented as a
fine-grained set of inter-related and sequenced
concepts Beyond concepts our observations
uncover learning materials instructional
activities Concepts LM IA Course Topography
58
ADEPT Education and Evaluation Examples of
Relationships Instances of relationships include
- Logical (often hierarchical) relations among
concepts as stated or implied by the instructor-
Sequential relations as concepts are described by
the instructor- Associations among artifacts
(documents, instructional materials) and
concepts
59
Tradisjonelle evalueringskriterier (1/2)

• Tradisjonelle bibliotek
• samlinger hensikt, grensesetting, autoritet,
  målgruppe, kostnad, format, preservering, ,,,
• informasjon nøyaktighet, egnethet, lenker,
  representasjon, sammenlignbarhet, presentasjon,,
• bruk tilgjengelighet (fysisk og psykisk),
  søkbarhet, brukervennlighet,,,
• standarder for en rekke elementer og prosesser

60
Tradisjonelle evalueringskriterier (2/2)

• Tradisjonelle IR kriterier
• relevans (måler presisjon og recall)
• tilfredshet, suksess,,,
• Tradisjonelle HCI / brukergrensesnitt kriterier
• brukbarhet, funksjonalitet, innsats,,,
• tilpassing til oppgaven, feil,,

61
Multifaceted Evaluation Design
G. Marchionini, UNC-CH
62
CSCW What is it?

• CSCW is an identifiable research field focused
  on the role of the computer in group work
• (Greif, 1988)
• CSCW should be conceived as an endeavor to
  understand the nature and the characteristics of
  cooperative work with the objective of designing
  adequate computer-based technologies.
• (Bannon and Schmidt, 1991)

63
Groupware What is it?

• A multi-user program that lets the members of a
  distributed group work together by
• Providing group members with communication
  facilities,
• Letting them share their files and data,
• Making them aware of each others existence.
• Groupware is the product (program) resulting from
  the research done in CSCW field.

64
Groupware or CSCW?

• CSCW
• Focus on
• workplace activities,
• organizational impact of technology,
• co-evolution of the technology and the groups
  using it,
• Interdisciplinary Social scientists and
  technologists.
• Groupware
• Focus on
• product,
• the design of the product,
• Mainly a technical discipline.

65
Cooperation is complex!

• Groupware is different from single-user
  applications
• Affects directly the social context of the work,
• Requires broad acceptance,
• Groupware faces new challenges with respect to
  mainframe systems
• Mandating adoption is often not possible,
• It may appear in less hierarchic and more
  egalitarian, settings (with all the demands that
  this imposes!)

66
(No Transcript)
67
A taxonomy of Groupware (Johansen, 1988)
Time
Same
Different
Place
Meeting facilitation
Workflow over Intranet
Same
Tele/video desktop conferencing
Different
Electronic mail
68
Cooperative work Definitions

• ...multiple individuals working together in a
  planned way in the same production process or in
  different but connected production processes.
  Marx (1867),
• All work is essentially cooperative, in that it
  depends on others for its successful
  performance. (Ehn 1988)
• The interdependence of multiple actors who
  interact through changing the state of a common
  field of work. (Divitini et al. 1996).

69
Groupware technology categories

• Communication media space, video conferencing,
  e-mail, ...
• Information sharing whiteboards, meeting
  facilitation,
• Coordination calendar, scheduler, workflow
• Most of the systems cover more than one category!!

70
Co-located
meeting rooms
teamrooms
Synchronous
Asynchronous
organizational memory workflow web based
applications collaborative writing
distributed meetings shared drawing video
conferencing collaborative writing
Remote
71
CSCW and Digital Libraries

• Libraries are meeting placeswhere
  collaborations can and do happen
• (Levy Marshall, 1994)

72
Consequences of seeing work as collaborative

• A digital library needs to provide tools
  supporting collaboration and communication among
  1) users, 2) librarians and 3) across the two
  groups.
• Examples are
• collaborative annotation, construction and
  maintenance of local sub-collections (work
  spaces), e-mail, on-line recommendation, ...

73
Collaboration in Digital Libraries (1)

• Groupware can be used for supporting DL
• in the same vein that any (distributed)
  organization (meeting, collaboration among
  distributed members, both synchronous
  asynchronous)
• in the DL specific activities (collection
  creation, document searching, etc.)

74
Collaboration in Digital Libraries (2)

• Strategic collaboration among different
  organizations for sharing resources or
  information, unite collections and/or catalogues,
  
• Every-day collaboration ad-hoc collaboration
  , normally for short period of time
• between library staff,
• between library staff and users,
• between library users.

75
Eksempler CSCW

• Same time, same place meeting support tools
• Same time, different place video conferencing
• Different time, same place a design teams
  shared room containing specialist equipment
• Different time, different place email systems

76
Discussion Beyond collecting ...

• Possible services that a DL can provide includes
  support for the following activities
• locating and selecting among relevant resources,
• retrieving information from them,
• interpreting what was retrieved,
• managing the filtered-out information locally,
• sharing results with others.

77
Searching ...

• generally requires browsing a variety of
  information sources, often starting from needs
  that are initially vague and evolve during the
  search process.
• is seen as an individual activity, but social
  aspects are essential (for example, in the form
  of consults).

78
The role of technology

• Technology can support the existing interactions
  and make it possible new ones reducing the
  problems connected to geographic distribution
• Different types of support could be provided for
  the sharing of the results and searching processes

79
Collaboration during searching

• Ask explicitly for information to a colleague
  that could know the answer (or how to get one)
• Help voluntarily someone that is facing a
  specific problem
• Pass information to someone that could find it
  interesting
• ...

80
In conventional libraries ...

• Most of the interactions in searching are
  co-located and synchronous.
• How can digital libraries modify this? They seem
  to change interactions into remote/asynchronous.
• Which support should be provided?

81
A typology of cooperative activities

• (in conventional and digital libraries)
• collaboration through interactions during the
  searching process
• collaboration through the sharing of the search
  product
• collaboration through the sharing of the
  searching process.

82
Interaction during the searching

• Do you know? electronic mail
• Does anyone know? newsgroups
• Who might know? active support of a groupware
  system keeping user profiles
• brainstorming GroupSystems, shared whiteboards,
  ...

83
Sharing the search product

• By recommendation agents for identifying people
  that could be interested in the result of the
  research
• By annotation groupware supporting the
  annotation of documents (possibly identifying the
  source of the annotation)
• By rating groupware allowing to easily rate the
  found documents.

84
Sharing the search process

• Saving and visualizing the search this can be
  used by someone else for eventually correct
  mistakes or avoid to follow the same pattern of
  search
• a support can be provided for
• attach comments to relevant points
• editing and correcting the search path
• communicating with/to other users,
• ...

85
personal help reference interview issue of book
loan face - to - face interactions user education
(taught)
notice boards letters post - it notes
memos documents for study user education
(reference material)
Co-located
Synchronous
Asynchronous
use of OPACs database search video
conferencing telephone
social information filtering email, voice
email distance learning postal services
Remote
86
Reference

• BROWSING IS A COLLABORATIVE PROCESS by MICHAEL B.
  TWIDALE , DAVID M. NICHOLS and CHRIS D. PAICE,
  available at
• http//www.comp.lancs.ac.uk/computing/research/cse
  g/projects/ariadne/docs/bcp.html