Focus Group Methodology

1
Focus Group Methodology

• Five focus groups science educators (n 38)
• K-5, 6-12 (inservice and preservice group),
  undergraduate faculty (two groups)
• Instructed to provide advice to DLESE and NSDL on
  policies, priorities, and best practices for
  building useful digital collections for classroom
  use
• Each focus group lasted between 60 and 90
  minutes
• Briefed on DLESE and NSDL, and demo of DLESE
  collection
• Briefed on relationships between the World Wide
  Web, the DLESE Broad Collection, and the DLESE
  Reviewed Collection
• Explanation of the 3Ps, how to develop
  importance categories
• Each person directed to an individual computer
  and given 20 - 30 minutes to evaluate 4
  educational web sites
• Getting to resources from bookmarks, not DLESE
• Group discussion - contribute the positive and
  negative criteria, then 3Ps

2
Evaluation Rubric
3
Results (1) - Policies, Priorities, Practices

• Scientific accuracy (policy, all groups but k-5)
• Complement, dont replicate traditional materials
  (priority, all groups)
• Libraries should label biased sites in the
  item-level metadata (all groups except K-5
  either policy or priority)
• I know that you know
• Library policies or priorities should limit the
  accessioning of sites with advertising (all
  groups)
• Discrete advertising, preferably related to
  scientific or educational products, that does not
  detract from learning (i.e. no pop-ups)
• Frustrated with current state of web resource
  design (all groups)
• All sites had usability problems, but
  particularly ambiguously rated (0-1) sites
• Presence of Distractions - Student time on task a
  concern across all grade levels

4
Results (2) - Calibration, Added Value

• Gestalt ratings - people agree more on good
  resources (lower stdev) than bad (higher stdev)
  
• Consistency between rankings of teachers and
  DLESE collections expert
• Remarkable consistency in recommendations across
  middle, high, and undergrad groups
• K-5 have very different concerns
• Participants expect library metadata to add value
  over the content of resources, not just to
  summarize
• High expectations for NSF-funded projects
• Noting bias and classroom uses (but how?)
• Accurate and more precise designation of grade
  levels (and standards)
• K-5 reading level often assumed to be too high
• Many grade ranges too broad

5
Recommendations - from JCDL

• Shift from growing the collection quickly to
  growing it selectively
• Higher standard of selection criteria for
  collections accessioning and funded cataloging
  efforts (distractions, advertising,
  interactivity)
• Tightly couple collections development and
  library use
• Co-development model provides context for design
  improvements, assessment, generating contextual
  information and reviewing
• Library school district content provider
  partnerships
• Reviewing
• Identifying the problematic is as important as
  the good

6
Designing for Quality

• Role of mediating tools such as digital libraries
  
• Models unaided human mind lt-gt informational site
• Predictive and Evaluative Judgments (Rieh 2002)

• Relevance of search results
• Metadata descriptions
• Resources
• Annotations and reviews

Library systems, policies, work processes