Are They Really Getting It

1
Are They Really Getting It?

• The Evolution of Information Literacy Assessment
  for First Year Students
• Sarah Burns
• sburns_at_pace.edu
• Pace University Library
• Pleasantville, New York
• Online Presentation Materials
• http//www.pace.edu/library/pages/instruct/firstye
  ar.html

2
Goals for the Session

• To share the plan we use to assess Information
  Literacy skills in first-year students
• To hear your ideas
• To hopefully provide each session participant
  with at least one new idea

3
Pace University

• 14,000 students
• 9,000 undergrad
• 5,000 graduate
• 1,000 full and part-time faculty
• Multi-campus, urban and suburban university
• 500 new students in Westchester County
• 750 new students in New York City

4
Pace University

• Lubin School of Business
• Lienhard School of Nursing
• School of Education
• School of Computer Science Information Systems
• Dyson College of Arts Sciences
• School of Law

5
Information Literacy at Pace

• Included as one of the twelve student learning
  objectives of the New Core Curriculum
• Information Literacy and Research - Locate,
  evaluate and make efficient and ethical use of
  information resources.

6
Information Literacy at Pace

• Course-embedded library instruction sessions
• First Year Library Instruction program
• UNV 101 Freshmen Seminar
• ENG 101 English I
• ENG 102 English II

7
Learning Objectives

• Information Literacy Competency Standards for
  Higher Education
• Objectives for Information Literacy Instruction
  A Model Statement for Academic Librarians
• http//www.ala.org/acrl

8
Assessment Methods

• Library Experience Survey
• Sample Syllabus matching exercise
• Session evaluations
• One minute paper exercise
• Multiple-choice exercise
• Portfolio review
• Faculty evaluations

9
UNV 101 Fall 1999

• Instructional activity
• Students work in small groups, are given a
  Research Scenario, and explore the Library home
  page to determine how they would start the
  research for their given Scenario. Groups report
  back on the steps they took.
• No assessment tools in place

10
UNV 101 Fall 2000

• Library Experience Survey introduced
• Students self-report on previous library skills
  and experiences
• Sample Syllabus introduced
• Students match items on a syllabus with buttons
  on the Library home page

11
UNV 101 Fall 2000

• Library Experience Survey results
• 26 report no OPAC experience
• 16 report no OPAC instruction
• 65 had used a library database for articles
• 97 had used the Internet
• 53 agreed Everything is on the Web.
• 93 are comfortable with a Web browser
• 95 are comfortable asking a librarian for help

12
UNV 101 Fall 2000

• Sample Syllabus results
• 83 knew to use the Catalog to find a book
• 58 knew where to locate Reserves
• 42 knew where to locate articles in magazines or
  newspapers
• 12 knew where to locate articles in scholarly
  journals (Bad question!)
• 41 knew where to locate MLA citation help

13
UNV 101 Fall 2001

• Change to Library Experience Survey
• Changed from Yes/No to more options
• Sample Syllabus revisions
• Italicized, made bold and added arrows to draw
  attention to what they should look for
• Removed question on reading the New York Times
  every day too many options were correct

14
UNV 101 Fall 2001

• Changes to Instruction
• Scenarios were revised to focus the students
  choices more
• Revised one Scenario to specifically address
  locating Reserve materials
• Revised one Scenario to address locating help
  with MLA format

15
UNV 101 Fall 2001

• Library Experience Survey results
• 30 report no OPAC experience (up 4)
• 48 report no OPAC instruction (up 32)

16
UNV 101 Fall 2001

• Sample Syllabus results
• 80 knew to use the Catalog to find a book (down
  3)
• 53 knew where to locate Reserves (down 5)
• 59 knew where to locate articles in magazines or
  newspapers (up 17)
• 45 knew where to locate articles in scholarly
  journals (up 33)
• 68 knew where to locate MLA citation help (up
  27)

17
UNV 101 Fall 2002

• Changes to Instruction
• Complete revision of Research Scenarios now
  called Research Exercises
• Reduced to seven Exercises, each with one
  specific objective
• Created two Exercises for Reserve materials
• More structured questions to guide students in
  pursuing the items and reporting back to the
  class.

18
UNV 101 Fall 2002

• Changes to Library Experience Survey
• Added question What was the first thing you did
  to begin research for a paper in high school?
• Added question What method did you use to search
  the Internet?
• Changes to Sample Syllabus
• Switched item on Reserve materials from an
  article to a book too many correct options for
  locating an article

19
UNV 101 Fall 2002

• Library Experience Survey results
• 48 report no OPAC experience (up 18)
• 54 report no OPAC instruction (up 6)
• 57 had used a library database for articles
  (down 9)
• 64 agreed Everything is on the Web.
• (up 9)

20
UNV 101 Fall 2002

• Sample Syllabus results
• 79 knew to use the Catalog to find a book (down
  1)
• 75 knew where to locate Reserves (up 22)
• 61 knew where to locate articles in magazines or
  newspapers (up 2)
• 44 knew where to locate articles in scholarly
  journals (down 1)
• 68 knew where to locate MLA citation help

21
UNV 101 - Summary

• Weakest area is knowing where to locate articles
  in scholarly journals
• Knowing to use the Catalog to find a book is
  dropping just slightly
• Use of online catalogs is dropping
• Use of databases is dropping slightly

22
ENG 102 Spring 2001

• Two-hour session
• Demonstration of online catalog, literature
  databases, types of citations, hands-on use of
  online catalog and databases, small-group
  evaluation and mini-presentation of reference
  tools
• Multiple-choice assessment exercise

23
ENG 102 Spring 2001

• Multiple Choice Exercise results
• 39 knew the catalog contained items owned by the
  library
• 29 knew journal articles were not in the library
  catalog
• 39 knew how to search for books about an author
• 47 could interpret a periodical holdings record
  from the catalog

24
ENG 102 Spring 2001

• 74 knew to search a library database to locate
  an article from a periodical
• 62 recognized the value of a journal article
• 63 knew what an abstract was
• 67 knew what a citation was
• 74 knew what a periodical was

25
ENG 102 Spring 2002

• Changes to Multiple choice exercise
• Changed question 2 reserve materials choice to
  videos
• Piloted online with a small number of classes
• Changes to Instructional materials
• Incorporated a Research Log into the session, as
  we knew the ENG 102 classes would be doing
  Portfolio Review at the end of the semester

26
ENG 102 Spring 2002

• Multiple Choice Exercise results
• 39 knew the catalog contained items owned by the
  library
• 23 knew journal articles were not in the library
  catalog (down 6)
• 48 knew how to search for books about an author
  (up 9)
• 48 could interpret a periodical holdings record
  from the catalog (up 1)

27
ENG 102 Spring 2002

• 80 knew to search a library database to locate
  an article from a periodical (up 3)
• 72 recognized the value of a journal article (up
  10)
• 62 knew what an abstract was (down 1)
• 75 knew what a citation was (up 8)
• 81 knew what a periodical was (up 7)

28
ENG 102 Spring 2002

• Portfolio Review
• 48 student research papers and bibliographies
  reviewed
• Rated holistically following system the English
  department used to rate the entire portfolio
• 1poor, 2good, 3excellent
• Criteria based on an existing Research Paper
  Rubric and the English department criteria

29
ENG 102 Spring 2002

• Portfolio Review results
• 13, or 27 of portfolios received 1
• 22, or 45 of portfolios received 2
• 13, or 27 of portfolios received 3
• Saw some excellent resources (scholarly,
  credible, timely) and some not so excellent
  resources (SparkNotes, etc.) cited
• Saw big problems with MLA format
• Inconsistent mention of research or the Library
  in reflective statements

30
ENG 102 Spring 2003

• Changes to Multiple Choice Exercise
• Changed question 2 videos to books that you
  can check out and books to reference books
• All sessions completed the exercise online

31
ENG 102 Spring 2003

• Changes to instructional activities
• Pre-session visit to all classes
• Revised Research Log included examples and
  instructions
• Added a worksheet with examples for use of MLA
  citation format reviewed format for citing
  articles from a library database
• Students were given IL reflective questions and
  asked to reflect on research in their reflective
  statements

32
ENG 102 Spring 2003

• Multiple Choice Exercise results
• 50 knew the catalog contained items owned by the
  library (up 11)
• 38 knew journal articles were not in the library
  catalog (up 15)
• 53 knew how to search for books about an author
  (up 5)
• 50 could interpret a periodical holdings record
  from the catalog (up 2)

33
ENG 102 Spring 2003

• 83 knew to search a library database to locate
  an article from a periodical (up 3)
• 72 recognized the value of a journal article
• 62 knew what an abstract was
• 79 knew what a citation was (up 4)
• 86 knew what a periodical was (up 5)

34
ENG 102 Spring 2003

• Portfolio Review results
• 40 portfolios reviewed
• 6, or 15, of portfolios received 1 (down 12)
• 22, or 55, of portfolios received 2 (up 10)
• 12, or 30 of portfolios received 3 (up 3)

35
ENG 102 Spring 2003

• Portfolio Review results
• We saw vast improvement in citing resources using
  MLA format
• Almost all students included reflection on
  research
• Students reported on their research plans, and
  how they had to revise them some indicated the
  value of outside sources to provide evidence for
  their arguments many reported the research paper
  was the item of which they were most proud

36
ENG 102 - Summary

• We are very happy with the apparent improvement
  in MLA citation skills
• Need to improve our results for the first four
  questions, which measure if students know
• The catalog contains items owned by the library
• Journal articles are not in the library catalog
• How to search for books about an author
• How to interpret a periodical holdings record
  from the catalog

37
ENG 101

• ENG 101 sessions have been offered since 1996,
  but starting reaching 100 of sections in 2000
• Instructional activities
• Introduction to brainstorming search terms,
  combining them using Boolean operators, demo of
  online catalog and interdisciplinary database,
  hands-on searching, and a worksheet they complete
  during the session, and is theirs to keep

38
ENG 101 - 1997

• No outcomes assessment tool in place
• Students asked to complete a session evaluation
  designed to measure qualitative items
• Students are asked to use a Likert scale to rate
  aspects of the instruction, including session
  contents and librarian performance

39
ENG 101 - 1998

• Added questions to check for validity
• Was the session too long? too short?
• Added questions to determine if students had
  received previous library instruction
• Asked them if they had learned concepts in the
  current session that they could integrate into
  their research process, and what were they?

40
ENG 101 2001 and 2002

• Removed some of the Likert scale questions
• Revised question on concepts learned to One
  thing I learned in todays session that I will
  use in another class or research project is

41
ENG 101 - 2002

• Tallied results of Likert scale questions for the
  first time
• For all questions except the two dealing with
  length of the session, majority of students
  either Agreed or Strongly Agreed
• For the questions dealing with length of session,
  most students disagreed that it was too long or
  it was too short.

42
ENG 101 - 2002

• Portfolio Review
• Of 33 portfolios reviewed
• 9, or 27, were assigned a value of 1
• 20, or 61, were assigned a value of 2
• 4, or 12, were assigned a value of 3
• About split between credible and non-credible web
  sites
• About half the students reported using a library
  database to locate articles

43
ENG 101 2002

• Variation on the one minute paper during
  pre-session visit to ENG 102
• Students asked to explain the difference between
  the library catalog and databases
• 22 of students could explain the difference
• 27 could half explain got one right
• 12 explained them incorrectly
• 11 did not know

44
ENG 101 - Summary

• Still work to do on outcomes assessment
• Qualitative evaluations are valuable to the
  individual librarians and help them to improve
  their instructional methods and materials

45
Faculty Evaluations

• Most faculty Agree or Strongly Agree with
  statements regarding librarian performance
• Most agree the length was adequate
• Most agree that the session included the needed
  skills and information
• Most agree the session was at the appropriate
  level

46
Faculty Evaluations

• Most disagree essential points were missing
• Most found handouts useful
• Things found most useful were
• Hands on time, web evaluation, MLA citation,
  accessing databases
• Suggestions for improvement
• More time for CAP classes, go slower, repeat
  more, more hands-on time

47
Faculty Evaluations

• We attempted to ask if the level of student
  research was improved as a result of the library
  session
• Hard to determine as many instructors have always
  brought their classes in
• In 2003, asked if it was their first time
  bringing their class, did they see a difference
• A very small number said yes, most did not answer

48
The Future

• The New Core!
• UNV 101 more substantive due to changes in the
  sequence of English classes
• ENG 110 focus on controlled research
• ENG 120 focus on locating primary and secondary
  sources, annotated bibliographies
• ENG 201 research in the disciplines, an
  opportunity to reach sophomores and juniors in a
  more comprehensive fashion

49
Thank you!

• Sarah Burns
• sburns_at_pace.edu
• Pace University Library