The Evolution of Instruction Assessment - the UGA Perspective

1
The Evolution of Instruction Assessment - the UGA
Perspective

• Diana Hartle
• Deb Raftus
• Amy Watts

2
What came before

• Half sheet handout to BIOL1107 (and other)
  classeswww.libs.uga.edu/ref/evalform.pdf
• SurveyMonkey for Academic Enhancement classes

3
A dream born, a committee formed

• Charge from the University Librarian to assess
  instruction program. More than attendance
  numbers were desired.
• Committee formed with reference and other
  instruction librarians.

4
Our Quiz

• Who will be assessed?
• Students in introductory level classes across the
  curriculum.

5
How do we give the quiz?

• Rejected ideas
• Internet based quiz services
• Expense
• Reliability
• Paper quizzes
• Collecting and correlating data
• Disseminating data

6
WebCT is the winner!

• Availability
• Technical and administrative support
• Student familiarity with interface
• Data collection possibilities

7
What to ask?

• Learning outcomes were defined by
• ACRL standards for Information Literacy
• Local concerns

8
Skills / knowledge to be assessed

• Research strategies
• Moving from a question to a search strategy
• Narrowing/broadening topic
• Identifying keywords, synonyms
• Truncation and wildcards
• Boolean operators
• Choosing which database/resource to search
• Revising search based on results

9
(No Transcript)
10
Skills / knowledge to be assessed

• Locating full text of a journal article using a
• Direct full-text link in a database
• Link to another database (FindIt_at_UGA)
• Specialized database (Electronic Journal Locator)
• Library OPAC for print
• Using citation information and OPAC record to
  identify and locate specific item needed

11
(No Transcript)
12
Revising the quiz after Year 1

• Merged Science/Humanities into one quiz
• Added questions for evaluative comments
• Did away with sequential elements to quiz (Based
  on answer to question 2)
• Added graphics to some questions
• Changed wording on some questions

13
Results

• Snapshot
• 1st semester (Fall 2005)
• Humanities quiz
• 199 students, Mean score 74.7
• Science quiz
• 437 students, Mean score 83.5

14
Results from Quiz 2, Fall 2006

• Graded questions

Question Mean Discrimination
1 94.3 0.48
2 81.6 0.56
3 92.6 0.52
4 91.5 0.48
5 43.3 -0.37
6 71.7 0.48
7 95.7 0.51
15
Results from Quiz 2, Spring 07

• Graded Questions

Question Mean Discrimination
1 92.8 0.67
2 84.3 0.61
3 92.1 0.68
4 91.0 0.71
5 54.2 0.54
6 76.6 0.58
7 94.4 0.65
16
Results evaluative questions

• Weeding responses determining what counts as a
  valid response
• Coding responses applying a standardized set of
  response possibilities to valid responses
• Crosscheck the coding
• If needed, combine categories

17
What did they learn
18
Selected responses to question 9

• I learned so much it was crazy.
• I didn't know the best way to use GALILEO. Now I
  know how to use it quickly and efficiently in
  order to get the results I want.
• I found out how to use Galileo, which I will now
  use a lot because of its credibility.
• I also did not know the availability of help by
  the librarians.
• That it is more important how you search than
  what you search.
• Being more specific with searches greatly affects
  how much time you have to put into a search.

19
Still confused about
20
Selected responses to 10

• Why we weren't told about this earlier.
• Ask me when I'm doing it on my own. I feel like
  I have it now, but that doesn't mean I can do it
  when I don't have someone leading me through it.
• I am overwhelmed with the system and am not sure
  how to find the article that I need.
• Not so much as confused, but a little
  overwhelmed. I like the fact that the instructor
  used an example to lead us through the search. I
  just think that it will take personal practice to
  really understand how to use the library.
• I think it would be helpful to have us work on
  something outside of class immediately to see if
  we have any questions and make sure we can do it
  on our own.
• I am not really confused about the search engines
  but overwhelmed because of the amount of
  information given to me in a little amount of
  time.
• Why I had to sit through an hour lecture . . .
  when only 30 minutes was necessary.

21
There must be 50 ways to spell truncation

• tranctuating
• truction
• truncationg
• trunkation
• trumcations
• trunkating
• troncation
• trunk
• trucation

• asterix trick
• asterick
• asterik
• asteric
• astricks
• asteriks
• astricts
• astericks
• asterics

22
We dont know whether to laugh or to cry

• That we actually have librarians!
• Well, first off, I finally learned where the
  library was.
• Nothing this guy made it perfectly clear. He is
  amazing. He should get a raise and take over
  President Adams job.
• Why would there be a physical subscription to a
  journal but not an electronic when generally
  speaking the electronic is or should be
  cheaper...
• Nothing! It was brilliant..............BRILLIANT!
• Why everyone thinks Galileo is so great. I never
  have needed to use it.
• All the articles I need are available any time
  online through the UGA Libraries homepage.
• I reviewed library skills I learned in 9th grade.

23
Future of quiz

• Possible additional questions
• Year of student
• Have they had instruction session before?
• Data Analysis
• Frequency of administration
• Migrate to new version of classroom management
  software

24
Thanks for coming!

• Dont forget to take a bookmark.
• Your presenters may be contacted at
• ugaref_at_gmail.com