Accountability in Academic Libraries

1
Assessment and Evaluation

• Accountability in Academic Libraries

2
Focus on Higher Education

• Federal government has intensified focus on
  higher education and its performance
• Started with Spellings Commission
• Continues with Obama Adminstration
• DOE criticizes North Central Association of
  Schools and Colleges
• Guide to Accreditation Recognition
• Increased focus on quality
• Time to graduation
• Cost

3
Topics

• Accountability (institutional effectiveness)
  state government, accrediting organizations
• Perspectives library, institutional, customer,
  stakeholders
• Types of measures input, output, customer,
  outcome (impact)
• Outcomes student learning, research, student
• Levels course, program, college/school,
  institution
• Assessment plan Which outcomes/measures to focus
  on

4
Accountability and Institutional Effectiveness

• People are asking if the benefits of education
  equals or exceeds its cost
• Federal and state government, the private sector,
  and education consumers (students and parents)
  want education institutions held more accountable
  for the funds, time and other resources allocated
  and expended in the education process, especially
  when gaining an undergraduate degree that can be
  quite expensive

5
Accountability and Institutional Effectiveness

• Attention to accountability has resulted in
  increased attention being given to measures of
  institutional effectiveness
• Traditional measures employed were inputs and
  outputs. Most recently, outcomes have increased
  in importance as colleges and universities must
  demonstrate their effectiveness through
  efficiency and quality

6
Accountability and Institutional Effectiveness

• As a result, there is growing interest in
  answering such questions such
• What should students learn?
• How well are they learning it?
• What measures and procedures does the institution
  utilize to determine that it is effective?
• To what extent does the institution offer
  evidence that demonstrates its effectiveness to
  the public?
• What does the institution plan to do with this
  evidence to improve outcomes?

7
Accountability and Institutional Effectiveness

• Results-oriented questions such as these have
  resulted in applying outcomes assessment
  processes in an effort to measure institutional
  effectiveness. Institutional effectiveness is
  concerned, in part, with measuring
• accountability (e.g., institutional fiscal
  efficiency) and
• educational quality and improvement (e.g.,
  student learning)

8
Accountability versus Assessment

• Simply
• Assessment when an educational institution or
  one of its components assesses its own
  performance
• essentially an internal regulatory process,
  designed to assure institutional conformity to
  specified norms
• Accountability when others assess performance
• accountability is a set of initiatives others
  take to monitor the results of the institution's
  actions, and to penalize or reward the
  institution based on the outcomes

9
Accountability versus Assessment

• Assessment for educational quality
• address numerous self-evaluative questions
  related to how well the institution accomplishes
  its education mission and goals

10
Outcomes versus Learning Outcomes

• Student outcomes and student learning outcomes
  are not the same
• Student outcomes refer to aggregate statistics on
  groups of students (e.g., graduation rates,
  retention rates, transfer rates, and employment
  rates for a graduating class)
• Such outcomes are institutional outcomes and are
  used to compare institutional performance
• They do not measure changes in students
  themselves due to their college experience
• these outcomes are outputs and reflect what the
  institution has accomplished they do not reflect
  what (or how much) students learned

11
Outcomes versus Learning Outcomes

• Student learning outcomes encompass assorted
  attributes and abilities, both cognitive and
  affective, that reflect how the student
  experiences at the institution supported their
  development as individuals
• Cognitive outcomes include demonstrable
  acquisition of specific knowledge and skills
• What do students know that they did not know
  before?
• What can they do that they could not do before?

12
Accountability and Institutional Effectiveness

• Major Players
• American Association for Higher Education (AAHE)
• Association of College and Research Libraries
  (ACRL)
• Council for Higher Education Accreditation (CHEA)
• United States Department of Education

13
Accountability and Institutional Effectiveness

• Regional Higher Education Institutional
  Accreditation Associations
• develop standards by which to evaluate
  institutions
• in the past, their standards have been inputs and
  outputs based, focusing on processes and
  structures concerning the delivery and use of
  library services.
• to increase accountability, these bodies are
  placing more pressure on institutions to measure
  what students learn through assessment processes.
  
• by focusing on results, rather than counting
  heads and library books, the regionals say, they
  are holding colleges accountable
• want institutions to prepare assessment plans
  that measure outcomes within the institution.

14
Accountability and Institutional Effectiveness

• Several regional accrediting agencies have
  revised their standards to include an emphasis on
  information literacy as a student learning
  outcome, and have identified participating roles
  for academic libraries.
• NEASC - Standard Seven Library and Information
  Resources
• Standard 7.4
• Professionally qualified and numerically adequate
  staff administer the institutions library,
  information resources, and services. The
  institution provides appropriate orientation and
  training for use of these resources, as well as
  instruction in basic information literacy

15
Four Perspectives (separate but interrelated?)

• Library
• Users customers

• Institutional
• Broader community
• Government (e.g., legislatures and departments of
  education)
• National and regional accrediting bodies

16
Measures of Library Use

• Inputs
• the materials used to provide services such as
  collections, staffing, the physical facility and
  installed information technologies
• usually described in financial terms
• Outputs
• measures quantifying workload undertaken and/or
  completed, such as the number of books circulated
  or the number of reference questions answered
• valuable measures for making decisions about
  staffing levels, setting hours of operations, etc.

17
Measures of Library Use

• It is important to track the library's inputs and
  outputs
• libraries organize, track and evaluate measures
  by applying internal management information
  systems
• report these measures via annual reports and
  surveys
• Problems with library measures
• Librarians often misidentify outputs as outcomes
• There is also confusion concerning definitions
  and applications
• What is a login? What is a hit? Is a login a hit?
• What is a page retrieval? An abstract or a page
  from a full text article?
• Lack of standards leads to this confusion

18
Measures of Library Use

• Input and output measures are limited
• If the mission of the University is teaching and
  research, how do these outputs measure that?
• Outputs measure use of inputs they do not
  measure results
• Inputs and outputs are insufficient for assessing
  outcomes
• Therefore, assessment is conducted to measure
  outcomes

19
Outcomes Assessment

• Defined change in ...
• Local focus
• Planning/ implementation/ analysis
• Types of measures
• Student outcomes
• Student learning outcomes (information literacy
  or research)

20
Outcome Measures in Libraries

• Outcome assessment
• User oriented rather than institution centered
• Focus on statements about what students will
  know/think/be able to do as a result of our
  programs.
• Not statements about what the library
  should/could do to bring about desired outcomes.
• Measures changes in library users as a result of
  their contact with an academic librarys
  programs, resources and services (but how do we
  know the impact results from the library?)
• student known content, developed skills and
  abilities, and acquired attitudes and values

21
Outcome Measures in Libraries

• measures the contributions the library made to
  the university or colleges educational mission
  as a whole
• conducting student learning outcomes assessment
  is designed to improve library services.

22
Developing Partnerships with Faculty

• Kenneth Smiths report, New Roles and
  Responsibilities for the University Library
  Advancing Student Learning Through Outcomes
  Assessment, states that the library must move
  from a content view (books, subject knowledge) to
  a competency view (what students will be able to
  do).
• He contends that the student learning outcomes
  identified by ACRL are often common to learning
  outcomes identified by faculty for departmental
  courses and programs.

23
Developing Partnerships with Faculty

• Therefore, the library could assist the student
  learning and assessment efforts of the academic
  programs by
• delivering offerings defined as units of
  learning materials designed to develop competency
  in specific learning outcomes that are considered
  important by the library and by other academic
  programs.
• incorporate the offerings into required courses

24
Developing Partnerships with Faculty

• The offerings are a means to give the library a
  curriculum (its own set of course segments) and
  an opportunity to connect this curriculum to
  other academic programs.
• The library must internally develop these
  learning units and then proactively convince the
  faculty that the library can contribute to the
  learning outcomes of the academic program,
  because it is unlikely that the academic
  department will formally request such assistance
  from the library.

25
Direct Methods

• Qualitative developmental portfolios,
  think-aloud protocol, and directed conversations
• Quantitative content analysis, evaluation of
  theses/dissertations, tests (even ones
  administered as pre- and post- tests), videotape
  and audiotape evaluation, and nationally
  developed tests

26
Indirect Measures

• Qualitative focus group interviews, curriculum
  and syllabus evaluation, exit interviews,
  external reviewers, observation, self-assessment
• Quantitative general surveys

27
Example (Kathleen Dunn, p. 28)

• Core Competency locate and retrieve
  information, in all of its various formats using,
  when appropriate, technological tools.
• Scenario Youve been selected to lead a group
  of students on a four-week trip to another
  country. In preparation you need to find
  background information on the countrys history,
  current events, language, local customs, art, and
  music. How would you locate the information? What
  type of information would you expect to find?
• How can we assess the extent to which scenario
  met and change over time?

28
Another Example

• Objective to reduce the number of retrievals
  (hits) per search on aggregate and specialized
  databases
• Is this objective measurable? What types of
  measures would you apply?
• Can this objective be taught? How?
• How would you measure if anything was learned?
• What will the results tell you?

29
How should a library set assessment priorities?
So
30
Some Relevant Readings

• An Action Plan for Outcomes Assessment in Your
  Library (ALA, 2002)
• The Journal of Academic Librarianship (Jan.-Mar.
  2002) (Nov. 2002)
• Outcomes Assessment in Higher Education
  (Libraries Unlimited, 2004)
• Assessing for Learning, by Peggy Maki (Stylus,
  2004)
• Textbooks from Middle States Commission

31
Some Relevant Readings

• Association of College and Research Libraries.
  Information Literacy Competency Standards for
  Higher Education. lthttp//www.ala.org/ala/acrl/ac
  rlstandards/informationliteracycompetency.htmgt
• Association of College and Research Libraries.
  Standards for College Libraries 2000 Edition.
  lthttp//www.ala.org/ala/acrl/acrlstandards/standar
  dscollegelibraries.htmgt

32
Some Relevant Readings

• Association of College and Research Libraries.
  Task Force on Academic Library Outcomes
  Assessment Report, June 27, 1998.
• ERIC Clearinghouse on Assessment and Evaluation.
  lthttp//ericae.net/gt
• Smith, Kenneth R. New Roles and Responsibilities
  for the University Library Advancing Student
  Learning Through Outcomes Assessment.
  lthttp//www.arl.org/stats/newmeas/outcomes/HEOsmit
  h.htmlgt

33
Management Information Systems (MISs)

• Used to compile and report information concerning
  how many and how much to help libraries
  demonstrate accountability
• Is an integrated reporting system specifically
  planned and designed to provide managers with
  specific information they need to perform their
  managerial functions effectively and efficiently

34
MIS

• An MIS uses information that is internally
  produced
• Data generated by the integrated automated
  library system including counts (e.g.,
  circulation) and reports (e.g., collection age
  report)
• Other counts (e.g., number of reference questions
  received and of students attending instruction
  sessions)
• Measurements (e.g., square feet)
• User/use surveys (concerning user feedback)

35
MIS

• An MIS uses information that is externally
  produced
• Institutional status reports (e.g., monthly
  expenditures)
• Institutional reports (e.g., reports to
  governments)
• Information from the librarys vendors (e.g.,
  database use)
• Demographics (e.g., number of students and
  program population)

36
Uses of a MIS

• Planning
• Use analysis
• Cost analysis
• Gap analysis

• Information system for decision making
• Compile numbers to determine status
• As a management control mechanism
• For evaluation (inputs and outputs)
• Sawyer Library MIS

37
Uses of a MIS (for Benchmarking)

• Ratios how do we compare with others using the
  same measures?
• ACRL Standards for College Libraries 2000 Edition
  Points of Comparison http//www.ala.org/Content
  /NavigationMenu/ACRL/Standards_and_Guidelines/Stan
  dards_for_College_Libraries_2000_Edition.htmpoint
  s
• Governors State University Implementation of
  Standards for College Libraries (2000 edition),
  http//www.govst.edu/uploadedFiles/GSU20Implement
  ation202000(1).pdf
• ARL Statistics Measurement Program,
  http//www.arl.org/stats/
• Table of ratios used
• http//www.rdugan.org/LIS451/mis/arl_stats_ratios.
  pdf

38
Statistics

• ACRL 2003 Statistical Summaries
• NCES Compare Academic Libraries

39
Even More Uses

• Compliance with standards
• Regional accrediting bodies
• NEASC Standard 7, Libraries and Information
  Resources, http//www.rdugan.org/LIS451/mis/NEASC_
  standard_7.pdf
• Reporting to
• Federal agencies (IPEDS survey instrument)
• the Institution
• Annual report, etc.
• the library community
• surveys from organizations, associations, and
  other libraries (ACRL Statistical Questionnaire)