1

Florida Association of Accounting Educators Fall
2004

• Strategies for
• On-Line
• Assessment
• Presented by
• JulieAnne Adamich Ph. D., CPA
• St. Petersburg College
• E-mail address adamichj_at_spcollege.edu

2
Define the purpose of the Assessment Process

• It is a Formative evaluation process
• based on the Continuous collection and analysis
  of data
• with the primary purpose of emphasizing
  Improvement of
• - student learning
• - student development
• - the program within the institution

3
Assessment accomplishes two important goals

• Improvements in learning will depend on how
  well assessment, curriculum, and instruction are
  aligned and reinforce a common set of learning
  goals Pellegrino, et al (2001,p. 25)
• It also is a way of ensuring that students have
  been exposed to and have acquired the major
  learning outcomes . essential for successful job
  placement.

4
Assessment is an integral part of the Planning
and Evaluation Process
Formative Goals Continuous Improvement
Both Formative and Summative Goals Focus
Allocating Resources
Summative Goals Accountability, Performance
Evaluation
5
Another reason- Accreditation Requirements for
SACS Reaffirmation

• The institution engages in ongoing, integrated,
  and institution-wide research-based planning and
  evaluation processes that incorporate a
  systematic review of programs and services that
  (a) results in continuing improvement and (b)
  demonstrates that the institution is effectively
  accomplishing its mission. (Core Requirement 5)
• The institution identifies expected outcomes for
  its educational programs and its administrative
  and educational support services assesses
  whether it achieves these outcomes and provides
  evidence of improvement based on analysis of
  those results. (Comprehensive Standard 16)

6
Goals for Developing On-Line Assessments

• Select questions that assess learning objectives
  taught in a course/program.
• - Textbook End of Chapter assignments and test
  banks are typically categorized by learning
  objective.
• Select questions that match the taxonomy level of
  the students learning development.
• Utilize technology incorporated into Learning
  Management Systems (such as WebCT) which include
  quizzing modules to minimize instructor
  intervention during the assessment process to
  permit time for instructor focus on the analysis
  of detailed reports available to the instructor
  and the provision of student feedback.
• Incorporate quiz features/settings that
  discourage or prevent cheating.

7
On-Line Assessment Resources

• Quizzing Assessments using WebCT (which is
  similar to other Learning Management Systems such
  as Blackboard, eCollege, ANGEL etc.)

8
Pre-formatted matching and multiple-choice
quizzes are provided to assess each chapter in
Thomson Learnings Course E-Packs.
9
Quizzes can also be created directly in WebCT (or
imported from a Test Bank, Respondus etc.)
10
Students know the status of each question at all
times.
For matching questions, select the correct answer
by clicking on the drop down box.
Answers are saved by clicking on Save Answer
11
For multiple choice questions, students click on
the button to the left of the correct answer.
12
On-Line Quizzing Security Features

• Develop question sets and randomize answers to
  generate a unique quiz for each student.
• Deliver questions one at a time to discourage
  students from printing the entire quiz.
• Limit the quiz duration to discourage students
  from using/sharing resources.
• Passwords, quiz availability, selective release
  settings limit the availability of the quiz.

13
There are a variety of question delivery options
to promote a secured environment.
14
Question sets can also be developed in WebCT.
For example, the system randomly selects either
MC1 OR MC2. Each exam is therefore unique to
discourage cheating.
15
There are also settings for each question, where
the answer order can be randomized.
16
and upon completion of the quiz click Finish
Students verify that all questions have been
answered
17
Webct provides the opportunity for students to
receive immediate feedback.
18
There are a variety of options for student
feedback.
19
Scores can be released at various stages
Question feedback (such as the correct answer,
the total score, and graders comments) are
available options.
20
Instructors can also view statistical information
regarding the performance of the entire class on
each quiz.
21
Test Item AnalysisReporting Features available
in WebCT

• Item Difficulty Index
• Item Discrimination Index
• Group Performance Indices
• - Standard Deviation
• - Mean

22
Item Difficulty Index
Upper 25 the percentage of students in the
upper 25 of quiz scorers who answered this
question correctly.
Whole Group the percentage of students who
answered the question correctly.
Lower 25 the percentage of students in the
lower 25 of quiz scores who answered this
question correctly.
23
Item Discrimination Index
Discrimination an indication as to how well the
question relates between high versus low scorers.
Higher values indicate better discrimination. A
negative value indicates that the lower-scoring
portion of the class performed better on this
question than the high-scorers.
Good Discrimination
Low Discrimination
24
SD the standard deviation of the results from
the average score.
Group Performance
N the total number of all student responses for
the question.
Mean the average score of all students for the
question.
25
Analyze assessment results and prepare an
assessment report

• Analyze each major learning outcome, based on
  the statistical results. Include the following
• Criteria for Success, i.e., expected statistical
  results
• Summary of Assessment Findings, i.e., actual
  statistical results
• Use of Results, i.e., analyze results to
  determine if planned program improvements are
  deemed necessary
• Action Plan and Timetable for Implementation
• Budgetary and Planning Implications

26
AICPA Educational Competency Assessment (ECA) Web
Site

• The Educational Competency Assessment (ECA) Web
  site is an online resource for educators. The
  software is designed to guide accounting
  educators through the process of assessing a
  program or course.
• The purpose of the site is to help educators
  develop or reform curricula and courses to
  support the development of a set of competencies,
  consistent with the findings of the CPA Vision.

Register free at http//eca.aicpaservices.org
27
Organizers(A series of templates to develop a
database for your institution with the capability
to sort by selected course and/or program)

• Evaluate Competency Performance (ECC) Organizer
• Documents your comparison of the actual level to
  your desired level of coverage of competencies to
  determine the degree to which the AICPA Core
  Competencies are covered in your courses and
  programs.
• Organized by Competency Categories, evaluate your
  perceived degree of
• Actual Competency Coverage
• Desired Competency Coverage
• Desired Learning Outcome
• Overall Conclusion

• Assess Student Performance (ASP) Organizer
• Documents the process of gathering, analyzing and
  using assessment information to determine how
  well students are developing the AICPA Core
  Competencies.
• Areas of input include
• Describing the Assessment Activity
• Documenting Findings
• Drawing Conclusions
• Recommending Improvements

28
Quizzing Assessments Major Pros

• Explicit demonstration of knowledge
• Efficiently examines a wide range of content
  knowledge
• Can use well-established practices to evaluate
  reliability and validity
• Inexpensive to administer and evaluate for a
  large number of students
• If embedded in courses, allow for more
  representative or complete student samples
• If test is used multiple times, can obtain
  longitudinal/trend or pre/post data

Source AICPA Educational Competency
Assessment (ECA) Web Site Register free
at http//eca.aicpaservices.org
29
Major Cons Mitigated using On-line Assessments

• Labor-intensive to develop valid and reliable
  questions Test Bank questions can be imported
  into WebCT.
• May be unable to generalize performance to other
  tasks/ Less authentic because they do not
  simulate conditions of work Personal Trainer
  simulates detailed exercises and problems.
• Student participation rates and motivation levels
  may be low for non-graded examinations. Limited
  feedback to students if they are not allowed
  access to results for individual questions Quiz
  settings provide options of hints, solution, and
  IMMEDIATE feedback upon completion of the quiz.
  
• Standardized tests may encourage instructors to
  teach to the test and direct student focus away
  from competencies Question sets may be
  developed to randomize questions based on
  criteria (such as learning outcome, competency
  level, etc.).

30
Resources for Faculty

• BEEP Best Educational E-Practices
• http//www.spcollege.edu/eagle/research/beep/beep3
  8.htm
• Archive includes information associated with the
  following resource categories
• Assessment and Evaluation
• Innovative Technologies
• Instructional Resources
• Merlot. Well-known and growing collection of
  peer-reviewed online learning materials and more.
  Officially titled Multimedia Educational Resource
  for Learning and Online Teaching.
  http//www.merlot.org