Learning Impact Assessment in Online Mathematics

1

• Learning Impact Assessment in Online Mathematics
  Statistics Classes at Pace University
• David Sachs, Nancy Hale,
• Barbara Farrell, Patricia Giurgescu
• Pace University
• The Ninth Sloan-C International Conference
• on Asynchronous Learning Networks

November 16, 2003 Session 5
2
Data from two mathematics statistics online
courses(second-generation) MAT 125 MAT 234

• MAT 125 Technical Mathematics CSIS/NACTEL
  Program
• MAT 234 Introduction to Probability
  Statistical Analysis
• Course enrollment --steady from year to year and
• Grade statistics --found consistent with those in
  comparable traditional classes no loss in the
  quality of learning outcomes (no significant
  difference phenomenon)
• Course materials developed through an iterative
  process, aimed at exposing students to
• a variety of content presentation resources
  (combining text with dynamic elements and
  visualization tools),
• multiple forms of testing, to track student
  learning performance and give student feedback
• early student interaction, so instructors can
  identify the intervention and assistance needed
  by individual students.

3
Students frames for managing knowledge in
online courses

• Online students demand focus, order and
  structure, particularly in math stat online
  courses
• Frames (Redish 2002) filter students knowledge
  management
• Social (who will I interact with during this
  course? instructor, peers)
• Material (what course materials will I use (and
  how)?)
• Skills (what will I be doing here? what is
  expected of me?)
• Affect (how will I feel about what Im doing?)

4
Focus on developing successive layers of
mathematical abilities ? conceptual
understanding, ? procedural knowledge, ? problem
solving, waving through five basic content
strands, leading to competency in ? mathematical
reasoning, ? connections and ? communication.
5
Learning effectiveness targets (measures)

• cognitive outcomes
• conceptual understanding
• procedural fluency
• strategic competence (for problem solving)
• (knowing what, why, how, when and where certain
  knowledge applies)
• communication outcomes
• ability to express quantitative information
  clearly and rigorously, using the most
  appropriate technological tools,
• attitude/productive disposition, civility
  integrity
• affective and ethical dimension

6
Learning Assessment

• Learning outcomes assessment (objective)
  quizzes, weekly homework, proctored exams
  individual projects.
• Learning experiences assessment (students
  perception) online student satisfaction surveys
• Bloom's learning achievement function
• S f(x,y,z)
• x cognitive entry characteristics
• y quality of instruction,
• z affective characteristics (attitude,
  motivation)

7
Learning modes in online math/stat courses

• Supervised learning --learning from examples,
  provided by instructor
• seek to minimize error, i.e., deviation between
  learner instructors responses
• insufficient for learning to act optimally in
  new problem domains.
• Unsupervised learning --student looks for
  association rules, concept clustering, patterns,
  without instructors direct guidance or "training
  set
• performance measures are more difficult to
  establish and calculate (e.g., can be assessed
  from student class projects open ended
  assignments, with perfection-based grading)
• includes incidental learning (as ability to make
  sense out of related material, e.g. gathered from
  discussion board interactions)
• Reinforcement learning student is goal-directed
  and seeks to maximize reward, by interacting with
  the problem domain
• trade-off between minimal investment (exploiting
  what student can easily acquire in order to
  obtain reward) vs. further exploration (investing
  in more knowledge, in order to make better
  decisions/choices in the future)

8
Math/Stat course design principles for learning
effectiveness

• Good structure of materials / logical sequencing
  --so that students can move easily and
  systematically through content
• Actively involve online students through
  exercises embedded in the lecture notes and
  classroom tasks for the Discussion Board.
• Adaptive instruction --opportunities for peer to
  peer instruction, to enhance interaction of
  students and the instructor
• Systematic use of embedded assessments and
  student self-assessment tools.
• Tracking of student learning --to identify
  nodes of student understanding or
  misunderstanding (node key point in
  understanding a particular content area or
  process Zygielbaum, 2001).
• A correct outcome to a node-task leads to
  subsequent activities, incorrect outcomes lead to
  remedial tasks and then move to the subsequent
  activities.
• Involve students in solving real-life problems,
  with real-life data, using technology
• Student-centeredness --communication
  coaching/support, to help students clarify their
  thinking process and strengthen problem solving
  skills.

9
Challenges

• distinguish inadequate presentation or faulty
  assessment items from poor student performance
  --item response theory analysis can help
• blind-spot of help-based interaction --common
  assumption is that students, as mature learners,
  are willing and able to ask for help when needed
  but students with weak metacognitive skills are
  least able to seek assistance.
• moving from easily assessable procedural
  mathematics tasks to assessing higher order
  skills (complex problem solving and modeling).
• A popular assessment component is the assignment
  of a comprehensive class project, reflecting
  students competencies at the end of the course
  the instructor gives individual guidance to
  students throughout the semester, for completing
  the project, which is then presented to the class
  in a valid electronic format and may be included
  in students electronic portfolio. The emphasis
  is on tackling real-life problems, with real-life
  data and tools, and strengthening the
  communication skills and technological fluency.
  The grading is perfection-based (student has to
  revise and resubmit project, within given
  deadline, until it passes pre-set quality
  standards)
• grading misclassification error --trade-off
  between a-risk and b-risk

10
Major shifts in math assessment practice
11
Enduring characteristics of assessment

• relevance --how closely the outcomes are related
  to marketable employment or institution's
  mission
• utility --potential usefulness for individuals
• applicability --extent to which the information
  is relevant for multiple user groups
• interpretability --likelihood of understanding by
  multiple users
• credibility --level of trust of different users
  regarding assessment information on an outcome
• fairness --balance of perspective among groups of
  diff. ability
• scope --size and breadth of sample
• availability --accessibility, feasibility
• measurability --reliability, and validity
• cost --appropriateness of expenditures to produce
  it

12
Recent influences on assessment

• online instruction scaffolding instructor
  continually adjusts the level of help in response
  to the students level of performance
• cognitive psychology, learner-centric approach
  takes into account expanding the zone of proximal
  development range of potential each person has
  for learning the subject, when the learning is
  facilitated by someone with greater expertise
  (Vygotsky) target both the level of actual
  development the level of potential achievement.
  
• With modern instructional technology, assessment
  focuses on building frequent and accurate
  feedback loops directly into the learning
  process
• Formative assessment ? allows students to
  structure their learning experiences around their
  individual needs encourages self-efficiency,
  self-appraisal, reflection.

13
Slide summary /conclusions

• Target classes Technical Math Introd. to
  Statistics Probability
• Online students perspective frames
• Desired competency math ability layers
• Learning effectiveness targets
• Assessment of learning outcomes (obj.)
  experiences (subj.)
• Modes of learning supervised, unsupervised,
  reinforcement
• Design for learning effectiveness in math
• Challenges in assessment
• Shifts in assessment practice
• Characteristics of assessment
• Recent influences on assessment
• DSachs_at_pace.edu NHale_at_pace.edu
• BFarrell_at_pace.edu PGiurgescu_at_pace.edu