Don

1
Dont Guess . . . Know When (and How) to Assess

• William F. McComas, Ph.D.
• Director of Science Education Programs
• Fellow, Center for Excellence in Teaching
• Rossier School of Education
• University of Southern California
• Los Angeles, CA 90089-0031
• mccomas_at_usc.edu

2
What is Assessment?

• Assessment comes from the Latin meaning to sit
  with a learner and verify that the responses
  really mean what they seem to mean
• In most assessment plans such assurance is
  lacking
• Much assessment is suspect in terms of validity
  and reliability

3
What Can We Assess?

• Knowledge
• Attitudes
• Skills

4
Assessment Plans Must

• Be free of bias
• Reflect what is (or should have been) taught
• Provide information to enhance instruction
  (teaching or curriculum)
• Reveal misconceptions
• Adapted from Champagne, Lovitts and Calinger
  (1990). Assessment in the Service of Instruction.
  Washington, AAAS.

5
Assessment should

• Be a learning experience for both students and
  teachers
• Involve the evaluation of reasonable standards or
  benchmarks known in advance to the learners
• Be based on a well-designed table of
  specifications
• The table of specifications lists the learning
  goals associated with the number, kind, and
  nature of the individual assessment items

6
Timing is Everything

• Diagnostic Evaluation (pretest)
• Used for gauging prior conceptions on which to
  base instructional strategies
• Formative Evaluation (in process)
• Useful for midcourse corrections
• Summative Evaluation (posttest)
• A way of seeing how well you did as a teacher

7
Sorry, Wrong Number!

• Norm-referenced measures of performance against
  that of others
• Scores are typically reported in percentiles
• Jane was in the 95th percentile on the SAT
• Criterion-referenced absolute measures
• Scores are typically reported in percents
• Bob got 82 of the questions correct
• Problems occur when instructors confuse the two
  types of assessment goals

8
Major Assessment Issues

• High Stakes Test a single assessment the result
  of which (positive or negative) will have a large
  impact on the future of the examinee
• Certification, licensure, entrance exams, etc.
• Low Stakes Test a single assessment the result
  of which does not have a large impact on the
  future of the examinee
• Most teacher-made tests

9
Major Issues in Assessment Design

• Reliability a measure of the consistency of the
  results of a given instrument (mathematically
  determined)
• Validity a measure of the degree to which the
  instrument measures what it is designed to
  measure (qualitatively determined)

10
What is Validity?
Validity refers to the accuracy of a measure A
measurement is valid when it measures what it is
designed to measure and performs the functions
that it purports to perform. Does an indicator
accurately measure the variable that it is
intended to measure?
11
What is Reliability?
Reliability refers to the dependability and
consistency of the results provided by an
instrument or technique. A measurement is
reliable when it produces the same result
repeatedly with the same examinees or those from
the same sample. Does an indicator repeatedly
report the same measure that it is intended to
measure?
12
Forms of Assessment
Traditional typical forms of style and substance
Enhanced new forms of style, substance and goals
13
Traditional Assessment

• Goals assignment of grades, student progress
  reporting and fault finding
• Target learners
• Timing summative
• Methods objective exams at the recall level of
  the knowledge domain

14
Enhanced Assessment

• Goals assignment of grades, student and program
  assessment
• Target learners, instructors and the curriculum
• Timing formative and summative
• Methods objective exams and expanded methods
  targeting KSA at all levels

15
Enhanced Assessment Includes

• More data points (not just summative assessment)
• More domains (not just knowledge) but attitudes,
  skills, creativity, etc.,
• Higher levels of all domains (such as synthesis
  rather than memorization)
• More techniques such as portfolio, authentic
  assessment and empirically derived exams

16
Why Expand Assessment?

• The nature and focus of assessment tell students
  (and teachers) what is important
• Enhanced assessment encourages students to
  achieve in nontraditional realms (ex creativity)
  and in nontraditional ways
• Expanded assessment results provides the data
  necessary to make more thoughtful decisions about
  the curriculum, instruction, and student progress

17
Multi-Domain Assessment

• Knowing and Understanding
• Exploring and Discovering (process skills --
  particularly in math and science)
• Using and Applying Knowledge
• Imagining and Creating
• Feeling and Valuing (attitudes)
• Understanding the nature of the discipline (such
  as science)

18
Domain Knowledge Understanding

• Facts
• Information
• Concepts
• Laws (principles)
• Explanations and theories
• Processes inherent to the discipline

19
Discipline-Based Skills and Processes

• Primary Science Process Skills
• Observing
• Using Space/Time Relationships
• Classifying (Grouping and Ordering)
• Using Numbers (Quantifying)
• Measuring
• Communicating
• Inferring
• Predicting

20
Discipline-Based Skills and Processes

• Integrated Science Process Skills
• Controlling and Identifying Variables
• Interpreting Data
• Formulating Hypotheses
• Defining Operationally
• Experimenting

21
Domain Using Applying

• Seeing learned concepts in everyday life
• Applying learned concepts to everyday life
• Evaluating reports of scientific developments
• Making personal decisions based on legitimate
  knowledge
• Becoming involved in science-related pursuits
• Taking actions based on what has been learned

22
Domain Imagining Creating

• Visualizing by producing mental images
• Combining objects and ideas in unusual or useful
  new ways
• Producing alternate uses for objects or ideas
• Pretending
• Converging, Diverging and Synthesizing

23
Domain AttitudesValues and Feelings

• Developing positive attitudes toward science in
  general and school science
• Exploring and discussing both sides of an issue
• Expressing personal feelings in a constructive
  fashion
• Making decisions based on values

24
Domain Understanding the Nature of the Discipline

• Appraising the strengths and limitations of the
  discipline and its methods
• Evaluating and applying appropriate
  knowledge-production modes
• Using criteria to place boundaries on the
  discipline (what is in and what is out)
• Understanding the social aspects of work in the
  field

25
Blooms Taxonomy of the Cognitive Domain

• Knowledge (Recall of data)
• Comprehension (Understanding)
• Application (Using data)
• Analysis (Separating ideas into their parts)
• Synthesis (Building something out of parts)
• Evaluation (Making judgments)

26
TechniqueAuthentic Assessment

• These are modes of assessment in which examinees
  are asked to perform in ways highly related to
  real situations (or situations in which the
  learning originally occurred)
• Real situations are those that exist in study,
  work, or life itself
• The best authentic assessments are both faithful
  (life-like) and comprehensive (wide-ranging)

27
Authentic Assessment Involves

• Worthy problems or questions of importance
• Faithful representations of real situations
• Options, constraints, and access to resources
  that are legitimate and not arbitrary
• Problems that require a combination of knowledge,
  judgment and creativity
• Judgment standards that are clearly stated in
  advance
• Adapted from Wiggins (Nov / 93) Phi
  Delta Kappan

28
Technique Empirically-Derived Exams

• Determine the standards or bench-marks for
  achievement
• Create an open-ended assessment tool
• Collect data and collapse similar responses into
  typical response items
• Now, use the typical responses on a multiple
  choice type examination

29
Defining science is difficult because science is
complex and does many things. But, MAINLY
science is

• the study of fields like biology, physics, etc.
• a body of knowledge (laws and theories) which
  explain the world around us
• exploring the unknown and making discoveries
• carrying out experiments to solve problems
• inventing or designing things like computers
• an organization of people (scientists) who have
  ideas for discovering new knowledge
• None of these choices fits my basic viewpoint

30
Technique Performance Assessment

• Practical examinations,
• Performances,
• Exhibits
• The task must be legitimate and contextualized,
  not artificial and remote
• in physical science - students measure the
  temperature of a liquid vs.
• in ecology - students determine the chemicals in
  a sludge sample

31
Technique Portfolio Assessment

• . . . a systematic and organized collection of
  evidence used by the teacher and student to
  monitor growth of the students knowledge,
  skills, and attitudes in a specific subject
  area.
• Vavrus, L. (August, 1990). Put portfolios to
  the test. Instructor. pp. 48-53.

32
What Can Portfolios Contain?

• Artifacts tests, lab reports, photographs,
  meaningful journal entries
• Reproductions documents about typical events not
  usually captured (videos of presentations, photos
  of projects, etc.)
• Attestations something produced by someone else
  (i.e. notes from the teacher)
• Productions documents prepared for the portfolio
  (goal statement or reflection)

33
The Portfolio Process

• Collect materials for the portfolio
• Select materials to be in the portfolio
• Reflect on why those materials were selected
• Project make a presentation based on the
  portfolio contents
• Respect the contents and presentation
• Show the Effect of the portfolio

34
Issues in Enhanced Assessment

• How do we evaluate the results of the new
  schemes?
• How do we report the results?
• What do we do with the results in terms of
  advisement and promotion?
• What are the solutions to questions of time and
  expense?

35
Typical Small Toolbox of Assessment Methods
Midterms and Finals Multiple Choice Items True
False Items Summative Assessment Philosophy
36
Larger Toolbox of Assessment Methods
Summative Formative

• Not graded
• Perhaps on-line
• Feedback to all concerned
• Immediate feedback
• Allow for real time adjustments

• Graded
• End of a Unit
• Feedback to students

True/False Multiple Choice Term Papers Labs /
Practical exams
Minute Paper Empty Outlines Approximate
Analogies Etc.
37
Dont Guess . . . Know you Know How to Assess

• William F. McComas, Ph.D.
• Director of Science Education Programs
• Fellow, Center for Excellence in Teaching
• Rossier School of Education
• University of Southern California
• Los Angeles, CA 90089-0031
• mccomas_at_usc.edu