Modeling Assessments of Innovative Physics Courses

1
Modeling Assessments of Innovative Physics Courses

• A Symposium

2
Institutions Educational Outcomes

• To prepare students who
• possess a breadth of integrated, fundamental
  knowledge in basic physics
• build a depth of knowledge in exploring the
  contradictions between personal assumptions and
  evidence from the physical world.

3

• can frame and resolve ill-defined problems
• can communicate effectively
• are independent learners
• can work effectively with others
• are intellectually curious
• can apply knowledge and skills to the unique
  tasks of a profession

4
Physics Departments Curriculum Goal

• to develop introductory curriculum which
  exploits the use of new technology in order to
  promote the development of critical thinking and
  problem solving skills, thus enhancing career
  preparation.

5
ZX Approach to Teaching Physics

• required inventive problem solving that
• addressed complex situations which had
  ill-defined approaches and unclear solutions
• deluged students with relevant and irrelevant
  information and
• required analysis, criticism, and synthesis

6
ZX Approach to Teaching Physics

• modeled conditions met by human beings in real
  world vs. working within an abstracted or
  generalized framework
• encouraged scholarly research using the CD-ROM
  resources
• planned for individualized problem-solving
• required learner creation of problems to be
  solved by peers.

7
Policy Support for ZX Approach

• Part of the education students receive is
  responding to stress and allocating precious time
  to a variety of demands. Responding to the
  demands of coursework is one of the areas in
  which students must often make critical
  decisions. While the faculty should understand
  that students have many things to do in a given
  day, this level of activity no way justifies
  diminishing the rigor of our courses. Students
  must also learn responsibility, and we must allow
  them the chance to fail as well as succeed.
• Faculty Handbook, p. 30.

8
Action Research OverviewPurpose

• To track the impact of innovation in the
  approach to teaching physics for a semester
• Implementation of CD-ROM resource
• Use of e-technology as major instructional tool
• Attitudes of instructors and students
• Use of ill-defined problems as central
  instructional approach

9
Multiple Methods ElementsQuantitative Data
CollectedPre- and Post-data

• Multiple aptitude entry measures (Pre-)
• Nature of Science Profile (Pre- Post-)
• Strategies in Teaching Physics (Pre- Post-)
• Attitude to Science (Physics) (Pre- Post-)
• Force Concept Inventory (Pre- Post)
• Common summative examination (Post-)

10
Multiple Methods ElementsQualitative Data
Collected

• 3 student interview sessions
• Weekly email student diaries
• Instructional staff interviews and notes on
  change process

11
Theoretical FrameworksAdult Intellectual
Development

• William Perry (1970, 1981)
• Dualism -- instructor and textbook is authority
• Multiplicity everyone has right to an opinion
• Relativism analytical thinking skills emerge
  
• ability to critique own
  ideas and those of others
• Validity not all ideas/positions are equally
  valid

12
Theoretical FrameworksExpert vs. Novice

• Ability to chunk info into familiar patterns
• Knowledge organized around core concepts or big
  ideas
• Knowledge reflects contexts of applicability s
• Automaticity in retrieving relevant
  characteristics for problem solving
• A need to possess content pedagogical knowledge
• Mental flexibility, a positive view of ones own
  accomplishments and a belief that there is much
  more to learn.

Experts have.
13
Theoretical FrameworksRIGOR DEFINED

• Rigor is the goal of helping students develop the
  capacity to understand content that is complex,
  ambiguous, provocative, and personally or
  emotionally challenging.
• Strong. R., H. Silver and M. Perini. (2001)
  Teaching What Matters Most Standards and
  Strategies for Raising Student Achievement.
  Alexandria, VA Association of Supervision and
  Curriculum Development. P. 7.

14
Interview Protocol

• 3 instructors on experimental team
• 12 students, 6 from each experimental course
• Selected randomly, ensuring gender balance
• 3 interviews
• Entry level first week of semester
• Mid-point
• Exit level 2 weeks before final exam
• Feedback from interviews given to instructors

15
Interview Cohort12 students from experimental
group

• Reasons for volunteering.
• 9 use of computers to learn
• 7 CD-ROM instead of textbook
• 3 no books to carry
• 1 many new resources available
• 1 a new opportunity
• 1 can explore Physics deeper

16
Interview CohortPre-study interviews

• Previous Physics experience
• 4 one year High School
• 3 two years High School
• 2 one year advanced level physics
• 1 one year College plus 1 year High School
• 2 none

17
Interview CohortPre-study interviews

• Subjects described learning style
• 5 as visual
• 3 as hands-on
• 2 as problem-solvers
• 2 need an interesting instructor

18
Interview CohortPre-study interviews
LESLIE

• Primary subject interest
• 5 Math and Science
• 1 Science
• 2 Math
• 1 Computer Science
• 1 History
• 1 Political Science
• 1 Spanish

• Least favorite subject
• 6 English
• 3 Writing
• 2 History
• 1 Math

19
Interview CohortPost-study interviewsReported
experiences and attitudes

• Appreciated
• working with others toward a common goal
• using technology as a tool for studying Physics

20
Interview CohortPost-study interviewsReported
experiences and attitudes

• Concerns
• negative impact of time constraints
• confusion in choice-making activities
• concern about adequate preparation for final
  departmental assessment

21
Quantitative Research ReviewInstruments Used

• Strategies in Teaching Physics (SITP) Pre-Post
• Nature of Science Profile (NOSP) Pre-Post
• Attitude to Science (Physics) (ATP) Pre-Post
• Force Concept Inventory (FCI) Pre-Post
• Multiple aptitude entry measures Pre-
• Common summative final exam Post-

22
Why Use Surveys?Some of the Research Questions

• What are the students and instructors
  approaches to the teaching and learning
  (curriculum) context of physics? (SITP
  Questionnaire)
• What are the students and instructors
  approaches to the philosophical context of
  science (physics)? (NOSP Questionnaire)
• What are the students and instructors attitudes
  to social and educational contexts of physics -
  including the use of electronic technology? (ATP
  Questionnaire)

23
Strategies in Teaching Physics(SITP) Pre-
Post-Administered to Instructors and Students
Pre-post80 items - 3 subtests - 6 point scale

• SITP Examines Three Curriculum Models
• Sub-test 1 Neo-classicism (reproduction)
• (memorize and recall physics facts, concepts and
  processes)
• Sub-test 2 Liberal-progressivism
  (reconstruction)
• (active personal construction of physics
  knowledge)
• Sub-test 3 Social-criticism (interactive
  reconstruction)
• (students participate as co-learners of physics)
  

24
ONE TEACHING-LEARNING CONTEXT INSTRUCTORS
RESULT (SITP)
Four instructors exhibited noticeable differences
in preferred teaching strategies.
25
Nature of Science Profile (NOSP)Instructors
Students Pre- Post- 24-item, 11 point Likert
scale, 5 subtests

• Examines the philosophical context of science
  (physics) along five axes
• Relativism lt -- gt Positivism (RPP axis)
• Inductivism lt -- gt Deductivism (IDP axis)
• Contextualism lt -- gt Decontextualism (CDP axis)
• Process lt -- gt Content (PCP axis)
• Instrumentalism lt -- gt Realism (IRP axis)

26
ONE PHILOSOPHICAL CONTEXT INSTRUCTORS RESULTS
(NOSP)
For the Relativist-Positivist subtest (RPP), the
more negative the score, the stronger the
Relativist position. The higher the positive
score, the stronger the Positivist position
similarly for IDP, CDP, PCP IRP.
27
Conflicts of Instructors Ideas about the Nature
of Science (Physics) Context

• The NOSP graph shows strongly differing ideas
  amongst the instructors about the philosophical
  context of science (physics).
• Such philosophical differences about the nature
  of science (physics) are NOT INCONSEQUENTIAL.
• Such fundamental differences of philosophy
  undercut attempts at curriculum reform.

28
Comparing SITP Curriculum Contexts with the NOSP
Philosophy of Science Contexts

• Neo-classicism (NC)
  vs Liberal-progressivism (LP)
  (knowledge reproduction) (knowledge
  construction)
• Realism lt-gt Instrumentalism (IRP)
• Inductivism lt-gt Deductivism (IDP)
• Positivism lt-gt Relativism (RPP)
• Decontextualism lt-gt
  Contextualism (CDP)
• Content lt-gt Process (PCP)
• Social Criticism is not represented in the table
  above because the philosophical differences
  between LP SC are far more subtle than between
  NC LP (or NC SC).

29
Attitude to Physics and E-Technology(ATP) 1
87 questions - 7 Subtests - 7 point scale

• Subtests
• Social implications of physics SP
• Attitude to scientific inquiry in physics AP
• Attitude to teaching/learning physics TP
• Physics as a personal threat PTP
• Physics as a practical hands-on activity HP
• Socio-cultural value of physics VP
• E-technology in learning/teaching physics ET

30
The bar graph shows instructors variability.
Note SP AP HP VP.
31
Attitude to E-Technology (ATP) 2 27 Extra
Items 4 sub-tests

• Created specifically for the research study at
  the request of the focus group
• Interest in E-technology (7 ZIP items)
• Useful aid to study (7 ZUP items)
• Aid to Multi-tasking (6 ZMP items)
• Personal Control (7 ZCP items)

32
Instructors do not exhibit a uniformly high
response. Some head in opposite directions.
33
Action Research Summary 1 Instructors
Quantitative Result

• If a Physics Department is genuinely interested
  in developing a coherent and effective approach
  to collaborative teaching and shared
  instructional goals, then the skills, attitudes,
  and dispositions of the instructors must be
  examined and taken into account in the process of
  curriculum reform.

34
Students Quantitative Result 1Aptitude
Pre-tests c.f. Final Exam

• 7 Aptitude Predictors Employed
• 2 Standardised Achievement Tests
• Math Aptitude Algebra Aptitude Calculus
  Aptitude
• Chemistry Aptitude Physics Aptitude
• ACCOM Score (Academic Composite of Aptitudes)
  used to allocate students to Honours Classes.
• THE ONLY PREDICTOR TO PRODUCE A SIGNIFICANT
  ASSOCIATION WITH FINAL PHYSICS EXAMINATION
  RESULTS WAS THE CHEMISTRY APTITUDE TEST. WHY?

35
Some Little-known Facts AboutNeo-classical
Instruction and Examinations.

• The Neo-classical Instruction Effect
• Teaching procedure Equal exposure/Equal rate
  
• Student progress By differential entry
  aptitude/ability rate
• Slow students progress slower (differentially)
  by aptitude
• Fast students progress faster (differentially)
  by aptitude.
• Student performance order Entry aptitude
  order should be THE SAME AS exit summative exam
  order (except for minor accidental influences
  poor teaching, illness, lack of motivation, etc)
• Student performance distribution Remains
  normative but with a differentially greater
  spread/variance (caused by differential
  spreading of rates of progress by aptitude).
• Grading Could be done fairly and equally at
  the beginning of semester using valid
  aptitude/ability test order (theoretically).
• If you are a neo-classical teacher, just for
  fun, compare your pre-course aptitude order
  with your post-course final exam order using
  Spearman rho!!
• You have a problem somewhere if there is a
  significant difference.

36
Students Pre-Post Quantitative Result 2Force
Concept Inventory

• Hº No difference amongst average scores by
  treatment groups RETAINED (Kruskal-Wallis
  One-way Anova test)
• If the FCI is a valid and reliable independent
  test of understanding of Newtonian mechanics,
  then
• What did the physics course instruction teach the
  students about Newtonian Mechanics during the
  semester?

37
Students Pre-Post Quantitative Result
3Strategies in Teaching Physics (SITP)

• Neo-classical Sub-test (NCP)
• Pre-Post Preference Scores DECLINED
• Liberal-progressive Sub-test (LPP)
• Pre-Post Preference Scores NO DIFFERENCE
• Socially-critical Sub-test (SCP)
• Pre-Post Preference Scores NO DIFFERENCE
• Further No difference by gender, class grouping,
  or instructor.

38
Students Quantitative Result 4 (NOSP) Favoured
Ideas Profile

• Relativismlt-gtPositivism (RPP) Undiscriminated
• Inductivismlt-gtDeductivism (IDP) Undiscriminated
• Contextualismlt-gtDecontextualism (CDP)
  Decontextualism
• Processlt-gtContent (PCP) Process
• Instrumentalismlt-gt Realism (IRP) Realism
• Further No difference by gender, class grouping,
  or instructor.

39
Students Pre-Post Quantitative Result 5Nature
of Science (Physics) (NOSP)

• Relativismlt-gtPositivism (RPP) No difference
• Inductivismlt-gtDeductivism (IDP) No difference
• Contextualismlt-gtDecontextualism (CDP) No
  difference
• Processlt-gtContent (PCP) No difference
• Instrumentalismlt-gt Realism (IRP) No difference
• CONCLUSION Students retained the ideas formed in
  High School. Physics 110 had no effect on
  students views about the nature of science
  (physics).

40
Students Pre-Post Quantitative Result 6Attitude
to Physics (ATP)

• Pre-Post Sub-test Results
• Social implications of physics (SP)
  Significantly Lower
• Attitude to scientific inquiry in physics (AP)
  Significantly Lower
• Attitude to teaching/learning physics (TP) No
  difference
• Physics as a personal threat (PTP)
  Significantly Lower
• Physics as a practical hands-on activity (HP)
  No difference
• Socio-cultural value of physics (VP)
  Significantly Lower
• Further All pre-post attitudes had positive
  support.
• No pre-post difference by gender.
• There were significant differences by
  class section and instructor.

41
Students Pre-Post Quantitative Result 7 (ATP)
Attitude to E-Technology

• Pre-Post Sub-test Results
• Interest in E-technology (ZIP) Significantly
  lower
• Useful aid to study (ZUP) Significantly
  lower
• Aid to Multi-tasking (ZMP) Significantly
  lower
• Personal Control/Vulnerability (ZCP)
  Significantly lower
• Further Female ZIP, ZMP ZCP post-tests were
  significantly lower. Section grouping post-tests
  were all significantly lower. Instructor grouping
  post-tests were significantly lower for ZIP only.

42
Students Final Summative Written Exam Results - 8

• Overview of Final Summative Exam Results
• There were no significant differences amongst
  Final Exam scores by experimental section, by
  instructor, or by treatment group.
• In general, students were not disadvantaged
  through participation in the CD-ROM project.
• It is not possible to make any quantitative
  statement about academic gains by the CD-ROM
  project groups through the use of the CD-ROM.
• In considering the total student population,
  Final Exam results for females were significantly
  lower than for males.

43
Some of the Conclusions and Recommendations 1

• No pre-course aptitude measures were valid
  outcome predictors except the chem aptitude
  test.
• There is no value in using the FCI with the
  current physics curriculum and final examination.
• The current teaching and final examination system
  discriminates against females (shown by pre-post
  FCI results and course entry aptitude measures).
• There were no differences in final performance
  across sections, instructors, or treatment group.

44
Some of the Conclusions and Recommendations 2

• There was a significant drop in studentspre-post
  dispositions to the neo-classical teaching model.
• Dispositions to liberal-progressive and
  socially-critical models of curriculum and
  instruction remained stable.
• There were no pre-post changes to sub-test
  dispositions by gender, section grouping or
  instructor.
• Both instructors and students may be ill-prepared
  for the teaching of physics using inquiry
  procedures to support critical thinking and
  problem-solving approaches directly related to
  the institutional objectives.
• Instructors could benefit from workshops
  involving history and philosophy of science,
  curriculum theory, educational psychology, and
  educational measurement.

45
Some of the Conclusions and Recommendations 3

• Four of six ATP attitudes declined significantly
  across the semester. Section attitudes converged
  so that all sections became more alike.
• Students E-technology attitudes declined across
  the semester. Some attitudes remained more
  positive than others. Sections by instructor
  attitudes converged to become more alike.
  Reasonable support remained for E-technology.
• The Physics 110 programme made no impact on the
  sophistication of students metaphysical
  viewpoints about the nature of science (physics).
• Physics 110 students attitudes tend NOT to
  support inquiry, critical thinking and complex
  problem-solving teaching and learning procedures.

46
Some of the Conclusions and Recommendations 4

• Immersion in, exposure to, and coverage of
  will NOT be sufficient to attain the departmental
  and institutional goals.
• Instructors most aligned with departmental and
  institutional goals are those who receive the
  least favourable end-of-semester evaluation of
  teaching .
• Students need to be re-educated to think
  differently about what it means to learn,
  understand, do, and be competent in physics.
• Physics 110 students misunderstand, disagree,
  oppose, and show discontent with teaching that is
  not seen to focus directly upon final examination
  specifications.

47
Some of the Conclusions and Recommendations 5

• The Physics 110 curriculum and instruction model
  must be reviewed in the light of the last 30
  years of research on the intellectual development
  of young adults.
• The Physics 110 curriculum and instruction model
  must respond to the cognitive agility,
  flexibility, and innovation required by the
  ambiguity, complexity and change features of the
  21st century world.
• Students need to be re-educated to think
  differently about what it means to learn,
  understand, do, and be competent in physics and
  be moved from a passive to an active
  responsibility for their learning and
  understanding.
• The Physics Department must re-orient its
  approach to academic rigour so that the term no
  longer implies the brain is a muscle that must be
  subjected repeatedly to severity, hardship,
  adversity and stress to improve its functioning
  and performance.

48
Features of Action Research

• Aim
• Create and study change.
• Minimal requirements
• Social practice (not content) is the research
  subject.
• Proceed systematically in repeated cycles of
  observing, planning, doing, and reflecting.
• Involve those responsible for current practice
  and future change.
• Collaborate with all stakeholders through
  appropriate representation.
• Useful words
• Re-examine reinvent reconceptualize
  re-imagine refine refresh reform rebuild
  reconstruct revise remodel regenerate revisit.

49
Action Research Procedures

• Procedure (? What we did.)
• Form a reference group.
• Identify or define the problem(s) or issue(s).
• Identify clearly what happens now.
• Reflect and refocus on how and what might,
  should, and can be done differently.
• Seek disagreement.
• Redefine the problem as necessary.
• Settle on the research question(s)
• Develop a strategy for collecting data, solving
  the problem or implementing an idea.
• Use multiple flexible methods.
• Collect data, implement action, problem-solve,
  test ideas.
• Observe, evaluate, conclude study the
  consequences of actions, specify learnings, make
  sense of experience, describe, explain. Ask, So
  what? and What next?
• Continue with ongoing action research cycles to
  make further improvements.
• Share your learning with others.

50
What We Learned.Still requires work..

• What were strengths of study?
• Institutional, departmental, staff, and student
  involvement.
• Cooperative extensive participation of staff and
  students
• Extensive data gathering was possible.
• Non-threatening
• Non-invasive
• Non-personal
• Thorough
• Unpleasant or unexpected findings were fairly
  reported
• Recommendations were made without fear or favour
• What was missing?
• Impact of time and budget?
• Influence of institutional culture?
• Influence on instructors risks?

51
Responder Frits Gravenberch

• 20 minutes

52
Question and AnswerSession

• 15 minutes