Calibrated Peer ReviewTM CPR Past, Present, and Future http:cpr'molsci'ucla'edu

1
Calibrated Peer ReviewTM (CPR) Past,
Present, and Future http//cpr.molsci.ucla.edu

• Arlene A. Russell, UCLA
• CPR Symposium 2007

2
CPR Symposium 2007

• WALS
• Writing and Assessment of Learning in
  Mathematics and the Natural Sciences
• Nancy Simpson Principal Investigator
• Adalet Baris Gunersel conference organizer

3
http//cpr.molsci.ucla.edu

• Calibrated Peer Review TM (CPR)
• Molecular Science Project
• NSF DUE 95-55605
• Comprehensive Curriculum Development Systemic
  Reform Project in Chemistry
• Awarded to UCLA in 1995

4
Molecular Science Project

• Produce students who
• have a deep understanding of chemistry
• have scientific problem-solving skills
• have good scientific writing skills
• can use the tools of the discipline
• collaborate
• have positive attitudes towards science

5
Molecular Science Project

• Integrate telecommunications and technology into
  instructional processes so that students learn to
• explore ideas independently
• collaborate
• write about chemistry
• work to a mastery level
• take responsibility for their own learning

6
Molecular Science Project

• Orville L. Chapman
•  
•  
• June 26, 1932 January 22, 2004

7
Initial Rationale for CPR

• Expository writing promotes understanding
• Clear writing demonstrates clear thinking
• Peer review fundamental to academic research
• _____________________________________
• Only way to handle the grading load of adding
  writing to a science course

8
Actual Outcomes of CPR

• Expository writing promotes understanding
• Clear writing demonstrates clear thinking
• _____________________________________
• Evaluation process of peer review requires
  higher-order, critical thinking skill

9
1998 First Research Study

• Research questions
• What level of student can handle the CPR process?
  
• How comparable are student and instructor
  evaluations?
• Organic chemistry, UCLA
• General chemistry, CSUF
• AP chemistry, Crossroads HS

10
First CPR StudyMethane Halogenation

• UCLA sophomores gt CSUF freshmen Crossroads AP
  Chem
• Two external graders HS and University
• Consistent between instructors
• Students gave lower evaluations than instructors.
• Conclusion Instructors give benefit of doubt
  Students very rigid in interpreting rubric

11
1995-2000 Molsci Project

• At the end of the grant period
• 38 Assignments
• Mainly in chemistry with a few in biology
• 70 schools had registered for accounts

12
Multi-Initiative Dissemination Project 2001-2004

• NSF DUE 01-96527
• 34 workshops in 23 states
• Chemistry focus but all faculty interested in the
  results from the four systemic reform projects
  were invited
• Over 900 attendees

13
Texas AM

• MID Workshop
• CTE Workshop
• CPR adoption by Chemistry and others
• Writing for Assessment and Learning in
  Mathematics and the Natural Sciences
• NSF 02-43209 2003-2008

14
2004 Victim of Own Success!

• 138 assignments posted
• more than 1500 in personal libraries!
• 71,000 student accounts issued
• 462 institutions
• 1,938 courses set up
• FERPA, Unposted assignments, enormous use

15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
1800 Assignments
455
330
307
Percentage
213
78
78
56
50
46
44
43
38
20
Augmenting CPR NSF CCLI grant (2005 - 2008)

• New data base structure
• student records separate from authoring and
  assignments
• Student records on local campus sites
• Assignments on central servers
• Assignments modifiable by others
• Citation credit to authors

21
Augmenting CPRA Distributed System
22
Author Home
23
Author Assignment
24
Library Search
25
Search Results
26
Citation Information
27
Timetable

• Summer 2007 Select assignments to move to
  distributed system
• Fall 2007 Contact authors
• Winter 2007 Finalize distribution costs
• Develop user manuals
• Fall 2008 CPR Version 4.0

28
Acknowledgements

• National Science Foundation
• Howard Hughes Medical Institute
• University of California, Office of the
  President
• University of California, Los Angeles
• 2006 Brian P. Copenhaver Award

29
Acknowledgements

• Colleagues
• Tim Su
• Stephen Schimpf
• CCLI Advisory Board
• Texas AM
• authors

30

• Thank You

31
(No Transcript)
32
June 2008

• Perhaps the most difficult topic in introductory
  analytical chemistry. Many concepts involved in
  solving the problem.
• Make a solution of a known concentration and
  precision. Dont waste reagents.

33
(No Transcript)
34
E-mail from a colleague

• I just had two 14BL students in my office
  hours. After asking me questions about the
  reports and about the midterm, they said thank
  you that I gave them the second CPR assignment on
  the buffer system (the one with carbonic acid and
  bicarbonate). They said they learned so much
  about buffers compared to their 14A lecture in
  which they simply did calculations on pH, pKa and
  pKb. They said they never learned in depth about
  buffers and how they work until now.

35
Peer Review Feedback
36
Peer Review Feedback
37
Does CPR make a difference?

• Does CPR in a course enhance learning?
• Writing-to-Learn?
• Does using CPR in a course improve writing?
• Learning-to-Write?
• Does CPR teach reviewing skills?
• Critical thinking skills?

38
(No Transcript)
39
Introductory Organic Chemistry

• Research University
• lecture three times/week
• no discussion
• separate lab course
• most students premed
• one CPR assignment Study of Cycloalkanes
• uses CHIME interactive tutorial to study ring
  strain
• given as extra credit assignment at end of
  quarter
• feedback e-mail to instructor required for credit

40
Introductory Organic ChemistryStudent Buy-In

• 219 of 372 students did the assignment and
  submitted comments!

41
Student Perceptions
42
Student Comments Technical Issues

• The chime plugin was a little problem because I
  too use netscape 6.1 so I ended up using Internet
  Explorer 6.0. It handled the plugin well and did
  not give me any problems. The web site performed
  decently with a 56K modem connection, except the
  chime illustrations took about 30sec to 45sec to
  load but acceptable. Now that the calibrations
  are available the calibration screens seem very
  user friendly but I would tell everyone that it
  is best to use a screen resolution of at least
  800x600 because there is so much to info to
  display. The questions take half the screen and
  the essay takes up the other half.

43
Source Material/Reviewing

• I thought that the CPR was really interesting.
  I liked the fact that they made us look at 3D
  models before we could answer the questions
  because it gave us a good idea of the concept
  involved in the question. I think that the
  process we had to take before writing the essay
  was great because it took us step by step to the
  answer, allowing us to thoroughly comprehend the
  strain (angle and torsional) in cycloalkanes in
  comparison to their alkane counterparts and to
  other larger or smaller cycloalkanes. I also
  like that we will get to grade other students
  essays and later see if our own essays were of
  good quality. I believe that it will let us know
  if we really understand the part that strain
  plays in cycloalkanes.

44
Workload, Course Value

• This program is similar to something that my
  girlfriend is doing in her Christian theology
  class at BIOLA. She writes a 400-word response
  to a "poll" question based on the 400-500 page
  reading assignment that week, every week. Then
  she reviews the work of three others (but not her
  own), followed by an in class discussion once a
  week. Of course, an in-class discussion is not
  appropriate for a chemistry course and the
  material covered requires more work than a simple
  read and response. This is why I feel that the
  CPR program would be more beneficial if it were
  assigned more than once a quarter. Something
  more like one assignment every 2-3 weeks. I
  really feel that this would improve the overall
  information retention of the class.

45
Writing

• By writing this assignment in a mini-essay form I
  was able to come at my own conclusions and
  express the material in my words. I believe that
  this was very helpful because in order to
  understand the material it is extremely useful to
  be able to describe and put the material one
  learned into his/her own words

46
Workload, Course Value

• I enjoyed the CPR assignment. I liked having
  another format to think about organic chemistry.
  If you do this again, I think optical rotation
  would be a great subject. I thought the writing
  assignment was a good length--not too much to
  read six examples of writing

47
Workload, Value, Reviewing

• I thought the idea of CPR was a very good idea
  because it helped expanding students' knowledge
  and developing insight into the materials we had
  studied. However, I didn't really enjoy this
  project. I didn't like it because there was too
  much research involved and questions were very
  hard to answer. This project itself was time
  consuming as well. It took me a long time to
  finish the practice before I moved onto the
  actual project. Another time-consuming part was
  the peer reviewing. It took me about an average
  of 10 minutes each to review one paper. It
  required a lot of concentration as well

48
Course Value

• I thought CPR to be good learning tool. The
  project made the participants review the course
  material. As a result a better understanding of
  the material is attained. Its like having two
  lectures instead of one. Making an environment
  where students have to think is a good way to
  make the material understandable.

49
Reviewing, Course Value

• Truthfully, I enjoyed CPR. Writing the essay was
  a great way to  review the specifics of
  cyclohxane strain, but when I reviewed the  peer
  essays I think I learned most. I was able to see
  what strong  points my essay had, and more
  importantly where I was lacking. I  think CPR can
  be a very useful tool for students to use if they
  take  it seriously and take the time to explore
  the website. Also, it is  another chance for
  points, which is always good. By the way, thanks
   for the extra credit.

50
Timing, Workload

• this project was sort of like high school work.
  it was only  assigned to keep me busy. first of
  all i don't have a computer so i  had to go out
  of my way to use one. we studied cyclohexanes for
  the  first midterm. it would've been a lot better
  if it was material from  the current chapters.

51
Word Count, Technical Issues

• I just wanted to give you my opinion of cpr. I
  felt like it  was to much of a hassle. It was
  hard to log onto and frustrating to  deal with.
  When I logged on and began writing my text the
  HTML tutor  wasn't accessible so I didn't know
  the editing commands. I know that  we were
  supposed to be concise, but I felt that 230-390
  words just  wasn't enough to truly answer the
  questions and write a good essay.  Anyways, I
  didn't think cpr was really worth it because it
  was to  much of a hassle.

52
Foreign Students

• The calibrated peer review helped me to review
  many knowledges about the conformation of
  cyclohexane. I learned a lot from it

53
Technical Difficulties and Attitude to CPR

• Positive Negative
• Comment Comment
• Mentioned problems
• with CHIME 59 41
• (/or UCR and CPR login)
• No technical difficulties 82
  18
• mentioned

54
(No Transcript)
55
Augmenting CPR NSF CCLI grant (2005 - 2008)

• New data base structure
• student records separate from authoring and
  assignments
• Student records will be on local campus sites
• Assignments on central servers

56
UCLA (2005-2008)

• Listserv is the main vehicle for information on
  the development of the new version of the program
• CPR will continue to be free until the
  distributed version is available

57
(No Transcript)
58
Peer Review

• Goal of peer review is improved products
• Robert Lichter 3/22/99
• Goal of Calibrated Peer Review is improved
  student understanding of the discipline, writing
  ability, and critical-thinking skills.

59
Conclusion

• The development of CPR, which began during
  the NSF-funded Molecular Science Project, has
  changed the teaching practices of hundreds of
  STEM faculty and enriched the instruction and
  enhanced the learning of thousands of students
• The impact of CPR continues to grow.

60
Workshops

• MID (2-hour introduction)
• February 6,7 - Univ of Arkansas, Little Rock
• March 5, 6 - Housatonic Comm College, Bridgeport,
  CT
• April 16, 17 - Macalester College, St. Paul MN
• April 30-May1 - Portland Community College
• Chautauqua (3-day intensive)
• June 23-25, UCLA, Los Angeles, CA

61
(No Transcript)
62
Writing An Assignment

• Good Assignments have
• Clear goals and focused writing topics for the
  essays
• Interesting source material, preferably with a
  study guide
• One calibration essay that demonstrates the ideal
  answer (the key), and the other two that
  require students to confront common errors and
  misconceptions
• Calibration answer keys that include explanations
  of the strengths and weaknesses of the critical
  points in each essay.

63
So What? Do CPR Assignments Make a Difference?

• Is content knowledge improved?
• Are evaluation skills developed?
• Do students learn to collaborate?
• What do students think about CPR?

64
Evaluation of CPRChemistry - UCLA

• General chemistry
• Midterm question on theory of two-component phase
  diagram
• Two quarters traditional lecture and problems
• One-quarter with an additional CPR assignment
• Russell, A. A.

65
Exam Scores

• Quarter CPR Actual Percent
  use grade
• Winter 1998 No 6.2/10 62
• Winter 2000 No 9.2/15 61
• Spring 2000 Yes 15.1/20 71

66
Evaluation of CPR-writing Skills Chemistry- UCLA

• Senior level environmental chemistry all
  evaluation based on writing
• Midterm - 3 essays on issues -topics specified
• Final - 6 essays on issues, two on topics covered
  by CPR assignments
• 3 CPR assignments given between midterm and final
• Chapman, O.L.

67
(No Transcript)