Revised GRE General Test: A Prelaunch Update

1
Revised GRE General Test A Prelaunch Update

CGS Annual Meeting December 8, 2006

• David G. Payne, PhD
• Educational Testing Service

2
The Revised GRE General Test

• Brief Background
• Reasons for Change
• Value of the Revised Test
• Key Aspects of Changes to the General Test
• Contingency Planning for Launch
• Communications Efforts
• Additional Slides for Deans to Use on Campus

3
(No Transcript)
4
What Is the Value of the Revised GRE General
Test?

• Improved security Administration plan that
  directly addresses current and potential future
  security challenges over the long term
• Improved predictive validity The revised test
  reduces the possible effects of memorization.
• Improved construct validity - Measures skills
  more directly related to graduate study
• Linear Testing Format is more test taker friendly
  than current Computer Adaptive Test (CAT)

5
Why Are We Changing the General Test?

• Test delivery is changing from continuous testing
  to fixed administrations in order to address
  security issues
• Eliminate the potential security risk that exists
  with continuous testing regarding the exposure of
  test questions
• Test content is changing to increase the validity
  of the test by reducing the effects of
  memorization
• Eliminate single-word verbal questions
• Introduce new question types and tasks
• Reduce possibility of non-original essay materials

6
Additional Goals for the Revision of the GRE
General Test

• Provide faculty with better information on
  applicants performance
• Essay distribution
• New Verbal and Quantitative Score Scales
• Better measurement of skills
• New question formats other than traditional
  multiple choice
• Allow calculator use in order to reduce emphasis
  on computation in Quantitative section
• Improved test access for examinees worldwide

7
Verbal Reasoning andQuantitative Reasoning
Sections
8
Value of the New Verbal Reasoning Section

• Emphasis on skills related to graduate work such
  as complex reasoning
• Increased emphasis on inferential reasoning
• Increased emphasis on verbal reasoning in context
• Increased number of reading comprehension
  questions based on a greater variety of reading
  passages
• Reduced emphasis on vocabulary out of context
• No Antonyms or Analogies
• Inclusion of new question formats other than
  traditional multiple choice (e.g., highlighting a
  sentence in a passage that serves the function
  described in the question)

9
Value of the New Quantitative Reasoning Section

• Quantitative reasoning skills that are similar to
  skills typically used in graduate school
• Increased emphasis on questions involving
  real-life scenarios
• Increased emphasis on data interpretation
  questions
• Reduced emphasis on Geometry
• On-screen 4-function calculator with square root
• Reduced emphasis on computation
• Inclusion of new question formats other than
  traditional multiple choice (such as entering a
  numeric answer via the keyboard)

10
Why Are We Changing the GRE Verbal and
Quantitative Score Scales?

• Sound measurement practice and professional
  standards (developed by the AERA, APA, and NCME)
  recommend a change in the score scale when
  significant changes to test specifications are
  made.
• The ways in which we are measuring the verbal and
  quantitative reasoning skills (i.e., question
  types) and the relative emphasis among the
  component skills in the revised Verbal and
  Quantitative sections are sufficiently different
  from those on the current test, so that it is
  appropriate to use a new scale for the revised
  test.

11
New Verbal and Quantitative Score Scales

• Change will allow us to address two situations
  that have developed over the life of the current
  test
• Mean Verbal scores (470) and Quantitative scores
  (593) have drifted apart over time
• Quantitative scores cluster at the top of the
  scale

12
Ranked Preference for New Score Ranges
Preferred Most
Preferred Least
13
Comparison of Old and New Verbal and Quantitative
Score Scales
NOTE The Critical Thinking and Analytical
Writing score scale will not be changed.
14
Advantages of the New Score Scale

• The new scale will make more apparent the
  differences between candidates and will
  facilitate more appropriate comparisons between
  candidates.
• Score users will be less likely to interpret
  small score differences as meaningful differences
  between candidates.
• The Verbal and Quantitative means will be aligned
  which will allow score users to see relative
  strengths for a particular candidate.

15
Information to Assist GRE Score Recipients

• Percentiles (based on first 3 administrations)
  will be available on the GRE Web site
  (www.ets.org/gre) and printed on score reports to
  assist score users in interpreting scores
• A concordance table to assist score users in
  determining the relationship between old and new
  Verbal and Quantitative scores will be available
  on the GRE Web site.
• For Verbal and Quantitative scores earned prior
  to September 2007 and reported on score reports,
  approximate score equivalents on the new scale
  will be included on score reports to assist Score
  Recipients in comparing old and new scores.
• Broad major field score distributions will be
  available on the GRE Web site.

16
Sample Revised General Test Score Report

• For tests taken beginning in September 2007,
  scores will be printed in the New Score column
  and dashes will be printed in the Original
  Score column. For tests taken before September
  2007, scores will be printed in the Original
  Score column and approximate equivalents on the
  new scale will be printed in the New Score
  column. These equivalents are based on a
  concordance study the full table showing the
  relationship between the two scales will be
  posted on the GRE Web site at www.ets.org/gre.

17
Analytical Writing Section
18
Familiarity with Skills Measured in Sections of
the GRE General Test
Very Familiar
Not at all Familiar
19
Appropriateness of Alternative Names
Very appropriate
Not at all appropriate
20
Value of the New Critical Thinking and
Analytical Writing Section

• A performance-based measure that integrates the
  assessment of critical thinking and analytical
  writing
• Consists of two complementary analytical writing
  tasks
• 30-minute Analyze an Issue" task
• 30-minute Analyze an Argument" task
• New, more focused prompts
• Requires a more specific response from the test
  taker
• Will reduce possibility of reliance on memorized
  materials
• Limited or no reuse of prompts

21
The New Critical Thinking and Analytical Writing
Section

• Measures the ability to
• Articulate complex ideas clearly and effectively
• Examine claims and accompanying evidence
• Support ideas with relevant reasons and examples
• Sustain a well-focused, coherent discussion
• Control the elements of standard written English
• Note Essay responses are typed on computer,
  with a basic word processor, and no spell checker
  or grammar checker.

22
Essay Distribution

• Essay responses will be made available
  electronically to designated score recipients.
• The details of operation of this new service will
  be announced in early 2007.
• It is critical that faculty understand that test
  takers construct their essay responses, which are
  essentially first drafts, under these conditions
• Each essay must be completed in 30 minutes
• It is a high stakes testing situation
• Test takers use an elementary word processing
  system with no spell check or grammar check

23
Contingency Planningfor theRevised GRE General
Test
24
rGRE Contingency Planning

• Issues
• Need to set the new Score Scales for Verbal and
  Quantitative
• Requires 30,000 representative test takers in
  first 3 administrations
• Only possible to do this in the Fall
• Not possible to do a phased launch like the one
  used with TOEFL iBT
• Constraints on Score Reporting Timelines
• Concerns about capacity in new IBT Network
• Primarily in large domestic urban areas

25
rGRE Contingency Planning

• Strategies
• Actively Recruiting Test Centers Apply now!
• Exploring options for alternative testing sites
• Large Flexible Testing Centers (e.g., Commercial
  sites)
• Mobile Test Centers
• Exploring options for increasing the number of
  administration dates
• Goal is to allow every test taker to be able to
  schedule a test within 30 days in their preferred
  area for testing.

26
rGRE Communications

27
Key Communications Activities

• Print Materials
• Web Seminars
• rGRE eNewsletter
• Annotated PowerPoint Presentations
• Outreach to Academic Discipline Societies
• Conference Presentations
• Academic Discipline Societies
• CGS Regional Meetings, CHBGS, NAGAP
• DVD/Video Presentations
• General Overview
• Specific topics (e.g., Using the Revised GRE
  Scores for Verbal and Quantitative)

28
Key Communications Activities

• Just-in-time communications to Score
  Recipients, or What happened to the 200-800
  scores and what the heck do these new scores
  mean?
• Reason for Score Scale Changes
• Nature of New Score Scales
• How to Use Score Scales

29
Need More Information?

• To get the most up-to-date information about the
  changes to the GRE General Test
• Visit the GRE Web site at www.ets.org/gre/revgente
  st.html
• Subscribe to the quarterly GRE eNewsletter (see
  link on www.ets.org/gre)
• If you have questions
• E-mail David Payne at dpayne_at_ets.org

30
Test Details and Additional Support Information
31
Benefits of Using GRE Test Scores

• Measure skills faculty and graduate deans have
  identified as essential to graduate school
  success, including verbal reasoning, quantitative
  reasoning, critical thinking and analytical
  writing
• Provide the only common measures of these skills
  for comparing the qualifications of applicants
  from different educational backgrounds and
  countries of origin

32
Benefits of Using GRE Test Scores (contd)

• Furnish independent information to supplement the
  evaluation of grades and recommendations
• Offer a valid predictor of graduate school
  performance, as confirmed by a recent independent
  study using 82,000 graduate students and over
  1,700 studies containing validity data for GRE
  tests
• Ensure that essay responses are the original work
  of the examinee

33
Verbal Reasoning and Quantitative Reasoning
Sections
34
The New Verbal Reasoning Section

• Measures the ability to
• Understand the meanings of words, sentences, and
  entire texts understand relationships among
  words and among concepts
• Select important points distinguish major from
  minor or irrelevant points summarize text
  understand the structure of a text
• Analyze and draw conclusions from discourse
  reason from incomplete data identify
  authors/speakers assumptions and/or
  perspective understand multiple levels of
  meaning (such as literal, figurative, texts
  intent, etc.)

35
Sample Verbal Question

• While chocolate was highly esteemed in
  Mesoamerica, where it originated, its adoption in
  Europe was initially slow. There is a common
  belief that Europeans needed to transform
  chocolate to make it appetizing. However, while
  Spaniards did put sugar, which was unknown to
  indigenous Americans, into chocolate beverages,
  this additive was not completely innovative.
  Mesoamericans were already sweetening chocolate
  with honey, and the step from honey to sugar
  increasingly more available than honey because of
  expanding sugar plantations in the Americas is a
  small one. Likewise, although Spaniards adjusted
  Mesoamerican recipes by using European spices,
  the spices chosen suggest an attempt to replicate
  harder-to-find native flowers. There is no
  indication the Spaniards deliberately tried to
  change the original flavor of chocolate.
• Click on the sentence that presents a
  misconception that the passage challenges.

36
The New Quantitative Reasoning Section

• Measures the ability to
• Understand quantitative information
• Interpret and analyze quantitative information
• Solve problems using mathematical models
• Apply basic mathematical skills and elementary
  mathematical concepts of arithmetic, algebra,
  geometry, probability, and statistics

37
Sample Quantitative Question

• The table shows the distribution of prices of
  45 houses for sale in a certain region. Select
  two of the following choices and place them in
  the blanks below so that the resulting statement
  is true.
• 175,000 185,000 190,000 at
  most at least
• 
  42,000 57,000
• If the highest price of the 45 houses is
                     , then the range of the prices
  of the 45 houses is                    .
• Click on a choice, then click on a blank.

38
Best Ways to Use Revised General Test Scores (V
and Q)

• The new scaled scores (and new score equivalents)
  are a good way to evaluate all applicants.
• Percentile information provides information about
  the relative standing of the applicant compared
  to the fall 2007 test-taking population.
• Broad intended graduate major field score
  distributions provide a good way to evaluate
  applicants within a particular group of
  disciplines.

39
Best Ways to Use Revised General Test Scores
(continued)

• For applicants with multiple scores (both old and
  new), use the new score scales (for old General
  Test scores, use the approximate score equivalent
  new scores) whichever method used (e.g.,
  average, most recent, or highest) should be used
  consistently
• Programs should use multiple pieces of
  information in making admissions decisions (e.g.,
  GPA, GRE scores, letters of recommendation).

40
Best Ways to Use Revised General Test Scores
(continued)

• When there is a mismatch between the level of
  skill in application writing samples (e.g.,
  personal statement) and the Critical Thinking and
  Analytical Writing scores, it may be helpful to
  view the applicants essay responses on the
  Critical Thinking and Analytical Writing section.

41
If You Have Score Use or Score Interpretation
Questions

• The GRE Board has a long history of encouraging
  appropriate score use
• See the GRE Board Score Use Guidelines on the GRE
  Web site at www.ets.org/gre/edupubs.html for a
  fuller description
• Visit the URL above to view score interpretative
  information for the General Test and Subject
  Tests
• The GRE Program is happy to answer score
  interpretation questions from GRE score users.
  Individuals should contact the GRE Program
• By telephone at 609-683-2002
• Via email at gretests_at_ets.org

42
Critical Thinking and Analytical Writing
43
Sample Argument Topic

• Hospital statistics regarding people who go to
  the emergency room after roller-skating accidents
  indicate the need for more protective equipment.
  Within that group of people, 75 percent of those
  who had accidents in streets or parking lots had
  not been wearing any protective clothing
  (helmets, knee pads, etc.) or any
  light-reflecting material (clip-on lights,
  glow-in-the-dark wrist pads, etc.). Clearly, the
  statistics indicate that by investing in
  high-quality protective gear and reflective
  equipment, roller skaters will greatly reduce
  their risk of being severely injured in an
  accident.
• Write a response in which you examine the
  arguments unstated assumptions making sure to
  explain how the argument depends on the
  assumptions and what the implications are if the
  assumptions prove unwarranted.

44
Sample Issue TaskReliance on Technology

• "As people rely more and more on technology to
  solve problems, the ability of humans to think
  for themselves will surely deteriorate.
• Discuss the extent to which you agree or disagree
  with the statement above and explain your
  reasoning for the position you take. In
  developing and supporting your position, you
  should consider ways in which the statement might
  or might not hold true and explain how those
  considerations shape your position.

45
How Essay Responses Are Evaluated

• GRE readers, who are college and university
  faculty, read each essay response and evaluate
  its overall quality according to how well the
  test taker
• Responds to the specific instructions on the
  issue
• Considers the complexities of the issue
• Organizes, develops, and expresses his/her ideas
• Supports his/her position with relevant reasons
  and/or examples
• Controls the elements of standard written English

46
Understanding How the Critical Thinking and
Analytical Writing Section is Scored

• Each essay is scored by two trained readers,
  using a 6-point holistic scale
• Two scores that differ by more than one point are
  adjudicated by a third reader
• Scores from the two readings of an essay are
  averaged
• The final scores on the two essays are then
  averaged and rounded up to the nearest half-point
  interval
• A single score is reported for the section
• Scoring guides and score level descriptions are
  available on the GRE Web site at www.ets.org/gre

47
Selected Score Level Descriptions

• 5.5 and 6 Sustains insightful, in-depth
  analyses of complex ideas develops and supports
  main points with logically compelling reasons
  and/or highly persuasive examples is well
  focused and well organized skillfully uses
  sentence variety and precise vocabulary to convey
  meaning effectively demonstrates superior
  facility with sentence structure and language
  usage but may have minor errors that do not
  interfere with meaning
• 3.5 and 4 Provides competent analysis of
  complex ideas develops and supports main points
  with relevant reasons and/or examples is
  adequately organized conveys meaning with
  reasonable clarity demonstrates satisfactory
  control of sentence structure and language usage
  but may have some errors that affect clarity.
• 1.5 and 2 Displays serious weaknesses in
  analytical writing. The writing is seriously
  flawed in at least one of the following ways
  serious lack of analysis or development lack of
  organization serious and frequent problems in
  sentence structure or language usage, with errors
  that obscure meaning.

48
Revised General TestOverall Information
49
Test Structure
Questions in this section are either being tried
out for possible use in future tests or are being
used to ensure that scores on new editions of the
test are comparable to scores on earlier test
editions.
50
Test Administration Changes
For statistical equating purposes, some test
questions need to be reused across test
editions.
51
Test Administration Changes (continued)
52
Timeline for Major Events

• Fall 2006
• Sample questions available on GRE Web site
  (www.ets.org/gre)
• Score use information available
• New format for score reports described
• Spring 2007
• Test preparation materials available
• Test administration schedule available

53
Timeline for Major Events (continued)

• July 2007
• Registration for the revised General Test opens
• September 2007
• Revised GRE General Test launched
• Internet-Based testing in a new network
• October 2007
• New Verbal and Quantitative score scales
  finalized

54
Timeline for Major Events (continued)

• November 2007
• Scores reported for first 3 administrations
  (approx 4-5 weeks after 3rd administration)
• Percentiles available on GRE Web site and printed
  on score reports
• Concordance table available on the GRE Web site
• Approximate score equivalents on the new scale
  included on score reports for Verbal and
  Quantitative scores earned prior to September
  2007
• Essay responses made available to Score
  Recipients
• January 2008
• Broad major field score distributions available
  on the GRE Web site.

55
GRE Validity Research
56
Independent GRE Validity Research

• In order to examine the predictive validity of
  the GRE General Test and Subject Tests, Nathan R.
  Kuncel, Sarah A. Hezlett, and Deniz S. Ones
  conducted a meta-analysis of all available GRE
  validity studies that had been conducted by
  independent researchers over the past several
  years.
• Their study included
• 1,753 independent samples of students
• Over 80,000 graduate students
• Refinements in study included
• A very large database of studies
• Statistical corrections for restriction of range
• Statistical corrections for unreliable criterion
  measures

57
Kuncel, Hezlett Ones

• General Test was found to be a generalizably
  valid predictor of
• First-year graduate grade point average
• Overall graduate grade point average
• Comprehensive exam scores
• Publication citation counts
• Faculty ratings
• Correlations with degree attainment and research
  productivity were consistently positive, but not
  statistically significant in all cases.

58
Results of Kuncel et al. Research

• A follow-up study by Kuncel indicated the GRE
  General Test has similar levels of predictive
  validity for both master's and doctoral level
  programs.
• The GRE General Test has better predictive
  validity than undergraduate grades or letters of
  recommendation.
• Subject Tests were found to be better predictors
  of success than were either the General Test or
  undergraduate grade point average.
• These data strongly suggest that the GRE is a
  valid predictor of graduate student performance
  across disciplines, degree levels, age of
  students, etc.

59
Results of Kuncel et al. Research (continued)
For more information about the Kuncel et al.
research study, see A Comprehensive
Meta-Analysis of the Predictive Validity of the
Graduate Record Examinations Implications for
Graduate Student Selection and Performance Natha
n R. Kuncel Sarah A. Hezlett Deniz S.
Ones Psychological Bulletin 2001, Vol. 127, No.
1, 162-181
60
GRE Communications
61
rGRE eNewsletter

• Content
• Articles about aspects of the revised General
  Test and/or GRE tests/services
• Links to upcoming conferences
• Links to important information on GRE Web site
• Quarterly publication

62


63
Web Seminars

• Online, web-based seminars about the revised
  General Test
• Conducted by GRE Program Staff
• Multiple Sessions, multiple topics
• Overview of the Revised General Test
• How to Use RGRE Scores in Admissions
• Overview of the Analytical Writing Measure
• Email invitations and online registration
• Multiple Audiences
• Score recipients/faculty/administrators
• Undergraduate faculty
• Test Prep trainers
• Promote the Website and rGRE eNewsletter

64
PowerPoint Presentations

• Presentations about rGRE will be available on the
  GRE Web site
• Multiple topics
• Include detailed speaker notes
• Presentations for various audiences
• Administrators
• Faculty
• Test prep trainers
• Promotes the Website and eNewsletter