Analyzing

1
Analyzing Interpreting Data

• Assessment Institute
• Summer 2005

2
Categorical vs. Continuous Variables

• Categorical Variables
• Examples ? Students major, enrollment status,
  gender, ethnicity also whether or not the
  student passed the cutoff on a test
• Continuous Variables
• Examples ?GPA, test scores, number of credit
  hours.
• Why make this distinction?
• Whether a variable is categorical or continuous
  affects whether a particular statistic can be
  used
• Doesnt make sense to calculate the average
  ethnicity of students!

3
Averages

• Typical value of a variable
• In assessment we commonly compare averages of
• Different groups
• Each group consists of different people
• Avg. score on a test for students in different
  classes
• Different occasions
• Same people tested on each occasion
• Avg. score on a test for students who took the
  test as freshmen an then again when they were
  seniors

4
Before calculating an average

• Check to make sure that the variable
• Is continuous
• Has values in your data set that are within the
  possible limits
• Check minimum and maximum values
• Does not have a distribution that is overly
  skewed
• If so, consider using median
• Does not have any values that would be considered
  outliers

5
Histogram
6
Correlations (r)

• Captures linear relationship between two
  continuous variables (X and Y)
• Ranges from -1 to 1 with values closer to 1
  indicating a stronger relationship than values
  closer to 0 (no relationship)
• Positive values
• High X associated with high Y low X associated
  with low Y
• Negative values
• High X associated with low Y low X associated
  with high Y

7
In this example, dropping cases that appeared to
be outliers did not change the relationship
between the two administrations (r .30),
nor their averages.
Scatterplot Does relationship appear linear? Is
there a problem with restriction of range? Does
there appear to be outliers?
8
Standards

• May want to use standard setting procedures to
  establish cut-offs for proficiency on the test
• Could be that students are gaining
  knowledge/skills over time, but are they gaining
  enough?
• Another common statistic calculated in assessment
  is the of students meeting or exceeding a
  standard

9
A. Are the 29 senior music majors in Spring 2005
scoring higher on the Vocal Techniques 10-item
test than last years 20 senior music majors?

• Compare averages of different groups

Yes, this years seniors scored higher (M 6.72)
than last years (M 6.65).
10
B. Are senior kinesiology majors in different
concentrations (Sports Management vs. Therapeutic
Recreation) scoring differently on a test used to
assess their core kinesiology knowledge?

• Compare averages of different groups

11
(No Transcript)
12
C. On the Information Seeking Skills Test (ISST),
what percent of incoming freshmen in Fall 2004
met or exceeded the score necessary to be
considered as having proficient information
literary skills?

• Percent of students meeting and exceeding a
  standard

Of the 2862 students attempting the ISST, 2751
(96) met or exceeded the proficient standard.
13
D. Are the well-being levels (as measured using
six subscales - e.g., self-acceptance, autonomy,
etc.) of incoming JMU freshmen different than the
well-being levels of adults?

• Compare averages of different groups (JMU
  students vs. adults)
• More than one variable (six different subscales)

14
While the practical significance of the
differences for Self-Acceptance and Purpose in
Life are considered small (d.14 and d.25), the
differences for Autonomy (d.50) and
Environmental Mastery (d.35) are considered
medium and small to medium, respectively.
Similarities JMU Incoming Freshmen seem to be
similar to the adult sample (N 1100) in
Positive Relations with Others and Personal
Growth. Differences JMU incoming freshmen have
significantly lower Autonomy and Environmental
Mastery well-being compared to the adult sample
and significantly higher Self-Acceptance and
Purpose in Life.
15
E. Are students scoring higher on the Health and
Wellness Questionnaire as sophomores compared to
when they were freshmen? Does the difference
depend on whether or not they have completed
their wellness course requirement?

• Comparing Means Across Different Occasions for
  Different Groups

16
(No Transcript)
17
F. Are the writing portfolios collected in the
fall semester yielding higher ratings than
writing portfolios collected in the spring
semester?Are the differences between the
semesters the same across three academic years?

• Compare averages of different groups
• Six different groups (fall and spring for each
  academic year)

18

• In the 2001-2002 and 2002-2003 academic years,
  fall portfolios were rated slightly higher than
  spring portfolios. In the most current academic
  year, the fall and spring portfolio averages were
  about the same. There doesnt seem to be
  overwhelming evidence that the difference between
  fall and spring portfolios is of importance.

19
G. Are students who obtained transfer or AP
credit for their general education sociocultural
domain course scoring differently on the 27-item
Sociocultural Domain Assessment (SDA) than
students who completed their courses at JMU?

• Compare averages of different groups

• JMU students N 369, M 18.63, SD 3.83
• AP/transfer students N 29, M 18.55, SD
  3.68
• Difference was not statistically, t(335).11, p
  .92, nor practically significant (d .02).

20
G. What is the relationship between a students
general education sociocultural domain course
grade and their score on the 27-item
Sociocultural Domain Assessment (SDA)?

• Relationship between two variables, finally!

21
Inferential Statistics

• How likely is it to have found results such as
  mine in a population where the null hypothesis is
  true?
• Comparing Averages of Different Groups
• Independent Samples T-test
• Null ? Groups do not differ in population means
• Comparing Averages Across Different Occasions
• Paired Samples T-test
• Null ? Occasions do not differ in population
  means
• Correlation
• Null ? No relationship between variables in the
  population

Typically, want to reject the null p-value
22
Effect Sizes and Confidence Intervals

• Statistical significance is a function of both
  the magnitude of the effect (e.g., difference
  between means) and sample size
• Supplement with confidence intervals and effect
  sizes
• SPSS provides you with confidence intervals
• Can use Wilsons Effect Size Calculator to obtain
  effect sizes

23
Wellness Domain ExampleGoals Objectives
Students take one of two courses to fulfill this
requirement, either GHTH 100 or GKIN 100.
24
Knowledge of Health and Wellness (KWH) Test
Specification Table
25
Data Management PlanWellness_Data.sav (N 105)
Missing data indicated for all variables by "."
26
Item Analysis
Item 1
27
Item Difficulty

• The proportion of people who answered the item
  correctly (p)
• Used with dichotomously scored items
• Correct Answer - score1
• Incorrect Answer - score0
• Item difficulty a.k.a. p-value
• Dichotomous items
• Meanp
• Variancepq, where q 1-p

28
SPSS output for 1st 6 items of 35 item GKIN100
Test3 Spring 2005
Std Dev is a measure f the variability in the
item scores
Mean is item difficulty (p)
Mean Std
Dev Cases 1. ITEM1 .5609
.4972 271.0 2. ITEM2
.9520 .2141 271.0 3.
ITEM3 .8598 .3479
271.0 4. ITEM4 .7454
.4364 271.0 5. ITEM5
.6089 .4889 271.0 6. ITEM6
.5793 .4946 271.0
Sample size on which analysis is based
58 of the sample obtained the correct response
to Item 6. The difficulty or p-value of Item 6 is
.58
29
Easiest Hardest Items
p .99 EASIEST

• 25. Causes of mortality today are
• A. the same as in the early 20th century.
• B. mostly related to lifestyle factors.
• C. mostly due to fewer vaccinations.
• D. a result of contaminated water.
• 34. Which of the following is a healthy lifestyle
  that influences wellness?
• A. brushing your teeth
• B. physical fitness
• C. access to health care
• D. obesogenic environment

p .14 HARDEST
30
Item Difficulty Guidelines

• High p-values, item is easy low p-values, item
  is hard
• If p-value1.0 (or 0), everyone answering
  question correctly (or incorrectly) and there
  will be no variability in item scores
• If p-value too low, item is too difficult, need
  revision or perhaps test is too long
• Good to have a mixture of difficulty in items on
  test
• Once know difficulty of items, usually sort them
  from easiest to hardest on test

31
Item Discrimination

• Correlation between item score and total score on
  test
• Since dealing with dichotomous items, this
  correlation is usually either a biserial or
  point-biserial correlation
• Can range in value from -1 to 1
• Positive values closer to 1 are desirable

32
Item Discrimination Guidelines

• Item discrimination can the item separate the
  men from the boys (women from the girls)
• Can the item differentiate between low or high
  scorers?
• Want high item discrimination!
• Consider dropping or revising items with
  discriminations lower than .30
• Can be negative, if so check scoring key and if
  the key is correct, may want to drop or revise
  item
• a.k.a. rpbis or Corrected Item-Total Correlation

33
Scatterplot of relationship between item 2 score
(0 or 1) and total score rpbis .52
If I know you item score, I have a pretty good
idea as to what your ability level or total score
is.
0 1
Scatterplot of relationship between item 17 score
(0 or 1) and total score rpbis .18
If I know you item score, I DO NOT have a pretty
good idea as to what your ability level or total
score is.
0 1
34
SPSS output for 1st 6 items of 35 item GKIN100
Test3 Spring 2005
Corrected Item-Total Correlation is Item
Discrimination (rpbis)
Why is it called corrected item-total
correlation? The corrected implies that the total
is NOT the sum of all item scores, but the sum of
item scores WIHTOUT including the item in
question.
35
Percentage of sample choosing each alternative.
Average total test score for students who chose
each alternative.
A 1 B 2 C 3 D 4 9 Missing
Notice how the highest average total test score
(M 27.65) is associated with the correct
alternative (B). All other means are quite a bit
lower. This indicates that the item is
functioning well and will discriminate.
This information is for item 2, where the item
difficulty and discrimination were p .95,
rpbis .52
36
Percentage of sample choosing each alternative.
Average total test score for students who chose
each alternative.
A 1 B 2 C 3 D 4 9 Missing
Notice how the highest average total test score
(M 27.91) is associated with the correct
alternative (C). Unlike item 2, with this item
all other means are fairly close to 27.91. This
indicates that the item does not discriminate as
well as item 2.
This information is for item 17, where the item
difficulty and discrimination were p .697,
rpbis .18
37

• Did this mainly for those items that were
  difficult (p (rpbis

• Took information from SPSS distractor analysis
  output and put it in the following graph.

4. The DSHEA of 1994 has A. labeled certain
drugs illegal based on their active
ingredient. B. caused health food companies to
lose significant business. C. made it easier for
fraudulent products to stay on the
market. D. caused an increase in the cost of
many dietary supplements.
38
Hard item - but pattern of means indicates it is
not problematic.
31. aging relates to lifestyle. A. Time-depende
nt B. Acquired C. Physical D. Mental
39

• 10.Chris MUST get a beer during the commercials
  each time he watches the NFL. Which stage of
  addiction does this demonstrate?
• Exposure
• Compulsion
• Loss of control
• This is not an example of addiction.

This item may be problematic - students choosing
"C" scoring almost as high on the test overall as
those choosing "B".
40
Other Information from SPSS

• Descriptive Statistics for total score.

N of Statistics for Mean
Variance Std Dev Variables SCALE
27.1033 19.1152 4.3721 35
items on the test
Average total score
Average of points by which total scores are
varying from the mean

• An measure of the internal consistency
  reliability for your test called coefficient
  alpha.

Alpha .7779
Ranges from 0 ? 1 with higher values indicating
higher reliability. Want it to be .60
41
Test Score Reliability

• Reliability defined extent or degree to which a
  scale/test consistently measures a person
• Need a test/scale to be reliable in order to
  trust the test scores! If I administered a test
  to you today, wiped out your memory, administered
  it again to you tomorrow you should receive the
  same score on both administrations!
• How much would you trust a bathroom scale if you
  consecutively weighed yourself 4 times and
  obtained weights of 145, 149, 142, 150?

42
Internal Consistency Reliability

• Internal consistency reliability extent to which
  items on a test are highly intercorrelated
• SPSS reports Cronbachs coefficient alpha
• Alpha may be low if
• Test is short
• Items are measuring very different things
  (several different content areas or dimensions)
• Low variability in your total scores or small
  range of ability in the sample you are testing
• Test only contains either very easy items or very
  hard items