Title: Assessment Using Online Assignment Management Systems
1Assessment Using Online Assignment
ManagementSystems
Glenn Ledder Department of Mathematics University
of Nebraska-Lincoln gledder_at_math.unl.edu
2Assessment Using Online Assignment
ManagementSystems
- Defining the terms
- OAM advantages and disadvantages
- Low-maintenance administration
- Accurate and reproducible results
- Getting meaningful results
3Defining the Terms
4Assessment
5Assessment
- Conflicting meanings
- General meaning measuring academic success in
any context
6Assessment
- Conflicting meanings
- General meaning measuring academic success in
any context - Specific meaning measuring academic success in a
broad context -
7Assessment
- Conflicting meanings
- General meaning measuring academic success in
any context - Specific meaning measuring academic success in a
broad context - (larger than a single course)
8Assessment vs Evaluation
- Dictionary definition
- assess to determine the importance, size, or
value of - evaluate to determine or fix the value of
9Assessment vs Evaluation
- Dictionary definition
- assess to determine the importance, size, or
value of - evaluate to determine or fix the value of
- Connotation
- assessment usually emphasizes the making of
summary judgments - evaluation usually emphasizes the measurement
of quantitative value
10Assessment vs Evaluation
- Assessment
- measuring knowledge and understanding
- Evaluation
- measuring achievement
11Assessment vs Evaluation
- Assessment
- measuring knowledge and understanding
- used for placement
- Evaluation
- measuring achievement
- used to determine course grades
12Assessment vs Evaluation
- Assessment
- measuring knowledge and understanding
- used for placement
- Evaluation
- measuring achievement
- used to determine course grades
- Test an instrument used for assessment,
evaluation, or both
13OAM Advantages and Disadvantages
14OAM Advantages/Disadvantages
- Immediate grading
- (no wait for grading or machine scoring)
15OAM Advantages/Disadvantages
- Immediate grading
- General purpose testing center
- (no need for direct faculty involvement)
16OAM Advantages/Disadvantages
- Immediate grading
- General purpose testing center
- Administrative difficulties
- (How do you maintain the database?)
17OAM Advantages/Disadvantages
- Immediate grading
- General purpose testing center
- Administrative difficulties
- Algorithmic Qs and symbolic As
- (individual answers, no prompts)
18OAM Advantages/Disadvantages
- Immediate grading
- General purpose testing center
- Administrative difficulties
- Algorithmic Qs and symbolic As
- Difficulties in writing questions
- (How to avoid tricky, picky, sticky?)
19Low-Maintenance Administration
20Low-Maintenance Administration
- Lots of instructors and proctors
- Should require minimal knowledge
21Low-Maintenance Administration
- Lots of instructors and proctors
- Should require minimal knowledge
- One or two faculty manage the system
- Should require minimal time
22Low-Maintenance Administration
- Lots of instructors and proctors
- Should require minimal knowledge
- One or two faculty manage the system
- Should require minimal time
- Questions are permanent.
- Student records are transient.
23The Math 106 EDU folder structure
- 106 COURSE
- Question banks
- Gateway exam
- Practice assignments
The 106 COURSE data is permanent. The only
regular changes are to the Gateway exam
dates. Maintained by faculty manager.
24The Math 106 EDU folder structure
- 106 COURSE
- Question banks
- Gateway exam
- Practice assignments
- 106 CLASS
- Question banks
- Gateway exam
- Practice assignments
- Student records
The 106 COURSE data is permanent. The only
regular changes are to the Gateway exam
dates. Maintained by faculty manager.
Students register for their own 106 CLASS. Tests
are inherited. Student records are
local. Maintained by individual
instructors/proctors.
25Accurate and Reproducible Results
26Reproducibility
- Tests should measure something.
- The test has to yield consistent results in
multiple administrations for a single student.
27Reproducibility
- Tests should measure something.
- The test has to yield consistent results in
multiple administrations for a single student. - But questions must change when a test is given at
different times. - This is especially important for evaluations.
28Reproducibility
- Use template problems to get a great variety of
answers. - Use template problems to get uniformity of
content and difficulty. - Group problems into categories that are
consistent in content and difficulty.
29The Math 106 Gateway Exam
10 questions, 8 correct to pass
- Elementary functions
xn, sin(ax), cos(ax),
tan(ax), eax, ln x, nx - 2. Products 3. Quotients 4. Compositions
- 5. Compositions of compositions
- 6. Products with a composite factor
- 7. Compositions of products
- 8. Quotients with an embedded composition
- 9. Quotients with an embedded product
- 10. Functions defined by equations
30Category 4 - Compositions
- A, C, N gt 0 B ? 0 K ? 0, 1
The algorithm chooses 5 random integers
satisfying certain requirements.
31Category 4 - Compositions
- A, C, N gt 0 B ? 0 K ? 0, 1
- X t, u, v, w, x, y, z
The algorithm randomly chooses an independent
variable.
32Category 4 - Compositions
- X t, u, v, w, x, y, z A,C,Ngt0 B?0
K?0,1 - P XNB, XNBX
- Q AXNB, AXNBX, sqrt(X)B
- S sin AX, cos AX, tan AX
- T e -CXB, eKXBX
- U Ae -CXB, AeKXBX, A ln X, ANX
The algorithm randomly creates functions P, Q, S,
T, U, using some of the chosen integers.
33Category 4 - Compositions
- X t, u, v, w, x, y, z A,C,Ngt0 B?0
K?0,1 - P XNB, XNBX
- Q AXNB, AXNBX, sqrt(X)B
- S sin AX, cos AX, tan AX
- T e -CXB, eKXBX
- U Ae -CXB, AeKXBX, A ln X, ANX
- F1 sqrt(P), sqrt(S), sqrt(T)
- F2 SN, TN
- F3 ln Q, ln CS
- F4 eQ, eCS
- F5 sin Q, cos Q, sin U, cos U
The algorithm chooses one of 5 functions based on
prior elements. There are 38 templates, each with
7 independent variables and at least one
parameter
34Reproducibility
- Tests should measure something.
- The test has to yield consistent results in
multiple administrations for a single student.
35Accuracy
- Tests should measure what they were intended to
measure.
36Accuracy
- Tests should measure what they were intended to
measure. - Questions should have to be done right to be
counted right.
37Accuracy
- Tests should measure what they were intended to
measure. - Questions should have to be done right to be
counted right. - Students who can do what the test is supposed to
measure should get a good score.
38Counted right ? Done right
- Savvy test takers can do well on multiple choice
tests even when they dont understand the
material.
39Counted right ? Done right
- Savvy test takers can do well on multiple choice
tests even when they dont understand the
material. - For calculations, students should have to type in
the answer.
40Counted right ? Done right
- Savvy test takers can do well on multiple choice
tests even when they dont understand the
material. - For calculations, students should have to type in
the answer. - For conceptual questions, use multiple selection.
(Which of the following)
41Can do ? Good score
- Tricky Misinterpretation or small mistakes
account for many wrong answers.
42Can do ? Good score
- Tricky Misinterpretation or small mistakes
account for many wrong answers.
Find the derivative of cos e2x and Find the
derivative of e-2 cos x are tricky
43Can do ? Good score
- Tricky Misinterpretation or small mistakes
account for many wrong answers.
Find the derivative of cos e2x and Find the
derivative of e-2 cos x are tricky, compared
to Find the derivative of e2 cos x
44Can do ? Good score
- Tricky Misinterpretation or small mistakes
account for many wrong answers.
Find the derivative of cos e2x and Find the
derivative of e-2 cos x are tricky, compared
to Find the derivative of e2 cos x
The problem is meant to test the chain rule, not
elementary algebra.
45Can do ? Good score
- Picky Too many details connect minor errors with
wrong answers.
Find the derivative of cos2(2x3) 4 sin x.
Find the derivative of 4x5-2x cos (ex2).
46Can do ? Good score
- Picky Too many details connect minor errors with
wrong answers.
Find the derivative of cos2(2x3) 4 sin x.
Find the derivative of cos2(2x3).
Find the derivative of 4x5-2x cos (ex2).
Find the derivative of ex5 cos (ex2).
47Can do ? Good score
- Picky Too many details connect minor errors with
wrong answers.
Find the derivative of cos2(2x3) 4 sin x.
Find the derivative of cos2(2x3).
The simpler problems are hard enough.
Find the derivative of 4x5-2x cos (ex2).
Find the derivative of ex5 cos (ex2).
48Can do ? Good score
- Sticky Difficulties in transferring answer from
paper to computer.
2x2 x3
4x(x3)-2x2 (x3)2
Find the derivative of .
49Can do ? Good score
- Sticky Difficulties in transferring answer from
paper to computer.
2x2 x3
4x(x3)-2x2 (x3)2
Find the derivative of .
2x2 x3
32 25
Find the derivative of at x 2.
50Can do ? Good score
- Sticky Difficulties in transferring answer from
paper to computer.
2x2 x3
4x(x3)-2x2 (x3)2
Find the derivative of .
2x2 x3
32 25
Find the derivative of at x 2.
2x2 (x3)2
Type 4x(x3)-2x2 / (x3)2, get
4x(x3)-
51Getting Meaningful Results
52Accuracy
- Tests should measure what they were intended to
measure.
53Getting Meaningful Results
- Tests should measure what they were intended to
measure. - What should OAM evaluations and assessments be
intended to measure?
54Dont give up paper evaluations
55Dont give up paper evaluations
- Use paper exams for questions that demand partial
credit and questions where the answer is an
integral, a graph, or an explanation.
56Dont give up paper evaluations
- Use paper exams for questions that demand partial
credit and questions where the answer is an
integral, a graph, or an explanation. - Use OAM evaluation for routine computations and
basic concepts.
57Dont give up paper evaluations
- Use paper exams for questions that demand partial
credit and questions where the answer is an
integral, a graph, or an explanation. - Use OAM evaluation for routine computations and
basic concepts. - Testing takes time and effort. Decide what is
important and then create the test to match. - Set high standards and allow retakes.
58Do give up paper assessments
59Do give up paper assessments
- Assess routine computations and basic concepts.
60Do give up paper assessments
- Assess routine computations and basic concepts.
- Expect very few retakes. Qualified students
should pass on the first try. - Provide necessary formulas online or as a printed
supplement. Test conceptual understanding and
knowledge, not memory for trivia.