The Ppite project

1
The Pépite project
Automatic Multi-criteria Assessment of Open-Ended
Questions a case study in school algebra
Élisabeth Delozanne, Paris Universitas, UPMC D.
Prévit, B. Grugeon, F. Chenevotot
ITS2008
2
Cognitive modeling authoring tool

• Problem
• Multi-step reasoning, multiple equivalent
  reasonings
• Our approach
• An expert teacher (or a researcher) defines
  diagnosis exercises
• A cognitive engineer implements templates that
  generalize these particular diagnosis exercises
• A teacher clones these diagnosis exercises by
  filling template forms
• A domain specific application
• generates the clone and a set of plausible
  correct and incorrect anticipated solutions
• matches the students reasoning with anticipated
  solutions

3
Outline

• An introductory example
• Pépite a specific diagnosis tool
• PépiGen a system to clone Pépite
• Authors and Students points of view
• Automatic Diagnosis
• How does it work ?
• Pépinière
• Formal processing of expression trees
• Conclusion

in French tree nursery
4
Blandine
5
Aliou
6
Definitions

• Diagnosis exercise
• An exercise (statement and user interface)
• an analysis grid to assess every plausible
  solution anticipated by experts
• Clone
• A similar exercise
• has the same kind of statement and user interface
• gives the same kind of information on students
  competence
• an analysis grid
• to assess every plausible solution
• automatically generated by the system

7
PépiGen

• A system to clone the Pépite diagnosis tool
• An author (a teacher)
• Chooses an exercise to be cloned
• Enters the statement of the clone
• PépiGen generates
• The students interface
• Each plausible solution (correct or incorrect)
  and its assessment on several dimensions

8
The Authors interface
9
The Students interface
10
The Automatic Diagnostic
11
Outline

• An introductory example
• Pépite a specific diagnosis tool
• PépiGen a system to clone Pépite
• How does it work ?
• Pépinière
• Expanding the tree of plausible steps of correct
  and incorrect algebraic transformations
• Walking through the tree to anticipate different
  solutions and their assessment
• Diagnosing the students reasoning
• Conclusion

12
Plausible steps
(x6)3-3x
Correct rules
R1
R2
R1 (AB)C ACBC R3 ABAC
A(BC)
x363-3x
x63-3x
Incorrect rules
R3
R5
R2 (AB)C ABC R4 ABC
B(AC) R5 ABC (AB)C
-2x18
3x18-3x
3x18-3x
R3
R4
R3
R4
18
21x-3x
21x-3x
18
R3
R3
18x
18x
V3,EA31EA42
V3,EA31
V1,EA1
V3,EA42
V3,EA32
13
Analysis grid generation

• PépiGen
• sends the algebraic expression to Pépinière that
  returns a tree of plausible steps
• Validity and Algebraic Expression Writing
• completes the plausible solutions set with
• Non optimal algebraic
• completes each solution assessment on the 5
  dimensions
• V, EA, L, T, J
• saves each algebraic solution and its assessment
• XML file solution analysis grid
• Note
• arithmetic reasonings are analyzed by the
  diagnosis system

14
Analysis grid (extract)

• ltUnexpectedCorrectSolutionsgt ()
• ltCommentgtAlgebraic proof the student interprets
  the statement as an equationlt/Commentgt
• ltSolutiongt
• ltAssessmentgtV2,EA1,L1,T1,J1lt/Asssessmentgt
  
• ltExpressiongt(x6)3-3x 18lt/Expressiongt
• ltExpressiongtx363-3x 18lt/Expressiongt
• ltRulegtC,3lt/Rulegt
• ltExpressiongtx318-3x 18lt/Expressiongt
• ltExpressiongt18 18lt/Expressiongt
• lt/Solutiongt
• lt/UnexpectedCorrectSolutiongt

15
Automatic diagnosis
XM L
loaded
Students reasoning
save
Diagnosis system
XM L
loaded
Equivalent expression tree ?
Students reasoning assessment
True/False
XM L
Analysis grid
ExpressionsTree processor Pépinière
16
Diagnosis algorithm

• Numerical or algebraic approach?
• Loop on each expression of the students
  reasonning
• Build the expression tree (ST)
• Loop on each Plausible solution in the analysis
  grid
• Build the expression tree (PT)
• If numerical approach
• substitute the numerical value in PT
• If ST ? PT
• keep PT, the rule and the comment and stop
• At the end
• walk through PT to set up the final assessment
• save the final assessment, the comment and the
  applied rules

17
Results and tests

• On going work
• A demonstration prototype implements a complex
  exercise cloning
• Authoring clones
• Solving
• Diagnosing
• Preliminary Tests
• assessment of a corpus of 141 students solutions
• Multi-step reasoning
• Multiple equivalent reasonings
• 3 teachers tested it in the lab

18
Discussion

• Diagnosis
• compared with model tracing
• Tree of plausible steps correct and incorrect
  rules
• Emphasis whole reasoning/step-by-step
• Several students types of reasoning derived from
  a single solution branch
• Multidimensional assessment
• Authoring
• Filling template forms
• Limited to specified exercises
• Automatic multidimensional diagnosis validated by
  experts
• No programming, no modeling for teachers

19
Automatic Multi-criteria Assessment

• Our proposal
• Teachers clone a diagnosis tool previously
  designed by experts
• The cloning process relies on
• A preliminary educational study in the domain
• An implementation of templates of diagnosis
  exercises
• A specific application to analyze reasonings that
  are not pre-formated
• Demo Friday afternoon
• http//pepite.univ-lemans.fr