Evaluating the Effectiveness of Feedback in SQL-Tutor

1
Evaluating the Effectiveness of Feedback in
SQL-Tutor

• Antonija Mitrovic, Brent Martin
• Intelligent Computer Tutoring Group
• Computer Science Department
• University of Canterbury
• Christchurch, New Zealand

2
Overview

• SQL-Tutor
• Feedback types
• Evaluation study
• Results

3
Architecture of the standalone version of
SQL-Tutor
4
(No Transcript)
5
SQLT-Web a Web-enabled SQL tutor

• http//ictg.cosc.canterbury.ac.nz8000/sql-tutor/
  
• Interface developed in CL HTTP-server
• Used in COSC courses since 1999
• Open for outside users on March 29, 2000

6
Architecture of SQLT-Web
7
(No Transcript)
8
Feedback types

• Positive/negative
• Error flag
• Hint
• All errors
• Partial solution
• Complete solution

9
Hypothesis

• Positive/negative and Error flag too general
• Low-level feedback contraproductive
• (partial and complete solution)
• Constraint-based feedback most effective
• (hint and all errors)

10
Evaluation study

• COSC313, May 1999 (Web)
• Single 2-hour session
• 6 lectures 8 hours of labs
• 33 students

11
Three groups

• Detailed (complete and partial solution)
• General (hint and all errors)
• Limited (pos/neg and error flag)

12
Results 1
Group Solved () NoAttempts Time/attempt
Detailed 87.07 2.21 65.26s
General 83.49 2.16 47.80s
Limited 84.10 2.24 78.05s
Difference in times is significant
13
Results 2
Feedback Const Succ Fail Learn Corr ()
Pos/neg 436 254 636 29 78.0
Error flag 116 98 126 44 81.8
Hint 43 33 43 43 74.4
All errors 64 72 81 47 80.0
Partial 26 22 10 69 91.6
Full sol 18 6 16 27 44.2
14
(No Transcript)
15
Initial learning rate

• All errors 0.44
• Error Flag 0.40
• Pos/Neg 0.29
• Hint 0.26
• Partial 0.15
• Full solution 0.13

16
Conclusions

• General feedback is most effective
• Detailed feedback is detrimental to learning
• Feedback level needs to be adapted