AutoGrader Automatic Grading of programming Assignments

1
AutoGraderAutomatic Grading of programming
Assignments

• By
• Chandandrasekhar Pasupuleti
• Under guidance of
• Dr.Steven Zeil

2
Introduction

• Testing Automation and its prominence
• programming assignments is one domain that poses
  tough challenges to Testing Automation.
• Efficient solution for grading programming
  assignments accurately with minimum effort.

3
Steps in Grading programming assignments

• Design an assignment
• Gather conditions/testing scenarios to test the
  program
• Upon submission, build the program and run it
  against a set of data to check for correctness
• Grade the assignment based on the testing
  performed.
• Send out the results.

4
Automation view of Grading

• Automation has to be achieved right from the
  first step in the process, designing the
  assignment.
• Grading is considered a process, it has to be
  designed from first step to last keeping
  automation in mind. Every step in the process
  should fit in automation model.

5
Why Automation?

• Repeatable
• Fast
• Accurate
• Efficient in many ways

6
Complete Automation

• The process has to be automated.
• Design assignments with probes.
• Look for the probes for grading.
• Testing files to hit the probes.
• Combined with Automated manual labor forms
  Complete Automation.

7
Automation Grading Model

• General information phase
• Configuring the testing environment and program
  to grade
• Build the program
• Execute the program with set of predefined data
• Evaluate the output of the previous step
• Late penalty
• Mail the results
• Clean up

8
The AutoGrader Harness

• The AutoGrader harness is a java bases
  application that runs the scripts written in XML
  .
• Grading file itself is the configuration file.

9
Hot to Install and Run?

• The Package comes in a jar file and can be run by
  using the command
• Java jar grader.jar ltgradingfile.xmlgt
  lttrue/falsegt ltlist of student directoriesgt.
• The AutoGrader Variables
• CurrentStudentDir
• WorkDir
• SubmissionDir
• InstructorCodeDir
• InstructorDataDir

10
Phase-IGeneral Phase

• ltPhase name"GeneralInfo" run"true"gt
• ltTitle value" Assignment Number2 -
  httpchecker"/gt
• ltInstructorsCodeDir value"D\Eclipse\workspace\A
  utoGrader\InstCodeDir"/gt
• ltInstructorsDataDir value"D\Eclipse\workspace\A
  utoGrader\InstDataDir"/gt
• ltStudentSubmissionDir value"D\Eclipse\workspace
  \AutoGrader\Submissions"/gt
• ltStudentEmailList value"D\Eclipse\workspace\Aut
  oGrader\email.xml"/gt
• ltWorkDir value"D\Eclipse\workspace\AutoGrader"/
  gt
• ltExecute function"CheckDir"gt
• ltReportsMessage value"Checking the Directory
  for Copying"/gt
• ltParams value"InstructorsCodeDir"/gt
• lt/Executegt
• lt/Phasegt

11
Phase-IIConfiguration

• ltPhase name"Configure" run"true"gt
• ltExecute function"CopyDir" maxscore "2"
  maxpenalty"2"gt
• ltReportsMessage value"Checking the Directory
  for Copying"/gt
• ltParams value"CurrentStudentDir"/gt
• ltParams value"WorkDir"/gt
• lt/Executegt
• ltExecute function"CheckFile" maxscore"4"
  maxpenalty"4gt
• ltReportsMessage value"Checking the executable
  file.Points will be deducted"/gt
• ltParams value"WorkDir\htmlcheck.exe"/gt
• lt/Executegt
• ltExecute followup"true" function"DeleteFile"
  maxscore"2" Maxpenalty"2"gt
• ltReportsMessage value"Checking the executable
  file.Points will be deducted"/gt
• ltParams value"WorkDir\htmlchec.exe"/gt
• lt/Executegtlt!-- This is we follow this only if
  previous text failed. --gt
• lt/Phasegt

12
Phase-IIIBuild

• ltPhase name"Build" run"true"gt
• ltExecute function"make" maxscore "25"
  maxpenalty"25"gt
• ltReportsMessage value"Executing Make file. On
  Success points will be awarded"/gt
• ltMaxTime value"100" maxscore "10"/gt
• ltMaxFileSize value"1000" maxscore "10"/gt
• ltCaptureOutput value"compile.txt"/gt
• lt/Executegt
• lt!-- now checking for output files after
  execution--gt
• ltExecute function"CheckFile" maxscore "10"
  maxpenalty "10"gt
• ltParams value"htmlcheck.exe"/gt
• lt/Executegt
• lt/Phasegt

13
Phase-IVExecute

• ltPhase name"Execute" run"true"gt
• ltExecute function"htmlcheck.exe" maxscore"20"
  maxpenalty "20" Loop"1,3" Increment"1"gt
• ltReportsMessage value"Checking for making sure
  that .exe is successful"/gt
• ltParams value"InstructorsDataDir\dataN.dat"/gt
  
• ltMaxTime value"100" maxscore "10"/gt
• ltMaxFileSize value"1000" maxscore "10"/gt
• ltCaptureOutput value"outputN.dat"/gt
• lt/Executegt
• lt/Phasegt

14
Phase-ivEvaluate

• ltScoringgt
• ltUnit percentage"10" penalty"10"/gtlt!--
  //first 10 --gt
• ltUnit percentage"20" penalty"10"/gtlt!-- //next
  20 --gt
• ltLinear percentage"30" penalty"10"/gtlt!-- --gt
• lt/Scoringgt
• ltExecute function"Diff" maxscore "50"
  maxpenalty"50" loop"1,3" increment"1"gt
• ltReportsMessage valueChecking the Program
  output"/gt
• ltMaxTime value"100"/gt
• ltMaxFileSize value"1000"/gt
• ltInstructorsData value"InstructorsDataDir\out
  N.dat"/gt
• ltProgramOutputData value"outputN.dat"/gt
• ltIgnoreWhiteSpaces value"true"/gt
• ltIgnoreBlankLines value"true"/gt
• ltIgnoreOrder value"false"/gt
• ltIgnoreExtraLines value"true"/gt
• ltIgnoreCase value"true"/gt
• lt/Executegt

15
Phase-VEvaluate

• ltOracle ClassNamePatternSearch" maxscore "50"
  maxpenalty"50"
• ltScoringgt
• ltUnit percentage"10" penalty"10"/gt
• ltUnit percentage"20" penalty"10"/gt ltLinear
  percentage"30" penalty"10"/gt
• lt/Scoringgt
• ltReportsMessage value"Executing Oracle"/gt
• ltMethod namesetFiletoSearch"gt
• ltParams Type"String" valueWorkDir\output01.d
  at"/gt
• ltReturn Type"null"/gt
• lt/Methodgt
• ltMethod namesetSearchPattern"gt
• ltParams Type"String" value"hello"/gt
• ltReturn Type"null"/gt
• lt/Methodgt
• ltMethod name"getResult"gt
• ltReturn Type"boolean"/gt
• lt/Methodgt
• lt/Oraclegt

16
Phase-VILatePenalty

• ltPhase name"Latepenalty" run"true"gt
• ltDueDate date"05/16/2005" time"100"/gtlt!--
  stick to this format,time 24 hr time frame --gt
• ltScoringgt
• ltUnit hours"6" penalty"50"/gt
• ltUnit hours"12" penalty"3"/gt
• ltUnit hours"6" penalty"15"/gt
• ltUnit hours"6" penalty"1"/gt
• ltLinear hours"6" penalty"10"/gt
• lt/Scoringgt
• ltExecute function"GetLatepenalty" maxscore "5"
  maxpenalty"20"gt
• ltParams value"CurrentStudentDir"/gt
• ltReportsMessage value"Latepenalty"/gt
• lt/Executegt
• lt/Phasegt

17
Phase -VIISend Results

• ltPhase name"MailResults" run"true"gt
• ltSubject value"Assignment1-Grades"/gt
• ltReportAddendum value Please contact grader
  at grader_at_abc.edu for any issues with grades."/gt
• ltMailServer value"10.113.40.16"/gt
• ltFromAddress valueprof_at_abe.edu"/gt
• ltToAddress valuestudent"/gt
• lt/Phasegt
• lt?xml version"1.0"?gt
• ltStudentListgt
• ltcpasupul EmailAddress"cpasupul_at_cs.odu.edu"/gt
• ltlkasi EmailAddress"lkasi_at_cs.odu.edu"/gt
• ltsmallep EmailAddress"xyz_at_yahoo.com"gt
• lt/StudentListgt

18
Phase-VIIIClean Up

• ltPhase name"CleanUp" run"true"gt
• ltBaseDir value"WorkDir"/gt
• ltExecute function"Clean" MaxScore"0"
  MaxPenalty"0"gt
• ltParams value"WorkDir/gt
• lt/Executegt
• lt/Phasegt

19
Environment Setup

• The AutoGrader requires J2SE 1.4.5 or later, but
  nor 1.5
• The AutoGrader requires mail.jar and
  activation.jar to be in the class path. These are
  required for sending mails to the students.
• The System should have the compilers for
  compiling the students grades
• Any Text Editor for editing the XML configuration
  files, for driving the AutoGrader.

20
Demonstration

• Scoring The Assignment Grades?
• How does the Scoring Pattern Works in
  LatePenalty?
• Debug runs?
• What if AutoGrader Returns Unable to parse XML
  Error?
• What if AutoGrader doesnt do what you expected?
• How Looping Works in Execute?
• Integrating Oracles?
• Configuring Mailing Report?

21
Future Developments

• UI for configuring the AutoGrader and designing
  the Grading Files.
• A System to keep track of mistakes in students
  subsequent submissions, so that hints can be
  issued and graded accordingly.
• More Oracles to give flexibility for grading.
• Interactive Diff functionality.
• Configurable reporting system.

22
Questions????

• All you wanted to ask.