Pros and Cons of online testing using Moodle

1
Pros and Cons of on-line testing using Moodle
TEL_at_York2006 April 25, 2006

• A case study of the elementary Japanese course

by Norio Ota, Noriko Yabuki-Soh, Gergana Ivanova
(DLLL), Mike Street (IT Consultant, ATS)
2
Preamble

• TEL initiatives by the Japanese Section of DLLL
• Server-based web course development
• Developing interactive instructional materials
  for self-study
• Developed a distance education course for the
  elementary level Japanese language course (tested
  at Glendon for 4 yrs) using videoconferencing
• Introducing Media Site Live for video-streamed
  lectures
• Developing web-based on-line tests for the
  elementary Japanese course assisted by ATS
  (2005-06)
• Developing a distance education course for the
  advanced Japanese course for St. Marys
  University in Halifax (to be implemented in
  September 2006)

3
Web-based Testing (WBT)

• Computer-based Testing (CBT) vs. WBT
• On-line course delivery software products
• WebCT
• Hot Potatoes
• Sakai
• FLE3 (Future Learning Environment 3)
• Moodle (Modular Object-Oriented Dynamic Learning
  Environment)
• Moodle
• Free
• Open source customizable
• Non-proprietary
• User-friendly
• Ease of installation
• Flexible
• Language support
• Modular Comprehensive
• Why on-line testing?
• Reducing marking time
• Developing distance education courses

4
Challenges for on-line testing for languages

• Concerns by language teaching professionals
• limited types of questions
• lack of analytical tools (natural language
  parsing)
• lack of qualitative evaluation
• lack of evaluation for communicative competence
• security issues

5
Implementation and Objectives

• On-line testing is NOT comprehensive
• On-line testing is to access each learners
  knowledge and recognition of
• Vocabulary
• Expressions
• Conjugations
• Sentence structures
• Basic kana characters and basic sino-Japanese
  characters (kanji)
• Simple context (communicative understanding)
• Sociolinguistic and pragmatic aspects

6
Developing tests

• Transforming paper-based tests into web-based
  versions
• Modifying types of questions
• Developing new types of questions
• Reviewing students answers and modifying
  possible answers
• Readjusting answers to introduce partial marking
  system
• Developing a questionnaire for students feedback
  

7
Overview of Moodle

• Course Management System (CMS aka LMS)
• Similar to WebCT, or Blackboard
• Our Moodle runs on Linux OS using PHP and a MySQL
  database backend.
• Open-source, modular and flexible
• Large, growing community
• Community-driven support

8
Overview of the Japanese Tests

• Tests run in the Multimedia Language Lab (MLC)
• 50 mins per test
• 2 to 5 tests run a day, over 2 days, 5 mins
  between tests
• Students self-register for accounts using Moodle
• Ran an Orientation Session at beginning to create
  accounts and try out sample test
• Tests were originally developed in MS Word.
• Imported into Moodle via web interface

9
Question Types

• Aside from the normal types of questions, the
  tests also included
• Fill-in-the-blanks (multiple blanks within a
  paragraph, using Cloze or Embedded Answers type
  questions)
• Wildcard characters in the answers (for partial
  marks)
• Custom question type
• 50 Japanese characters in a large image (png),
  each given a unique number
• Students answered the question by typing the
  number of the character into the box, in the
  correct order. (able to use wildcards for partial
  marks here too).

10
Security Measures

• IP-restricted (MLC only)
• Each test has unique password
• Time-based release of each test
• During the allowed test time, students can review
  answers
• Allowed a maximum of 2 attempts
• Moodle secure window test

11
Technical Challenges

• Accurate representation of Japanese characters
• Clarity of images
• Import not functioning in Moodle, so questions
  were cut/pasted
• Time-consuming to enter questions initially
• Cloze type question still sensitive to illegal
  characters
• Various minor software glitches (eg. Half marks
  getting rounded down)

12
For the Future

• Moodle has proven to be a successful way of
  deploying language tests online
• All of the major/minor software problems are
  going to be fixed over the summer 2006
• Eventually will be bringing Language Placement
  Tests on board (French and Italian)

13
What is tested?

• Evaluation criteria for JP1000
• Attendance Participation (Sept.-April)
  10
• Oral Presentation (4 times a year)
  25
• Quizzes (dictation, every week) Homework
  (practice of Japanese characters)
  15
• Tests (4 times a year, on-line testing)
  50
• -grammar structure vocabulary idiomatic
  expressions
• -reading of Japanese scripts

14
Types of questions

• Pen Paper vs. On-line
• Reading words
• What would you say when?
• Translation (E?J)
• -Complete a dialogue
• -Short answers vs. jumbled words
• Kanji (sino-Japanese characters)
• Translation (J?E)
• -Short answers vs. multiple-choice

15
New attempts

• Short answers
• Multiple-choice questions
• Questions using a table
• Questions using pictures
• Read a passage and answer questions

16
Cons for Students

• Less lenient marking compared to paper-based
  tests
• Stressful for those uncomfortable with computers
  and bad eye-hand coordination displaying time
  left in the corner
• Save without Submitting function tedious and
  demanding
• Difficult to return to an unanswered or
  problematic question, slow in viewing answers
• Easier to cheat
• Blurry screens

17
Pros for Students

• Quiet atmosphere, easier to focus
• Easier to answer questions (multiple-choice,
  kanji recognition, re-attempts)
• Easier to correct answers
• Immediate feedback
• No concern about bad handwriting hands do not
  get tired from writing no need to use pencils
• Less stress for those familiar with technology,
  more fun
• Gain more knowledge about computer technology -
  important for academic and professional life

18
On-line vs. PaperTest Average ()
Test 1 Test 2 Test 3 Test4 Paper 78.44 55.16
64.90 66.98 (2004-05) On-line 64.57 59.09 59.
52 62.50 (2005-06)
19
Students Self-reflection (1)

• Q How satisfied are you with the time you have
  allotted to prepare for the tests?
• A dissatisfied 6.67
• somewhat dissatisfied 46.67
• satisfied 34.67
• very satisfied 12

20
Students Self-reflection (2)

• Q How satisfied are you with the efforts you
  have allotted to prepare for the tests?
• A dissatisfied 8.79
• somewhat dissatisfied 44.59
• satisfied 38.51
• very satisfied 8.11

21
Pros for Instructors (1)

• Less time spent on marking, less tired, improved
  quality of teaching
• A test can be automatically re-graded for the
  entire class if a mistake is discovered in a
  question or more possible answers are added to
  the system
• Easier to deliver the correct answers to students
• Typed text easier to read than unintelligible
  handwriting

22
Pros for Instructors (2)

• Easier coordination between instructors regarding
  
• reviewing students answers, modifying possible
  answers, marking
• Faster processing of the test results
• Objective assessment (not a comprehensive
  approach)
• Increased marking consistency

23
Cons for Instructors

• Technical problems unsolvable by instructors
• Too sensitive to spaces, hyphens, unpredictable
  bugs in the program
• Location and scheduling restrictions
• Limitations regarding the types of questions
• Academic honesty issues

24
Cheating

• ? Weaker classroom control
• Assigned seats, gaps in the seating
• ? Inability to cut off the Internet
• ? Monitored activities
• ? Multiple attempts, higher test scores
• ? Limited attempts (two attempts only)
• ? Easy to duplicate the test and export it to
  Word
• ? Question pools, randomized questions, different
  order of answers run Moodle from a CD

25
Outcome

• Learned more about limitations, bugs and positive
  features re Moodle
• Why were test scores lower this year?
• Academic honesty issues
• Learning process for faculty
• Facultys willingness to learn and no quick
  resistance to Moodle
• Cooperation between IT consultant and faculty
  with good working relationship and initiatives
• Tech problems are hard to deal with by faculty
  alone requires a tech support person at the test
  site.
• Limitations re the test site

26
References

• Abhijeet Chavan (2004) Open-Source Learning
  Management with Moodle
• http//www.linuxjournal.com/article/7478
• Moodle (2006) Moodle for Language Teaching
• http//moodle.org/course/view.php?id31
• Aditya Nag (2005) Moodle An open source learning
  management system
• http//business.newsforge.com/article.pl?
  sid05/05/09/2117200
• Den Pain and Judy Le Heron (2003) WebCT and
  Online Assessment  The best thing since SOAP?
• http//www.ifets.info/journals/6_2/7.html
  
• Röver, C. (2000)Web-Based Language Testing
  Opportunities and Challenges http//www2.hawaii.ed
  u/roever/wbt.htm
• _________(2001) WEB-BASED LANGUAGE TESTING
• http//llt.msu.edu/vol5num2/roever/defaul
  t.html
• Sabine Siekmann (2006) CALICO Software Report
• Which Web Course Management System is
  Right for Me?A Comparison of WebCT 3.1 and
  Blackboard 5.0
• http//calico.org/CALICO_Review/review/we
  bct-bb00.htm
• University of Ontario (2006) WebCT
• http//www.uoit.ca/EN/main/11258/12122/17
  767/learning_webct.html