: Flexible Open Learner Modeling

1
Flexible Open Learner Modeling

• Sergey Sosnovsky,PAWS_at_SIS_at_PITT

2
References

• Susan Bull, , UK.
• Mabbott, A. Bull, S. (2004). Alternative Views
  on Knowledge Presentation of Open Learner
  Models, ITS2004, 689-698.
• Mabbott, A. Bull, S. (2006). Student
  Preferences for Editing, Persuading and
  Negotiating the Open Learner Model, ITS2006,
  481-490.
• Kerly, A. Bull, S. (2006). The Potential for
  Chatbots in Negotiated Learner Modelling,
  ITS2006, 443-452.

3
Outline

• Open User Model
• Flexi-OLM
• Viewing LM
• Editing LM
• Persuading LM
• Negotiating LM
• Multiple LM Presentations
• Evaluation
• Demo

4
Open Learner Modeling

• WhatVisualization of the learner model,
  providing a learner with a mechanism to explore
  it, sometimes, negotiate it.
• WhyWhen a learner is engaged in the analysis of
  the learner model he is reflecting upon his
  domain knowledge and experience re-calling and
  re-considering ideas of which he is aware.

5
Flexi-OLM

• Models student understanding of basic C
  programming
• Uses color coding for telling students about the
  concept knowledge levels
• While limited understanding
• Pale yellow somewhat limited
• Yellow/green moderate
• Bright green excellent
• Red misconception
• Grey insufficient data
• Large topics include smaller concepts. Clicking
  on a topic in the model brings more detailed
  concept-wise information about this topic
  understanding.
• Knowledge are assessed with the help of short
  questions
• After playing with the system
• Questions correspond to only one concept
• No knowledge Inference between concepts
• Very simple modeling formula (seems like average
  with linear thresholds for knowledge levels)

6
Editing LM

• Flexi-OLM allows direct editing of LM
• Possible scenarios for this will be
• new learner wishes to inform the system about
  topics she already understands
• the learner grasps a concept outside the system
  and wants LM to reflect this
• the learner correctly guesses a series of answers
  gt LM has a higher knowledge level than she
  believes she has.

7
Scrutinizing and Persuading LM

• Less direct method
• A learner registers a disagreement with the LM
  and propose a change
• Flexi-OLM explains its believes by presenting the
  evidence supporting these beliefs
• If the learner still wishes to proceed, she has
  the opportunity to persuade the LM by answering
  a series of test questions.
• Possible Scenarios
• A learner believes her knowledge may be different
  than the system asserts, but lacks the confidence
  to edit it unchallenged,
• A learner seeks the satisfaction of proving the
  system wrong

8
Negotiating LM

• Flexi-OLM supports conversation-based negotiation
  of LM
• A learner is chatting with the system (as he
  thinks)
• Flexi-OLM maintains separate believe models for
  LM and for a learner
• It is ensured that the same dialogue moves are
  available to both parties gt Each party
• has full control over their own beliefs,
• can challenge the others belief,
• can seek justification for the others belief (on
  the LM side justification is based on the past
  learners answers),
• may request justification before changing their
  own beliefs,
• may ultimately decide to leave their belief
  unchanged.
• If the difference between LMs and Students
  beliefs is
• 1 level The LM accepts the learners
  suggestion
• 2 levels A compromise is offered (of
  changing both beliefs by one level)
• 3 levels The systems seeks a
  justification (the learner will be asked to
  answer a question)
• possible hack gradual change of the LM belief
  by one level

9
Negotiating LM (cont.)The Wizard-of-Oz Paradigm

• Human experimenter takes the role of the chatbot
  Wizard
• The Wizard follows a protocol designed to
  ensure
• that responses to students remained consistent
  between users, and
• that the chatbot was believable to users.
• To enact the protocol, the Wizard was provided
  with some 350 pre-authored chatbot negotiation
  initiations and responses to user inputs.
• Typical conversation

10
LM Presentations in Flexi-OLM

• hierarchy, a logical grouping of related
  concepts
• lectures, where topics are organized the same as
  in the related lecture course
• concept map showing relationships between the
  topics
• prerequisites, showing possible sequences for
  studying topics
• index, an alphabetical list
• ranked, where topics are listed in order of
  proficiency
• textual summary.

11
Experiment 1 (2004)

• Two questions
• Is it beneficial for students to have a choice
  over presentation of open LM, or it causes
  information overflow?
• Is there any strong preference for a particular
  LM view among individuals and if so, can it be
  predicted on the basis of learning style?
• 23 undergraduate students
• Experiment flow
• Pre-test (control flow in C) to populated LM
• Browsing session, where students can choose among
  4 different presentations

12
Experiment 1 Results
13
Experiment 2 (2006)

• Main question
• What are the students preferences concerning
  editing, negotiating and persuading LM?
• 8 third-year undergraduate students
• Experiment flow
• Initial testing to populate LM
• 1-hour LM exploring session (edit persuade)
• 20 minutes of negotiating with LM

14
Experiment 2 Results
15
Experiment 3 (2006)

• The goal
• To explore the feasibility of using a chat-based
  interface in an OLM system
• 11 final year undergraduates
• Experiment flow
• Self-assessment of student knowledge of each
  concept, providing the initial users beliefs to
  the system.
• Interacting with the system to populate LM and
  provide it with its beliefs about student
  knowledge
• Then students were shown how to use the chatbot
  and asked to interact with it for at least 20
  minutes.
• Post-experiment questionnaire

16
Experiment 3(2006)
17
Demo

• http//olm.eee.bham.ac.uk/flexi-olm/login.php

18
Thank You for Your Questions