Vorlesung KI

1
Raph Koster (about games) Fun from games
arises out of mastery. It arises out of
comprehension. It is the act of solving puzzles
that makes game fun. In other words, with games,
learning is the drug. Thats what games are,
in the end. Teachers. Fun is just another word
for learning. Koster, Raph Wright, Will A
Theory of Fun for Game Design ISBN 1932111972,
Paraglyph Press, 2004
Thesis The satisfaction of some kind of fun
factor is certainly a basis of humans
evaluations of any (learning) environment.
2
Towards an Evaluation of (e-) Learning
Systems Rainer Knauf Faculty of Computer
Science and Automation Chair of Artificial
Intelligence Technical University of
Ilmenau Ilmenau, Germany Klaus P. Jantke German
Research Center for Artificial Intelligence The
Competence Center for e-learning
(CCeL) Saarbrücken, Germany
3
1 Introductory Remarks
What is learning?

• Learning is a process of (re-)constructing
  knowledge
• by actively dealing with
• content represented as media objects
  (textbooks, slides, talks, movies, - course
  material)
• human actors teachers, tutors, co-learners
• in a certain location characterized by
• room conditions
• presentation equipments
• by a certain form of interaction between the
  (human and non-human) agents
• influenced by soft factors like
• the agents moods
• formal and social relations of agents to each
  other
• individuals outside the learning process
• driven by
• the learners pre-conditions and needs

Since at least (4) and (5) are very individual,
different learners may construct different
knowledge in the same environment settings (1),
(2) and (3).
4
What kind of answer do we expect when asking for
the quality of a learning system?

• At a first view, the key to the answer is
  considering the success of the learning process,
  but
• this does not reveal the particular weaknesses
  of the process and thus,
• no chance to derive improvement measures.

To point out particular weaknesses of any
learning process, we need an explicit
representation of both the (1) topical content
and (2) the didactic design of its presentation.
? Fortunately, the first requirement is met by
the nature of media objects. ? Unfortunately,
the latter requirement is not really met in
todays (not only e-) learning systems.
5
2 Evaluation Scenarios
What is the subject of evaluation ?

• learning process
• by rating the learning activity (by an evaluation
  form)
• as a black-box object
• by providing maybe criteria-associated
  ratings for the entire activity
• in a white-box manner
• by considering the particular learning scenes and
  their sequence
• learning result
• by rating the learners skills after passing
  through the process (by an examination)
• in a black-box manner
• by evaluating solutions to test tasks
  independently from the way of developing them
• as a white-box object
• by evaluating solving methods and paths

Who is the source of the evaluation assessment ?

• teachers, topical experts,
• who are involved in the learning process
• who are external experts
• learners, students,

6
3 Current Customs
in three example learning environments

• example 1
• Learning fundamental knowledge in schools
• Learning processes
• are mostly not evaluated at all.
• teachers didactic skills are evaluated during
  their study, .. .
• but rarely during the 40 years of their
  practice as long as (1) nobody complains and (2)
  their pupils examination results sound
  acceptable.
• Learning results
• are evaluated all the time by tests,
  examinations,
• includes test task solutions and in some cases
  the manner to develop it.
• The evaluation of learning results is performed
• by their own teachers

Checklist Summary
?
(?)
?
7

• example 2
• Learning high-level knowledge in (German)
  universities
• Learning processes
• are evaluated
• once as a (very small) part of the habilitation
  process and
• more and more often regularly by students
  ratings
• Learning results
• are evaluated all the time by tests, homework,
  students talks, examinations,
• includes both test task solutions and mostly the
  manner to develop it.
• The evaluation of learning results is performed
• by their own teachers
• external reviewers for Thesis (Master, Diploma,
  Ph.D., )

Checklist Summary
?
?
?
?
(?)
8

• example 3
• Learning fundamental but survival essential
  knowledge in Driving Schools
• Learning processes
• are not evaluate.
• Learning results
• are evaluated by stringent examinations, which
• is limited to task solutions.
• The evaluation of learning results is performed
• external reviewers who
• developed the questionnaire and the rules to
  evaluate it
• sit in the back seats during the practical
  examination.

Checklist Summary
?
?
9
4 Generally,

• The learning process is rarely evaluated.
• The design of this process is significant for the
  learning success.
• Thus, this process needs to become a subject of
  evaluation and refinement.
• The evaluation is rarely performed by the
  learners themselves.
• Since learning is a very interactive process,
  both parties of interaction (learners and
  teachers) need to be included into the evaluation
  process.
• Ignoring the learners didactic experience wastes
  a lot of improvement opportunities.
• The evaluation is often limited to
• the evaluation of test task solutions (and not
  the way to develop them)
• by the own teachers (not external experts).
• The higher the the academic level of a skill to
  be learnt or the higher this skill is mission
  critical, the more the evaluation shifts
• from black-box to white box evaluation
• from internal to external reviewing
• of learning results (only, ?) .

10

• The didactics of a learning process needs to be
  represented explicitly to
• evaluate the process,
• point out its particular weaknesses and
• to improve and refine this learning process to
  finally reach better learning results.

• Therefore, we introduced an AI-typical approach
  to face these issues
• We Proposed a modeling concept for didactic
  knowledge Storyboards
• We suggested a (standard-) tool to develop and
  process such models Viso

11
5 Storyboarding - Our Concept

• Objectives and differences to concepts so far
• driven by the human learning process,
• not by software-technological concepts
• supporting the development of technology enhanced
  learning,
• not the design of e-learning systems
• organizing learning experience,
• not learning materials
• concentration on the learners activities,
• not on the use of e-learning systems
• More generally, technological progress
• has to support the satisfaction of natural
  human wishes (like learning, e.g.) by providing
  tools that help to perform appropriate activities
  and
• must not force humans to adapt their natural
  desires and activities to current (software-)
  technological standards or tools.
• Requirements to the Storyboard approach to
  support didactic design
• clarity
• simplicity
• visual appearance

12

• Core notion
• A storyboard is a graph with annotations to its
  nodes and edges
• Nodes scenes, episodes
• edges transitions between nodes
• Scenes are atomic, implemented in different ways
• Episodes are composite, described by sub-graphs
• Key annotations to nodes specify actors and
  locations
• Free text annotations to nodes and edges may
  represent didactic intensions

13
An example Annotations to a (atomic) scene
14
Peculiarities of the proposed concept

• Again, storyboards do not organize materials, but
  experience related to materials.
• The agents are not limited to computers (as the
  source of knowledge transfer) and humans (as the
  destination of knowledge transfer).
• The documents are not limited at all to any
  e-learning system's content, but allow for a
  large variety of document types (scripts, slides,
  books, ).
• The setting doesn't have to be a learner sitting
  in front of a computer, but also a group of
  learners, co-learners, teachers, software agents,
  in a lecture room, at a coffee place, at a
  green meadow,
• Both online and offline experiences are involved.
• Episodes may have alternative implementations
  (online vs. offline, e.g.)
• Didactic concepts result in developing patterns
  of graph structures and thus, become explicit,
  visible, and subject of quality assurance.

15
6 Storyboarding in Practice

• Basic principles
• Top-down design
• Keep an overview by
• starting with a top-level storyboard of about 6
  nodes and becomes subject of refinement
• developing small graphs only and
• nesting them appropriately
• Collect scenes and episodes first and discus
  their appropriate linking later

An Example A Data Mining Course

• Here, we chose a problem-oriented, explorative
  style of transferring knowledge.
• principle 1 start by a top-level graph
• principles 2 and 3 ended up with
• an 11 nodes initial top-level graph with one
  episode unlinked at all
• which turned to a 13 nodes graph with all nodes
  linked and introducing
• different design variants
• different colors of edges to reflect different
  learners preferences

16
Top Level Storyboard of the Example

• Episode Introductory Case Study can be
  implemented in different ways
• Individual studies with a Web-based system
  (JANTKE)
• Classical lecture (KNAUF)
• Discussion group offline (an option for others)
• What about the lonely node Stories of Success ?
• In the early design stages its integration has
  been left open until we found a didactically
  motivated integration

17
Didactically Driven Paths - Embedding Stories
of Success

• Generally, all transitions are accessible by all
  learners.
• Guidance offered to particular learners are
  didactically driven
• Light blue edges transitions recommended for
  illustration oriented learners

Variant 1 Using this episode for repeated
motivation
Variant 2 Using this episode for final
illustration
18
Alternative Node Implementations - Refining
the episode Introductory Case Study Revisited
Depending on the available resources, the actors
preferences, the implementation of a node may
differ.
This node has three implementation variants

• An episode State of the Affair Summary
• A scene Spam Mail Classification Revisited
  implemented by lecture slides (KNAUF)
• A training-oriented Open Space scene (JANTKE)
• extended by the human actors (Learner,
  Co-learner, Trainer)
• commented by an informal description of (required
  prerequisites and didactic intentions)

The latter variant nicely illustrates that the
approach goes far beyond the issue of designing
e-learning environments!
19
Deriving Design Patterns

• As a vision, the comprehensive use of this
  approach will lead to typical design patterns of
  successful storyboards.
• Thus, it is the first step towards exploring and
  learning (new) general didactic knowledge as
  graph-templates by analyzing particular
  successful storyboards.

A problem-driven, explorative didactic template
seems to be
20
7 The Evaluation Approach

• Attachment of validities to
• Scenes ( documents that provide the source for
  topical knowledge)
• Episodes ( compound objects containing both
  topical and didactic knowledge)
• Edges ( the didactic knowledge about shifting
  scenarios)
• provided by all involved human sources (learners,
  teachers, external experts, )

Whats a learning process ?

• A learning process is defined by a sequence of
  visited scenes.
• Their content characterizes topical knowledge.
• Their sequence defines didactic knowledge.

and in terms of our storyboard concept ?
A learning process is the traversed path in each
of the nested graphs.
21

• For computability, we introduce validity degrees
  within 0 , 1
• 0 is the worst degree of validity (completely
  invalid, did never contribute to a successful
  learning process)
• 1 is the best degree of validity (completely
  valid, did always contribute to a successful
  learning process)
• For simplification, we ask every person to
  provide just the sharp ratings
• 0 (bad)
• 1 (good)

Initially, the only people who rate the
storyboard are the authors, who intend to make
their job well, usually. ? The initial
validity of each storyboard element (nodes and
edges) is 1

• Whenever a learning process has been rated by
  someone, the validity degrees of all elements of
  the traversed paths are subject of change
• Downwards, if the process has been rated by 0
• Upwards, if the process has been rated by 1
• by using the principle of exponential smoothing.

22
Fixing the storyboards validities by a learning
process rating
nh of counter-examples to halve the validity
of an item in the storyboard w the related
influence of a new rating on a storyboard items
validity vi validity of a storyboards item after
the i-th rating
relation between nh and w
fixing a validity vi with a new rating r
23
Enjoyable features of this approach

• By fixing w, the approach can be adapted to
  various requirements to the evaluation
• A conservative strategy can be implemented by a
  very small value.
• A quick adaptation can be implemented by choosing
  a large value.
• More recent ratings are promoted and historic
  ones are outdated step by step. Thus, it reflects
  long term trends.
• The validity degree is related to the number of
  available ratings. Different numbers of visits
  (of a node or an edge) dont distort validity
  degrees

24
How to interpret the validity degrees?

• A good validity degree of a scene ( atomic node)
  reflects appropriate topical content as well as
  no need to refine didactics (no need to develop a
  sub-graph)
• A good validity degree of an episode reflects
  both, a good topical and didactic quality (well
  structured sub-graphs)
• A good validity degree of an edge reflects a
  proper didactic decision to shift from its source
  node to its destination node

How to use validity degrees for the refinement of
the learning system ?

• Scenes with bad validity degrees point out
• topical weaknesses of the content and/or
• the need to refine the didactics by developing a
  sub-graph
• Suggestion refine the didactics after
  confirming/fixing the topical quality
• Episodes with the bad validity degrees need to be
  analyzed more detailed by jumping into the
  related sub-graph and considering the validity
  degrees of its elements.
• Edges with bad validity degrees point out
  improper decisions to shift from their source
  node to their destination node.

25
6 Summary and Outlook

• Currently, learning processes are rarely
  evaluated.
• The evaluation is mostly limited to learning
  results.
• External experts as well as the learners
  themselves are rarely included.
• Evaluating learning processes and fixing the
  revealed weaknesses requires an explicit
  representation of the didactic design of learning
  systems.
• Storyboarding is a way of representing,
  processing, evaluating and refining didactic
  knowledge.
• The proposed storyboard concept leads far beyond
  the limits of software engineering.
• All didactic forms including collaborative and
  competitive work and classical learning forms may
  be included.
• As a vision, the comprehensive use of this
  approach will lead to typical design patterns of
  successful storyboards.
• A learning process is defined by a path of
  visited nodes and performed node shifts in the
  (nested) storyboard graph(s).
• Exponential smoothing is an approach to estimate
  validity degrees attached to each element of a
  storyboard based on humans ratings for a
  learning process.
• The evaluation concept needs to be developed
  further with respect to
• a concept to distinguish different sources of
  ratings teachers, external experts, learners, ..
• a refined scale of ratings
• learning objectives, which enforce different
  paths for different learning purposes with the
  same storyboard.