Christian G

1
Christian Gütl, IICM, TU-GRAZInstructions for
Experiment

• Natural Language Processing

2
The Experiment Procedure

• Topic Natural Language Processing (NLP)
• Objective of the Experiment
• Gain insight about extracting and evaluating
  concepts and generating questions
• Informed consent
• Please fill in the informed consent

3
The Experiment Procedure

• Step 1 Learning the text
• Step 2 Questionnaire 1
• Step 3 Learning Activity I (Questionnaire 2)
• Step 4 Test (Questionnaire 3)
• Step 5 Learning Activity II (Questionnaire 4)
• Step 6 Questionnaire 5a
• Please allow yourself breaks between tasks!

4
Step 1
Please learn the text (max. 30 minutes) ? See
email (Subject NLP text)

5
Step 2

• Please fill in Questionnaire 1 (max. 10 min)
• See email (Subject phd Questionnaire 1)
• Please give all answers in English!
• Please do not use the text while answering the
  questions!
• Click on the Submit-Button once you have
  finished the task!

6
Step 3a Extracting concepts

• Please extract the main concepts of the
  NLP-text, using Questionnaire 2 (max. 10 min)
• (Example Speed of light theory of relativity,
  Einstein).
• Please separate your concepts with semicolons ()
  or commas (,)
• ? See email (Subject phd Questionnaire 2)
• After you have finished this step, please
  continue working on Questionnaire 2 (Step 3b) as
  described on the next slide

7
Step 3b Generating questions

• Please generate eight questions from the
  NLP-text, using Questionnaire 2.
• Construct two Open Ended questions using the text
• (Example What do you know about theory of
  light?)
•  Construct two Fill In The Blank questions using
  the text
• (Example The founder of this university was
  ________.)
•  Construct two Single Choice questions using the
  text. Their answer should be no! (Example
  Albert Einstein was born in Norway)
•  Construct two Multiple Choice questions with
  three distractors for each chapter. Only one
  of the distractors should be correct! (Example
  The sun is _____. blue, green, yellow, red)
• Note that you have to generate one question for
  each category first and then another one
  afterwards.

8
Step 4

• Please fill in Questionnaire 3 (max 15 Minutes)
  and allow yourself a break afterwards.
• See email (Subject phd Questionnaire 3).
• Please do not use the text while answering the
  questions!

9
Step 5a Evaluation of concepts

• Evaluation Task I (max. 10 min.)
• Please evaluate the concepts you will find in
  Questionnaire 4. Do you think the concept is
  relevant for the text?
• 1 not relevant 5 very relevant
• See email (Subject phd Questionnaire 4).
• After you have finished this task, please
  continue working on Questionnaire 4 (Step 5b) as
  described on the next slide

10
Step 5b Evaluation of questions

• Evaluation Task II (max. 35 min)
• Please evaluate the questions provided in
  Questionnaire 4, using the criteria as described
  below. For each criterion described below, please
  assign a score between 1 ( very bad) and 5
  (very good) to each question.
• Pertinence Relevancy of the formulated
  questions. Has the question a precise and logical
  connection to the topic?
• Terminology Appropriateness of the words chosen
  to formulate the question.
• Level Extent in approaching the topic. Is
  the question trivial or expresses it a
  significant meaning? (1 trivial, easy 5
  difficult, significant meaning)
• Answer (only if provided!) Is the provided
  answer relevant?
• Distractors (only for multiple choice!) Quality
  of the computed distractors of the multiple
  choice questions

11
Step 5c General Survey

• Please fill in the last section of Questionnaire
  4 (max. 15 min)

12
Step 6 Evaluating questions

• Evaluation Task III
• Please evaluate the questions provided in
  Questionnaire 5a (see email), using again the
  criteria as described below. For each criterion,
  please assign a score between 1 ( very bad) and
  5 (very good) to each question.
• Pertinence Relevancy of the formulated
  questions. Has the question a precise and logical
  connection to the topic?
• Terminology Appropriateness of the words chosen
  to formulate the question.
• Level Extent in approaching the topic. Is
  the question trivial or expresses it a
  significant meaning? (1 trivial, easy 5
  difficult, significant meaning)
• Answer (only if provided!) Is the provided
  answer relevant?
• Distractors (only for multiple choice!) Quality
  of the computed distractors of the multiple
  choice questions

13

• Thank you for participating in this Experiment!