A Taxonomy of Questions for Question Generation

1
A Taxonomy of Questions for Question Generation

• Rodney D. Nielsen1,2, Jason Buckingham, Gary
  Knoll, Ben Marsh and Leysia Palen
• 1 Boulder Language Technologies, Boulder, CO
• 2 Center for Computational Language and Education
  Research, CU, Boulder

2
The Goal of the Taxonomy

• HCI design for a Socratic tutoring system
• Key output was mapping from a classification of
  learner interactions to an appropriate tutoring
  response
• Included elaborate taxonomies for learner tutor
• Process
• Start with current research on tutoring
• Analyze human tutoring transcripts
• Six subject areas spanning elementary school
  reading comprehension through college-level
  conceptual physics

3
Relation to Question Generation

• Roadmap for question generation
• View QG as a 3-step process
• Concept Selection
• Question Type Determination
• Question Construction
• Pedagogical theory
• Automatic question classification

4
Question Branch of Taxonomy

• Started with question type list in Graesser
  Person (1994), adapted from Lehnert (1978)
• Added Blooms (1956) Taxonomy of Educational
  Objectives and other schemes
• Iterative process (4 annotators)
• Analyzed human tutoring transcripts
• Annotated dialog acts according to taxonomy
• Separately, revised taxonomy per transcripts
• Met to review and integrate separate taxonomies
• Redundancy, usefulness, completeness
• Would change facilitate more effective tutoring /
  learning

5
Comparison with Prior Work

• Addition of secondary dimensions
• Hierarchical structure of primary taxonomy
• Added a few question classes
• Moved some classes to secondary dimensions
• Verification and Disjunctive questions

6
Question Taxonomy

• PRIMARY CLASSIFICATION
• Description Questions
• Concept Completion Who, what, when, where?
• Definition What does X mean?
• Feature Specification What features does X have?
• Composition What is the composition of X?
• Example What is an example of X?
• Method Questions
• Calculation Compute or calculate X.
• Procedural How do you perform X?
• Explanation Questions
• Causal Antecedent What caused X?
• Causal Consequence What will X cause?
• Enablement What enables the achievement of X?
• Rationale Questions
• Goal Orientation What is the goal of X?
• Justification Why is X the case?
• Comparison Questions
• Concept Comparison Compare X to Y?

ORTHOGONAL CLASSIFICATIONS Collins Question Type
(Collins, 1985) Form hypothesis, Test
hypothesis, Make prediction, Trace consequences,
Entrapment, or None. Blooms Taxonomy of
Educational Objectives, top level (Bloom, 1956)
Knowledge, Comprehension, Application, Analysis,
Synthesis, or Evaluation. Content Level Amount
of answer content in the question Example Usage
(adapted from Collins, 1985) Positive Paradigm
Case, Negative Paradigm Case, Negative Exemplar
for a Necessary Factor (near miss), Positive
Exemplar for an Unnecessary Factor (near hit),
Generalization Exemplar for a Factor (maximal
pair), Differentiation Exemplar for a Factor
(minimal pair), Exemplar to Show the Variability
of a Factor, Exemplar to Show the Variability of
the Dependent Variable, Counterexample for
Insufficient Factors, Counterexample for
Unnecessary Factors, Analogy, Continuation of
Example, Reuse of Example, None. Response Form
This indicates the type and length of the
expected response Boolean, Multiple Choice,
Word, Phrase, Sentence or Paragraph. Question
Relation Relationship of the current question to
preceding questions Connection Question
7
Question Taxonomy

• PRIMARY CLASSIFICATION
• Description Questions
• Method Questions
• Explanation Questions
• Comparison Questions
• Preference Questions

• ORTHOGONAL CLASSIFICATIONS
• Collins Question Type
• Blooms Taxonomy of Educational Objectives
• Content Level
• Example Usage
• Response Form
• Question Relation
• Connection Question

8
Examples

• What happens to the pitch as a guitar string gets
  longer? It becomes a) higher, b) lower, c)
  louder, or d) softer
• Causal Consequence, Comprehension, Multiple
  Choice,
• Describe one way to lower the pitch produced by
  plucking a wire?
• Causal Antecedent, Application, Sentence,

9
Roadmap

• Most current question generation
• Blooms Knowledge level
• Subclasses of Description questions
• Progress across Blooms Taxonomy
• Some Method Comparison questions
• Explanation questions question series
• Finally Collins categories examples
• All types are within reach w restrictions

10
Summary of Taxonomy Uses

• Roadmap
• QG as a 3-step process
• Concept Selection
• Question Type Determination - output
• Question Construction input
• Question Classification
• Learning appropriate question types
• Evaluating system output

11
Thanks!

• Thanks to Steve Bethard, the CU Computational
  Semantics Group and the anonymous reviewers for
  helpful feedback.
• This work was partially funded by Award Numbers
• NSF 0551723,
• IES R305B070434, and
• NSF DRL-0733323.