MHCI PSLC Data Shop Project

1
MHCI PSLC Data Shop Project

• Spring Progress Report

2
The Team
Jason Hum Design Lead
Sandi Lowe Project Manager
Sam Zaiss Documents Manager
Meghan Myers Client Liaison
Jeff Wong Tech Lead
3
Outline

1. What is the PSLC?
2. What is the Data Shop?
3. Analyzing Data
4. Exporting Data
5. Project Themes
6. Preliminary Design
7. Project Plan
8. Final Takeaways

4
What is the PSLC?
PSLC
LearnLab
Data Shop
Collect
Process
Access
Pre- Defined
Free- Form
Export
5
Background on PSLC

• Further current education research
• Enable new education research
• Support collaboration
• Support LearnLab courses

6
Background on LearnLab Courses

• 7 course areas under PSLC
• 3 Language (Chinese, French, ESL)
• 2 Math (Algebra, Geometry)
• 2 Science (Physics, Chemistry)
• Various stages of development

7
What is the Data Shop?
PSLC
LearnLab
Data Shop
Collect
Process
Access
Pre- Defined
Free-Form
Export
8
Collecting the Data

• Allows researchers to set up an in-vivo LearnLab
  experiment
• Facilitates logging of data in these experiments

9
Processing the Data

• Strip away student information
• Perform computations on data to streamline
  analysis
• Create supporting data tables

10
Accessing the Data

• Built as a web application
• Selected data analysis techniques
• Data filtering and exportation

11
Analyzing Data Pre-Defined Reports
PSLC
LearnLab
Data Shop
Collect
Process
Access
Pre- Defined
Free-Form
Export
12
Learning Curves

• Displays students performance over time on a
  particular knowledge component
• Log Files must include
• Knowledge Components
• Student performance on individual opportunities
• Match Knowledge Components to Problems /
  Sub-Goals
• Supporting Data
• Requirements Solicitation (Koedinger, Aleven,
  Ritter)
• General Research Contextual Inquiries (U2)
• LearnLab Research Contextual Inquiries (U13)

Courtesy of Data Shop
13
Help Effectiveness Report

• The probability of the Section getting task or
  sub-goal x incorrect after a particular help
  message
• Log Files must include
• Help / Hints
• Whether a student received help, and if the
  answer that followed was correct or incorrect
• Measure of students knowledge before help
  message
• Supporting Data
• Requirements Solicitation (VanLehn, Ritter)
• General Research Contextual Inquiries (U1, U10)
• LearnLab Research Contextual Inquiries (U11, U12)

14
Error Report

• Determine common mistakes that students make on
  tasks
• Log Files must include
• Erroneous Input Tutor Response to Error
• Error Classifications
• Supporting Data
• Requirements Solicitation (VanLehn, Koedinger,
  Ritter)
• General Research Contextual Inquiries (U3, U8,
  U10)
• LearnLab Research Contextual Inquiries (U11, U12)

15
Problem Profile

• Context and student performance on a particular
  problem
• Log Files must include
• Problem Text
• Record of students answers on the problem and
  whether the answers were correct or incorrect
• Supporting Data
• General Research Contextual Inquiries (U1, U3,
  U7, U8, U9, U10)
• LearnLab Research Contextual Inquiries (U11, U12)
• Course Committee Survey (Chem)
• Think Aloud Pilot (U2)

16
Analyzing Data Free-Form Analysis
PSLC
LearnLab
Data Shop
Collect
Process
Access
Pre- Defined
Free-Form
Export
17
Timeline Visualization

• Visualizes the pattern of each students actions
• Log Files must include
• Time measurements (ms)
• Supporting Data
• Users want to drill down and tell stories
• General Research Contextual Inquiries (U1, U5,
  U8, U9, U10)

Courtesy of Andrea Knight
18
Session Browser
Courtesy of Jack Mostow

• Visualizes the pattern and context of each
  students actions
• Log Files must include
• Time measurements (ms)
• Event Hierarchy
• Supporting Data
• Users want to drill down and tell stories
• Requirements Solicitation (VanLehn)
• General Research Contextual Inquiries (U1, U3,
  U4, U5, U7, U8, U9, U10)

19
Event Filtering (aka Rollup Table)

• Allows researcher to specify their variables and
  calculations over those variables
• Supporting Data
• Requirements Solicitation (Koedinger, VanLehn)
• General Research Contextual Inquiries (U5, U8)
• U5 By the time I get the data I need out of the
  system, Ive forgotten the question I wanted to
  ask

20
Exporting Data
PSLC
LearnLab
Data Shop
Collect
Process
Access
Pre- Defined
Free-Form
Export
21
Project Themes
PSLC
LearnLab
Data Shop
Collect
Process
Access
Pre- Defined
Free-Form
Export
22
Project Themes

• 1. Support All LearnLab Tutors

23
Project Themes

• 2. Facilitate Inter-Report Navigation

24
Project Themes

• 3. Let Researchers See Only the Data They Want to
  See, When They Want to See It.
• Reports show only whats relevant
• Find out how researchers want to narrow down
  their data in reports
• Minimize interface elements for each report

25
Preliminary Design
26
Design for Theme 2

• Large buttons to move between reports
• Key Considerations
• Data transfer from report to report
• Saving reports with interesting data along the
  way
• Issues of scale

27
Design for Theme 3

• Specialized filtering for each report
• Key Considerations
• Condensing report specification modification
  into one set of controls
• Distinguishing between singular multiple
  selection
• Enabling fast selection
• Design Support
• List boxes for long lists of variables
  (StatTransfer)
• Clear representation of curriculum hierarchy
  (Froogle)
• Design Challenges
• Minimizing space while maximizing speed
• Providing sufficient context to the user

28
User Testing Findings

• 2 users doing an analysis with learning curves
• 1st year HCI researcher PhD, familiarity with
  tutor studies
• 4th year education researcher PhD, no tutor
  studies
• Needs discovered
• Compare curves against pre- and post-tests
• Show p-value and regression equation
• Describe sample
• Click on a Learning Curve spike to understand
  context
• Show text of the problem

29
Project Plan
1 wk
Pre-Workshop 6 wks
Post-Workshop 5 wks

• Pre-Summer Workshop
• Add breadth to current structure
• Error Reports
• Help Effectiveness
• Problem Profiles
• Develop Filtering Designs (Mid-Fi)
• Prototype Session Browser Timeline
  Visualization (Low-Fi)
• Heuristic Evaluation
• User Testing

30
Project Plan
1 wk
Pre-Workshop 6 wks
Post-Workshop 5 wks

• Summer Workshop (Week of June 27th)
• High-Intensity User Testing, Iteration
• Post-Summer Workshop
• Database Connections for Pre-Defined Reports
• Session Browser Timeline Visualization (Mid-Fi)
• User Testing
• Revisions

31
Final Takeaways

• 1. Rules for Data Sharing
• Sharing outside research group
• Competitive Analysis revealed need for sharing
  regulations
• U5 Im happy to share my data if the
  researchers follow PSLCs guidelines
• For now, no open sharing
• Sharing inside research group
• Seen as a need in General Research CIs (U1 - U8)
• Designed tested by end of summer

32
Final Takeaways

• 2. Context Matters
• Understanding data is hard without context
• Issues discovered to date
• Complex naming gt complex design (CA)
• Tutor writers use intimate knowledge of the tutor
  frequently (U5, U7, U9, U11, U12)
• Researchers unfamiliar to the tutor need that
  intimate knowledge (TA Pilot Test)
• TA Pilot Test U2 So youd have a master book
  that would tell you what all the problems are?
  Because Id never remember.

33
Final Takeaways

• 3. Courses in Development Should Log Necessary
  Data
• Survey course committees as needed
• Proposed reports will function with current PSLC
  tutors
• Cant predict what future tutors may and may not
  log

34
Fin
Feedback?MHCI PSLC Project Teamhcii-mhciprojec
t-pslc_at_lists.andrew.cmu.edu
35
Behavior Graph

• Illustrates the strategies students are using to
  solve problems
• Log Files must include
• Sub-goals
• Supporting Data
• Requirements Solicitation (Koedinger)
• General Research Contextual Inquiries (U8, U10)
• Course Committee Survey (Chem)
• Think Aloud Pilot (U2)

Courtesy of Data Shop
36
Physics Survey
Tutor Element ANDES WhyAtlas
Problem Problem Task
Sub goal Entries Expectations / Responses
Knowledge Comp Principles Principles
Right Answers Right Answers Correct Features
Wrong Answers Recognized Errors Recognized Errors
Unanticipated Errors Unrec. Errors Unrec. Errors
Wrong Ans Feedback Red Incorrect Error State
Hint Next Step Help / Whats Wrong Help N/A
Error Cluster Error Classes Misconceptions
37
Chinese/French Survey
Tutor Element Chinese/French
Problem Activities
Sub goal N/A
Knowledge Comp Skills
Right Answers Correct Answers
Wrong Answers Incorrect Answers
Unanticipated Errors Unpredicted Errors
Wrong Ans Feedback Error Feedback
Hint Funtional / Contextual Help
Error Cluster (e.g., Orthography, Meaning, Form)