An Interactive System for Hiring

1
An Interactive System for Hiring Managing
Graduate Teaching Assistants

• Ryan Lim
• Venkata Praveen Guddeti
• Berthe Y. Choueiry
• Constraint Systems Laboratory
• University of Nebraska-Lincoln

2
Outline

• Task Motivation
• System Architecture Interfaces
• Scientific aspects
• Problem Modeling
• Problem Solving
• Comparing Characterizing Solvers
• Motivation revisited Conclusions

3
Task

• Hiring managing GTAs as instructors graders
• Given
• A set of courses
• A set of graduate teaching assistants
• A set of constraints that specify allowable
  assignments
• Find a consistent satisfactory assignment
• Consistent assignment breaks no (hard)
  constraints
• Satisfactory assignment maximizes
• number of courses covered
• happiness of the GTAs
• Often, number of hired GTAs is insufficient

4
Motivation

• Context
• Most difficult duty of a department chair
  Reichenbach, 2000
• Assignments done manually, countless reviews,
  persistent inconsistencies
• Unhappy instructors, unhappy GTAs, unhappy
  students
• Observation
• Computers are good at maintaining consistency
• Humans are good at balancing tradeoffs
• Our solution
• An online, constraint-based system
• With interactive automated search mechanisms

5
Outline

• Task Motivation
• System Architecture Interfaces
• Scientific aspects
• Problem Modeling
• Problem Solving
• Comparing Characterizing Solvers
• Motivation revisited Conclusions

6
System Architecture
7
GTA interface Preference Specification
8
Manager interface TA Hiring Load
9
Outline

• Task Motivation
• System Architecture Interfaces
• Scientific aspects
• Problem Modeling
• Problem Solving
• Comparing Characterizing Solvers
• Motivation revisited Conclusions

10
Constraint-based Model

• Variables
• Grading, conducting lectures, labs recitations
• Values
• Hired GTAs ( preference for each value in
  domain)
• Constraints
• Unary ITA certification, enrollment, time
  conflict, non-zero preferences, etc.
• Binary (Mutex) overlapping courses
• Non-binary same-TA, capacity, confinement
• Objective
• longest partial and consistent solution (primary
  criterion)
• while maximizing GTAs preferences (secondary
  criterion)

11
Outline

• Task Motivation
• System Architecture Interfaces
• Scientific aspects
• Problem Modeling
• Problem Solving
• Comparing Characterizing Solvers
• Motivation revisited Conclusions

12
Problem Solving

• Interactive decision making
• Seamlessly switching between perspectives
• Propagates decisions (MAC)
• Automated search algorithms
• Heuristic backtrack search (BT)
• Stochastic local search (LS)
• Multi-agent search (ERA)
• Randomized backtrack search (RDGR)
• Future Auction-based, GA, MIP, LD-search, etc.
• On-going Cooperative/hybrid strategies

13
Manager interface Interactive Selection
14
Dual perspective
Task-centered view
Resource-centered view
15
Heuristic BT Search

• Since we dont know, a priori, whether instance
  is solvable, tight, or over-constrained
• Modified basic backtrack mechanism to deal with
  this situation
• We designed tested various ordering heuristics
  
• Dynamic LD was consistently best
• Branching factor relatively huge (30)
• Causes thrashing, backtrack never reaches early
  variables

16
Stochastic Local Search

• Hill-climbing with min-conflict heuristic
• Constraint propagation
• To handle non-binary constraints (e.g.,
  high-arity capacity constraints)
• Greedy
• Consistent assignments are not undone
• Random walk to avoid local maxima
• Random restarts to recover from local maxima

17
Multi-Agent Search (ERA) Liu et al. 02

• Extremely decentralized local search
• Agents (variables) seek to occupy best positions
  (values)
• Environment records constraint violation in each
  position of an agent given positions of other
  agents
• Agents move, egoistically, between positions
  according to reactive Rules
• Decisions are local
• An agent can always kick other agents from a
  favorite position even when value of global
  objective function is not improved
• ERA appears immune to local optima
• Lack of centralized control
• Agents continue to kick each other
• Deadlock appears in over-constrained problems

18
Randomized BT Search

• Random variable/value selection allows BT to
  visit a wider area of the search space
  Gomes et al. 98
• Restarts to overcome thrashing
• Walsh proposed RGR Walsh 99
• Our strategy, RDGR, improves RGR with dynamic
  choice of cutoff values for the restart strategy
  Guddeti Choueiry 04

19
Optimizing solutions

• Primary criterion solution length
• BT, LS, ERA, RGR, RDGR
• Secondary criterion preference values
• BT, LS, RGR, RDGR
• Criterion
• Average preference
• Geometric mean
• Maximum minimal preference

20
More Solvers

• Interactive decision making
• Automated search algorithms
• BT, LS, ERA, RGR, RDGR.
• Future Auction-based, GA, MIP, LD-search, etc.
• On-going Cooperative / hybrid strategies

21
Outline

• Task Motivation
• System Architecture Interfaces
• Scientific aspects
• Problem Modeling
• Problem Solving
• Comparing Characterizing Solvers
• Motivation revisited Conclusions

22
Comparing Solvers

• Using the same CSP encoding, students implements
  solvers separately and competed for best results
• Experience lead to the identification of
  behavioral criteria and regimes that characterize
  the performance of the various solvers in the
  context of GTAP

23
Characterizing Solvers

• General criteria
• Stability, solution length, vulnerability to
  local optima, deadlock, thrashing, etc.
• Tight but solvable instances
• ERA ? RDGR ? RGR ? BT ? LS
• Over-constrained instances
• RDGR ? RGR ? BT ? ERA ? LS

24
Outline

• Task Motivation
• System Architecture Interfaces
• Scientific aspects
• Problem Modeling
• Problem Solving
• Comparing Characterization Solvers
• Motivation revisited Conclusions

25
Motivation (revisited)

• Most difficult duty of a department chair
• Keeps the manager in the decision loop while
  removing the need for tedious and error-prone
  manual assignments
• Helps producing quick (3 weeks down to 2 days)
  and satisfactory (stable) assignments
• Initially, assignments were manually done on
  paper
• Now, on-line data acquisition process
• Enabled department to streamline standardize
  GTA selection, hiring, and assignment
• Overworked staff, unhappy GTAs
• Overjoyed staff (relieved from handling
  application forms and massive paperwork)
• Enthusiastic anonymous online reviews from
  applicants

26
History Evaluation

• System entirely built by students
• Modeling started in January 2001
• Prototype system used since August 2001
• Features improved and added as needs arised
• No formal longitudinal study
• Since August 2003 109 GTA users, 23 feedback
  responses
• Since April 2004, CSE implemented on-line GTA
  evaluation by faculty on top of GTAAP

27
GTA Online Feedback
Navigation
Data entry
23 responses
28
Conclusions

• Integrated interactive automated
  problem-solving strategies
• Reduced the burden of the manager
• Lead to quick development of stable solutions
• Our efforts
• Helped the department
• Trained students in CP techniques
• Paved new avenues for research
• Cooperative, hybrid search
• Visualization of solution space

29
ltltlt end of presentation

• I welcome your questions
• Please contact me for a live demo

30
(No Transcript)
31
Manager interface Course Load Specification
32
Manager interface Preassignment
33
Manager interface Constraint Specification