Generative vs' Discriminative models

1
Generative vs. Discriminative models

• Vidit Jain
• IR Seminar (Spring 2008)

2
Readings

• Required
• Ng and Jordan. NIPS01
• Nallapati. SIGIR04
• Lafferty Zhai. Workshop on language modeling
  and IR, 2001.
• Required Skims
• Jain et al. CVPR08
• Jain et al. ICCV07

3
Readings (contd.)

• Recommended
• Wei and Croft. RIAO07
• Cao et al. SIGIR06
• Raina et al. NIPS04
• Others
• Metzler. Tech Report, CIIR
• Jaakkola and Hausler. NIPS98
• Ulusoy and Bishop. CVPR05
• Long and Servedillo. COLT06

4
Agenda

• Asymptotic results for these models (Ng Jordan)
• Relevance of these results in the context of IR
• Language models as generative models (Lafferty
  Zhai)
• Topic models (Wei)
• Discriminative models for IR (Nallapati)
• IR as classification
• Is the paradigm choice task / evaluation
  dependent?
• vision (Jain), ranking (Metzler)
• Hybrid models

5
Ng Jordan (NIPS01)

• Generative models have better performance than
  discriminative models for less data, but the
  asymptotic error is higher for the former.
• Is it true for any pair of models?
• VC-dimension of a classifier vs. the complexity
  of the problem.
• Does smoothing change anything?

6
Relevance for IR

• Is generative modeling always preferable for IR
  tasks ?
• Less data
• Can we compress representation / feature space?
• Expansion of representation ?
• Effect of smoothing

7
Lafferty Zhai

• Relevance models and language models are
  probabilistically equivalent but different from
  statistical point of view.
• Assumptions for language models
• Dr and Qr are independent.
• D and R are independent.
• Independence of query terms.
• Incorporating relevance judgments in the two
  models.

8
Wei and Croft, RIAO07

• Vocabulary mismatch
• User context
• Manually-built ideal topics why is it the
  best?
• Topic model construction
• Results not good mostly because of the
  ideal-iness.
• Analysis

9
Discriminative models

• Is IR classification?
• Reasoning for language models being generative
  not complete.
• Benefits for discriminative
• Expressiveness, burstiness, arbitrary features,
  relevance.
• Results analysis
• Is the comparison fair?

10
Paradigm choice

• Evaluation task dependent
• Metzlers observations
• IR not concerned with accuracy or likelihood
• Direct optimization of the evaluation metric

11
Task dependent (Vision papers)
People as topics (Jain, ICCV07)
Event Classification (Jain, CVPR08)
12
Hybrid models

• Use hybrid models to use benefits of both types
  of models.
• Time-dependent learning parameter ?
• Over-sampling weight ?
• Hybrid optimization functions ? (Raina, Ulusoy )