Title: Semantic Representations with Probabilistic Topic Models
1Semantic Representations with Probabilistic Topic
Models
Mark Steyvers Department of Cognitive
Sciences University of California, Irvine
Joint work with Tom Griffiths, UC
Berkeley Padhraic Smyth, UC Irvine
2Topic Models in Machine Learning
- Unsupervised extraction of content from large
text collection - Topics provide quick summary of content / gist
- What is in this corpus?
- What is in this document, paragraph, or sentence?
- What are similar documents to a query?
- What are the topical trends over time?
3Topic Models in Psychology
- Topic models address three computational problems
for semantic memory system - Gist extraction what is this set of words about?
- Disambiguation what is the sense of this word?
- - E.g. football field vs. magnetic field
- Prediction what fact, concept, or word is next?
4Two approaches to semantic representation
Semantic networks
Semantic Spaces
BAT
BALL
LOAN
CASH
GAME
FUN
MONEY
PLAY
THEATER
BANK
STAGE
RIVER
STREAM
Can be learned (e.g. Latent Semantic Analysis),
but is this representation flexible enough?
How are these learned?
5Overview
- I Probabilistic Topic Models
- generative model
- statistical inference Gibbs sampling
- II Explaining human memory
- word association
- semantic isolation
- false memory
- III Information retrieval
-
-
6Probabilistic Topic Models
- Extract topics from large text collections
- ? unsupervised
- ? generative
- ? Bayesian statistical inference
- Our modeling work is based on
- pLSI Model Hoffman (1999)
- LDA Model Blei, Ng, and Jordan (2001, 2003)
- Topics Model Griffiths and Steyvers (2003, 2004)
7Model input bag of words
- Matrix of number of times words occur in
documents - Note some function words are deleted the,
a, and, etc
documents
words
8Probabilistic Topic Models
- A topic represents a probability distribution
over words - Related words get high probability in same topic
- Example topics extracted from NIH/NSF grants
Probability distribution over words. Most likely
words listed at the top
9Document mixture of topics
20
Document ------------------------------- --------
--------------------------------------------------
---- ---------------------------------------------
------------------------------------------
80
100
Document ------------------------------- --------
--------------------------------------------------
---- ---------------------------------------------
------------------------------------------
10Generative Process
- For each document, choose a mixture of topics
- ? ? Dirichlet(?)
- Sample a topic 1..T from the mixture z ?
Multinomial(?) - Sample a word from the topic w ?
Multinomial(?(z)) ? ? Dirichlet(ß)
Nd
D
T
11Prior Distributions
- Dirichlet priors encourage sparsity on topic
mixtures and topics
Topic 3
Word 3
Topic 1
Topic 2
Word 1
Word 2
? Dirichlet( a )
? Dirichlet( ß )
(darker colors indicate lower probability)
12Creating Artificial Dataset
Two topics
16 documents
Docs
Can we recover the original topics and topic
mixtures from this data?
13Statistical Inference
- Three sets of latent variables
- topic mixtures ?
- word mixtures ?
- topic assignments z
- Estimate posterior distribution over topic
assignments - P( z w )
- (we can later infer ? and ?)
14Statistical Inference
- Exact inference is impossible
- Use approximate methods
- Markov chain Monte Carlo (MCMC) with Gibbs
sampling
Sum over Tn terms
15Gibbs Sampling
count of topic t assigned to doc d
count of word w assigned to topic t
probability that word i is assigned to topic t
16Example of Gibbs Sampling
- Assign word tokens randomly to topics
(?topic 1 ?topic 2 )
17After 1 iteration
- Apply sampling equation to each word token
(?topic 1 ?topic 2 )
18After 4 iterations
(?topic 1 ?topic 2 )
19After 8 iterations
(?topic 1 ?topic 2 )
20After 32 iterations
?
(?topic 1 ?topic 2 )
21Algorithm input/output
INPUT word-document counts (word order is
irrelevant)
OUTPUT topic assignments to each word P( zi
) likely words in each topic P( w z ) likely
topics in each document (gist) P( ? d )
22Software
- Public-domain MATLAB toolbox for topic modeling
on the Web - http//psiexp.ss.uci.edu/research/programs_data/t
oolbox.htm
23Examples Topics from New York Times
Terrorism
Wall Street Firms
Stock Market
Bankruptcy
WEEK DOW_JONES POINTS 10_YR_TREASURY_YIELD PERCENT
CLOSE NASDAQ_COMPOSITE STANDARD_POOR CHANGE FRIDA
Y DOW_INDUSTRIALS GRAPH_TRACKS EXPECTED BILLION NA
SDAQ_COMPOSITE_INDEX EST_02 PHOTO_YESTERDAY YEN 10
500_STOCK_INDEX
WALL_STREET ANALYSTS INVESTORS FIRM GOLDMAN_SACHS
FIRMS INVESTMENT MERRILL_LYNCH COMPANIES SECURITIE
S RESEARCH STOCK BUSINESS ANALYST WALL_STREET_FIRM
S SALOMON_SMITH_BARNEY CLIENTS INVESTMENT_BANKING
INVESTMENT_BANKERS INVESTMENT_BANKS
SEPT_11 WAR SECURITY IRAQ TERRORISM NATION KILLED
AFGHANISTAN ATTACKS OSAMA_BIN_LADEN AMERICAN ATTAC
K NEW_YORK_REGION NEW MILITARY NEW_YORK WORLD NATI
ONAL QAEDA TERRORIST_ATTACKS
BANKRUPTCY CREDITORS BANKRUPTCY_PROTECTION ASSETS
COMPANY FILED BANKRUPTCY_FILING ENRON BANKRUPTCY_C
OURT KMART CHAPTER_11 FILING COOPER BILLIONS COMPA
NIES BANKRUPTCY_PROCEEDINGS DEBTS RESTRUCTURING CA
SE GROUP
24Example topics from an educational corpus
PRINTING PAPER PRINT PRINTED TYPE PROCESS INK PRES
S IMAGE
PLAY PLAYS STAGE AUDIENCE THEATER ACTORS DRAMA SHA
KESPEARE ACTOR
TEAM GAME BASKETBALL PLAYERS PLAYER PLAY PLAYING S
OCCER PLAYED
JUDGE TRIAL COURT CASE JURY ACCUSED GUILTY DEFENDA
NT JUSTICE
HYPOTHESIS EXPERIMENT SCIENTIFIC OBSERVATIONS SCIE
NTISTS EXPERIMENTS SCIENTIST EXPERIMENTAL TEST
STUDY TEST STUDYING HOMEWORK NEED CLASS MATH TRY T
EACHER
Example topics from psych review abstracts
SIMILARITY CATEGORY CATEGORIES RELATIONS DIMENSION
S FEATURES STRUCTURE SIMILAR REPRESENTATION ONJECT
S
STIMULUS CONDITIONING LEARNING RESPONSE STIMULI RE
SPONSES AVOIDANCE REINFORCEMENT CLASSICAL DISCRIMI
NATION
MEMORY RETRIEVAL RECALL ITEMS INFORMATION TERM REC
OGNITION ITEMS LIST ASSOCIATIVE
GROUP INDIVIDUAL GROUPS OUTCOMES INDIVIDUALS GROUP
S OUTCOMES INDIVIDUALS DIFFERENCES INTERACTION
EMOTIONAL EMOTION BASIC EMOTIONS AFFECT STATES EXP
ERIENCES AFFECTIVE AFFECTS RESEARCH
25Choosing number of topics
- Bayesian model selection
- Generalization test
- e.g., perplexity on out-of-sample data
- Non-parametric Bayesian approach
- Number of topics grows with size of data
- E.g. Hierarchical Dirichlet Processes (HDP)
26Applications to Human Memory
27Computational Problems for Semantic Memory System
- Gist extraction
- What is this set of words about?
- Disambiguation
- What is the sense of this word?
- Prediction
- what fact, concept, or word is next?
28Disambiguation
FOOTBALL FIELD
P( zFIELD w )
P( zFIELD w )
FIELD MAGNETIC MAGNET WIRE NEEDLE CURRENT COIL POL
ES
BALL GAME TEAM FOOTBALL BASEBALL PLAYERS PLAY FIEL
D
FIELD MAGNETIC MAGNET WIRE NEEDLE CURRENT COIL POL
ES
BALL GAME TEAM FOOTBALL BASEBALL PLAYERS PLAY FIEL
D
29Modeling Word Association
30Word Association(norms from Nelson et al. 1998)
CUE PLANET
31Word Association(norms from Nelson et al. 1998)
CUE PLANET
people EARTH STARS SPACE
SUN MARS UNIVERSE SATURN GALAXY
associate number 1 2 3 4 5 6 7 8
(vocabulary 5000 words)
32Word Association as a Prediction Problem
- Given that a single word is observed, predict
what other words might occur in that context - Under a single topic assumption
Cue
Response
33Word Association(norms from Nelson et al. 1998)
CUE PLANET
people EARTH STARS SPACE
SUN MARS UNIVERSE SATURN GALAXY
model STARS STAR SUN
EARTH SPACE SKY PLANET UNIVERSE
associate number 1 2 3 4 5 6 7 8
First associate EARTH has rank 4 in model
34Median rank of first associate
TOPICS
35Median rank of first associate
LSA
TOPICS
36Episodic Memory Semantic Isolation
EffectsFalse Memory
37Semantic Isolation Effect
Study this list PEAS, CARROTS, BEANS, SPINACH,
LETTUCE, HAMMER, TOMATOES, CORN, CABBAGE, SQUASH
HAMMER, PEAS, CARROTS, ...
38Semantic isolation effect / Von Restorff effect
- Finding contextually unique words are better
remembered - Verbal explanations
- Attention, surprise, distinctiveness
- Our approach
- assume memories can be accessed and encoded at
multiple levels of description - Semantic/ Gist aspects generic information
- Verbatim specific information
39Computational Problem
- How to tradeoff specificity and generality?
- Remembering detail and gist
- Dual route topic model topic model encoding
of specific words
40Dual route topic model
- Two ways to generate words
- Topic Model
- Verbatim word distribution (unique to document)
- Each word comes from a single route
- Switch variable xi for every word i
- xi 0 ? topics
- xi 1 ? verbatim
- Conditional prob. of a word under a document
41Graphical Model
Variable x is a switch x0 ? sample from
topic x1 ? sample from verbatim word
distribution
42Applying Dual Route Topic Model to Human Memory
- Train model on educational corpus (TASA)
- 37K documents, 1700 topics
- Apply model to list memory experiments
- Study list is a document
- Recall probability based on model
43RETRIEVAL
ENCODING
Study wordsPEAS CARROTS BEANS SPINACH LETTUCE
HAMMER TOMATOES CORN CABBAGE SQUASH
Special
verbatim
44Hunt Lamb (2001 exp. 1)
- OUTLIER LIST
- PEAS
- CARROTS
- BEANS
- SPINACH
- LETTUCE
- HAMMER
- TOMATOES
- CORN
- CABBAGE
- SQUASH
CONTROL LIST SAW SCREW CHISEL DRILL
SANDPAPER HAMMER NAILS BENCH RULER ANVIL
45False Memory(e.g. Deese, 1959 Roediger
McDermott)
Study this list Bed, Rest, Awake, Tired, Dream,
Wake, Snooze, Blanket, Doze, Slumber, Snore, Nap,
Peace, Yawn, Drowsy
SLEEP, BED, REST, ...
46False memory effects
Number of Associates
3
6
9
MAD FEAR HATE SMOOTH NAVY HEAT SALAD TUNE COURTS C
ANDY PALACE PLUSH TOOTH BLIND WINTER
MAD FEAR HATE RAGE TEMPER FURY SALAD TUNE COURTS C
ANDY PALACE PLUSH TOOTH BLIND WINTER
MAD FEAR HATE RAGE TEMPER FURY WRATH HAPPY FIGHT C
ANDY PALACE PLUSH TOOTH BLIND WINTER
(lure ANGER)
Robinson Roediger (1997)
47Modeling Serial Order Effects in Free Recall
48Problem
- Dual route model predicts no sequential effects
- Order of words is important in human memory
experiments - Standard Gibbs sampler is psychologically
implausible - Assumes list is processed in parallel
- Each item can influence encoding of each other
item
49Semantic isolation experiment to study order
effects
- Study lists of 14 words long
- 14 isolate lists (e.g. A A A B A A ... A A )
- 14 control lists (e.g. A A A A A A ... A A )
- Varied serial position of isolate (any of 14
positions)
50Immediate Recall Results
Control list A A A A A ... A
Isolate list B A A A A ... A
51Immediate Recall Results
Control list A A A A A ... A
Isolate list B A A A A ... A
52Immediate Recall Results
Control list A A A A A ... A
Isolate list A B A A A ... A
53Immediate Recall Results
Control list A A A A A ... A
Isolate list A A B A A ... A
54Immediate Recall Results
55Modified Gibbs Sampling Scheme
- Update items non-uniformly in Gibbs sampler
- Probability of updating item i after observing
words 1..t - ? Words further back in time are less likely to
be re-assigned
item to update
Current time
Parameter
56Effect of Sampling Scheme
l1
l0.3
l0
Study order
57Normalized Serial Position Effects
DATA
MODEL
58Information Retrieval Human Memory
59Example
- Searching for information on Padhraic Smyth
60Query Smyth
61Query Smyth irish computer science department
62Query Smyth irish computer science department
weather prediction seasonal climate fluctuations
hmm models nips conference consultant yahoo
netflix prize dave newman steyvers
63Problem
- More information in a query can lead to worse
search results - Human memory typically works better with more
cues - Problem how can we better match queries to
documents to allow for partial matches, and
matches across documents?
64Dual route model for information retrieval
- Encode documents with two routes
- contextually unique words ? verbatim route
- Thematic words ? topics route
65Example encoding of a psych review abstract
Contextually unique words ALCOVE, SCHAFFER,
MEDIN, NOSOFSKY
Kruschke, J. K.. ALCOVE An exemplar-based
connectionist model of category learning.
Psychological Review, 99, 22-44.
Topic 1 (p0.21) learning phenomena acquisition
learn acquired ... Topic 22 (p0.17) similarity
objects object space category dimensional
categories spatial Topic 61 (p0.08)
representations representation order alternative
1st higher 2nd descriptions problem form
66Retrieval Experiments
- For each candidate document, calculate how likely
the query was generated from the models
encoding
67Information Retrieval Results
Evaluation Metric precision for 10 highest
ranked docs
FRs
APs
Method Title Desc Concepts
TFIDF .406 .434 .549
LSI .455 .469 .523
LDA .478 .463 .556
SW .488 .468 .561
SWB .495 .473 .558
Method Title Desc Concepts
TFIDF .300 .287 .483
LSI .366 .327 .487
LDA .428 .340 .487
SW .448 .407 .560
SWB .459 .400 .560
68Information retrieval systems in the mind web
- Similar computational demands
- Both retrieve the most relevant items from a
large information repository in response to
external cues or queries. - Useful analogies/ interdisciplinary approaches
- Many cognitive aspects in information retrieval
- Internet content is produced by humans
- Queries are formulated by humans
69Recent Papers
- Steyvers, M., Griffiths, T.L., Dennis, S.
(2006). Probabilistic inference in human semantic
memory. Trends in Cognitive Sciences, 10(7),
327-334. - Griffiths, T.L., Steyvers, M., Tenenbaum,
J.B.T. (2007). Topics in Semantic Representation.
Psychological Review, 114(2), 211-244. - Griffiths, T.L., Steyvers, M., Firl, A. (in
press). Google and the mind Predicting fluency
with PageRank. Psychological Science. - Steyvers, M. Griffiths, T.L. (in press).
Rational Analysis as a Link between Human Memory
and Information Retrieval. In N. Chater and M
Oaksford (Eds.) The Probabilistic Mind Prospects
from Rational Models of Cognition. Oxford
University Press. - Chemudugunta, C., Smyth, P., Steyvers, M.
(2007, in press). Modeling General and Specific
Aspects of Documents with a Probabilistic Topic
Model. In Advances in Neural Information
Processing Systems, 19.
70Text Mining Applications
71Topics provide quick summary of content
- Who writes on what topics?
- What is in this corpus? What is in this document?
- What are the topical trends over time?
- Who is mentioned in what context?
72Faculty Browser
- System spiders UCI/UCSD faculty websites related
to CalIT2 California Institute for
Telecommunications and Information Technology - Applies topic model on text extracted from pdf
files - Browser demohttp//yarra.calit2.uci.edu/calit2/
73one topic
most prolific researchers for this topic
74one researcher
topics this researcher works on
other researchers with similar topical interests
75Inferred network of researchers connected through
topics
76Analyzing the New York Times
330,000 articles 2000-2002
77Extracted Named Entities
Three investigations began Thursday into the
securities and exchange_commission's choice of
william_webster to head a new board overseeing
the accounting profession. house and
senate_democrats called for the resignations of
both judge_webster and harvey_pitt, the
commission's chairman. The white_house expressed
support for judge_webster as well as for
harvey_pitt, who was harshly criticized Thursday
for failing to inform other commissioners before
they approved the choice of judge_webster that he
had led the audit committee of a company facing
fraud accusations. The president still has
confidence in harvey_pitt, said dan_bartlett,
bush's communications director
- Used standard algorithms to extract named
entities - People
- Places
- Organizations
78Standard Topic Model with Entities
79Topic Trends
Tour-de-France
Proportion of words assigned to topic for that
time slice
Quarterly Earnings
Anthrax
80Example of Extracted Entity-Topic Network
81Prediction of Missing Entities in Text
Shares of XXXX slid 8 percent, or 1.10, to 12.65 Tuesday, as major credit agencies said the conglomerate would still be challenged in repaying its debts, despite raising 4.6 billion Monday in taking its finance group public. Analysts at XXXX Investors service in XXXX said they were keeping XXXX and its subsidiaries under review for a possible debt downgrade, saying the company will continue to face a significant debt burden,'' with large slices of debt coming due, over the next 18 months. XXXX said
Test article with entities removed
82Prediction of Missing Entities in Text
Shares of XXXX slid 8 percent, or 1.10, to 12.65 Tuesday, as major credit agencies said the conglomerate would still be challenged in repaying its debts, despite raising 4.6 billion Monday in taking its finance group public. Analysts at XXXX Investors service in XXXX said they were keeping XXXX and its subsidiaries under review for a possible debt downgrade, saying the company will continue to face a significant debt burden,'' with large slices of debt coming due, over the next 18 months. XXXX said
fitch goldman-sachs lehman-brother moody morgan-stanley new-york-stock-exchange standard-and-poor tyco tyco-international wall-street worldco
Test article with entities removed
Actual missing entities
83Prediction of Missing Entities in Text
Shares of XXXX slid 8 percent, or 1.10, to 12.65 Tuesday, as major credit agencies said the conglomerate would still be challenged in repaying its debts, despite raising 4.6 billion Monday in taking its finance group public. Analysts at XXXX Investors service in XXXX said they were keeping XXXX and its subsidiaries under review for a possible debt downgrade, saying the company will continue to face a significant debt burden,'' with large slices of debt coming due, over the next 18 months. XXXX said
fitch goldman-sachs lehman-brother moody morgan-stanley new-york-stock-exchange standard-and-poor tyco tyco-international wall-street worldco
wall-street new-york nasdaq securities-exchange-commission sec merrill-lynch new-york-stock-exchange goldman-sachs standard-and-poor
Test article with entities removed
Actual missing entities
Predicted entities given observed words (matches
in blue)
84Model Extensions
85Model Extensions
- HMM-topics model
- Modeling aspects of syntax
- Hierarchical topic model
- Modeling relations between topics
- Collocation topic models
- Learning collocations of words within topics
86Hidden Markov Topic Model
87Hidden Markov Topics Model
- Syntactic dependencies ? short range dependencies
- Semantic dependencies ? long-range
q
Semantic state generate words from topic model
z1
z2
z3
z4
w1
w2
w3
w4
Syntactic states generate words from HMM
s1
s2
s3
s4
(Griffiths, Steyvers, Blei, Tenenbaum, 2004)
88Transition between semantic state and syntactic
states
OF 0.6 FOR 0.3 BETWEEN 0.1
0.8
z 1 0.4
z 2 0.6
HEART 0.2 LOVE 0.2 SOUL 0.2 TEARS 0.2 JOY
0.2
SCIENTIFIC 0.2 KNOWLEDGE 0.2 WORK
0.2 RESEARCH 0.2 MATHEMATICS 0.2
0.7
0.1
0.3
0.2
THE 0.6 A 0.3 MANY 0.1
0.9
89Combining topics and syntax
x 2
OF 0.6 FOR 0.3 BETWEEN 0.1
x 1
0.8
z 1 0.4
z 2 0.6
HEART 0.2 LOVE 0.2 SOUL 0.2 TEARS 0.2 JOY
0.2
SCIENTIFIC 0.2 KNOWLEDGE 0.2 WORK
0.2 RESEARCH 0.2 MATHEMATICS 0.2
0.7
0.1
0.3
0.2
x 3
THE 0.6 A 0.3 MANY 0.1
0.9
THE
90Combining topics and syntax
x 2
OF 0.6 FOR 0.3 BETWEEN 0.1
x 1
0.8
z 1 0.4
z 2 0.6
HEART 0.2 LOVE 0.2 SOUL 0.2 TEARS 0.2 JOY
0.2
SCIENTIFIC 0.2 KNOWLEDGE 0.2 WORK
0.2 RESEARCH 0.2 MATHEMATICS 0.2
0.7
0.1
0.3
0.2
x 3
THE 0.6 A 0.3 MANY 0.1
0.9
THE LOVE
91Combining topics and syntax
x 2
OF 0.6 FOR 0.3 BETWEEN 0.1
x 1
0.8
z 1 0.4
z 2 0.6
HEART 0.2 LOVE 0.2 SOUL 0.2 TEARS 0.2 JOY
0.2
SCIENTIFIC 0.2 KNOWLEDGE 0.2 WORK
0.2 RESEARCH 0.2 MATHEMATICS 0.2
0.7
0.1
0.3
0.2
x 3
THE 0.6 A 0.3 MANY 0.1
0.9
THE LOVE OF
92Combining topics and syntax
x 2
OF 0.6 FOR 0.3 BETWEEN 0.1
x 1
0.8
z 1 0.4
z 2 0.6
HEART 0.2 LOVE 0.2 SOUL 0.2 TEARS 0.2 JOY
0.2
SCIENTIFIC 0.2 KNOWLEDGE 0.2 WORK
0.2 RESEARCH 0.2 MATHEMATICS 0.2
0.7
0.1
0.3
0.2
x 3
THE 0.6 A 0.3 MANY 0.1
0.9
THE LOVE OF RESEARCH
93Semantic topics
PLANTS PLANT LEAVES SEEDS SOIL ROOTS FLOWERS WATER
FOOD GREEN SEED STEMS FLOWER STEM LEAF ANIMALS RO
OT POLLEN GROWING GROW
GOLD IRON SILVER COPPER METAL METALS STEEL CLAY LE
AD ADAM ORE ALUMINUM MINERAL MINE STONE MINERALS P
OT MINING MINERS TIN
BEHAVIOR SELF INDIVIDUAL PERSONALITY RESPONSE SOCI
AL EMOTIONAL LEARNING FEELINGS PSYCHOLOGISTS INDIV
IDUALS PSYCHOLOGICAL EXPERIENCES ENVIRONMENT HUMAN
RESPONSES BEHAVIORS ATTITUDES PSYCHOLOGY PERSON
CELLS CELL ORGANISMS ALGAE BACTERIA MICROSCOPE MEM
BRANE ORGANISM FOOD LIVING FUNGI MOLD MATERIALS NU
CLEUS CELLED STRUCTURES MATERIAL STRUCTURE GREEN M
OLDS
DOCTOR PATIENT HEALTH HOSPITAL MEDICAL CARE PATIEN
TS NURSE DOCTORS MEDICINE NURSING TREATMENT NURSES
PHYSICIAN HOSPITALS DR SICK ASSISTANT EMERGENCY P
RACTICE
BOOK BOOKS READING INFORMATION LIBRARY REPORT PAGE
TITLE SUBJECT PAGES GUIDE WORDS MATERIAL ARTICLE
ARTICLES WORD FACTS AUTHOR REFERENCE NOTE
MAP NORTH EARTH SOUTH POLE MAPS EQUATOR WEST LINES
EAST AUSTRALIA GLOBE POLES HEMISPHERE LATITUDE PL
ACES LAND WORLD COMPASS CONTINENTS
FOOD FOODS BODY NUTRIENTS DIET FAT SUGAR ENERGY MI
LK EATING FRUITS VEGETABLES WEIGHT FATS NEEDS CARB
OHYDRATES VITAMINS CALORIES PROTEIN MINERALS
94Syntactic classes
BE MAKE GET HAVE GO TAKE DO FIND USE SEE HELP KEEP
GIVE LOOK COME WORK MOVE LIVE EAT BECOME
ONE SOME MANY TWO EACH ALL MOST ANY THREE THIS EVE
RY SEVERAL FOUR FIVE BOTH TEN SIX MUCH TWENTY EIGH
T
HE YOU THEY I SHE WE IT PEOPLE EVERYONE OTHERS SCI
ENTISTS SOMEONE WHO NOBODY ONE SOMETHING ANYONE EV
ERYBODY SOME THEN
MORE SUCH LESS MUCH KNOWN JUST BETTER RATHER GREAT
ER HIGHER LARGER LONGER FASTER EXACTLY SMALLER SOM
ETHING BIGGER FEWER LOWER ALMOST
ON AT INTO FROM WITH THROUGH OVER AROUND AGAINST A
CROSS UPON TOWARD UNDER ALONG NEAR BEHIND OFF ABOV
E DOWN BEFORE
THE HIS THEIR YOUR HER ITS MY OUR THIS THESE A AN
THAT NEW THOSE EACH MR ANY MRS ALL
GOOD SMALL NEW IMPORTANT GREAT LITTLE LARGE BIG
LONG HIGH DIFFERENT SPECIAL OLD STRONG YOUNG COMMO
N WHITE SINGLE CERTAIN
SAID ASKED THOUGHT TOLD SAYS MEANS CALLED CRIED SH
OWS ANSWERED TELLS REPLIED SHOUTED EXPLAINED LAUGH
ED MEANT WROTE SHOWED BELIEVED WHISPERED
95NIPS Semantics
IMAGE IMAGES OBJECT OBJECTS FEATURE RECOGNITION VI
EWS PIXEL VISUAL
KERNEL SUPPORT VECTOR SVM KERNELS SPACE FUNCTION
MACHINES SET
NETWORK NEURAL NETWORKS OUPUT INPUT TRAINING INPUT
S WEIGHTS OUTPUTS
EXPERTS EXPERT GATING HME ARCHITECTURE MIXTURE LEA
RNING MIXTURES FUNCTION GATE
MEMBRANE SYNAPTIC CELL CURRENT DENDRITIC POTENTI
AL NEURON CONDUCTANCE CHANNELS
DATA GAUSSIAN MIXTURE LIKELIHOOD POSTERIOR PRIOR D
ISTRIBUTION EM BAYESIAN PARAMETERS
STATE POLICY VALUE FUNCTION ACTION REINFORCEMENT L
EARNING CLASSES OPTIMAL
NIPS Syntax
IN WITH FOR ON FROM AT USING INTO OVER WITHIN
I X T N - C F P
IS WAS HAS BECOMES DENOTES BEING REMAINS REPRESENT
S EXISTS SEEMS
SEE SHOW NOTE CONSIDER ASSUME PRESENT NEED PROPOSE
DESCRIBE SUGGEST
MODEL ALGORITHM SYSTEM CASE PROBLEM NETWORK METHOD
APPROACH PAPER PROCESS
HOWEVER ALSO THEN THUS THEREFORE FIRST HERE NOW HE
NCE FINALLY
USED TRAINED OBTAINED DESCRIBED GIVEN FOUND PRESEN
TED DEFINED GENERATED SHOWN
96Random sentence generation
LANGUAGE S RESEARCHERS GIVE THE SPEECH S THE
SOUND FEEL NO LISTENERS S WHICH WAS TO BE
MEANING S HER VOCABULARIES STOPPED WORDS S HE
EXPRESSLY WANTED THAT BETTER VOWEL
97Nested Chinese Restaurant Process
98Topic Hierarchies
- In regular topic model, no relations between
topics
topic 1
topic 2
topic 3
- Nested Chinese Restaurant Process
- Blei, Griffiths, Jordan, Tenenbaum (2004)
- Learn hierarchical structure, as well as topics
within structure
topic 6
topic 4
topic 5
topic 7
99Example Psych Review Abstracts
THE OF AND TO IN A IS
A MODEL MEMORY FOR MODELS TASK INFORMATION RESULTS
ACCOUNT
SELF SOCIAL PSYCHOLOGY RESEARCH RISK STRATEGIES IN
TERPERSONAL PERSONALITY SAMPLING
MOTION VISUAL SURFACE BINOCULAR RIVALRY CONTOUR DI
RECTION CONTOURS SURFACES
DRUG FOOD BRAIN AROUSAL ACTIVATION AFFECTIVE HUNGE
R EXTINCTION PAIN
RESPONSE STIMULUS REINFORCEMENT RECOGNITION STIMUL
I RECALL CHOICE CONDITIONING
SPEECH READING WORDS MOVEMENT MOTOR VISUAL WORD SE
MANTIC
ACTION SOCIAL SELF EXPERIENCE EMOTION GOALS EMOTIO
NAL THINKING
GROUP IQ INTELLIGENCE SOCIAL RATIONAL INDIVIDUAL G
ROUPS MEMBERS
SEX EMOTIONS GENDER EMOTION STRESS WOMEN HEALTH HA
NDEDNESS
REASONING ATTITUDE CONSISTENCY SITUATIONAL INFEREN
CE JUDGMENT PROBABILITIES STATISTICAL
IMAGE COLOR MONOCULAR LIGHTNESS GIBSON SUBMOVEMENT
ORIENTATION HOLOGRAPHIC
CONDITIONIN STRESS EMOTIONAL BEHAVIORAL FEAR STIMU
LATION TOLERANCE RESPONSES
100Generative Process
THE OF AND TO IN A IS
A MODEL MEMORY FOR MODELS TASK INFORMATION RESULTS
ACCOUNT
SELF SOCIAL PSYCHOLOGY RESEARCH RISK STRATEGIES IN
TERPERSONAL PERSONALITY SAMPLING
MOTION VISUAL SURFACE BINOCULAR RIVALRY CONTOUR DI
RECTION CONTOURS SURFACES
DRUG FOOD BRAIN AROUSAL ACTIVATION AFFECTIVE HUNGE
R EXTINCTION PAIN
RESPONSE STIMULUS REINFORCEMENT RECOGNITION STIMUL
I RECALL CHOICE CONDITIONING
SPEECH READING WORDS MOVEMENT MOTOR VISUAL WORD SE
MANTIC
ACTION SOCIAL SELF EXPERIENCE EMOTION GOALS EMOTIO
NAL THINKING
GROUP IQ INTELLIGENCE SOCIAL RATIONAL INDIVIDUAL G
ROUPS MEMBERS
SEX EMOTIONS GENDER EMOTION STRESS WOMEN HEALTH HA
NDEDNESS
REASONING ATTITUDE CONSISTENCY SITUATIONAL INFEREN
CE JUDGMENT PROBABILITIES STATISTICAL
IMAGE COLOR MONOCULAR LIGHTNESS GIBSON SUBMOVEMENT
ORIENTATION HOLOGRAPHIC
CONDITIONIN STRESS EMOTIONAL BEHAVIORAL FEAR STIMU
LATION TOLERANCE RESPONSES
101Collocation Topic Model
102What about collocations?
- Why are these words related?
- PLAY - GROUND
- DOW - JONES
- BUMBLE - BEE
- Suggests at least two routes for association
- Semantic
- Collocation
- ? Integrate collocations into topic model
103Collocation Topic Model
TOPIC MIXTURE
If x0, sample a word from the topic If
x1, sample a word from the distribution based on
previous word
...
TOPIC
TOPIC
TOPIC
WORD
WORD
WORD
...
X
X
...
104Collocation Topic Model
TOPIC MIXTURE
Example DOW JONES RISES JONES is more likely
explained as a word following DOW than as word
sampled from topic Result DOW_JONES recognized
as collocation
...
TOPIC
TOPIC
DOW
JONES
RISES
...
X1
X0
...
105Examples Topics from New York Times
Terrorism
Wall Street Firms
Stock Market
Bankruptcy
WEEK DOW_JONES POINTS 10_YR_TREASURY_YIELD PERCENT
CLOSE NASDAQ_COMPOSITE STANDARD_POOR CHANGE FRIDA
Y DOW_INDUSTRIALS GRAPH_TRACKS EXPECTED BILLION NA
SDAQ_COMPOSITE_INDEX EST_02 PHOTO_YESTERDAY YEN 10
500_STOCK_INDEX
WALL_STREET ANALYSTS INVESTORS FIRM GOLDMAN_SACHS
FIRMS INVESTMENT MERRILL_LYNCH COMPANIES SECURITIE
S RESEARCH STOCK BUSINESS ANALYST WALL_STREET_FIRM
S SALOMON_SMITH_BARNEY CLIENTS INVESTMENT_BANKING
INVESTMENT_BANKERS INVESTMENT_BANKS
SEPT_11 WAR SECURITY IRAQ TERRORISM NATION KILLED
AFGHANISTAN ATTACKS OSAMA_BIN_LADEN AMERICAN ATTAC
K NEW_YORK_REGION NEW MILITARY NEW_YORK WORLD NATI
ONAL QAEDA TERRORIST_ATTACKS
BANKRUPTCY CREDITORS BANKRUPTCY_PROTECTION ASSETS
COMPANY FILED BANKRUPTCY_FILING ENRON BANKRUPTCY_C
OURT KMART CHAPTER_11 FILING COOPER BILLIONS COMPA
NIES BANKRUPTCY_PROCEEDINGS DEBTS RESTRUCTURING CA
SE GROUP