Mobile Information Access with Spoken Query Answering

1
Mobile Information Access with Spoken Query
Answering
POSH (Perceived Quality Of Service in
Heterogeneous Environments) Project

• Computer Science and Engineeering Department
• Aalborg University Esbjerg
• Center of Teleinfrastructure (CTIF)

2
POSH Project - the two tracks

• Track A
• Simulation of heterogeneous environments to asses
  the perceived QOS in mobil comunication systems

speech text
videospeech text
Wireless Service Provider
3
POSH Project - the two tracks

• Track A

Users location
Application
TCP/IP
Simulation
Network
Data
Physical
4
POSH Project - the two tracks

• Track B
• Mobil information access with query aswering
  capabilities
• Well focus on the Track B project on this talk
• Both tracks were demostrated in the First
  International Wireless Summit 2005 17-22
  September in Aalborg Kultur- og Kongrescenter

5
POSH Project Track-B Participants

• AAUE - Computer Science Department - Wireless
  Computing and Security (WCS)
• Dr. Henrik Legind Larsen
• Dr. Daniel Ortiz-Arroyo
• Research assistant Dan Majgaard
• Master students
• AAU Speech and Multimedia Communication group

6
POSH-Track B Motivation

• Increasing demand of services in mobile devices
• Small mobil devices with limited capabilities
• Lack computing power and resources to run
  sophisticated applications
• Small keypad
• Screen size limitations

7
POSH-Track B Phase 0 Goals

• Proof of concept on Mobile Information Access
  with Spoken Query Answering capabilities
• Service accessibility on mobile devices with
  limited resources
• Integration of state of art technologies
• Collaborative research among groups inside
  AAU-CTIF from Esbjerg/Aalborg

8
POSH-Track B Functionality
www
Speech Recognition Server
Who won the match?
Information Retrieval Server
10011
Who won..
Who won..
Esbjerg won
9
POSH -Track B

• Sophisticated services on resource-limited
  wireless devices
• Speech Recognition
• Intelligent Information Retrieval
• Natural Language Processing

10
POSH-Track B Features

• Spoken Query Processing
• Danish language
• Intelligent Information Retrieval
• Learning user needs/preferences
• Knowledge of a domain
• Football soccer (Danish league)
• Fuzzy logic
• Direct Answering
• Information extraction

11
POSH-Track B Features (cont.)

• Simple User Interface
• Single window
• Fully Distributed Architecture
• Distributed speech recognition
• Distributed search engine

12
Technologies

• 3-Tier client-server
• architecture - developed in multiple languages
• Thin client on PDA
• Server
• Server logic
• Information extraction
• Information retrieval engine
• Speech recognizer

Speech Recognition Server
Information Retrieval Server
PDA or mobile phone
13
Technologies
WWW
Other WWW Sources
14
Mobil devices

• Currently client implemented on PDA
• C .NET compact

Recording button
Search
Microfone
15
Systems Architecture
16
Systems Architecture UML Package Diagram
17
Example Class Diagram
18
Example UML Sequence Diagram of a Search
Java
C
19
Speech Recognition

• Distributed Speech Recogition
• Voice is filtered and compressed in the client
• Recognition performed in server
• Speech recognizer features
• Speaker independent
• Vocabulary - 709 terms (players, clubs etc)
• Tested with 260 sentences with 90 accuracy

20
Information Retrieval/Extraction

• Distributed Information Retrieval
• SOAP protocol
• Sockets used for testing debugging

21
Information Retrieval
DB
Danish football league Web sites
Text
Information Retrieval
Filter
RSS feed client
XML/HTML
22
Intelligent Information Retrieval
won/match/game/..
team/player/score/..

• Concept Clusters
• Fuzzy Logic
• Users Profile

index
Information retrieval
Selection
Result
DB
Users query
Users profile
23
Intelligent Information Retrieval

• Concept Clusters

Lexical chain links related lexical terms
Concept clusters For Yoga
24
Information Extraction
WWW
DB
ER model for football soccer
Sports web sites
Football Soccer HTML Tables
Information Extraction
Filter
Crawler
HTML Tables
Answer
Question recognizer
Users query
25
POSH -Track B Phase 1Challenges

• Moving from web searching to query-answering
  systems
• Searching/Retrieval of multimedia
• Audio indexing
• Video streaming

26
POSH -Track B Phase 1Challenges

• Supporting English
• Context/dialog management
• Extending/choosing a better knowledge domain
  representation
• Implementation on cell phones

27
POSH-track B Phase 1Challenges

• Scaling
• Supporting thousands of users
• Multiple knowledge domains
• Multiple servers
• Others
• Improving speech recognition
• Vocabulary
• Context and user profile to select N-best option
• Grammar
• Security

28
Thanks

• Questions or comments?