Internet Searching and Browsing in a Multilingual World

1
Internet Searching and Browsing in a Multilingual
World

• An Experiment on the Chinese Business
  Intelligence Portal

Acknowledgment NSF/NIJ Grant
2
Outline

• Motivation
• The Chinese Business Intelligence Portal
• System Description
• Results of Usability Study
• Conclusions

3
Introduction
4
Motivation

• As the Internet grows in popularity worldwide,
  more users want to access Web content in their
  native languages
• The majority of the total global online
  population (63.5) lives in non-English-speaking
  areas (Global-Reach, 2002)
• Such population is estimated to grow rapidly,
  much faster than English-speaking population
• However, existing search engines may not serve
  their needs, because most technologies have been
  developed for English-speaking users

5
This Presentation

• The following slides present our efforts in
  creating and evaluating intelligent Web portals
  that address the above needs
• The Chinese business information serves as our
  research testbed
• Through the studies, we aim to achieve better
  understanding of human interaction and analysis
  with automated systems developed for Internet
  searching and browsing in a multilingual world

6
The Chinese Business Intelligence Portal
(CBizPort)
7
CBizPort

• The Chinese Business Intelligence Portal
  (CBizPort)
• Two versions of user interface Simplified
  Chinese and Traditional Chinese
• URLs
• Introduction http//ai.bpa.arizona.edu/go/dl/cbiz
  port.html
• Portal http//ai17.bpa.arizona.edu8080/big5biz/i
  ndex.html
• Each version has the same user interface and
  provides the same functions
• Encoding conversion
• Meta searching major Chinese information sources
• Summarization, Categorization
• Providing links to major Chinese business Web
  resources
• The following slides show the system architecture
  and screen shots of CBizPort

8
(No Transcript)
9
Keywords
10
Search Page
Result Page
Web pages grouped by key phrases extracted by
mutual information algorithm (non-exclusive
categorization)
Categorizer
Summarizer
11
Evaluation of CBizPort

• Objectives
• To evaluate the performance of summarizer as a
  preview function and categorizer as an overview
  function
• To compare CBizPort with regional Chinese search
  engines to study its effectiveness and usability
• To evaluate, in comparison with existing regional
  Chinese search engines, the information quality
  obtained from CBizPort and its capability of
  searching for cross-regional business information

12
Experimental Design

• Searching and browsing were studied
• Scenario-based, culturally oriented tasks, e.g.,
• A search task (4 min) Find two cities in
  mainland China that Motorola has set up its
  manufacturing operations
• A browse task (5 min) Describe, in a number of
  distinct themes, the economic impacts of removing
  trade barriers between mainland China and Taiwan
  towards Hong Kong
• Theme identification method (Chen et al., 2001)
• Pilot test 3 subjects used up all the time in
  most tasks ? only focused on effectiveness but
  not efficiency

13
S search task B browse task O Basic
searching (with neither summarizer nor analyzer)
M Basic searching with summarizer only A
Basic searching with categorizer only G
General searching and browsing C
Cross-regional searching and browsing same
number signals the same question across different
regions (Random assignment of tasks is used for
different settings)
14
Comparisons
15
Subjects

• 30 subjects, 10 from each region, were recruited
• Rationale equal influence of regional impacts
• Each subject used CBizPort and another search
  tool according to his/her origin

16
Experts

• Three experts, one from each region, were
  recruited to provide answers to all browse tasks
• First, the experts identify the set of relevant
  answers (organized into themes) to a browse task
• Then, they modified the answers by adding some of
  subjects responses that they judged as relevant
• The above two steps are repeated for all the
  other browse tasks

Bla bla bla
17
Hypotheses

• Three sets of hypotheses were tested
• CBizPorts Enhanced Analysis Capabilities
• Searching and browsing
• With or without summarizer/categorizer
• SE Performance Comparison
• Searching and browsing capabilities
• Individual settings and combination
• Users Subjective Evaluation
• Information quality
• cross-regional searching capability
• overall satisfaction
• Auxiliary hypotheses Performance of the three
  regions are not significantly different

We tried to mimic a situation that each subject
was allowed to use both CBizPort and benchmark
search engine together to solve the same problem
18
Benchmark SE
19
(No Transcript)
20
Performance Measures

• Accuracy Percentage of correct answers
• Precision number of correct themes identified
  by users / total number of themes identified by
  users
• Recall number of correct themes identified by
  users / total number of themes identified by an
  expert
• F value 2RecallPrecision / (Precision
  Recall)
• Information quality accessibility,
  appropriateness of amount, believability,
  completeness, , etc. (Wang Strong, 2002)
• Subjective evaluation cross-regional searching
  capability, overall satisfaction, protocol
  analysis, post-hoc test (to study whether the
  three SEs yield significantly different results)

21
Accuracy of search tasks
22
Precision of browse tasks
23
Recall of browse tasks
24
F value of browse tasks
25
Information Quality
26
Users Subjective Evaluation
27
Subjects Verbal Comments

• Subjects liked summarizer and categorizer
• Subj.15 good performance in summarization
  and categorization, more focused results can be
  found 26 very handy 6 useful tools
  to enhance the searching ability (11 subjects)
• CBizPort provides a wide coverage and variety of
  searching options
• Subj.2 Yahoo Search Engine is more limited
  when search certain term in a specific region
  While CBizport can fulfill what Yahoo couldnt
  do. 4 more search engines to choose from
  (4 subjects)

28
Subjects Verbal Comments (2)

• Subjects are familiar with benchmark SEs
• Subj27 I am familiar with the format of
  Openfind. So that's the reason that I am more
  satisfied with it than CBizPort. (4 subjects)
• Benchmark SEs are not good at cross-regional
  information searching
• Subj15 Sina gives many results but they are
  not focused, and is poor at searching HK and
  Taiwan results 5 provide more accurate
  regional searching
• CBizPort is user friendly but slow
• 3 Yahoo not as precise as CBizPort 28
  easier to search (7 subjects) slow (3
  subjects)

29
Conclusions

• CBizPorts summarizer and categorizer provide
  helpful analysis capabilities for users search
  and browse tasks
• CBizPorts searching and browsing performance is
  comparable to that of regional Chinese search
  engines
• CBizPort can significantly augment the searching
  and browsing ability of regional Chinese search
  engines, thus improving human integration of
  regional information and analysis
• Information quality, cross-regional searching
  capability and overall satisfaction of CBizPort
  are comparable to those of regional Chinese
  search engines
• CBizPort is better than regional Chinese search
  engines in terms of analysis functions,
  cross-regional searching capabilities and
  user-friendliness