Information Reputation, Ratings, and Quality on the Web

1
Information Reputation, Ratings, and Quality on
the Web

• Nick Diakopoulos nad_at_cc.gatech.edu
• School of Interactive Computing

2
Reputation, Ratings, Recommender Systems

• Amazon.com
• Movielens.org (U Minnesota)
• Digg.com
• Slashdot.com
• Newstrust.net

3
Reputation, Ratings, Recommender Systems

• Datatypes numeric, text, tags
• Explicit / Implicit
• Manual / Automatic (content based)
• Anonymous? Blind review?
• Aggregation of Evaluations
• Timescale, type of content
• Incentives for honest contribution money,
  influence, fun, somatic?
• Impact of liars
• Transparency of rating system

• Paul Resnick and Hal Varian. Recommender Systems.
  CACM March 1997.
• Thomas Adler, Luca de Alfaro. A Content-Driven
  Reputation System for the Wikipedia. WWW 2007.
• Sonja Buchegger et al. Reputation Systems for
  Self-Organized Networks. IEEE Technology and
  Society. Spring 2008.

4
Properties and Limitations

• Temporal Information Dynamics
• Takes time, sparse ratings, early influencers
• Effect of visibility of rating, raters identity
• Polarization
• Information Cascades
• Non-disclosure

5
Information Quality and Reputation

• Computational Journalism
• Journalism as a discipline of verification,
  providing quality information to public
• Importance of reputation in computing credibility
  and trust

6
Information Quality Facets

• Accurate
• Free of mistakes, exact
• Reliable and Credible
• Consistent, dependable / trustworthy, expert
• Valid
• Justifiable, logically correct
• Techniques
• Corroboration / fact-checking / triangulation,
  falsifying evidence / sources, chains of evidence
• Other concepts from journalism balance, context,
  evidence, fairness, importance, sources, trust,
  style

7
Information Quality Facets

• Sources
• Documents name, author, link, publication, trust
• Entities name, expertise, trust
• Claims
• Claimant, belief, subjectivity, inaccuracy, bad
  assumptions
• Free text comments
• Tags omitted info, context needed, omitted
  information, stereotyping,

8
Information Quality in Video

• Survey need
• 89 reads comments, but 63 think comments are
  not valuable
• Need ability to filter comments
• Multimedia transcript video annotations
• Much more complex interface
• Easy to get cluttered
• Granularity

9
Videolyzer

• Video Overview

10
Videolyzer

• Moving towards a ratings SYSTEM
• Dynamics of ratings between people
• Micro-Vetting
• Each rating / comment on the video sent to 1-3
  other people in the system for evaluation of ITS
  quality
• Chains of ratings (triangulation / corroboration)
• Rounds of ratings in a game like fashion
• Scoring?