Chapter 1: Overview

1
Chapter 1 Overview
2
What is multimedia?

• Multimedia too many cables?
• Multimedia combines audio and visual materials to
  provide computerized interaction of text, sound,
  graphics, images, animation video to enhance
  communication and to enrich its presentation.
• Multimedia systems handle at least one type of
  continuous media as well as static media.

3
Multimedia applications

• Videodisc applications
• A DVD can hold 2-8 hrs of high-quality video.
• Electronic games
• Web browsers
• Multimedia presentation systems
• An engine that displays, synchronizes, provides
  interaction with, and generally manipulates
  multimedia material (e.g., Macromedia Flash).

4
Multimedia applications

• Multimedia mail systems.
• Teleconferencing
• A computer equipped with microphone, speakers,
  and a video camera, and placed on a multimedia
  network, can establish audio and video
  connections between other, similarly equipped,
  machines.
• Multi-user tools, such as group editors. A group
  editor allows conference participants to share
  documents, and to edit the documents
  simultaneously.

5
Multimedia applications

• Multimedia services
• Interactive TV
• Interactive shopping
• Education
• Medical services (telemedicine)
• Video-on-demand

6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
Analog vs. Digital

• Two ways to process information analog and
  digital
• Examples
• Vinyl LP vs. CD
• conventional radio vs. web-radio
• slide-rule vs. digital calculator

13
Analog recording

• Sound is caused by a variation of air pressure
  a sound wave

14
Analog recording

• An audio tape records an analog signal that
  represents the sound wave.
• An analog signal is continuous there are an
  infinite number of points (values) and each value
  has infinite precision.

15
Digital recording

• To record an analog signal by a computer (which
  has a finite amount of storage), we need to
  perform analog-to-digital conversion (ADC)
• Sampling
• Quantization
• Coding

16
ADC

• Consider an analog signal

time
17
Sampling

• convert continuous signal into discrete values by
  taking samples

18
Sampling

• The original signal can be approximated by
  interpolation using the sampled values.

19
Sampling

• More samples ? more accurate approximation

20
Quantization

• A computer cannot record the values with infinite
  precision. A value has to be quantized.

a quantization Level/interval
21
Quantization

• A computer cannot record the values with infinite
  precision. A value has to be quantized.

a coarser quantization
22
Quantization

• Each sample value is replaced by the nominal
  value of its quantization level.

a sample value
a nominal value
23
Quantization

• Each sample value is replaced by the nominal
  value of its quantization level.

24
Quantization

• The difference between a sample value and its
  quantization value is called the quantization
  error.

quantization error
25
Quantization

• More quantization levels gives a more accurate
  representation.

26
Quantization

• More quantization levels gives a more accurate
  representation.

27
Quantization

• More quantization levels gives a more accurate
  representation.

28
Quantization

• Note that quantization does not necessarily have
  to be uniform or linear.

wider quantization level
narrower quantization level
29
Coding

• assign a codeword to each quantization level.

0101
0100
0011
0010
0001
0000
1000
1001
1010
1011
1100
1101
30
Coding

• An analog signal could then be represented
  digitally by a string of 0s and 1s
• 0010 0011 0011 0000 1010 1011

31
Why digital?

• ADC introduces error
• The waveform constructed by interpolating samples
  is not a verbatim copy of the original.
• Quantization introduces quantization error.
• so
• Why digital representation?
• What should be the sampling frequency?
• How many quantization levels (or bits/sample)
  shall we use?
• Example for CD-DA (music CD)
• sampling frequency 44,100 samples per second
• 16 bits per sample (i.e., 65,536 quantization
  levels)

32
Why digital?

• easy integration sharing of resources (storage,
  transmission network).
• can be processed by a computer, e.g., encryption,
  watermarking, compression, digital effects, etc.
• more reliable storage and transmission (through
  error-correction and replication).
• can be used and copied many times without losing
  quality.

33
Why digital?
analog signal transmission
channel
channel
digital signal transmission
channel
reconstruct
34
Media types

• Non-temporal (Discrete) do not have a time
  dimension, and their contents meanings do not
  depend on the presentation time.
• Text
• Image
• Graphics

35
Media types

• Temporal (Continuous, Isochronous) have a time
  dimension. They convey meanings only if
  displayed at a specific rate.
• Video
• Digital Audio
• Music

36
Text

• not visually exciting
• conveys essential and precise information
• text representation e.g., ASCII, BIG5, GB
• storage friendly
• sometimes, certain information is too abstract to
  be captured by words

37
Digital images
To digitize a photo, we again perform ADC
38
Digital images
sampling
39
Digital images
pick a sample color from each box to record
Each sample is called a pixel ? picture element
40
Digital images
finer sampling gives a clearer image
41
Digital images
finer sampling yet
42
Digital images

• The resolution of a digital camera refers to the
  number of samples that the camera takes.
• Some typical numbers

You needat least 150ppi (pixel per inch)for
printing
For better quality,make it 300 ppi
43
Digital cameras

• A Charge-coupled-device (CCD) camera has a
  2-dimensional array of photosites that convert
  the amount of light intensities into equivalent
  electrical charges.
• Note there are also less-expensive digital
  cameras that use CMOS instead of CCD. CMOS
  sensors are usually
• More susceptible to noise
• Less light-sensitive
• Consume less power
• Both technologies have advanced in recent years
  that they are comparable against each other now.

44
Capturing color

• There are a number of ways that a CCD camera
  captures color
• Split a light beam into its red, green and blue
  components and use 3 separate CCD grids.

45
Capturing color

• There are a number of ways that a CCD camera
  captures color
• A more economical way is to use 1 CCD panel and
  cover each photosite with a different filter.

46
Capturing color
The amount of charge at each photosite tells the
intensity of a color component of light detected
at that point.
R
G
B
B
R
G
G
B
R
47
Capturing color
R
G
B
B
R
G
G
B
R
red
48
Capturing color
R
G
B
B
R
G
G
B
R
green
49
Capturing color
R
G
B
B
R
G
G
B
R
blue
50
Capturing color
Bayer filter
51
Digital images

• two-dimensional arrays of pixels (picture
  elements) of varying color and intensity.
• color model how to specify the color of a pixel
  (coding)?
• RGB colors can be represented by numeric
  triplets specifying red (R), green (G), and blue
  (B) intensities.
• YCRCB (for digital images and video)
• CMY(K) (for printing)
• Cyan (no red)
• Magenta (no green)
• Yellow (no blue)
• K black ink

Additivemixing
Colordifferences
Subtractivemixing
52
Digital images

• Compression
• A page-sized 24-bit color image with 300 pixels
  per inch (PPI) takes up about 20Mbytes.
• Lossless and lossy compression.
• Many different standards JPEG, GIF (Graphic
  Interchange Format), ...Example run-length
  encoding.
• Image transformation and processing.
• E.g., morphing (one image transformed into
  another).

53
Graphics

• Graphics data are represented by a geometric
  model a set of graphics operations.
• Geometric model
• A collection of 2D/3D geometric
  primitives(lines, circles, polygons, curves).
• Transformations rotation, translation, scaling.

54
Graphics
55
3D models
Wireframe
taken from Planet Architecture
56
Graphics

• Graphics operations are applied to make the scene
  more realistic
• Coloring
• Shading
• How a surface reflect light
• Lighting
• Ambient light (from all directions)
• Point light (inverse square law)
• Spot light (a cone-shaped volume)
• Viewing
• Where the camera is

57
Graphics

• Texture mapping
• applying an image onto a surface
• Rendering
• converts a model shading, lighting viewing
  ... into an image.
• Animation
• Eye-catching
• Good for demonstration

58
Graphics
Taken from SIGGRAPH 97
59
Graphics
60
Graphics
61
Graphics
taken from Planet Architecture
62
Analog video

• A sequence of images called frames, persistence
  of vision.
• Originally, motion pictures are shown at an
  (insufficient) frame rate of 16fps.
• It was found that for smooth motion, 24fps is
  needed.
• Attributes frame rate, resolution, aspect ratio,
  interlacing, refresh rate.

63
Analog video

• Formats.
• Examples
• NTSC (National Television Systems Committee)
• PAL (Phase Alternation Line).

format frame rate scan lines
aspect ratioNTSC 30
525 43PAL 25
625 43HDTV(US) 30
1125 169HDTV(EURO) 25
1250 169 MUSE(Japan) 30 1125
169
picture width to picture height
64
Analog video

• Theoretically, most color can be produced by
  mixing 3 primary colors (red, green, blue). An
  analog video camera produces 3 distinct
  continuous signals, one for each color component.

scan line signal
horizontal blanking
black
white

black
vertical blanking
white
65
Analog video

• Theoretically, most color can be produced by
  mixing 3 primary colors (red, green, blue). An
  analog video camera produces 3 distinct
  continuous signals, one for each color component.

scan line signal
horizontal blanking
black
white

black
vertical blanking
white
66
Analog video

• Theoretically, most color can be produced by
  mixing 3 primary colors (red, green, blue). An
  analog video camera produces 3 distinct
  continuous signals, one for each color component.

scan line signal
horizontal blanking
black
white

black
vertical blanking
white
67
Analog video

• Theoretically, most color can be produced by
  mixing 3 primary colors (red, green, blue). An
  analog video camera produces 3 distinct
  continuous signals, one for each color component.

scan line signal
horizontal blanking
black
white

black
vertical blanking
white
68
Analog video

• A video signal is applied to a TV (a CRT) to
  control the power of an electron beam that
  strikes the phosphors on the inside of the CRT
  surface.
• With the 625/50 system (PAL), for example, a
  frame is displayed every 1/25 seconds.
• Unfortunately, the phosphors do not stay lit that
  long ? flickering
• To prevent flicker, the picture has to be
  refreshed at least 50 times per second.
• Interlacing a frame is divided into 2 fields
  odd-lines and even-lines, a field is displayed
  every 1/50 seconds.

69
Analog video
frame 1
progressive
frame 2
time
1/50
1/25
0
interlaced
70
Analog video

• Luminance/chrominance principle the three
  primary colors can be converted into 2 parts
• Luminance information on the lightness of the
  image.
• Chrominance information on the color of the
  image.
• Because the human eye is not very sensitive to
  color information, the bandwidth of the color
  component is reduced before transmission.

71
Analog video

• Video storage
• video tape
• e.g., VHS tape
• about 240 scan lines resolution
• problem loss of quality when copying and
  repeated playback due to stretches and magnetic
  material wearing off.

72
Digital video

• A video can also be represented by a sequence of
  digital images.
• broadcast quality video (uncompressed)
• 1 sec ? 20 MB
• For lesser quality, and a good compression
  technique, it is possible to achieve
• 1 sec 1.2 Mb ?
• transfer rate of (single-speed) CD-ROM ?
• VCD
• Data rate of a DVD movie is about 2GB/hr.
• Compression lossless and lossy.

B byte b bit
73
Digital video

• For lossy compression, we can achieve a 501 or
  higher compression ratio.
• MPEG The Moving Pictures Expert Group
• MPEG-1 1.5Mbps VHS quality video
• MPEG-2 4-10 Mbps (digital TV)
• MPEG-4 a system which allows a scene structure
  to be composed of multiple different objects
  (video, audio, natural, synthetic)
• MPEG-7 multimedia information retrieval

74
Digital video formats

• CCIR 601 (422 chromatic subsampling)
• for video exchange
• 525/60 system 720 ? 480 (take 720 samples from
  each active scan line)
• 625/50 system 720 ? 576
• 420
• for video broadcast

75
Digital video formats

• SIF (Standard Input Format)
• for storage
• 525/60 system 352 ? 240
• 625/50 system 352 ? 288
• 411 chromatic subsampling
• CIF (Common Interchange Format)
• for video conferencing
• 352 ? 288 at 30 fps using 411 chromatic
  subsampling

76
Digital audio

• Digital audio representation
• produced by sampling a continuous signal
  generated by a sound source
• ADC (again)

77
Digital audio

• Sampling frequency
• Nyquists Theorem
• Sampling rate ? 2 ? highest signal frequency
• Human ear is sensitive to frequencies of up to
  about 20kHz (c.f., rat 1k 10k Hz cat 100
  60k Hz)
• Sampling frequency ? 40kHz
• audio-CD ? 44.1kHz, 16 bits per sample (? 96dB
  max. SNR)
• DVD audio ? 196kHz (max), 24 bits per sample
  (max)
• (Q what is the throughput requirement?)
• For stereo, 2 channels.

78
Digital audio

• Storage
• An hour of high quality stereo digital audio
  requires 500MBytes of storage.
• A CD-ROM can store about 650MBytes of data.
• A CD-DA can store about 74 minutes of audio data.

79
Digital audio

• Why 16 bits per sample?
• Let b number of bits per sample, q
  quantization step, Q number of quantization
  levels. We have Q 2b.
• Max. signal amplitude (q2b)/2
• Max. quantization noise q/2.
• SNR 20log10(q2b/q) dB ? 6b dB.
• Threshold-of-pain / audibility-threshold 100 to
  120 dB.
• Quantization noise is inaudible ? SNR is at least
  100.
• 6b 100 ? b 16.7.

80
Digital audio
(q2b)/2
q
2b levels
81
Music

• MIDI -- Musical Instrument Digital Interface
• Digital musical instruments send MIDI messages to
  a sequencer.
• The sequencer composes the music according to the
  messages received.
• The sequencer/synthesizer has a palette of
  sounds for each type of instrument

MIDI sequencer
82
Challenges

• Multimedia stresses all components of a computer
  system (data volume time constraints)
• CPU processing power
• fast speed for data capturing, CODEC, data
  enhancement
• large amounts of data being processed in
  real-time
• Storage and Memory
• high capacity, fast access time, high transfer
  rates
• System architecture
• high bus bandwidth, efficient I/O

83
Challenges

• Software
• tools for retrieval and data management of
  continuous media data
• Operating systems
• support for new data types, real-time scheduling,
  multimedia file systems, time-critical
  synchronization
• Networks
• high bandwidth, low latency, low jitter

84
Research areas

• 1. fast processors
• 2. high-speed networks
• 3. large capacity storage devices
• 4. video audio compression algorithms
• 5. graphics systems
• 6. human-computer interfaces

• 7. real-time operating systems
• 8. information storage and retrieval
• 9. hypertext hypermedia
• 10. languages for scripting
• 11. parallel processing methods

85
Compression

• MM systems require data compression for 3
  reasons
• the large storage requirements of MM data.
• relatively slow storage devices that cannot play
  MM data in real time
• network bandwidth that does not allow real-time
  video data transmission

86
Some compression standards
87
Multimedia networking

• Many multimedia applications, such as video mail,
  video conferencing, and video-on-demand, require
  the support of a high performance network system.
• In these applications, the multimedia objects are
  stored at a server and played back at the
  clients sites.
• Remote retrieval of multimedia objects has
  stringent time constraints.

88
Multimedia networking

• Delay the amount of time it takes to transmit a
  data unit (e.g., a video frame) from a sender to
  a receiver.
• Jitter delay variation.

delay
jitter
source
destination
89
Multimedia networking
90
Multimedia information retrieval

• To retrieve a text document from the Web, we use
  keyword search via Alta Vista, for example.
• To retrieve a record from a relational database,
  such as Oracle, we use an SQL statement.
• How shall we formulate a query to retrieve
  pictures?
• What about audio? How do we describe a sound? How
  do we describe a song?

91
Multimedia software tools

• Music sequencing
• Cakewalk
• supports general MIDI
• provides several editing views (staff, piano
  roll, event list) and Virtual Piano

92
Multimedia software tools

• Image and video editing
• Adobe Photoshop
• allows layering of images, graphics and text
• includes many graphics drawing and painting tools
• sophisticated lighting effects, various image
  processing filters
• Adobe Premiere
• provides many video and audio tracks,
  superimposition and virtual clips
• supports various transitions, filters and motions
  for video clips
• a reasonable desktop video editing tool

93
Multimedia software tools

• Multimedia authoring
• Microsoft Power Point
• building slide show sequence
• slides include objects of different media, e.g.,
  sound and video
• limited animation ability
• Macromedia Director, Flash
• Movie metaphor (cast of bitmapped sprites,
  scripts, music, sound, and palettes)
• Lingo script language allows more control of the
  presentation sequence. Control of devices, e.g.,
  VCRs and disk players.
• ready for building interactivities using buttons,
  etc.