Telecommunications for Multimedia

1
Telecommunications for Multimedia

• Summer Semester 2006
• G. Menegaz
• menegaz_at_dii.unisi.it

2
Prologue
Can you believe your eyes?
3
(No Transcript)
4
The importance of semantics
5
Scale
6
Scale
7
Scale
8
Context
9
Can you believe your eyes?
10
Course overview

• Goal
• The course is about the state-of-the-art image
  processing tools for the multimedia
• Analyses the exploitability of human vision
  models in such a framework (still images and
  video)
• Approach
• Acquire the fundamentals of signal processing AND
  vision sciences and gather them into a unified
  approach

Signal processing (tools)
Vision sciences (methodology)
Exactness of engineering sciences (systematic,
reproducible) BUT non accounting for of human
perception
Modeling of perceptual issues that hold a strong
potential impact on applications BUT are usually
too much simplistic
Vision-based models
11
Understanding vision
A problem of reverse engineering!
12
Framework
human sensory system
brain
Perceptual units or perceptons
Vision Audition Olfaction Haptic Taste
Human perception of reality is the result of the
interplay of many processes of different nature
and occurs in a perceptual feature space The
features and mechanisms involved in the
projection of the world to the perceptual space
are mostly unknown
13
Telecommunications for Multimedia

• Schedule
• 42 hours
• Lessons and labs
• 4 hours/week lessons (Thu. 9.00-13.00)
• 2 hours/week lab. (Wed. 9.00-13.00)
• Exercices
• Structure
• mid-term exam
• final exam
• other dates early July (TBC)

14
Course structure

• Part 1 Basics
• Mathematical tools
• Review of the Fourier transform
• Wavelets and pyramids
• Discrete wavelet transform (DWT)
• Overcomplete bases
• Advanced bases (curvelets)
• Compression and Coding
• Entropy coding
• State-of-the-art coding systems for still images
  and video
• Coding standards for images and video
• JPEG2000, MPEG4
• Image and video quality
• Metrics for perception-driven quality assessment

• Part 2 Applications Advanced Issues
• The human visual system
• Basics
• Color vision
• Color
• Colorimetry
• Color naming

15
Telecommunications for Multimedia

• Good news
• It is fun!
• Get in touch with the state-of-the-art technology
• Convince yourself that the time spent on
  mathsstats was not wasted
• Learn how to map theories into applications
• Acquiring the tools to contribute to the field

• Bad news
• Some theoretical background is unavoidable
• Mathematics
• Fourier transform
• Linear operators
• Digital filters
• Wavelet transform
• (some) Information theory
• Statistical data analysis
• Psychophysics

16
Framework
Digital image
Natural scene
capture sampling quantization color space
filtering transforms coding ....
Is this good quality
What is the best I can get over my phone line?
Network
Image Processing System
How can I protect my data?
How much will it cost?
Image rendering
17
Main Issues

• Broadcasting ? High information carrying capacity
• Efficient data representation
• Projection into suitable (perception based?)
  spaces
• Color processing
• Efficient encoding
• Reduction of redundancy
• Classical information theoretical principles
  (entropy based)
• Novel approaches based on visual perception
  (perception based)
• Standardization
• Openness
• Ability to adapt to new technologies
• Flexibility
• Ability to interact with different media
• JPEG2000, MPEG4, MPEG7

18
Main Issues

• Quality of Service
• Objective measure of the quality of service
• Determination of the cost of the service as a
  function of the features of the media available
  to the user
• Perception based metrics for the automatic
  assessment of the quality of images and videos
• Entails the investigation and modeling of the
  Human Visual System as well as the measure of the
  perceived quality of the signal by the design of
  ad-hoc subjective tests

19
Capture devices

• Optics
• lenses, diaphrams
• Analog cameraA/D converter
• Digital cameras
• CCDs (Charge Coupled Devices)
• CMOS technology

• Features
• Size and number of photosites
• Gain
• Noise
• Transfer function of the optical filter

Matrices of photo sensors collecting photons of
given wavelength
20
Basics graylevel images
Images ? Matrices of numbers Image processing ?
Operations among numbers bit depth ? number of
bits/pixel N bit/pixel ? 2N-1 shades of gray
21
Sampling
2D spatial domain

• Sampling in p-dimensions
• Nyquist theorem

Tsx
Tsy
2D Fourier domain
?y
?ymax
? x
?xmax
22
Spatial aliasing
23
Quantization

• A/D conversion ? quantization

f?L2(?)
discrete function f ?L2(?)
Quantizer
uniform
perceptual
fqQf
fqQf
rk
f
f
tk tk1
24
Quantization
Signal before (blue) and after quantization (red)
Q
25
Quantization

• Distortion measure
• The distortion is measured as the expectation of
  the mean square error difference between the
  original and quantized signals.
• Towards Image quality...
• Even though this is a very natural way for the
  quantification of the quantization artifacts, it
  is not representative of the visual annoyance due
  to such artifacts.

26
Quantization
original
27
Color images
C1
C2
C3

• Each colored pixel corresponds to a vector of
  three values C1,C2,C3
• The characteristics of the components depend on
  the chosen colorspace (RGB, YUV, CIELab,..)

28
The physical perspective
29
The perceptual perspective
30
Color
31
Color

• Human vision
• Color encoding (receptoral level)
• Color perception (post-receptoral level)

• Physics
• Spectral properties of radiation
• Physical properties of materials

Color categorization and naming (understanding
colors)
Color vision (Seeing colors)
Colorimetry (Measuring colors)
Models
32
Why do we care about color?

• Chromatic adaptation transforms

33
Why do we care about color?

• Gamut mapping

34
Mathematical tools
35
Signals as functions

• Continuous functions of real independent
  variables
• 1D ff(x)
• 2D ff(x,y) x,y
• Real world signals (audio, ECG, images)
• Real valued functions of discrete variables
• 1D ffk
• 2D ffi,j
• Sampled signals
• Discrete functions of discrete variables
• 1D fdfdk
• 2D fdfdi,j
• Sampled and quantized signals

36
Images as functions

• Gray scale images 2D functions
• Domain of the functions set of (x,y) values for
  which f(x,y) is defined 2D lattice i,j
  defining the pixel locations
• Set of values taken by the function gray levels
• Digital images can be seen as functions defined
  over a discrete domain i,j 0ltiltI, 0ltjltJ
• I,J number of rows (columns) of the matrix
  corresponding to the image
• ffi,j gray level in position i,j

37
Example 1 ? function
38
Example 2 Gaussian
Continuous function
Digital function
39
Example 3 Natural image
40
Example 3 Natural image
41
The Fourier kingdom

• Frequency domain characterization of signals

Signal domain
Transformed domain
42
The Fourier kingdom
Gaussian function
43
The Fourier kingdom
rect function
44
The Fourier kingdom
45
The Fourier kingdom
46
Wavelet domain
47
What wavelets can do?
48
WaveletsPyramids
Basis functions are square waves!
49
WaveletsPyramids
50
Fourier vs Wavelets

• Fourier
• Basis functions are sinusoids
• More in general, complex exponentials
• Switching from signal domain t to frequency
  domain f
• Either spatial or temporal
• Good localization either in time or in frequency
• Transformed domain Information on the sharpness
  of the transient but not on its position
• Good for stationary signals but unsuitable for
  transient phenomena

• Wavelets
• Different families of basis functions are
  possible
• Haar, Daubechies, biorthogonal
• Switching from the signal domain to a
  multiresolution representation
• Good localization in time and frequency
• Information on both the sharpenness of the
  transient and the point where it happens
• Good for any type of signal

51
WaveletsPyramids
N
N
N
N
52
WaveletsPyramids
53
WaveletsPyramids
54
WaveletsPyramids
55
WaveletsFilterbanks
56
WaveletsFilterbanks
H
?2
H
?2
G
?2
H
?2
?2
G
G
?2
Very efficient implementation by recursive
filtering
57
Emerging wavelet families

• Contourlets, curvelets, ridgelets....
• More suitable for representing line
  discontinuities (edges)
• Could be representative of the receptive fields
  of complex cells
• Used to model second order channels for texture
  perception
• Suitable for shape representation and model-based
  pattern recognition

c2-j/2
2-j
2-j
2-j
58
Coding
Desirable features Flexibility User-data
interactivity Openness Easy to use User
interactivity Security
Standardization
59
Image Quality

• The human visual system and beyond

60
What is image quality?
Same PSNR, different perceived quality
61
Coding specific artifacts
JPEG
JPEG2000
62
Coding specific artifacts
MPEG4
63
Colorfulness
64
Sharpness
65
Modeling the visual system

• Vision is a very complex process which concerns
  the brain
• It cannot be explained only in terms of physical
  quantities because the stimuli trigger cognitive
  processes
• Models for the visual system should account for
  both low-level and high-level processes

Stimulus encoding (early vision)
Stimulus interpretation (cognitive processes,
high level)
image
score
Simple cells, receptive fields, multi-resolution
Low-level visual attributes
66
How to define Image Quality?

• No golden rule!
• The perceived quality of an image depends on many
  variables which are not directly measurable
• Not all the measurable distortions are visible
  (masking) and viceversa, there are some induced
  perceptual distortions which do not correspond to
  physical distortions (illusions)
• Cognitive processes also play a role, making the
  judgment depending on content of the image, the
  background of the subject, the task..
• Many definitions are possible
• Image quality is the integrate set of perceptions
  of the overall degree of excellence of an image
  Engeldrum, Psychometric scaling, 2000

67
Image Quality
Validation
Subjective tests
Subjective evaluation
Vision model
Quality metric
Psychometric scaling
Image
Perceptual features
Objective evaluation