Xilinx XC4000 FPGA devices

1
Introduction and Overview Image and Video
Processing
By Theerayod Wiangtong, PhD., DIC Mahanakorn
University of Technology
2

• Human Visual Perception

3
Eye Anatomy
4
Eye Vs. Camera
5
The Human Visual System (HVS)
6
Human Perception of Color

• Retina contains two classes of receptors
• Cones (6-7 millions)
• Located in the fovea

• Each one connected to its own nerve end
• Day vision, can perceive color tone
• Red, Green, and Blue cones.
• Different cones have different freq. responses
• Rods (75-100 millions)
• Several rods are connected to a single nerve
• Night vision, perceive brightness only

7
Range of Perception

• Light consists of an electromagnetic wave with
  wavelength in the range 380-780 nm.

8
Light Sources
Illuminating Source
Reflecting Source

• Emits light (e.g. sun, bulb, TV)
• Perceived color depends on spectral contents of
  the emitted light
• Follows the additive rule (the perceived color
  of several mixed illuminating light sources
  depends on the sum of the spectra of the light
  sources)

• Reflects incoming light (e.g. all
  non-illuminating objects)
• Perceived color depends on spectral contents of
  the reflected light
• Follows the subtractive rule (the perceived
  color of several mixed reflective light sources
  depends on the remaining (unabsorbed) wavelengths)

9
Trichromatic color mixing theory

• Trichromatic color mixing theory
• Any color can be obtained by mixing three primary
  colors with a right proportion
• Primary colors for illuminating sources
• Red, Green, Blue (RGB)
• Color monitor works by exciting red, green, blue
  phosphors using separate electronic guns
• Primary colors for reflecting sources (also known
  as secondary colors)
• Cyan, Magenta, Yellow (CMY)
• Color printer works by using cyan, magenta,
  yellow.

10
Primary Vs Secondary Colors

• Primary Colors

Secondary Colors
11
HSV Visual Illusion
12
HSV Visual Illusion
Which lines are straight?
13
Motion Illusion
14
Motion Perception is Tricky
15
Where is Human Head
16
Test
17
(No Transcript)
18
Image Processing
)
19
Digital Image Formation
20
Matrix Representation
Divide into 8x8 blocks
21
Image Resolution
22
Image Resolution
23
Bitplanes
Bitplane 7
Bitplane 6
Original 8bits/pixel
one 8-bit byte
Bitplane 7
Bitplane 5
Bitplane 4
Bitplane 0
24
Bitplanes
Bitplane 3
Bitplane 2
Original 8bits/pixel
one 8-bit byte
Bitplane 7
Bitplane 1
Bitplane 0
Bitplane 0
25
Dimensionality of Digital Images

• Images and videos are multi-dimensional ( 2
  dimensions) signals.

26
Color
27
Color Space Model
YDbDr
CMYK
RGB
Lab
YCrCb
YIQ
YUV
28
Color RGB Cube
29
Conversion between RGB and YIQ in NTSC

• RGB -gt YIQ

Y 0.299 R 0.587 G 0.114 B I 0.596 R
-0.275 G -0.321 B Q 0.212 R -0.523 G 0.311 B

• YIQ -gt RGB

R 1.0 Y 0.956 I 0.620 Q, G 1.0 Y - 0.272
I -0.647 Q, B 1.0 Y -1.108 I 1.700 Q.
30
YIQ Component
Y image
Color Image
Q image (green-purple)
I image (orange-cyan)
31
YUV in PAL
32
YDbDr in SECAM
33
YCbCr in Digital Image/Video
Analog video
Digital video
34
Different Color TV Systems
35
Applications
Segmentation
Manipulation
Tracking
Digital Image Processing
Recognition
Coding
Detection
36
Image Processing Image Enhancement
Enhance
37
Image Processing Image De-noising
Denoise
38
Image Processing Image Deblurring
Deblur
39
Image Processing Image Inpainting




40
Image Analysis Edge Detection
41
Image Analysis Face Detection
42
Image Analysis Image Segmentation
43
Image Analysis Image Matching
44
Image Coding Image Compression
original image 262144 Bytes
compressed bitstream 00111000001001101 (2428
Bytes)
image encoder
image decoder
compression ratio (CR) 1081
45

• Video Processing

46
Digital Video Processing

• Digital
• Use a digital data format
• Video
• a sequence of images along the temporal axis
• We will talk this more next
• Processing
• A running software program or computing operation
  
• Driven by the real-world applications (e.g.,
  compression, filtering, retrieval)

47
Overview of Video Processing
Video Manipulation
Video Display
Video Compression
Video Database
Video Acquisition
Video Transmission
Computer Graphics
Video Analysis
Computer Vision
48
Video Acquisition
Video camera
VHS digitization
computer-generated
49
Acquisition-related Problems

• Video camera
• What if camera is not kept still? (e.g.,
  jittering in carphone sequence)
• Why is it difficult to improve the spatial
  resolution of video cameras?
• VHS digitization
• What if VHS contains some scratches?
• How to handle interlaced video?
• Computer-generated
• How is this type of video different? Shouldnt we
  have a separate coding algorithm for this type of
  video?

50
Interlaced vs Progressive scan
Interlaced
Progressive
51
Digital Video
Sampling
Quantization
52
Sampling and Quantization
53
Sampling
Quantization
Minimum of 3 bits!
54
BT.601 Video Format
BT.601 is formerly known as CCIR601
55
Chrominance Subsampling Formats
56
Digital Video Formats
57
Overview of Video Processing
Video Manipulation
Video Display
Video Compression
Video Database
Video Acquisition
Video Transmission
Computer Graphics
Video Analysis
Computer Vision
58
Video Manipulation

• Why?
• Fight against a non-ideal video acquisition
  (e.g., analog heritage, film scratches, limited
  resolution) or transmission environment
• Create new and artificial video content (e.g.,
  spatio-temporal interpolation, background/foregrou
  nd modification)

59
Video De-jittering
PDE-based approach by Jackie Shen
http//epubs.siam.org/sam-bin/dbq/article/41869
60
Video Inpainting
Cool application remove the annoying texts added
by various video conversion software
61
Error Concealment
corrupted blocks are recovered From surrounding
neighbors in space and time
some blocks are corrupted due to channel errors
62
Super Resolution
LR sequence




HR sequence
63
Post-processing
Deblocking suppress block artifacts in video
decoded video frame at very low bit rate
processed video frame after deblocking
64
Overview of Video Processing
Video Manipulation
Video Display
Video Compression
Video Database
Video Acquisition
Video Transmission
Computer Graphics
Video Analysis
Computer Vision
65
Video Coding Overview

• The grand challenge
• We still face insufficient storage space for
  video data even with Gigabyte hard disks
• Video transmission through limited bandwidth
  channels
• Existing approaches
• Three-dimensional waveform coding
• Motion-compensated hybrid coding
• Model-based coding

66
Video Coding Standards
H.263 8-64Kbps, videophone
H.263/ 8-64Kbps, videophone
H.261 p64Kbps
MPEG-4 8-1024Kbps, videophone
MPEG-2 2-10Mbps, DVD
MPEG-1 1.5Mbps, VCD
67
Overview of Video Processing
Video Manipulation
Video Display
Video Compression
Video Database
Video Acquisition
Video Transmission
Computer Graphics
Video Analysis
Computer Vision
68
Video Transmission (I)

• Downloading
• Pro you can have your own copy and can watch it
  offline
• Con you have to wait!!!
• Streaming
• Pro no need to store (we seldom watch a movie
  again and again)
• Con you have to have a good network connection
  and pray for less traffic

69
Video Transmission (II)
6Mbps
3Mbps
1.5Mbps
6Mbps
3Mbps
1.5Mbps
Multi-cast mode
Scalable mode
70
Scalable vs. Multicast

• What is scalable coding?

foreman.yuv
foreman.yuv
foreman128k.cod
foreman.cod
foreman256k.cod
foreman512k.cod
foreman1024k.cod
1024
512
256
128
Multicast
Scalable coding
71
Spatial scalability
1 0 1 1 1 0 1 0 1 0 0 0 1 1 0 1 0 0
72
Temporal scalability
1 0 1 1 1 0 1 0 1 0 0 0 1 1 0 1 0 0
Frame 0,1,2,3,4,5,
Frame 0,2,4,6,8,
Frame 0,4,8,12,
7.5Hz
30Hz
15Hz
73
SNR (Rate) scalability
1 0 1 1 1 0 1 0 1 0 0 0 1 1 0 1 0 0
PSNRavg40dB
PSNRavg30dB
PSNRavg35dB
PSNRi PSNR of frame i
74
Overview of Video Processing
Video Manipulation
Video Display
Video Compression
Video Database
Video Acquisition
Video Transmission
Computer Graphics
Video Analysis
Computer Vision
75
Motion Tracking
76
Video-based Lifeguard
Application in swimming pool monitoring to
prevent drowning
77
Overview of Video Processing
Video Manipulation
Video Display
Video Compression
Video Database
Video Acquisition
Video Transmission
Computer Graphics
Video Analysis
Computer Vision
78
Video Database Management

• Database management
• Indexing, parsing, browsing, querying
• Retrieval
• What is special about video?
• Formidable amount of data
• Difficulty with query (content-based)
• MPEG-7 is designed for database management

79
Content-Based Video Retrieval (CBVR)

• How to provide a compact and complete video
  sequence representation?
• Spatial analysis (histogram, color, texture)
• Temporal analysis (cut, dissolve, wipe)
• How to provide easy-to-use and efficient query
  interface to user
• Video browsing (slide vs. 3D)
• Video querying (example-based, text-based)

80
Video Summarization
Extract important motion pictures such as
home-runs