Introduction of Video Retrieval and Archiving of MPEG Compressed Video

1
Introduction of Video Retrievaland Archiving of
MPEG Compressed Video

• By Yang Yongsheng
• Base on Vikrant Kobla and David Doermann,
  (1)Indexing and Retrieval of MPEG Compressed
  Video
• (2)Archiving,indexing,and retrieval of video in
  the compressed domain

2
Introduction of Video Retrieval

• In a Video Database, there are three types of
  metadata indexes were used
• Bibliographic Data
• Structural Data
• Content Data

3
Introduction of Video Retrieval

• 1.Bibliographic Data
• This category includes information about the
  video(e.g.,title,abstract, subject, and genre)
  and the individuals involved in the
  video(e.g.,producer, director, and cast).

4
Introduction of Video Retrieval

• 2.Structural Data
• Videos and movies can be described by a hierarchy
  of movie, segment, scene, and shot where each
  entry the hierarchy is composed of one or more
  entries at a lower level(e.g., a scene is
  composed of a sequence of shots and a segment is
  composed of a sequence of scenes).

5
Introduction of Video Retrieval

• 3.Content Data
• Users want to retrieve videos based on their
  content. Examples of content are
• 1.sets of key frames that represent key images in
  a video
• 2.keywrod indexed built from the sound track
• 3.object indexes that indicate entry and exit
  frames for each appearance of a significant
  object or individual

6
Introduction of Video Retrieval

• Figure1Video Retrieval in MPEG Compressed Domain

7
Overview of MPEG Stream

• Two basic techniques of MPEG(Moving Picture
  Expert Group) encoding algorithm
• Block-based motion compensation to capture
  temporal redundancy
• Transform domain based compres-sion to capture
  spatial redundancy

8
Overview of MPEG Stream

• A MPEG stream consists of three types of frames
  I, P and B frames
• I frame anchor frame, JPEG encoding
• P frame predicted from its preceding I or P
  frame.
• B frame predicted from both its preceding and
  following I and/or P frames.

9
Overview of MPEG Stream
10
Overview of MPEG Stream

• I,P,B frames are encoded using the 2D
  DCT(Discrete Cosine Transform).
• P and B frames include motion vector information,
  except I frames.
• Each frame is divided to blocks of 1616 pixels
  called Macroblocks(MBs)
• The computation of using Inverse DCT to
  decompressing MPEG video is expensive.

11
Archiving of MPEG Compressed Video

• Procedure
• Video Segmentation divides the incoming video
  into shots or scenes
• Key Frame Selection selects one or more key or
  representative frames for each shot
• Feature Extraction extracts features from key
  frames and uses to creat a database index

12
Video Segmentation

• Video segmentation, or shot change occurs between
  two frames,called a cut or a break.
• Most of the cuts can be detected by performing an
  analysis of the macroblock information only.
• In some special cases, DCT analysis is needed to
  detect shot changes.

13

• Figure3 Example of Shot Change

14
Cut detection based MBs Info.

• Three cases
• A P frame contains primarily intra-coded MBs, a
  shot change having occurred between the previous
  I/P frame and the current P frame.
• In a B frame, A majority of the MBs are
  forward-predicted from the previous I/P frame,a
  shot change will occur between current frame and
  the next I/P frame.
• In a B frame, majority of the MBs are
  backward-predicted,a shot change having occurred
  between the previous frame and the current frame.

15
DCT validation

• DCT validation is used to confirm the existence
  of certain shot changes, When the MB information
  is found to be insufficient.
• DCT validation involves counting the number of DC
  coefficients of corresponding blocks that differ
  in value by more than a given threshold, t0.
• If this count exceeds another threshold value,
  t1, then a shot change could possibly have taken
  place between the two frames.

16
Video Segmentation

• Figure4 Example of without a shot change, DCT
  validation is required

17
DCT validation algorithm

• Let be the I-frame in domain D, be the
  previous I-frame. Define
• Where stands for the DC
  coefficient of the frame.
• A shot change is validated if

18
Key Frame Selection

• 1.Ideal method
• Comparing each frame to every other frame in the
  scene and select the frame with the least
  difference from others.
• This method requires extensive computation and it
  is not practical for most applications.

19
Key Frame Selection

• 2.Choosing the first frame key frame
• It seems to be natural choice.All the rest of the
  frames in the scene can be considered to be
  logical and continuous extensions of the first
  frame.
• But, It may not be the best match for all the
  frames in the scene.

20
Key Frame Selection

• 3.Choosing frames with the greatest content or
  index potential as key frame
• Such as frames with text, or text, or frames with
  clear unoccluded objects

21
Key Frame Selection

• 4.Other factors frequency of I frames
• I frames represent the best candidates for key
  frame selection
• If I frames occur fairly frequently, then the
  first I frame can be chosen as the key frame
• In case of an entire scene with no I frame, other
  measures are required

22
Feature Extraction

• Challenge
• Frames can be of different types,i.e., I,P or B
  frames.
• I,P or B frames can occur in many patterns
• I,P or B frames may have complex
  inter-relationships.

23
Feature Extraction

• Solution removing dependencies of one frames on
  others.
• Normalizing DCT coefficients
• Normalizing flow vectors
• Normalizing frame size

24
Summary

• Introduction of Video Retrieval
• An overview of MPEG Stream
• Approach to archiving MPEG Compressed Video
• 1.Video Segmentation
• 2.Key Frame Selection
• 3.Feature Extraction