Creating Music Videos using Automated Media Analysis

1
Creating Music Videos using Automated Media
Analysis

• Authored by Jonathan Foote, Matthew Cooper, and
  Andreas Girgensohn
• Presented by Sukhyung Shin, Ninad Dewal

2
One neat usage

• Home videos are LONG
• AND generally have poor quality video audio
• Video has fast motion
• Video has moments of extreme brightness
• Too tedious to watch
• Too precious to throw away
• Solution
• Automatic Music Video Creation

3
Key Guidelines to Keep in Mind

• Soundtrack quality ? video quality
• You think the video is better
• Synchronization helps both
• Enhanced perception of quality
• Users choose clips
• Fully automated not optimal
• Need mix of both

4
What they did, in a Nutshell

• Automatic/Semi-automatic creation
• Source video
• Arbitrary audio soundtrack
• Video clips aligned w/ audio changes
• Audio looked for tempo
• Video looked for unsuitability
• High level of synchronization

5
Audio Parameterization

• Self-similarity (SS) analysis
• Independent of type of music
• Past and future regions
• Novel point between high SS regions
• Standard spectral parameterization
• Based on STFT (short term Fourier transform)
• Sampled at 22 kHz, quantized into 30 bins

6
Audio Self-Similarity Analysis

• Parameterized ? 2D representation
• Key Dis-similarity measurement (cosine)
• Can yield large scores for low magnitude vectors
• Similarity Matrix S
• Serves as visualization of audio file structure
• High similarity bright

7

• Not similar regions darker
• Look for regions of
• Low cross-similarity
• Then high self-similarity
• Compare with to obtain novelty N(i) for frame i

8
Segmenting and Editing Video

• Video boundaries into takes and clips
• Discarding Unsuitable Video
• Excessive camera motion or poor exposure
• Unsuitability score
• First estimate camera speed and direction
• Compare this estimate vs. current camera motion
• Test exposure/brightness
• Discard clips with score gt 0.5

9
(No Transcript)
10
Aligning Audio and Video

• So far, you have
• Peaks from audio
• Clips from video boundaries
• Simple solution
• Rank audio peaks and match w/ video boundaries
• Assuming video longer than audio (what if not?)
• Clip video clips even further if too big
• Assuming High suitability score w/ audio region
• Focus on audio segmenting video usually poor
• For fully automated Algorithms used sort, DP

11
User Control

• Hitchcock System

12
More Uses

• Home Videos ? Music Videos
• Precious but tedious
• Music artists
• MTV, VH1
• Movie, TV Show, Anime Fans
• Creating free MV as hobbies

13
Improvements

• Rhythmic synchronization
• Distinctive tempo or beat
• Combining source and soundtrack audio
• Has issues with edit boundaries

14
Conclusions

• Preliminary studies had positive outlook
• Users could interact w/ Hitchcock
• Authors realized that
• source videos audio should be used
• Hitchcock interface combined w/ automated
  ordering worked well.