The Human AudioVisual System

1
The Human Audio-Visual System

• Topics covered
• Digitisation and Quantisation.
• Sounds and the way we hear them.
• Colour and contrast.
• Images and the way we see them.

2
Digitisation and Quantisation

• The real world is smooth and continuous.
• The digital world is made of little chunks.
• Smaller chunks lead to a better illusion of
  smoothness.
• For example A screen with small pixels can
  display a smoother image than a similar screen
  with larger pixels.

3
Digitisation and Quantisation

• To quantise an image or sound we approximate the
  analogue signal with a digital signal.

4
Colour and contrast

• Colour can be thought of in terms of hue,
  lightness and saturation.
• The hue is the colour type (e.g. green).
• The lightness is how light or dark the colour is.
• The saturation is how intense the colour is.
• But computers tend to use a system based on
  mixing red, green and blue light (RGB).

5
Colour and contrast

• Most people can distinguish about 8 million
  different colours.
• A 24-bit computer display can show up to 16.7
  million colours.
• Some colours have good contrast in their hue but
  not in their lightness and saturation. This
  causes problems on monochrome (grey-scale)
  displays.

6
Images and the way we see

• The world is three dimensional but photographs,
  movies and TV are all two dimensional.
• How do we cope with this loss of information?
• If digital images are built from a regular,
  rectangular grid of dots...
• Why do we see pictures not pixels?

7
Images and the way we see

• Detail is not always important.
• You can see a man standing by this text.
• The picture is not a photograph and doesnt even
  match reality very well.
• So how can you see a man standing here?

8
Still images

• If an image is 640 pixels wide and 480 pixels
  high with 64K colours, how big is it?
• 640 480 2 614400 bytes 600KB.
• The calculation relies on knowing
• 16 colours 4 bits 0.5 byte per pixel
• 256 colours 8 bits 1 byte per pixel
• 65536 colours 16 bits 2 bytes per pixel
• 16.7 Mcolours 24 bits 3 bytes per pixel

9
Still images

• If an image is 640 pixels wide and 480 pixels
  high with 64K colours, how big is it after
  compression?
• Probably less than 600KB.
• Compression is usually image dependent.
• The amount of compression depends on the type of
  compression used.

10
Some still image formats

• bmp 24-bit, 8-bit and 4-bit colour supported,
  usually without compression.
• gif 8-bit and 4-bit colour, good lossless
  compression, often 21.
• jpg 24-bit colour, lossy compression, selectable
  quality, up to 501 compression.
• png 32-bit, 24-bit, 16-bit, 8-bit colour,
  lossless compression, better than gif.

11
Sounds and the way we hear

• Natural sounds are analogue.
• Analogue sounds are continuously variable in both
  frequency (pitch) and amplitude (loudness).
• Digital sounds are quantised to a certain number
  of levels and samples per second.
• Digital sounds can only approximate the original
  analogue signal.

12
Sounds and the way we hear

• Even though digital sound is stepped rather
  than smooth, it can still sound good.
• CD quality sound is played at about 44,100
  samples per second, in stereo.
• Each sample is a 16-bit value, capable of 65,536
  different amplitude levels.
• Humans cannot detect any inaccuracies in the
  sound at this sample rate.

13
Sounds and recording quality

• Digital sound can sound very bad if the sampling
  rate is too low.
• 44kHz 16-bit CD quality
• 22kHz 16-bit Very good
• 22kHz 8-bit Often good enough
• 11kHz 16-bit Often good enough
• 11kHz 8-bit Game quality, just okay
• 8kHz 8-bit Rough - telephone quality

14
Digital sound

• If a sound is recorded at 22000 samples per
  second at 16 bits per sample for ten seconds, how
  big is the file?
• 22000 2 10 440000 bytes, about 430KB.
• Sound is usually sampled at 8kHz, 11kHz, 22kHz or
  44 kHz.
• Sound is usually sampled at 4, 8 or 16 bits per
  sample.

15
Digital sound

• More samples per second and more bits per sample
  usually result in a higher quality recording.
• 44kHz, 16-bit stereo sound is considered to be
  CD quality - about 172KB per second!
• Telephone quality is more like 8kHz at 8 bits per
  sample, about 8KB per second.

16
Sound and compression

• Sound can be compressed through simple or complex
  methods, mostly lossy.
• Some types of sound can be compressed in special
  ways.
• Example Human speech can be compressed in a way
  that just doesnt work for music.

17
Some sound file formats

• wav 16-bit, 8-bit, 4-bit and other
  formats, some uncompressed, some lossy.
• aiff 16-bit, 8-bit compressed and uncompressed.
  
• snd 8-bit uncompressed.
• au 8-bit compressed and uncompressed.

18
Watching a movie

• What makes a movie move?
• A movie is really a sequence of still images.
• When still images are presented often enough, we
  perceive smooth motion.
• Cinema films run at 24 frames per second.
• Normal TV runs at 25 or 30 frames per second.
• Most cartoons run at 12 frames per second.

19
Watching a movie

• Still images have to have quite high quality - we
  may look at them for many seconds.
• Movie frames can have quite low quality - we only
  see each frame for a fraction of a second.
• In analogue TV broadcasts, the signal is split
  into two segments
• 70 Luminance (black white TV) value.
• 30 Colour signal.

20
Digital video

• If a movie is recorded digitally at 320 by 240
  pixels in 24-bit colour at 30 frames per
  second...
• How big will an uncompressed 5 minute video clip
  be?
• 5 60 30 320 240 3 about 1.98GB!
• The example uses a data rate of about 7MBps.
• Normal TV has a data rate of less than 6MBps.

21
Size of Digital Video

• Solutions
• Reduce the size, full frame is a luxury we cant
  afford
• Reduce the colour depth, use 256 colours rather
  than 16 million
• Reduce the frame rate, try 15fps
• Use compression techniques

22
Digital video

• Video compression relies on many tricks.
• Each frame could be compressed just like a still
  image, but that misses the point.
• Most compression methods rely on the difference
  between any two frames of video being quite
  small.
• Recording the differences and then compressing
  again can reduce file size by over 100 times.

23
Video Compression

• Compression reduces amount of information stored
  per frame which gives better playback performance
• Compression can be time consuming
• can take 10-15 minutes to compress a small video
  sequence
• Decompression is fast

24
Some digital video formats

• mov Quicktime movies, a few different compression
  methods avaliable. Often good quality and quite
  good compression.
• avi Audio-Video Interleaved, many different
  compression schemes possible. Quality and
  compression comparable to Quicktime.
• mpg MPEG, the Motion Picture Experts Group
  format. PC and TV quality movies.