Image restoration by deconvolution

1
Image restoration by deconvolution

• Volker Bäcker
• Montpellier Rio Imaging http//www.mri.cnrs.fr/Pi
  erre Travo
• IFR3Giacomo CavalliFrederic BantigniesPatrice
  MollardNicole Lautrédou-Audouy Jean-Michel
  Poulinvolker.baecker_at_crbm.cnrs.fr

2
Overview

• Part 1
• introduction
• what is deconvolution ?
• how does it work ?
• when should it be used ?
• Part 2
• what are the parameters to know and care about
  for image restoration by deconvolution?

3
fluorescence microscopy

• specimen has to be in focal distance
• to image 3d specimen
• move focal plane through specimen
• creating stack of slides

• fluorescence microscopy
• specimen marked with dye that emists light of one
  wavelength while being stimulated by light of
  another wavelength
• Microscope types
• widefield whole specimen bathed
  in light
• confocal image is constructed
  point by point to keep out out-of-focus light
• two photon two photons needed to
  stimulate emission, similar effect as confocal

4
Example 2d widefield
Image from microscope

5
Example 3d confocal
After deconvolution
Image from microscope
6
Example time series 2 photon
Image from microscope
After deconvolution

7
The aquired image is not the real image

• Images are degraded due to the limited aperture
  of the objective
• Deconvolution can be used to get an image nearer
  to the real object
• by using knowledge of the imaging process and
  the properties of the microscope
• Deconvolution can be used for all kinds of
  fluorescence microscope images
• 2D, 3D, time series, widefield, confocal, 2
  photon

8
Sources of image degradation

• Noise
• Blur
• Can be handled by image restoration
• Scatter
• random distribution of light due to heterogenous
  refrection index within specimen
• Glare
• random distribution of light that occurs within
  the optical train of the microscope

9
Causes of image degradationNoise

• Geben Sie eine Zusammenfassung der momentanen
  Situation

10
Causes of image degradationNoise

• Where does the noise come from ?
• random fluctuations in the signal intensity
• variation of the incident photon flux
• interfering signals from electronic system of the
  captor device

11
Causes of image degradation Blur
Before restoration
After restoration
12
Causes of image degradationBlur

• Where does the blur come from ?
• contributions of out-of-focus light to the
  imaging plane
• diffraction
• a result of the interaction of light with
  matter
• diffraction is the bending of light as it passes
  the edge of an object

13
How does deconvolution work

• Image restoration
• Get rid of noise
• assume random noise with Poisson distribution
• remove it
• Get rid of blur
• Compute real image from sample
• by applying a model of how the microscope
  degraded the image

14
Point Spread Function

• Point spread function (psf)
• Model of how one pointis imaged by microscope
• Experimental
• aquired by taking an image ofpoint like
  objects - beads
• Alternatevely, point like object
• present in the acquired image
• itself can be usedf.
• Theoretical
• computed from the microscopeand captor parameters

15
Convolution (Faltung)

• aquired image real image convolved with psf
• Convolution is an integral that expresses
• amount of overlap of functions as g is shifted
  over f.
• N pixel gt O(NN) operations to compute it

16
Fourier Transform (FT)

• Signal can be represented as sum of sinoids
• FT transforms from spacial to frequency domain

17
Convolution theorem

Object function
psf
FT
Fourier transform (FT)
inverse FT
FT can be computed in O(n log n)
Object function convolved with psf
18
Deconvolution

• Deconvolution find object function f for given
  image i and psf g
• Unfortunatly it is not practicable to compute
• G has zeros outside certain regions
• Setting F zero for these would create artefacts
• In practice there is noise
• N/G would amplify noise
• It's not possible to reconstruct the real object
  function

19
Deconvolution algorithms

• Solution
• Find an algorithm that computes a function f' so
  that
• f' estimates f as good as possible
• works in the presence of noise
• Different deconvolution algorithms exist
• In general best for fluorescent microscopy
• (Classical) Maximum Liklihood Estimation - MLE

20
Maximum Likelihood Estimation

• Tries to optimise f' iteratively
• The basic principal is (but there's more to it)
• g(ij) psf - the fraction of light from true
  location j that is observed in pixel i

Richardson and LucyR-L Iteration
21
A
B
C
D
1
2
3
psf
4
image
5 51 / (51 0.34 0.26) 0,14 /(51
0.34 0.26) 0,26/(51 0.34 0.26)
5 5 / 7.4 0.4/7.4 1.2/7.4
fraction of light lost
5 (5 0.4 1.2)/7.4
5 6.6 /7.4
5 0.891891
New estimate
aquired image
last estimate
4.459459
last estimate
fraction of light from others
22
Summary and conclusions 1

• image from microscope is degraded
• it contains noise and blur
• blur can be described as a convolution of object
  function and psf
• image nearer to the object function can be
  obtained by image restoration yielding higher
  resolution and better contrast
• MLE is a deconvolution algorithm approriate for
  fluorescent microscope images
• imaging process is not finished finished without
  deconvolution
• do it whenever high quality images are needed

23
Image restoration in practice
Many deconvolution software packages are
commercially available
They use various types of deconvolution algorithms
In addition to these algorithms, they might
incorporate other imaging tools, such as filters
of different kinds. Moreover, different types of
algorithms may introduce or not some
 assumptions  concerning the image sent to
restoration.
In general, it is important to test the software.
One basic  rule of thumb  is also that the
restoration should respect the acquired image in
terms of objects visible and of their relative
intensity. Objects  appearing ,
 disappearing  or changing relative intensity
with respect to neighboring structures are
diagnostic of problems. These problems might be
due to the setting of relevant parameters or, in
the worst case, of poor quality of the software
24
Image restoration using the huygens2 software
from SVI

• http//www.svi.nl/ - website of Scientific Volume
  Imaging (SVI)
• It is the software used at the
• Institute of Human Genetics

25
Relevant parameters in deconvolution Setting
Microscope parameters

• microscope type
• widefield and multipoint confocal
• work with ccd camera
• single point confocal and two photon
• work with photomultiplier
• different point spread functions
• if you don't know
• Ask your imaging facility and look at the
  specifications of your microscope

26
Microscope parameters

• Numerical aperture
• measure of ability to gather light and resolve
  fine specimen detail at a fixed object distance
• higher magnification doesn't yield higher
  resolution, higher NA does
• Maximal value written on objective
• Can't be larger than the the refractive index n
  of the medium

27
Sampling theorem

• Imaging converts an anlog signal into a digital
  signal
• When converting an analog signal into a digital
  signal the sampling theorem applies
• Nyquist-Shannon sampling theoremthe sampling
  interval must not be greater than one-half the
  size of the smallest resolvable feature of the
  optical image
• sampling at nyquist rate means using exactly
  this interval
• sampling interval is the pixel size in our
  image

28
Undersampling and oversampling

• under sampling
• loss of information
• aliasing artefacts

• over sampling
• higher computation times and storage
  requirements
• longer acquisition times, photobleaching.

under sampling example. An object of a given
shape (dashed line) can be interpreted as a
different shape (thick line) if too few points
are acquired along any of the x,y,z axes
29
Changing the Numerical Aperture (NA)
forwidefield / two photon

• huygens2 allows under/oversampling within a
  range
• at the borders of this range deconvolution can be
  done but results are not good
• In this case better results when lying about
  NA
• if sampling size not in range change NA

nyquist sample size
30
Microscope parameters

• Excitation and emission wavelength
• fluorescent dye absorbs light of one wavelength
  and emits light of another wavelength
• filter cubes are used to ensure that only light
  of a wanted wavelength passes.
• exitation and emission wavelengths depend on the
  cube used
• GFP 473, 525

31
Microscope parameters

• The objective magnification used
• determines the pixel size in the image
• ccd camera
• Pixel size ccd element size /
  magnification(eventually modified by other
  parameters)
• photomultiplier
• pixel size depends on resolutionand
  magnification

32
Microscope parameters

• Refractive index n of the objective medium
• oil 1,51500
• water 1,33810
• air 1,00000
• Should match the refractive index of the sample
  medium
• Otherwise
• Magnification error in axial direction
• Spherical aberration (psf deteriorates with
  increasing depth)

33
Microscope parameters

• Cmount factor
• adaptor that attaches the camera to the
  microscope
• might contain additional optic that
• changes the overall magnification
• and therefore the pixel size
• value is 1 if no additional optic present

34
Microscope parameters

• Tube factor
• the tube might contain additional optics to
  change the tube length
• this changes the overallmagnification
• and therefore the pixel size

35
Microscope parameters

• sample medium
• refractive index n
• default (all media for example water)
  1,33810
• liquid Vectashield (not polymerized)
  1,49000
• 90-10 (vv) glycerol - PBS ph 7.4
  1,49000
• prolong antifade
  1,4
• limits the NA and therefore the possible
  resolution

36
Captor parameters

• size of the unitary ccd captor
• image sensor of the camera
• ccd charge coupled device
• diodes that convert light into electrical charge
• property of the camera
• Coolsnap 6450 nm
• Micromax 6700 nm

37
Captor parameters

• Binning
• take nxn elements as one
• more light per pixel
• reduces noise
• higher signal to noise ratio
• lower resolution

38
Captor parameters

• in case of XZY
• z step size
• in case of time series
• time interval

39
Captor parameters

• in case of confocal
• pinhole radius
• pinhole
• keep out out of focus light
• pinhole either fixed or adjustable
• Backprojected radius in nm
• Size of pinhole as it appears in the specimen
  plane
• size should match airy disk (2d psf) size

40
task parameter

• Style of processing
• step
• process image slide by slide
• converts stack into time series for processing
• converts result back into stack
• volume
• use 3d information
• step combined
• do step processing
• followed by volume processing with fixed
  parameters

41
Full restoration parameters

• signal/noise ratio
• the ratio of signal intensity to noise intensity
  
• high noise case
• can be measured in the image
• Single photon hit intensity
• find low intensity voxels from one photon hit
  add values subtract background
• Max voxel value
• value of brightest voxel
• low noise case
• single photon hits cant be seen
• rough guess is sufficient

42
Full restoration parameters

• background offset
• empty regions should be black
• but contain some light in reality
• subtract mean background to see object clearly

43
Full restoration parameters

• number of iterations
• too low
• optimal restoration not yet achieved
• too high
• takes longer to compute
• some signal may be removed
• Usually between 30-70

44
Summary and conclusions 2

• deconvolution should be used to obtain high
  quality imagesfor all kind of fluorescent
  microscope images
• parameters of the imaging system have to be
  entered to create a model of the image
  degradation

45
End of presentation

• volker.baecker_at_crbm.cnrs.fr
• Giacomo.Cavalli_at_igh.cnrs.fr

46
Links

• participants
• Montpellier RIO Imaging http//www.mri.cnrs.f
  r/
• IFR3 / CCIPEhttp//www.montp.inserm.fr/ifr3.htm
• IGHhttp//www.igh.cnrs.fr/
• CRIChttp//www.iurc.montp.inserm.fr/cric/index.ht
  m
• literature
• Introduction to Fluorescence Microscopyhttp//www
  .microscopyu.com/articles/fluorescence/fluorescenc
  eintro.html
• How does a confocal microscope work?http//www.ph
  ysics.emory.edu/weeks/confocal/
• Two-Photon Fluorescence Microscopyhttp//www.fz-j
  uelich.de/ibi/ibi-1/Two-Photon_Microscopy/
• Deconvolution in Optical Microscopyhttp//micro.m
  agnet.fsu.edu/primer/digitalimaging/deconvolution/
  deconintro.html

47
Links

• literature
• Diffraction of Lighthttp//micro.magnet.fsu.edu/p
  rimer/java/diffraction/basicdiffraction/
• Image restoration getting it righthttp//www.svi
  .nl/support/talks/GettingItRight.pdf
• Image Restoration in Fluorescence
  Microscopyhttp//www.ph.tn.tudelft.nl/Publication
  s/PHDTheses/GMPvanKempen/thesis_kempen.html
• Image restoration in one- and two-photon
  microscopyhttp//www.svi.nl/support/talks/Vancouv
  er97.pdf
• Introduction to Convolutionhttp//cnx.rice.edu/co
  ntent/m11542/latest/
• An Introduction to Fourier Theoryhttp//aurora.ph
  ys.utk.edu/forrest/papers/fourier/
• A Self Contained Introduction to Fourier
  Transformshttp//www.doc.eng.cmu.ac.th/course/cpe
  496/notes/fourier.pdf
• Convolution theoremhttp//www.fact-index.com/c/co
  /convolution_theorem.html

48
Links

• literature
• Three-Dimensional Imaging by Deconvolution
  MicroscopyArticle ID meth.1999.0873, available
  online at http//www.idealibrary.com on IDEAL
• Deconvolution of confocal images of
  dihydropyridine and ryanodine receptors in
  developing cardiomyocyteshttp//www.sfu.ca/tibbi
  ts/research/JAP04.pdf
• Maximum likelihood estimation via the ECM
  algorithm A general frameworkhttp//www.jbs.agrs
  ci.dk/lfo/talks/ECM_talk.pdf
• The influence of the background estimation on the
  superresolution properties ofnon-linear image
  restoration algorithmshttp//www.ph.tn.tudelft.nl
  /People/lucas/publications/1999/SPIE99GKLV/SPIE99G
  KLV.pdf
• Numerical Aperture and Resolutionhttp//micro.mag
  net.fsu.edu/primer/anatomy/numaperture.html
• User guide for Huygens Professional and
  Deconvolution Recipeshttp//www.svi.nl/download/
• Digital Image Sampling Frequencyhttp//www.olympu
  smicro.com/primer/java/digitalimaging/processing/s
  amplefrequency/index.html
• Filter Cubeshttp//www.olympusmicro.com/primer/te
  chniques/fluorescence/filters.html
• Filters for fluorescence microscopyhttp//www.nik
  on-instruments.jp/eng/products/option/index1.aspx

49
Links

• literature
• Immersion Mediahttp//www.olympusmicro.com/primer
  /anatomy/immersion.html
• How Digital Cameras Workhttp//electronics.howstu
  ffworks.com/digital-camera2.htm
• Pixel Binninghttp//micro.magnet.fsu.edu/primer/d
  igitalimaging/concepts/binning.html
• CCD Signal-To-Noise Ratiohttp//www.microscopyu.c
  om/tutorials/java/digitalimaging/signaltonoise/