Robin Hogan

1
Towards unified radar/lidar/radiometer
retrievals for cloud radiation studies

• Robin Hogan
• Julien Delanoe
• Department of Meteorology, University of Reading,
  UK

2
Motivation

• Clouds are important due to their role in
  radiative transfer
• A good cloud retrieval must be consistent with
  broadband fluxes at surface and top-of-atmosphere
  (TOA)
• Increasingly, multi-parameter cloud radar and
  lidar are being deployed together with a range of
  passive radiometers
• We want to retrieve an optimum estimate of the
  state of the atmosphere that is consistent with
  all the measurements
• But most algorithms use at most only two
  instruments/variables and dont take proper
  account of instrumental errors
• The variational framework is standard in data
  assimilation and passive sounding, but has only
  recently been applied to radar
• Mathematically rigorous and takes full account of
  errors
• Straightforward to add extra constraints and
  extra instruments
• In this talk it will be shown how radar, lidar
  and infrared radiometers can be combined for ice
  cloud retrievals
• Demonstrated on ground-based and satellite
  (A-train) observations
• Discuss challenges of extending to other clouds
  and other instruments

3
Surface/satellite observing systems
Ground-based sites ARM and Cloudnet NASA A-Train Aqua, CloudSat, CALIPSO, PARASOL ESA EarthCARE For launch in 2013
Radar 35 and/or 94 GHz Doppler, Polarization 94 GHz CloudSat 94 GHz CPR Doppler
Lidar Usually 532 or 905 nm Polarization 532 1064 nm CALIOP Polarization 355 nm ATLID Polarization, HSRL
VIS/IR radiometers Some have infrared radiometer, sky imager, spectrometer MODIS, AIRS, CALIPSO IIR (Imaging Infrared Radiometer) Multi-Spectral Imager (MSI)
Microwave radiometers Dual-wavelength radiometer (e.g. 22 28 GHz) AMSR-E (6, 10, 18, 23, 36, 89 GHz) Polarization None
Broadband radiometers Surface BBR Europe/Africa sites have GERB overhead CERES (TOA only) BBR (TOA only)

• Broadband radiometers used only to test
  retrievals made using the other instruments

4
Radar and lidar

• Advantages of combining radar, lidar and
  radiometers
• Radar Z?D6, lidar b?D2 so the combination
  provides particle size
• Radiances ensure that the retrieved profiles can
  be used for radiative transfer studies
• Some limitations of existing radar/lidar ice
  retrieval schemes (Donovan et al. 2000, Tinel et
  al. 2005, Mitrescu et al. 2005)
• They only work in regions of cloud detected by
  both radar and lidar
• Noise in measurements results in noise in the
  retrieved variables
• Elorantas lidar multiple-scattering model is too
  slow to take to greater than 3rd or 4th order
  scattering
• Other clouds in the profile are not included,
  e.g. liquid water clouds
• Difficult to make use of other measurements, e.g.
  passive radiances
• Difficult to also make use of lidar molecular
  scattering beyond the cloud as an optical depth
  constraint
• Some methods need the unknown lidar ratio to be
  specified
• A unified variational scheme can solve all of
  these problems

5
Why not invert the lidar separately?

• Standard method assume a value for the
  extinction-to-backscatter ratio, S, and use a
  gate-by-gate correction
• Problem for optical depth dgt2 is excessively
  sensitive to choice of S
• Exactly the same instability for radar
  (Hitschfeld Bordan 1954)
• Better method (e.g. Donovan et al. 2000)
  retrieve the S that is most consistent with the
  radar and other constraints
• For example, when combined with radar, it should
  produce a profile of particle size or number
  concentration that varies least with range

Implied optical depth is infinite
6
First step target classification

• Combining radar, lidar with temperature from a
  model allows the type of cloud (or other target)
  to be identified
• Example from Cloudnet processing of ARM data
  (Illingworth et al., BAMS 2007)

Example from US ARM site Need to distinguish inse
cts from cloud
7
Formulation of variational scheme

• For each ray of data we define
• Observation vector State vector
• Elements may be missing
• Logarithms prevent unphysical negative
  values

8
The cost function

• The essence of the method is to find the state
  vector x that minimizes a cost function

Smoothness constraints
9
Solution method
New ray of data Locate cloud with radar
lidar Define elements of x First guess of x

• An iterative method is required to minimize the
  cost function

Forward model Predict measurements y from state
vector x using forward model H(x) Predict the
Jacobian H?yi/?xj
Gauss-Newton iteration step Predict new state
vector xk1 xkA-1HTR-1y-H(xk)
-B-1(xk-b)-Txk where the Hessian
is AHTR-1HB-1T
No
Has solution converged? ?2 convergence test
Yes
Calculate error in retrieval
Proceed to next ray
10
How do we solve this?

• The best estimate of x minimizes a cost function
• At minimum of J, dJ/dx0, which leads to
• The least-squares solution is simply a weighted
  average of m and b, weighting each by the inverse
  of its error variance
• Can also be written in terms of difference of m
  and b from initial guess xi

• Generalize suppose I have two estimates of
  variable x
• m with error sm (from measurements)
• b with error sb (background or a priori
  knowledge of the PDF of x)

11
The Gauss-Newton method

• We often dont directly observe the variable we
  want to retrieve, but instead some related
  quantity y (e.g. we observe Zdr and fdp but not
  a) so the cost function becomes
• H(x) is the forward model predicting the
  observations y from state x and may be complex
  and non-analytic difficult to minimize J
• Solution linearize forward model about a first
  guess xi
• The x corresponding to yH(x), is equivalent
  to a direct measurement m
• with error

y
Observation y
x
xi
xi1
xi2
(or m)
12

• Substitute into prev. equation
• If it is straightforward to calculate ?y/?x then
  iterate this formula to find the optimum x
• If we have many observations and many variables
  to retrieve then write this in matrix form
• The matrices and vectors are defined by

Where the Hessian matrix is
State vector, a priori vector and observation
vector
Error covariance matrices of observations and
background
The Jacobian
13
Radar forward model and a priori

• Create lookup tables
• Gamma size distributions
• Choose mass-area-size relationships
• Mie theory for 94-GHz reflectivity
• Define normalized number concentration parameter
• The N0 that an exponential distribution would
  have with same IWC and D0 as actual distribution
  
• Forward model predicts Z from extinction and N0
• Effective radius from lookup table
• N0 has strong T dependence
• Use Field et al. power-law as a-priori
• When no lidar signal, retrieval relaxes to one
  based on Z and T (Liu and Illingworth 2000,
  Hogan et al. 2006)

Field et al. (2005)
14
Lidar forward model multiple scattering

• 90-m footprint of Calipso means that multiple
  scattering is a problem
• Elorantas (1998) model
• O (N m/m !) efficient for N points in profile and
  m-order scattering
• Too expensive to take to more than 3rd or 4th
  order in retrieval (not enough)
• New method treats third and higher orders
  together
• O (N 2) efficient
• As accurate as Eloranta when taken to 6th order
• 3-4 orders of magnitude faster for N 50 ( 0.1
  ms)

Wide field-of-view forward scattered
photons may be returned
Narrow field-of-view forward scattered
photons escape
Ice cloud
Molecules
Liquid cloud
Aerosol
Hogan (Applied Optics, 2006). Code
www.met.rdg.ac.uk/clouds
15
Poster P3.10 Multiple scattering
CloudSat multiple scattering
New model agrees well with Monte Carlo

• To extend to precip, need to model radar multiple
  scattering

16
Radiance forward model

• MODIS solar channels provide an estimate of
  optical depth
• Only very weakly dependent on vertical location
  of cloud so we simply use the MODIS optical depth
  product as a constraint
• Only available in daylight
• Likely to be degraded by 3D cloud effects
• MODIS, CALIPSO and SEVIRI each have 3 thermal
  infrared channels in atmospheric window region
• Radiance depends on vertical distribution of
  microphysical properties
• Single channel information on extinction near
  cloud top
• Pair of channels ice particle size information
  near cloud top
• Radiance model uses the 2-stream source function
  method
• Efficient yet sufficiently accurate method that
  includes scattering
• Provides important constraint for ice clouds
  detected only by lidar
• Ice single-scatter properties from Anthony
  Barans aggregate model
• Correlated-k-distribution for gaseous absorption
  (from David Donovan and Seiji Kato)

17
Ice cloud non-variational retrieval
Donovan et al. (2000)
Aircraft-simulated profiles with noise (from
Hogan et al. 2006)
Observations State variables Derived
variables
Retrieval is accurate but not perfectly stable
where lidar loses signal

• Donovan et al. (2000) algorithm can only be
  applied where both lidar and radar have signal

18
Variational radar/lidar retrieval
Observations State variables Derived
variables
Lidar noise matched by retrieval
Noise feeds through to other variables

• Noise in lidar backscatter feeds through to
  retrieved extinction

19
add smoothness constraint
Observations State variables Derived
variables
Retrieval reverts to a-priori N0
Extinction and IWC too low in radar-only region

• Smoothness constraint add a term to cost
  function to penalize curvature in the solution
  (J l Si d2ai/dz2)

20
add a-priori error correlation
Observations State variables Derived
variables
Vertical correlation of error in N0
Extinction and IWC now more accurate

• Use B (the a priori error covariance matrix) to
  smooth the N0 information in the vertical

21
add visible optical depth constraint
Observations State variables Derived
variables
Slight refinement to extinction and IWC

• Integrated extinction now constrained by the
  MODIS-derived visible optical depth

22
add infrared radiances
Observations State variables Derived
variables
Poorer fit to Z at cloud top information here
now from radiances

• Better fit to IWC and re at cloud top

23
Convergence

• The solution generally converges after two or
  three iterations
• When formulated in terms of ln(a), ln(b) rather
  than a, b, the forward model is much more linear
  so the minimum of the cost function is reached
  rapidly

24
Radar-only retrieval
Observations State variables Derived
variables
Use a priori as no other information on N0
Profile poor near cloud top no lidar for the
small crystals

• Retrieval is poorer if the lidar is not used

25
Radar plus optical depth
Observations State variables Derived
variables
Optical depth constraint distributed evenly
through the cloud profile

• Note that often radar will not see all the way to
  cloud top

26
Radar, optical depth and IR radiances
Observations State variables Derived
variables
27
Ground-based example
Observed 94-GHz radar reflectivity Observed
905-nm lidar backscatter Forward model radar
reflectivity Forward model lidar backscatter
Lidar fails to penetrate deep ice cloud
28
Retrieved extinction coefficient a Retrieved
effective radius re Retrieved normalized number
conc. parameter N0 Error in retrieved extinction
Da
Radar only retrieval tends towards a-priori
Lower error in regions with both radar and lidar
29
Ground based example

• Radagast Campaign (AMMA)
• Based in Niamey, Niger
• ARM Mobile Facility
• MMCR cloud radar
• 532-nm micropulse lidar
• SEVIRI radiometer aboard MeteoSat 2nd Generation
  8.7, 10.8, 12µm channels
• Ice cloud case, 22 July 2006

30
Example from the AMF in Niamey
94-GHz radar reflectivity
Observations
532-nm lidar backscatter
31
Results radarlidar only

• Retrievals in regions where radar or lidar
  detects the cloud

Retrieved visible extinction coefficient
Retrieved effective radius
Retrieval error in ln(extinction)
32
Results radar, lidar, SEVERI radiances
Retrieved visible extinction coefficient

• TOA radiances increase retrieved optical depth
  and decrease particle size near cloud top

Retrieved effective radius
33
CloudSat/CALIPSO retrieval
Oct 13, 2006 0352-0358
AVHRR
Radar Reflectivity from CloudSat
Height km
0352 0355
0358
Attenuated lidar backscatter from CALIPSO
Height km
34
Forward model
Observed radar reflectivity, 95 GHz Attenuated
lidar backscatter, 532 nm Radar reflectivity
forward model Attenuated lidar backscatter
forward model
35
Preliminary results (radarlidar)
October 13th 2006 Granule 2006286023036_02443
between 3h52 and 3h58 UTC
36
MODIS radiances
Radiances not used in retrieval, just forward
modeled for comparison
Radar Reflectivity from CloudSat
Height km
Attenuated lidar backscatter from CALIPSO
Height km
Radiances W sr-1 m-2 Forward model MODIS
8.48.7 micron
10.7811.25 micron
11.77 12.27 micron
37
CloudSat/CALIPSO example
2006 Day 286
Radar Reflectivity from CloudSat
Attenuated lidar backscatter from CALIPSO
38
Conclusions and ongoing work

• New radar/lidar/radiometer cloud retrieval scheme
  
• Applied to ground based or satellite data
• Appropriate choice of state vector and smoothness
  constraints ensures the retrievals are accurate
  and efficient
• Can include any relevant measurement if forward
  model is available
• Could provide the basis for cloud/rain data
  assimilation
• Extension to other cloud types
• Retrieve properties of liquid-water layers,
  drizzle and aerosol
• Incorporate microwave radiances and wide-angle
  radar/lidar multiple-scattering forward models
  for deep precipitating clouds
• Other activities
• Validate using aircraft underflights
• Use in radiative transfer model to compare with
  TOA surface fluxes
• Build up global cloud climatology to evaluate
  models

39
CloudSat/CALIPSO example
2006 Day 172
Radar Reflectivity from CloudSat
Attenuated lidar backscatter from CALIPSO
40
Enforcing smoothness 1

• Cubic-spline basis functions
• Let state vector x contain the amplitudes of a
  set of basis functions
• Cubic splines ensure that the solution is
  continuous in itself and its first and second
  derivatives
• Fewer elements in x ? more efficient!

Forward model Convert state vector to high
resolution xhrWx Predict measurements y and
high-resolution Jacobian Hhr from xhr using
forward model H(xhr) Convert Jacobian to low
resolution HHhrW
Representing a 50-point function by 10 control
points
The weighting matrix
41
Enforcing smoothness 2

• Twomey matrix, for when we have no useful a
  priori information
• Add a term to the cost function to penalize
  curvature in the solution ld2x/dr2 (where r is
  range and l is a smoothing coefficient)
• Implemented by adding Twomey matrix T to the
  matrix equations