What is the Space of Camera Responses

1
What is the Space of Camera Responses?

• Michael Grossberg and Shree Nayar
• CAVE Lab, Columbia University
• IEEE CVPR Conference
• June 2003, Madison, USA
• Partially funded by NSF ITR Award, DARPA/ONR MURI

2
The Camera Response
Linear Function (Optical Attenuation)
Image Irradiance
Scene Radiance
E
L
s
3
Impact of Camera Response
Accurate scene radiance required for
Color Constancy
Creating Accurate High Dynamic Range Images
Shape from Shading
Photometric Stereo
Inverse Rendering
Measuring BRDF from Images
4
Response Model for Recovery
Charts Known Reflectance
Multiple Images Changing Exposure
Sawchuk, 77 Chang and Reid, 96
Debevec and Malik, 1997, Mann, 2000, Mann and
Picard, 1995, Mitsunaga and Nayar, 1999, Tsin et
al., 2001
Model Required for
Interpolation
Breaking Ambiguities Grossberg, Nayar, 2002
5
Response Normalization and Monotonicity
Saturation Level
Camera Response
1
f
Intensity B
Intensity B
0
0
1
Irradiance E
Dark Current Level
Irradiance E
Monotonicity Key Property Makes response
invertible
6
Space of Response Functions
Space of theoretical response functions
0
1
7
Linear Model of Response
f
f
h
h2
f0
h1
f0

• M-order linear approximation model

base response
parameters of model
basis functions
8
Choosing a Basis

• Possible basis h1, h2,

h4
Polynomial basis
h3
Intensity
h2
h1
Irradiance
h1
Trigonometric basis
h3
Intensity
h4
h2

• Which basis is best?
• Depends on which response functions occur

Irradiance
9
Database of Response Functions (DoRF)
Collected 201 response curves from
Film Positive, negative, consumer, professional,
color, b/w Agfacolor Futura Agfachrome
RX-II Fuji F125 Fuji FDIC Kodak Advanced Kodak
Gold
Digital/Video Sony DC 950 Canon Optura Gamma
curves
CCDs Kodak's KAI and KAF series
10
Sample curves
Cannon Optura
Kodak Ektachrome-100plus Green
1
Kodak DCS 315 Green
Kodak Ektachrome-64 Green
0.9
Sony DXC-950
Agfachrome CTPrecisa100 Green
0.8
Agfachrome RSX2 050 Blue
0.7
Agfacolor Futura 100 Green
Agfacolor HDC 100 plus Green
0.6
Agfacolor Ultra 050 plus Green
0.5
Normalized Brightness
Intensity
Agfapan APX 025
0.4
Agfa Scala 200x
Fuji F400 Green
0.3
Fuji F125 Green
0.2
g
gamma curve,
0.6
Kodak Max Zoom 800 Green
0.1
g
gamma curve,
1.0
Kodak KAI0372 CCD
g
gamma curve,
1.4
0
Kodak KAF2001 CCD
g
gamma curve,
1.8
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Irradiance
Evaluate Bases Using DoRF Good basis provides
good approximation with few parameters
11
Empirical Model of Response (EMoR)

• Build Basis using DoRF
• 175 training curves, 26 testing curves
• Apply PCA to DoRF f0, h1, h2, .
• 99.5 of energy in first 3 dimensions

Response
Response
Mean Curve
Principal Components
Energy
1
h4
h2
0.04
Normalized
Normalized
0.8
0
0.6
Intensity
Intensity
Percent
0.4
h3
h1
-0.04
0.2
-0.08
Percent of Energy
0
0
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
0.8
1
Irradiance
Principal Components
Irradiance
12
Log Model of Response (Log EMoR)
basis functions
base response

• Log basis
• Log model
• Generalizes gamma curves
• Log EMoR
• Apply PCA to Log DoRF
• 99.6 of energy in first 3 dimensions

parameters of model
Mean Curve
Principal Components
Energy
1
h
h
4
0.04
2
0.8
0
0.6
Intensity
Intensity
h
Percent
h
3
0.4
-0.04
1
0.2
-0.08
0
0
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
0.8
1
Irradiance
Irradiance
Principal Components
13
Monotonic Approximation
Monotonicity Linear inequalities in cn
Least Squares error

• Quadratic Programming

2 Principal Components
6
Gamma Curves
Gamma 2.5
Monotonic functions
Other DoRF Curves
5
4
Gamma 0.2
3
Derivative at Unity
2
1
0
Derivative at Unity
-1
- 5
0
5
10
15
20
25
30
35
40
45
Derivative at Origin
14
EMoR/Log EMoR Model Evaluation
EMoR Model
Log EMoR Model
Mean Disparity
Mean RMSE
Mean Disparity
Mean RMSE
4.12E-02 1.94E-02 8.87E-03 4.15E-03 2.82E-03
1.91E-03 1.58E-03 1.15E-03 9.10E-04 7.60E-04
6.02E-04
9.07E-02 4.77E-02 2.60E-02 1.51E-02 1.05E-02 8.09E
-03 7.46E-03 4.96E-03 4.30E-03 3.95E-03 3.12E-03
1 2 3 4 5 6 7 8 9 10 11
7.04E-02 1.71E-02 8.83E-03 4.99E-03 3.44E-03 2.80E
-03 2.52E-03 1.67E-03 1.36E-03 1.79E-03 9.54E-04
1 2 3 4 5 6 7 8 9 10 11
1.11E-01 3.44E-02 1.85E-02 1.11E-02 8.14E-03 6.58E
-03 5.77E-03 4.12E-03 3.36E-03 4.14E-03 2.49E-03
Parameters
Parameters
Accuracy 6.8 bits
Accuracy 6.8 bits
9.0 bits
8.3 bits
8.4 bits
8.6 bits
15
Models Compared
RMSE Error
Parameters
Model Gamma Polynomial Trigonometric EMoR
1 2 3 4
5 6
7 3.46E-02 N. A. N. A. N. A.
N. A. N. A. N. A. 7.37E-02
3.29E-02 1.71E-02 1.06E-02 6.93E-03 4.95E-03
3.65E-03 6.83E-02 3.91E-02 2.58E-02 1.89E-02
1.44E-02 1.16E-02 9.46E-03 4.00E-02 1.73E-02
6.27E-03 2.54E-03 1.77E-03 1.07E-03 9.55E-04
Accuracy 4.8 bits
5.9 bits
5.3 bits
7.3 bits
16
Response from Sparse Samples
Camera Response
1
0.8
Normalized Intensity
0.6
Brightness
Monotonic EMoR
0.4
Monotonic polynomial
EMoR
Normalized
Polynomial
Chart values used for fit
0.2
Other chart values
0
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.9
0.8
Normalized Irradiance
17
Response from Multiple Images
Inverse Camera Response
1
Monotonic EMoR
Mitsunaga-Nayar (Polynomial)
l
Debevec-Malik
8 (Log space)
0.8
l
Debevec-Malik
32 (Log space)
l
Debevec-Malik
128 (Log space)
Data from chart
0.6
Normalized Intensity
0.4
0.2
0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Normalized Irradiance
18
Summary

• Determined Space of Response functions
• Intersection of cone and plane
• Linear and Log approximation models
• Generalized previous models
• Database of Response Functions (DoRF)
• Evaluate models
• Empirical Model of Response (EMoR)
• Superior model of camera response based on DoRF
• DoRF and EMoR available for download
• from www.cs.columbia.edu/CAVE

19
Colors