Error Diffusion Halftoning Methods for Image Display

1
Error Diffusion Halftoning Methods forImage
Display
Prof. Brian L. Evans
Embedded Signal Processing Laboratory The
University of Texas at Austin Austin, TX
78712-1084 USA http//www.ece.utexas.edu/bevans
Ph.D. Graduates Dr. Niranjan
Damera-Venkata (HP Labs)
Dr. Thomas D. Kite (Audio
Precision) Dr. Vishal Monga (Xerox
Labs) Ph.D. Student Mr. Hamood Rehman
November 7, 2007
2
Outline

• Introduction
• Grayscale error diffusion
• Analysis and modeling
• Enhancements
• Color error diffusion
• Vector quantization withseparable filtering
• Matrix-valued error filtermethods
• Conclusion

Barbara image
Peppers image
3
Need for Digital Image Halftoning
Introduction

• Examples of reduced grayscale/color resolution
• Laser and inkjet printers (9.3B revenue in 2001
  in US)
• Facsimile machines
• Low-cost liquid crystal displays
• Halftoning is wordlength reduction for images
• Grayscale 8-bit image to 1-bit image (ink dot,
  or no ink dot)
• Color 24-bit RGB image to 12-bit RGB display
• Halftoning tries to reproduce full range of gray/
  color while preserving quality spatial
  resolution
• Screening methods quantize each pixel value
  (feedforward)
• Error diffusion methods quantize each pixel value
  and filter resulting quantization error
  (feedback)

4
Binarize Image Using a Fixed Threshold
Introduction

• 8-bit grayscale image
• Each pixel value represents image intensity
• Each pixel value is unsigned 8-bit number in 0,
  255
• For display black is 0, white is 255, and
  mid-gray is 128
• Threshold each pixel using same fixed threshold
• If pixel value ? 128, output 255 otherwise,
  output 0

5
Screening (Masking) Methods
Introduction

• Periodic thresholds to binarize image
• Periodic application leads to aliasing (gridding
  effect)
• Clustered dot screening is more resistant to ink
  spread
• Dispersed dot screening has higher spatial
  resolution
• Blue noise screening uses larger masks (e.g. 1
  by 1)

Clustered dot mask
Dispersed dot mask
index
Threshold Lookup Table
6
Adding Feedback to Quantization Example
Introduction

• Consider four-bit signal on two-bit display
  device
• Unsigned data
• Feedback quantization error
• For constant input of 1001

• Average output value
• 1/4(10101011) 1001
• 4-bit resolution at DC !
• Wordlength reduction
• Truncating from 4 bits to 2 bits increases noise
  by 12 dB
• Feedback eliminates noise at DC at expense of
  increasing noise at high frequency

7
Error Diffusion Halftoning
Introduction

• Nonlinear feedback system
• Quantize each pixel
• Filter quantization error (noise)
• Add filtered output to future grayscale pixels
  to diffuse quantization error

8
Conversion to One Bit Per Pixel Spatial Domain
Introduction
9
Conversion to One Bit Per Pixel Magnitude Spectra
Introduction
10
Human Visual System Modeling
Introduction

• Contrast at particular spatialfrequency for
  visibility
• Bandpass non-dimbackgroundsManos Sakrison,
  1974 1978
• Lowpass high-luminance officesettings with
  low-contrast imagesGeorgeson G. Sullivan,
  1975
• Exponential decay Näsäsen, 1984
• Modified lowpass versione.g. J. Sullivan, Ray
  Miller, 1990
• Angular dependence cosinefunction Sullivan,
  Miller Pios, 1993

11
Grayscale Error Diffusion Halftoning
Introduction

• Aperiodic halftones
• Shape quantization noise into highfrequencies,
  which are less visible
• Design of error filter key to quality

12
Analysis of Error Diffusion I
Analysis and Modeling

• Error diffusion as 2-D sigma-delta
  modulationAnastassiou, 1989 Bernard, 1991
• Error image Knox, 1992
• Error image correlated with input image
• Sharpening proportional to correlation
• Serpentine scan places morequantization error
  along diagonalfrequencies than raster Knox,
  1993
• Threshold modulation Knox, 1993
• Add signal (e.g. white noise) to quantizer input
• Equivalent to error diffusing an input image
  modified by threshold modulation signal

13
Analysis and Modeling
Example Role of Error Image

• Sharpening proportional to correlation between
  error image and input image Knox, 1992

Floyd-Steinberg(1976)
Limit Cycles
Jarvis(1976)
Error images
Halftones
14
Analysis of Error Diffusion II
Analysis and Modeling

• Limit cycle behavior Fan Eschbach, 1993
• For a limit cycle pattern, quantified likelihood
  of occurrence for given constant input as
  function of filter weights
• Reduced likelihood of limit cycle patterns by
  changing filter weights
• Stability of error diffusion Fan, 1993
• Sufficient conditions for bounded-input
  bounded-error stability sum of absolute values
  of filter coefficients is one
• Green noise error diffusionLevien, 1993 Lau,
  Arce Gallagher, 1998
• Promotes minority dot clustering
• Linear gain model for quantizerKite, Evans
  Bovik, 2000
• Models sharpening and noise shaping effects

Minority pixels
15
Linear Gain Model for Quantizer
Analysis and Modeling

• Extend sigma-delta modulation analysis to 2-D
• Linear gain model for quantizer in 1-D Ardalan
  and Paulos, 1988
• Linear gain model for grayscale image Kite,
  Evans, Bovik, 1997
• Error diffusion is modeled as linear,
  shift-invariant
• Signal transfer function (STF) quantizer acts as
  scalar gain
• Noise transfer function (NTF) quantizer acts as
  additive noise

Ks us(m)
us(m)
Signal Path
n(m)
un(m)
un(m) n(m)
Noise Path
16
Linear Gain Model for Quantizer
Analysis and Modeling
n(m)
Quantizermodel
x(m)
u(m)
b(m)
Ks
_

f(m)
_
Put noise in high frequencies H(z) must be
lowpass

e(m)
STF
NTF
2
1
1
w
w
w
-w1
-w1
-w1
w1
w1
w1
Also, let Ks 2 (Floyd-Steinberg)
Pass low frequencies Enhance high frequencies
Highpass response(independent of Ks )
17
Linear Gain Model for Quantizer
Analysis and Modeling

• Best linear fit for Ks between quantizer input
  u(m) and halftone b(m)
• Does not vary much for Floyd-Steinberg
• Can use average value to estimate Ks from only
  error filter
• Sharpening proportional to Ks Kite, Evans
  Bovik, 2000
• Value of Ks Floyd Steinberg lt Stucki lt Jarvis
• Weighted SNR using unsharpened halftone
• Floyd-Steinberg gt Stucki gt Jarvis at all viewing
  distances

18
Enhancements I Error Filter Design
Enhancements

• Longer error filters reduce directional
  artifactsJarvis, Judice Ninke, 1976 Stucki,
  1981 Shiau Fan, 1996
• Fixed error filter design minimize mean-squared
  error weighted by a contrast sensitivity function
• Assume error image is white noise Kolpatzik
  Bouman, 1992
• Off-line training on images Wong Allebach,
  1998
• Adaptive least squares error filter Wong, 1996
• Tone dependent filter weights for each gray level
  Eschbach, 1993 Shu, 1995 Ostromoukhov, 1998
  Li Allebach, 2002

19
Example Tone Dependent Error Diffusion
Enhancements

• Train error diffusionweights and
  thresholdmodulationLi Allebach, 2002

Highlights and shadows
FFT
Graylevel patch x
Halftone pattern for graylevel x
FFT
20
Enhancements II Controlling Artifacts
Enhancements

• Sharpness control
• Edge enhancement error diffusion Eschbach
  Knox, 1991
• Linear frequency distortion removal Kite, Evans
  Bovik 1991
• Adaptive linear frequency distortion
  removalDamera-Venkata Evans, 2001
• Reducing worms in highlights shadowsEschbach,
  1993 Shu, 1993 Levien, 1993 Eschbach, 1996
  Marcu, 1998
• Reducing mid-tone artifacts
• Filter weight perturbation Ulichney, 1988
• Threshold modulation with noise array Knox,
  1993
• Deterministic bit flipping quant. Damera-Venkata
  Evans, 2001
• Tone dependent modification Li Allebach, 2002

DBF(x)
x
21
Example Sharpness Control in Error Diffusion
Enhancements

• Adjust by threshold modulation Eschbach Knox,
  1991
• Scale image by gain L and add it to quantizer
  input
• Low complexity one multiplication, one addition
  per pixel
• Flatten signal transfer function Kite, Evans
  Bovik, 2000

L
b(m)
u(m)
x(m)
_

_

e(m)
22
Results
Enhancements
Original
Floyd-Steinberg
Edge enhanced
Unsharpened
23
Enhancements III Clustered Dot Error Diffusion
Enhancements

• Feedback output to quantizer input Levien, 1993
• Dot to dot error diffusion Fan, 1993
• Apply clustered dot screen on block and diffuse
  error
• Reduces contouring
• Clustered minority pixel diffusion Li
  Allebach, 2000
• Block error diffusion Damera-Venkata Evans,
  2001
• Clustered dot error diffusion using laser pulse
  width modulation He Bouman, 2002
• Simultaneous optimization of dot density and dot
  size
• Minimize distortion based on tone reproduction
  curve

24
Results
Enhancements
Block error diffusion
Green-noise
DBF quantizer
Tone dependent
25
Color Monitor Display Example (Palettization)
Color Error Diffusion

• YUV color space
• Luminance (Y) and chrominance (U,V) channels
• Widely used in video compression standards
• Contrast sensitivity functions available for Y,
  U, and V
• Display YUV on lower-resolution RGB monitor use
  error diffusion on Y, U, V channels separably

u(m)
b(m)
24-bit YUV video
12-bit RGB monitor
x(m)

_
_

RGB to YUV Conversion
h(m)
e(m)
26
Non-Separable Color Halftoning for Display
Color Error Diffusion

• Input image has a vector of values at each pixel
  (e.g. vector of red, green, and blue components)
• Error filter has matrix-valued coefficients
• Algorithm for adaptingmatrix coefficientsbased
  on mean-squarederror in RGB spaceAkarun,
  Yardimci Cetin, 1997
• Optimization problem
• Given a human visual system model, findcolor
  error filter that minimizes average visible noise
  power subject to diffusion constraints
  Damera-Venkata Evans, 2001
• Linearize color vector error diffusion, and use
  linear vision model in which Euclidean distance
  has perceptual meaning

u(m)
b(m)
x(m)
_

_
t(m)
e(m)

27
Matrix Gain Model for the Quantizer
Color Error Diffusion

• Replace scalar gain w/ matrix Damera-Venkata
  Evans, 2001
• Noise uncorrelated with signal component of
  quantizer input
• Convolution becomes matrixvector multiplication
  in frequency domain

u(m) quantizer inputb(m) quantizer output
Grayscale results
Noisecomponentof output
Signalcomponentof output
28
Linear Color Vision Model
Color Error Diffusion

• Undo gamma correction to map to sRGB
• Pattern-color separable model Poirson Wandell,
  1993
• Forms the basis for Spatial CIELab Zhang
  Wandell, 1996
• Pixel-based color transformation

B-W
R-G
B-Y
Opponent representation
Spatial filtering
29
Example
Color Error Diffusion
Original
Optimum vectorerror filter
SeparableFloyd-Steinberg
30
Evaluating Linear Vision ModelsMonga, Geisler
Evans, 2003
Color Error Diffusion

• Objective measure improvement in noise shaping
  over separable Floyd-Steinberg
• Subjective testing based on paired comparison
  task
• Online at www.ece.utexas.edu/vishal/cgi-bin/test.
  html
• In decreasing subjective (and objective) quality
• Linearized CIELab gt gt Opponent gt YUV ?
  YIQ

original
halftone A
halftone B
31
Image Halftoning Toolbox 1.2
Conclusion

• Grayscale andcolor methods
• Screening
• Classical diffusion
• Edge enhanced diff.
• Green noise diffusion
• Block diffusion
• Figures of merit
• Peak SNR
• Weighted SNR
• Linear distortion measure
• Universal quality index

Figures of Merit
http//www.ece.utexas.edu/bevans/projects/halfton
ing/toolbox
32
Backup Slides
33
Uniform Quantization Using Thresholding
Introduction

• Round to nearest integer (midtread)
• Quantize amplitude to levels -2, -1, 0, 1
• Step size D for linear region of operation
• Represent levels by 00, 01, 10, 11 or10, 11,
  00, 01
• Latter is two's complement representation
• Rounding with offset (midrise)
• Quantize to levels -3/2, -1/2, 1/2, 3/2
• Represent levels by 11, 10, 00, 01
• Step size

Qx
1
x
1
-2
2
-1
34
Error Diffusion
Introduction
Original
Halftone
u(m)
35
Digital Halftoning Methods
Introduction
Clustered Dot Screening AM Halftoning
Dispersed Dot Screening FM Halftoning
Error Diffusion FM Halftoning 1976
Blue-noise MaskFM Halftoning 1993
Green-noise Halftoning AM-FM Halftoning 1992
Direct Binary Search FM Halftoning 1992
36
Compensation for Frequency Distortion
Analysis and Modeling

• Flatten signal transfer function Kite, Evans,
  Bovik, 2000
• Pre-filtering equivalent to threshold modulation

x(m)
u(m)
g(m)
b(m)
_

_

e(m)
37
Visual Quality Measures Kite, Evans Bovik,
2000
Analysis and Modeling

• Sharpening proportional to Ks
• Value of Ks Floyd Steinberg lt Stucki lt Jarvis
• Impact of noise on human visual system
• Signal-to-noise (SNR) measures appropriate when
  noise is additive and signal independent
• Create unsharpened halftone ym1,m2 with flat
  signal transfer function using threshold
  modulation
• Weight signal/noise by contrast sensitivity
  function Ck1,k2
• Floyd-Steinberg gt Stucki gt Jarvis at all viewing
  distances

38
Example 1 Green Noise Error Diffusion
Enhancements

• Output fed back to quantizer input Levien, 1993
• Gain G controls coarseness of dot clusters
• Hysteresis filter f affects dot cluster shape

f
G
u(m)
b(m)
x(m)
_

_

e(m)
39
Example 2 Block Error Diffusion
Enhancements

• Process a pixel-block using a multifilterDamera-
  Venkata Evans, 2001
• FM nature controlled by scalar filter prototype
• Diffusion matrix decides distribution of error in
  block
• In-block diffusions constant for all blocks to
  preserve isotropy

40
Block FM Halftoning Error Filter Design
Enhancements

• FM nature of algorithm controlled by scalar
  filter prototype
• Diffusion matrix decides distribution of error
  within a block
• In-block diffusions are constant for all blocks
  to preserve isotropy

41
Vector Quantization but Separable Filtering
Color Error Diffusion

• Minimum Brightness Variation Criterion
  (MBVC)Shaked, Arad, Fitzhugh Sobel, 1996
• Limit number of output colors to reduce luminance
  variation
• Efficient tree-based quantization to render best
  color among allowable colors
• Diffuse errors separably

42
Results
Color Error Diffusion
Original
MBVC halftone
SeparableFloyd-Steinberg
43
Linear Color Vision Model
Color Error Diffusion

• Undo gamma correction on RGB image
• Color separation Damera-Venkata Evans, 2001
• Measure power spectral distribution of RGB
  phosphor excitations
• Measure absorption rates of long, medium, short
  (LMS) cones
• Device dependent transformation C from RGB to LMS
  space
• Transform LMS to opponent representation using O
• Color separation may be expressed as T OC
• Spatial filtering included using matrix filter
• Linear color vision model

is a diagonal matrix
where
44
Designing the Error Filter
Color Error Diffusion

• Eliminate linear distortion filtering before
  error diffusion
• Optimize error filter h(m) for noise shaping
• Subject to diffusion constraints
• where

45
Generalized Optimum Solution
Color Error Diffusion

• Differentiate scalar objective function for
  visual noise shaping w/r to matrix-valued
  coefficients
• Write norm as trace and differentiate trace
  usingidentities from linear algebra

46
Generalized Optimum Solution (cont.)
Color Error Diffusion

• Differentiating and using linearity of
  expectation operator give a generalization of the
  Yule-Walker equations
• where
• Assuming white noise injection

• Solve using gradient descent with projection onto
  constraint set

47
Implementation of Vector Color Error Diffusion
Color Error Diffusion
Hgr
Hgg

Hgb
48
Generalized Linear Color Vision Model
Color Error Diffusion

• Separate image into channels/visual pathways
• Pixel based linear transformation of RGB into
  color space
• Spatial filtering based on HVS characteristics
  color space
• Best color space/HVS model for vector error
  diffusion? Monga, Geisler Evans 2002

49
Linear CIELab Space TransformationFlohr,
Kolpatzik, R.Balasubramanian, Carrara, Bouman,
Allebach, 1993
Color Error Diffusion

• Linearized CIELab using HVS Model by
• Yy 116 Y/Yn 116 L 116
  f (Y/Yn) 116
• Cx 200X/Xn Y/Yn a 200
  f(X/Xn ) f(Y/Yn )
• Cz 500 Y/Yn Z/Zn b 500
  f(Y/Yn ) f(Z/Zn )
• where
• f(x) 7.787x 16/116 0lt x lt
  0.008856
• f(x) (x)1/3
  0.008856 lt x lt 1
• Linearize the CIELab Color Space about D65 white
  point
• Decouples incremental changes in Yy, Cx, Cz at
  white point on (L,a,b) values
• T is sRGB ? CIEXYZ ?Linearized CIELab

50
Spatial Filtering
Color Error Diffusion

• Opponent Wandell, Zhang 1997
• Data in each plane filtered by 2-D separable
  spatial kernels
• Parameters for the three color
  planes are

Plane Weights wi Spreads si
Luminance 0.921 0.0283
  0.105 0.133
  -0.108 4.336
Red-green 0.531 0.0392
  0.330 0.494
Blue-yellow 0.488 0.0536
  0.371 0.386
51
Color Error Diffusion
Spatial Filtering

• Spatial Filters for Linearized CIELab and YUV,YIQ
  based on
• Luminance frequency Response Nasanen and
  Sullivan 1984

L average luminance of display, the radial
spatial frequency and
K(L) aLb
where p (u2v2)1/2 and
w symmetry parameter 0.7 and
effectively reduces contrast sensitivity at odd
multiples of 45 degrees which is equivalent to
dumping the luminance error across the diagonals
where the eye is least sensitive.
52
Color Error Diffusion
Spatial Filtering
Chrominance Frequency Response Kolpatzik and
Bouman 1992
Using this chrominance response as opposed
to same for both luminance and
chrominance allows
more low frequency chromatic error not perceived
by the human viewer.

• The problem hence is of designing 2D-FIR filters
  which most closely match the desired Luminance
  and Chrominance frequency responses.
• In addition we need zero phase as well.
• The filters ( 5 x 5 and 15 x 15 were
  designed using the frequency sampling approach
  and were real and circularly symmetric).
• Filter coefficients at http//www.ece.utex
  as.edu/vishal/halftoning.html
• Matrix valued Vector Error Filters for each of
  the Color Spaces at
• http//www.ece.utexas.edu/vishal/mat_filter.html

53
Color Spaces
Color Error Diffusion

• Desired characteristics
• Independent of display device
• Score well in perceptual uniformity Poynton
  color FAQ http//comuphase.cmetric.com
• Approximately pattern color separable Wandell et
  al., 1993
• Candidate linear color spaces
• Opponent color space Poirson and Wandell, 1993
• YIQ NTSC video
• YUV PAL video
• Linearized CIELab Flohr, Bouman, Kolpatzik,
  Balasubramanian, Carrara, Allebach, 1993

Eye more sensitive to luminance reduce
chrominance bandwidth
54
Monitor Calibration
Color Error Diffusion

• How to calibrate monitor?
• sRGB standard default RGB space by HP and
  Microsoft
• Transformation based on an sRGB monitor (which is
  linear)
• Include sRGB monitor transformation
• T sRGB ? CIEXYZ ?Opponent RepresentationWandell
  Zhang, 1996
• Transformations sRGB ? YUV, YIQ from S-CIELab
  Code at http//white.stanford.edu/brian/scielab/s
  cielab1-1-1/
• Including sRGB monitor into model enables
  Web-based subjective testing
• http//www.ece.utexas.edu/vishal/cgi-bin/test.htm
  l

55
Spatial Filtering
Color Error Diffusion

• Opponent Wandell, Zhang 1997
• Data in each plane filtered by 2-D separable
  spatial kernels
• Linearized CIELab, YUV, and YIQ
• Luminance frequency response Näsänen and
  Sullivan, 1984
• L average luminance of display
• r radial spatial frequency
• Chrominance frequency response Kolpatzik and
  Bouman, 1992
• Chrominance response allows more low frequency
  chromatic error not to be perceived vs. luminance
  response

56
Subjective Testing
Color Error Diffusion

• Binomial parameter estimation model
• Halftone generated by particular HVS model
  considered better if picked over another 60 or
  more of the time
• Need 960 paired comparison of each model to
  determine results within tolerance of 0.03 with
  95 confidence
• Four models would correspond to 6 comparison
  pairs, total 6 x 960 5760 comparisons needed
• Observation data collected from over 60 subjects
  each of whom judged 96 comparisons
• In decreasing subjective (and objective) quality
• Linearized CIELab gt gt Opponent gt YUV ?
  YIQ

57
Selected Open Problems
Conclusion

• Analysis and modeling
• Find less restrictive sufficient conditions for
  stability of color vector error filters
• Find link between spectral characteristics of the
  halftone pattern and linear gain model at a given
  graylevel
• Model statistical properties of quantization
  noise
• Enhancements
• Find vector error filters and threshold
  modulation for optimal tone-dependent vector
  color error diffusion
• Incorporate printer models into optimal framework
  for vector color error filter design

58
Joint Bi-Level Experts Group
Compression of Error Diffused Halftones

• JBIG2 standard(Dec. 1999)
• Binary document printing, faxing, scanning,
  storage
• Lossy and lossless coding
• Models for text, halftone, and generic regions

• Lossy halftone compression
• Preserve local average gray level not halftone
• Periodic descreening
• High compression of ordered dither halftones

Generate (M21) patterns ofsize M x M from a
clustereddot threshold mask
Construct Pattern Dictionary
Lossless Encoder
JBIG2 bitstream
Halftone
Compute Indices into Dictionary
Count black dots in each M x M block of
input Range of indices 0... M21
59
JBIG2 Halftone Compression Model

• JBIG2 assumes halftones produced by small
  periodic screen
• Stochastic halftones are aperiodic

Existing JBIG-26.1 1
Proposed Method 6.6 1
60
Lossy Compression of Error Diffused Halftones
Compression of Error Diffused Halftones

• Proposed method Valliappan, Evans, Tompkins,
  Kossentini, 1999
• Reduce noise and artifacts
• Achieve higher compression ratios
• Low implementation complexity

Linear distortion measure (LDM)
High Quality Ratio 6.6 1 WSNR 18.7 dB LDM 0.116
High Compression Ratio 9.9 1 WSNR 14.0
dB LDM 0.158
512 x 512 Floyd-Steinberg halftone
61
Lossy Compression of Error Diffused Halftones
Compression of Error Diffused Halftones

• Fast conversion of error diffused halftones to
  screened halftones with rate-distortion tradeoffs
  Valliappan, Evans, Tompkins, Kossentini, 1999

• modified multilevel Floyd Steinberg
• error diffusion
• L sharpening factor

• 3 x 3 lowpass
• zeros at Nyquist
• reduces noise
• 2n coefficients

Prefilter
Decimator
Quantizer
Lossless Encoder
JBIG2 bitstream
Halftone
17
16 M2 1
graylevels
N
2

• M x M lowpass averaging filter
• downsample by M x M

Symbol Dictionary
Free Parameters L sharpening M downsamping
factor N grayscale resolution

• N patterns
• size M x M
• may be angled
• clustered dot

62
Rate-Distortion Tradeoffs
Compression of Error Diffused Halftones
Linear Distortion Measure for downsampling
factor M ? 2, 3, 4, 5, 6, 7, 8
Weighted SNR for downsampling factor M ? 2, 3,
4, 5, 6, 7, 8(linear distortion removed)