Unsupervised%20Automation%20of%20Photographic%20Composition%20Rules

1
Unsupervised Automation of Photographic
Composition Rules
Serene Banerjee and Brian L. Evans http//www.ece.
utexas.edu/bevans/projects/dsc/index.html
Computer Engineering Area Dept. of Electrical
and Computer Engineering The University of Texas
at Austin
2
Motivation

• Problem Amateur photographers often
  takelow-quality pictures with digital still
  cameras
• Personal use
• Professionals who need to document
• (e.g.. realtors and architects)
• Goal Automate photographic composition rules and
  find alternatives to the picture being acquired
• Analyze scene, including detection of main
  subject
• Develop algorithms to automate rules

3
Solution

• Solution 1 Automatically detect main subject
• Independent of indoor/outdoor setting or scene
• Low implementation complexity, fixed-point
  computation
• Solution 2 Automate a few photograph
  composition rules
• Rule of thirds for placing the main subject
• Simulated background blur for motion pictures or
  depth-of-field

4
Digital Still Cameras

• Converts optical image to electric signal using
  charge coupled device (CCD)
• Software control
• Zoom
• Focus, e.g. auto-focus filter
• Shutter aperture and speed
• White balance Corrects color distortions
• Settings that can be controlled (with added
  hardware)
• Camera angle
• Aspect ratio Landscape or portrait mode
• Produces JPEG compressed images

5
Main Subject Detection Methods

• Two differently focused photographs Aizawa,
  Kodama, Kubota 1999-2002
• One has foreground in focus, and other has
  background in focus
• Significant delay involved in changing the focus
• Bayes nets based training Luo, Etz, Singhal,
  Gray 2000-2001
• Bayesian network trained on example set and
  tested later
• Training time involved suited for offline
  applications
• Multi-level wavelet coefficients Wang, Lee,
  Gray, Wiederhold 1999-2001
• Expensive to compute and analyze wavelet
  coefficients
• Iterative classification from variance maps Won,
  Pyan, Gray 2002
• Optimal solution from variance maps and
  refinement with watershed
• Suitable for offline applications involving
  iterative passes over image

6
Proposed Main Subject Detection

• User starts image acquisition
• Focus main subject using auto-focus filter
• Partially blur background and acquire resulting
  picture
• Open shutter aperture (by lowering f-stop) which
  takes about 1 s
• Foreground edges stronger than background edges
• While acquiring user-intended picture, process
  blurred background picture to detect main subject
• Generate edge map (subtract original image from
  sharpened image)
• Apply edge detector (Canny edge detector performs
  well)
• Close boundary (e.g. gradient vector flow or
  proposed approximation)

7
Generate Edge Map

fsmooth(x,y)
-
g(x,y)
fsharp(x,y)
Smoothing filter


f(x,y)

Sharpening filter
Model for an image sharpening filter

• Symmetric 3 x 3 sharpening filter
• For integer a and b, coefficients are
• Integer when dropping 1/(1 a) term
• Fractional when -1 2a ? b lt 1 and 1/(1 a) is
  power-of-two
• Generate edge map
• Subtract original image from sharpened image
• Main subject region now has sharper edges

8
Boundary Closure

• Gradient vector flow method Xu, Yezzi, Prince
  1998-2001
• Compute gradient
• Outer boundary of detected sharp edges is initial
  contour
• Change shape of initial contour, depending on
  gradient
• Shape converges in approximately 5 iterations
• Disadvantage computationally and memory
  intensive
• Approximate lower complexity method
• Select leftmost rightmost ON pixel and make row
  between them ON
• Can detect convex regions but fails at concavities

9
Automation of Rule-of-Thirds

• Goal Center of mass of the main subject at 1/3
  or 2/3 of the picture width (or height) from the
  left (or top) edge
• Solution
• For n-D, define function that attains minimum
  when center of mass placed as desired and
  increases otherwise
• Shift picture so that minimum is attained
• Implementation
• For 2-D, sum of Euclidean distance from the 4
  points
• Measure which of the 4 points is closest to the
  current position of the center of mass
• Shift picture so that the rule-of-thirds is
  followed

10
Simulated Background Blurring

• Goal Filter the image background and add
  artistic effects keeping the main subject intact
• Solution
• Original image masked with detected main subject
  mask
• Region of interest filtering performed on masked
  image
• Possible motion blurs
• Linear blur subject or camera motion
• Radial blur camera rotation
• Zoom change in zoom
• Applications
• Enhance sense of motion where the main subject is
  moving
• Digitally decrease the depth-of-field of the
  photograph

11
Proposed Module
12
Implementation Complexity
Processing step Multiply-Accumulates /pixel Comparisons/pixel Memory accesses/pixel
Main subject detection 18 4 10
Rule of thirds 2 1 1 or 3
Background blurring 9 4

• Number of computations and memory accesses per
  pixel
• Main subject detection convolution with
  symmetric 3x3 filter, edge detection, approximate
  boundary closure
• Rule-of-thirds center of mass (1 division, 4
  compares) , shift pixels
• Background blurring convolution with symmetric
  3x3 filter
• Digital still cameras use 160 digital signal
  processor instruction cycles per pixel

13
Results (1)
Original image with main subject(s) in focus
Detected strong edges with proposed algorithm
Detected main subject mask
Rule-of-Thirds Main subject repositioned
Simulated background blur
14
Results (2)
Original image with main subject(s) in focus
Detected strong edges with proposed algorithm
Detected main subject mask
Rule-of-Thirds Main subject repositioned
Simulated background blur
15
Results (3)
Original image with main subject(s) in focus
Detected strong edges with proposed algorithm
Detected main subject mask
Rule-of-Thirds Main subject repositioned
Simulated background blur
16
Conclusion

• Developed automated low-complexity one-pass
  method for main subject detection in digital
  still cameras
• Processes picture taken with blurred background
• All calculations in fixed-point arithmetic
• Automates selected photographic composition rules
• Rule-of-thirds Placement of the main subject on
  the canvas
• Simulated background blur motion and
  depth-of-field
• Applications digital still cameras,
  surveillance, constrained image compression, and
  transmission and display
• Copies of MATLAB code, poster, and paper,
  available at
• http//www.ece.utexas.edu/bevans/projects/dsc/in
  dex.html