ProjectorGuided Painting

1
Projector-Guided Painting

• Matthew Flagg and James M. Rehg
• Computational Perception Lab / GVU Center
• College of Computing
• Georgia Institute of Technology

2
Everyone is an artist
3
Everyone appreciates art
4
Not everyone has the courage to try
5
Goal

• Create an enjoyable painting experience
• Support novice artists in the creation of an oil
  painting
• Composition, brushwork, and color mixing
• Approach An on-canvas interface, consisting of a
  set of interaction modes, that provide guidance
  throughout the painting process

6
Outline

• System overview (play video)
• Motivation
• Process model for painting
• Technology for on-canvas interface
• Design of interaction modes
• User evaluations
• Future work

7
System Overview
8
Motivation

• Novice painters face a variety of technical
  challenges
• Managing the composition
• Trying to get one area right before moving on
  to another
• Difficulty of visualizing the finished effect
• Mixing paint
• Subtractive color space
• How much paint do I need?
• Brushwork
• How do I move the brush?

9
Motivation

• Guide books and art projectors are of limited
  usefulness
• Advances in projector-camera technology which can
  enable an adaptive on-canvas display
• With our system, a novice can have an enjoyable
  painting experience
• A feeling of I can do this

10
Process Model

• We assume the painting will be constructed from a
  series of layers
• Layers will be planned in advance and executed in
  sequence
• Within each layer, colors are mixed according to
  the palette and applied to give the desired
  brushstroke texture

11
Classical Layer-Based Styles

• Painting on toned or dark ground
• Alla Prima, Fa Presto, Direct Painting

Wet onto Dry
Wet onto Wet
12
Challenges

• How to control the appearance of the canvas and
  adapt to paint reflectance?
• How to provide assistance while
• Not interfering with the natural painting process
• Allowing the artist total control over the canvas
  at all times
• Where do the layers come from?

13
Display on Canvas

• Virtual Rear Projection (VRP) creates illusion of
  embedded display using redundant projection

14
Geometric Calibration
Projected image Camera view
User view
T projector-camera homography C
screen-camera homography P C-1T Warp image
by P-1
Screen
P
C
T
Projector
Camera
15
Problem Projecting onto a Painting
Desired appearance
Photo of Canvas
Van Gogh projected on Starry Night painting
16
Adaptation with Multiple Projectors
Desired appearance
Projector 1 output
Projector 2 output
Photo of Canvas
Appearance with projector 1
Final canvas appearance
17
Geometric and PhotometricCalibration Video
18
Painting Augmentation

• User sees real paint, virtual paint, or a
  combination of both

Real paint
Virtual paint
Projector 1
Canvas lit by grey illumination from 2 projectors
Canvas lit by 2 aligned projectors
Projector 2
19
Interaction Modes

• Preview Mode
• Helps to focus attention on layer at hand
• Provides visualization of colors to be painted
• Color Selection Mode
• Identifies all regions with a specific color
• Color Mixing Mode
• Paint mixing guidance based on color wheel
• Orientation Mode
• Brushstroke guidelines
• Blank mode
• Everything is off (standard painting environment)

20
Interaction Modes Video
21
Color Mixing Mode
Color wheel interface
Live camera view
22
Color Mixing Mode
User selection of mixture
23
Where Do Layers Come From? - Artists
expert snapshot 1
expert snapshot 2
expert snapshot 3
expert snapshot 4
expert snapshot 5
expert snapshot 6
24
Where Do Layers Come From? Graphics

• Employ NPR system by Hays and Essa to generate
  series of independent layers
• Each layer represents a frequency band in the
  image

James Hays, Irfan Essa. Image and Video Based
Painterly Animation. In Proceedings of NPAR 2004
25
Where Do Layers Come From? Graphics
input photograph
layer 1 from NPR
layer 2 from NPR
layer 3 from NPR
layer 4 from NPR
layer ordering
26
Result
painting following layer 1
painting following layer 2
painting following layer 3
finished painting
27
Result
28
Evaluation

• User study to assess change in confidence of
  novice painters after a painting experience
• Evaluation by art professors to assess technical
  quality
• Baseline for comparison is standard art projector
  and art books

29
User Study Design

• 20 subjects with no painting experience
• Asked to reproduce a model painting, given
• Canvas containing initial sketch (produced by art
  projector)
• Layer printouts (conventional art book)
• Experimental group used our system
• Sequence of events
• Warm up painting
• Confidence survey
• Model painting
• Confidence survey

30
Model Painting
Challenging perspective
Challenging brushwork
Challenging color mixture
Challenging shading
31
Confidence Survey
Please rate your confidence in ability to execute
the following paintings on a scale of 1 to 5,
where 1 means Not confident at all and 5 means
Very Confident
Easy
Medium
Difficult
Before
After
32
Survey Results Medium Difficulty
Significant at p lt 0.05
33
User Study Gallery
Model Painting
34
Paintings Made Using System
35
Paintings Made Using System
36
Paintings Made Using System
37
Paintings Made Without System
38
Paintings Made Without System
39
Paintings Made Without System
40
Quality Ratings by Painting Professors
41
Conclusions

• Using visual feedback, we can create an on-canvas
  interface which adapts to the underlying paint
• We can address common problems of novice painters
  through a layer-based process model and a set of
  interaction modes
• Self-reported user confidence scores increased
  after using our system
• Art professors ranked the paintings produced with
  our system as being of higher quality

42
Future Work

• Applications to capture and replay in classroom
  settings (distance learning)
• Extensions to other art forms
• Mural and decorative painting for interiors
• Sculpture
• Wood working

43
Projector-Guided Painting

• Matthew Flagg and James M. Rehg
• Computational Perception Lab / GVU Center
• College of Computing
• Georgia Institute of Technology