Framework for digital camera in Linux

1
Framework for digital camerain Linux

• Dongsoo Kim (dongsoo45.kim_at_samsung.com)
• Heungjun Kim(riverful.kim_at_samsung.com)Samsung
  Electronics

2
Contents

• Introducing mobile camera devices
• SoC camera subsystem
• Mobile camera module devices
• Framework for digital camera
• Video4Linux2 and camera device
• New APIs for high-end mobile camera devices
• Dual camera framework for 3G handset
• Porting issues in SoC camera subsystem

3
Terms

• AP Application Processor
• Camera interface interface device which is
  syncing image data from external camera device
  and gives some commands to them
• Camera module external camera device packaged
  with lens, sensor and ISP on it
• ISP Image Signal Processor. Also supports lots
  of additional functions
• V4L2 Video for Linux 2

4

• This document contains some animations.
• Slideshow mode is recommended

5
What makes camera a camera?

• Strobe support
• Various program mode slow sync, red-eye
  reduction and on.

• Exposure control
• Aperture (Iris)
• Shutter
• ISO

• Lens control
• Focus
• Zoom

6

• Introducing mobile camera devices

7

• History of camera phone

First Camera Phone in 2000 with 3.5MP
1.3MP in 2003
5MP in2004
3MP in2004
7MP in 2005
2MP in 2004
10MP in 2006
12MP in 2009
8
Mobile camera module devices

• Old and low-end mobile camera modules

9
Mobile camera module devices

• Brand new and high-end mobile camera modules

10
SoC camera subsystem

• What is SoC camera subsystem?

ARM SoC solution
Peripherals
External Camera module
ARM Core
11
SoC camera subsystem
Image data in expected format
Request for data
Camera interface
Buffer handling
Resolution handling
Data format handling (YUV/RAW/JPEG)
Command interface
I2C
Camera module
Exposure control
Lens control
Strobe control
White balance control
Effect control
Face detect
Zoom control
. . .
12
SoC camera subsystem

• Various ways of preview (Live view) in SoC camera
  subsystems

User Application
COPY!
M e m o r y
Frame buffer
DATA
Camera Interface
DATA
Camera Module
OK
Cancel
13
SoC camera subsystem

• Various ways of preview (Live view) in SoC camera
  subsystems

User Application
COPY!
Save as file
DATA
Camera Interface
DATA
Camera Module
Frame buffer
Through DMA
OK
Cancel
14
SoC camera subsystem

• How to capture still shots in SoC camera
  subsystem
• Your camera module has a dedicated JPEG encoder
  on it

User Application
Camera module
COPY!
DATA
RAW DATA PROCESS
JPEG PROCESS
M e m o r y
DATA
Save as JPEG file
Camera Interface
DATA
MEMORY
15
SoC camera subsystem

• How to capture still shots in SoC camera
  subsystem
• No JPEG encoder in your camera module

User Application
COPY!
RAW DATA
M e m o r y
DATA
Codec S/W
Camera Interface
DATA
Camera Module
Save as JPEG file
16
SoC camera subsystem

• How to take motion pictures in SoC camera
  subsystem

Frame buffer
COPY!
User Application
M e m o r y
OK
Cancel
DATA
Codec S/W
Save as
Camera Interface
DATA
Camera Module
COPY!
17
What weve got in Linux to control camera

• In Kernel
• V4L2 (AKA Video For Linux 2)
• The second version of the Video For Linux API
• Kernel interface for analog radio and video
  capture and output drivers

• In user space
• Multimedia middleware
• Gstreamer
• Pipeline based multimedia framework written in
  the C programming language with the type system
  based on GObject
• OpenMAX
• Cross-platform set of C-language programming
  interfaces that provides abstractions for
  routines especially useful for audio, video, and
  still images

18
Mobile camera module devices
Is V4L2 enough for mobile digital camera modules?
No way
19

• Framework for digital camera

20
Video4Linux2 and camera device

• Camera subsystem in Linux kernel aspect

videobuf
videodev
V4l2-dev
V4l2-ioctl
V4l2-subdev (new)
V4l2-int-device (old)
V4L2
videobuf-core
videobuf-sg-dma
V4l2-device
Camera interface
V4L2 driver
Buffer handling
Resolution handling
Data format handling (YUV/RAW/JPEG)
Command interface
Camera module
V4L2 driver
Exposure control
Lens control
Strobe control
White balance control
Effect control
Face detect
Zoom control
. . .
21
Abstract view of V4L2 working with video capture
device

• How does it work with V4L2?

Application or Middleware
Open device
Copy memory to userspace
VIDIOC_QUERYCAP
VIDIOC_S_INPUT
VIDIOC_REQBUF
VIDIOC_QBUF
VIDIOC_QBUF
VIDIOC_DQBUF
VIDIOC_QUERYBUF
VIDIOC_STREAMON
22

• New V4L2 APIs for digital camera
• Exposure control

23
New V4L2 APIs for exposure control

• Need for detailed control in exposure
• Weve got camera devices with mechanical shutter
• Iris could be handled also
• ISO controlled output is somehow useful

Why not using enhanced exposure control like a
regular digital camera?
24
New V4L2 APIs for exposure control

• What weve got in mainline V4L2 right now?

define V4L2_CID_EXPOSURE_AUTO
(V4L2_CID_CAMERA_CLASS_BASE1) enum  v4l2_exposure
_auto_type          V4L2_EXPOSURE_AUTO  0,      
   V4L2_EXPOSURE_MANUAL  1,         V4L2_EXPOSURE
_SHUTTER_PRIORITY  2,         V4L2_EXPOSURE_APERT
URE_PRIORITY  3 define V4L2_CID_EXPOSURE_ABSO
LUTE  (V4L2_CID_CAMERA_CLASS_BASE2) define V4L2_
CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_
BASE3)

• No way to control iris
• No way to control exposure metering method

25
New V4L2 APIs for exposure control

• Additional exposure control

enum v4l2_exposure_auto_type V4L2_EXPOSURE_AU
TO 0, V4L2_EXPOSURE_MANUAL
1, V4L2_EXPOSURE_SHUTTER_PRIORITY
2, V4L2_EXPOSURE_APERTURE_PRIORITY 3, /
Additional features for digital camera
/ V4L2_EXPOSURE_ISO_PRIORITY 4, define
V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS
_BASE2) define V4L2_CID_EXPOSURE_AUTO_PRIORITY
(V4L2_CID_CAMERA_CLASS_BASE3) . . . define
V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE
18)
26
New V4L2 APIs for exposure control
define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLA
SS_BASE19) define V4L2_CID_CAM_SHUTTER (V4L2_CI
D_CAMERA_CLASS_BASE20) define
V4L2_CID_CAM_ISO (V4L2_CID_CAMERA_CLASS_BASE21)

• Supported aperture stages, shutter speed, and ISO
  speed could be different between every different
  camera module products

Supported values for aperture, shutter and ISO
should be made in V4L2_CTRL_TYPE_MENU
    .id  V4L2_CID_CAM_APERTURE,     .type  V4L
2_CTRL_TYPE_MENU,     .name  "Aperture",     .min
imum  0,     .maximum  4,     .step  1,     .de
fault_value  0, ,
static const char camera_iris_stages   / Thi
s module supports 5 Iris stages   on it's own
but F number depends  how you package
Lens module / "2.8, "4, "5.6, "8,
"11, NULL
27
New V4L2 APIs for exposure control

• New exposure metering control

/ Exposure Methods / define
V4L2_CID_PHOTOMETRY (V4L2_CID_CAMERA_CLASS_BASE17
) enum v4l2_photometry_mode
V4L2_PHOTOMETRY_MULTISEG 0,
V4L2_PHOTOMETRY_CWA 1, V4L2_PHOTOMETRY_SPO
T 2, V4L2_PHOTOMETRY_AFSPOT 3,
28

• New V4L2 APIs for digital camera
• Lens control

29
New V4L2 APIs for lens control

• Need for enhanced Focus mode API
• Current V4L2 framework just slightly covers
  MANUAL AUTO FOCUS mode for usb webcam
• High-end camera modules support for various focus
  mode presets like
• AF Macro
• AF-S / AF-C

For compatibility Focus control API should be
defined
30
New V4L2 APIs for lens control

• Focus Mode Enumeration values

define V4L2_CID_FOCUS_ABSOLUTE
(V4L2_CID_CAMERA_CLASS_BASE10) define
V4L2_CID_FOCUS_RELATIVE (V4L2_CID_CAMERA_CLASS_BA
SE11) define V4L2_CID_FOCUS_MODE
(V4L2_CID_CAMERA_CLASS_BASE13) / Focus
Methods / enum v4l2_focus_mode
V4L2_FOCUS_AUTO 0, V4L2_FOCUS_MANUAL
1, V4L2_FOCUS_MACRO 2,
V4L2_FOCUS_CONTINUOUS 3,
31

• New V4L2 APIs for digital camera
• Object recognition

32
What is object detection?

• Object detection

Gee! It recognized my face! Hello everyone
33
What should we consider?

• What do we set?
• Detect mode
• Type of object
• How many objects to detect
• Triggered action
• What do we get?
• How many objects are detected
• Detected objects type
• Detected objects coordinates

34
New V4L2 APIs for object recognition

• New API for object recognition
• New capability

define V4L2_CAP_ASYNCIO 0x02000000 / async
I/O / define V4L2_CAP_STREAMING 0x04000000 /
streaming I/O ioctls / define
V4L2_CAP_OBJ_RECOGNITION 0x10000000

• New IOCTL

define VIDIOC_S_RECOGNITION _IOWR ('V', 85,
struct v4l2_detect) define VIDIOC_G_RECOGNITION
_IOR ('V', 86, struct v4l2_detect)
35
New V4L2 APIs for object recognition

• New API for object detection

/ Object detection and triggered actions
/ enum v4l2_recog_mode V4L2_RECOGNITION_MO
DE_OFF 0, V4L2_RECOGNITION_MODE_ON 1,
V4L2_RECOGNITION_MODE_LOCK 2, enum
v4l2_recog_action V4L2_RECOG_ACTION_NONE
0,/ only detection / V4L2_RECOG_ACTION_BLINK
1,/ Capture on blinking /
V4L2_RECOG_ACTION_SMILE 2,/ Capture on smiling
/ enum v4l2_recog_pattern
V4L2_RECOG_PATTERN_FACE 0 / Face /
V4L2_RECOG_PATTERN_HUMAN 1 / Human /
V4L2_RECOG_PATTERN_CHAR 2 / Character /
36
New V4L2 APIs for object recognition

• New API for object detection

struct v4l2_recog_rect enum v4l2_recog_pat
tern p / What pattern detected /
struct v4l2_rect o / detected area /
__u32 reserved4 struct
v4l2_recognition_data __u32 detect_cnt /
detected object counter / struct v4l2_recog_r
ect obj / detected area /
__u32 reserved4 struct v4l2_recognition
enum v4l2_recog_mode mode enum
v4l2_recog_pattern pattern / What pattern to
detect / __u32 obj_num / How many object
to detect / __u32 detect_idx / select
detected object / struct v4l2_recog_data data
/ read only / enum v4l2_recognition_action
action __u32 reserved4
37
What should we consider?

• What do we set?
• Detect mode
• How many objects to detect
• Triggered action
• What do we get?
• How many objects are detected
• Detected objects coordinates

38
New V4L2 APIs for object recognition

• New API for object recognition
• New capability

define V4L2_CAP_ASYNCIO 0x02000000 / async
I/O / define V4L2_CAP_STREAMING 0x04000000 /
streaming I/O ioctls / define
V4L2_CAP_OBJ_RECOGNITION 0x10000000

• New IOCTL

define VIDIOC_S_RECOGNITION _IOWR ('V', 85,
struct v4l2_detect) define VIDIOC_G_RECOGNITION
_IOR ('V', 86, struct v4l2_detect)
39
New V4L2 APIs for object recognition

• New API for object detection

/ Object detection and triggered actions
/ enum v4l2_recognition_mode
V4L2_RECOGNITION_MODE_OFF 0,
V4L2_RECOGNITION_MODE_ON 1,
V4L2_RECOGNITION_MODE_LOCK 2, enum
v4l2_recognition_action V4L2_RECOG_ACTION_NO
NE 0,/ only detection /
V4L2_RECOG_ACTION_BLINK 1,/ Capture on
blinking / V4L2_RECOG_ACTION_SMILE 2,/
Capture on smiling / struct
v4l2_recognition_data __u8 detect_cnt /
detected object counter / struct v4l2_rect o
/ detected area / struct
v4l2_recognition enum v4l2_recognition_mode
mode __u8 obj_num / How many object to
detect / __u8 detect_idx / select
detected object / struct v4l2_recognition_dat
a data / read only / enum
v4l2_recognition_action action
40

• New V4L2 APIs for digital camera
• Dual camera

41
Dual camera for 3G handset

• What is dual camera?
• Dual camera applications
• Video telephony
• Camera applications
• Taking self portrait shots

42
Dual camera for 3G handset

• H/W restrictions in Application Processors
  camera peripheral interface

43
Why should we make single device node?

• H/W restrictions in Application Processors
  camera peripheral interface

SINGLE camera interface on Application
Processor !
Only one camera could be handled at a time
Only one camera could be using data pins at a time
Each camera is different in
Working resolution (SYNC)
Working MCLK
Working PCLK
44
What do we have in V4L2?

• We already have API for dual camera
• Enumerate video inputs
•  Query or select the current video input

VIDIOC_ENUMINPUT
VIDIOC_G_INPUT
VIDIOC_S_INPUT
45
What do we need to consider for dual camera?

• What do we need to consider?

• Let user space application get noticed about
  camera input devices
• Dont forget VIDIOC_ENUMINPUT
• Check out supported pixel format using
  VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES

• Context of each camera module
• Capability Supported color spaces, resolutions,
  additional functions
• Latest working status resolution, color space,
  effect and so on

• Context of camera interface
• Working information of the camera module getting
  switched Clock information and so on

46
Example of switching camera
Mega camera
VGA camera

• How to switch active camera

Mega Enable
VGA Enable
Condition 1 How to enable camera module Give
power source Enable expecting camera Give MCLK
Mega Reset
VGA Reset
CAM POWER
MCLK
DATA PINS
CPU
Condition 2 Shared things Camera power enable
pin Camera data pins MCLK
Condition 3 Dedicated things Camera enable
pins Camera reset pins
47
Abstract work flow of switching camera
Cam power off
Camera (B) Enable pin on
Camera (A) enable pin off
Cam power on
Switch to camera (B)
STREAMOFF
STREAMON
Camera (A) initialize
Start preview
Camera (B) initialize
Stop preview
G_INPUT/ S_INPUT
Start preview
Camera interface reset setting
Reset camera interfaces current setting
Make a proper MCLK for camera (B)
Setup resolution, color-space, DMA
Stop preview Stop DMA
48

• Porting issues in SoC camera subsystem

49
I2C issue

• I2C protocol has start and stop condition
• Some camera modules generate unexpected noise
  until their power up sequence
• Makes I2C bus in bus busy state

50
I2C issue

• How to prevent this malfunctioning issue

Use a noise free on power up camera module
OR
Use a level shifter to block and ignore noise
from camera module
OR
Change functionality of I2C pins while turning on
camera module
51
Resolution issue

• All about between camera interface and camera
  module
• Expected resolution from camera interface
• Served resolution from camera module
• Symptoms depend on camera interface
• Expecting resolution is bigger than served
  resolution
• A select time out error in user space
  application or
• Preview image is floating in some direction
• Serving resolution is bigger than expected
  resolution
• A select time out error in user space
  application or
• Preview angle looks like to be zoomed in

52
Color space issue

• Color order mismatched between camera interface
  and camera module

53
Lens control issue

• Preview freezing while checking lens response
• Using while() loop while it checks response,
  preview will freeze until focusing job gets
  finished
• If the ISP driver use the kernel thread when it
  check response from current lens status could
  make smooth preview on the LCD while lens is on
  focusing.

54
Lens control issue

• Solving preview freezing while checking lens
  status

struct platform_isp const struct platform_data
pdata struct v4l2_int_device
v4l2_int_device struct i2c_client
i2c_client int reschk . . . . . . . .
. static int ioctl_s_ctrl(struct
v4l2_int_device s, struct v4l2_control vc)
struct platform_isp isp s-gtpriv struct
i2c_client client isp-gti2c_client
switch (vc-gtid) case V4L2_CID_FOCUS_AUTO
isp-gtvfm vc-gtvalue if (vc-gtvalue
V4L2_CID_FOCUS_MACRO) err
i2c_command_for_isp( client, focus_macro )
isp-gtreschk 1 break
If isp-gtreschk is Set, The kernel thread is
active.
55
Lens control issues

• Solving preview freezing while checking lens
  status

56
JPEG data synchronization issues

• JPEG has no synchronization protocol when it gets
  transferred
• All about camera interface and camera modules
  data synchronization
• When camera interface supports JPEG sync
• Follow user manual and make camera module to
  follow that protocol
• When camera interface does not support JPEG sync
• Make synchronization protocol make JPEG data to
  be synchronized with VSYNC

57

• QA

58

• Thank you