FPGA

1
FPGA Camera Sensors
CISC 849
11/16/2006

• by Juergen Vogt

2
Outline
Research Background
Camera Design
Class Project

• Research Background
• The Application
• Application Requirements
• The Whole System
• Camera Design
• The Acquisition System
• CCD Technologies
• The Camera
• Software Overview
• Class Project
• Reinhard Tone Mapping

3
Application Neuro Imaging
Research Background
Camera Design
Class Project

• What is neuro imaging ?
• Injecting a fluorescent substance into an object
  of interest (e.g. brain)
• Stimulating the object (e.g. breathing)
• Watching the change in florescence upon
  stimulation
• What is the goal of neuro imaging ?
• Understanding which areas of the brain are more
  or less neuronal active or responsible for actions

4
The Resulting Sequence
Research Background
Camera Design
Class Project
Image Sequence mapped to respiratory cycle
5
Application Requirements
Research Background
Camera Design
Class Project

• Low noise
• fluorescent dyes provide only 0.05-10 changes in
  intensity
• High-spatial resolution Individual neurons are
  5-40 mm
• Either high speed acquisition or long exposure
• Programmable camera depending on experiments
• Fast recording capabilities
• Individual events may only have 2ms duration
• Long term recording
• Additional electrophysiology data feeds
• Merge data in real time with the images
• Triggered imaging
• Additional trigger inputs to capture images
• Lab environment
• Easy to use Plug Play camera

6
The Whole System
Research Background
Camera Design
Class Project
Custom HW Custom SW
Off shelf HW Custom SW
Off shelf HW Off shelf SW
Camera
Control, Setup programming
Non real time storage evaluation
Real time acquisition recording
7
How do we start ?
Research Background
Camera Design
Class Project

• Choice of CCD
• Low noise
• HDR
• Reasonable priced
• Choice of interconnect
• Large throughput
• Sustained throughput
• Reliable throughput
• Choice acquisition system
• Enough short term storage
• Enough processing power
• Compatible with interconnect

8
The Usual CCD
Research Background
Camera Design
Class Project
Light
Unenclosed light sensitive area for acquisition
Enclosed light sensitive area for storage
Serial read out register
9
Electron Multiplier CCD
Research Background
Camera Design
Class Project
Light
Unenclosed light sensitive area for acquisition
Enclosed light sensitive area for storage
Electron multiplier
Serial read out register
10
EMCCD Requirements
Research Background
Camera Design
Class Project

• EMCCD requires 13 control signals
• Low noise high speed clock signals with huge
  voltage swing required
• Accurate phase shifted or random pattern
• All frame rate dependent

11
The Interconnect
Research Background
Camera Design
Class Project

• CameraLink benefits
• Compatibility with existing imaging systems
• Proven functionality
• Open standard
• 24 bit _at_ 85 MHz 2 GBit/s
• 250 MB/s
• Standard causes cheaper and more reliable
  development hardware, software and parts
• (e.g. 3 ft. of cable for just 90,-)

12
The Acquisition System
Research Background
Camera Design
Class Project

• Dual Xeon 3.2 GHz HT
• up to 16 GB RAM
• 150 MB/s RAID 0
• Gidel prototyping board
• PCI-X 100 compatible
• Altera Cyclone II 35 FPGA
• DRAM 64MB
• Expandable to up to 16GB
• PCI-X sustained throughput 400MB/s
• I/O connections 314
• CameraLink interface daughter board
• Providing 2 CameraLink 26 pin MDR ports

13
Camera Requirements
Research Background
Camera Design
Class Project

• EMCCD
• Voltage regulators
• Low noise
• A/D Converter
• Digitization of the image with a high speed
• FPGA
• CCD exposure and readout control
• Post processing of the image, Compression,
  filtering requires a powerful FPGA
• Additional inputs
• Trigger inputs for capture on demand
• Electrophysiology data
• CameraLink

14
Chosen Components
Research Background
Camera Design
Class Project

• EMCCD TI TC253SPD-BD 656x496, 14 bit HDR with
  build in peltier cooler
• Linear Technology LTC2284 ADC
• 14 bit 105 Msps dual ADC
• Xilinx Virtex II Pro (speed grade -6)
• Power PC 405 RISC processor
• 11088 logical cells (FF 4 bit LUT Carry
  Logic)
• National Semiconductors DS90287 LVDS Serializer
• Dimensions 30x10 cm (12x4 inches)
• 6 layer PCB to provide enough space for routing

15
Decisions Restrictions
Research Background
Camera Design
Class Project

• Peltier controller will be mounted on a socket on
  top of the board to save space
• Early digitization close to the imaging sensor
• CCD needs to be placed
• at the end of the board
• not to block the viewer
• Mounting holes need
• to be specified early in order
• to manufacture a case
• Split up in analog and digital
• side to reduce noise
• Analog signal path shielding

Camera
16
The PCB Board
Research Background
Camera Design
Class Project
Analog power reg.
Digital Power reg.
CLK
Switches
SerDes
CCD
ADC
MDR26
FPGA
Proms
Analog signal path
CLK
Analog power reg.
Analog side
Digital side
17
Timeline
Research Background
Camera Design
Class Project

• Feasibility studies for CameraLink transmission
• Draw check schematics / limited simulations
• Generating pads, footprints symbols
• Part layout and physical considerations
• Power wiring
• Signal wiring
• Fabrication
• Assembly
• Test
• FPGA programming
• PCI-X card programming
• Software adaptation
• Feature implementation

18
Software Overview
Research Background
Camera Design
Class Project
Async. Control
Camera VHDL
PCI-X Card VHDL
DMA
RAM
Async. Control
Interrupt
Imaging Software C
Display
HDD
19
Camera State Machine
Research Background
Camera Design
Class Project
Exposing
Storing
Reading
Sending to PC
Async. Receive
Different colors represent different clock
domains or asynchronous signals
20
Camera State Machine
Research Background
Camera Design
Class Project
Exposing
Storing
Reading
Pixel Processing
Sending to PC
Image Processing
Async. Receive
Different colors represent different clock
domains or asynchronous signals
21
Class Project
Research Background
Camera Design
Class Project

• Should be implemented with or on camera device
• Needs easy to computation
• Needs fully running camera
• A filter or processing method that applies to
  neuronal imaging
• Tone mapping
• Complex evaluation algorithms
• Should improve the overall system performance
• Take load off the acquisition system
• Should encourage further development
• Need to be easy expandable
• Need to be a separable part of the system

Leads to Reinhard Tone Mapping
22
Reinhard Tone Mapping
Research Background
Camera Design
Class Project

• Tone Mapping
• Easy to compute
• Pixel processing and image processing involved
• Photography background
• Parameters can be adjusted to neuronal imaging
• VHDL template needed
• Further filters can be implemented easily
• Output is highly valuable
• Ability to run the camera without an acquisition
  system

23
Local Global Tone mapping
Research Background
Camera Design
Class Project

• Global TM
• Zones are defined once
• Mapping is only defined once
• Easy to compute
• Bad detail preservation

• Local TM
• Zones are defined once
• Mapping is defined for each pixel
• Hard to compute
• Details are preserved

24
Tone Mapping Zones
Research Background
Camera Design
Class Project

• Zones are on a logarithmic scale
• Real world zones have to be mapped to the zones
  of the display medium
• 14 bit HDR image maps to 14 zones
• 8 bit grey scale is only 8 zones

Real time 641 compression
25
Global Tone Mapping
Research Background
Camera Design
Class Project
Computing the average world luminance
Scaling the luminance by a percentage a of the
average
Scaling by 1/L for large L Scaling by 1 for small
L
Scaling by 1/L for large L Scaling by 1 for small
L Highest luminance mapped to white
26
Global Tone Mapping Curve
Research Background
Camera Design
Class Project
27
Project proposal
Research Background
Camera Design
Class Project

• Implementation of Global Tone Mapping in VHDL
• Proof of concept for real time filtering inside
  the camera
• Gaining a VHDL filter template for further use
• Reduce processing in acquisition system
• Camera could be used without PC in the future

28
Summary
Research Background
Camera Design
Class Project

• Flexible research platform for
• Neuro imaging
• Signal processing
• Image processing
• Electrical engineering
• Open platform with available hardware design
  source code
• FPGA inside the camera
• High flexibility to adopt camera for different
  needs in different experiments or complete
  different applications
• Cheaper camera for research
• Class project contributes to current research