Greg Wilson

1
CSC407 Software ArchitectureSummer 2006Pipes
and Filters

• Greg Wilson
• BA 3230
• gvwilson_at_cs.utoronto.ca

2
Overview

• Some things can only really be learned by doing
• So let's do some software architecture
• Which really means doing design
• First target pipes and filters
• One of the reasons for Unix's success and
  longevity
• The world's first (and simplest) component object
  model

3
High-Level View

• Want to be able to do complex data processing by
  composing simple operations in flexible ways
• More specifically
• Data to process is a stream of records (possibly
  of infinite length)
• Processing is done by filters, whose only
  interactions are the effects they have on the
  data streams
• Restrict model to directed acyclic graph (DAG)

4
System Components

• Filters encapsulate operations
• Pipes connect filters
• Sources provide data
• Sinks consume data

source
sink
in.txt
grep
sort
out.txt
filter
pipe
5
A More Elaborate Example
program text
semantic checker
bytecode generator
scanner
parser
program




error stream

• We need a join as well
• which implies a tee

6
A More Elaborate Example
in.txt
cat
tee
out.txt
sort
comm
sort

• mknod pipeA p
• mknod pipeB p
• sort pipeA gt pipeB \
• cat in.txt tee pipeA sort u \
• comm 13 pipeB gt out.txt

7
Going All The Way
8
Back to the Future

• Microsoft PowerShell (formerly Monad) passes
  objects instead of strings
• "Revolution" is not too strong a word
• Shame about the syntax, though

• get-process where _.handles -gt 500 sort
  _.handles
• Handles NPM(K) PM(K) WS(K) VS(M) CPU(s) Id
  ProcessName
• ------- ------ ----- ----- ----- ------ ----
  ----------
• 516 13 1676 1476 23 68.6 1088
  svchost
• 546 10 4464 2760 44 7772.7 832
  lsass
• 582 24 27148 7708 129 1777.3 1440
  msnmsgr
• 599 60 7868 4644 56 110.3 776
  winlogon

9
CRC Cards
Class pipe
Collaborators Source Sink Filter
Class filter
Collaborators Pipe

• Responsibility
• Gets input data
• Transforms data
• Supplies output data

• Responsibility
• Transfer data
• Buffer data
• Synchronize

Class sink
Collaborators Pipe
Class source
Collaborators Pipe

• Responsibility
• Deliver data

• Responsibility
• Consume data

10
Who's Driving?

• Option 1 active filters push data into pipes
• But what about sinks?
• Option 2 passive filters are driven by pipes
• But what about sources?
• Option 3 a controller manages the data flow
• Note have not yet considered
• Whether processing runs in parallel or not
• Whether components can be distributed

11
Active Filters

• Filter is a loop that pulls input from one pipe
  and pushes output into another
• E.g., stdin and stdout by default

while true data inputPipe.read() data
transform(data) outputPipe.write(data)
12
Passive Filters (Pull)
source
filter 1
filter 2
sink

• The push strategy is the converse

13
Controllers

• There are no "pipes" per se
• Just call filters in a loop
• Problems with concurrency and composability
• But it makes it a lot easier to pass objects
• and avoids data-copying performance problems

while true data input.read() for f in
filters data f(data) output.write(data)
14
Design Questions

• Data format?
• Just one, or many?
• How to signal end-of-stream?
• How to implement pipe/filter connections?
• How to implement error handling?
• Use the Decorator design pattern?
• What others apply?