Constraints for Multimedia Presentation Generation

1
Constraints for Multimedia Presentation Generation

• Joost Geurts,
• Multimedia and Human-Computer Interaction
• CWI Amsterdam
• email Joost.Geurts_at_cwi.nl

2
Talk overview

• Generating multimedia automatically
• Cuypers multimedia generation engine
• Multimedia and constraints
• Quantitative constraints
• Qualitative constraints
• Cuypers demo
• Conclusion, future directions

3
Multimedia Presentation

• Multimedia Presentation
• Image, Text, Video, Audio
• Based on Temporal and
• Spatial Synchronization
• Multimedia Document
• SMIL, SVG, HTML
• WYSIWYG
• Static Content
• Problem Dynamic Content

4
Generating adaptive multimedia

• Content
• Large multimedia database
• System profile
• PC, PDA, WAP
• Network profile
• Modem, Gigabit
• User profile
• Language, Interests, Abilities, Preferences
• Too costly to author manually

5
Cuypers multimedia generation engine
6
Cuypers multimedia generation engine

• Cuypers is based on
• media independent presentation abstractions
• transformation rules with built-in backtracking
  andconstraint solving

7
Semantic structure
Author does not specify complete
presentationbut only rhetoric relations
8
Communicative Devices
rhetoric relations are than transformed into
presentation independent communicative devices
9
Automatic multimedia generation

• Designer does not specify complete
  presentationbut only specifies requirements
• System automatically finds a solution which meets
  requirements
• How should the requirements be specified?
• Declarative constraints

10
Constraint satisfaction

• Constraints occur often in our daily lives
• Agenda, Travelling, Shopping
• Constraint paradigm for Problem Solving
• Declarative
• Used for problems with
• Many variables
• Large domains
• Based on domain reduction paradigm

11
Intelligent reduction of possible values

• X ? 1,2,3,4,5,
• Y ? 1,2
• X ? Y

• X ? 1,2,
• Y ? 1,2
• X ? Y

12
Traditional use of constraints

• Quantitative constraints
• Integer domain
• Reduction by arithmetic relations
• Greater than (gt)
• Less than (lt)
• Equals ()
• Example
• (x lt y x ? 0..10, y ? 5..10 )
• (x ? y ? z 3 , x u ? 1 x ? ? , y ? ? ,
  z? ? , u ? ? )

13
Solving a Constraint Satisfaction Problem

• Problem
• SEND
• MORE
• MONEY
• Modeling
• 1000 x S 100 x E 10 x N
  D
• 1000 x M 100 x O 10
  x R E
• 10000 x M 1000 x O 100 x N 10 x E Y
• Domain reduction / Search
• Solution
• S9, E5, N6, D7, M1, O0, R8, Y2

14
Quantitative Constraints in Multimedia
Communicative devices generate constraint-graph
which the system tries to satisfy
15
Drawbacks of quantitative constraints

• Too many (trivial) solutions that differ by
• 1 pixel position, or
• 1 milliseconds in timing
• Not sufficiently expressive
• cannot specify no overlap constraint
• Too low level
• A.X2 ? B.X1

16
Allens 13 temporal relations
Allens relations are used for both spatial and
temporal lay-out
17
Solution qualitative constraints

• For non-typical domains
• Boolean,
• Three valued logics,
• Allens relation
• Advantages for Multimedia generation
• More intuitive
• More expressive
• Smaller domains

18
Domain Reduction Rules

• Inverse
• A before B ? B after A
• A equal B ? B equal A
• Transitive
• A before B , B before C ? A before C
• A overlaps B, B during C ? A overlap C or
• A during C or
• A starts C
• Equals
• A overlap C, A o,d,s C ? A overlap C

19
Qualitative Constraints
Qualitative solutions translate automatically
to lower level quantitative constraints
20
New problem What if constraints are insoluble?
Solution Constraint Logic Programming

• Combine Prolog unification and backtracking with
  constraint solving
• Use Prolog rules to generate constraints
• Backtrack when constraints are insoluble

21
Cuypers generation engine

• Multiple layers
• Communicative devices generate constraints
• Qualitative constraints translate to quantitative
  constraints
• Solution of both constraints provides sufficient
  information for final presentation

22
Cuypers demo scenario

• Client
• Server
• Server
• Server
• Server
• Client

• User is interested in Rembrandt and wants to
• know about about the chiaroscuro technique
• Query database
• Generate constraints according to
• System profile
• User profile
• Network profile
• Solve constraints / revise constraints
• Generate SMIL presentation
• Play presentation

23
Conclusions

• Quantitative constraintsare insufficient for
  automatic multimediapresentation generation.
  Also need
• Qualitative constraintsto allow intuitive and
  effectivehigh level specification, and
• Backtrackingfor revising specific
  constraintswhich otherwise cause the entire set
  to fail

24
Discussion

• Labeling
• Choice of candidate variable
• Choice of candidate value
• Transitive Reasoning Rule
• Infer implicit relations
• Redundant
• Allens Relations
• Not very well suited for generating MM
• Non interactive

25
Future directions

• Best-first instead of depth-first
• Choose best among possible solutions
• Needs evaluation criteria
• Improve knowledge management
• Make design knowledge declarative and explicit
• Preserve metadata in final presentation
• Use standardized and reusable profiles

26
Thank you
27
Need to make trade-offs

• Semantics
• Convey message
• Aesthetics
• Clear / nice layout
• Resources
• Screen size, bandwidth
• Dimension may result in conflicting goals

28
Quantitative Constraints

• csp(Ids, -Boxes)
• csp(IdA,IdB,box(IdA,x1AX1, ),
  box(IdB,x1BX1,)) -
• get values
• maxX(MaxX), maxY(MaxY),
• height(IdA,HeightA),
• widtht(IdA,WidthA),
• define domains
• AX1,AX2,BX1,BX20..MaxX,
• AY1,AY2,BY1,BY20..MaxY,
• set width height
• AX2 AX1 WidthA,
• AY2 AY1 HeightA,
• constraints
• AX2 lt BX1, A left-of B
• AY1 BY1, A top-align B,

29
Multimedia and Constraints

• Constraint Logic Programming
• Domain reduction
• Backtracking
• Unification (matching rules)
• Qualitative Constraints
• Non-integer domain
• Allens 13 temporal interval relations in three
  dimensions

30
Qualitative Constraints

• Example
• Two images, A,B
• A left or right of B
• A not above or below B

31
Qualitative Constraints
csp(Ids, -Graph) csp(IdA, Idb,
edge(IdA,IdB,x,NoOverlap),) - define
domains NoOverlap b,b-,m,m-, Overlap
d,d-,s,s-,f,f-,e, constraints edge(IdA,Id
B,x,NoOverlap), B not-overlap
A edge(IdA,IdB,y,Overlap), B overlap
A true.
32
Qualitative Constraints

• Reasoning
• Inverse
• edge(A,B,D,Value) ltgt inverse(Value,RValue),edge(
  B,A,D,RValue).
• Equality
• edge(A,B,D,V1), edge(A,B,D,V2) gt V1 V2
• Transitive
• edge(A,B,D,VAB), edge(B,C,D,VBC) gt
• tr(VAB,VBC,VAC), rule generation algorithm
• edge(A,C,D,VAC).
• Translation rules to quantitative domain
• edge(A,B,D,b) gt node(A,D/2,V2), node(B,D/1,V1)
• V1 lt V2.

33
Problems in generating multimedia

• Text documents are flexible
• Add page, scrollbar,
• Template models
• Wrap text around images
• Multimedia documents are less flexible
• No pages or scrollbars, no line-breaking or
  hyphenation
• Not based on text-flow
• Feedback needed
• Linear process model does not work for multimedia

34
Quantitative Constraints

• Example
• Two images, A,B
• A left-of B
• A top-align B

35
Cuypers generation engine

• Rhetoric/Semantic
• Sequence, Example
• Communicative devices
• Bookshelf, Slideshow
• Qualitative Constraints
• A before B
• Quantitative Constraints
• A.X2 lt B.X1
• Presentation
• SMIL