eScience Collides with the Performing Arts

1
e-Science Collides with thePerforming Arts

• Dr R P Fletcher
• University of York

2
So what is this e-Science??

• At the inception of the UK e-Science programme
  John Taylor said that
• e-Science is about global collaboration in key
  areas of science and the next generation of
  infrastructure that will enable it
• Another source says it is
• science increasingly done through distributed
  global collaborations enabled by the Internet,
  using very large data collections, terascale
  computing resources and high performance
  visualisation
• Malcolm Atkinson has said that
• e-Science is the development and exploitation of
  advanced informatics methods to support
  collaborative research in all disciplines

3
OVERVIEW

• Academic Background am I a real scientist?
• Involvement in Performing Arts am I a fraud?
• Involvement with technology am I a
  technologist?
• Technology
• Bespoke hardware and software
• Computers
• Networks
• Software
• Engaging with this technology
• Right people in the right place
• Future some ideas
• Questions whatever you want to throw at me!

4
My Academic background

• 1968 Degree in Biology
• General
• 1971 D.Phil in Neurophysiology
• Information processing in single sensory nerve
  cells
• 1975 Research Fellow in Psychology
• Risk of schizophrenia
• Psychophysiology
• 1981 Computing Service
• Graphics
• Database systems
• Programming
• Mathematical and statistical systems and
  libraries

5

• Academic Background
• Involvement in Performing Arts
• My involvement with technology
• Technology
• Bespoke hardware
• Computers
• Networks
• Software
• Engaging with this technology
• Right people in the right place
• Future
• Questions

6
My Performing Arts background

• 1965 Staged school bands
• 1968 Staged University bands
• 1971 President of the Electronic Arts Society
• 1972 Recording engineer York Musicians
  Cooperative
• 1972 Worked with Trevor Wishart
• 1973 Recording/mixing engineer campus band
• 1974 Built own synthesiser system
• 1976 Programmed FX on PDP-11
• 1978 Recording/mixing engineer campus band
  (again)
• 1991 SGI Indigos bootstrap Music Technology
• 1996 Chaotic Constructions
• 1999 RIMM
• 2005 UKLight for Collaborative Composition/Perform
  ance
• 2006 PEACHES committee of TERENA
• 2007 Content Creation tools for Digital Cinema?

7
Technology Input and Output

• Specialised software to edit synthetic speech
• User interfaces for Music Technology
• Courses in graphics for Music Technology
• External Examiner MA Digital Arts (Middlesex)
• Midi I/O systems for Sun Workstations
• Developer for GIMP (video plugins for Unix)
• Developer for XMX (collaborative X-Windows)
• Developer for .ogg audio formats
• Still ambisonic developments
• Tracking technology for Chaotic Constructions
• Graphics system designer/developer for RIMM
• Real-time Interactive Multiple Media
• PEACHES committee at TERENA
• Panel of Experts for Arts, Culture and Humanities
  E-Science

8

• Academic Background
• Involvement in Performing Arts
• My involvement with technology
• Technology
• Bespoke hardware
• Computers
• Networks
• Software
• Engaging with this technology
• Right people in the right place
• Future
• Questions

9
Technologies

• Aim to lead you through some technologies
• Ones I have worked on over the last 15 years (or
  so!)
• At some point technology overlaps e-Science
• Need to engage with the new technologies
• Need to help drive their development
• Performing Arts need new/different facilities
• either from the existing infgrastructures
• or via new facilities
• But, unless we know whats possible
• How do we know what to ask for?

10
YORKTALK

• Worked with the Language and Linguistics staff
• Their technology allowed about six synthesis runs
  
• PER DAY!
• Data transferred via Kermit file transfer
  program!
• Real informed trial and error to fix problems
• I created a graphical interface to the data
• Ran on a then high performance system
• Could interact with the data
• Synthesis was up to near real time
• Combined visualisation to see errors
• Used interactive graphics to fix the problems

11
YORK TALK

• Research vehicle for speech synthesis
• DecTalk example (may recognise the voice)
• YorkTalk examples
• Same words as DecTalk
• A sentence (with a cold!)

12
Engaging with artists

• Next project was with staff in Music
• Needed help to create interfaces for a system
• Needed some lateral thinking
• Definitely moving into the Arts domain
• High performance computing was needed
• Multiple tasks by same computer
• Video
• Audio
• Graphics
• Midi

13
Chaotic Constructions - 1994

• Certain amount of serendipity
• Peter Fluck talking on Radio 4 about Chaotic
  motion in kinetic scuplture
• Tony Myatt talked a few weeks later about Chaos
  in Music
• Phoned each other
• classic WE SHOULD TALK moment
• By 1996 developed a project
• to use the chaotic motion of a kinetic sculpture
  to drive a music composition system
• PROBLEM how to capture the motion
• ANSWER ask me

14
Chaotic Constructions the Tracking System

• Request came to me
• How can we do this?
• and
• It will be installed in Tate Modern in March 1997
• and
• It was now January 1997!
• First thoughts were to use shape tracking systems
• Problems
• most systems were proprietary and expensive
• shapes to be tracked changed because sculpture
  was 3 dimensional
• could not use trajectory prediction due to the
  chaotic motion
• ANSWER simply track unique coloured items

15
VIDEO EXAMPLE
The Making of CHAOTIC CONSTRUCTIONS
16
Stage 1 Define a Colour Fingerprint

• Program creates a set of unique fingerprints

17
Tuning the Fingerprints

• Check motion does not lose sync with colour

18
Peter Fluck Hand Cranking the Sculpture
19
Running with Tracking and Music Program

• Fingerprint used by second program
• Motion tracking is just one element

20
Changing a Tracking Point

• What are the best points to track?
• Due to chaotic motion some trial and error!

21
And finally the prototype

• we get there a small excerpt

22
And the real thing
23
Credits and Awards

• Exhibited to great acclaim in Tate Modern
• Other UK Art Galleries
• Travelled to Hong Kong and Corsica
• Herald Angel Award by the Glasgow Herald for the
  best piece of visual art at the 1997 Edinburgh
  Festival
• International Critics Award as part of the 1997
  Demarco European Art Foundation exhibition

24
Not really Performing Arts I hear you say

• In St. Ives staff gave children coloured bats
• Switched off the sculpture
• Children waved the bats around
• And, of course, made music
• Security was called once to remove a visitor
• Would not leave at closing time
• Transfixed by the whole thing
• Some people came 2 or 3 times a week
• For a fix

25
How can this Technology be used

• for Performing Arts
• Mount a camera above a stage
• Give actors coloured hats
• Now you have their position on stage
• Link their movement around the stage
• To
• Orchestrate lighting changes
• Hotspots
• Move to one place on stage and make a sound
• Detect collisions
• Allow actors to fully interact with their
  environment

26
or

• Point the camera at a tropical fish tank
• Let the fish make music

27
Bringing e-Science into the Frame

• So far York work done on a single modest computer
• Scale up
• To multiple specialist computers
• Networks
• Wireless devices
• Complex tracking
• 3D sound
• Interactive graphics
• All controlled by
• The PERFORMER

28
Real Time Interactive Multiple Media - RIMM

• Construction 3
• Realised by the Real-time Interactive Multiple
  Media Project
• University of York Music Department (UK), IRCAM
  (France) and SIMPK (Germany)
• European Commission Information Society
  Technologies Programme
• Incorporates a live performer on soprano
  saxophone
• Three-dimensional sculpture
• Computer generated graphics
• Ambisonic surround sound

29
construction 3 is a synergetic composition

• The musical and graphical elements of the piece
  have been composed concurrently
• Both the computer graphics and music are
  triggered by the saxophonists natural gestures
• Through infra-red video tracking
• Use i/r device and i/r filter on camera
• Electronic sensors
• Footswitches, accelerometers
• Machine listening
• Score following
• Gesture recognition using neural networks
• e.g. throw sound into the audience

30
Block Diagram of System
31
System Diagram
32
The Team

• Lots of people needed to make this happen
• Project manager to keep us in line
• The Composer/Sound stage designer
• The Graphic Designer
• The Graphics Programmer
• The Audio Programmer
• The Electronics technical wizard
• The Local site organiser
• The commercial suppliers
• And
• THE PERFORMER

33
Team Photo
34
Screens and Performer
35
Examples of the RIMM Computer Graphics
36
Technology becomes e-Science

• Clearly a lot of technology to make it work
• Some is bespoke electronics
• Majority of tasks were programmed
• The SGI Origin 2000
• 8 fast processors
• Highly parallel
• jMax
• 24-bit optical output for audio
• High speed network interconnect to graphics
  computer
• SGI O2 graphics
• Optimised for OpenGL
• Texture memory drawn from main memory

37

• Academic Background
• Involvement in Performing Arts
• My involvement with technology
• Technology
• Bespoke hardware
• Computers
• Networks
• Software
• Engaging with this technology
• Right people in the right place
• Future
• Questions

38
Technologies for All

• Many of these technologies are available
• Many of the ideas are now embodied in software
• Many are free
• Pure Data aka PD (with GEM add-in)
• Eyesweb
• VJ-ing is growing
• Many VJ products around
• Grid Computing is available
• Not easily accessible
• Lacks some functionality for Performing Arts
• Photonic networks are available
• High speed links point to point
• Up to 10 gb/s possible now
• 1 gb/s across the globe

39
jMax, Pure Data/GEM

• Graphics Programming
• Lots of modules
• Connect up inputs and outputs
• Uses the modular synthesiser metaphor
• GEM
• Graphics Environment for Musicians
• Link audio to graphics
• Limited by modules unless you write your own

40
eyesweb

• Video programming with modules
• Huge number of modules
• Includes man tracking capability
• Can link to rudimentary audio
• But
• Can also link to PD
• using Open Sound Control protocols - OSC

41
REACTABLE

• Collaborative electronic music instrument
• Tabletop tangible multi-touch interface
• Several simultaneous performers
• Share complete control over the instrument
• by moving and rotating physical objects
• on a luminous round table surface.
• Objects represent components of a classic modular
  synthesizer
• can create complex and dynamic sonic topologies
• a kind of tangible modular synthesizer or
• graspable flow-controlled programming language

42
REACTABLE
43
REACTABLE - demo
Click on Image to go to site with Videos will
open a browser
44
More Video Tracking and Performance
Messa di Voce Tmema,Blonk,La Barbara http//www.tm
ema.org/messa/messa.html
45
Messa di Voce - demo
Click image to go to the Host site will try to
play Quicktime in a browser
46
And just for fun another demo
47

• Academic Background
• Involvement in Performing Arts
• My involvement with technology
• Technology
• Bespoke hardware
• Computers
• Networks
• Software
• Engaging with this technology
• Right people in the right place
• Future
• Questions

48
So what is this Grid Computing?

• The term originated in the early 1990s
• Metaphor for making computer power as easy to
  access as an electric power grid
• Many years on and still mainly in the domain of
  BIG SCIENCE
• Most Grids are run like 80s batch style
  computing
• Submit a job description to a scheduler
• It decides when and where the job runs more or
  less
• This is a computational grid
• Many computers and clusters are interconnected
• Can be globally distributed via photonic networks

49
What is this Data Grid?

• A grid computing system that deals with data
• The controlled sharing and management of large
  amounts of distributed data
• These are often, but not always, combined with
  computational grid computing systems
• Finding data is not easy
• Accessing it is not easy
• Mainly in the domain of BIG SCIENCE
• Some shift to Arts and Humanities
• We have lots of data too!
• More later in context of National Grid Service

50
What are these Photonics Networks

• A worldwide network of optical fibres
• Speeds are 1gbit/s to 40 gbit/s
• Run by different organisations
• UK - Janet
• Europe - GEANT2
• USA - StarLight
• USA - National Lambda Rail
• Canada CANARIE
• etc
• A connection is usually point-to-point
• Very exclusive
• Lots of bandwidth
• Low latency and jitter (more of this later)

51
What might I use such links?

• Many examples
• Interactive dance
• Use HDTV rather then Lo-res AccessGrid technology
• Collaborative audio and video
• Master classes are more commonplace
• Collaborative tools for distributed, remote teams
• Digital cinema (4k x 2k) is used for
• Entertainment, media, art and culture
• Science, medicine, education and research
• and also
• Military, intelligence, security and police

52
Nagano Winter Olympics 1998

• Massive endeavour with live networked music
• Opening ceremony had a chorus in
• Berlin
• Cape Town
• Beijing
• New York
• Sydney
• Performed the final movement of Beethoven's 9th
• Together with the full symphony orchestra in
• Nagano Kenmin Cultural Centre Hall.
• Also
• 2000-person chorus in the Olympic stadium

53
Nagano Winter Olympics

• The final result was mixed and broadcast
  worldwide
• This setup required
• 7 satellites
• 12 uplinks with encoders and decoders
• Multiple-delay compensation
• The longest delay was to and from Berlin
• Around 4 seconds!
• Incredibly
• It all worked and the result was orchestras and
  choirs all in time

54
Orchestra Rehearsals

• New World Symphony in Miami rehearsed pieces by
  French composers
• The composers participated live from IRCAM in
  Paris
• This transatlantic rehearsal enabled live
  exchanges between musicians in Miami and the
  composers in Paris
• Via RENATER-4 in France, GEANT2 in Europe, and
  Internet2
• These advanced research and education networks
  provided the bandwidth and quality of service to
  ensure high quality audio and video

55
Using High Speed Networks for Arts

• Cinegrid is a good Example
• Cinegrids Mission Statement
• To build an interdisciplinary community that is
  focused on the research, development, and
  demonstration of networked collaborative tools to
  enable the production, use and exchange of
  very-high-quality digital media over photonic
  networks
• See www.cinegrid.org for more details

56
Cinegrid

• Converging on the digital
• Fast networking
• Access to shared instruments, computers, storage
• Collaboration tools
• Robust security for intellectual property
• High visual quality
• Greater speed
• More distributed applications
• and, importantly
• Next generation of trained professionals

57
Cinegrid_at_AES October 25th 2006

• This event was structured in four acts
• Each demonstrating a different facet of the
  CineGrid philosophy of networked extreme media
• In Act 1, a sequence of 4K digital shorts at 24
  frames per second (fps), together with fully
  mixed synchronized audio, were pulled in real
  time from network-connected servers in Los
  Angeles and San Diego.

58
Cinegrid_at_AES October 25th 2006

• In Act 2, 4K tele-presence was used for
  interactive video-conferencing and
  ultra-realistic reproduction of a classical music
  performance from Tokyo

59
Cinegrid_at_AES October 25th 2006

• Acts 3 and 4 were designed to prove the concept
• networked, remote audio post-production for
  digital cinema
• by creative teams spread around the world
• who demand the highest-quality production values
• In Act 3
• 4K motion pictures were sent compressed from
  Tokyo
• 24-channel non-compressed digital audio was
  streamed from San Diego
• In Act 4
• The performance system was re-configured
• to use uncompressed 2K motion pictures coming
  from ILM servers in the LDAC facility,
  synchronized to 24-channel, non-compressed
  digital audio streaming from San Diego

60
Cinegrid_at_AES October 25th 2006

• Was a good test of the CineGrid concept of
• Using 4K cameras
• Multi-channel, non-compressed audio
• Ultra-realistic live experiences of music
  concerts
• and
• other kinds of performing arts
• To distant audiences in theatres
• Connected by high-speed networks

61
Cinegrid_at_AES October 25th 2006
Tomonori Aoyama, Professor at Keio/DMC and
Chairman of the Digital Cinema Consortium of
Japan (DCCJ) said

• "I believe the very high-speed optical networks
  such as now being deployed by research
  organizations will become an essential
  infrastructure for digital cinema production and
  distribution
• "But, we still have to learn how to integrate
  systems that creative people can use to make
  beautiful 4K content - picture and sound - in new
  ways appropriate for the 21st century.
  Demonstrations such as CineGrid_at_AES force us, in
  a good way, to learn by doing."

62
UK National Academic Networks

• UKERNA
• UK Education and Research Networking Association
• Now renamed to JANET
• Operate the standard IP service
• What most of use use daily
• With the rollout of SUPERJANET-5 we now have
• JANET Lightpath Service
• Evolved from UKLight
• See www.uklight.ac.uk for details
• Various research projects
• Many coordinated from NeSC and the eSI

63
What is a lightpath?

• End-to-end Network Capacity
• Point-to-point circuits
• Capacities from about 50Mbit/s to 10Gbit/s
• Presentations
• Ethernet 100/1G/10G (WAN/LAN)
• SDH STM-1/4/16/64
• Implemented over JANET optical transmission
  infrastructure
• including SDH equipment re-deployed from UKLight

64
Janet Networks
Research capacityUKLight
JANET IP
AdditionalWavelengths
JANET Optical Transmisson
65
SUPERJANET-5 Integrated with UKLIGHT
66
Getting a Lightpath - JANET Policy

• Subject to capacity and budget availability,
  lightpaths will be free at point of use up to
  1Gbit/s
• Tariffs available for 10Gbit/s
• Delivered to Institutions at the JANET Access
  Point
• Institutions must commit to extending the
  lightpath across their networks
• See www.ja.net/lightpath for details
• Documents shortly
• First public version in final editorial stage
• Service description and application lightpath
  application form

67
High Performance Computing

• The more power we need the less we can afford it
• Not sensible to have massive systems duplicated
• Look towards the grid for power
• But, what if we need it in real time?
• Current technology geared towards batch computing
• Some notable exceptions
• Interactive visualisation
• Computational steering
• VLBI
• Some CERN LHA experiments
• Data reduction in real time

68
Grid Computing in the UK

• National Grid Service (NGS)
• NGS-1 4 core sites with compute and TB storage
• NGS-2 Officially in pre-production as of 20th
  June
• Sevices
• Job Submission via
• Globus, GridSAM, Condor, gLite
• or via portals and applications repositories
• to simplify job submission for users who do not
  wish to delve into the technical details of grid
  computing
• Grid Support Centre to help users

69
Data Grid in the UK

• NGS offers
• Oracle 10g
• Storage Resource Broker (SRB)
• OGSA-DAI
• VOMS
• Oracle 10g high performance database system
• SRB - data storage service
• store your data in geographically distributed
  locations
• access them all using only the logical filename
• Reliabilty and efficiency are improved
  dramatically
• OGSA-DAI
• The Open Grid Services Architecture Data Access
  and Integration
• middleware that helps users access and integrate
  data
• from separate structured data sources e.g. XML or
  Oracle

70
VOMS

• Virtual Organisation Membership Service
• To help grids manage the authorization of their
  users
• Delegates the approval of users to the to the VO
  itself
• Thus removing the onus upon the end user to
  register with each resource they might use as
  part of the VO
• NGS resources are in the process of adopting VOMS
  to authorize NGS members
• In the future the number of VOMS-aware resources
  is likely to increase

71

• Academic Background
• Involvement in Performing Arts
• My involvement with technology
• Technology
• Bespoke hardware
• Computers
• Networks
• Software
• Engaging with this technology
• Right people in the right place
• Future
• Questions

72
High Performance Computing

• Wavefield Synthesis is a good example
• Computationally demanding
• Desktop machines and small clusters not capable
• Can we have real time grid computing please?
• If not, will someone fund a big lorry
• With lots of computers
• Lots of aircon
• Plug in 3-phase power at one end
• Gigabit ethernet comes out the other end!

73
Wavefield Synthesis what, why, when?

• The limitation of stereo or ambisonic techniques
• it only works perfectly well for one listener
• positioned on the so-called sweet spot
• Thus in common concert environments
• the intended effect of movement of the sound will
  in these cases not be heard by a majority of the
  listeners
• Wave field synthesis can overcome this limitation
• Can provide a good perceptual localisation in a
  relatively large listening area
• This makes the technique ideal for concert
  environments

74
Wavefield Synthesis Simplistic overview

• Sound is waves
• Next wave can be synthesised by a infinite number
  of small speakers
• Need to approximate infinite

75
Wavefield Synthesis the reality

• Various experimental speaker arrays

76
Exemplar Venue

• RECOMBINANT MEDIA LABS, SF
• Founded to create, research, and portray Spatial
  Media Synthesis
• The science of projecting image and sound objects
  in 3-dimensional space
• RML cultivates radical methodologies for
  experiential engineering
• Exploring performative processes which expand the
  formal aesthetic and technological boundaries of
  immersive installation and surround cinema

77
RML

• The San Francisco facility contains
• A flexible black box environment that houses
• A high definition multi-channel audio-visual
  system
• known as Surround Traffic Control
• Full fidelity array consists of a design
  specification for
• 10 screens in 360 degrees
• supported by an ultra impact 16.8.2 horizontal
  and vertical sound diffusion system
• scalable for all types of rectangular rooms
• High Speed Networking

78
Summary

• Seen the evolution of some technology
• The newer technologies require
• High performance computing
• Very high speed networks
• Massive storage
• New breeds of creative artists
• Many different experts in the technologies
• Networking of experts
• New theatres
• Investment
• and much much more

79
ANY QUESTIONS?