Andrew Salway a.salwaysurrey.ac.uk

1
Computational Semiotics an analogy with
Computational Linguistics?

• Andrew Salway (a.salway_at_surrey.ac.uk)
• Dept. of Computing, University of Surrey
• 29 June 2000
• Workshop on Computational Semiotics for New Media

2
Compound Terms

• Computation the action or process of
  reckoning calculation the use of a computer
  (SOED)
• data structures and algorithms for information
  processing GUIs for I/O
• Semiotics science of signs
• Computation Semiotics
• ?? doing semiotics on computational systems
• ?? doing computation grounded in semiotics

3
Computational

• biology, mathematics, physics, LINGUISTICS
• Symbiotic relationship between the disciplines
  e.g. linguistics elaborates structures and
  processes for dealing with language (lexical,
  syntactic, semantic, pragmatic) computing
  science necessitates and assists in the formal
  descriptions of these
• Texts (inc. speech/handwriting) ? Representations
  of meaning Keyword (IR) Template (IE) Deep
  semantics (NLU) also text generation
• Consider machine translation from speech signal /
  ASCII strings to speech signal / ASCII strings
  directly
• ? Degree to which computational techniques equate
  with theory

4
Surreys Computational Semiotics

• Digital reproductions of multimedia artefacts,
  including paintings and dances, fail to recognise
  the human processes of creation and
  understanding, and loose the content of these
  artefacts
• Perhaps there is a need for a theoretical
  framework for multimedia computing through the
  synthesis of semiotic theories and computational
  techniques
• The resulting computational techniques would be
  more sensitive to the idiosyncrasies and
  intricacies of different kinds of multimedia
  information

5
Surreys Computational Semiotics

• Systems for retrieving and browsing multimedia
  artefacts, attending to how they are produced and
  understood by experts in
• Art Galleries
• Dance Libraries
• Crime Scenes (NB. detection and semiotics)
• Work in the AI Group, led by Prof. Khurshid
  Ahmad, with Andrew Salway, David Boulton, Bogdan
  Vrusias and Mariam Tariq

6
Why Dance?

• Composite multimodal signal complex but
  patterned
• Can be analysed at different levels signal
  processing ? aesthetic interpretation and
  evaluation aesthetic framework available
• Collateral media musical scores, movement
  notations, texts (programme notes, reviews,
  journals, textbooks)

7
Semiotics of Dance

• Verbal reports elicited from experts as they
  Describe and Interpret dances (plus corpus of
  extant texts)
• Linguistic analysis characterises information
  processing aspects of task
• The collateral texts can be used to index and
  to segment the dance sequences (KAB system)
• PhD thesis, Video Annotation the role of
  specialist text

8
Semiotics of Dance current research

• Towards an ontology of dance (Corpus Linguistics)
• Modelling of composite signal and, modelling its
  generation and understanding contribution of
  different codes and temporal integration
  (Hess-Lüttich)
• Visual versus verbal representations (Mitchell)
• Cross-modal metaphor (Hodges)
• Intertextuality and hypermedia (Lansdale)
• Narratives in moving images and collateral texts
  (Chafe)
• Links with signal processing research (CVSSP)
• Interactive video and performance??

9
Closing Remarks

• In the long-term a symbiotic relationship between
  semiotics and computing science may be realised
  in new computational techniques
• Much may depend on whether semiotics can provide
  formal enough descriptions of structures and
  processes

10
What others have said

• Minneman and Smoliar (1996), emphasise the
  processes of producing and understanding media
  documents (as well as refining structural models
  of the documents).
• Such an inquiry is not well served by the
  intellectual foundations of computer science but
  must instead turn to principles laid down by
  disciplines such as semiotics and hermeneutics.

11
What others have said

• Cooperation between artificial intelligence and
  semiotics would bring enrichment to both
  areas. (Entry for AI in Encyclopedic
  Dictionary of Semiotics, Sebeok ed. 1994).
• Peirce himself might have suggested
  computational semiotics as a less contentious
  name for AI (John Sowa, Knowledge
  Representation, 2000402).

12
Bibliography for Computation and Semiotics

• Gonzalez, R. (1997). Hypermedia Data Modeling,
  Coding and Semiotics, Procs. of the IEEE 85(7).
  Discusses semiotic theories about the syntactic
  aspects of texts, images, videos, etc. in
  relation to the structured representation of
  hypermedia.
• Purchase, H. (1998). Defining Multimedia, IEEE
  Multimedia (Jan-March). A taxonomy of
  representational systems for multimedia
  computing based on Peirces typology of signs.
• Andersen, P. A. (1990), A Theory of Computer
  Semiotics, CUP. Andersen, Holmqvist and Jensen
  (1993 eds.), The Computer as Medium, CUP. A
  semiotic study of computer programming / system
  development and a semiotic perspective of HCI.
• Jorna, van Heusden and Posner (1993 eds.), Signs,
  Search and Communication Semiotic Aspects of
  Artificial Intelligence, Walter de Gruyter.
  Semiotic descriptions of computation relating to
  inferencing and learning in intelligent systems.
• International Workshop on Computational
  Semiotics, May 1997. Semiotics of text analysis
  narratives through large amounts of data (e.g.
  WWW) organisational semiotics (semiotic analyses
  of interpersonal and corporate exchanges)
  biosemiotics (agent-based systems, evolving
  agents, GAs, NNs)
• Special Session on Computational Semiotics at
  Conference on Intelligent Systems and Semiotics,
  1998. Simulating the constitution of meanings
  and the interpretation of signs as a form of
  procedural structuring, emerging from algorithmic
  processing of natural language data without being
  interpreted (and represented) in a predicative
  and/or propositional form.
• Computational Semiotic Systems II workshop
• Smoliar et al. at www.fxpal.xerox.com
• Computational Semiotics Group, Uni of Campinas,
  Brazil. The attempt at emulating the semiosis
  cycle within a digital computer