'Interactivity' is often limited to the user moving their cursor to an appropriate place, clicking their mouse, then something happens. Choice follows a tree structure with limited branches and is limited to a series of predetermined outcomes, nothing new or unplanned for originally can develop, emerge or challenge the original script. Multimedia programming which produces an over-determined interaction creates a fundamentally passive user experience, with little more user input than a television remote control provides. Before thinking of ways to help design 'interactives', it is necessary to ask "what is interaction?".
Interaction is a recursive process
Interaction is a recursive process: a complex back-and forth, a dialectical process, an act of unfolding, a feedback loop between the user and the computer. Recursion a fundamental trope in computing and is becoming an increasingly important trope in the way we understand our world. Recursion as a trope combines ideas of self-reference, self-similarity, circularity, repetition and nesting. It reveals a path from the finite to the infinite, from the same to the different. It also reveal similarities between disparate phenomenon.
Recursion is difficult to define. A recursive definition is a definition which refers to that which is being defined in its own definition. A recursive process is a process that refers to itself - a computer program that calls itself being the paradigmatic example. This seems paradoxical but it is very common in mathematics and logic. Computers are recursion machines. Recursion seems paradoxical due to its circular self-reference. However, the interesting thing is - as long as there is at least one way out of a circular loop, recursion does not necessarily lead to infinite regress and paradox. Recursion is both the thing which makes our computers work and which makes them crash.
Recursion also holds the key to a very specific problem in multimedia production: How can we make a CD ROM that has a finite set of media components - text fragments, audio, video and images - but is capable of a potentially infinite variety of novel behaviours? We can do this by using recursion - rules which can be repetitively applied to the results of their own previous application. By recombination and reuse, a finite set of elements can be combined in a infinite variety of ways. This is how language works to produce linguistic creativity. It is also how mathematics, logic, number, divination, myth, DNA, sampling DJs, and artistic pastiche work.
Lippman's definition of interactivity
According to Alan Lippman from MIT's Media Lab, interactivity involves "mutual and simultaneous activity on the part of both participants, usually working towards some goal, but not necessarily", and offers these principals:
- Interruptibility - that an interactive system allows participants to interrupt each other, not just alternate complete turns
- Graceful Degradation - that an interactive system will always encounter something it can't respond to fully, in which case it needs to gracefully "degrade"
- Limited Look-Ahead - that an interactive system is capable of looking ahead in order to anticipate the other
- Impression of an Infinite Database - that an interactive system gives the impression of being able to handle an infinite variety of interactions with an infinite number of responses
- No Default - that an interactive system does not push a default trajectory for the interaction
Interaction as mutual and simultaneous activity
If we think of "interaction" as a mutual and simultaneous activity between participants (our user(s) and our software), then - adopting Lippman's ideas - the following pincipals will help designing interactive multimedia:
- Each participant is able to interrupt the other at any stage.
- The direction of the flow of information may change during the interaction.
- In order to maintain the flow of interaction it must not get stuck in dead-ends halt the interactive process. Thus the trajectory and outcomes of the interaction is contingent on the user's actions and the program's actions.
- Avoid pre-planned paths to interaction.
- The path is developed in interaction, not prior to interaction as no one party can determine the course of the interaction.
- Each participant is (to a certain extent) semi-autonomous because they must have the ability to change the course of interaction whilst staying responsive to the cues provided by the other participant.
Application of these principals
Rather than describing an interactive multimedia project through a series of screens and pathways which can be detailed in a storyboard, think of your software as a semi-autonomous agent in a 'conversation' with the user. Whilst individual scenarios and activities can be storyboarded, they way they are manifested is an outcome of interaction. Therefore, think in terms of how your software can receive user input (how it 'listens' to the conversation), how does it process these inputs (out does it 'think' about what it hears), and how is its response manifested (what it 'says' back to the user).
For example, the PAL9000 Project, an attempt to a self-referential multimedia title which plays with the conventions of human computer interaction. A key idea expressed by PAL is that we no longer give 'commands' to machines but we 'enter into dialogues' with them. PAL 9000 takes this one step further by engaging users in a (literal) conversation that are partly determined by the user's actions and partly by PAL's mood (which in turn is influenced by a range of factors such as past conversations with that user, the time of day, the availablility of certain resources on the user's computer and so on.). PAL constantly generates unique experiences for the user, so that each time the user has a different 'conversation' with PAL. Some of the elements of the conversation might be the same - like a real person PAL sometimes repeats itself when it runs out stories, gets bored or forgets. PAL's face is the primary interface. His/her visual appearance is fluid because PAL likes to wear masks, to play games and to impersonate other computers and software for dramatic effect. There is no pre-planned paths to interaction. For some users, PAL is a like a 'tamagotchi' to be pampered and cajoled in performing cute tricks with caresses and flattery. For others, PAL is an adversory to be tricked and beaten into submission or exposed as a 'pretend' artificial creature with linguistic booby traps.
A short comment on graphic user interfaces
A typical way an interactive multimedia application 'listens' to a user is via a GUI (graphic user interface). Over the years, an informal standard or orthodoxy has developed for many visual devices - such as buttons, sliders, radio buttons and checkboxes. There are some advantages to maintaining and supporting such a standardised approach since they provide an easily recognisable 'common language' for users and software to use when interacting (and it certainly makes it easier to start a conversation with a user when they immediately recognise the language your application is speaking). Making an obtuse and mysterious GUI does not necessarily help create a richer, deeper level of interaction. Whilst a well designed, seductive 'GUI' (and some 'interesting content') will help stimulate the curiosity, users interact with the 'whole' of the software over time, not just its visual interface.