8 The third dimension
THE THIRD DIMENSION: FROM ‘READER’ TO ‘USER’
So far we have restricted ourselves to still rather than moving images, and to two dimensional forms of visual communication rather than three-dimensional ones such as sculpture, product design, architecture or stage set design. In this chapter we will explore to which degree the descriptive framework we have developed in this book can be applied also to the three-dimensional visual and the moving image. Our three-dimensional examples will mainly be drawn from the fields of sculpture and children’s toys. Toys are of particular interest as they occupy a space somewhere in between sculptures, which are primarily symbolic objects, objects for contemplation and veneration, and ‘designed objects’, which are primarily objects for use, even though they may also convey symbolic messages. In addition we will consider everyday objects such as cups and motor cars, and architecture. The chapter is intended as a first exploration. We are not able, within the space of this book, to present a systematic account of all aspects of three-dimensional visual communication, as this would require too many additional concepts (some of these are discussed in van Leeuwen, 2003). Instead, we will concentrate on the concepts we have described in the preceding chapters, to show what role they play in three-dimensional visual communication. In this way we will at least be able to indicate in which ways three-dimensional visual communication is similar to and different from two-dimensional communication, and to outline the theoretical issues which follow from this.
Starting with the issue of visual representation, we found that the key categories we introduced in chapters 2 and 3 can be applied also to three-dimensional visuals, and do indeed seem sufficient to describe a wide range of such objects. Many sculptures, for example, have what we called in chapter 2 a ‘narrative’ structure. Take Epstein’s Jacob and the Angel (1940), shown in figure 8.1: the arms of Jacob and the Angel form powerful vectors, relating the two participants in a complex and interesting way. The Angel’s action is transactional. It has a Goal, as he holds Jacob in a firm grip. But Jacob’s action, though foregrounded, is non-transactional – his arm hangs limply, and he does not hold or grab anything.
Kenneth Armitage’s People in the Wind (1952), shown in figure 8.2, also has strong vectors, formed by the way the figures are bent forwards as they struggle against the wind. But here (as in sculptures of discus throwers, ballerinas and other active subjects) the action is ‘non-transactional’. The vectors do not point at or lead to another participant, a Goal. The figures, it seems, ’strain forwards’, but they do not ’strain towards something’.
‘Reactions’ are also common, although in sculpture the eyes do not usually form as strong a focus of attraction as they do in two-dimensional images, because they lack the strong tonal contrast between the whites of the eyes and the pupils, which, in pictures as in nature, makes eyes so salient. It is as if the added naturalism of the third dimension must be counteracted by greater abstraction in other means of expression, such as colour, to prevent sculpture from crossing the line between art and the uncannily real make-believe of the waxworks show (or of certain contemporary forms of art). Nevertheless, in a sculpture like Rodin’s The Kiss (1880), the man and the woman not only hold each other with their arms, they hold each other with their gaze as well, in a ‘transactional reaction’, and it is not difficult to find examples of ‘non-transactional reactions’ also. Jacob (figure 8.1) looks up, in a non-transactional reaction, while the Angel looks at Jacob, in a transactional reaction. Again, the Angel acts on Jacob, but Jacob does not act on the Angel. Arnheim’s description of Michelangelo’s Moses provides another example. ‘The deflection of the lawgiver’s head and the fierce concentration of his glance introduce an oblique vector that moves outwards like the beam of a lighthouse. But no goal object is included’ (Arnheim, 1982: 46).
Even the design of objects and buildings can be vectorial and hence ‘narrative’, as shown in figure 8.3. The vector formed by the tailfins of 1950s cars, for instance, represented (in an abstract way) the idea of dynamic motion, as if it was not enough that cars are in fact dynamic moving objects, regardless of whether they have tailfins or not. The fact that cars do not have tailfins now points to the ideological dimension of sculptural representation: the meanings represented were those of the ‘jet age’. But here an important complication occurs. In the case of objects, the ideational relations we have discussed in chapters 2 and 3 can be realized in two ways: they can be realized by the designer, as forms to be ‘read’ by a viewer, as when a cup has a ‘dynamic’, vectorial handle (and also, of course, in the case of the pictorial or decorative designs printed or painted on, or moulded or carved in, the cup); or they can be realized by the user of the object, as when the cup is held or drunk from, in a ‘transactional action’ with its user – the vectorial handle is then a ‘non-transactional action’ from the point of view of the design of the cup, and a potential for transactional action, a means, from the point of view of its use.
Reactions can even occur in objects. The toy telephone shown in figure 8.4 not only includes the dog’s tongue, as an oblique vector signifying a non-transactional speaking; it also has eyes. The inclusion of eyes is in fact quite common in toys for young children, particularly in toys with the themes of time (clocks), communication (toy telephones) and transport (toy locomotives and cars), as if to encourage the child to form an emotive, personalized bond with these three key technologies as early as possible.
In contrast to two-dimensional visuals, sculptures rarely include a Setting. Their setting is the environment in which they are displayed, a gallery, a niche in a church, or a public square. It is not a represented Setting. Of course, sculptures can include a Setting, as in the works of Edward Kienholz and George Segal, for instance, or Asian sculpture gardens, such as the Tiger Balm Gardens in Singapore. But in contemporary Western sculpture the inclusion of a Setting is relatively rare. Decontextualization, it seems, has to counteract the added naturalism of the third dimension. Sculptures do, however, often have another participant, the pedestal on which they stand. Such pedestals can be (mere) framing devices, creating a degree of separation between the sculpture and its environment, and so enhancing its status as a representation, an object for contemplation, set apart from its environment. But they can also and at the same time form part of the representation, as in Canova’s Paolina Borghese (1805), which has Paolina resting on a couch which forms also a sarcophagus-like support for her reclining body. The absence of such framing can have a strong effect, as in the lifesize bronze of a middle-aged, corpulent man in a raincoat and hat placed as if mingling with the shoppers in the middle of the footpath of a busy shopping street in Amsterdam.
Turning now to ‘conceptual’ rather than ‘narrative’ structures of representation, these, too, can be found in sculpture. Miró’s Woman (1970), reproduced in figure 8.5, is what, in chapter 3, we called an ‘analytical’ representation. The sculpture does not just play with the forms of found objects, making eyes of the headlights and a mouth of the windscreen of a car; it is also an ‘analysis’ of ‘Woman’. ‘Woman’, in all its generality, is the ‘Carrier’, and the parts, the ‘Possessive Attributes’, are, in Miró’s conception: a head which is also the empty shell of a car; an upper body which is also a tray on which two aggressively pointed breasts are presented to the viewer; and a lower body which is a barrel-shaped container with a vagina-like slit and two handles to hold her by. When we visited an
exhibition of Miró’s sculptures we noted that the misogynistic quality of Miró’s ‘analysis’ was not lost on the viewers. There was a guest book in which the visitors of the gallery could write down their impressions. Many had used the opportunity to draw a quick caricature of a Miro woman with a contemptuous comment such as ‘WOMAN????’ These readings contrasted sharply with the artspeak in the catalogue, which described the formal qualities of the sculptures only, and did not dwell on Miro’s way of representing the female gender.
Giacometti’s Hour of the Traces (1930), shown in figure 8.6, is an analysis of the genderless human being, of the human condition in general. The parts: a kind of antenna, with an abstract eye, forms an active sensory tentacle and protrudes from the sculpture at an angle, constituting an oblique vector; a rigid rusty frame, the body; and, within the open frame, a plaster heart, suspended on a thin string, and moving slightly to and fro. Thus two actions are embedded in the analysis, both non-transactional: the action of the sensory apparatus, and the movement of the heart – the human being as a skeletal frame that is alive and surveys its environment.
We might add that analytical sculpture is used not only in art but also in science – for instance, to show the construction of a molecule; or as a teaching aid, for instance in anatomy, in which case the parts can often be detached from the whole. The kinetic design of sculptures and other objects, the way they can move or be made to move, taken apart and put back together again, and so on, is a subject to which we cannot do justice in this chapter, as it would again demand the introduction of a new set of concepts (but see van Leeuwen and Caldas-Coulthard, 2004).
The third dimension creates an additional option in representation, a relation between the representational structure and the position of the viewer. Seen from the side Epstein’s Jacob and the Angel (figure 8.1) has a narrative structure (‘transactional action’). It is in the first place about what Jacob and the Angel do. But if we look at the Angel from behind (figure 8.7), we are faced with an ‘analysis’ of the Angel, and a very striking one: the three principal ‘Possessive Attributes’ Epstein emphasizes are the Angel’s long hair, his wings – and his balls.
Not all sculptures use this possibility. One can imagine a continuum running from reliefs, which perhaps differ from two-dimensional images only in terms of modality, to
fully ‘multifaceted’ sculptures such as Jacob and the Angel. In between there are sculptures which, though free-standing, are clearly not designed to be seen from behind and leave the back ‘unworked’, perhaps because they were meant to be placed against a wall or in a niche. And even when a sculpture is a fully multifaceted representation, like Jacob and the Angel, its placement in a particular environment can block access to alternative viewing positions, and hence to alternative readings. This may be because the work is placed with its back against the wall, or because a barrier prevents the viewer from access to other than more or less frontal viewing positions. But it may also be done in subtler ways. When we first analysed Jacob and the Angel, it was placed in the centre of the octagonal entrance hall of the Tate Gallery, in such a way that the viewer first saw it from the side, with the Angel on the left. In other words, its position favoured the narrative reading, the drama of the sculpture, rather than Epstein’s striking ‘analysis’ of the Angel.
But the viewer did have access to the other sides, as the sculpture was placed in the centre of the hall. Playmobil figures such as those shown in figure 8.8 are also analytical structures. They show the significant attributes, the significant characteristics, of (for example) an ‘ethnic family’. The family has-five members – a father, a mother and three children. Each member of the family has black hair and dark skin. Note the difference from the neutrally labelled ‘family set’: the composition of the family is the same, and all the members of the family have pink skin, but they differ in the colour of their hair and therefore have individual characteristics as well as social characteristics, whereas (lesson number one) the members of the ‘ethnic family’ are ‘others’ who ‘all look the same’. The children of the two families are dressed identically, but the parents are not. Lesson number two: second-generation immigrants are already ‘more like us’. The Playmobil company brochure says that these toys ‘form an aid to the training of your minds’ and will ‘acquaint children with what they will meet in the big real world’ – but not in an entirely neutral fashion. As we have said in chapter 3, any analytical structure is only one of the many ways in which a given ‘Carrier’ can be analysed.
We have to remember, of course, that Playmobil figures enter into representation in two ways, like our earlier example of cups with vectorial handles (figure 8.4). On the one hand, they are like sculptures, pre-designed representations, to be ‘read’ by the child; on the other hand, they have movable limbs and detachable parts and they can hold objects in their hands. Children can therefore use them to create a variety of representational structures, narrative ’scenes’, and they can even subvert the pre-designed representations, for instance by giving an ‘ethnic’ child red hair. They can also create their own classificatory or analytical arrangements, for instance by making a display of different kinds of Playmobil children on their toy shelf or by creating a new analysis of the ‘family’, with only a mother, perhaps, or with five children of different ‘ethnic’ origins (the arrangement of Our Society and Others, figure 3.29, could be reconstructed with Playmobil figures!). In the same way a cup can be used, not only in the ‘transactions’ of holding or drinking, but also: to create an analytical structure, as when the cup is arranged on a sideboard together with the other parts of the set to which it belongs, or on the kitchen shelf, to become a ‘Possessive Attribute’ of the ‘Carrier’ ‘dishes’; or to create a classificational structure, as when a number of different cups are arranged symmetrically in a shop window or in a design exhibition.
The Vitrinen (display cases) of the German avant-garde artist Joseph Beuys are an intriguing example of the classificational sculpture. They are the kind of sculpture which is usually referred to as an ‘installation’ – glass display cases containing a variety of objects, some of them altered by Beuys. V/ti-/ne 2 (1960-70), for instance, contains a film can (itself containing a film which features a performance by Beuys), a pair of boxing gloves, a sausage, a cassette player (with a tape of music performed by Beuys), a Beethoven score with a small blackboard eraser on it, two wineglasses (one coated with a white substance, the other looking as though something has been burnt in it), and a zinc box. Clearly a work like this raises the question of what these objects have in common (what their ’superordinate’ is, in our terms), and this is exactly the question Beuys’ interpreters have asked. According to Theewen (1993: 139), Verwandschaft (‘relatedness’) is the key to understanding Beuys’ Vitrinen: ‘By bringing related objects together an association between them is created’, and in the case of the Vitrine we have just described this association is that all the objects ‘have played a role in performances of Beuys, and [that they] all contain something’ (1993: 29).
The third kind of conceptual structure discussed in chapter 3 was the symbolic relation. We saw how in pictures an overall colour – a blue haze, or a golden glow – could realize what we called a ’suggestive symbolic’ process, endowing the depicted scene with an overall significance, ’saying’, as it were, ‘this scene is cold and desolate’ (in the case of the blue haze) or ‘these objects are very valuable’ (in the case of the golden glow). Colour can play this role also in the case of three-dimensional objects – think of the colours of cars, for instance, the difference between a black, a bright-red and a white Mercedes, say.
But in addition to colour there are other factors, such as the material from which an object is made, the way the surface of a sculpture or other object is ‘worked’, or the overall shape of the objects, in so far as these are not determined by considerations of naturalistic representation, or by the functions served by an object. Giacometti’s Man Pointing (1947) is much less ‘analytical’ than the sculptures we have so far discussed. It places no emphasis on the distinctness of the parts of the human body. Even the facial features are hardly stated – small indentations to indicate a mouth and eyes. The sculpture does of course have a clear vector, as the man is making an expansive oratorical gesture. The most striking characteristic of this sculpture, however, is its rough, black, craggy surface. It is difficult to put into words exactly what is suggested by this surface, but whatever transcoding we attempt it will have to express somehow that this figure is ‘weatherbeaten’, affected by exposure to – but here we can fill in a number of things – the elements suffering, ageing and so on. The sculpture is abstract enough to allow all these readings and more.
The same applies, again, to other kinds of three-dimensional objects. Cups can be smooth, made of delicate china, suggesting, perhaps, an overall quality of elegance and refinement; or they can be sturdy and solid and made of brick-red terracotta, suggesting, perhaps, an overall quality of down-to-earth simplicity. Cars can be elongated and streamlined, suggesting power and speed, or, as in the case of the currently fashionable ‘retro’ look, rounded and egglike, suggesting a safe cocoon, a kind of womb.
The second kind of synbolic relation we discussed was the ’symbolic attributive’ process, where one represented participant has no other function than to endow another with symbolic significance. This occurs, for instance, in some of Miró’s sculptures, where birds and eggs attribute symbolic qualities to the figures depicted (usually ‘women’). But it occurs also in toys for young children, where telephones can have wheels (an early lesson about the concept of communication as ‘transport of information’ and ‘bridging the distance’ rather than ’sharing of information’), and ‘interactive learning centres’ for children aged 21/2 to 5 (‘6 built-in functions which teach the alphabet, numbers, shapes, colours, sound effects and nursery rhymes’, and all this for £29.50) have a steering wheel and dashboard, as a symbol of the power and control afforded by knowledge. Or in adult toys: emblems and other decorations on cars, for instance. Or in architecture, where sculptures and murals can become symbolic attributes for buildings.
On the whole, then, we feel that the account of visual representation we have presented in chapters 2 and 3 can be applied to three-dimensional visual communication. Yet there are some significant differences. First, three-dimensional objects can be placed on a continuum which runs from objects that allow only one reading (by offering the reader only one aspect, usually the front) to objects which allow more than one reading, depending on the position of the viewer relative to the object.
Second, three-dimensional objects can be placed on a continuum which runs from objects which have been designed only to be looked at, only to be ‘read’, to objects which enter into representational relations in three ways: (1) the relations encoded in the design of the object itself, to be ‘read’ only by the viewer; (2) interactive relations between the object and its user (e.g. holding the cup, or drinking from it); and (3) conceptual relations created by the user (e.g. creating a classificational syntagm with a number of different cups).
Third, even when an object does have a potential for multifaceted representation and/or for being ‘used’ as well as ‘read’, external conditions can inhibit this potential, block the viewer’s access to alternative reading positions, or to interactive engagement with the representational potential of the object.
INTERACTIVE VIEWING
We will now turn to the interactive relations we discussed in chapter 4, trying to explore, again, how applicable they are to three-dimensional visual communication. In that chapter we distinguished between ‘demand’ pictures from which represented participants address the viewer directly with their gaze and ‘want something from the viewer’, and ‘offer’ pictures which position the viewer as an observer only, and offer the represented participants as ‘information’ to be taken in by the viewer.
Clearly, this distinction can be applied also to sculpture – but, again, with a difference. Henry Moore’s Recumbent Figure (1938), shown in figure 8.9, addresses the viewer
powerfully. Although the eyes are little more than indentations in the surface of the stone, the whole attitude of the figure suggests a concentrated look. But, as viewers, we can take up a position from which that look w/// directly address us (as did the photographer, in the case of figure 8.9), so that the picture forms a ‘demand’; or a position from which the figure looks past us, at something else, or at nothing in particular, in any case, at something not included in our view, and in that case the look will become a ‘non-transactional reaction’. In the two-dimensional medium we cannot, as viewers, decide whether or not we will allow ourselves to be directly addressed by a represented participant; the decision has been made for us. In the three-dimensional medium we can -that is, if the placement of the sculpture allows us to do so. In the Tate Gallery, Moore’s sculpture could have been placed in such a way that the figure’s gaze would fix the viewer immediately upon entering the room. But this was not done when we viewed the sculpture there, and as a result the figure became just one of a number of Moore’s works, presented as part of a class ificational syntagm, and favouring tie ‘offer’ rather than the ‘demand’.
The same would be true of the toy telephone (figure 8.4). The gaze of this telephone can only become a ‘demand’ by virtue of an active decision on the part of its user. Some toys, of course, lend themselves more to this than others. Playmobil characters have small black dots for eyes. They are biased more towards the ‘offer for information’ than towards the interactive ‘demand’. And the eyes of many ‘boys’ ‘ dolls (Batmen, Crash-dummies, Megazords, etc.) are often obscured by helmets, masks or dark glasses. The eyes of ‘girls’ dolls (and of many cuddly animal toys), on the other hand, tend to be large and highly detailed. While boys are steered towards a more manipulative relation to their dolls, for girls the look, the interactive dimension, is made to matter more. And the same is true for very young children: even their bedclothes, pillowcases, cups, plates may have eyes, and are thus personalized, animated, capable of entering into a ‘direct address’ relation with the child. As with many other things, some of this may well live on, unconsciously, in the adult relation with objects.
The same reasoning can be applied to the other interactive dimensions we discussed in chapter 4. In principle the viewer can decide whether to see the object from close up or from a distance, frontally (hence with ‘involvement’) or from an oblique angle (hence with ‘detachment’); from above (hence from a position of power over the object) or from below (hence from a position in which the object has power over the viewer). We say ‘in principle’, because here, too, the viewer’s choice may be restricted by external factors, by barriers that prevent viewers from coming up close or seeing the object from a different angle. And large objects can make the high-angle viewpoint and the close distance impossible. What towers over us has, by design, power over us, and is, by design, socially distant: the vertical dimension is the dimension of power and reverential distance, the dimension of ‘highly placed’ people, places and things. In this connection it is also significant that sculptures, as works of ‘high’ art, cannot usually be approached from the most intimate distance, the distance that makes touching possible: as soon as the gallery visitor comes too close, a guard will become alert.
When sculptures are taken out of their original context and moved into another, their interactive meanings may change significantly. They may be, literally, taken down from the pedestal – in a church perhaps – where they were to be looked at from below, with reverence, to be moved into a gallery, where they are positioned at a level of equality, and viewed from a more ‘familiar’ distance: Michelangelo’s David, removed to the rotunda of a museum, no longer calls to the citizens of Florence and is unaware of their calling on him, and now ‘can be explored by the viewer, but makes no advances to him’ (Arnheim, 1982: 50; also Hodge and l<ress, 1988: 201-3).
MODALITY IN THREE DIMENSIONS
In chapter 5 we described visual modality as resulting from the degree to which certain means of pictorial expression (colour, representational detail, depth, tonal shades, etc.) are used. Each of these dimensions can be seen as a scale, running from the absence of any rendition of detail to maximal representation of detail, or from the absence of any rendition of depth to maximally deep perspective. And on each of these scales there is a point that represents the way the given pictorial dimension is used in what could be called standard naturalism. To the degree that the use of a dimension is reduced, it becomes, at least in one respect, more abstract, ‘less than real’. To the degree that it is amplified, it becomes ‘more than real’, and we associated this with a ’sensory coding orientation’, an emphasis on sensory pleasure (or displeasure, as in the case of ‘more than real’ horror images), and an attempt to come as close as possible to a representation that involves all the senses.
Some of these play much the same role in three-dimensional visuals. Clearly, sculptures and toys can represent what they represent in naturalistic detail or more abstractly. And when the shape of everyday design objects no longer betrays their function, when, for instance, refrigerators, washing machines and kitchen storage cabinets all become sleek featureless white boxes, there is also a strong sense of abstraction. The same can be argued for buildings. Also, like pictures, three-dimensional representations can include several levels of modality. 
The heads of Henry Moore’s King and Queen (1952-3), for instance, shown in figure 8.10, are abstract symbols, while their hands are rendered in naturalistic detail. This expresses the contradictory nature of the powerful. Their minds may have lost touch with the detail of everyday concrete reality, but look at their hands – they are after all still human, and their work, their doing, is still the work of humans. Machin and Suleiman (2004) have pointed out that in American computer war games the weaponry is represented in realistic detail, while the settings have lower modality, forming a generic desert that could be anywhere. This foregrounds American technological supremacy and backgrounds the specifics of specific conflicts. In a Lebanese computer war game produced by Hezbollah, the landscape is reconstructed from photographs of the sites of actual conflicts and represented in more detail. Here the specifics of historical and geographical accuracy matter.
The representation of detail in toys is particularly interesting. Barthes’ still highly readable essay about the semantics of toys (1973: 53ff.) is now perhaps overtaken by semiotic events. He describes French toys as highly detailed, highly naturalistic miniature versions of adult objects – and notes how unsatisfactory they are, therefore, both from the point of view of pleasure, of the sensory dimension, and from the point of view of their interactive potential, as objects to play with: ‘the child can only identify himself as owner, as user, never as creator; he does not invent the world, he uses it: there are, prepared for him, action without adventure, without wonder, without joy’ (1973: 54). Toys of this kind can be seen in museums such as London’s Museum of Childhood. One’s first impression on seeing Victorian toys is that children are addressed as miniature adults, their subjectivity ’scaled down’, but not ‘reduced’, from that of the adult world. This is represented through a large number of semiotic modes: the materials (glass, clothes of various kinds, metals, etc. – all of them rarely, if at all, used in contemporary toys) as much as the miniature naturalism of the represented objects. Today’s toys vary in their detail. Toys for young children are abstract. Shapes and textures are brought down to their essentials. Locomotives have featureless wheels, one featureless chimney, two yellow circles for windows. They are, from the point of view of detail of representation, like the simplest line drawings. As the child gets older, detail increases. The wheels of the locomotives get spokes and driveshafts. The texture of the machine’s body acquires detail. Headlights with miniature Fresnel lenses are added. But on the whole the contemporary toy remains simple, essentialized, as for instance (again) in the popular Playmobil figures, where the eyes are two dots, the mouth a curved line, the hair an almost featureless helmet, with a few indentations suggesting texture and the length signifying gender. Girls’ dolls, on the other hand (and other girls’ toys: realistic washing machines, ‘beauty shops’, vacuum cleaners, dolls’ cots), imitate the adult world, or at least that of the glamour girl, the housewife and the mother, much more so than boys’ toys, which depict a make-believe world of sciencefiction vehicles and weapons, or a world of dinosaurs and other monsters. The latter are often ‘more than real’, with highly textured, glistening scales, irregular teeth, and menacing eyes, set behind wrinkled lids. They are designed to create the kind of sensory, visceral reaction also sought, for instance, in horror films. Cuddly animal toys also tend to be ‘more than real’, with exaggeratedly soft furs and large moist eyes, this time to enhance the sensory pleasures of holding and touching. Researching this chapter, we spent many hours in toy shops as well as in toy museums, and could not help being struck by the contrast between the ‘bourgeois’ naturalistic toys Barthes described and today’s make believe world of brightly coloured plastic and creatures and objects from fantasy stories.
The role of colour in the modality of three-dimensional visual representation also resembles that of the two-dimensional visuals. Some of Mir6’s sculptures of ‘women’, for instance, are painted in bright, primary colours – yellow, blue and red. As a result they are schematic and analytical from the point of view of representational detail, a simplified, abstract view of ‘woman’, but ‘more than real’, ’sensory’ from the point of view of colour. Mir6’s women are not just machine-like assemblages of parts, they are also pleasurably colourful (‘woman’ as a machine for pleasure). Many toys, especially toys for young children, have exactly the same kind of modality configuration: abstract and schematized, ‘conceptual’, from the point of view of colour; ‘un-naturalistic’ primary colours, colours for the sake of pleasure rather than naturalistic representation. This ’sensory’ aspect is then further enhanced by the way these toys appeal to all the senses, and include touch and sound, as stressed in this description of the ‘Chicco animal train’: ‘a locomotive featuring eight different animal sounds and four different train sounds. Sounds are activated by pressing appropriately shaped buttons.’ Again, for older children colour can become more naturalistic, as in Thomas the Tank Engine, which replaces bright reds, yellows, blues and greens with more mute steel blue, grey and black, and just a touch of red. And, while the colour in all these examples is unmodulated, the colour of girls’ dolls becomes more varied, with blushes on the cheeks, shadows under the eyes, a shine on the lips. And the same applies to the slimy greens and greys and pinks on dinosaurs and monsters.
The materials used in three-dimensional representation, similarly, can be motivated naturalistically, as when toy cars are made of metal, or cuddly animals of soft, furry materials (though, on the other hand, even toy pigs can be furry!). But they can also be ‘less than real’, abstracting from the variety and specificity of the range of materials available, as in the case of plastic. To quote Barthes again.
Many toys are now moulded from complicated mixtures; the plastic material of which they are made has an appearance at once gross and hygienic, it destroys all the pleasure, the sweetness, the humanity of touch …. Henceforth, toys are chemical in substance and colour; their very material introduces one to coenaesthetics of use, not pleasure. (1973: 55)
Again, materials can have ‘more than real’ modality, when their choice is motivated not by an attempt to make the object look like what it represents, but by an attempt to create pleasure or displeasure. And this can apply also to everyday objects, and to buildings, if we replace naturalism by an attempt to reveal the material from which the object is actually made and divergence from naturalism by attempts to conceal these in one direction (say, plain sheets of a synthetic material over bricks) or another (say, timber cladding over bricks or concrete).
In other respects, however, three-dimensional modality differs from two-dimensional modality. There is no need to represent depth: the object already has depth, by virtue of its three-dimensionality. And there is no need to represent the play of light and shade: it already occurs, naturally. It is to compensate for these intrinsic naturalistic qualities that most Western sculpture (1) is decontextualized, lacking a Setting; (2) refrains from using colour as a means of representation, except in the sense of overall, symbolic colour; and (3) increasingly tends towards highly reduced forms of representation. In this way ‘high art’, which seeks to go beyond the mere replication of reality by representing an ideal of beauty or an abstract truth, distinguishes itself from everyday sculpture – from the dummies in shop windows, from girls’ dolls, from miniature dinosaurs, and so on. These need not themselves be naturalistic in every dimension in any case, and may use reduced naturalism either for didactic purposes, as in toys for young children, or to create class distinctions in taste, as with the more stylized dummies in expensive shops.
COMPOSITION IN THREE DIMENSIONS
Many sculptures and other three-dimensional objects do not clearly polarize between left and right, top and bottom, centre and margin, but when they do, the values of Given and New, Ideal and Real and Centre and Margin apply, we think, in the same way as they do in two-dimensional visual communication. Yet the third dimension does introduce additional factors.
In figure 8.1 we saw Epstein’s Jacob and the Angel as it would have been seen by visitors entering the Tate Millbank Gallery when the work was positioned in the centre of the gallery’s octagonal entrance hall. Like God in figures 6.5 and 6.9, the Angel against whose force Jacob is so helpless is Given, and Jacob’s helplessness is New, the focus of the drama. But moving to the other side, the viewer can reverse this, and make Jacob Given and the Angel New, provided access to the other side is not blocked. The same is true for other multifaceted sculptures with two or more ‘polarized’ participants, such as Rodin’s The Kiss, where, depending on your point of view, either the man can be Given and the woman New, or the woman Given and the man New. This is why photographs cannot do justice to multifaceted sculptures – a photo can always give only one angle, and hence one reading. The same is not true, however, for Ideal and Real and Centre and Margin. These relations cannot be inverted by changing the angle from which the work is viewed. In other words, the horizontal dimension allows interactivity; the vertical dimension and centrality do not.
Ideal and Real and Centre and Margin are often the most significant compositional dimensions in three-dimensional visual composition. Architecture provides perhaps the clearest example. Left and right are not usually polarized. Horizontally there is symmetry, but vertically there is not. The vertical dimension is used to polarize, to produce difference, with the Ideal, the element(s) that give(s) the building its more general and ‘ideal’ significance on top – the tower, for instance, with its significant emblems, the cross, or the clock, the gable stone on Amsterdam canal houses, the frieze high up on the Greek temple. Below, on the other hand, is the space of the Real -the forecourts where we meet, the doors through which we enter. More generally, the façade of a building, its vertical dimension, is the building we ‘read’; the horizontal dimension, the floor plan, is the building we ‘use’: the compositional spectacle in the upright dimension is essentially visual. It restricts the user to observation from a distance …. In the horizontal plane, the corresponding dynamics involves the user directly and is therefore largely social. The level plan is the arena of human action. (Arnheim, 1982: 213)
Figure 8.11, shows an example of centrality in an architectural façade: the canopy with the Madonna, in the centre of the façade of the Church of Santa Maria della Spina in Pisa.
Multifaceted objects add further dimensions to three-dimensional composition, and allow (at least in principle) front and back, and the left and right side (and, in the case of open structures, an interior centre and the exterior) to be used for the production of difference. Of the latter we have already seen an example in figure 8.6-the plaster heart in the centre of the rigid frame of Giacometti’s Hour of the Traces. But not all multifaceted objects use these dimensions. As in nature, where trees or mountains do not have a front or back (other than one which stems from our positions towards them, e.g. which side of the mountain we live on), objects can be the same, and have the same meaning, whichever side we view them from. Sculptures of the ‘Three Graces’, for instance, are usually composed in the round, offering essentially the same view from whichever side one approaches them. And the same pattern can be observed in some children’s toys, for instance a kind of ball with symmetrically distributed pictures illustrating nursery rhymes (although ‘the one that ends up on top determines which song is heard’). Or you find it in buildings, such as the Queen Victoria Building in Sydney, which, in its latest reincarnation as a shopping centre, has, on every side, a central entrance and shop windows; or perfectly round buildings such as the Temple of Vesta in Rome, or the Pantheon. In other cases, however, front and back especially differ in meaning. It is tempting here to take the human body as a metaphor, with the front as the public side, the side where we articulate how we want to be read, and the back as the non-social side, the private side which is not meant to be viewed and often has no meaning except perhaps for those with whom we are most intimate. This is why, paradoxically, it can also be the most revealing, as in the case of Epstein’s Angel (figure 8.7). This idea is worked out in more detail in van Leeuwen (2003).
In chapter 3 we argued that the structures of diagrams (the top-down path of the taxonomy, the left-right path of the flowchart, the digital network) are modelled on forms of social organization. Perhaps it can be argued along the same lines that composition, both two- and three-dimensional, is ultimately modelled either on the ‘non-social’ roundness of the natural forms such as trees and mountains, or on the polarized human body, with the head as the Ideal, the feet as the Real, the heart as the Centre, movement and action as the more interactive and dynamic horizontal dimension, and the front and the back as, respectively, the social and public and the non-social and private side. As Lakoff and Johnson said, Spatial orientations arise from the fact that we have bodies of the sort we have and that they function as they do in our physical environment. Orientational metaphors have a basis in our physical and cultural experience. Though the polar opposites, up-down, in-out, etc., are physical in nature, the orientational metaphors based on them can vary from culture to culture. (1980: 14, our italics)
THE MOVING IMAGE
The representational, interactive and compositional patterns we have discussed in this book also apply to the moving image, as shown by a number of our examples in this book. Yet the addition of movement does of course lead to differences, and it is these differences we will discuss in the final section of this chapter.
Starting with the narrative processes we discussed in chapter 2, here the principal difference is that the role of the vector is taken over by movement. Instead of, for instance, a vector formed by an outstretched arm, as in figure 2.15, the process will be realized by the action of raising the arm and pointing the hand. Usually these actions are figurative, recognizable as driving, walking, jumping, pointing and so on. But they may also be abstract – as, for instance, in Walt Disney’s animation film Fantasia (1941) – or in technical films, where arrows may be animated, unfolding in front of our eyes.
But there is a complication. In moving images the relation between Actors and Goals may be represented in a single shot, showing both Actor and Goal; or in two subsequent shots, the first showing the Actor, the second the Goal (or vice versa), as demonstrated in figure 8.12. In both cases we see a soldier (Actor) and civilians (Goal). But in the one case they are spatially connected, shown together in the same shot; in the other they are disconnected, shown in separate shots. As every film and television director knows, the two shots in such a disconnected syntagm (usually referred to as a pair of ‘reverse angle shots’) have to be ‘matched’ carefully, to restore the connection. They have to be taken from the same side of the imaginary line running between the participants and from approximately the same horizontal angle, to make it appear that the participants are facing each other and looking at each other. The tonality of the two shots has to match as well, for instance by ensuring that they are shot under the same lighting conditions, and through colour grading, removing any discrepancies between the colour rendition of the shots.
Should we see such a ‘disconnected’ narrative process as one unit of meaning or two? Is it the equivalent of a sentence like ‘The soldier shoots the villagers’, or of a formulation that expresses the soldier’s agency less directly – for instance, ‘The soldier fires. The villagers are shot’? But such an attempt at translating moving images into words cannot fully capture the difference. Filmic ‘disconnection’ has no parallel in language. It does have semiotic potential, however. It can, for instance, show people as ‘isolated’ individuals, even while they are interacting with others, and it can radically disconnect Actors from the Goals of their actions, and from the effect of their actions on these Goals, just as happens, for instance, in long-distance telephone calls or the firing of long-range missiles.
The disconnection between Actors and Goals is an aspect of ‘film language’ that only developed twenty-five years or so after the invention of the medium, and it has been the subject of much discussion in film theory, not least because it allows ‘faking’. No history of the medium omits the experiments of ‘Constructivist’ film-makers in the Soviet Union of the early 1920s. In one of these experiments, film maker Lev Kuleshov cut together shots of two actors meeting and greeting each other. Each actor was filmed separately, in a different location. They had therefore never actually played out the scene together. Once the two shots were spliced together, however, the two appeared to meet in one and the same place, an effect which l<uleshov called ‘creative geography’. In 1930s Tarzan films, encounters with wild animals were often faked in this way, by intercutting stock shots of wild animals with shots of actors acting out the appropriate reactions and actions in a studio set. The famous French film critic André Bazin (1967), on the other hand, favoured the ‘connected’ method. To see a real event happening in real time was for him the quintessential film experience and the quintessential power of the medium. He praised the seal hunt in the documentary Nanook of the North (1921), a scene taken as one long, unedited shot of a seal being harpooned through a hole in the ice.
In contrast to the still image, the moving image can realize events that have neither an Actor nor a Goal. Shots of shimmering light on softly rippling water create a sense of pure process, pure movement, in which it is hardly possible to disentangle process and participants, and in which participants, if they can be discerned at all, are ‘caught up’ in the process in a way that is neither ‘active’ nor ‘passive’. The still image equivalent of such a shot would be a kind of abstract pattern, lacking the dynamic sense of ‘action’ or ‘event’.
The choice between ‘connection’ and ‘disconnection’ also exists in the case of reactions. Films can show Reacters and Phenomena either in one and the same shot, or in two subsequent shots. This pattern, known as ‘the point-of-view shot’, ties together three shots, with the Reacter reappearing in the third shot, so that the Phenomenon is wedged in between two shots of the Reacter. Here too the shots have to carefully ‘matched’. If, for instance, the Reacter looks down, the Phenomenon has to be shot from above, and if the Reacter looks at a moving Phenomenon, the angle of his or her head and the direction of his or her gaze should have changed in the third shot, to match the distance travelled by the Phenomenon during the second shot. Disconnected reactions have a particularly strong ’subjective’, ‘first-person’ feel, as the viewer is looking at the Phenomenon ‘through the eyes of the Reacter’. A variant shows the Reacter and the Phenomenon in the same image, ‘over the shoulder’ of the Reacter, hence also from the Reacter’s point of view. But here we do not look at the Phenomenon ‘through the Reacter’s eyes’, and the effect is less emotionally involving, as we see the Reacter from behind and therefore do not see his or her reactions to the Phenomenon. It is the angle used in contemporary computer war games, to make players identify with the ’special-ops’ soldier characters they play (figure 8.13).
Finally, while still images have developed dialogue balloons to realize verbal processes, in moving images dialogue is not represented visually, through writing, but directly, through speech. The synchronization between the speech and the Speaker’s lip movements replaces the vector that connects Speaker and Speech. Without such synchronization, moving images cannot signify that the speech we hear is actually spoken by the Speaker we see. Once the link between the Speaker and the dialogue has been established, the dialogue may become ‘off screen’, continuing, for instance, while viewers watch the reaction of a listener.
Turning now to the interactive dimension, in chapter 4 we have already seen how camera positions can create symbolic relations between viewers and what is depicted in an image. Moving images are no different in this respect, with one proviso: in moving images the relationship becomes dynamic. It can change in front of our eyes. The camera can zoom in to a closer shot, or zoom out to a wider shot; it can crane up to a high angle or crane down to a low angle; and so on. And even when the camera is not moving, the participants themselves can move, walk away from or towards the camera, or walk up or down a flight of stairs with the camera tilting up or down to follow them, thus changing the angle from which viewers see the participants. In other words, the moving image can represent social relations as dynamic, flexible and changeable. Distance and angle can be dynamicized, and this in two ways: subject-initiated, with the represented participants initiating the change, or camera-initiated, with the image-maker initiating the change (the contrast of course also applies to synthetic images where a camera is not involved). In the first case, the visual text takes a ‘neutral’ stance, a stance of ‘recording’ what is taking place (even though the events may of course be staged). In the second case, the image-maker more overtly positions viewers towards what is being represented.
In most films distance and angle change constantly. In other words, what in the case of still images has never moved in the mainstream, cubism, the use of multiple perspectives, has become so commonplace in movies that it is now hardly noticed. The only difference is that films show the different perspectives one after the other, rather than at the same time. Figure 8.14 shows how such changes of distance and angle can be used to signify both the relations between ‘characters’, between the people we see on the screen, and the ongoing, constantly shifting relations between these characters and the viewers. It is the opening scene from Howard Hawks’ The Big Sleep (1947). ‘Private Eye’ Marlowe (Humphrey Bogart) has been called to the house of General Sternwood (Charles Waldron) to help him deal with a case of blackmail involving his youngest daughter, Carmen (Martha Vickers). As Marlowe waits in the hall to be shown in by the butler, Carmen provocatively confronts him. The interview with Sternwood then follows.
In news and current affairs television, distance and angle create a symbolic relation between the people on the screen and the viewer. Anchorpersons are shown frontally, from slightly below eye level, and in a wider shot than most other participants in the programme. This enhances their authority. They are literally ‘higher up’ than the viewers and shown from a respectful distance – initial shots may even show them from a very long distance, sitting behind large, gleaming desks at the far side of an empty, palatial hail.
The distinction between ‘offer’ and ‘demand’ (see chapter 4) also applies to moving images, and it too can be dynamicized: represented participants can turn towards the camera and look at the lens (and hence at the viewer), or can avert their gaze. But the camera cannot initiate this; it must be initiated by the participant, whether on their own initiative or as a result of following instructions from a director.
‘Offers’ are still the rule in naturalistic drama, in the theatre as much as in film and television. Bertolt Brecht famously sought to reintroduce the ‘demand’ stance in the theatre, especially by means of interpolated songs, and film-makers like Jean-Luc Godard have followed him in this. In these contexts ‘demands’ were thought to create an ‘alienation effect’, to break with conventions meant to naturalize the fictional world of stage and screen, and so to make the audiences more aware that they were watching a fiction and invite them to reflect on its content. In many other contexts – for example, television news the ‘demand’ is the accepted convention, although not everyone is given the right to address the viewer directly. Anchorpersons and on-camera reporters may look at the camera, but interviewees may not; in chat shows hosts may look at the camera, but guests may not, and so on. In other words, the ‘demand’ is a privilege which media professionals have reserved for themselves.
The concept of modality (see chapter 5) is also fully applicable to moving images, but a further facto movement, needs to be added to the list of means of expression that can cue modality. Like visual detail, background, depth, light and shade, colour, etc., movement can be represented with different degrees of realism or abstraction and hence play a role in modality judgements. Representations of walking, for instance, can range from simple animations in which stick figures raise and lower their legs without any articulation of the joints or any movement of the rest of the body, to highly detailed animations showing the rippling of every muscle involved.
Most films invite us to use the naturalistic criterion, although this is perhaps changing as the use of synthetic images and animation increases. In many animated cartoons, the background has higher (naturalistic) modality than the foreground, a reversal of what normally happens in ‘live action’ films. There is, of course, a technical reason for this. Backgrounds do not have to be animated and can therefore be painted in detail without breaking the budget. Again, in computer games different characters and actions may be animated more or less intricately. In a Delta Force game one of us played, the movement of enemies falling down as they were killed was decidedly unconvincing and unnatural. Again, there may be pragmatic reasons. Detailed animation costs time and money, and may slow down the action. But that does not negate the semiotic effect of reducing the naturalistic impact of killing.
Finally, the elements of composition discussed in chapter 6 (information value, salience and framing) apply to the composition of the shots in a film or television programme just as much as they apply to still images and other visual compositions, with, again, the proviso that the moving image can make composition dynamic. Something that starts out as Given can move into the New position in front of our eyes. Something that has low salience can become highly salient in the middle of a shot – for instance, by moving or being moved into the light, or by a change of focus of the camera. In chapter 6 we showed how, in figure 6.1, the left edge of the door of the shed frames the two characters in the shot, causing them to inhabit different spaces and so emphasizing the lack of communication between them. But in a moving image characters can move into each other’s space and undo the framing between them. And all of these ways of dynamicizing composition can be subject-initiated or camera-initiated.
This brief discussion does not exhaust the ‘language of film and television’. It has concentrated on the spatial patterns of individual shots and on two specific time-ordered patterns, the ‘reverse angle’ and ‘point of view’. But film is also, and perhaps above all, a temporal mode, structured by intricate semantic and rhythmic patterns of editing (see van Leeuwen, 2004, for a social semiotic approach), and it is also characteristically multimodal, involving not just the visual, but also speech, sound and music. These aspects of the medium fall outside the scope of this book. But we do hope we have shown that the ideas presented in this book can usefully be applied to the spatial aspects of moving images or, more precisely, since movement is a temporal phenomenon, to an area where the spatial and the temporal interact and overlap.
