DEBRECENI EGYETEM FINNUGOR NYELVTUDOMÁNYI TANSZÉK FOLIA URALICA DEBRECENIENSIA 23. DEBRECEN, 2016 Conveying Nonverbal Messages in Translated Comics Eliisa PITKÄSALO University of Tampere Eliisa.Pitkasalo@staff.uta.fi 1 Introduction According to comics theorist Scott McCloud (1994: 9), comics are [j]uxtaposed pictorial and other images in deliberate sequence, intended to convey information and/or to produce an aesthetic response in the viewer. In other words, a comic consists of consecutive panels and possibly text, pictorial symbols, sound effects, and other effects attached to the images in various ways. The aesthetic response mentioned by McCloud results from someone reading any comic (a comic strip, a graphic novella, or a graphic novel) as a whole comprised of the aforementioned parts. Comics are a versatile art form that combines techniques from literature, visual arts, and films. They can be displayed as a part of a performance, or distributed through various types of media (for example, webcomics). Therefore, it is possible to utilise research methodology related to different art forms to study comics. This can be done by borrowing methods from narratology and textual research, as well as from the semiotics and visual culture studies. Methods from film studies can be applied when analysing still images, image layout, and various effects that imply resemblance between comics and movies (see Herkman 1998: 94 95.) As speech bubbles usually contain dialogue and other types of speech (Herkman 1998: 42), their contents can be used as material for discourse analysis just as dialogue can be analysed in literature. Discourse analysis can be applied even if literary speech is summarised, and is therefore only an illusion of speech (cf. Tiittula Nuolijärvi 2013). In this article, I examine the uniqueness of translating comic books. Comics are unique due to their multimodality: image and text, the nonverbal 233
ELIISA PITKÄSALO and verbal are combined into one message, which is then conveyed to the reader. Comics also contain references to other systems of signs, not just images and text. The panels incorporate narrative techniques that are typical for comics. These include speech bubbles and pictorial symbols (for example, a light bulb to signify an idea), as well as sound effects and other effects (such as motion lines and onomatopoetic letter combinations describing sound). In other words, the space in a comic is filled with different intermediate forms between images and text that describe sounds, time, movement, emotional states, and so forth (Manninen 1995: 38). When translating a multimodal text that combines written text, images, and graphic signs that represent aspects such as sounds or gestures, a translator must have a diverse set of skills in addition to language skills. They must not only be familiar with the language conventions in comics, but also be able to analyse and interpret culture-specific elements in images and other visual material, and to transfer these meanings to another language and culture. In an article about comics translation and humour by comics theorist Federico Zanettin (2010: 39), he notes that if a reader is not familiar with the cultural allusions in an illustration, a comic cannot fulfil its humorous purpose. In other words, even if the words are translated correctly, the comic cannot make the reader laugh. This applies to the translation of all types of comics. A translator should therefore also be aware of the feelings that a comic s real-world based illustrations evoke in the reader. This article examines how a comic conveys an experience of scenery, sensations, and situations to the reader, and how these experiences and perceptions can be transferred to another language and cultural environment. The examples I present in this article are from a Finnish graphic novel titled Sarasvatin hiekkaa (2008) (The Sands of Sarasvati, 2013) and its Hungarian translation Elsodort világok from 2010.The graphic novel was translated by Bella Lerch, a student at the University of Debrecen, who was a member of a translation project I managed. The graphic novel by Petri Tolppanen and Jussi Kaakinen is based on Risto Isomäki s 2005 eco-thriller also called Sarasvatin hiekkaa. Their work, The Sands of Sarasvati, introduces a group of scientists who study how the matter of water and sand are changing in different parts of the world. Russian oceanographers observe the changes occurring at the bottom of the Norwegian Sea, a group of Indian archaeologists study the ruins at the bottom of the Gulf of Cambay (today known as the Gulf of Khambat), glaciologist Susan Cheng observes the changes in the ice sheets in Greenland, and Finnish researcher Kari Alanen, located on the south coast of Finland, attempts to find solutions for the melting ice sheets. 234
CONVEYING NONVERBAL MESSAGES IN TRANSLATED COMICS Towards the end of the novel, these seemingly unrelated findings interweave, and gradually begin to form a frightening picture of what might await humankind in near future. 2 Comic as a Narrative The shapes, texts, images, and effects of a comic are often considered the grammar of comics (Herkman 1998: 68). When combined, these elements form a narrative which can be completely fictional or contain factual knowledge embedded in a fictional narrative. The original story of the examined graphic novel is mainly based on factual knowledge and includes a large amount of scientific jargon typical for science fiction as Markku Soikkeli observes (2015: 10). According to Soikkeli s definition, the first part of the term, science, means in this context research-based presentation where matters link to each other in the course of the text and/or critical presentation of information, and even a consistent use of metaphors. However, the science of science fiction is popularised and can also be fictional. (ibid. 10, 13.) Due to its visuality, a graphic novel is a particularly apt channel for science fiction because many of the events that would require special vocabulary to be described can simply be portrayed in single panels or in a series of panels. Hence, complicated matters can be depicted swiftly and in a simplified manner, which McCloud (1994: 42 43) considers to be the strong point of expression in comics. Time and space in comics are divided by panels, (McCloud 1994: 99), which are the basic units of comics, separated by gutters. The shape of a panel plays an essential part in the narrative. The conventional shape is a rectangle and any diverging shapes can be used to emphasise the significance of the events depicted in the panel. The size and especially the width of a panel usually illustrate the duration of the event in the panel: a tall and narrow panel signifies short duration, whereas a wide horizontal panel decelerates the action or even stops the course of events (McCloud 1994: 101 102). For instance, a panel depicting a landscape often spans the whole page and is as detailed as a photograph. Since the panels in comic books usually do not include motion (except multimedia comics), motion has to be depicted in other ways, such as using speed lines or broken panel borders. A sense of motion can also be created by using sound effects. These effects can sometimes overlap panel borders and thus depict motion that continues from one panel to another (see Manninen 1995: 38). 235
ELIISA PITKÄSALO Dialogue and any other speech in comics is written in speech bubbles which vary in shape depending on the volume of the speech. The volume can also be illustrated by typographic choices. First of all, normal speech is written in a normal-sized font in round, oval-shaped, or sometimes even in square speech bubbles. Secondly, loud sound, such as screaming, are written in bold, large letters, and thirdly, whispers are often written using a small font and placed in a speech bubble drawn with a dash line. Finally, a cloudshaped speech bubble indicates a thought and a serrated bubble expresses sound coming from an electronic device, such as a radio (Herkman 1998: 44 45). Comics often feature sounds other than speech. To add motion and action to the comic, various effects describing sound are used as a narrative technique. These types of sound effects can help highlight dramaturgical high points, lending them a cinematic sense of motion. (Huitula 2000). In Figure 1, the sound effect travels across gutters, which is one of the narrative devices used in comics. Effects that travel across panels, sound effects included, indicate the direction of motion or sound as well as the rising and fading of sounds, and they connect the events of separate panels more closely (Herkman 1998: 45 46). Figure 1. Isomäki Tolppanen Kaakinen (2008: 34) 236
CONVEYING NONVERBAL MESSAGES IN TRANSLATED COMICS In addition to sound effects, Figure 1 also illustrates how the size and shape of panels can indicate the duration of an event. The bottom panel depicts a scene in which the events unfold at a much slower pace than in the panels on top. The typographical signs illustrating the sound effect, in this case the letters decreasing in size, also indicate that the greatest danger has passed and the reader may relax. As Huitula (2000) notes, it is typical in a comic to depict fast-paced events in tall, narrow panels and slower ones in wide panels. According to Huitula, this is primarily due to the psychological way we interpret images. The reader s gaze scans narrow panels quicker than wide ones. On the other hand, the design of the panels forces the reader to perceive the images at the same time. As McCloud (1994: 99) observes, even though gutters divide the page, one panel does not correspond to one event only. The graphic composition of the sound effect that travels through the panels encourages the reader to interpret the page as one, big event, almost as continuous motion spanning over most of the page. 3 Impact of Multimodality on Comics Translation The visual composition and narrative graphic signs used in a comic evoke varied sensations in the reader. The reading experience elicits the sensation of movement, and the reader can hear the sound effects or taste the flavours of the visual symbols and feel blinded by the brightness of the art. The effects are always context specific: a cloud above someone s head might in one instance imply cold weather, and in another it could illustrate fatigue (air wheezing out of a character), or an odour in the air. In addition, visual symbols can depict various states of being, such as sleepiness, anger, or a stroke of genius (Herkman 1998: 46). The reader interprets these symbols through their own language and culture. Hence, whilst translating a graphic novel, it is important to acknowledge that the effects referring directly to the real world and the conventional symbols are both marked differently in each linguistic and cultural environment (Fiske 2000: 70 72). The translator of a comic must consider the target culture s system of signs since the meanings behind the symbols are adopted and thus culturally specific. According to Lehtonen (1998: 33), these meanings are historical as well as social; they are bound to time and place and can shift from one time and place to another. For example, Klaus Kaindl (2004: 177 178) examines the relation between images and word, and especially between symbol language, such as gesture language, and the wordplay portrayed in images. Al- 237
ELIISA PITKÄSALO though Kaindl focuses on humour and how it transfers to the reader of graphic novels, his article can also be applied to the examination of conveying other meanings. For instance, if the nonverbal messages of visual symbols or other visual gestures are foreign in the target culture, they do not automatically carry their meaning to the target text. In most cases, the translator cannot interfere with the actual image. Instead, the translator receives a template, into which they can insert only the translated text. If a panel includes culture specific, nonverbal messages that the target language reader cannot comprehend, the translator must have the capability to transfer the visual content into the target language s linguistic and cultural environment through words alone. However, words tend to fail in expressing the combined multimodal content of art and text. Thus, as the message conveyed by the image is lost, the combined meaning is not transferred to the reader or at the very least, it will fall short of the original. The same can naturally occur in a science fiction comic if the reader does not recognise a pictured appliance that is essential for the story. Images such as these, as in the cases observed by Kaindl (2004: 185 186), can be intertextual references, allusions to cultural information that the source language audience is assumed to possess. When this occurs, the translator should help the reader by adding information to the text, information that the image itself does not provide the target language audience with. This said, the reader is affected more by the events depicted in science fiction the more accurately and thus more plausibly the details have been given. Difficulties in translating The Sands of Sarasvati were caused by scientific language, jargon, and the ambiguity of words, as well as having to create the illusion of speech with a word limit and within the restricted space of a speech bubble. Also challenging were sounds and other effects being culture specific and the interplay of images, words, effects, and symbols. The graphic layout of comic effects merges words, images, and effects together, and the combination thus becomes an illustrated expression (Herkman 1998: 43). Transferring the meaning of comic effects from one language to another becomes problematic for the translator if the meaning cannot be transferred to the translation without explanations. The restricted space reserved for the verbal expression causes the most significant problem when the comic s visual and sound effects as such do not become clear for the reader. Ideally, the effects in a comic panel are so universal that their meaning does not require clarification. Figure 2 and its Hungarian translation in Figure 3 demonstrate how sound effects are used to elaborate the meaning of an image. In the first panel, the ice block detaches (KRAK!), and the third panel 238
CONVEYING NONVERBAL MESSAGES IN TRANSLATED COMICS depicts the ice walls starting to move towards each other (RRRRR). A new sound effect (FOOOSHH) has been added to the last panel in Figure 3, which illustrates how a need for elaboration can occur in a translating process as well. Figure 2. Isomäki Tolppanen Kaakinen (2008: 33) Figure 3. Isomäki Tolppanen Kaakinen (2010: 33) 239
ELIISA PITKÄSALO Figures 2 and 3 depict glacier researcher Susan Cheng, who has descended into a glacier cave through a hole in the bottom of the ice lake to investigate where the water has disappeared. As she is hanging on a rope against the wall of the ice cave, an enormous boulder of ice breaks off, nearly crushing her. However, the boulder narrowly misses her and falls into the water far down below her. There is no sound effect in the original last panel, but one has been added to the Hungarian translation. Without a doubt, the image benefits from an effect that graphically illustrates the sound and the downward direction of the splash. Compared to the source text (Figure 2), the target text (Figure 3) has an increased depth effect and the reader hears the massive splash caused by the ice boulder the size of a small house falling into the water deep below. Adding effects to the images in this manner is probably rather unusual because the possibilities of making changes to the graphic form of a comic are limited (Diadori 2012: 181). In this case, the sound effect was added to the panel in the editorial stage. Sound effects are based on onomatopoeia and they create an illusion of sound. They are crucial for understanding when the source of the sound is not visible in the image and there are no other signs indicative of sound in the panel. In the first panel of Figure 4, there is a square speech bubble with two sound effects (KRÄKS and KILING). These sound effects are used in Finnish to indicate that something breaks. As becomes apparent from the next panel, the square speech bubble signifies sound coming through a Skype connection and Susan Cheng tells Sergei that she dropped a mug. Figure 4. Isomäki Tolppanen Kaakinen (2008: 58) 240
CONVEYING NONVERBAL MESSAGES IN TRANSLATED COMICS Figure 5. Isomäki Tolppanen Kaakinen (2010: 58) In the translated version (Figure 5), the noises made by the breaking mug are surprising to a Finnish speaker (CSATT and PUFF). With this I had to trust the Hungarian translator, who said that she thought that these effects best illustrate the sound of a breaking mug. Figure 4 and Figure 5 are a good example of a situation that allows a translator to use their own onomatopoetic ear precisely because the sound effects are placed in a speech bubble. As a result, it is not only easy for the translator to change them, but it is also expected of them. As Pekka A. Manninen (1995: 38) observes in his dissertation, it is a distinctive characteristic of comic effects that despite their established nature, they can be freely modified. Based on this notion, a translator should also have the freedom to modify the comic effects presented in a comic. However, as Pierangela Diadori (2012: 182) remarks, oftentimes the question whether or not to modify is for the editorial level to decide, not for the translator. When the translator has the opportunity to make the decision, they can also ascertain that the subconscious sensory experience of the effect will also reach the reader of the translation. 4 Conclusion When translating multimodal texts such as comics, it is crucial to take into account the entire work, so that the final translation is a both understandable and a pleasant reading experience for its audience. Symbols, other graphic signs, and the placement of panels are all means to convey sound and movement within a comic. They all have a direct effect on how the reader per- 241
ELIISA PITKÄSALO ceives the comic and it is therefore important for a translator to focus on their combined effect. The translator must also be prepared to make some unconventional decisions when necessary. The nonverbal messages in The Sands of Sarasvati have been transferred from the original to the translation, and in one case they have even been emphasised. The translator cannot have affected the graphic signs present in the panels, but in one example (Figure 3), the editor has come to an interesting solution: the sound of a falling block of ice has been emphasised with an additional sound effect that has been added to the panel. Despite the transcending nature of graphic novels, the translation of comics is regulated by the same rules that govern the entire field of translation. The translator interprets the graphic novel as a complete work and makes the decisions regarding the translation based on their overall view of the comic. The translator is simultaneously a translator and a reader, who interprets the text and makes decisions that affect the experience of reading the translated work. From a translator s point of view, comics are a very interesting multimodal art form as they combine visual, verbal, and nonverbal expression and evoke strong feelings in their readers, as my example comics do. In the examples presented here, the strength of the story is not based only on the combined effect of verbal and visual messages, but also on the reader s existing experience and knowledge of the world. Research material References Isomäki, Risto Tolppanen, Petri Kaakinen, Jussi 2008: Sarasvatin hiekkaa. Tammi, Helsinki. Isomäki, Risto Tolppanen, Petri Kaakinen, Jussi 2010: Elsodort világok. Translated by Bella Lerch. Nyitott Könyvműhely Kiadó, Budapest. Other references Diadori, Pierangela 2012: Teoria e tecnica della traduzione. Strategie, testi e contesti. Le Monnier, Firenze. Fiske, John 2000: Merkkien kieli. In: Pietilä, Veikko Suikkanen, Risto Uusitupa, Timo (toim.), Johdatus viestinnän tutkimiseen. Vastapaino, Tampere. 242
CONVEYING NONVERBAL MESSAGES IN TRANSLATED COMICS Herkman, Juha 1998: Sarjakuvan kieli ja mieli. Vastapaino, Tampere. Huitula, Kristian 2000: Sarjakuvan suhde ääneen ja multimediaan. http://www.huitula.com/sarjakuva_ja_multimedia.htm Kaindl, Klaus 2004: Multimodality in the translation of humour in comics. In: Ventola, Eija Cassily, Charles Kaltenbacher, Martin (eds), Perspective on multimodality. John Benjamins Publishing Company, Amsterdam Philadelphia. 173 192. Lehtonen, Mikko 1998: Merkitysten maailma. Kulttuurisen tekstintutkimuksen lähtökohtia. Vastapaino, Tampere. Manninen, Pekka A. 1995: Vastarinnan välineistö: Sarjakuvaharrastuksen merkityksestä. Dissertation. Tampere University Press, Tampere. McCloud, Scott 1994: Understanding Comics. The Invisible Art. Harper Collins, New York. Soikkeli, Markku 2015: Tieteiskirjallisuuden käsikirja. BTJ, Helsinki. Tiittula, Liisa Nuolijärvi, Pirkko 2013: Puheen illuusio suomenkielisessä kaunokirjallisuudessa. SKS, Helsinki. Zanettin, Federico 2010: Humour in Translated Cartoons and Comics. In: Chiaro, Delia (ed.), Translation, Humour and the Media. Continuum International Publishing Group, London New York. 34 52. * A comic book is a publication in the form of sequential juxtaposed panels which combine both verbal and visual expression in a variety of ways. The sequence of drawings includes not only images and written texts, but also many graphic signs, such as speech bubbles, speed lines, and sound effects as well as graphic and pictorial symbols. All of these reinforce the joint effect of the words and images. The reader thus receives an even stronger message. Through the overall multimodal effect of comics, the reader can feel the speed, hear the sounds, smell the aromas and even taste the flavours. The many effects and pictorial symbols used in comics are culture-specific. These are demanding to the translator, especially if there is no possibility to make any changes in the graphic features of the panel. When translating the entire multimodality of comics, the translator faces the challenging task of transferring the written text as well as the nonverbal messages of comics from one language and cultural environment to another. This article examines some of the nonverbal features that comics can transmit. ELIISA PITKÄSALO 243
ELIISA PITKÄSALO 244