Introduction
In Paper #3 I proposed the Objects of Study for multimodal composition to be twofold: new media objects (websites, images, videos) and student writing. While multimodal composition usually focuses on student production of new media, the Objects of Study in this case deal specifically with analysis, how students interpret new media objects and how teachers and researchers analyze student writing on new media as data. In this paper I outline theories and methods related to a social semiotic approach to multimodal composition. I will first provide a general overview of systemic functional linguistics (SFL) theory, and then relate these methods to analyzing new media and student writing.
SFL Theory
Table 1. SFL Resources (Schleppegrell)
A functional grammar is not just concerned with labeling grammatical elements according to the syntactic category they represent (nouns, verbs, adjectives, etc.) nor with identifying the role that these different elements play within a sentence (subject, object, etc.). Rather than analyzing linguistic structures in isolation or as abstract entities, a functional approach identifies the configuration of grammatical structures which is typical of or expected in different kinds of socially relevant tasks and links those linguistic choices with the social purposes and situations that “texts” (spoken or written) participate in. (Schleppegrell 45)
Functional grammar is not concerned with labeling the “formal” properties of language, instead looking at how meaning is made and the meaning options in social contexts. This functional approach to language can be very useful in education because “teachers also need a better understanding of the features of the language they aim for students to develop, and so the focus here is on the forms that language takes in academic contexts” (Schleppegrell 44). Providing a basic overview of functional grammar, language is used to create three different kinds of meaning: ideational, interpersonal, and textual. In terms of studying social context, or the register of a text, these variables are the “field (what is talked about), tenor (the relationship between speaker/hearer or writer/reader), and mode (expectations for how particular text types should be organized)” (Schleppegrell 46).
Within each of these functions is a set of tools that can be used for text analysis (See Table 1 above). In functional grammar, nouns and noun phrases are called participants, verbs are called processes (there are several kinds of verb processes: doing, saying, thinking, being, etc.), and adverbial phrases of time, place, and manner are called circumstances (see Table 2). By identifying the functional processes of language, research shows that students, including ELLs, are better able to interact with language and content on more meaningful levels than simply labeling sentences using formal grammatical rules (de Oliveira and Schleppegrell 20). Students can more readily understand the action of texts, the doers and the doings, by identifying the participants, processes, and circumstances.
Table 2. Functional Labeling of Sentences
Multimodal Analysis
Table 3. Semiotic Toolkit (Wise)
Researchers such as O’Toole, Kress, van Leeuwen, Lemke, Baldry, Thibault, and O’Halloran have adapted Halliday’s SFL metalanguage to analyze multimodal texts (images, videos, the Web, etc.). Their research is interdisciplinary, combining work in visual literacy, social semiotics, functional grammar, and design. Table 3 above is the visual analysis toolkit I created for my students in FYC based on Kress and van Leeuwen’s multimodal social semiotic framework in Reading Images: The Grammar of Visual Design.
This visual analysis framework shares many similarities with the SFL framework for textual analysis. Both are organized around the metafunctions. Examining the ideational metafunction, students label the participants of an image as actor and goal. Instead of processes, students identify vectors, the visible or invisible oblique lines formed from participants’ gaze, body parts, and objects. Vectors are the verbs of the visual world, expressing action. Moving to the interpersonal metafunction, students identify whether participants are looking directly at the viewer in a demand gaze, or looking away in an offer. Modality is a very useful tool, which is a way of judging the authenticity of an image on a sliding scale (see fig. 1). High modality, like black and white images, conveys a truthfulness, that the image is unaltered and authentic, while low modality refers to images that look unreal, highly saturated pictures, often photoshopped, such as on the cover of fashion magazines. For a full explanation of the visual analysis toolkit, you can download this PowerPoint.
Figure. 1 Modality Scale
Analyzing Student Writing on Multimodal Texts
Below I have provided a student’s recent visual analysis of a Ray Rice interview with his wife, Janay, on the Today Show addressing a domestic violence incident. I will not provide an in-depth analysis of this student text using SFL, though that would certainly be possible. Instead, I am interested in a more holistic rhetorical analysis and the effects learning a metalanguage has on student writing. First of all, I am surprised at the high level of detail and description in the analysis. The student vividly describes the participants, their mannerisms, gaze, as well as the textual features and arrangement of the screen. Secondly, the student uses the Aristotelian appeals and visual toolkit in creative ways. As I mentioned in Paper 3, students do not always uses the metalanguage in the “correct” way, for instance, saying Janay Rice is the Pathos of the image, yet students’ utilization of the metalanguage is always illuminating. The in-depth analysis is not just indicative of one student. All students that learn the metalanguage begin producing in-depth descriptions and analyses, which backs up the research that learning a metalanguage increases students’ powers of noticing (de Oliveira and Schleppegrell 34). While a rhetorical analysis is definitely necessary for students to analyze purpose, context, audience, etc., learning a metalanguage facilitates in-depth analysis of the complementarity of multimodal images:
The image used in the article “Janay Rice – Naughty or Nice” by Rachel Blitchington is of Ray Rice and his wife Janay being interviewed by the Today Show about his recent display of domestic violence. Janay is on the left side. She is wearing an orange shirt, her hair is down and slightly curled and her expression shows that she is somewhat distant from the interview, almost as if she was remembering something. Ray is on the right side. He is wearing a red shirt. He looks as if he is engaging in conversation with the interviewer. In front of both of them, a blue headline is stretched across the scene. In white lettering the headline states, “Ray Rice Speaks Out: NFL player on elevator incident and wife.” Inside the blue banner to the right is the Today Show’s logo ad slightly underneath that is the time. Underneath the blue headline there is a thinner white headline that states, “Boston marathon bombing suspect filed another request,” a completely unrelated topic. Above the blue headline there is a very thin orange banner with the words “Today Exclusive.”
Ray’s wife, Janay creates a vector through her gaze. She is not looking directly at the camera and through Logos, we can infer that she is embarrassed or even scared because of her facial expressions. She is the Pathos of this image. A domestic violence victim appeals to the public’s emotions because we sympathize with what she has gone through. Ray himself also creates a vector through his gaze. The goal of his vector is most likely the person who is interviewing him. Neither of the people shown in the image is creating a demand, they are not demanding the viewer’s attention. The reactor of this image would be the public, they are reacting to the news of domestic violence, especially because he is a NFL football player and someone who is, or was a respected member of the community. There is low modality in this image because it is in color, which shows little truthfulness. The Today Show would be showing Ethos, they are gaining credibility by having the perpetrator himself on their show.
Conclusion
At the moment, I do not see SFL or multimodal social semiotics holding much sway within English Studies. Although Kress has become very influential and authoritative within English Studies, his work within SFL has not translated to the composition community, perhaps, because composition is still operating within a Chomskyan formal approach to grammar. SFL and multimodal social semiotics have become extremely popular through much of Australia, Asia, and Europe, however. SFL has begun to make in-roads into public education in the U.S., as the literacy requirements of Common Core are demanding, and SFL offers cheap and functional solutions. The integration of SFL in the U.S. is due to a handful of researchers in genre pedagogy, such as Mary Schleppegrell. Genre Pedagogy Across the Curriculum: Theory and Application in U.S. Classrooms and Contexts is the first book to deal with a genre approach to language instruction in U.S. classrooms.
As new media becomes the dominant form of text, researchers in the U.S. will need to develop more responsive approaches to new media, not only focusing on production, but interpretation as well. As the need for interpretative frameworks is realized, SFL will be there as an already established, internationally recognized metalanguage for analyzing new media. As an added benefit, researchers in the U.S. can begin exploring functional approaches to language, breaking the stranglehold that Chomskyan, formal linguistics has had on the U.S. for half a century.
I recently had a conversation with Dr. Joanne Scheibman, a professor in the Applied Linguistics program at Old Dominion University, in which we discussed functional approaches to language. Though not a Hallidayan, Dr. Scheibman’s functional approach overlaps with Halliday in many ways. Her research, as well, differs from the “major Chomskyan paradigm.” She does not see humans as having a specific language acquisition device, “elevating a biological or built in component” to language learning. Instead, language learning is “usage based,” meaning structures arise out of social contexts and interaction. I was able to see many analogies in Dr. Scheibman’s work as a discourse linguist to work in multimodal social semiotics. Analyzing the minutiae of conversations is similar to analyzing the smallest details of multimodal texts to come to a better understanding of the social structures, which shape texts.
Works cited
De Oliveira, Luciana C., and Joshua Iddings, eds. Genre Pedagogy Across the Curriculum. Bristol: Equinox, 20
De Oliveira Luciana C., and Mary Schleppegrell. Focus on Grammar and Meaning. Oxford: Oxford UP, 2015. Print.
“Dr. Joanne Scheibman.” Online interview. 8 Oct. 2015.
Kress, Gunther R., and Theo Van Leeuwen. Reading Images: The Grammar of Visual Design. 2nd ed. London: Routledge, 2006. Print.
Schleppegrell, Mary. The Language of Schooling: A Functional Linguistics Perspective. Mahwah, NJ: Lawrence Erlbaum Associates, 2004. Print.